[logs] Re: Data mining

devdas@private

On 18/06/06 22:08 +0200, Stefano Zanero wrote:
> Devdas Bhagat wrote:
> > Does anyone have suggestions for data mining of logs for security
> > issues?
> > 
> > I am looking at a few gigabytes of daily logs (about 1.5 terabyte/month)
> 
> Could you send on or offlist a sanitized sample ? To let us know what
> you are looking at.
> 
Postfix logs. Slightly modified versions of what goes below:

Jun 19 00:56:54 dvb postfix/smtpd[659]: connect from mail.iocaine.com[209.169.14.18] 
Jun 19 00:57:01 dvb postfix/smtpd[659]: 437E566CA4: client=mail.iocaine.com[209.169.14.18]
Jun 19 00:57:02 dvb postfix/cleanup[662]: 437E566CA4: message-id=<20060616184441.GA11825@dvb> 
Jun 19 00:57:02 dvb postfix/qmgr[1980]: 437E566CA4: from=<loganalysis-bounces+devdas=dvb.homelinux.org@private>, size=3245, nrcpt=1 (queue active)
Jun 19 00:57:02 dvb postfix/local[663]: 437E566CA4: to=<devdas@private>, relay=local, delay=5.5, delays=5.3/0.06/0/0.14, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail)
Jun 19 00:57:02 dvb postfix/qmgr[1980]: 437E566CA4: removed
Jun 19 00:57:02 dvb postfix/smtpd[659]: disconnect from mail.iocaine.com[209.169.14.18]

Jun 18 05:10:28 dvb postfix/smtpd[28271]: NOQUEUE: reject: RCPT from
20158140252.user.veloxzone.com.br[201.58.140.252]: 550 5.1.1 <3cdevdas@private>: Recipient address rejected: User unknown in local recipient table; from=<arbd@0-0.com> to=<3cdevdas@private> proto=SMTP helo=<20158140252.user.veloxzone.com.br>

I am looking to be able to match excessive connections, figure out
dictionary attacks, spam/virus signatures, stuff like dynamic IP
addresses sending mail ....
(The reject message above is a virus infected host).

Once this is done, I want to tie it into user spam reports as well. And
then finally build a feedback system which feeds into our automated
feedback loop. The final goal is to get spam rejection up from 90% at
the edge to somewhere close to 98%+, identify worm infections and get
those hosts off the network as fast as possible (via admin action).

The above unsanitised logs are from my personal host, not the actual work
logs.

Devdas Bhagat
_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis