RE: [logs] Log Analysis (of Security Devices)

rgerhardsat_private

Shane,

We have started working on a related issue just at the begining of last
week. I today have put some of our findings into a paper. This paper is
available at

   http://www.monitorware.com/en/workinprogress/needle-in-haystack.asp

Please be warned that it is work in progress and definitely unfinished.
I thought twice before posting the link (and thus somehow publically
announcing it, as some on the list might like to point out ;)). But I
would really like to receive some feedback from all the experts on this
list. Even with the few information so far contained in it, I feel there
is enough to argue against. And I have written down some very basic
facts. If we do them wrong, the rest of our ideas can *never* be
right...

As such, I would deeply appreciate feedback on it.

If you have followed this list the past few weeks, you might also see
the reasoning behind many of my postings. For example, the recent
references posted were created in order to support the log analysis
algorithm.

Rainer

> -----Original Message-----
> From: Shane Amante [mailto:shaneat_private] 
> Sent: Wednesday, February 26, 2003 2:47 AM
> To: loganalysisat_private
> Subject: [logs] Log Analysis (of Security Devices)
> 
> 
> I'm curious what algorithms people are using to digest their 
> log files 
> in search of patterns, or other "interesting events", specifically as 
> it relates to firewall or NIDS devices?
> 
> Most of the tools that I've seen, mainly to analyze firewall logs, 
> output results in descending order of the frequency of individual 
> messages, or "attacks".  A simple example would be, assume 
> that in the 
> course of 24 hours src IP A launches 1,000 packets toward a single 
> dest-port on one of my servers (all get dropped); also during that 24 
> hours src IP B launches 200 packets toward a single dest-port 
> on one of 
> my servers (again, all get dropped); the resulting 
> loganalysis program 
> ranks src A highest, src B second highest, etc.  Although that's one 
> way of looking at the data, I'm interested in more sophisticated 
> analysis that covers other dimensions, specifically: time, 
> distribution 
> of src IPs + ports, distribution of attacks from the same src IPs + 
> ports over time, distribution of dest IPs + ports, 
> distribution of dest 
> IPs + ports over time, etc.
> 
> The end goal would be to spot attacks, or precursors to attacks, that 
> would otherwise get lost in the "noise" of less sophisticated 
> analyses 
> programs.  What are useful methods/algorithms/tools people 
> are using to 
> do this?  Or, do people not lose sleep at night worrying 
> about this :-) 
> ?
> 
> -shane
> 
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysisat_private 
> http://lists.shmoo.com/mailman/listinfo/logana> lysis
> 
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis