Re: [logs] most popular reports...?

download@private

I agree!  I used to consider my hourly cron job messages to be "not 
interesting" until I realized that I could use them to find NTP problems!  
(if the hourly cron job fires at an odd time, there is most likely an NTP 
problem on that host or on the syslog host, or a serious load on the host 
thats preventing it from running the cron job on time).  Now i'm very 
interested in my regular cron job messages.

I basically look for uninteresting messages in the set of messages that 
have no interesting variables.  Eg. if the time is not something I find 
interesting (eg. a message that could happen at any time of the day), then 
that field can be ignored for that message type.  If you can say that none 
of the fields could be interesting, then the message is uninteresting.

I also like to tell myself that my log analysis configs represent what I
am currently interested in.  I expect that the set of interesting messages
will change depending on what problem i'm trying to solve.  So, i'm not 
filtering because there is no value in the message, but rather because I 
don't need any of the potential value of that message to solve the problem 
i'm working on.

Jim

On Thu, 19 Aug 2004, Jian Zhen wrote:

> Just curious, how do you determine the stuff that you are *not* interested in?
> It seems to be a fairly subjective exercise and may result in losing 
> important data. (genuine question, not flame bait :)
> 
> As an example, deciding all accepts on a firewall are *not* interesting
> may not be valid. 
> 
> I do agree that some process of elimination is necessary for log analysis,
> where this happens is the question.
> 
> <RandomThougths>
> 
> It seems like in most real-world cases, log analysis is triggered by some
> stimuli, e.g. an alert (IDS, SIM, human) or a log report (text or graphical 
> format) showing something interesting. Most sysadmins are probably too busy
> to consciously go and review logs unless something happens.
> 
> It also seems like most of the time, the process of correlation is in 
> reverse chronological (temporal) order, as in, you go backwards in order 
> to find out what happened (when did the attack occur, where did it come from,
> how did it occur). Sometimes you will go forward in time to find out if other 
> similar incidents has happened or if the attacker used the host for other
> attacks.
> 
> There's also a spatial component to the process of correlation as well.
> We generally look at logs from multiple hosts/devices across the network
> in order to figure out the root cause.
> 
> Almost all of the SIM vendors are implementing their systems based on
> forward temporal correlation, this is probably useful in detecting some
> of the more well known scenarios. If the scenario is not known, most 
> likely the attack will be missed.
> 
> For most investigations, it seems like reverse spatial & temporal analysis
> is more useful.  Pro'ly less processor and memory intensive. However, 
> (hey! STRACE = Spatial & Temporal Reverse Analytics & Correlation Engine)
> 
> The question is can something like that built to perform the reverse
> correlation automatically (or if it's useful at all). There probably has 
> to be some rules to tell the engine what to look at next (in reverse).
> 
> </RandomThougths>
> 
> Tim Sailer (sailer@private) [040819 12:30]:
> > The way I've approached it, in my own worthless way, is to tell the tools
> > the stuff that I'm *not* interested in. Data reduction is a must in a
> > large centrally logging environment. You then pick through the rest
> > to find the good nuggets. Approaching logs looking at *all* the data is
> > simply a way to lose sleep/hair/braincells. You can then apply the
> > fancy tools to visualize the important stuff, and, by default, the 
> > stuff you never expected to see, since you didn't way you didn't care
> > about it.
> > 
> > Tim
> > 
> > -- 
> > Tim Sailer <sailer@private> 
> > Information and Special Technologies Program
> > Office of CounterIntelligence 
> > Brookhaven National Laboratory  (631) 344-3001
> > _______________________________________________
> > LogAnalysis mailing list
> > LogAnalysis@private
> > http://lists.shmoo.com/mailman/listinfo/loganalysis
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysis@private
> http://lists.shmoo.com/mailman/listinfo/loganalysis
> 

-- 
James E. Prewett                 "everything that is, that was, was not enough"
Systems Team Leader                                                505.277.8210
Designated Security Officer                download@private Jim@private
HPC Systems Engineer III @ HPC@UNM             OpenPGP key: pub  1024D/31816D93

_______________________________________________
LogAnalysis mailing list
LogAnalysis@private
http://lists.shmoo.com/mailman/listinfo/loganalysis