RE: [logs] what to log/what to look for: stateful log analysis?

toby.kohlenbergat_private

This is definitely the way to go and the thing I really saw as missing from
the previous conversation. There are a bunch of engines that can do this
(for instance any of the major systems management product- HP Openview,
Tivoli,
MasterCell), of course they usually won't translate differently named events
into a standard
nomenclature (which is really necessary when you have lots of products
feeding you
logs) so that a log entry from MS+Exchange about forwarding email looks the
same as
a log entry from SGI+Sendmail or Linux+procmail etc....

Some (free) correlation engines are:
http://www.estpak.ee/~risto/sec/
http://freshmeat.net/projects/ipfc/
http://www.hackertracker.org/cst/ape.tar
http://www.logtrend.org/english/index.shtml

There are tons of commercial ones (many of the products I listed in a
previous email have
some level of correlation engine available). The real trick is the
nomenclature stuff, IMHO.

toby

> -----Original Message-----
> From: Jose Nazario [mailto:joseat_private]
> Sent: Thursday, August 29, 2002 9:36 AM
> To: Anton Chuvakin
> Cc: loganalysisat_private; Tina Bird
> Subject: Re: [logs] what to log/what to look for: stateful 
> log analysis?
> 
> 
> On Thu, 29 Aug 2002, Anton Chuvakin wrote:
> 
> > Thoughts?
> 
> this seems to be where tina was hoping to go: state creation, 
> then state
> changes, then state deletion. ie smtpd is invoked, mail 
> commads are issues
> and the server enters a new state (ie transient ok, give me data then
> "."), message accepted, servce is quit by the client. those are easily
> logged. obviously some log more easily than others having 
> clearly defined
> states they enter and leave, but it shouldn't be too tough a mental
> exercise to think about how to list states of things which run,
> particularily daemons and services.
> 
> to get to what darren reed said in a message that came to my 
> inbox a few
> minutes ago, "look for bad stuff", well, that's of course an 
> obvious goal.
> look for bad stuff: intrusions (attempts, successes and 
> failures), system
> failures, terribly broken stuff (ie core dumps, 
> misconfigurations). but
> *how*? thats what this conversation is about.
> 
> using the state logging that anton was listing above we can 
> now analyze
> behaviors. there's already research into this, its easy to 
> build on that.
> as an example, a paper at this year's USENIX security 
> symposium looked at
> clusters of system calls as a way to detect intrusions:
> 
> 	http://www.usenix.org/events/sec02/liao.html
> 
> so, extending this, we can simply look at a sliding window of 3 or 4
> states and parameters which look ok, then filter those out and look at
> whats left.
> 
> this strikes me as a relatively direct approach to the 
> problem of how to
> analyze the data we log. now to get back to what we log, i agree with
> anton: state changes.
> 
> some examples. ftpd, smtpd, httpd: all of those have well 
> defined states
> they go through. lets look at something which has a loosely 
> logged state,
> login daemons. i say loosely logged because it seems that so 
> many systems
> log this in so many different ways. only when you standardize 
> on one type
> of login daemon (ie sshd) do you see any sort of standardization.
> 
> 	inbound connection request to the system
> 	connection handled by some process (ie sshd)
> 	authorization request by some method
> 	success or failure
> 		do we repeat?
> 	for success, what do we get passed off to?
> 		at what priviledge level?
> 
> 
> run ssh -v your.login.server and you'll see the information, 
> client side,
> listed there. or run sshd -d and you'll see all sorts of 
> debug info logged
> for the daemon (non forking of course).
> 
> just some ideas ... this information can then be processed to look for
> normal patterns and then deviations from this.
> 
> wow, this basically said a lot to agree with anton!
> 
> ___________________________
> jose nazario, ph.d.			joseat_private
> 					http://www.monkey.org/~jose/
> 
> _______________________________________________
> LogAnalysis mailing list
> LogAnalysisat_private
> http://lists.shmoo.com/mailman/listinfo/loganalysis
> 
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis