Since the list has been recently pretty quiet, perhaps it's time to start up a lively debate.... So... What kinds of data reduction strategies are people using for their logs? Lets say we have a database full of various types of events (IDS events, syslog messages, pacct data, and other assorted whatnot) what kinds of things are people doing to collapse this stuff over time and retain meaning, or are we all doomed to racks of disk servers and sql queries that take ever-longer to complete? One thought that's been rattling around in my head is to just collapse things into well understood events and shove them into rrd [1] files (login events, logins by user, cmd.exe IDS probes, portscans, so on and so on.....) for trending, but there's always the chance I'll want to do more with the data at some point; like using it to train an anomaly-bot, or go back and find things that I'd missed in the past... That, and maintainence of "well understood events" is going to be a big pain. Is this a problem someone's solved in a generic enough way that there's stuff I can just go download and install, or are we all still cobbling together our own little sets of widgets? [1] rrdtool & cricket, for the random few that haven't run into this yet, if there are still sysadmins that don't know what it is. Go see sourceforge. -- .emf.
This archive was generated by hypermail 2b30 : Sat Sep 29 2001 - 14:07:09 PDT