On Fri, Sep 28, 2001 at 10:11:56PM -0400, emf wrote: > So... What kinds of data reduction strategies are people using for their > logs? Lets say we have a database full of various types of events (IDS > events, syslog messages, pacct data, and other assorted whatnot) what kinds > of things are people doing to collapse this stuff over time and retain > meaning, or are we all doomed to racks of disk servers and sql queries that > take ever-longer to complete? The answer to this depends on what you want to do with the data later. If it's just trending, and you know the variables on which you want to trend, rrdtool-style summarization is a very good solution. For anamoly training, your data reduction is going to be directly dependent on the training methods you use. (Terran Lane at CERIAS has written some really good papers discussing some of the issues with data reduction for training sets, that should be available at CiteSeer; there are also some other good papers in the astronomy community that can be extrapolated to this type of use.) For being able to go back and look for anything that you missed, unfortunately, you really can't do much reduction--you can just move the data offline and compress the heck out of it until you need to look at a particular set; if you can define an event model that makes sense for your environment, you could instead move the historical data to an OLAP-style DB, which may or may not save you space (depending on how bad your denormalization gets). > Is this a problem someone's solved in a generic enough way that there's > stuff I can just go download and install, or are we all still cobbling > together our own little sets of widgets? I'm not aware of any product that deals with this in this context; there are a few tools I've read about (but not used) for reduction of astronomy data, but they don't really seem like they are flexible enough to be kluged into working on log data as such (unless someone can think of a way to map log data into a FITS export file). <blatant plug>There are also a couple of companies that provide services like this, including mine; I'm in the process of setting up a partnership with another company that might lead to our writing some commercial apps to deal with things like this. If there's any interest in such an app, please contact me off-list.</blatant plug> -- Sweth. -- Sweth Chandramouli ; <svcat_private> President, Idiopathic Systems Consulting
This archive was generated by hypermail 2b30 : Sun Sep 30 2001 - 10:54:30 PDT