[logs] Data Reduction Strategies?

From: emf (emfat_private)
Date: Fri Sep 28 2001 - 19:11:56 PDT

  • Next message: Tina Bird: "RE: [logs] Windows based Monitoring Tool"

    Since the list has been recently pretty quiet, perhaps it's time to start
    up a lively debate....  
    So... What kinds of data reduction strategies are people using for their
    logs?    Lets say we have a database full of various types of events (IDS
    events, syslog messages, pacct data, and other assorted whatnot)  what kinds of
    things are people doing to collapse this stuff over time and retain meaning, 
    or are we all doomed to racks of disk servers and sql queries that take
    ever-longer to complete?
    One thought that's been rattling around in my head is to just collapse things
    into well understood events and shove them into rrd [1] files (login events,
    logins by user, cmd.exe IDS probes, portscans, so on and so on.....) for 
    trending, but there's always the chance I'll want to do more with the data 
    at some point; like using it to train an anomaly-bot, or go back and find 
    things that I'd missed in the past...
    That, and maintainence of "well understood events" is going to be a big 
    Is this a problem someone's solved in a generic enough way that there's 
    stuff I can just go download and install, or are we all still cobbling 
    together our own little sets of widgets?
    [1] rrdtool & cricket, for the random few that haven't run into this yet, 
    if there are still sysadmins that don't know what it is.   Go see sourceforge.

    This archive was generated by hypermail 2b30 : Sat Sep 29 2001 - 14:07:09 PDT