[logs] syslog data volume management

From: Bennett Todd (bet@private)
Date: Wed Oct 15 2003 - 12:26:32 PDT

  • Next message: Rainer Gerhards: "Re: [logs] WinSyslog Long Syslog Message Denial of Service"

    In an enterprise setting, where you want to forward critical
    messages to a suitable collection of group-management servers, where
    log data is trawled to generate realtime alarms, a recurring problem
    shows up when a client goes berzerk and floods its logserver.
    Sometimes you can solve the problem by using simple stateless
    filtering to drop messages that aren't necessary for alert
    generation, but that doesn't always turn the trick; I've seen
    hardware errors provoke hideous floods of syslogged I/O errors, as
    an example. You want enough to trigger an alarm, but you want to
    shut the flood off fairly quickly, a second or two of it is more
    than plenty.
    I'm thinking about ways to solve this. So far, the cleanest I've
    figured out is to use SEC or something like it to do the data
    reduction, using it's "no more than N msgs that match a given
    pattern within T seconds" style limiting, having it write a named
    pipe, then forwarding whatever shows up there to a syslogd on
    another machine. Since syslog-ng can happily write data for SEC to
    analyze which can then be shoved into a TCP connection to another
    syslog-ng, I think these pieces will fit together without any
    Has anyone else done this sort of realtime data throttling? What
    tools have you used?
    [1] Don't get me wrong, I _love_ to hack, but in a corporate
        setting, it's cheaper to avoid having to maintain in-house

    _______________________________________________ LogAnalysis mailing list LogAnalysis@private http://lists.shmoo.com/mailman/listinfo/loganalysis

    This archive was generated by hypermail 2b30 : Wed Oct 15 2003 - 21:36:50 PDT