In an enterprise setting, where you want to forward critical messages to a suitable collection of group-management servers, where log data is trawled to generate realtime alarms, a recurring problem shows up when a client goes berzerk and floods its logserver. Sometimes you can solve the problem by using simple stateless filtering to drop messages that aren't necessary for alert generation, but that doesn't always turn the trick; I've seen hardware errors provoke hideous floods of syslogged I/O errors, as an example. You want enough to trigger an alarm, but you want to shut the flood off fairly quickly, a second or two of it is more than plenty. I'm thinking about ways to solve this. So far, the cleanest I've figured out is to use SEC or something like it to do the data reduction, using it's "no more than N msgs that match a given pattern within T seconds" style limiting, having it write a named pipe, then forwarding whatever shows up there to a syslogd on another machine. Since syslog-ng can happily write data for SEC to analyze which can then be shoved into a TCP connection to another syslog-ng, I think these pieces will fit together without any hacking. Has anyone else done this sort of realtime data throttling? What tools have you used? -Bennett  Don't get me wrong, I _love_ to hack, but in a corporate setting, it's cheaper to avoid having to maintain in-house forks.
This archive was generated by hypermail 2b30 : Wed Oct 15 2003 - 21:36:50 PDT