jamie rishaw <jamieat_private> wrote: > > What (presumedly freeware/opensource/open-dev) programs, home brew > perl madness, commercialware, etc, are people using for their log analysis? > > I think there will be two schools of solutions here, correct me if I'm > missing or off topic: > > > 1) Real-Time monitors to "tail" output and generate alerts/flags based > on certain situations or checkpoints/markers, and I rolled my own a couple of years ago, but this looks a lot better, though I haven't tried it yet: http://www.estpak.ee/~risto/sec > 2) Daily log parsing for reports, trend analysis, and longer term > watching I another roll-my-own: http://www.cise.ufl.edu/~jfh/jst/logstats-0.7.tar.gz Basically a rip-off of Craig Rowland's logcheck written in perl. Using a perl trick from the Perl Cookbook, it's pretty fast. Today it scanned 1.4M lines of log against almost 1700 regular expressions in about 3 hours (Ultra 5 w/9G SCSI Drive), though today was a special case, about half were in the ignore category, which is checked first (a given line goes in the category which has the first matching regexp). The most useful feature is the submatches category, which will match the groups from the regexp. E.g., one of my filesystems filled up, and I got this line in the report: 'keys... ufs:...NOTICE: alloc: /var: file system full' : 10589 : 0.756271% from the regexp (\S+).cise.ufl.edu( ufs:).*(NOTICE: alloc: \S+: file system full) Instead of getting the line 10000 times, I got the number of times that line matched. The array returned by the group matches is joined with "..." when put in the report. I wrote this a couple of years ago, so there may be better stuff out there now, but I still really like it for log reports. Adding the regexps by hand is a bear, but I don't know of a better way to do that. HTH. ---------------------------------------------------------------------- | Jim Hranicky, Senior SysAdmin UF/CISE Department | | E314D CSE Building Phone (352) 392-1499 | | jfhat_private http://www.cise.ufl.edu/~jfh | ---------------------------------------------------------------------- --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Wed Jan 09 2002 - 17:54:54 PST