I use perl to analyze logs and one of the things I always do is is create an extract of all the syslog lines that match the criteria I'm looking for in my analysis. That gives me a much smaller file with just the entries that are relevant to my analysis. One could write a script that globs say, a month's worth of these extracts as input, and do your trend analysis off that. Just an idea. John Campbell, GCWN Information Security Engineer Washington School Information Processing Cooperative (WSIPC) E-mail: jcampbellat_private -----Original Message----- From: Mike Blomgren [mailto:mike.blomgrenat_private] Sent: Tuesday, February 05, 2002 5:57 AM To: loganalysisat_private Subject: [logs] Tool for Statistical LogAnalysis over time? A while back I asked for tips on syslog daemons for Win2k and received great responses. Thanks to all! Now the project has moved on to be more administrative and practical. And my dilemma is this: With multiple logs from multiple machines, the amount of data is quite large to sift through, and perform analysis on. Specifically the Weblogs are around 1GB in size per day. Doing the analysis once per night is no problem, but, and it's a big BUT, how does one reasonably perform the same analysis for weekly or monthly reports? Or, heaven forbid, yearly reports? The analysis is done with homemade scripts using perl, and primarily serves as a security alerter. Not general visitor statistics, but more of an attempt to detect anomalous entries.(Abnormal logins, 404-errors, Cookie manipulation, etc etc) What I'm looking for, is some way to store the results from each daily analysis, and be able to reuse these results when creating the weekly reports. Similarly to the way WebTrends uses it's FastTrack (I think that's the term, anyway) proprietary database. As I see it, there are four alternatives: 1) Stor all data and perform the analysis on the 'raw' logfiles. Disk is cheap, but time isn't.... A monthly report is likely to take 20-30 hours. If perl doesn't segfault due to 'out of memory' errors... 2) Re-write some of the analysis in C. But this doesn't solve the problem of having to save all the logfiles on-line. Plus I need to learn C... 3) Skip the weekly/monthly/whatever reports, and only do a daily analysis. 4) Import everything into a database. But which database can handle these huge amounts of data? At a reasonable cost? Mysql went berzerk after 300MB... Oracle is in the house, but I don't think storing the logs in the database is the right way to go - possibly the results of every analysis. But, how do you store it in an intelligent way? Anyway, I figured someone on this list ought to have encountered similiar problems. And I'd be very happy to hear from your experience. Regards, ~Mike CCNOX Security Management & Technology Box 5227 102 45 Stockholm Tel: +46 (0)8 545 678 00 Mob: +46 (0)70 568 12 31 www.ccnox.com ________________________________________________ The information included in this e-mail is intended only for the person or entity to which it is addressed. Any use of this information by persons or entities other than the intended recipient is prohibited. If you receive this transmission in error, please delete this email and destroy any copies of it. Any opinions expressed in this email are those of the individual and not necessarily those of the company CCNOX. --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Tue Feb 05 2002 - 18:14:35 PST