[logs] Tool for Statistical LogAnalysis over time?

From: Mike Blomgren (mike.blomgrenat_private)
Date: Tue Feb 05 2002 - 05:56:55 PST

  • Next message: Frank Heyne: "Re: [logs] NT Logs"

    A while back I asked for tips on syslog daemons for Win2k and received
    great responses. Thanks to all! Now the project has moved on to be more
    administrative and practical. And my dilemma is this: With multiple logs
    from multiple machines, the amount of data is quite large to sift
    through, and perform analysis on. Specifically the Weblogs are around
    1GB in size per day. Doing the analysis once per night is no problem,
    but, and it's a big BUT, how does one reasonably perform the same
    analysis for weekly or monthly reports? Or, heaven forbid, yearly
    reports?
    
    The analysis is done with homemade scripts using perl, and primarily
    serves as a security alerter. Not general visitor statistics, but more
    of an attempt to detect anomalous entries.(Abnormal logins, 404-errors,
    Cookie manipulation, etc etc)
    
    What I'm looking for, is some way to store the results from each daily
    analysis, and be able to reuse these results when creating the weekly
    reports. Similarly to the way WebTrends uses it's FastTrack (I think
    that's the term, anyway) proprietary database.
    
    As I see it, there are four alternatives:
    1) Stor all data and perform the analysis on the 'raw' logfiles. Disk is
    cheap, but time isn't.... A monthly report is likely to take 20-30
    hours. If perl doesn't segfault due to 'out of memory' errors...
    
    2) Re-write some of the analysis in C. But this doesn't solve the
    problem of having to save all the logfiles on-line. Plus I need to learn
    C...
    
    3) Skip the weekly/monthly/whatever reports, and only do a daily
    analysis.
    
    4) Import everything into a database. But which database can handle
    these huge amounts of data? At a reasonable cost? Mysql went berzerk
    after 300MB... Oracle is in the house, but I don't think storing the
    logs in the database is the right way to go - possibly the results of
    every analysis. But, how do you store it in an intelligent way?
    
    Anyway, I figured someone on this list ought to have encountered
    similiar problems. And I'd be very happy to hear from your experience.
    
    Regards,
    
    ~Mike
    
    
    
    CCNOX Security Management & Technology
    Box 5227
    102 45 Stockholm
    Tel: +46 (0)8 545 678 00
    Mob: +46 (0)70 568 12 31
    
    www.ccnox.com
    ________________________________________________
    
    The information included in this e-mail is intended only for the person
    or entity to which it is addressed. Any use of this information by
    persons or entities other than the intended recipient is prohibited. If
    you receive this transmission in error, please delete this email and
    destroy any copies of it. Any opinions expressed in this email are those
    of the individual and not necessarily those of the company CCNOX.
    
    
    
    ---------------------------------------------------------------------
    To unsubscribe, e-mail: loganalysis-unsubscribeat_private
    For additional commands, e-mail: loganalysis-helpat_private
    



    This archive was generated by hypermail 2b30 : Tue Feb 05 2002 - 08:39:31 PST