Re: [logs] Re: what is normal ?

From: John Rowan Littell (littejoat_private)
Date: Fri Nov 15 2002 - 08:09:56 PST

  • Next message: Bob Staaf: "[logs] Logwatch vs Logcheck(logsentry)"

    -----BEGIN PGP SIGNED MESSAGE-----
    
    Lo, Will Partain and the coffee pot sang in unison:
    > Just an idle possibly-related thought: could any of the
    > principles of Bayesian spam filtering (quite the rage in
    > some circles...) be applied to logging?
    
    So I was sitting in one of the talks at LISA last week about log
    analysis, and I had the exact same thought.  It could work, and
    probably even one log line at a time, in real time, albeit probably
    with a lower statistical confidence than for spam.  I tried to find
    a good starting project from the list at Paul Graham's web site
    (www.paulgraham.com), but since I was sitting in a dim conference
    hall thinking about where I was going to get lunch, I didn't follow
    through very well.
    
    I think the problem we'd run into with this approach is building a
    sufficiently accurate training set.  It'd be easy to train the filter
    so that it could tell us what's unusual, but that's not the question
    we're asking most of the time.  I'd like it to tell me what's
    mailicious, or indicative of impending hardware failure, or the result
    of a config file goof, or whatnot.  The sets of "unusual events" and
    "bad events" are perhaps largely overlapping, but they're not
    identical.
    
    Of course, the other problem with the one-line-at-a-time approach is
    the same as with every other log file parser that just looks at single
    lines -- no knowledge of context or history.  Were the last 200 root
    login failures followed by a successful one?
    
    It's something on my list of potentially interesting projects to look
    at.  Right after I've finished dealing with the spam on our mail
    server...
    
      --rowan
    
    - -- 
    John "Rowan" Littell
    Systems Administrator
    Earlham College Computing Services
    http://www.earlham.edu/~littejo/
    2002-11-15 10:54
    -----BEGIN PGP SIGNATURE-----
    Version: GnuPG v1.0.6 (FreeBSD)
    Comment: Made with pgp4pine 1.76
    
    iQCVAwUBPdUcWpdUNSJ2nf/5AQH/tgQAwEJqP1Hf1UPfxxLotmnQq6iSADr0KSRA
    vE5kB077HCeR/sf2z1ntBVPk/kNoWAC6JjUN8558wwJySjTGdbKweSbQNYzPNRac
    YrM82vajIS8zeaY4ZboFFtamU/3zlSUCnFeyo1uj/ipkwewEwEgcJHrWAxhaN5ti
    x0RnfoRgMso=
    =CT9r
    -----END PGP SIGNATURE-----
    
    
    _______________________________________________
    LogAnalysis mailing list
    LogAnalysisat_private
    http://lists.shmoo.com/mailman/listinfo/loganalysis
    



    This archive was generated by hypermail 2b30 : Sun Nov 17 2002 - 20:10:44 PST