-----BEGIN PGP SIGNED MESSAGE----- Lo, Will Partain and the coffee pot sang in unison: > Just an idle possibly-related thought: could any of the > principles of Bayesian spam filtering (quite the rage in > some circles...) be applied to logging? So I was sitting in one of the talks at LISA last week about log analysis, and I had the exact same thought. It could work, and probably even one log line at a time, in real time, albeit probably with a lower statistical confidence than for spam. I tried to find a good starting project from the list at Paul Graham's web site (www.paulgraham.com), but since I was sitting in a dim conference hall thinking about where I was going to get lunch, I didn't follow through very well. I think the problem we'd run into with this approach is building a sufficiently accurate training set. It'd be easy to train the filter so that it could tell us what's unusual, but that's not the question we're asking most of the time. I'd like it to tell me what's mailicious, or indicative of impending hardware failure, or the result of a config file goof, or whatnot. The sets of "unusual events" and "bad events" are perhaps largely overlapping, but they're not identical. Of course, the other problem with the one-line-at-a-time approach is the same as with every other log file parser that just looks at single lines -- no knowledge of context or history. Were the last 200 root login failures followed by a successful one? It's something on my list of potentially interesting projects to look at. Right after I've finished dealing with the spam on our mail server... --rowan - -- John "Rowan" Littell Systems Administrator Earlham College Computing Services http://www.earlham.edu/~littejo/ 2002-11-15 10:54 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (FreeBSD) Comment: Made with pgp4pine 1.76 iQCVAwUBPdUcWpdUNSJ2nf/5AQH/tgQAwEJqP1Hf1UPfxxLotmnQq6iSADr0KSRA vE5kB077HCeR/sf2z1ntBVPk/kNoWAC6JjUN8558wwJySjTGdbKweSbQNYzPNRac YrM82vajIS8zeaY4ZboFFtamU/3zlSUCnFeyo1uj/ipkwewEwEgcJHrWAxhaN5ti x0RnfoRgMso= =CT9r -----END PGP SIGNATURE----- _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Sun Nov 17 2002 - 20:10:44 PST