http://www.internettradecraft.com/sids/ OK, so I mentioned this last week, and said I'd have something over the weekend... which would have been last weekend. Hey, a week late for something I do in my spare time isn't too bad for me. :) Briefly, SIDS is an anomaly detection/log reduction tool. So, I figure this would be a good mailing list for it. A present, what it does is go through an HTTP log file (Roxen right now... see below) and pulls out the less common entries. It does this in the most brain-dead way possible, by counting. The general idea is that if something has been seen 100 times, then it is probably "normal", and can be considered a candidate for no longer alerting on. Over a large enough amount of logs, this eventually ends up spitting out only the lines it hasn't seen before. Please understand that at present, SIDS is incredibly crude, hence the .2 version number. Probably the main things that keeps it from being useful at the moment is the output format, and the lack of ability to track "safe" items between session. Those will both be addressed in the next couple revs. What I would like from the subscribers of this list is some help with log file formats. What I'm looking for is samples from different HTTP log files, so that I can make compatible filters for them, so that less technical people who want to use it will hopefully not have to write their own. So, if you're interested, send me a few lines of your web logs (sanitized is fine, doesn't matter), and I'll write the appropriate config file for your web server. Please tell me what web server you're using too, so I can name it appropriately. Then, I can sent you the config file, you can try it on your full logs, and then tell me how useless SIDS is at present. :) (Actually, what that will do for you is have your log file format ready to go when SIDS is actually useful.) For the future, I'm looking to extend SIDS beyond HTTP logs, so I will also want to look at other log types. Anything that has a very regular single-line format is an immediate candidate. Others I will look at, so I can get ideas on how to handle them. This plays directly into the recent discussion here about how to eliminate unintersting log entries with swatch, and similar ideas. I would tend to think that Tina would want the replies off-list. I can't think what good the samples would do anyone else, but that is ultimately up to the moderator, of course. Oh.. and it's open-source, etc... haven't gotten around to picking a license yet.. but it is hardly important (take a look at how short the code is.) Ryan --------------------------------------------------------------------- To unsubscribe, e-mail: loganalysis-unsubscribeat_private For additional commands, e-mail: loganalysis-helpat_private
This archive was generated by hypermail 2b30 : Mon Aug 20 2001 - 09:42:26 PDT