On Thu, Aug 19, 2004 at 01:25:06PM -0700, Jian Zhen wrote: > Just curious, how do you determine the stuff that you are *not* interested in? > It seems to be a fairly subjective exercise and may result in losing > important data. (genuine question, not flame bait :) How do you lose it? I'm just not including it in the active dataset. > As an example, deciding all accepts on a firewall are *not* interesting > may not be valid. > > I do agree that some process of elimination is necessary for log analysis, > where this happens is the question. Wasn't it on this very list about a year ago, we talked about a tiered logging system? What I've implemented is a 3 (4 in one case that is VERY large) tier system that tier 3 collects all the raw data, and consists of many machines. Scaleable by adding more machines. Syslog-ng is running on the systems, and, through the regex and filter functions, *not* forward things to tier 2. The same thing happens there, and those machines forward to the 1 tier 1 machine, where you would do your basic reporting, etc. Now, on the tier 3 machines, you run something like logsurfer that looks for things, repeated within a time period, like failed logins for root, no matter what machine is logging it, or service looping messages, which, by themselves are usually not serious, but the same service across 100 machines in an hour will tall you something different. Make more sense? If it wasn't on this list that we talked about it, I'll have to go digging to find out where it was. It was a good discussion! Tim -- Tim Sailer <sailer@private> Information and Special Technologies Program Office of CounterIntelligence Brookhaven National Laboratory (631) 344-3001 _______________________________________________ LogAnalysis mailing list LogAnalysis@private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2.1.3 : Thu Aug 19 2004 - 22:01:51 PDT