> > As such, I intend to do the > > conversion so that I can design a single, generic back-end > log parser > > and display system... > > Nice but reality often intrudes, I don't expect one thing > will do everything forever. Strongly agree. That's why I said I hope to reach 50% in the first place and maybe 80% in the long run. I just hope that I will be able to provide at least a working basic solution, whith some customer code on top. > The apache example is good, sometimes KISS has value. If > we went to structured logging we'd have quite a bit of > overhead in dealing with the 3TB of apache logs we collect > and analyse monthly Definitely, there is no "one size fits all". But the Apache sample shows also how easily they could be integrated into the log structure: "just" take the log entries, they are already well defined. Then convert this to a name/value pair (which it essentially already is, just in some other representation). Feed this into the backend. Sure, you now need the ability to process these 3 TB(now probably 5 TB), but this is a general issue - router logs can become even heavier... > The other stuff that we usually syslog is minute by > comparison but more diverse and so worth genericising to > simplify handling Yes, this is the whole point. I think we are not trying to rule the app programmers (which wouldn't work anyway). We are just looking for some structure. I honestly believe that 90% to 98% of the syslog emitting programs and devies will NOT follow it, at least in the short term. BUT: we can create converters and funnel those messages to the tagged format. Than the analyzer people can focus on that format. After all, the standardization of web logs brought all this fine analyzers. Or look how NetIQ (relatively successful) tries to force vendors to use their WELF format... (but I don't intend to start a dicsussion on the NetIQ success with this approach). Bottom line is that you need some common format to do any intelligent analysis at all. Current log analysers often create this common format in in-memory tables, but they create it anyhow... Rainer _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Wed Jan 15 2003 - 06:15:44 PST