RE: [logs] Syslog payload format

From: Rainer Gerhards (rgerhardsat_private)
Date: Wed Jan 15 2003 - 01:18:20 PST

  • Next message: Rainer Gerhards: "RE: [logs] RE: syslog/tcp (selp)"

    > > As such, I intend to do the
    > > conversion so that I can design a single, generic back-end 
    > log parser 
    > > and display system...
    > 
    > Nice but reality often intrudes, I don't expect one thing 
    > will do everything forever.
    
    Strongly agree. That's why I said I hope to reach 50% in the first place
    and maybe 80% in the long run. I just hope that I will be able to
    provide at least a working basic solution, whith some customer code on
    top.
    
    > The apache example is good, sometimes KISS has value. If
    > we went to structured logging we'd have quite a bit of 
    > overhead in dealing with the 3TB of apache logs we collect 
    > and analyse monthly
    
    Definitely, there is no "one size fits all". But the Apache sample shows
    also how easily they could be integrated into the log structure: "just"
    take the log entries, they are already well defined. Then convert this
    to a name/value pair (which it essentially already is, just in some
    other representation). Feed this into the backend. Sure, you now need
    the ability to process these 3 TB(now probably 5 TB), but this is a
    general issue - router logs can become even heavier...
    
    > The other stuff that we usually syslog is minute by 
    > comparison but more diverse and so worth genericising to 
    > simplify handling
    
    Yes, this is the whole point. I think we are not trying to rule the app
    programmers (which wouldn't work anyway). We are just looking for some
    structure. I honestly believe that 90% to 98% of the syslog emitting
    programs and devies will NOT follow it, at least in the short term. BUT:
    we can create converters and funnel those messages to the tagged format.
    Than the analyzer people can focus on that format. After all, the
    standardization of web logs brought all this fine analyzers. Or look how
    NetIQ (relatively successful) tries to force vendors to use their WELF
    format... (but I don't intend to start a dicsussion on the NetIQ success
    with this approach). Bottom line is that you need some common format to
    do any intelligent analysis at all. Current log analysers often create
    this common format in in-memory tables, but they create it anyhow...
    
    Rainer
    
    _______________________________________________
    LogAnalysis mailing list
    LogAnalysisat_private
    http://lists.shmoo.com/mailman/listinfo/loganalysis
    



    This archive was generated by hypermail 2b30 : Wed Jan 15 2003 - 06:15:44 PST