RE: [logs] Syslog payload format

From: Frank O'Dwyer (fodat_private)
Date: Tue Dec 17 2002 - 17:40:01 PST

  • Next message: Jason Haar: "Re: [logs] Syslog payload format"

    Bennett Todd wrote:
    > I've yet to see a proposal for a tagged format that struck me as
    > convincing. If we could work out really valuable semantics for such
    > a format --- a good convincing taxonomy of loggable events --- then
    > maybe it'd motivate going to the trouble of introducing a tagged
    > format. Until we see one, though, I'll stick with the current
    > format, fixing only the broken timestamp format, and of course
    > tacking the unreliable and restrictive transport.
    >
    > Back to the payload, the way to guide the transition is to first
    > define your taxonomy.  Then write a converter that reads current
    > syslog data, together with a rules file describing known patterns
    > and their associated classification tags. Extend this rulebase to
    > cover an interesting range of real log types. Build tools that work
    > with it. Demonstrate their value. You'll need that converter almost
    > indefinitely anyway, until the very last embedded-OS gizmo gets
    > converted to the New Way Of Logging.
    >
    > Once you've demonstrated the real utility of the taxonomy, then
    > create a New Logging over-the-wire protocol, a client API, and a
    > companion server.
    
    This is all fair enough, but I think it's worth noting that a large
    proportion of business-significant events are never logged via the syslog
    API in the first place, and if they are ever carried over the syslog
    protocol, it's already going to be via some kind of adaptor/converter. And
    most of that work hasn't happened yet. It's not really a case of defining a
    New Way of Logging, but adapting a number of existing and possibly future
    approaches, only one of which (albeit an important one) is native syslog, to
    a centralised logging infrastructure.
    
    I would hazard a guess that a lot, maybe most, of business-significant
    events currently reside in web server logs, database tables, windows event
    log, and flat files, and the remainder aren't logged at all. The first three
    are pretty structured and machine readable, but all of that structure is
    hard to keep as soon as you try to ship it somewhere using syslog. The 4th
    case is similar to and inclusive of syslog, and for the 5th at least
    backwards compatibility is not an issue :)
    
    For that kind of scenario, syslog (the protocol) only figures as an event
    transport, a means for shipping event data about the place. The main things
    it lacks for that are those you mention (proper timestamp, reliability), but
    again a primary piece of information that is missing is some form of event
    ID. That is needed for routing events to entities that are interested in
    them, for filtering events that aren't of interest, for prioritising, and
    for quickly and reliably identifying the structure of the event data that
    may follow. The program[pid] part of syslog gets you some of the way there,
    and there may be ways to kludge this on top of the existing structure.
    
    > Remember you'll need to retain some kind of
    > backwards-compatibility interop with Syslog Classic more or less of
    > forever.
    
    Yes, but easily done since you could (for example) simply use a different
    port. Also arguably not genuinely as important as all that, since you can't
    assume that a Syslog Classic message will ever arrive, if by classic you
    mean UDP. Plus even if it does arrive it's not terrifically useful, since
    you're lucky if you can (automatically) figure out when it was generated,
    what it represents, whether anyone or anything is interested, where to send
    it, or who to tell. Having laboriously figured all that out, you then can't
    be sure of the event's authenticity.
    
    Cheers,
    Frank
    
    _______________________________________________
    LogAnalysis mailing list
    LogAnalysisat_private
    http://lists.shmoo.com/mailman/listinfo/loganalysis
    



    This archive was generated by hypermail 2b30 : Wed Dec 18 2002 - 12:01:57 PST