RE: [logs] Syslog payload format

From: Ogle Ron (Rennes) (ron.ogleat_private)
Date: Mon Jan 13 2003 - 03:25:31 PST

  • Next message: Bennett Todd: "Re: [logs] Trial SELP client implementation"

    It's amazing how Ethernet, IP, TCP, UDP, ICMP, etc. has been able to survive
    with you guys around.  Maybe it could be that computers handle this format
    just fine without all of the gold plating.  If we want humans to read it, we
    create other little programs that do the translating.  BTW, are humans the
    primary recipients of your new format or just another computer which will
    belch out (in a pretty format if you wish) the most important things for a
    human to take action on?
    LOGGING IS A RESOURCE USING ADD-ON that maybe necessary but shouldn't use
    EXCESSIVE RESOURCES.  Most OS developers know this.  It's also amazing that
    no matter what country or language someone speaks or writes; they all
    understand the "events" of Ethernet, IP, TCP, UDP, ICMP, etc.
    Your going to be making a behemoth that can't stand on it's own legs, and if
    it could won't be allowed by system administrators because it takes too many
    resources to propel it.  Trim down this beast if you want it to be used on
    the front end.  A sloth is ok on the back end.
    Ron Ogle
    Rennes, France
    P.S. IMNSHO, I can think of something more stupider.  Several people doing a
    lot of useless work.
    > -----Original Message-----
    > From: Marcus J. Ranum [mailto:mjrat_private]
    > Sent: Wednesday, December 18, 2002 10:39 PM
    > To: Frank O'Dwyer; Rainer Gerhards; loganalysisat_private
    > Cc: tcleary2at_private
    > Subject: RE: [logs] Syslog payload format
    > Frank O'Dwyer wrote:
    > >  log(event);
    > >
    > >Where 'event' is an event object, or something like a 
    > 'struct' if you're a
    > >'C' programmer.
    > Well, what we do now is bad and wrong - typical calls to
    > syslog() look like:
    > {
    >         char    buf[BUFSIZ];
    >         /* never mind overflows, this is an EXAMPLE... */
    >         sprintf(buf,"%s - file %s size %d",host,file,size);
    >         syslog(LOG_NOTICE,buf);
    > }
    > Well, that's just EVIL because here we already have it all
    > parsed out already and we're unparsing it into a string, then
    > handing that to syslog and EVENTUALLY someone tries to parse
    > the results out of syslog. Can you think of anything stupider
    > except using ice made out of distilled water to cool your tea?
    > So what I think we don't do is store it in a structure; it
    > needs to be open-ended.
    > RULE #1:
    >         - An event record is an arbitrary collection of tagged values
    > RULE #2:
    >         - Some tags are "known" public tags, some tags are 
    > application,
    >         site, or event specific - cannot know them all in advance
    >         therefore there must be some flexibility in letting app
    >         designers pick their own
    > RULE #3:
    >         - the API for tossing event records should be EASIER 
    > to use than
    >         the current syslog API and should inherently discourage
    >         implementation of buffer overruns
    > RULE #4:
    >         - defaults should be used wherever possible
    > so how about an event logging API that vaguely resembles
    > something like:
    > #include <eventlog.h>
    > eventlog_open("my program");
    > /* note this sets up a few defaults which may be
    > overridden by eventlog_setdefault() calls. i.e.:
    > EVENTLOG_PID is set to the results of getpid()
    > automatically */
    > eventlog_setdefault(EVENTLOG_PROTO,"ftp");
    > /* EVENTLOG_PROTO is a #define from eventlog.h and is the tag
    > for event log facility - i.e.: "PROTO"
    >  From now on unless an event record SPECIFIES a facility
    > it'll get assigned to whatever we defaulted it to "ftp"
    > because we're doing FTP stuff... */
    > EventRec        *newev;
    > newev = eventlog_new();
    > eventlog_addvalue(newev,EVENTLOG_PATH,pathname);
    > eventlog_addvalue(newev,EVENTLOG_PRIO,4);
    > eventlog_send(newev); /* note time gets added when SENT */
    > eventlog_free(newev);
    > Also have some basic routines that work on a statically
    > allocated default event record...
    > eventlog_clearstd();
    > eventlog_addstd(EVENTLOG_PATH,pathname);
    > eventlog_sendstd();
    > Note that this way you _could_ allocate an event record
    > and if you were logging (say) URLs you could just go in a
    > while(1) {
    >         eventlog_addvalue(newev,EVENTLOG_PATH,url);
    >         eventlog_send(newev);
    > }
    > The stuff that gets burped out the back would simply be
    > an representation of the tags with meta-character translation
    > applied. I.e.:
    > <REC>
    > <PATH>pathname</PATH>
    > <PRIO>4</PRIO>
    > <PROTO>ftp</PROTO>
    > <DATE>standard date</DATE>
    > </REC>
    > This is easy. We're looking at, what, a couple hours of coding?
    > Toss in some options that let you do useful things like tell the
    > subsystem to generate unique event-IDs automatically, etc, etc.,
    > and you're in good shape.
    > >So the first question is what does an event consist of? Things like
    > >generation time, event id, priority, source, human readable message,
    > >forwarding trail, maybe some application-specific payload, 
    > what else? It's
    > >easy to come up with a long shopping list, but what are the 
    > basics that
    > >people could agree on, and which are optional and which are 
    > mandatory?
    > You want to make the number of options small and the core stuff
    > should ALL be defaulted if not provided.
    > date/timestamp (standard)
    > >Once you know the content of the object/struct, you can then 
    > worry about
    > >getting it from A to B, and safely tucked into some log or other.
    > Yah, that's easy. :)
    > >[As an aside, anything you come up with will likely have an 
    > obvious XML
    > >equivalent, but that doesn't mean XML needs to be the 
    > on-the-wire format
    > >(although that or an XML subset is an obvious choice to consider).]
    > A subset of XML is so easy (see above) it's not even funny.
    > Where XML gets silly is when you get into complex nested
    > stuff and DTDs and all that houghmagandy.
    > mjr. 
    > ---
    > Marcus J. Ranum
    > Computer and Communications Security	mjrat_private
    > _______________________________________________
    > LogAnalysis mailing list
    > LogAnalysisat_private
    LogAnalysis mailing list

    This archive was generated by hypermail 2b30 : Mon Jan 13 2003 - 09:35:42 PST