Re: [logs] Syslog payload format

avalonat_private

I was kind of hoping someone would come back with this kind of email :)

In some mail from Balazs Scheidler, sie said:
> 
> On Sun, Dec 22, 2002 at 03:10:37PM +1100, Darren Reed wrote:
> > In some mail from Balazs Scheidler, sie said:
> > > 
> > > xnewsyslog(LOG_DAEMON|LOG_DEBUG, "debug: %(user) %(tty) from %(host)",
> > > 	   "marcus", tty, where);
> > [...]
> > > That is, a constant event description comes first without _any_ variable
> > > data, semicolon and a list of tag/value pairs. This makes the message easy
> > > to read by humans (the description itself is not a tag), and the event is
> > > still easily parseable. So the above xnewsyslog() call would become
> > > something like this:
> > > 
> > > xnewsyslog(LOG_DAEMON | LOG_INFO, "User logged in; %(user), %(tty), %(host)",
> > > 	"marcus", tty, where);
> > 
> > I just realised there's a "problem" with both of these messages, and
> > this API if something like XML is going to be used.
> > 
> > The problem is these formats suggest that a message is going to be
> > logged in a manner that is similar to the formatting string and it
> > is not.
> > 
> > The above message would be logged, at best, like:
> > 
> > <event user="marcus" tty="ttyp6" host="ranum.com">User logged in; , ,</event>
> 
> no, it would be logged as:
> 
> <event user="marcus" tty="ttyp6" host="ranum.com">User logged in</event>

And how do you arrive at that ?

Is some amount of arbitrary text following the $(macro) scrubbed from
the text message that gets logged ?

I know you as a human can look at the input string and arrive at what
you want the output to be, but are you going to write a logging interface
that's intelligent enough to do that ?  How then do I over-ride it when
I find your interface is removing information that I *want* there ?

Let me rewrite the xnewsyslog() like this:

xnewsyslog(LOG_DAEMON | LOG_INFO,
           "User %(user) logged in. On %(tty) from %(host) on %(date)",
           "marcus", tty, where);

Are you going to build an english parser into xnewsyslog() so it
knows what it can and cannot remove ?  What about for German, French,
Japanese and other languages ?  Is there some other magical way to
get from the above to what you think it means, algorithmicly ?

Modifying the input to the logging interface is bad karma, IMHO.

> which could be represented in a non-XML format for human processing. The
> problem with difficult APIs that they will simply be ignored by programmers.
> If the API is simple enough, the benefits it provides will overweigh the
> lazyness of programmers.
> 
> So my suggestion is this:
> 1) provide a clean API for sending tagged messages
> 2) provide a not-so-clean but easier to use interface based on the first

I think you need a (3) as well:

3) provide a replacement for the current syslog(3) API that produces
   tagged messages.

> my xnewsyslog() proposal is the 2nd case. It is quite easy to use from the
> programmer's viewpoint and still makes it possible to tag message parts.

I don't think you can use xnewsyslog() or syslog() or anything that has an
interface that consists of a formatted string plus args.  It kind of begs
the question why you have the macros at all if you're going to log them
separately.  Why not have it like this:

xnewsyslog(LOG_DAEMON | LOG_INFO, LOG_USERNAME | LOG_TTY | LOG_UTMPHOST,
           "User login");

(and get rid of %m too...)  If you want to allow formatted test in printf
style, you can do that, but the tag stuff should not go in the message
itself or if it is, it is unrelated to the text.  eg:

xnewsyslog(LOG_DAEMON | LOG_INFO, LOG_USERNAME | LOG_TTY | LOG_UTMPHOST,
           "User login by %s on %s from %s", "marcus", tty, where);

The problem here is the limitations on the number of tags inposed
by the number of bit positions that can be set.

In the XML output, you get repeated information but a sensible
message (as well).

In the non-XML output, you get a sensible message and no repeats.

> > User logged in; user="marcus", tty="ttyp6", host=ranum.com
> > 
> > But that's not XML.
> 
> It's a non-XML representation of an event, it is easier to read (for humans)
> and easier to produce (for programmers). It can be transformed to be XML
> easily.

I disagree with the latter.  You can transform it easily but unless
you're going to volunteer to install yourself in the library, I think
making the code that intelligent is another matter altogether :)

> Logging an event should not take more than a single function call,
> otherwise it will be too difficult to use.

Logging an event and building up a description of the event are two
separate exercises for the programmer.  For the former (buliding the
message), it's not that rare now - look at the use of s{,n}printf now
to build up text strings to fire out of syslog(3).

Darren
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis