RE: [logs] Re: Logging: World Domination

john.sanbornat_private

Just some thoughts from a non-programming syadmin ... 

Could some kind of tag be used that specifies which defined log format is
being used?  Then there is no question of the formatting.  And maybe a
separate tag that indicates that this single field is an exception to the
specified format, possibly including the field or 'tag' name.

Only if there is no formatting tag, would you need to worry about trying to
analyze the data.

Of course, then the big debate becomes, how to define the formats, but that
should only be a 'minor' argument.

John

-----Original Message-----
From: Bennett Todd [mailto:betat_private] 
Sent: Monday, September 09, 2002 8:59 AM
To: loganalysisat_private
Subject: [logs] Re: Logging: World Domination

(sorry about the late followup, just back from vacation)

2002-08-23-13:44:50 wolfgangat_private:
> Forget about XML for the moment, that's a secondary issue. The first 
> choice to be made is between a "tagged" format and a "defined" format.

A superb point, I agree entirely.

> The main difference, as I see it:
> - With "defined" format logs the developer of a "foo" application has
>   to find out that his application belongs to the "bar" group and 
>   therefor logs the timestamp as the 3rd token in a white-space separated
>   list. Your log parser has to know the log syntax of the "bar" group as
>   well to make any sense of the logs.

Actually, I think everyone advocating a defined format favours a set of
fixed fields common to all log records, always in the same place (at the
front of the record); these would include the timestamp. Also the
originating host. But if you replace "timestamp" with some category-specific
field your point remains valid.

> - With a "tagged" format, the developer of a "foo" application has to
>   know which tag to use for a timestamp. The log parser doesn't have
>   to know anything about "foo" or the "bar" group of applications.

Yes and no; if using a tagged format the developer has independant freedom
to specify their own tags or values or whatever, the log parser does need to
know about each developer's choices; and if they don't have that freedom,
then I think the advantage of a tagged format disappears, no? Whoops, I
maybe think I've noticed the point here; I retract the above, a pure
_parser_ wouldn't need to know the tags --- but an analyzer would. But then
too, a pure _parser_ for a defined format wouldn't need to know anything
special about the individual categories, it just wouldn't be able to do
anything intelligent with them unless it did --- just like with the tagged
format.

> So IMHO the "defined" format is all fine and well if you want to build 
> a logging infrastructure yourself for your own environment. But if we 
> try to define something that can be shared by people that don't know 
> anything about each others environment, then a "tagged" format is the 
> only workable solution.

If on the third hand we want to create a specification to allow us to build
a collection of platform-independant log analysis expertise, in the form of
portable code, then we really must be specifying the stuff that we want to
be able to portably analyze; flexibility here allowing individual developers
to hack off in their own directions defeats that purpose, no?

-Bennett
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis