[logs] Logging format: Defined vs. Tagged

ron.ogleat_private

> The main difference, as I see it:
> - With "defined" format logs the developer of a "foo" application has
>   to find out that his application belongs to the "bar" group and 
>   therefor logs the timestamp as the 3rd token in a 
> white-space separated
>   list. Your log parser has to know the log syntax of the 
> "bar" group as
>   well to make any sense of the logs.
> - With a "tagged" format, the developer of a "foo" application has to
>   know which tag to use for a timestamp. The log parser doesn't have
>   to know anything about "foo" or the "bar" group of applications.
> 
> So IMHO the "defined" format is all fine and well if you want to build
> a logging infrastructure yourself for your own environment. But if we
> try to define something that can be shared by people that don't know
> anything about each others environment, then a "tagged" format is the
> only workable solution.
> 
> Wolfgang

A good discussion topic, "defined" vs. "tagged".  We can better discuss the
merits of each type.  As always, each has good and bad points.  Maybe we can
quickly surmise the pros and cons of each format?

First a summarization of the requirement:
Provide a standard by which applications can specify information to a
logging mechanism.

The two proposed general formats: defined and tagged.  Specify the pros and
cons for each format with no particular order.

Defined
Pros:
1.  Backward compatible with current syslog.
2.  Compact format (less resources utilized for the same information
content)
3.  Well specified defined formats for many common applications
4.  Many tools available for parsing and tokenizing content
5.  Application programmer doesn't need to know the underlying log format
6.  Can be implemented with minimal changes to current logging systems
7.  Tokenizing of the information can be off loaded to the log server
8.  Can take advantage of new transports without affecting OS level features

Cons:
1.  Not all applications have a well defined format
2.  ASCII based
3.  Many custom solutions for parsing into a database

Tagged
Pros:
1.  Gives programmers more flexibility with logging (potentially)
2.  Supports more structured logs without the cost of defining many
"defined" formats
3.  Could support multiple human languages

Cons:
1.  Extended format (utilizes more resources for the same information
content)
2.  Not backward compatible with current logging systems and libraries
3.  Newer technology, not as well tested or supported in organizations
4.  Requires new infrastructure support on clients (ex. new libraries that
must be used by application developers).
5.  Depending upon implementations, may require clients to be updated with
new tags to define information on a continuing basis.
6.  Tokenizing of the information must be completed on the client.

As I see it, the "defined" format could be quicker to implement and provide
many of the same benefits as "tagged" with a lower cost.

Ron Ogle
Rennes, France
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis