[logs] Syslog payload format

rgerhardsat_private

Hi all,

This could be a reply to Frank O'Dwyer's recent posting. I have decided
to give it a new subject, as the original thread seems to have gone a
little bit out of the original scope ;)

The past days, some very valuable discussions have been going on on
syslog protocol. As I see it, two areas were addressed

- the transport format (UDP, TCP, BEEP...)
- the payload inside the packet (the message itself)

I would like to focus on the payload with this posting. Interestingly,
there never has been any standard for the exact format and also
interestingly nobody at the IETF seems to care (right now) about it.

Nevertheless, there have been aproaches to this area. I have compiled a
short list of things I find to be at least related. It is primarily
based on recent postings - but there might be one or two of my
additions. I am listing it, because I definitly would like to see some
(rough) concensus on the format so that we can put it together into a
paper and hope that somebody (on this list) will implement it (a dozen
is even better ;-).

So here comes the list:

- ftp://ftp.rfc-editor.org/in-notes/rfc3339.txt 
  (standardized time stamp format, also an ISO standard). States itself
that THIS time format
  should be used for newer RFCs. Also, there seems to be concensus in
most of the papers below
  that this is the format to go. Thus, I, too, opt for it ;)

- WELF
  A NetIQ proprietary format, but in common use. Details can be found at

http://216.239.39.100/search?q=cache:BO_ZHM7EYCwC:www.netiq.com/partners
/technology/prtnr_welf.doc+welf+format&hl=en&ie=UTF-8 
  (google cache as html). The original spec can be obtained from
  "www.netiq.com/partners/technology/ prtnr_welf.doc" - take care, it is
a word document ;)
  Reassmble the URL to obtain it.
  This is focussed on traffic reports, as that is the focus of the
WebTrends product it
  was designed for. So I don't see it can be used as a general-purpose
format as far
  as the actual tags go. However, it makes an excellent example of what
a Name="Value"
  format looks like. Also, if we'd go for that kind of format, it
definitly wouldn't hurt 
  to use the same tag names welf uses whereever possible.

- Logging Data Map
  http://www.ranum.com/logging/logging-data-map.html an approach to
define tags for common
  entities. I find this one very promising.

- A discussion threat already taken place on the above logging map (plus
more data on it)
  http://lists.shmoo.com/pipermail/loganalysis/2002-August/001089.html

- A current work of the IETF intrusion detection working group
  http://www.ietf.org/internet-drafts/draft-ietf-idwg-idmef-xml-09.txt
  A very complete approach to logging intrusions. In my personal
opinion, this
  is too complex for the fast implementation we intend to have. But I
think there
  are at least some ideas to take from. It also provides a good feel of
what
  an XMLized token format might look like (and how much space it takes
up).

- the DSHIELD format
  http://www.dshield.org/specs.html#dshield_format
  I think it is not the big hit for the format we are looking for, but
it was mentioned
  on the list, so I'd like to include the link.

- A (now expired) Internet Draft on the issue
  http://www.hsc.fr/gulp/draft-abela-ulm-05.txt
  Though it obviously did not receive too much support, I still find it
interesting and useful.
  But again, it was stuck, so this is a good warning sign...

If you look at the references, the usage of well-defined tags seem to be
a good idea and I think there is concensus on this (at least in the
mentioned references). So I think there are two related issues in the
content area:

1. Which tags to use
   This includes the tag name as well as the expected data type (and if
a typed solution should be
   used at all).

2. How to include the tags inside the message
   I just see two choices - either via Name="Value" pairs or via
dumb-simple XML
   in the format of "<name>Value</name>". The "dump-simple" part is
important. I wouldn't
   opt for anything that requires a "real" XML parser.

Having said all this, I ask if

- the list would like to discuss and drive this further, with the goal
to produce one or two papers
- if so, if anybody is interested in writing the papers (I could attempt
at least, but the more,
  the better ;))
- implementors on this list would be willing to implement it (if so,
please make yourself heared!)

Please note that I am not looking for a real standadization process. If
a number of people would agree on the format and at least some products
(Tina, the sentry would be nice ;)) would implement it, it would give us
a good start. Once we have rough consensus on the paper AND running
code, we could give the IETF another try, so it might not be a bad idea
to write the paper in RFC style (as suggested by Tom), which means as an
Internet Draft.

So - in short words: any suport here for doing so? ;)

Rainer
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis