[logs] EventLog library

bazsiat_private

Hi,

I have released the first public version of eventlog 0.1 which is available
at 

http://www.balabit.hu/downloads/eventlog/0.1/src/

I have tried to do my best on using consistent naming, simple & flexible
implementation, and although I'm quite sure there are some problems here and
there I am more or less satisfied.

I developed the library on Debian GNU/Linux 3.0 and did not compile it on
anything else, so I'm sure there are some portability problems left.

I have also included some documentation but I am sure it is not perfect now.

The library currently supports three event representations:

* plain, the one I was advocating for, with the ';' in it
* xmlattr, where log tags are represented as XML attributes of a single XML tag
* xmltag, where log tags are represented as XML tags, wrapped in a single XML tag

Of course the special characters are escaped as necessary. The only output
module currently supported is 'local' which uses the syslog() function in
libc to fire off a message. This might be changed when a syslog() wrapper
will be implemented.

A couple of features worth noting:
- TAG ordering
  Although log processing programs will not care about which order tags are
  listed, humans are a bit more picky. I assigned a priority to each tag
  which defines the order those tags are written.

- Implicit tags
  The set of implicit tags (e.g. generated automatically) is not burnt in,
  you can provide additional functions which add tags to freshly initialized
  event records. The current one implemented in the library optionally adds
  these fields to each message:

    PID, PROG, ISOSTAMP, UTCSTAMP, TIMEZONE, MSGID 

  The administrator is free to pick which attributes are to be actually
  used. 

  MSGID is not calculated yet, it is meant to be some kind of checksum of
  the message description. (first 32 bits of the MD5 sum? it seems to be too
  heavy. Does anyone know how msgid(1M) on Solaris works?)

- Configuration file
  The library reads /etc/eventlog.conf where the administrator can customize
  the way the library works. Its format is described in doc/DESIGN.txt

- TAGS are defined as strings, however a clear roadmap should be provided
  how the tag name should be chosen. I suggest a format like this:
  'namespace:tagname'. The tags in different namespaces can be standardized
  then. The namespace can be omitted in case of often used tags like
  PID.

Here's a sample program which builds an event record, shows it to the user,
and also sends it to the log:

#include "evtlog.h"
#include <stdio.h>
#include <errno.h>
#include <stdlib.h>

int
main(void)
{
  EVTREC *e;
  char *es;

  evt_open("evtfmt", LOG_AUTH);
  evt_rec_init(&e, LOG_INFO, "Test message with an embedded ';' in it. It also contains an <XML> like tag.");
  evt_tag_str(e, 0, "test:tag", "'value'");
  evt_tag_str(e, 0, "test:tag2", "\n\n\n\n");
  evt_tag_int(e, -10, "test:fd", fileno(stderr));
  evt_tag_errno(e, -5, "test:error", EAGAIN);
  evt_tag_printf(e, 0, "test:printf", "%d %d", 5, 6);

  es = evt_format(e);
  printf("%s\n", es);
  free(es);

  evt_log(e);

  evt_rec_free(e);
  return 0;
}

The way the message is formatted can be controlled by the /etc/eventlog.conf
file:

---- quote ----
outmethod local
format xmltag
implicit_tags 0x3f
---- quote ----

outmethod: only local is defined, which uses syslog()
format: one of plain|xmltag|xmlattr
implicit_tags: bit fields, look at the source which bit means what

I also faced a couple of problems I either solved or left open. These are:

* tag naming, namespaces 
  not really the scope of this library but should be documented somewhere

* tag ordering
  currently implemented as a number assigned to tags

* character set?	 
  should we support different character sets? I think we should but I'm not
  sure.

* output modules character conversion
  should we use a single character encoding on the wire (UTF8?) would it be
  mandatory or it should be configurable?

* how to store the (prio, tagname) tuple?
  I assigned a priority to tags to support tag ordering. This is done by
  using functions like this:
    evt_tag_str(e, 10, "testtag", "value")

  Currently I defined macros like this:
    #define EVT_TAG_TESTTAG	10, "testtag"

  which can be substituted in function calls like this:
    evt_tag_str(e, EVT_TAG_TESTTAG, "value")

  However I don't like this too much. Maybe a separate struct to specify 
  tags should be defined, but that would bloat the caller's code.

* syslog facility/priority mapping?

  The syslog facility and priority values are currently mandatory, they
  might be converted to simple tags, but how to send messages which do not
  specify them?

What still needs to be done:

Features of the library:
  * a friendlier function interface I've mentioned in one of my earlier
    mails
  * a syslog wrapper
  * more output modules: UDP, TCP, BEEP (BEEP is tricky to implement using a
    blocking scheme though)
  * some optimization here and there (small & many memory allocations for
    example)
  * better configuration parsing
  * portability fixes (configure script is present but is not really useful)

Administrative:
  * a policy to define tags
  * a set of standard tags for different applications

And finally:
  * applications, applications, applications

To help the final step, I'm going to convert syslog-ng to emit messages
using eventlog, and also schedule it into our Zorp project.

I think the 'administrative' part is really important to become successful.

I would appreciate some feedback.

PS: the library is about 1000 lines of code:
      13%    copyright headers
       5%    comments
     15-20%  structure & function definitions

-- 
Bazsi
PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1
_______________________________________________
LogAnalysis mailing list
LogAnalysisat_private
http://lists.shmoo.com/mailman/listinfo/loganalysis