Re: [logs] EventLog library

From: marc (marcat_private)
Date: Mon Jan 06 2003 - 17:41:10 PST

  • Next message: Rainer Gerhards: "RE: [logs] Charset selection (Was: Re: EventLog library)"

    Hello
    
    Well, I have been working in the opposite direction
    (creating a lite idsa version which just implements a very simple
    logging API), and have come to almost the same interface,
    while still remaining *ABI* upcompatible with the full IDS/A
    
    Its source is at 
    http://jade.cs.uct.ac.za/idsa/tmp/idsalite-0.1.tar.gz
    
    The full implementation is in the same directory. Due
    to popular demand, idsalite is released under a BSD license ;)
    
    Its example program looks like this
    
    #include <idsa.h>
    #include <idsa_schemes.h>
    
    int main(int argc, char **argv)
    {
      IDSA_CONNECTION *handle;
      IDSA_EVENT *msg;
    
      handle=idsa_open(argv[0], NULL, 0);
    
    
      /* get an opaque event */
      msg = idsa_event(handle);
    
      /* private / random fields */
      idsa_add_string(msg, "config-file", "/etc/httpd.conf");
      idsa_add_integer(msg, "status", 500);
    
      /* use logging data map tags */
      idsa_add_string(msg, IDSA_LDM_SRCDEV, "192.168.1.3");
      idsa_add_string(msg, IDSA_LDM_DESCRIPT, "Failed GET request due to internal er
    ror");
      idsa_add_printf(msg, IDSA_LDM_PROTO, "HTTP/%d.%d", 1, 0);
    
      /* report an error in error schema */
      idsa_add_string(msg, IDSA_ES, IDSA_ES_INTERNAL);
    
      /* use simple service model */
      idsa_add_string(msg, IDSA_SSM, IDSA_SSM_WFAIL);
    
      /* log the opaque event */
      idsa_log(handle, msg);
    
    
      idsa_close(handle);
    
      return 0;
    }
    
    We really seem close enough to construct a wrapper for both
    which we can put forward as a standard (hey, two implementations
    ;-). Note that for this discussion I am only interested in a
    standard API - since that is what the application programmers
    can then use. Individual logging library/daemon implementations
    can do whatever they want at the representation/transport
    layer until that gets sorted out.
    
    > The library currently supports three event representations:
    > 
    > * plain, the one I was advocating for, with the ';' in it
    > * xmlattr, where log tags are represented as XML attributes of a single XML tag
    > * xmltag, where log tags are represented as XML tags, wrapped in a single XML  
    See above, only interested in API for the time being.
    Idsafull can do almost arbitrary formatting.
    
    > Of course the special characters are escaped as necessary. The only output
    > module currently supported is 'local' which uses the syslog() function in 
    > libc to fire off a message. This might be changed when a syslog() wrapper 
    > will be implemented.
    
    That is what idsalite does at the moment, while idsafull does its own
    transport, using a much saner SOCK_STREAM.
    
    > A couple of features worth noting:
    > - TAG ordering
    >   Although log processing programs will not care about which order tags are
    >   listed, humans are a bit more picky. I assigned a priority to each tag   
    >   which defines the order those tags are written.
    
    This is what I do not have. I think that it is a presentation function and
    shouldn't be a concern at that level. But if you have it that is fine - I am 
    just in search of lowest common denominator. 
    
    > - Implicit tags
    >   The set of implicit tags (e.g. generated automatically) is not burnt in,
    >   you can provide additional functions which add tags to freshly initialized
    >   event records. The current one implemented in the library optionally adds 
    >   these fields to each message:
    > 
    >     PID, PROG, ISOSTAMP, UTCSTAMP, TIMEZONE, MSGID
    
    Not visible at the API level, so not considered. idsafull has its own
    PID, UID, GID, TIME, HOST, SERVICE, SCHEME, NAME, ARISK, CRISK, IRISK, HONOUR
    
    > - TAGS are defined as strings, however a clear roadmap should be provided
    >   how the tag name should be chosen. I suggest a format like this:
    >   'namespace:tagname'. The tags in different namespaces can be standardized
    >   then. The namespace can be omitted in case of often used tags like
    >   PID.
    
    For idsa I am using  .namespace.tagname  
    The leading dot is there to distinguish it from other tags, say
    if it is interleaved with regular syslog messages which use tag=values
    of their own. 
    
    Using a dot would also make subspaces more natural .namespace.subspace.tag
    And if necessary it could do MIBs without ASN.1, which might make some
    people quite happy and others upset ;)
    
    > Here's a sample program which builds an event record, shows it to the user,
    > and also sends it to the log:
    > 
    > #include "evtlog.h"
    > #include <stdio.h>
    > #include <errno.h>
    > #include <stdlib.h>
    > 
    > int
    > main(void)
    > {
    >   EVTREC *e;
    
        IDSA_CONNECTION *handle;
        IDSA_EVENT *msg;
    
    >   evt_open("evtfmt", LOG_AUTH);
    
        handle=idsa_open(argv[0], NULL, 0);
    
    So you are hiding global state from the user. Presumably
    that means thread locking in or around evt_open/log ? For the
    sake of getting a standard wrapper that is ok with me though, 
    I can hide the idsa_connection handle
    
    >   evt_rec_init(&e, LOG_INFO, "Test message with an embedded ';' in it. It also contains an <XML> like tag.");
    
        msg = idsa_event(handle);
    
    Almost the same. Though you are using the old syslog severities. 
    I think mjr proposed an 11 valued field, while I use 3 continuous 
    values in the range -1 to 1, to represent threat to availability,
    confidentiality and integrity. I can put in a message field.
    Though in my implementation the message is just 
    another tag - which I think is cleaner.
    
    >   evt_tag_str(e, 0, "test:tag", "'value'");
    
        idsa_add_string(msg, "config-file", "/etc/httpd.conf");
    
    Same except for priority
    
    >   evt_tag_str(e, 0, "test:tag2", "\n\n\n\n");
    >   evt_tag_int(e, -10, "test:fd", fileno(stderr));
    
        idsa_add_integer(msg, "status", 500);
    
    Ditto. PS: Idsalite doesn't do fancy escapes, idsafull does
    
    >   evt_tag_errno(e, -5, "test:error", EAGAIN);
    
    Idsalite doesn't have this, idsafull does, plus another 11 or so
    types. But for a simple standard, maybe we should stick to strings and
    integers for the time being ?
    
    >   evt_tag_printf(e, 0, "test:printf", "%d %d", 5, 6);
    
        idsa_add_printf(msg, IDSA_LDM_PROTO, "HTTP/%d.%d", 1, 0);
    
    >   es = evt_format(e);
    >   printf("%s\n", es);
    >   free(es);
    
    I don't have that in idsalite
    
    >   evt_log(e);
    
        idsa_log(handle, msg);
    
    >   evt_rec_free(e);
    
    So you have an explicit free. In idsalite I do the free inside
    the log to avoid leaks. But one also wants to re-use events.
    In idsafull one can control if the event needs to be free'ed
    explicitly. For the proposed standard I can implement a do_nothing
    function, but are you sure the risk of leaks is worth the gain ?
    I don't mind either way.
    
    > * tag naming, namespaces
    >   not really the scope of this library but should be documented somewhere
    
    I have tag names in idsa_schemes.h. I have included the logging 
    data map under the namespace .ldm-1. I hope Marcus and Paul don't
    object - I mailed them on Sunday.
    
    > * tag ordering
    >   currently implemented as a number assigned to tags
    > 
    > * character set?
    >   should we support different character sets? I think we should but I'm not
    >   sure.
    
    I think both are not required for the minimal/lowest common denominator
    standard API, as they are part of the "presentation layer".
    
    > * output modules character conversion
    >   should we use a single character encoding on the wire (UTF8?) would it be
    >   mandatory or it should be configurable?
    
    Just one. I prefer plain ASCII ;)
    
    > * how to store the (prio, tagname) tuple?
    >   I assigned a priority to tags to support tag ordering. This is done by
    >   using functions like this:
    >     evt_tag_str(e, 10, "testtag", "value")
    > 
    >   Currently I defined macros like this:
    >     #define EVT_TAG_TESTTAG     10, "testtag"
    > 
    >   which can be substituted in function calls like this:
    >     evt_tag_str(e, EVT_TAG_TESTTAG, "value")
    > 
    >   However I don't like this too much. Maybe a separate struct to specify
    >   tags should be defined, but that would bloat the caller's code.
    
    See above.
    
    > * syslog facility/priority mapping?
    > 
    >   The syslog facility and priority values are currently mandatory, they
    >   might be converted to simple tags, but how to send messages which do not
    >   specify them?
    
    I'd prefer it optional, so that the new API is not permanently tied
    to syslog. Actually the same holds for the second field of evt_open
    
    > Administrative:
    >   * a policy to define tags
    >   * a set of standard tags for different applications
    
    See idsa_schemes.h for suggestions.
    
    > And finally:
    >   * applications, applications, applications
    > 
    > To help the final step, I'm going to convert syslog-ng to emit messages  
    > using eventlog, and also schedule it into our Zorp project.
    
    If we can get the common denominator we will have two implementations, 
    which should be a nice start.
    
    > I think the 'administrative' part is really important to become successful.
    > 
    > I would appreciate some feedback.
    
    regards
    
    marc
    
    _______________________________________________
    LogAnalysis mailing list
    LogAnalysisat_private
    http://lists.shmoo.com/mailman/listinfo/loganalysis
    



    This archive was generated by hypermail 2b30 : Wed Jan 08 2003 - 08:09:16 PST