Re: [logs] EventLog library

From: marc (marcat_private)
Date: Tue Jan 07 2003 - 05:20:48 PST

  • Next message: Buck Buchanan: "RE: [logs] Windows Event Log Analysis"

    You wrote:
    > > Its source is at 
    > > http://jade.cs.uct.ac.za/idsa/tmp/idsalite-0.1.tar.gz
    > 
    > gee... ;) another program with a 0.x version number. Darren?
    
    Ha, it is up to 0.2 by now ;)
    
    > > See above, only interested in API for the time being.
    > > Idsafull can do almost arbitrary formatting.
    > 
    > can you be more specific? 'arbitrary' is just too arbitrary for me ;)
    
    Oh, I just mentioned that in passing. The full idsa library
    has a little formatting language - it is internal and rather
    unpolished, but can be thought of as printf with loops. See lib/print.c
    
       struct lookup {
        char *key;
        char *value;
      } map[] = {
        { "csv", "\"%{:1}\"%{1>},\"%{:1}\"%{<}\n"},
        { "ulm", "%{*}=\"%{:1}\"%{1>} %{*}=\"%{:1}\"%{<}\n"},
        { "tulm", "%{*}:%{+}=\"%{:1}\"%{1>} %{*}:%{+}=\"%{:1}\"%{<}\n"},
        { "syslog", "%{time:100} %{host:1} %{service:1}[%{pid}]: %{scheme:1}.%{name:1} %{12>} %{:1}%{<}\n"},
        { "native", "%{time:102} %{host:1} %{uid}:%{gid} %{honour} %{arisk}:%{crisk}:% {irisk} %{service:1}:%{pid} %{scheme:1}:%{name:1}%{12>} %{*}=\"%{:1}\"%{<}\n"},
        { "xml", "<event>%{>}<%{*} type=\"%{+}\">%{:2}</%{*}>%{<}</event>\n"},
        { NULL, NULL}
      };
    
    But I don't think we should aim to standardise that (yet).
    
    > the point is that it is possible to add further transports. I think running
    > a UNIX-like system without local syslogd is an attainable target.
    
    Exactly, and idsad would be an example - hopefully one of several.
    
    > > Not visible at the API level, so not considered. idsafull has its own
    > > PID, UID, GID, TIME, HOST, SERVICE, SCHEME, NAME, ARISK, CRISK, IRISK, HONOUR
    > 
    > I think the first is to define the set of tags and their naming. Later the
    > library can pick which tags it wants to generate automatically. This part of
    > EventLog is meant as an example only.
    
    I agree completely - the automatically generated tags are easy
    to change, and can either be made specific to an implementation
    or changed.
    
    > It is again up to us to specify tag naming. I'd go for XML namespaces
    > instead of inventing our own naming scheme. So either ASN.1 OIDs (booo) or
    > XML namespaces.
    
    But XML can handle tag nesting, which we would have to simulate with
    something.something in which case there are subtle issues anyway. But
    I am open to suggestions on that.
    
    > I was under the impression that a single log 'connection' will be used by a
    > single application, therefore the global state is stored in a single, global
    > EVTCONFIG structure.
    > 
    > Thread safety is a must for me, so it will be added later, but I'd focus
    > more on evt_log() racing with another evt_log() call. (and not evt_open
    > racing with evt_log)
    
    Safety is the major problem. But also high performance (otherwise
    you block all threads), and then library components or dll's which
    want to do their own logging. Maybe tep has some comments ?
    
    > > Almost the same. Though you are using the old syslog severities. 
    > > I think mjr proposed an 11 valued field, while I use 3 continuous 
    > > values in the range -1 to 1, to represent threat to availability,
    > > confidentiality and integrity. I can put in a message field.
    > > Though in my implementation the message is just 
    > > another tag - which I think is cleaner.
    > 
    > The description should be as mandatory as possible. If it was provided as a
    > tag it would be possible to log an event without one.
    > 
    > I think it is difficult to completely forget about syslog severities though.
    > The major transport for these messages will be syslog. We either need a
    > function to map some 'new' priority field to standard syslog priority, _or_
    > we use syslog priority and require the application to define its own tag if
    > it wants more.
    
    I agree that it will be the major one, but if you make it make
    it part of that call the temptation is to make it "special", and
    note how the special status of the priority in "<12>Message..."
    leads to it being lost in the log file. Anyway - see previous mail 
    for my idea.
    
    > The 11 valued field mjr has proposed was quite IDS specific. The ones you
    > use are IDSA specific, and I don't either of you could work with the other's
    > priority definition.
    
    Actually, I like to think my approach can encode anything ;).
    Especially since it has a confidence value, which is useful, since
    most people who don't care about priority under syslog use "INFO", but
    info and don't know have different meaning. Anyway, a topic for
    another discussion.
    
    > > Ditto. PS: Idsalite doesn't do fancy escapes, idsafull does
    > 
    > Escaping is a must, otherwise you cannot warrant well-formedness. I'm for
    > dropping the priority field. It was a bad decision.
    
    Has been added as of idsalite-0.2. But how the escaping is done
    is a transport problem - so not part of the API which I would like
    to reach consensus on.
    
    > > >   evt_tag_errno(e, -5, "test:error", EAGAIN);
    > > 
    > > Idsalite doesn't have this, idsafull does, plus another 11 or so
    > > types. But for a simple standard, maybe we should stick to strings and
    > > integers for the time being ?
    > 
    > I don't think adding more types hurts
    
    Well, in a way I agree, as I have lots of types, but for the sake
    of a simple small standard it makes things difficult as people
    are likely to argue what types they want, and that windows pids
    don't map to unix, etc, etc,
    
    > What are these:
    >
    > int idsa_add_scan(IDSA_EVENT * e, char *n, unsigned int t, char *s);
    > int idsa_add_set(IDSA_EVENT * e, char *n, unsigned int t, void *p);           
    
    The full libidsa has a type associated with each tag (like you
    have priorities ;). Each type has a binary representation, and
    a string representation. These are the two low level handlers for it.
    idsa_add_string is implemented as
    
      idsa_add_set(e, "label", IDSA_T_STRING, value);
    
    idsa_scan allows the conversion:
    
      i=42;
      idsa_add_set(e, "lable", IDSA_T_INT, &i);
    has the same result as
      idsa_add_scan(e, "label", IDSA_T_INT, "42");
    
    I think it is neat to have, but probably beyond a standard, like your
    priorities.
    
    > > >   es = evt_format(e);
    > > >   printf("%s\n", es);
    > > >   free(es);
    > > 
    > > I don't have that in idsalite
    > 
    > This function is needed only if the application wants to perform its own
    > logging but still wants to use the tagged message format.
    
    Nice. But I don't think that should be part of the immediate
    standard, as that presupposes a pretty printing API which can
    be rather large, as above ;)
    
    > > So you have an explicit free. In idsalite I do the free inside
    > > the log to avoid leaks. But one also wants to re-use events.
    > > In idsafull one can control if the event needs to be free'ed
    > > explicitly. For the proposed standard I can implement a do_nothing
    > > function, but are you sure the risk of leaks is worth the gain ?
    > > I don't mind either way.
    > 
    > I think evt_log() should consume the event for the very reason you
    > mentioned. (thus: another change in the API) Event records are reference
    > counted so the application can keep a record if it wants to:
    > 
    > evt_rec_ref(e);
    > evt_log(e);
    
    Ok - so I'll have to update my wrapper too ;). See idsa_template()
    on how I chose to make event data available across logging calls.
    
    > > > * tag ordering
    > > >   currently implemented as a number assigned to tags
    > 
    > this should be dropped.
    
    Cool.
    
    > > > * output modules character conversion
    > > >   should we use a single character encoding on the wire (UTF8?) would it be
    > > >   mandatory or it should be configurable?
    > > 
    > > Just one. I prefer plain ASCII ;)
    > 
    > I'm not so sure ;) But make this a topic discussed later.
    
    Agreed.
    
    > > >   The syslog facility and priority values are currently mandatory, they
    > > >   might be converted to simple tags, but how to send messages which do not
    > > >   specify them?
    > > 
    > > I'd prefer it optional, so that the new API is not permanently tied
    > > to syslog. Actually the same holds for the second field of evt_open
    > 
    > but the primary transport of these messages will be syslog()
    > 
    > should we use a default priority for messages which do not specify syslog
    > priority on their own?
    
    I have this notion of using a very rare priority (see the comment
    in idsalite) so that a log processor can then assume that anything
    on that has the new format (whatever format it may be). Another reason
    for not giving special dispensation to the priority at the API level.
    
    regards
    
    marc
    _______________________________________________
    LogAnalysis mailing list
    LogAnalysisat_private
    http://lists.shmoo.com/mailman/listinfo/loganalysis
    



    This archive was generated by hypermail 2b30 : Wed Jan 08 2003 - 09:22:20 PST