Hello Well, I have been working in the opposite direction (creating a lite idsa version which just implements a very simple logging API), and have come to almost the same interface, while still remaining *ABI* upcompatible with the full IDS/A Its source is at http://jade.cs.uct.ac.za/idsa/tmp/idsalite-0.1.tar.gz The full implementation is in the same directory. Due to popular demand, idsalite is released under a BSD license ;) Its example program looks like this #include <idsa.h> #include <idsa_schemes.h> int main(int argc, char **argv) { IDSA_CONNECTION *handle; IDSA_EVENT *msg; handle=idsa_open(argv[0], NULL, 0); /* get an opaque event */ msg = idsa_event(handle); /* private / random fields */ idsa_add_string(msg, "config-file", "/etc/httpd.conf"); idsa_add_integer(msg, "status", 500); /* use logging data map tags */ idsa_add_string(msg, IDSA_LDM_SRCDEV, "192.168.1.3"); idsa_add_string(msg, IDSA_LDM_DESCRIPT, "Failed GET request due to internal er ror"); idsa_add_printf(msg, IDSA_LDM_PROTO, "HTTP/%d.%d", 1, 0); /* report an error in error schema */ idsa_add_string(msg, IDSA_ES, IDSA_ES_INTERNAL); /* use simple service model */ idsa_add_string(msg, IDSA_SSM, IDSA_SSM_WFAIL); /* log the opaque event */ idsa_log(handle, msg); idsa_close(handle); return 0; } We really seem close enough to construct a wrapper for both which we can put forward as a standard (hey, two implementations ;-). Note that for this discussion I am only interested in a standard API - since that is what the application programmers can then use. Individual logging library/daemon implementations can do whatever they want at the representation/transport layer until that gets sorted out. > The library currently supports three event representations: > > * plain, the one I was advocating for, with the ';' in it > * xmlattr, where log tags are represented as XML attributes of a single XML tag > * xmltag, where log tags are represented as XML tags, wrapped in a single XML See above, only interested in API for the time being. Idsafull can do almost arbitrary formatting. > Of course the special characters are escaped as necessary. The only output > module currently supported is 'local' which uses the syslog() function in > libc to fire off a message. This might be changed when a syslog() wrapper > will be implemented. That is what idsalite does at the moment, while idsafull does its own transport, using a much saner SOCK_STREAM. > A couple of features worth noting: > - TAG ordering > Although log processing programs will not care about which order tags are > listed, humans are a bit more picky. I assigned a priority to each tag > which defines the order those tags are written. This is what I do not have. I think that it is a presentation function and shouldn't be a concern at that level. But if you have it that is fine - I am just in search of lowest common denominator. > - Implicit tags > The set of implicit tags (e.g. generated automatically) is not burnt in, > you can provide additional functions which add tags to freshly initialized > event records. The current one implemented in the library optionally adds > these fields to each message: > > PID, PROG, ISOSTAMP, UTCSTAMP, TIMEZONE, MSGID Not visible at the API level, so not considered. idsafull has its own PID, UID, GID, TIME, HOST, SERVICE, SCHEME, NAME, ARISK, CRISK, IRISK, HONOUR > - TAGS are defined as strings, however a clear roadmap should be provided > how the tag name should be chosen. I suggest a format like this: > 'namespace:tagname'. The tags in different namespaces can be standardized > then. The namespace can be omitted in case of often used tags like > PID. For idsa I am using .namespace.tagname The leading dot is there to distinguish it from other tags, say if it is interleaved with regular syslog messages which use tag=values of their own. Using a dot would also make subspaces more natural .namespace.subspace.tag And if necessary it could do MIBs without ASN.1, which might make some people quite happy and others upset ;) > Here's a sample program which builds an event record, shows it to the user, > and also sends it to the log: > > #include "evtlog.h" > #include <stdio.h> > #include <errno.h> > #include <stdlib.h> > > int > main(void) > { > EVTREC *e; IDSA_CONNECTION *handle; IDSA_EVENT *msg; > evt_open("evtfmt", LOG_AUTH); handle=idsa_open(argv[0], NULL, 0); So you are hiding global state from the user. Presumably that means thread locking in or around evt_open/log ? For the sake of getting a standard wrapper that is ok with me though, I can hide the idsa_connection handle > evt_rec_init(&e, LOG_INFO, "Test message with an embedded ';' in it. It also contains an <XML> like tag."); msg = idsa_event(handle); Almost the same. Though you are using the old syslog severities. I think mjr proposed an 11 valued field, while I use 3 continuous values in the range -1 to 1, to represent threat to availability, confidentiality and integrity. I can put in a message field. Though in my implementation the message is just another tag - which I think is cleaner. > evt_tag_str(e, 0, "test:tag", "'value'"); idsa_add_string(msg, "config-file", "/etc/httpd.conf"); Same except for priority > evt_tag_str(e, 0, "test:tag2", "\n\n\n\n"); > evt_tag_int(e, -10, "test:fd", fileno(stderr)); idsa_add_integer(msg, "status", 500); Ditto. PS: Idsalite doesn't do fancy escapes, idsafull does > evt_tag_errno(e, -5, "test:error", EAGAIN); Idsalite doesn't have this, idsafull does, plus another 11 or so types. But for a simple standard, maybe we should stick to strings and integers for the time being ? > evt_tag_printf(e, 0, "test:printf", "%d %d", 5, 6); idsa_add_printf(msg, IDSA_LDM_PROTO, "HTTP/%d.%d", 1, 0); > es = evt_format(e); > printf("%s\n", es); > free(es); I don't have that in idsalite > evt_log(e); idsa_log(handle, msg); > evt_rec_free(e); So you have an explicit free. In idsalite I do the free inside the log to avoid leaks. But one also wants to re-use events. In idsafull one can control if the event needs to be free'ed explicitly. For the proposed standard I can implement a do_nothing function, but are you sure the risk of leaks is worth the gain ? I don't mind either way. > * tag naming, namespaces > not really the scope of this library but should be documented somewhere I have tag names in idsa_schemes.h. I have included the logging data map under the namespace .ldm-1. I hope Marcus and Paul don't object - I mailed them on Sunday. > * tag ordering > currently implemented as a number assigned to tags > > * character set? > should we support different character sets? I think we should but I'm not > sure. I think both are not required for the minimal/lowest common denominator standard API, as they are part of the "presentation layer". > * output modules character conversion > should we use a single character encoding on the wire (UTF8?) would it be > mandatory or it should be configurable? Just one. I prefer plain ASCII ;) > * how to store the (prio, tagname) tuple? > I assigned a priority to tags to support tag ordering. This is done by > using functions like this: > evt_tag_str(e, 10, "testtag", "value") > > Currently I defined macros like this: > #define EVT_TAG_TESTTAG 10, "testtag" > > which can be substituted in function calls like this: > evt_tag_str(e, EVT_TAG_TESTTAG, "value") > > However I don't like this too much. Maybe a separate struct to specify > tags should be defined, but that would bloat the caller's code. See above. > * syslog facility/priority mapping? > > The syslog facility and priority values are currently mandatory, they > might be converted to simple tags, but how to send messages which do not > specify them? I'd prefer it optional, so that the new API is not permanently tied to syslog. Actually the same holds for the second field of evt_open > Administrative: > * a policy to define tags > * a set of standard tags for different applications See idsa_schemes.h for suggestions. > And finally: > * applications, applications, applications > > To help the final step, I'm going to convert syslog-ng to emit messages > using eventlog, and also schedule it into our Zorp project. If we can get the common denominator we will have two implementations, which should be a nice start. > I think the 'administrative' part is really important to become successful. > > I would appreciate some feedback. regards marc _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Wed Jan 08 2003 - 08:09:16 PST