On Tue, Jan 07, 2003 at 03:41:10AM +0200, marc wrote: > Hello > > Well, I have been working in the opposite direction > (creating a lite idsa version which just implements a very simple > logging API), and have come to almost the same interface, > while still remaining *ABI* upcompatible with the full IDS/A > > Its source is at > http://jade.cs.uct.ac.za/idsa/tmp/idsalite-0.1.tar.gz gee... ;) another program with a 0.x version number. Darren? > > The full implementation is in the same directory. Due > to popular demand, idsalite is released under a BSD license ;) EventLog might also slip to a BSDish license. > We really seem close enough to construct a wrapper for both > which we can put forward as a standard (hey, two implementations > ;-). Note that for this discussion I am only interested in a > standard API - since that is what the application programmers > can then use. Individual logging library/daemon implementations > can do whatever they want at the representation/transport > layer until that gets sorted out. ok. > > > The library currently supports three event representations: > > > > * plain, the one I was advocating for, with the ';' in it > > * xmlattr, where log tags are represented as XML attributes of a single XML tag > > * xmltag, where log tags are represented as XML tags, wrapped in a single XML > See above, only interested in API for the time being. > Idsafull can do almost arbitrary formatting. can you be more specific? 'arbitrary' is just too arbitrary for me ;) > > > Of course the special characters are escaped as necessary. The only output > > module currently supported is 'local' which uses the syslog() function in > > libc to fire off a message. This might be changed when a syslog() wrapper > > will be implemented. > > That is what idsalite does at the moment, while idsafull does its own > transport, using a much saner SOCK_STREAM. the point is that it is possible to add further transports. I think running a UNIX-like system without local syslogd is an attainable target. > > A couple of features worth noting: > > - TAG ordering > > Although log processing programs will not care about which order tags are > > listed, humans are a bit more picky. I assigned a priority to each tag > > which defines the order those tags are written. > > This is what I do not have. I think that it is a presentation function and > shouldn't be a concern at that level. But if you have it that is fine - I am > just in search of lowest common denominator. You only process messages by software but you cannot avoid human reading of the log many times. Tag ordering helps human processing. I also think that adding priority directly to the tag add function is not the best solution. Maybe tags should be registered to an internal hash table with their associated 'priority'. If a given tag is not registered, a default priority is simply assigned. As I think about it more, this should be completely left out, as the message is consistent, it is very easy to reformat the logs. So tag priorities should be dropped. (it seemed a good idea to me ;) > > > - Implicit tags > > The set of implicit tags (e.g. generated automatically) is not burnt in, > > you can provide additional functions which add tags to freshly initialized > > event records. The current one implemented in the library optionally adds > > these fields to each message: > > > > PID, PROG, ISOSTAMP, UTCSTAMP, TIMEZONE, MSGID > > Not visible at the API level, so not considered. idsafull has its own > PID, UID, GID, TIME, HOST, SERVICE, SCHEME, NAME, ARISK, CRISK, IRISK, HONOUR I think the first is to define the set of tags and their naming. Later the library can pick which tags it wants to generate automatically. This part of EventLog is meant as an example only. > > > - TAGS are defined as strings, however a clear roadmap should be provided > > how the tag name should be chosen. I suggest a format like this: > > 'namespace:tagname'. The tags in different namespaces can be standardized > > then. The namespace can be omitted in case of often used tags like > > PID. > > For idsa I am using .namespace.tagname > The leading dot is there to distinguish it from other tags, say > if it is interleaved with regular syslog messages which use tag=values > of their own. > > Using a dot would also make subspaces more natural .namespace.subspace.tag > And if necessary it could do MIBs without ASN.1, which might make some > people quite happy and others upset ;) It is again up to us to specify tag naming. I'd go for XML namespaces instead of inventing our own naming scheme. So either ASN.1 OIDs (booo) or XML namespaces. > > > Here's a sample program which builds an event record, shows it to the user, > > and also sends it to the log: > > > > #include "evtlog.h" > > #include <stdio.h> > > #include <errno.h> > > #include <stdlib.h> > > > > int > > main(void) > > { > > EVTREC *e; > > IDSA_CONNECTION *handle; > IDSA_EVENT *msg; > > > evt_open("evtfmt", LOG_AUTH); > > handle=idsa_open(argv[0], NULL, 0); > > So you are hiding global state from the user. Presumably > that means thread locking in or around evt_open/log ? For the > sake of getting a standard wrapper that is ok with me though, > I can hide the idsa_connection handle I was under the impression that a single log 'connection' will be used by a single application, therefore the global state is stored in a single, global EVTCONFIG structure. Thread safety is a must for me, so it will be added later, but I'd focus more on evt_log() racing with another evt_log() call. (and not evt_open racing with evt_log) > > > evt_rec_init(&e, LOG_INFO, "Test message with an embedded ';' in it. It also contains an <XML> like tag."); > > msg = idsa_event(handle); > > Almost the same. Though you are using the old syslog severities. > I think mjr proposed an 11 valued field, while I use 3 continuous > values in the range -1 to 1, to represent threat to availability, > confidentiality and integrity. I can put in a message field. > Though in my implementation the message is just > another tag - which I think is cleaner. The description should be as mandatory as possible. If it was provided as a tag it would be possible to log an event without one. I think it is difficult to completely forget about syslog severities though. The major transport for these messages will be syslog. We either need a function to map some 'new' priority field to standard syslog priority, _or_ we use syslog priority and require the application to define its own tag if it wants more. The 11 valued field mjr has proposed was quite IDS specific. The ones you use are IDSA specific, and I don't either of you could work with the other's priority definition. > > > evt_tag_str(e, 0, "test:tag", "'value'"); > > idsa_add_string(msg, "config-file", "/etc/httpd.conf"); > > Same except for priority > > > evt_tag_str(e, 0, "test:tag2", "\n\n\n\n"); > > evt_tag_int(e, -10, "test:fd", fileno(stderr)); > > idsa_add_integer(msg, "status", 500); > > Ditto. PS: Idsalite doesn't do fancy escapes, idsafull does Escaping is a must, otherwise you cannot warrant well-formedness. I'm for dropping the priority field. It was a bad decision. > > > evt_tag_errno(e, -5, "test:error", EAGAIN); > > Idsalite doesn't have this, idsafull does, plus another 11 or so > types. But for a simple standard, maybe we should stick to strings and > integers for the time being ? I don't think adding more types hurts. What are these: int idsa_add_scan(IDSA_EVENT * e, char *n, unsigned int t, char *s); int idsa_add_set(IDSA_EVENT * e, char *n, unsigned int t, void *p); ? > > > evt_tag_printf(e, 0, "test:printf", "%d %d", 5, 6); > > idsa_add_printf(msg, IDSA_LDM_PROTO, "HTTP/%d.%d", 1, 0); > > > es = evt_format(e); > > printf("%s\n", es); > > free(es); > > I don't have that in idsalite This function is needed only if the application wants to perform its own logging but still wants to use the tagged message format. > > > evt_log(e); > > idsa_log(handle, msg); > > > evt_rec_free(e); > > So you have an explicit free. In idsalite I do the free inside > the log to avoid leaks. But one also wants to re-use events. > In idsafull one can control if the event needs to be free'ed > explicitly. For the proposed standard I can implement a do_nothing > function, but are you sure the risk of leaks is worth the gain ? > I don't mind either way. I think evt_log() should consume the event for the very reason you mentioned. (thus: another change in the API) Event records are reference counted so the application can keep a record if it wants to: evt_rec_ref(e); evt_log(e); ... keep using e ... > > > * tag naming, namespaces > > not really the scope of this library but should be documented somewhere > > I have tag names in idsa_schemes.h. I have included the logging > data map under the namespace .ldm-1. I hope Marcus and Paul don't > object - I mailed them on Sunday. I'll try to write something about tag naming. > > > * tag ordering > > currently implemented as a number assigned to tags this should be dropped. > > > > * character set? > > should we support different character sets? I think we should but I'm not > > sure. > > I think both are not required for the minimal/lowest common denominator > standard API, as they are part of the "presentation layer". > > > * output modules character conversion > > should we use a single character encoding on the wire (UTF8?) would it be > > mandatory or it should be configurable? > > Just one. I prefer plain ASCII ;) I'm not so sure ;) But make this a topic discussed later. > > > * how to store the (prio, tagname) tuple? > > I assigned a priority to tags to support tag ordering. This is done by > > using functions like this: > > evt_tag_str(e, 10, "testtag", "value") > > > > Currently I defined macros like this: > > #define EVT_TAG_TESTTAG 10, "testtag" > > > > which can be substituted in function calls like this: > > evt_tag_str(e, EVT_TAG_TESTTAG, "value") > > > > However I don't like this too much. Maybe a separate struct to specify > > tags should be defined, but that would bloat the caller's code. > > See above. I've decided to remove tag priorities completely. > > > * syslog facility/priority mapping? > > > > The syslog facility and priority values are currently mandatory, they > > might be converted to simple tags, but how to send messages which do not > > specify them? > > I'd prefer it optional, so that the new API is not permanently tied > to syslog. Actually the same holds for the second field of evt_open but the primary transport of these messages will be syslog() should we use a default priority for messages which do not specify syslog priority on their own? > > > Administrative: > > * a policy to define tags > > * a set of standard tags for different applications > > See idsa_schemes.h for suggestions. I'd prefer using XML namespaces for tag naming. Other opinions? -- Bazsi PGP info: KeyID 9AF8D0A9 Fingerprint CD27 CFB0 802C 0944 9CFD 804E C82C 8EB1 _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Wed Jan 08 2003 - 08:08:54 PST