> > > My current thoughts are : > > * they should be archived to tamper proof (write once) > media, such as CD- > > or DVD-R. > > Many people share this view. It's pricey, but doable. It may be > cheaper to instead engineer good secure off-site replication to a > site so tightly secured and monitored that it can be defended as > "tamper-resistent". This may be cheaper than archiving all your log > data to slow, small media. The volume of logs i'm presently dealing with allow this to be feasible. I'm not anticipating having to change disks more then once a week (something that can be worked into regular schedules) The backup system being used here is not secure however, and i don't think i've got the time or the energy to figure out a way to secure it. > > > * they should have as a minimum a hash applied and stored > with them (how to > > implement is the question) > > I've not pursued that line of research myself, but I'm pretty sure I > remember some papers from Bruce Schneier about it. Can't seem to > turn 'em up on a search of counterpane.com, though. Google did turn > up an email[1] with some references. I've not seen an appealing > implementation of this stuff, though; key management is a pain. If > you have to securely hold a key on a logging machine, you have to > secure that machine. If you've done that, then I'd rather just treat > it as tamper-resistent in its own right:-). What are the PEO features of msyslog for then? i was under the impression that this was the primary goal of them. > > > * They should ideally be organized for easy perusing. > > If I were trying to build such a lashup, I'd _definitely_ decouple > this one from the "write-once media" aspect. Browsing huge numbers > of slow disks is no fun. I'd maybe discuss alternatives for how to > make the 'tamper-resistent logs', but I think it's inarguable that > for nice browsing you want online archives on big fast disk, stored > in compressed files, with helper indices to enable fast searching > for what you want. One possible reasonable approach is full-text > indexing on the messages, plus auxiliary indices with something like > cdb hashes for the fixed fields of interest. > > My own preference would be to identify just a couple of indexing > criteria (host, day), and store logfiles like YYYY/mm/dd/hostname.bz2, > where the violently compressed logfiles have the arrival timestamps > in high-resolution prepended to them, in a nice format for sorting > --- I like UTC ISO 8601 mostly, although that "T" is butt-ugly, and > I replace it with a space, or a dash if I want a single field with > no internal whitespace. Before compressing, generate full-text > indices over the files. Then it's easy to script up whatever search > you need, and with nicely sortable arrivale-time-stamps it's easy to > reconstruct various merged logs from various sets of files. This is more along the lines of what i was looking at.. probably something similar to what syslog-ng currently does, e.g. /YYYY/MM/DD/HOST/Service. I'm assuming most syslog implementations can handle this (e.g. SDSC when it's released as a final, msyslog, syslog-ng) Thanks for the info! _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis
This archive was generated by hypermail 2b30 : Wed Dec 11 2002 - 15:09:37 PST