Re: [logs] Log archival

betat_private

2002-12-11-11:26:15 Blaise St-Laurent:
> I'm in the process of setting up a centralized log server here, and i was
> wondering, from a potential forensics point of view, what are the
> requirements for archiving logs such that they are maximally useful down the
> road.
> 
> My current thoughts are :
> 	* they should be archived to tamper proof (write once) media, such as CD-
> or DVD-R.

Many people share this view. It's pricey, but doable. It may be
cheaper to instead engineer good secure off-site replication to a
site so tightly secured and monitored that it can be defended as
"tamper-resistent". This may be cheaper than archiving all your log
data to slow, small media.

> 	* they should have as a minimum a hash applied and stored with them (how to
> implement is the question)

I've not pursued that line of research myself, but I'm pretty sure I
remember some papers from Bruce Schneier about it. Can't seem to
turn 'em up on a search of counterpane.com, though. Google did turn
up an email[1] with some references. I've not seen an appealing
implementation of this stuff, though; key management is a pain. If
you have to securely hold a key on a logging machine, you have to
secure that machine. If you've done that, then I'd rather just treat
it as tamper-resistent in its own right:-).

> 	* They should ideally be organized for easy perusing.

If I were trying to build such a lashup, I'd _definitely_ decouple
this one from the "write-once media" aspect. Browsing huge numbers
of slow disks is no fun. I'd maybe discuss alternatives for how to
make the 'tamper-resistent logs', but I think it's inarguable that
for nice browsing you want online archives on big fast disk, stored
in compressed files, with helper indices to enable fast searching
for what you want. One possible reasonable approach is full-text
indexing on the messages, plus auxiliary indices with something like
cdb hashes for the fixed fields of interest.

My own preference would be to identify just a couple of indexing
criteria (host, day), and store logfiles like YYYY/mm/dd/hostname.bz2,
where the violently compressed logfiles have the arrival timestamps
in high-resolution prepended to them, in a nice format for sorting
--- I like UTC ISO 8601 mostly, although that "T" is butt-ugly, and
I replace it with a space, or a dash if I want a single field with
no internal whitespace. Before compressing, generate full-text
indices over the files. Then it's easy to script up whatever search
you need, and with nicely sortable arrivale-time-stamps it's easy to
reconstruct various merged logs from various sets of files.

-Bennett

[1] <URL:http://http://lists.jammed.com/forensics/2001/07/0094.html>