Re: [logs] Log Analysis for Law Enforcement

From: Bennett Todd (betat_private)
Date: Fri Jan 17 2003 - 10:20:17 PST

  • Next message: swatch swatch: "Re: [logs] swatchrc file"

    2003-01-16T16:05:03 Buck Buchanan:
    > The Institute for Security Technology Studies at Dartmouth College has a
    > paper out titled "Law Enforcement Tools and Technologies for Investigating
    > Cyber Attacks" and you can get it via
    > http://www.ists.dartmouth.edu/TAG/lena.htm.
    
    Nice pointer, thanks!
    
    > Another part suggests the need for a tool to merge multiple logs
    > from multiple machines into a timeline.
    
    Different folks will have different approaches they favour for this
    problem. My own favourite approach is to (a) canonicalize the log
    records to have a timestamp that collates in chronological order, at
    the beginning of the record; e.g. RFC 3339 / ISO 8601 canonicalized
    to a single uniform timezone, and with uniform precision (making a
    uniform-length timestamp field). Then (b) append to the timestamp
    record numbers (relative to the input files to be merged), to make
    a stable sort. (c) Concatenate and sort the lot, then (d) strip the
    record numbers back out.
    
    > Hopefully this tool would have the smarts to figure out the
    > differences between the clocks on the system supplying the logs,
    > and to be able to determine the clock drift.
    
    That'd be nice, but it requires more information than I've seen
    preserved in syslogs. The easiest fix to this problem is to just
    eliminate the clock skew and drift, by running good time
    synchronization code like ntp (which I'm not all that fond of) or
    clockspeed (which I prefer).
    
    Failing that, the next easiest would be to determine the clock skew
    by external means, then use it to retroactively adjust the logs.
    Clockspeed is probably the most convenient tool I know of for
    determining clock skew.
    
    The folks who are working on enhancing the payload of syslog are
    also hoping to collect the info needed for this clock-deskew
    process, by preserving both original log timestamp and additional
    stamps applied by each relay daemon.
    
    But a really totally completely robust solution to this is hard
    enough that I doubt it'll ever be accomplished.
    
    For folks who really, really care about the best possible timestamps
    for logfile entries, the best solution I know of is to synchronize
    all systems with clockspeed, keeping them on TAI. The "UTC" time
    as implemented by ntp follows the mandate of Posix, and shifts
    its "epoch" every time there's a leap second, making historical
    timestamps ambiguous during any leap second.
    
    -Bennett
    
    
    

    _______________________________________________ LogAnalysis mailing list LogAnalysisat_private http://lists.shmoo.com/mailman/listinfo/loganalysis



    This archive was generated by hypermail 2b30 : Sat Jan 18 2003 - 21:12:50 PST