[ISN] Audit trails are vital for post-compromise investigations.

From: InfoSec News (isnat_private)
Date: Thu Nov 07 2002 - 02:52:29 PST

  • Next message: InfoSec News: "[ISN] Wormholes in hacker case"

    |  Linux Security: Tips, Tricks, and Hackery                       |
    |  Published by Onsight, Inc.                                      |
    |                                                                  |
    |  06-November-2002                                                |
    |  http://www.hackinglinuxexposed.com/articles/20021106.html       |
    Audit trails are vital for post-compromise investigations.
    By Brian Hatch
    Summary: It is imperative that you can determine what state a machine
    was in before you can ascertain how crackers compromised your
    security measures.
    When your machine is cracked[1] it's a good idea to figure out
    exactly what happened. If you don't know what was or who was on your
    machine, there's little chance you can ascertain what occurred to
    breach the security. Regardless of your post-compromise preference
    (fixing the problem or reinstalling from scratch) it's imperative
    that you understand what happened. If you end up leaving the exact
    same vulnerability open, you haven't fixed anything.
    For example, I once consulted at a pharmecutical company that was
    required by FDA regulations to keep detailed paper logs of every
    piece of software installed, every configuration change, ever user
    added or removed. While most of us grumbled about the annoyance of
    doing so, it made it absolutely clear what the state of a machine was
    over time, which made it easier to correlate problems with
    configuration changes, and find other bugs created by users^H^H^H^H
    Unfortunately, most folks don't have the time or energy to list
    everything they do to a system. However there are many steps you can
    take to make things more automated.
    Backup configuration directories
        In addition to your normal system backups, create simple cron
        jobs that back up your configuration settings - The /etc
        directory may be sufficient in itself. Backups should ideally go
        to removable media (tape, CD, etc) for long term storage, but you
        may find it's also helpful to have it stored on a separate
        machine as well for faster access in a crisis. Take, for example,
        the following script:
          cd /etc
          tarball=`mktemp` || exit 1
          tar cvzf $tarball .
          scp $tarball backup-server:/backups/`hostname`.etc.`date '+%Y-%m-%d'`.tgz
          rm $tarball
        This shell script, ripe for use from cron, will create a tarball
        of /etc and scp it over to a backup-server. (This, of course,
        assumes that you have some form of passwordless scp access
        available. I'll probably cover that in a later article.)
        Certainly not the most elegant of solutions, but it will capture
        the essence of the machine's state.
        Of course, if you liberally comment your configuration files,
        this makes everything much clearer as well.
        My personal favourite method is to keep all your configuration in
        CVS. You can put all of /etc in a CVS repository and maintain
        your configuration via CVS that way. This makes it easy for you
        to see exactly how a machine was configured at any time, and
        makes it trivial to recover a compromised system or clone a
        functional one.
        You can use RCS locally, if you prefer, and in combination with
        some sort of backups such as the previous example you'll have a
        similar revision history without the overhead of learning and
        implementing CVS.
    Internal Mail Archives
        If you have multiple administrators at work, it's a good idea to
        create a mailing list that goes to all parties, and have each
        mail archived automatically. Though not as seamless as the
        previous two options, it does provide a minimal audit trail that
        may be helpful.
        This assumes, of course, that you're all keeping each other
        informed about what you're doing. Communication is good for
        systems stability. Then again, keeping secret what you do is good
        for job security.
    This article is written partially in response to a request from a
    reader. He found that several processes were running as uid 1032, but
    no such user existed on the system. In fact, more processes kept
    being run over time - it wasn't just leftover processes that were
    still running. It took a while to track down how the processes were
    being started before they were all killed off.
    There used to be an actual user with uid 1032, but it had been
    deleted some time in the past. Since the record from /etc/passwd was
    gone, it was not immediately obvious who the user himself actually
    Next week I'll teach you a very simple trick you can use to lock out
    user without loosing their actual login information. However until
    that time, I offer my latest challenge...
    The challenge
        Come up with fun, ingenious, or wacky ways that you think a user
        who is removed from the system (entry in /etc/passwd is deleted,
        for example) can have processes still running, and/or the ability
        to create new processes at will or periodically.
        I'll send out a post card from lovely yet rainy Seattle to an
        entry (or entries) of my choice based on Originality,
        Performance, and Lip Sync, or any other criteria that comes to
        (Send your responses to me at briat_private Don't
        reply to the list address.)
    [1] Yes, "when", not "if". Sooner or later, it happens to all of us.
    Brian Hatch is Chief Hacker at Onsight, Inc and author of Hacking
    Linux Exposed and Building Linux VPNs. He is a bit CVS-happy. His
    websites are all stored as WML in CVS, and any cvs commit
    automatically triggers a rebuild of the site and rsync to the
    webservers. Now if only he could get this to work for the
    non-computing areas of his life. Brian can be reached at
    This newsletter is distributed by Onsight, Inc.
    The list is managed with MailMan (http://www.list.org). You can
    subscribe, unsubscribe, or change your password by visiting
    http://lists.onsight.com/ or by sending email to
    Archives of this and previous newsletters are available at
    Copyright 2002, Brian Hatch.
    ISN is currently hosted by Attrition.org
    To unsubscribe email majordomoat_private with 'unsubscribe isn'
    in the BODY of the mail.

    This archive was generated by hypermail 2b30 : Thu Nov 07 2002 - 05:01:14 PST