[ISN] /etc/inittab - The Most Overlooked Cracker Haven

From: InfoSec News (isnat_private)
Date: Thu Dec 05 2002 - 22:43:51 PST

  • Next message: InfoSec News: "[ISN] I shut radio site, boasts teen hacker"

    |  Linux Security: Tips, Tricks, and Hackery                       |
    |  Published by Onsight, Inc.                                      |
    |                                                                  |
    |  05-December-2002                                                |
    |  http://www.hackinglinuxexposed.com/articles/20021205.html       |
    This issue sponsored by: Hacking Linux Exposed, Second Edition.
    Hacking Linux Exposed, Second Edition will be hitting the stores any
    day now. This new version of the best-selling first edition includes
    almost 200 new pages of hacking and cracking materials. Also, if you
    order online through our web page (via Amazon or Barnes and Noble),
    we'll donate any commissions to the Electronic Frontier Foundation.
    For more information, visit http://www.hackinglinuxexposed.com/
    /etc/inittab - The Most Overlooked Cracker Haven
    By Brian Hatch
    Summary: Crackers can cause their software to be run by adding
    entries to /etc/inittab, a file frequently missed by administrators.
    Last week I provided a challenge, which can be summarized as follows:
      * A cracker broke into a server,
      * Said cracker installed various back doors,
      * The administrator removed everything he could find, but,
      * Each time the machine was booted, even in single user mode,
        mysterious processes ran from /tmp/.qw,
      * The /tmp directory was created clean from scratch at every
        bootup, so the files could not be stored there and usable between
      * All xinetd and /etc/rc?.d services were cleaned out and nothing
        therein could have started these mystery processes, or start
        processes that started those processes, etc,
      * No LKMs or kernel modifications were used to start these
    Many[2] people wrote in with ideas which fell into the following
      * changes to the master boot record
      * changes to /etc/inittab
      * changes to bootloader (lilo/grub) command-line arguments
      * changes to initrd ram disks
      * changes to /root/.bashrc, /etc/profile, and such.
    I told folks to discount any change which required kernel
    modifications. A change to the MBR probably doesn't qualify in that
    regard, but it would certainly seem out of scope nonetheless.[3]
    Similarly, a change to the bootloader or initrd ram disk[4] fall
    under kernel changes to some degree, and thus weren't the correct
    answer, based on my phrasing.
    Many folks suggested that it was root's .bashrc, or global /etc/
    profile and similar files that was the culprit. By logging in, root's
    shell would execute commands that were hidden in those files. That's
    definately a possibility, but the fact that moniker had PID 15 meant
    that it had to have started before root logged in, so that's not the
    In fact, the problem on this compromised box was caused by cracker
    changes to /etc/inittab. To explain, let's recap some of the
    evidence. The processes were visible via ps as:
      # ps -ef | grep qw
      UID      PID  PPID  C STIME TTY          TIME CMD
      root     15    1    0 11:24 ?        00:00:00 /tmp/.qw/moniker
      root     242   15   0 11:25 ?        00:00:00 /tmp/.qw/cr8
      root     244   1    0 11:25 ?        00:00:00 /tmp/.qw/rucc
    Several folks said that it is quite obvious that 'monkier' and 'rucc'
    were started from init since their parent is process #1. (Init always
    has process ID #1). Unfortunately, that's faulty logic. Any
    daemonized process will be inherited by init when as seen in this "ps
    -ef" snippet:
    $ ps -ef | grep '  1  '
      UID        PID  PPID  C STIME TTY          TIME CMD
      root         1     0  0 Jan23 ?        00:00:19 init [2]
      root         2     1  0 Jan23 ?        00:00:00 [powerd]
      root         3     1  0 Jan23 ?        00:00:00 [keventd]
      root        10     1  0 Jan23 ?        00:00:00 [scsi_eh_0]
      root        11     1  0 Jan23 ?        00:00:00 [khubd]
      root        88     1  0 Jan23 ?        00:00:04 perl /etc/swatch/swatch
      root       483     1  0 Jan23 ?        00:00:00 /sbin/klogd
      root       574     1  0 Jan23 ?        00:02:23 /usr/lib/postfix/master
      daemon     612     1  0 Jan23 ?        00:00:00 /usr/sbin/atd
      root       619     1  0 Jan23 ?        00:00:13 /usr/sbin/cron
      root       632     1  0 Jan23 ttyS0    00:00:00 /sbin/getty 9600 ttyS0
      root       859     1  0 Jan23 ?        00:02:15 /sbin/syslogd
      root       893     1  0 Jan23 ?        00:28:28 /usr/sbin/sshd
    These processes also have a PPID of 1, but most were started by
    standard /etc/rc?.d scripts. The fact that our cracker's processes
    had init as the parent doesn't prove they were started by init.
    However the 'monker' program has a PID of 15 - a very very low number
    for processes. Assuming it was started as the 15th process (as
    opposed to being several thousand process IDs later when the PIDs
    wrapped around) then it must be extreemly early in the boot process,
    before many rc?.d scripts run, and likely is started from init.
    The administrator had re-installed all his software to be safe. By
    design, these re-installs did not wipe out his configuration files,
    so though he reinstalled init, it did not wipe /etc/inittab clean.
    He scanned the machine for any files that had been touched since the
    breach, but this is not reliable. A cracker can easily change the
    modified-date timestamp on a file using the 'touch' command. Doing so
    will change the inode change time, however:
      $ cd /tmp
      $ ls -l somefile; ls -cl somefile
      -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
      -rw-------    1 bri      bri         20476 Sep 17 14:49 somefile
      $ date
      Wed Dec  4 13:31:37 PST 2002
      $ echo "blah" >> somefile
      $ ls -l somefile; ls -cl somefile
      -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile
      -rw-------    1 bri      bri         20481 Dec 04 12:00 somefile
      $ touch 09201419 somefile
      $ ls -l somefile; ls -lc somefile
      -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
      -rw-------    1 bri      bri         20481 Dec  4 13:32 somefile
    Note that after our 'touch' command, the file modification time
    (mtime) was set to the original Sept 17th date. However the inode
    change time (ctime) was not. The ctime is always updated whenever the
    inode has any changes made, such as file modifications, chmod
    permissions changes, etc. The administrator was smart by looking for
    files with mtime or ctime changes during the window where the cracker
    had access.
    If the cracker modified /etc/inittab, even though he could reset the
    mtime with touch, he shouldn't be able to change the ctime back. Thus
    you'd think that his find process would have caught the /etc/inittab
    changes. Unfortunately, the cracker can still modify the ctime of a
    file simply by changing the system clock:
      $ date 09201419 
      $ touch 09201419 somefile
      $ date 12041200 
      $ ls -l somefile; ls -lc somefile
      -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
      -rw-------    1 bri      bri         20481 Sep 17 14:19 somefile
    So, faking both the mtime and ctime is trivial, and this is why he
    missed the changes to /etc/inittab - his find command didn't indicate
    any change, so he didn't examine the file at all.
    So, what were those changes? Here's a snippet of the file
      # The default runlevel.
      # Boot-time system configuration/initialization script.
      # This is run first except when booting in emergency (-b) mode.
      # What to do in single-user mode.
      # /etc/init.d executes the S and K scripts upon change
      # of runlevel.
      l0:0:wait:/etc/init.d/rc 0
      l1:1:wait:/etc/init.d/rc 1
      l2:2:wait:/etc/init.d/rc 2
      l3:3:wait:/etc/init.d/rc 3
      l4:4:wait:/etc/init.d/rc 4
    The line that begins with C1 above was the new entry. It tells init
    to run the program /dev/st0z at boot time, regardless of runlevel.
    This entry came before the /etc/init.d scripts (the l# entries) which
    is how it was able to get such a low process id.
    The /dev/st0z file was the cracker's program. Sticking the file in /
    dev/ made it less likely the administrator would find it, since many
    admins don't understand what all the /dev devices do. This file was a
    simple shell script:
      # head /dev/st0z
      cat <<'EOF' > /tmp/st0z.uu
      begin 700 st0z
      cd /tmp
      uudecode st0z.uu
      tar xzf st0z
      rm  st0z.uu st0z
      exec /tmp/.qw/moniker
    This script contains a uuencoded file within it, which it puts in /
    tmp. This uuencoded file is just a compressed tarball, so it
    uudecodes and untars it. It then deletes the temporary files, and
    then runs the moniker program (another shell script). The actual
    script had more comments at the top to make it look more like a
    system program, and obfuscated the commands a bit, but you get the
    Monkier was responsible for running the cr8 program -- obvious from
    ps output, since moniker is cr8's parent -- and for running rucc,
    which is not obvious from ps, but obvious if you read monkier's
    This st0z script also explains why the files in /tmp/.qw were always
    coming back, even though they weren't there between reboots. The
    files were copies out of the /dev/st0z file each time, so wiping /tmp
    had no effect.
    So what were the problems this administrator faced when trying to
    track down and remove the mysterious cracker code?
      * He blindly trusted timestamps on files, thus missing /etc/
      * Had no file integrity tools on the system, thus missing inittab
        and /dev/st0z.
      * He didn't understand the Linux boot process enough to know that
        programs can be launched by init directly.
    Of course, you may argue having a week root password was another
    problem. There were a few other backdoors found later, and eventually
    they decided to wipe the machine clean and start from scratch - a
    very good idea given how long the machine had been out of their
    Thanks to the many folks who wrote in with ideas - you came up with
    some things I hadn't though of either. And congratulations to Mandi
    Walls, who nailed pretty much everything that the cracker did. A
    beautiful copy of Hacking Linux Exposed, Second Edition will be
    coming your way.
    Also, thanks to Simon Walter who pointed out the 'From Power Up To
    Bash Prompt' webpage at http://www.tldp.org/HOWTO/
    From-PowerUp-To-Bash-Prompt-HOWTO.html. If you want to understand the
    bootup process in more depth, take a look.
    [1] This wasn't obvious based on the facts presented, but was offered
    by me since I already knew the actual cause, in order to narrow the
    scope of the challenge.
    [2] Proof that a free book is much better incentive than a postcard -
    I shall have to remember this.
    [3] A change to the MBR would need to run the kernel in a way
    different than the default, and really is no different than modifying
    the kernel or the bootloader. Thus a theoretical MBR change would
    seem to fit within the other alternatives anyway.
    [4] Inirtd ram disks are typically used to help a Linux kernel load
    needed modules, and thus this fits into the kernel-changes category
    [5] Becoming a daemon process involves code such as "fork && exit;
    fork && exit" which creates a child process and kills off the parent
    processes, such that they have no running parent process, and are
    inherited by init.
    [6] Rucc contained code to daemonize, which is why it's parent is
    Brian Hatch is Chief Hacker at Onsight, Inc and author of Hacking
    Linux Exposed and Building Linux VPNs. He launches programs from /etc
    /inittab quite regularly, or at worst through DJB's supervise. It's a
    heck of a lot easier than writing your own daemon checking and
    restarting code. Brian can be reached at
    This newsletter is distributed by Onsight, Inc.
    The list is managed with MailMan (http://www.list.org). You can
    subscribe, unsubscribe, or change your password by visiting
    http://lists.onsight.com/ or by sending email to
    Archives of this and previous newsletters are available at
    Copyright 2002, Brian Hatch.
    ISN is currently hosted by Attrition.org
    To unsubscribe email majordomoat_private with 'unsubscribe isn'
    in the BODY of the mail.

    This archive was generated by hypermail 2b30 : Fri Dec 06 2002 - 01:25:35 PST