<< Aleph; Where I work, the policy has often been that security is not relevant, or a pain in the neck, or the paranoid delusions of systems admin-types. Well, last thursday, we discovered a rather annoying intrusion, and I wrote a brief, almost 'manager-level' paper on hwo bad things get when they go wrong ... Perhaps other people on Bugtraq will have use of it? >> What Happened: ------------- 8:00am, Thursday, January 28th, I recieved email that one of my servers a) had been compromised, and b) had been used in the compromise of a remote university site, who was now very displeased with our existence. I immediately logged into the offending machine, and investigated what evidence the cracker had left behind. The first thing discovered was trojan'ed copies of rshd, telnetd, and ftpd, in a supposedly hidden ... directory. Much to my annoyance, I also found out that /usr/bin/ls was trojan'ed, at least not to list ... and '. ' files. Switching to /usr/ucb/ls, which the cracker missed, a rootkit trojan script was discovered, which replaced several executables in /usr/bin and /usr/sbin. I believe that the network services were manually trojan'ed. The logs looked 'too clean', causing me to suspect that they had been sanitised in some fashion. As an offhand guess, we think that ftpd was compromised, in early January, but lack concrete evidence. My general opinion is that we most likely were dealing with what a friend of mine calls a 'script kiddie.' However, he did a few things that struck me as somewhat abnormal for a standard kiddie [namely the apparent manual replacement of the rshd, et al], and I felt it prudent to continue to the next step: the machine was sentenced to death -- unplugged from the network, backed up, formatted and reinstalled. While we were at it, we sentenced all the client machines to the same death, as they were supported through NFS of binaries and rdist'ed account information. Mistakes Made in Incidence Response: ----------------------------------- 1) Don't log in as root on a machine that most likely has been compromised. Bsd things can happen. 2) Don't go around blithely executing binaries. (I feel rather stupid about that) 3) Do *immediately* take the machine offline, and mount the disks on another system for analysis. Mistakes Made in Security Implementation: ---------------------------------------- 1) we have no firewall nor tcpd running, so there is no effective access control or access logging. We have incredibly primitive router filtering, which eliminates only the most basic of IP-spoofing attacks. 2) we did not audit our services effectively, nor secure more reliable alternatives to services we needed to offer. [examples: telnetd, rshd, ftpd, httpd/cgi-bin, pop, imap] 3) we did not keep up with the security mailing lists, versioning of important freeware services [sendmail, apache, ssh, bind, pop, imapd], nor did we maintain operating system patches. 4) we did not audit our workstations or compute servers for unnecessary SUID/SGID programs. 5) we did not engage in security sweeps/random audits with portscanners, publicly available tools [SAINT], or commercial utilities [ISS]. 6) we did not purchase or implement any Intrusion Detection Software. [IDS] 7) we did not run tripwire, or any other cryptographic checksum program against the executables on the server. 8) we did not log activity to a secure loghost. 9) we did not manually analyse the log files, nor did we run scripts to correlate accesses against the log files. 10) we combined multiple services on one machine, and sometimes known-insecure services on NIS or rdist-passwd clients. [web/ftp/mail on a general use server] 11) we did not have a trusted archive of all the tools we compiled/installed on the server, nor on the clients Direct Costs Incurred: --------------------- 1) Over twelve hours of system unavailability. 2) Over 48 man-hours of work to restore services. Indirect Costs Incurred: ----------------------- 1) secondary DNS server unavailable. 2) two days where researchers and staff cannot perform work. 3) open-ended period of time compiling and installing tools that we missed the first time. Where Our Security Implementation Hurt/Hindered: ----------------------------------------------- Not providing more than the most rudimentiary access control denied us the possibility of delaying, or even denying the compromise. Not providing for a secure loghost resulted in the destruction of the relevant log files, making analysis of the incident difficult, apprehension of the felon unlikely and conviction impossible. Not auditing services and not removing/replacing/securing especially odious ones left systems openly available to the Internet[tm] in a fragile state. Not keeping up with security information and/or patch revisions left a number of exploits 'lying around' on our system -- one or more of which were used by the cracker. Not auditing SUID/SGID programs leaves a wonderful toolkit of mayhem, with which a malicious user can hurt you. (How many Solaris admins have ufsrestore rsxr-xr-x?) Not engaging in random security sweeps and roving audits denied us the opportunity to notice something amiss, specifically in services offered on our machines. Not having trusted logfiles to analyse cost us the possibility of noticing portscans or 'security sweeps' from the cracker, before they struck. Not using tripwire cost us a lot, in that a) we had to rebuild every last GNU program from source, and b) we did not have it available as a means of detecting 'wrongness' on a production system. Not divorcing services meant that we were running web/ftp on machines that had the complete account/password information for the lab, 'just ripe for the picking'. In addition, the compromise rendered the entire lab unavailable as we rebuilt, rather than just one or two functions. Not having a secure archive of tools (or even the systems in general!) meant that we had to rebuild everything from scratch. Lessons Learned: --------------- When you think 'security,' think 'defense in depth.' The French demonstrated very neatly that putting all their resources into the Maginot Line was not very bright, and we should make every effort *not* to recreate the Maginot Line. Security is *not* cost-intensive, if you build it in the first time, or add it in as you upgrade your environment, especially as you value it against the total loss of your environment. Find a way to control outside access. Either throttle it through a firewall, run it through router filters, or use tcpd. (in decending order of preference) Scan. Scan. Scan. Scan your machines for vulnerabilities, scan your network for machines that shouldn't be there, and services that shouldn't be running. Create/purchase and use Intrusion Detection Software (IDS) across common avenues of attack [phf exploit in cgi-bin, port scans, nfs traffic from across the network, executing common buffer overflow exploits, etc -- this could get exciting if you run a scan and the IDS sets off your pager .... 9000 times.] Use a checksum sanity checking program, such a tripwire, on your binaries; and run it periodically as insurance against anything going wrong. Keep secure, trusted backups of important things -- cut a tape after you've freshly installed/configured a server; and/or periodically cut a CD of your favourite compiled tools. Audit the services that you are offering, shut off the annoying, upgrade the rest, and find ways of tightening security on them. (apop vs. pop, or perhaps pop through an ssh tunnel?) Try to keep track of security -- Bugtraq, at www.netspace.org, is a very good resource. Keep up to date on your patches. It is truly embarrassing to get nailed by an exploit that was patched last year. Divorce global services (httpd/ftp) from internal ones (mail/file); important services from not so important, and *don't* run them on the same machine! CGI-BIN scripts are generally annoying, and generally hard to keep secure. In my opinion, they should not be allowed to live. However, if you must let them stay, consider: using perl with the 'taint' and 'warn' flags; consider cgi_wrappers; consider 'mod_perl' for Apache; php; or sperl, for so called 'secure PERL.' Get a secure loghost, and run analysation algorythms against your logfiles, perhaps: tabulating accesses to your network from external IP number/domain name; by date; by port number; etc -- and look for abnormalities. [This can even be scripted and crontabbed to death] Investigate other security software: kerberos, AFS, etc. By all means, attempt to break into your site. [NOTE: make sure you have everything signed, notarised, dotted and witnessed before doing so!] Probe remotely, discover what an 'outsider' can discover, and research how to shut them off. And, whatever you do, don't get complacent. -- -- Dragons slain; Demons banished; Castles stormed--all in one low price! -- -- John E. Jasen // DNRC Ambassador to Earth \\ jjasen1at_private -- -- These views have been approved by the Y2K Panic Facilitation Committee --
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 14:32:20 PDT