[ISN] How to Survive a Hack Attack

From: mea culpa (jerichoat_private)
Date: Sat Nov 28 1998 - 01:46:34 PST

  • Next message: mea culpa: "[ISN] Hacker's Death: Murder or Suicide?"

      This message is in MIME format.  The first part should be readable text,
      while the remaining parts are likely unreadable without MIME-aware tools.
      Send mail to mimeat_private for more info.
    Content-Type: TEXT/PLAIN; CHARSET=us-ascii
    Content-Transfer-Encoding: QUOTED-PRINTABLE
    Content-ID: <Pine.SUN.3.96.981126155026.19243vat_private>
    How to Survive a Hack Attack
    November 23, 1998
    Its late, you're sleeping, you get a call waking you . . .  "Fred, we're
    being attacked, someone is trying to do a denial of service attack against
    our web server". You quickly realize that lots of people aren't going to
    be very happy, so you leap into action . . . after all, you're the
    security GURU, right?=20
    So what do you do about this "alert"?=20
    Well, for many, this has already happened and, for the most part, they've
    experienced what I'm going to describe. However, many of you who read
    NTBugtraq and follow security alerts on the 'net haven't had it happen to
    you yet . . . and it is you I'd like to talk to.=20
    You see the number of times the alert is sounded, versus the number of
    times an actual attack occurs, is extremely disproportionate. Usually,
    chances are you are not under attack.=20
    Don't get me wrong, I'm not saying attacks don't happen=97they do,
    especially to ISPs. The key to responding to this type of alert--the key
    to determining if you're really under attack--is how you handle the facts.=
    1. Do you know what's supposed to be running on the box? =20
    Making sure that someone can tell you precisely what's supposed to be
    running on the box is crucial to handling an alert. Without this
    knowledge, you're at a loss to know whether what's supposed to be there is
    there. Has something been placed there by an attacker? Knowing the right
    version numbers and names of the various executable programs that are
    running (or could be running), when they're supposed to be running, and
    what each of them actually does, is crucial to keeping control of your
    And of course, knowing all of this will make it easier to search through
    Knowledgebase articles and/or vendor's support sites . . . but don't go
    there yet.=20
    2. Do you have a backup?=20
    Before you go trying to stop an attack, you need to be sure you can
    recover from it once you do. Since you don't know what the attack may or
    may not have done to your system, figuring out when your last backup was
    made, and more importantly how useful it might be, is crucial to
    determining how to handle the alert. If you know you have a backup from
    just prior to any indications of an alert you could, if necessary,
    completely replace the box.  That makes attacking the problem easier:
    Should you have to delete or alter something sensitive in the process, you
    know you can go to the backup if all else fails. If you don't have a
    reasonably up-to-date backup, then you'll have to be a whole lot more
    cautious in your approach. And you'll have to work harder to determine
    exactly what might have changed due to the alert=97trashing the system and
    doing a full "restore" won't be an option.=20
    3. Who discovered the alert? What exactly did they discover?=20
    Who discovered the alert is important in diagnosing the problem because
    their knowledge and expertise could greatly affect the validity of their
    observations. Not to be harsh, but: Do they know what they're talking
    about?  Should they have been able to even see the alert? If not, what
    were they doing on the box that allowed them to discover it? The answer to
    that question has solved more than a few alerts I've dealt with (um, they
    were just poking around the registry on that box and then all of a sudden
    they noticed the CPU went to 100% . . . ;-])=20
    Additionally, what the alert actually is can be highly subjective. That is
    to say that there may be a difference between what the person thinks the
    problem is and the problem itself. We've all likely heard the claim "its
    running much slower than it usually does . . ." only to find out it
    happens to be in the middle of a full system backup . . . ;-] Performance
    counters are complex, and can easily be mis-read. Telling someone that the
    number of "HTTP Connection Attempts" is way above normal doesn't
    necessarily mean you're under attack. Especially if they take that Perfmon
    value and compare it with a web statistics program's "Hits" value (when
    you consider that a connection attempt doesn't necessarily get logged as a
    hit if the transfer was unsuccessful).=20
    Too many people spend far too little time analyzing the alert and
    verifying it by multiple mechanisms=97sad but true. Do it: analyze & verify=
    If you think you're under a DOS attack, you should:=20
       * run Netmon and see whether network utilization is being sustained at a
         higher than normal level. And remember to compare your LAN bandwidth
         with that of your ISP connection. Your utilization shouldn't be able t=
         sustain rates higher than your Internet connection allows. If it is,
         then the attack is either coming from another machine on your LAN, or
         you've got some other problem causing the saturation.
       * look at "netstat -n 1" and see whether or not you have a lot of
         connection attempts from the same, or similar, IP addresses. Chances
         are a DOS attack is going to come from a single IP address, or a range
         of addresses from a single subnet. When the Teardrop2 attack happened
         earlier this year, the source address was always the same. Despite it
         being spoofed (i.e. not from the address you actually saw), it was
         still the same address repeatedly.
       * Have a look at your web logs. Do they reflect the traffic you're
         seeing? The Net is a funny place. I know=97I had a spike on my web sit=
         once that seriously made me consider it an alert. Turns out something =
         wrote had just been linked from Microsoft's web site and the viewers
         rose exponentially in a matter of minutes.
    If the results of the above verifications show some inconsistencies, then
    you can assume (as I usually do) that someone has changed something on the
    box, and that it's causing a problem. When did the alert start? Did
    someone add or remove something from the box around that time? What's new
    on the box? Can you revert back to what you had before, and if you do,
    does the problem still occur? I can't stress this enough: chances are
    something you (or your team) did is the cause of your alert. More often
    than not, it's a change to the box that's caused things to run afoul.=20
    If, on the other hand, all verifications check out and you're convinced
    your being attacked, then what?=20
    4. Contact your ISP.=20
    If you can identify a particular IP address (or range of IP addresses) as
    the root cause of the alert, you should contact your ISP and ask them to
    filter out that address at their routers. Of course you could filter out
    the offending address at your routers, but that'll amount to a hill o'
    beans most likely. Remember, the bandwidth from your ISP to you is your
    bandwidth=97it will still be consumed by the attacking traffic as it comes
    down to your router to be rejected! Getting your ISP involved should help
    avoid this bandwidth loss, as well as set their security folks in motion.
    An attack against you is also an attack against your ISP! Once you and
    your ISP have cut off the attack, you're looking at fixing what ever was
    damage (say hello to your backup) and patching you box.=20
    Of course it goes without sayting that you and your ISP should have a plan
    in place before an attack occurs. Talk to your ISP now and find out what
    they will do in such a situation . . . if they don't have an answer:
    change your ISP!=20
    5. Log everything you can.=20
    While actually going to court against an attacker is extremely rare, you
    won't be able to do anything without logs of the attack. Further,
    companies like Network Flight Recorder (http://www.nfr.net) and ISS Real
    Secure (http://www.iss.net), who make products that can detect attack
    signatures on the wire, would greatly appreciate detailed logs of actual
    attack events.  The more detailed the information you can provide to such
    vendors, the more they'll be able to build products to combat such
    6. Let me know about it!=20
    NTBugtraq is here to keep NT folk informed about such attacks. The damage
    done by Teardrop2, while devastating to quite a few people for a short
    period of time, was limited at least in part due to our ability to
    coordinate lots of different people onto the same problem . . . virtually
    in real time.=20
    Chances are, if you follow these steps, your problem will be resolved.=20
      1. know what's running on your box
      2. keep current backups
      3. verify the attack as real
      4. cut it off at the ISP level
      5. log it
      6. report it
    Bottom line to all of this is to make sure you have a plan. Make sure you
    know what you're going to do when/if you should ever get that call in the
    middle of the night. Trust me, when it happens, your heart is going to
    start pumping and you're going to be scrambling. Having a plan, a list of
    questions to ask, a list of things to do, a list of people to call (with
    numbers!) will help calm you down and let you take the right actions to
    resolve it quickly.=20
    Subscribe: mail majordomoat_private with "subscribe isn".
    Today's ISN Sponsor: Repent Security Incorporated [www.repsec.com]

    This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 13:12:43 PDT