Hi Andrew, The message you are quoting from me was my attempt to refute the possible causes suggested by another poster, not as a suggestion for the reasoning behind the nmap problem. I was attempting to indicate that I did not think that ARP was the problem, but was waiting for further testing and details, which I did not have at the time. Apologies, as I was not clear in my message. I believe my later follow up message of 9/15/99 is exactly in-line with what Kenny had tested in house. Just further clarification. Thank you, Lisa Napier Product Security Incident Response Team Cisco Systems At 01:44 PM 9/22/1999 -0700, Lancashire, Andrew wrote: >This is to clarify what is being put out by Cisco and what we are being told >by Cisco. > >Two e-mails below is what Cisco is telling us and makes allot more sense >than what Cisco is telling Bugtraq. The last post to Bugtraq made mention >that the arp cache was filling up and allocating memory for both reachable >hosts and unreachable hosts (incompletes). Although what Lisa describes is >true regarding the arp cache, it would not be true for our or most other >sane persons environment. Since routers will only arp for what is local, >that would mean that for the arp cache to fill up and us all the memory all >networks in the 10.x.x.x range would need to be local. So that's not gonna >happen but if you read the e-mail below that from Kenny (also at Cisco ) his >explanation makes allot more sense considering we have hundreds of routers. > >Thank You > >Andrew > >P.S. Congratulations on the re-opening of PacketStorm >___________________________________ > > >Subject: Re: Cisco and Nmap Dos >To: BUGTRAQat_private > > >Hi all, > > >I wanted to address the items listed here. We are still investigating this >problem, and hope to have more information later on in the week. > > >Item 1, OSPF is not an issue. According to the configuration information >provided to us by the customer, OSPF is not in use. > > >Items 2, 3 & 4. IOS actually handles ARP in the following manner: > > >When we receive a packet destined for something not already in our ARP >table, we enter an "incomplete" entry in the ARP table. Then we will rate >limit ARPs to once every 2 seconds to that destination. Any additional >packets to that same destination will be dropped until the ARP entry is >completed. This is on a per destination basis. > > >"Incompletes" (ARP requests that have not been responded to) get dropped >after approximately 1 minute from the last time we sent an ARP request for >that non existent address. > > >Incomplete entries MAY stay around longer, as the process that is >responsible for cleaning up the ARP table may not have enough time to >complete before it is called again, if we have a lot to clean up, which may >be relevant to this case. The incomplete entries will eventually get >cleaned up, but it may take two or three minutes, two or three cycles of the >process that cleans up the table. > > >Under a dedicated, intense nmap scan, a very large number of ARP requests >may be generated, causing the ARP table to grow very large with "incomplete" >entries. These entries consume memory. As the amount of free memory >declines and demand on the processor to handle outstanding packets >increases, ARP processing falls behind and throughput on the router may >decline significantly. Once the scan is stopped, processing catches up and >things return to normal. > > >So, to my knowledge IOS should be doing the right thing, we only queue one >ARP request at a time, every 2 seconds, until the ARP entry is resolved, we >rate limit requests, dropping all additional packets, until the ARP entry is >resolved, and we clean up the outstanding incomplete requests within a few >minutes. > > >I hope that helps address some of the concerns put forth. Hopefully we will >have further updates shortly. > > >Thank you, > > >Lisa Napier >Product Security Incident Response Team >Cisco Systems >___________ > >_______________ > >-----Original Message----- >From: khollis [SMTP:khollisat_private] >Sent: Wednesday, September 15, 1999 11:31 AM >To: wescotdat_private >Subject: Regarding Case Number V44290 > >Hello Dave, I've done some testing here with Nmap. I was able to create a >test bed that can cause problems & symptoms similar to those you reported. >But in summary, the router is functioning normally & depending on the >network topology the behavior you experienced would be expected. > > >From show processor memory, the ip input process is the process that >maintains the ip fast switching cache. This fast switching cache is used >when forwarding packets to avoid interrupting the cpu for repetitive route >table lookups. The key issue is the behavior of the fast cache and the way >it gets built. > >There are a number of scenarios that will cause the fast cache to install an >entry that essentially looks like a host route. For instance, with only 1 >path to a destination, we will install an entry into the fast cache that >covers the entire network. Example: 100.0.0.0/8. However, when multiple >equal cost paths to a destination exist, we will install an entry into the >fast cache for each destination. Example: 100.0.0.1/32, 100.1.1.1/32, >100.2.2.2/32...and so on. This helps ensure load balancing. Additionally, >depending on whether routes are directly connected, and/or subnetted, or the >next hop of a static route, the results can vary. > >When running Nmap & scanning every address in a class A ip network, if >conditions warrant the installation of a /32 entry into the fast cache this >would allow the fast cache to consume a tremendous amount of memory and in >that scenario all available dram could be consumed. This creates additional >problems because there isn't enough memory to support other features on the >router. > >Since Nmap isn't a normal application ran on networks, this issue isn't a >concern in most networking environments. However, if this is a major concern >you could run CEF (Cisco Express Forwarding). The behavior I just explained >does not occur when running CEF. But you will need to run 12.0 on the Cat5 >RSM. Other workarounds such as disabling fast switching (no ip route-cache) >or using max-paths 1 aren't really feasible. CEF is the better solution. > >Thanks, >KennyH. > >_________________
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 15:05:01 PDT