This is to clarify what is being put out by Cisco and what we are being told by Cisco. Two e-mails below is what Cisco is telling us and makes allot more sense than what Cisco is telling Bugtraq. The last post to Bugtraq made mention that the arp cache was filling up and allocating memory for both reachable hosts and unreachable hosts (incompletes). Although what Lisa describes is true regarding the arp cache, it would not be true for our or most other sane persons environment. Since routers will only arp for what is local, that would mean that for the arp cache to fill up and us all the memory all networks in the 10.x.x.x range would need to be local. So that's not gonna happen but if you read the e-mail below that from Kenny (also at Cisco ) his explanation makes allot more sense considering we have hundreds of routers. Thank You Andrew P.S. Congratulations on the re-opening of PacketStorm ___________________________________ Subject: Re: Cisco and Nmap Dos To: BUGTRAQat_private Hi all, I wanted to address the items listed here. We are still investigating this problem, and hope to have more information later on in the week. Item 1, OSPF is not an issue. According to the configuration information provided to us by the customer, OSPF is not in use. Items 2, 3 & 4. IOS actually handles ARP in the following manner: When we receive a packet destined for something not already in our ARP table, we enter an "incomplete" entry in the ARP table. Then we will rate limit ARPs to once every 2 seconds to that destination. Any additional packets to that same destination will be dropped until the ARP entry is completed. This is on a per destination basis. "Incompletes" (ARP requests that have not been responded to) get dropped after approximately 1 minute from the last time we sent an ARP request for that non existent address. Incomplete entries MAY stay around longer, as the process that is responsible for cleaning up the ARP table may not have enough time to complete before it is called again, if we have a lot to clean up, which may be relevant to this case. The incomplete entries will eventually get cleaned up, but it may take two or three minutes, two or three cycles of the process that cleans up the table. Under a dedicated, intense nmap scan, a very large number of ARP requests may be generated, causing the ARP table to grow very large with "incomplete" entries. These entries consume memory. As the amount of free memory declines and demand on the processor to handle outstanding packets increases, ARP processing falls behind and throughput on the router may decline significantly. Once the scan is stopped, processing catches up and things return to normal. So, to my knowledge IOS should be doing the right thing, we only queue one ARP request at a time, every 2 seconds, until the ARP entry is resolved, we rate limit requests, dropping all additional packets, until the ARP entry is resolved, and we clean up the outstanding incomplete requests within a few minutes. I hope that helps address some of the concerns put forth. Hopefully we will have further updates shortly. Thank you, Lisa Napier Product Security Incident Response Team Cisco Systems ___________ _______________ -----Original Message----- From: khollis [SMTP:khollisat_private] Sent: Wednesday, September 15, 1999 11:31 AM To: wescotdat_private Subject: Regarding Case Number V44290 Hello Dave, I've done some testing here with Nmap. I was able to create a test bed that can cause problems & symptoms similar to those you reported. But in summary, the router is functioning normally & depending on the network topology the behavior you experienced would be expected. >From show processor memory, the ip input process is the process that maintains the ip fast switching cache. This fast switching cache is used when forwarding packets to avoid interrupting the cpu for repetitive route table lookups. The key issue is the behavior of the fast cache and the way it gets built. There are a number of scenarios that will cause the fast cache to install an entry that essentially looks like a host route. For instance, with only 1 path to a destination, we will install an entry into the fast cache that covers the entire network. Example: 100.0.0.0/8. However, when multiple equal cost paths to a destination exist, we will install an entry into the fast cache for each destination. Example: 100.0.0.1/32, 100.1.1.1/32, 100.2.2.2/32...and so on. This helps ensure load balancing. Additionally, depending on whether routes are directly connected, and/or subnetted, or the next hop of a static route, the results can vary. When running Nmap & scanning every address in a class A ip network, if conditions warrant the installation of a /32 entry into the fast cache this would allow the fast cache to consume a tremendous amount of memory and in that scenario all available dram could be consumed. This creates additional problems because there isn't enough memory to support other features on the router. Since Nmap isn't a normal application ran on networks, this issue isn't a concern in most networking environments. However, if this is a major concern you could run CEF (Cisco Express Forwarding). The behavior I just explained does not occur when running CEF. But you will need to run 12.0 on the Cat5 RSM. Other workarounds such as disabling fast switching (no ip route-cache) or using max-paths 1 aren't really feasible. CEF is the better solution. Thanks, KennyH. _________________
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 15:04:54 PDT