Hi all, I wanted to address the items listed here. We are still investigating this problem, and hope to have more information later on in the week. Item 1, OSPF is not an issue. According to the configuration information provided to us by the customer, OSPF is not in use. Items 2, 3 & 4. IOS actually handles ARP in the following manner: When we receive a packet destined for something not already in our ARP table, we enter an "incomplete" entry in the ARP table. Then we will rate limit ARPs to once every 2 seconds to that destination. Any additional packets to that same destination will be dropped until the ARP entry is completed. This is on a per destination basis. "Incompletes" (ARP requests that have not been responded to) get dropped after approximately 1 minute from the last time we sent an ARP request for that non existent address. Incomplete entries MAY stay around longer, as the process that is responsible for cleaning up the ARP table may not have enough time to complete before it is called again, if we have a lot to clean up, which may be relevant to this case. The incomplete entries will eventually get cleaned up, but it may take two or three minutes, two or three cycles of the process that cleans up the table. Under a dedicated, intense nmap scan, a very large number of ARP requests may be generated, causing the ARP table to grow very large with "incomplete" entries. These entries consume memory. As the amount of free memory declines and demand on the processor to handle outstanding packets increases, ARP processing falls behind and throughput on the router may decline significantly. Once the scan is stopped, processing catches up and things return to normal. So, to my knowledge IOS should be doing the right thing, we only queue one ARP request at a time, every 2 seconds, until the ARP entry is resolved, we rate limit requests, dropping all additional packets, until the ARP entry is resolved, and we clean up the outstanding incomplete requests within a few minutes. I hope that helps address some of the concerns put forth. Hopefully we will have further updates shortly. Thank you, Lisa Napier Product Security Incident Response Team Cisco Systems At 04:04 PM 9/2/1999 +0200, Mikael Olsson wrote: >There could be a couple of problems that I know of... > >The problem(s) here may be >1) OSPF : Remembering every single IP as separate routes. > I don't know the timeouts or memory usage for OSPF > routes, so I won't try to dig to deep into that. This should be > easily verifiable by just checking the routing table on > the router, should it be the case. >2) ARP resolution : Any number of packets to the same IP may be > queued, instead of just one per IP which is the sane thing to do. >3) ARP resolution : There are no checks to see if there's a huge > number of requests waiting. If there is, the router should just > kill old requests. >4) ARP resolution : The router remembers not only addresses that have > been found, > but also addresses that could NOT be found. This way, it > doesn't keep retrying several times per second but rather > once every 10-30 seconds (if someone still wants to talk to that > host, of course). > Here, the problem may be that it doesn't delete old "not found" > records if there's a disproportionate amount of them. > >The most probable one is number one. > >If you don't want the gory details in this discussion, >skip to </boringtechstuff>. >The conclusion is more interesting than the math. > ><boringtechstuff> > > >Let's do some math on the ARP part: > >Assume that all hosts are local to the router. >It took you 18 hours to complete 256^3 = 16M addresses. >16M hosts in 18 hours (64800 secs) means 250 hosts scanned per second. > >This would amount to app. 100 kbps load (assuming each host is only >pinged once). This doesn't make any sense, but I'll go ahead anyway :-) > >First we look at the case where packets are buffered. > >Assume the router buffers packets with unknown destination for >one second. >Each buffer would be 14 ether + 20 ip + 8 icmp + 4 data = 46 >bytes, plus internal data such as mbuf-equivalent links. Let's >assume each buffer eats 64 bytes. >64 * 250 = 16K used on average. >Hmm this does not seem to be the case. Even assuming nmap pings >four times per host and that each packet is buffered for >three seconds, it would only amount to 192K. > >Let's look at remembering ARP entries that are not found. >Assume each ARP entry eats 4 ip + 6 ether + 2 timeout + 2 flags >+ 2 bytes proto type + 2 byte hw type + ~8 bytes hash bucket >linkage (or something) = 26 bytes. >Assume each entry lives for 30 seconds. >250 * 30 * 26 = 195K. >Even with IPv6 address size allocation and long timeouts, >it doesn't amount to more than a couple of megs. > >Ack.. > >Even with both combined, it doesn't even come near to 35 MB. > >A quick look at the OSPF problem would say >so la la 20 bytes per route, route life time of 1 hour: >1 million hosts scanned per hour, 20 bytes per route, >gives an AVERAGE OF TWENTY MB RAM USED. > >This would seem more probable. > >By starting a scan and seeing how the memory grows, you >should see that it keeps growing for maybe 30 minutes, >maybe 1 hour, maybe 2 hours (route life) and then stays >at the same level. >When you stop scanning, it should take the same time >for the memory usage to decrease to normal levels. > ></boringtechstuff> > >Either there's a flaw in the routing kernel that I have no >idea about, or the problem is OSPF. > >As I said, you should be able to verify OSPF behaviour either >by checking the routing table from the console, or polling >it via SNMP. I've noticed on some brands that the console >only displays static and RIPped routes, but that SNMP >displays all; keep that in mind. > >You should be able to amend this problem by adding static >routes without destination for IP spans known to be empty. > >Hope this helps? >Regards, >Mikael Olsson > > >"Lancashire, Andrew" wrote: > > > > I don't know if you've ever seen this before. We ran nmap with ICMP > > discover and standard tcp scan. We ran the scan against the entire 10.0.0.0 > > network range. Although we were only looking for 2 ports, we found that the > > RSM in our 5500 series (our default route) was running out of memory and > > had to be rebooted by our Network Services group multiple times in the 18 > > hour stretch it took to complete. One of the interesting things is that we > > were only generating about 3-5 Mbs and the 5500 can pass Gigabits. I have > > not heard of this problem before. We contacted Cisco and sent them the > > details. Below is the response to one of our engineers. > > > > Andrew > > > > -----Original Message----- > > From: khollis [SMTP:khollisat_private] > > Sent: Tuesday, August 31, 1999 7:59 AM > > To: wescotdat_private > > Subject: Regarding Case Number V44290 > > > > Hi Dave, as I recall, the symptom we had to work/troubleshoot with was the > > router consumed lots of memory. Never heard about packets being dropped. So > > it seems like we forwarded everything nmap sent to us. The thing to keep in > > mind is that the router will dynamically allocate memory as necessary so > > that it can keep up with the load provided to it. Although we did not know > > nmap was running at the time, we noticed the memory consumed by the IP Input > > process dropped from 40M+ to an acceptable level of (4-5M) after nmap was > > shut down. This proves that the router need this much memory to process the > > entire load generated by nmap. > > > > I suspect nmap was doing much more than you've been able to calculate. It's > > obvious that running nmap continuously for 18-19 hours caused this problem. > > One possible explaination is constantly flooding the router w/64 byte > > packets for this timeframe could have caused the router's memory to become > > seriously fragmented. Also, I guess we can't tell, but another question > > would be how many tcp sessions were requested/open on the router after this > > timeframe? > > > > Port scanners have a reputation of helping identify potential security > > problems. However, they are also known to cause problems... > > > > Hope this helps, > > KennyH. > >-- >Mikael Olsson, EnterNet Sweden AB, Box 393, S-891 28 ÖRNSKÖLDSVIK >Phone: +46-(0)660-105 50 Fax: +46-(0)660-122 50 >WWW: http://www.enternet.se E-mail: mikael.olssonat_private
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 15:02:50 PDT