There could be a couple of problems that I know of... The problem(s) here may be 1) OSPF : Remembering every single IP as separate routes. I don't know the timeouts or memory usage for OSPF routes, so I won't try to dig to deep into that. This should be easily verifiable by just checking the routing table on the router, should it be the case. 2) ARP resolution : Any number of packets to the same IP may be queued, instead of just one per IP which is the sane thing to do. 3) ARP resolution : There are no checks to see if there's a huge number of requests waiting. If there is, the router should just kill old requests. 4) ARP resolution : The router remembers not only addresses that have been found, but also addresses that could NOT be found. This way, it doesn't keep retrying several times per second but rather once every 10-30 seconds (if someone still wants to talk to that host, of course). Here, the problem may be that it doesn't delete old "not found" records if there's a disproportionate amount of them. The most probable one is number one. If you don't want the gory details in this discussion, skip to </boringtechstuff>. The conclusion is more interesting than the math. <boringtechstuff> Let's do some math on the ARP part: Assume that all hosts are local to the router. It took you 18 hours to complete 256^3 = 16M addresses. 16M hosts in 18 hours (64800 secs) means 250 hosts scanned per second. This would amount to app. 100 kbps load (assuming each host is only pinged once). This doesn't make any sense, but I'll go ahead anyway :-) First we look at the case where packets are buffered. Assume the router buffers packets with unknown destination for one second. Each buffer would be 14 ether + 20 ip + 8 icmp + 4 data = 46 bytes, plus internal data such as mbuf-equivalent links. Let's assume each buffer eats 64 bytes. 64 * 250 = 16K used on average. Hmm this does not seem to be the case. Even assuming nmap pings four times per host and that each packet is buffered for three seconds, it would only amount to 192K. Let's look at remembering ARP entries that are not found. Assume each ARP entry eats 4 ip + 6 ether + 2 timeout + 2 flags + 2 bytes proto type + 2 byte hw type + ~8 bytes hash bucket linkage (or something) = 26 bytes. Assume each entry lives for 30 seconds. 250 * 30 * 26 = 195K. Even with IPv6 address size allocation and long timeouts, it doesn't amount to more than a couple of megs. Ack.. Even with both combined, it doesn't even come near to 35 MB. A quick look at the OSPF problem would say so la la 20 bytes per route, route life time of 1 hour: 1 million hosts scanned per hour, 20 bytes per route, gives an AVERAGE OF TWENTY MB RAM USED. This would seem more probable. By starting a scan and seeing how the memory grows, you should see that it keeps growing for maybe 30 minutes, maybe 1 hour, maybe 2 hours (route life) and then stays at the same level. When you stop scanning, it should take the same time for the memory usage to decrease to normal levels. </boringtechstuff> Either there's a flaw in the routing kernel that I have no idea about, or the problem is OSPF. As I said, you should be able to verify OSPF behaviour either by checking the routing table from the console, or polling it via SNMP. I've noticed on some brands that the console only displays static and RIPped routes, but that SNMP displays all; keep that in mind. You should be able to amend this problem by adding static routes without destination for IP spans known to be empty. Hope this helps? Regards, Mikael Olsson "Lancashire, Andrew" wrote: > > I don't know if you've ever seen this before. We ran nmap with ICMP > discover and standard tcp scan. We ran the scan against the entire 10.0.0.0 > network range. Although we were only looking for 2 ports, we found that the > RSM in our 5500 series (our default route) was running out of memory and > had to be rebooted by our Network Services group multiple times in the 18 > hour stretch it took to complete. One of the interesting things is that we > were only generating about 3-5 Mbs and the 5500 can pass Gigabits. I have > not heard of this problem before. We contacted Cisco and sent them the > details. Below is the response to one of our engineers. > > Andrew > > -----Original Message----- > From: khollis [SMTP:khollisat_private] > Sent: Tuesday, August 31, 1999 7:59 AM > To: wescotdat_private > Subject: Regarding Case Number V44290 > > Hi Dave, as I recall, the symptom we had to work/troubleshoot with was the > router consumed lots of memory. Never heard about packets being dropped. So > it seems like we forwarded everything nmap sent to us. The thing to keep in > mind is that the router will dynamically allocate memory as necessary so > that it can keep up with the load provided to it. Although we did not know > nmap was running at the time, we noticed the memory consumed by the IP Input > process dropped from 40M+ to an acceptable level of (4-5M) after nmap was > shut down. This proves that the router need this much memory to process the > entire load generated by nmap. > > I suspect nmap was doing much more than you've been able to calculate. It's > obvious that running nmap continuously for 18-19 hours caused this problem. > One possible explaination is constantly flooding the router w/64 byte > packets for this timeframe could have caused the router's memory to become > seriously fragmented. Also, I guess we can't tell, but another question > would be how many tcp sessions were requested/open on the router after this > timeframe? > > Port scanners have a reputation of helping identify potential security > problems. However, they are also known to cause problems... > > Hope this helps, > KennyH. -- Mikael Olsson, EnterNet Sweden AB, Box 393, S-891 28 ÖRNSKÖLDSVIK Phone: +46-(0)660-105 50 Fax: +46-(0)660-122 50 WWW: http://www.enternet.se E-mail: mikael.olssonat_private
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 15:01:38 PDT