Hi all, An update to this issue: we have been able to reproduce the problem in our labs, but only under specific conditions. At this point, the customer has not been able to confirm or deny the configuration items in effect during this problem. Essentially what we found was that if fast switching was in use, and if there are multiple equal cost routes for the same destination, the router will install host routes for each destination to ensure load balancing across equal cost paths. Under these conditions, scanning an entire class A network will use up all of the routers memory in short order. To avoid this problem, I would recommend using CEF (Cisco Express Forwarding) which handles equal cost paths differently, and more efficiently than the fast switching model detailed above. CEF is available in IOS version 12.0 for most platforms. Thank you, Lisa Napier Product Security Incident Response Team Cisco Systems At 07:12 PM 9/7/1999 -0700, Lisa Napier wrote: >Hi all, > >Sorry for the delay in response. > >At first glance, I had thought that this security advisory may have had >something to do with this issue. > >http://cco/warp/customer/770/iossyslog-pub.shtml > >This details a specific issue with the routers response to an invalid UDP >packet to the syslog port, however your report details strictly ICMP scan >and tcp port scans. > >In speaking with the engineer who is working the case, he was unable to >view the issue 'live'; the people running NMAP had turned it off, just as >he had logged into your network, and functionality was pretty much restored. > >We have not yet been able to reproduce the problem in house, but are >still working on it. > >The customer support engineer working the case is trying to cause the >problems you saw with the scan parameters that you specified, in that it >was only scanning hosts that responded to a ping. If we presume that the >scan was actually attempting to scan an entire class A network, then the >behavior seen of maxing out the input queues and therefore memory >resources is an expected and nasty side effect, and we have indeed seen >that behavior. > >Cisco's product security incident response team is monitoring the case at >this point, and expecting an update within the next few days. > >For those on the list unaware of us, the Cisco PSIRT is the best point of >contact for reporting a security issue with any Cisco product. The URL >below gives more details on the Cisco PSIRT. > >Thanks, > >Lisa Napier >Product Security Incident Response Team >Cisco Systems > >408 527-8747 > >http://www.cisco.com/warp/public/707/sec_incident_response.shtml > > >At 05:02 PM 8/31/1999 -0700, Lancashire, Andrew wrote: > >I don't know if you've ever seen this before. We ran nmap with ICMP > >discover and standard tcp scan. We ran the scan against the entire 10.0.0.0 > >network range. Although we were only looking for 2 ports, we found that the > >RSM in our 5500 series (our default route) was running out of memory and > >had to be rebooted by our Network Services group multiple times in the 18 > >hour stretch it took to complete. One of the interesting things is that we > >were only generating about 3-5 Mbs and the 5500 can pass Gigabits. I have > >not heard of this problem before. We contacted Cisco and sent them the > >details. Below is the response to one of our engineers. > > > >Andrew > > > >-----Original Message----- > >From: khollis [SMTP:khollisat_private] > >Sent: Tuesday, August 31, 1999 7:59 AM > >To: wescotdat_private > >Subject: Regarding Case Number V44290 > > > >Hi Dave, as I recall, the symptom we had to work/troubleshoot with was the > >router consumed lots of memory. Never heard about packets being dropped. So > >it seems like we forwarded everything nmap sent to us. The thing to keep in > >mind is that the router will dynamically allocate memory as necessary so > >that it can keep up with the load provided to it. Although we did not know > >nmap was running at the time, we noticed the memory consumed by the IP Input > >process dropped from 40M+ to an acceptable level of (4-5M) after nmap was > >shut down. This proves that the router need this much memory to process the > >entire load generated by nmap. > > > >I suspect nmap was doing much more than you've been able to calculate. It's > >obvious that running nmap continuously for 18-19 hours caused this problem. > >One possible explaination is constantly flooding the router w/64 byte > >packets for this timeframe could have caused the router's memory to become > >seriously fragmented. Also, I guess we can't tell, but another question > >would be how many tcp sessions were requested/open on the router after this > >timeframe? > > > >Port scanners have a reputation of helping identify potential security > >problems. However, they are also known to cause problems... > > > >Hope this helps, > >KennyH.
This archive was generated by hypermail 2b30 : Fri Apr 13 2001 - 15:04:10 PDT