Re: cisco 7200 performance issue

From: Wendy Garvin (wgarvinat_private)
Date: Fri May 23 2003 - 01:30:51 PDT

  • Next message: Jerry Shenk: "RE: Possible Intrusion Attempt?"

    -----BEGIN PGP SIGNED MESSAGE-----
    Hash: SHA1
    
    
    Hi Luciano,
    
    High CPU can be caused by a high volume of traffic destined to the processor
    on your 7200. The "%RCMD-4-RSHPORTATTEMPT:" message is commonly seen if you
    are getting scanned by Nessus, or an nmap based scanner. Your CPU
    utilization is not necessarily related to the rsh attempt specifically, but
    it could be related to the scanning activity as a whole.
    
    To narrow this down further, you can put up an input access list on all your
    interfaces with permit statements for specific types of traffic, and monitor
    the 'hit' counts with the 'show access-lists' command. You may also try
    specifically blocking rsh traffic just to verify that it's not the rsh
    connection attempts that are causing the high CPU.
    
    WARNING! Be careful not to cut off your remote access when applying an input
    access-list.
    
    An example input access list on a router with a interface address of 11.0.0.3
    might be:
    - - --
    !!These will help you characterize the scan
    access-list 101 permit tcp any host 11.0.0.3 eq cmd
    access-list 101 permit tcp any host 11.0.0.3 eq login
    access-list 101 permit tcp any host 11.0.0.3 eq nntp
    access-list 101 permit tcp any host 11.0.0.3 eq lpd
    access-list 101 permit tcp any host 11.0.0.3 eq finger
    access-list 101 permit tcp any host 11.0.0.3 eq echo
    access-list 101 permit tcp any host 11.0.0.3 eq chargen log-input
    !! These will help you identify what services you are running
    access-list 101 permit tcp any host 11.0.0.3 eq ftp
    access-list 101 permit tcp any host 11.0.0.3 eq telnet
    access-list 101 permit tcp any host 11.0.0.3 eq bgp
    access-list 101 permit ospf any host 11.0.0.3 
    access-list 101 permit ospf any host 224.0.0.6
    access-list 101 permit eigrp any host 224.0.0.10
    access-list 101 permit eigrp any host 11.0.0.3
    access-list 101 permit udp any host 11.0.0.3 eq snmp
    access-list 101 permit udp any host 11.0.0.3 eq ntp
    !!This will not allow you to identify all critical services
    !!Some you may still need to look for are icmp ttl exceeded
    !!and ftp/tftp for upgrades.
    access-list 101 permit ip any any
    
    
    interface fastethernet0/0
    ip access-group 101 in
    - - --
    
    The output of 'sh access-list 101' after a default namp scan might be:
    
    Router#show access-list 101
    Extended IP access list 101
        permit tcp any host 11.0.0.3 eq cmd (3 matches)
        permit tcp any host 11.0.0.3 eq login (3 matches)
        permit tcp any host 11.0.0.3 eq nntp (3 matches)
        permit tcp any host 11.0.0.3 eq lpd (3 matches)
        permit tcp any host 11.0.0.3 eq finger (9 matches)
        permit tcp any host 11.0.0.3 eq echo (10 matches)
        permit tcp any host 11.0.0.3 eq chargen log-input (2 matches)
        permit tcp any host 11.0.0.3 eq ftp (5 matches)
        permit tcp any host 11.0.0.3 eq telnet (10 matches)
        permit tcp any host 11.0.0.3 eq bgp (3 matches)
        permit ospf any host 11.0.0.3
        permit ospf any host 224.0.0.6
        permit eigrp any host 224.0.0.10
        permit eigrp any host 11.0.0.3
        permit udp any host 11.0.0.3 eq snmp
        permit udp any host 11.0.0.3 eq ntp
        permit ip any any (4867 matches)
    
    And in your log, you may see this:
    
    Dec  6 16:21:51.350: %SEC-6-IPACCESSLOGP: list 101 permitted tcp
    11.0.0.19(35265) (FastEthernet3/0 0002.559c.f66e) -> 11.0.0.3(19), 1 packet
    
    I deliberately chose 'chargen' as a service you aren't likely to have
    legitimate connections too, as 'log-input' can cause further increases in
    your CPU utilization. This gives you the source/destination of this packet.
    
    If you don't get immediate results, try replacing the address 11.0.0.3 in
    your access-list with your loopback address, and try placing the access-list
    on various input interfaces until you track down the source of the scan. The
    command 'show tcp brief all' may also be useful to you.
    
    At this point you can be fairly certain someone is scanning hosts on your
    network. It would be useful to know whether they're scanning your routers
    specifically or if they're hitting every host that answers. You may be able
    to track the source of the scan by following the traffic from the input
    interface back to the next hop router.
    
    If you continue to have high CPU utilization on your router, you may
    consider placing input access-lists on your interfaces which protect your
    router from traffic destined to the processor. The above access-list should
    help you recognize critical services destined to your router, now you must
    write an access list to permit only those from legitimate sources, and block
    everything else destined to the router.
    
    For example, if you are using ssh to connect to your routers from
    11.0.0.0/24, and the only other legitimate service you need to connect to
    the router is BGP from a directly connected peer, you could apply the
    following access-list:
    
    access-list 101 permit tcp host 11.0.0.2 host 11.0.0.3 eq bgp
    access-list 101 permit tcp  11.0.0.0 0.0.0.255 host 11.0.0.3 eq 22
    access-list 101 deny ip any any
    
    Please carefully review the traffic currently configured to connect to your
    router before applying such an access-list, you may disconnect critical
    services or block yourself from managing the router if you do not plan the
    access-list correctly.
    
    Please feel free to contact psirtat_private for any security concerns you
    have with Cisco products. You may also open up a case with our Technical
    Assistance Center by sending email to tacat_private
    
    Thanks,
    
    - --Wendy
    
    > Luciano Z <user_lucianoat_private> [2003-05-21 16:45] wrote:
    > 
    > Hi!
    > 
    > I was responding an incident last night and saw a
    > strange performance problem with a cisco 7200.
    > 
    > When I issued a "sh interface" on the two fast
    > ethernets of my box it was show that I got only 6Mbps
    > traffic and normal packet per second rate but when I
    > "sh logg" the box I got a lot of
    > "%RCMD-4-RSHPORTATTEMPT: Attempted to connect to
    > RSHELL from x.y.z.w" messages with spoofed sources.
    > 
    > Investigating a little more I discovered that this
    > traffic was pushing the CPU to 98% to 100% of
    > utilization. Back to the output of "sh logg" I saw
    > that the box was logging 2 to 3 RSHELL messages per
    > second. In my opinion this coulnd?t affect the CPU so
    > much. The router have 256M of RAM and it?s a 7200!
    > 
    > I coulnd?t gather more info about this incident
    > because it stopped before I could get the data. The
    > strange thing it?s that the high CPU utilization
    > stopped too.
    > 
    > I don?t know if this is a problem of this cisco model
    > or if I?m missing something. Any ideias?
    > 
    > []
    > lwulff
    > 
    > _______________________________________________________________________
    > Yahoo! Mail
    > O melhor e-mail gratuito da internet: 6MB de espa?o, antiv?rus, acesso POP3, filtro contra spam. 
    > http://br.mail.yahoo.com/
    > 
    > ----------------------------------------------------------------------------
    > *** Wireless LAN Policies for Security & Management - NEW White Paper ***
    > Just like wired networks, wireless LANs require network security policies 
    > that are enforced to protect WLANs from known vulnerabilities and threats. 
    > Learn to design, implement and enforce WLAN security policies to lockdown enterprise WLANs.
    > 
    > To get your FREE white paper visit us at:    
    > http://www.securityfocus.com/AirDefense-incidents
    > ----------------------------------------------------------------------------
    > 
    > 
    
    - - -- 
    Wendy Garvin - Cisco PSIRT - 408 525-1888 CCIE# 6526
    - - ----------------------------------------------------
               http://www.cisco.com/go/psirt
    
    
    -----BEGIN PGP SIGNATURE-----
    Version: PGP 6.5.2
    
    iQA/AwUBPs3b0M/6vhuARK9tEQLCwQCgwWSjs38+slcpABQLutgKXjErkJ0AniQr
    chJvuGNR13SLLnuBI0rjr2dX
    =2Nyl
    -----END PGP SIGNATURE-----
    
    ----------------------------------------------------------------------------
    *** Wireless LAN Policies for Security & Management - NEW White Paper ***
    Just like wired networks, wireless LANs require network security policies 
    that are enforced to protect WLANs from known vulnerabilities and threats. 
    Learn to design, implement and enforce WLAN security policies to lockdown enterprise WLANs.
    
    To get your FREE white paper visit us at:    
    http://www.securityfocus.com/AirDefense-incidents
    ----------------------------------------------------------------------------
    



    This archive was generated by hypermail 2b30 : Fri May 23 2003 - 10:33:32 PDT