Monday, March 27, 2017

Cisco Switch 2960x Memory Increasing Issue Troubleshooting - Memory Leak

Symptoms 

Our network monitoring software found memory usage on some new production switches keep increasing. Those switches are Cisco 2960X and coming with 15.0(2) EX3 IOS.

As we know there are two types of memory in Cisco IOS: process memory and IO memory.
  • When a feature is enable on an IOS device (e.g. PIM, HSRP, and etc), IOS allocates process memory for the process.
  • IO memory is used when a software switched traffic hits the CPU. The CPU allocates IO memory to store the frame temporarily.
Our case is process memory increasing. What we need to do is to find out which process.




Switch1# show version 
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2)EX3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Wed 11-Sep-13 02:04 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.0(2r)EX, RELEASE SOFTWARE (fc1)

CSWP3024 uptime is 5 days, 23 hours, 48 minutes
System returned to ROM by power-on
System restarted at 05:47:41 EST Sat Mar 11 2017
System image file is "flash:/c2960x-universalk9-mz.150-2.EX3/c2960x-universalk9-mz.150-2.EX3.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-48TS-LL (APM86XXX) processor (revision B0) with 262144K bytes of memory.
Processor board ID FOC1744W3V5
Last reset from power-on
2 Virtual Ethernet interfaces
1 FastEthernet interface
50 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 5C:A4:8A:5C:6F:80
Motherboard assembly number     : 73-14196-03
Power supply part number        : 341-0537-02
Motherboard serial number       : FOC17443CWD
Power supply serial number      : LIT17320EU6
Model revision number           : B0
Motherboard revision number     : B0
Model number                    : WS-C2960X-48TS-LL
Daughterboard assembly number   : 73-14200-03
Daughterboard serial number     : FOC17440VRF
System serial number            : FOC1744W3V5
Top Assembly Part Number        : 800-38011-02
Top Assembly Revision Number    : B0
Version ID                      : V02
CLEI Code Number                : CMMNG00ARA
Daughterboard revision number   : A0
Hardware Board Revision Number  : 0x05


Switch Ports Model              SW Version            SW Image                 
------ ----- -----              ----------            ----------               
*    1 50    WS-C2960X-48TS-LL  15.0(2)EX3            C2960X-UNIVERSALK9-M     


Configuration register is 0xF



Switch1#show memory statistics history 
------------------ History of Processor Mempool ------------------

                                                                  
                                                                  
                                                                  
      777777777777777777777777777777777777777777777777777777777777
      222222222222222222222222222222222222222222222222222222222222
  100                                                           
   90                                                           
   80                                                           
   70 **********************************************************
   60 **********************************************************
   50 **********************************************************
   40 **********************************************************
   30 **********************************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per second (last 60 seconds)

                                                                  
                                                                  
                                                                  
      777777777777777777777777777777777777777777777777777777777777
      222222222222222222222222222222222222222222222222222222222222
  100                                                           
   90                                                           
   80                                                           
   70 ##########################################################
   60 ##########################################################
   50 ##########################################################
   40 ##########################################################
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

                                                                              
                                                                              
                                                                              
      777777777777777777777777777777777777777777777777777777777777777777777777
      222222222233333333333333444444444444444555555555555555666666666666667777
  100                                                                       
   90                                                                       
   80                                        **#############################
   70 ######################################################################
   60 ######################################################################
   50 ######################################################################
   40 ######################################################################
   30 ######################################################################
   20 ######################################################################
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0  
                   Free memory per hour (last 72 hours)
                  * = maximum # = average


------------------ History of I/O Mempool ------------------

                                                                  
                                                                  
                                                                  
      555555555555555555555555555555555555555555555555555555555555
      888888888888888888888888888888888888888888888888888888888888
  100                                                           
   90                                                           
   80                                                           
   70                                                           
   60 **********************************************************
   50 **********************************************************
   40 **********************************************************
   30 **********************************************************
   20 **********************************************************
   10 **********************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per second (last 60 seconds)

                                                                  
                                                                  
                                                                  
      555555555555555555555555555555555555555555555555555555555555
      888888888888888888888888888888888888888888888888888888888888
  100                                                           
   90                                                           
   80                                                           
   70                                                           
   60 ##########################################################
   50 ##########################################################
   40 ##########################################################
   30 ##########################################################
   20 ##########################################################
   10 ##########################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               Free memory per minute (last 60 minutes)
              * = maximum # = average

                                                                              
                                                                              
                                                                              
      555555555555555555555555555555555555555555555555555555555555555555555555
      888888888888888888888888888888888888888888888888888888888888888888888888
  100                                                                       
   90                                                                       
   80                                                                       
   70                                                                       
   60 ######################################################################
   50 ######################################################################
   40 ######################################################################
   30 ######################################################################
   20 ######################################################################
   10 ######################################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0  
                   Free memory per hour (last 72 hours)
                  * = maximum # = average



From our management software , the free memory diagram shows it is continuous reducing until whole system lost responding .

Free Switch Memory Going Down in Monitoring Tools

Troubleshooting

There are two main reasons why a switch runs out of memory.

1. A process does not free up the memory after the memory is no longer needed. This is generally known as memory leak.

2. A process does not limit the amount of memory it allocates. Eventually, the process uses up all process memory on the switch.

The goal is to identify the misbehaving process. There are two approaches.
  •  Identify any process leaking memory
Show process memory debug leak  

Notes: This command only to be run during maintenance window or when the device is not in production .This command is handy to identify any process leaking memory. However, it does not help if the memory problem is caused by a process grabs all memory
  • Identify a process grabs all memory 


Potential fix:

After googled and I did find many related Cisco bug and suggestions.

1.  Cisco Bug: CSCut06609 - Memory leak seen in Auth Manager process- leaks from ASP component (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut06609

Last Modified

Jan 31, 2017

Product

Cisco IOS

Known Affected Releases

15.0(2)SE5

Description (partial)

Symptom:
++ Very slow memory leak in "Auth Manager" process seen on 2960s/2960x  switches running 15.0(2)SE5.
++ Everyday we see about 64KB leak in Auth Manager process.
++ Leak is seen on a setup having no "Auth session" present (Only auth hidden sessions) and no ports running MAB or dot1X.  It is not same as CSCuq53377.
++ The leak is seen in "Auth Manager" process but it is actually being caused by ASP component.

Conditions:
"macro auto monitor" need  to be enabled globally, it could be seen even when no switch port is running MAB/dot1x.
The Fix below fixed all out problems so far:

Apparently, in the IOS there's an "Auth manager" that can monitor all
sessions in a switch. Starting from the 15.0 IOS stream this feature is
enabled by default. As the bug describes:
"Auth Manager continues to hold more memory in Processor Pool locking
out access to the 3750 stack unless switch is rebooted"

To prevent this 'leakage' of memory to the processor, you have to
disable this session monitoring by issuing the following command:

no macro auto monitor

Tried this, but did not help in this case.

2. Cisco Bug: CSCut92215 - Memory Leak in hrpc_incoming_request_handler_process - 15.0(2)EX5 (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut92215

Last Modified

Jan 23, 2017

Product

Cisco Catalyst 2960-X Series Switches

Known Affected Releases

15.0(2)EX

Description (partial)

Symptom:
Show process memory sorted will display MallocLite holding memory but no freeing it

SW#show process memory sorted

Processor Pool Total:   XXXXXXXX Used:   XXXXXXXX Free:    XXXXXXX
      I/O Pool Total:   XXXXXXXX Used:   XXXXXXXX Free:    XXXXXXX
Driver te Pool Total:    XXXXXXX Used:         XX Free:    XXXXXXX

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   XXXXXXXX   XXXXXXXX   XXXXXXXX          0          0 *Init*          
   0   0          X          0   XXXXXXXX          0          0 *MallocLite*  


//////////////////
show mem lite-chunk statistics will show a large disparity between Inuse and Free


SW#show mem lite-chunk statistics

      Pool       Inuse       Free
   8-Bytes         XXX       XXXX
  20-Bytes      XXXXXX       XXXX    <<<<<<
  44-Bytes        XXXX        XXX
  68-Bytes         XXX        XXX
  96-Bytes         XXX        XXX
 128-Bytes          XX        XXX

Conditions:
SW running 15.0(2)EX5

3. SNMP disable (Not our case)
Suggestion to run following three commands to reduce memory usage
- no macro auto monitor
- no cdp running
- snmp completely disabled
Tried those three commands / methods, but still did not help.
4. Memory leak with hulc_flash init process (Our case)

Switch1#show processes memory sorted | i Total
Processor Pool Total:  174442388 Used:   50534880 Free:  123907508
      I/O Pool Total:   16777216 Used:    6924400 Free:    9852816
Driver te Pool Total:    1048576 Used:         40 Free:    1048536
                                 57447240 Total

Switch1#show processes memory sorted | i Total
Processor Pool Total:  174442388 Used:   50661608 Free:  123780780
      I/O Pool Total:   16777216 Used:    6924400 Free:    9852816
Driver te Pool Total:    1048576 Used:         40 Free:    1048536
                                 57573808 Total


You can see Total processor Pool Free memory is decreasing, Free I/O and Driver te Pool memory keep same.

Check more details from sorted processes memory command, *Dead* process memory is increasing by comparing the holding memory data with the data collected 5 minutes ago.

Switch1#show processes memory sorted 
Processor Pool Total:  174442388 Used:   50519544 Free:  123922844
      I/O Pool Total:   16777216 Used:    6924400 Free:    9852816
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   59344172   24818044   31046036          0          0 *Init*          
   0   0          0          0   19543880          0          0 *MallocLite*    
   0   0   40461580   39067064    1530300    6881007     949166 *Dead*          
  87   0     715684     146228     451160          0          0 Stack Mgr Notifi
 289   0     267228          0     277388     100548          0 EEM ED Syslog   
   1   0     234876      36256     205780          0          0 Chunk Manager   
  68   0     206732        232     188500          0          0 USB Startup     
 205   0    1520792    1070688     154044          0          0 802.1x switch   
  65   0     108532        400     118292          0          0 EEM ED Identity 
 180   0     153020       8916     109604          0          0 DHCPD Receive   
 102   0      93356        212     100304      53760          0 HRPC emac reques
 153   1    3418792    3336728      94288          0          0 SSH Process     
 300   0      70460          0      77620          0          0 EEM Server      
 178   0   13629416        300      75732     802008          0 CDP Protocol    
 138   0      65812        232      72740          0          0 REP Helper Proc 
  53   0     108644      84688      71160          0          0 IF-MGR control p
 175   0     530220     373064      62440          0          0 Auth Manager    
 128   0        232       2204      61160          0          0 HACL Acl Manager
 159   0  499433728  499416280      61160      10152          0 HRPC qos request
 173   0      49364          0      56524          0          0 ACCT Periodic Pr
 280   0     237076     202296      48632          0          0 Crypto CA       
  21   0      48776      64728      45920          0       9620 Entity MIB API  
  17   0  211669728  211633028      43860          0          0 ARP Input       
 189   0      33020          0      40180          0          0 IP ARP Retry Age
 297   0      12408          0      37440          0          0 LICENSE AGENT   
 230   0      50872       9248      34396          0          0 CEF: IPv4 proces
   8   0    1547520    3356300      32952    1401600    2400103 Pool Manager    

Switch1#show processes memory sorted          
Processor Pool Total:  174442388 Used:   50659912 Free:  123782476
      I/O Pool Total:   16777216 Used:    6924400 Free:    9852816
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   59344172   24818044   31046036          0          0 *Init*          
   0   0          0          0   19675040          0          0 *MallocLite*    
   0   0   44037920   42642260    1531252    6881007     949166 *Dead*          
  87   0     715684     146228     451160          0          0 Stack Mgr Notifi
 289   0     267228          0     277388     100548          0 EEM ED Syslog   
   1   0     234876      36256     205780          0          0 Chunk Manager   
  68   0     206732        232     188500          0          0 USB Startup     
 205   0    1520792    1070688     154044          0          0 802.1x switch   
  65   0     108532        400     118292          0          0 EEM ED Identity 
 180   0     153020       8916     109604          0          0 DHCPD Receive   
 102   0      93356        212     100304      53760          0 HRPC emac reques
 153   1     193080     111648      93592          0          0 SSH Process     
 300   0      70460          0      77620          0          0 EEM Server      
 178   0   13658484        300      75732     802008          0 CDP Protocol    
 138   0      65812        232      72740          0          0 REP Helper Proc 
  53   0     108644      84688      71160          0          0 IF-MGR control p
 175   0     530220     373064      62440          0          0 Auth Manager    
 159   0  510696628  510679180      61160      10152          0 HRPC qos request
 128   0        232       2204      61160          0          0 HACL Acl Manager
 173   0      49364          0      56524          0          0 ACCT Periodic Pr
 280   0     237076     202296      48632          0          0 Crypto CA       
  21   0      48776      64728      45920          0       9620 Entity MIB API  
  17   0  212097396  212060696      43860          0          0 ARP Input       
 189   0      33020          0      40180          0          0 IP ARP Retry Age
 297   0      12408          0      37440          0          0 LICENSE AGENT   
 230   0      50872       9248      34396          0          0 CEF: IPv4 proces
 112   0    4349612        232      33052          0          0 HLFM address lea
   8   0    1547520    3356300      32952    1401600    2400103 Pool Manager    
 321   0     288460     157320      31420          0          0 SNMP Traps      
 199   0       5724          0      30884          0          0 SMI Backup Proce
 197   0       3160        592      28360          0          0 SMI Director DB 
  60   0      74608      52800      27056          0          0 Per-Second Jobs 
 306   0        232          0      25392          0          0 tHUB  






From the command 'show memory dead totals' output showed there were many instances of hulc_flash init.


Switch1#show memory dead totals 
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor    49B2FA0   174442388    50508688   123933700   123791240   103785944
      I/O    E000000    16777216     6924444     9852772     9736080     9805164
Driver te    2B00000     1048576          44     1048532     1048532     1048532

         

Dead Proc Summary for: Processor

    PC          Total   Count  Name
0x020CC35C     942068      47  mifs NOR Sector
0x0036E178     201252      20  SSH Process
0x020D35AC     100220       5  mifs sector buffers
0x01B31C10      65580       1  AAA Large Chunk
0x0017E2D4      64696     517  hulc_flash init
0x020D6A08      25220       5  mifs cache_entry buffers
0x020CC91C      20044       1  mifs NOR Sector Info Pointers
0x0204DE2C      10576       1  master cfg mgr init process
0x02050388      10548       1  HTTP SSL PKI CERTIFICATE CREATE PROC
0x0099C134       9756       1  EEM Shell Director
0x020CC974       8656      10  hulc_flash init
0x0021CBE8       7316      54  Smartport macro autoexec subblock
0x02304ED0       6888      51  LLDP-MED local annex
0x020B0788       6188       1  hulc_flash init
0x01B5F2DC       4652       1  Watcher Info
0x022E3640       4100      52  LIM Index
0x014829EC       3716       3  OBFL UPTIME obfl0
0x020D3DE0       2780       5  hulc_flash init
0x020D3684       2720       5  mifs file_map buffers
0x02883318       2352       7  platform OBFL
0x00477224       2196       1  HTTP SSL PKI CERTIFICATE CREATE PROC
0x004ADE94       2124       1  HTTP SSL PKI CERTIFICATE CREATE PROC
0x020D3618       2040       5  mifs file_info buffers
0x014618C4       1796       5  OBFL UPTIME obfl0
0x020C7A68       1708      24  hulc_flash init
0x020C79F8       1668      24  hulc_flash init
0x020C79BC       1632      24  hulc_flash init
0x01B5E85C       1544       1  Watched Semapho
0x005C40A8       1460       4  HTTP SSL PKI CERTIFICATE CREATE PROC
0x020D6928       1020      15  hulc_flash init
0x020D3DA4        980       5  hulc_flash init
0x020CCA94        460       5  hulc_flash init
0x004ADD80        444       1  HTTP SSL PKI CERTIFICATE CREATE PROC
0x020CCB3C        400       5  hulc_flash init
0x020D08E0        340       5  hulc_flash init
0x020D6C40        340       5  hulc_flash init
0x000730D4        308       1  platform OBFL
0x000739B0        308       1  platform OBFL
0x00073814        308       1  platform OBFL
0x00073B14        308       1  platform OBFL
0x000732A4        308       1  platform OBFL
0x00072E5C        308       1  platform OBFL
0x00343B40        276       3  TTY timer block
0x0045A4EC        232       1  HTTP SSL PKI CERTIFICATE CREATE PROC
0x01489A94        216       1  OBFL Infra Device info
0x01B31A48        208       1  AAA Large Chunk
0x00C1869C        144       1  HTTP
0x003877F0        136       1  Parser Mode
0x020B0734        112       1  hulc_flash init
0x00BF1E80        104       1  HTTP
0x00BFE2FC        100       1  HTTP
0x0059FE88         92       1  crypto_handle_table
0x00BF1B6C         84       1  HTTP
0x020B0808         80       1  hulc_flash init
0x00363AB4         76       1  Parser Linkage
0x002FD83C         68       1  AAA MI SG NAME
0x0038785C         68       1  Parser Mode Q1
0x00387880         68       1  Parser Mode Q2
          

Dead Proc Summary for: I/O

    PC          Total   Count  Name


Dead Proc Summary for: Driver text

    PC          Total   Count  Name


Switch1#show memory debug leak
Adding blocks for GD...

                 Driver text memory                 


Address    Size   Alloc_pc  PID  Alloc-Proc       Name

                 I/O memory                 


Address    Size   Alloc_pc  PID  Alloc-Proc       Name

                 Processor memory                 


Address    Size   Alloc_pc  PID  Alloc-Proc       Name
Switch1#


Unfortunately, 'show memory debug leaks' did not find any process having leak issue.

I were able to find a post from High Memory Utilization due to hulc_flash to mention CSCth60511 and CSCua52463 are relating to this leak. It seems upgrade will fix this issue, but he did not post if issue resolved after upgraded.

Here is what I searched from Cisco Bug Search Website and there are three related bugs:
Cisco Bug Search Tool
Results from Cisco Bug Search Tool




4.1 CSCth60511 - Memory leak seen at hulc_flash init process

Description
Symptom :

Upon bootup a switch might leak a small amount of memory for hulc_flash init
Process


Further Info :

Since the leak is small it does not have any functionality impact.
Customer Visible

Was the description about this Bug Helpful?
(0)
Last Modified:
Feb 6,2017
Status:
Fixed
Severity:
3 Moderate
Product:
(1)
Cisco Catalyst 3750-E Series Switches
Support Cases:
14
Known Affected Releases:
(4)
12.2
15.0DPA
15.0DPB
15.0SID
Known Fixed Releases:
(23)
15.2(6.0.21i)E
15.2(5.1.57i)E
15.2(5.0)ST
15.2(4.0.64a)E
15.2(4.0)ST
15.2(1.30)PSR
15.2(1.1)EY
15.1(2)SG1.90
15.1(1)SG5.170
15.1(1)SG5.169
15.1(1)SG5.163
15.1(1)SG5.124
15.1(1)SG5.103
15.1(1)SG5.83.1
15.1(1)SG5.78.11
15.1(1)SG5.75
15.1(1)SG5.74
15.1(1)SG15.3
15.0(5.2)DPB1
15.0(4.6)SID
15.0(2)SE
15.0(2)EX
15.0(1)SE


Unfortunately our devices is 2960x and running on 15.0(2)EX3 , which is different from what this bug says.


4.2 CSCua52463 - C2960 - Memory leak in dead memory - not able to telnet/ssh to switch

C2960 - Memory leak in dead memory - not able to telnet/ssh to switch
CSCua52463
C2960 is having memory leak in *dead* memory.
Problem observed on multiple switches.

arnsw0034#sh mem dead
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 25D8FA0 23727744 19629156 4098588 3362220 4034820
I/O 2C00000 4194304 2473496 1720808 1646820 1609296
Driver te 1800000 1048576 44 1048532 1048532 1048532

-----------

arnsw0034#sh proc mem sor
Processor Pool Total: 23727744 Used: 19628532 Free: 4099212
I/O Pool Total: 4194304 Used: 2473356 Free: 1720948
Driver te Pool Total: 1048576 Used: 40 Free: 1048536

PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 22527504 4649688 14708156 0 0 *Init*
0 0 3014844 1663092 1290864 3804375 1341026 *Dead*
150 0 14687348 8409544 659316 2268 0 Auth Manager
71 0 736300 143256 472556 0 0 Stack Mgr Notifi
0 0 0 0 394484 0 0 *MallocLite*
157 0 25864596 43517164 335060 0 0 EAP Framework
179 0 2289784 23580 261832 0 0 CDP Protocol
57 0 198956 1686364 208516 0 0 EEM ED ND
-----------


From "show mem dead" we can see one process continuously increasing:
...
026A5598 0000000056 026A5538 026A55F8 001 -------- -------- 000196E0 hulc_flash init
026A55F8 0000000056 026A5598 026A5658 001 -------- -------- 000196E0 hulc_flash init
026A5658 0000000040 026A55F8 026A56A8 001 -------- -------- 005360BC hulc_flash init
026A56A8 0000000028 026A5658 026A56EC 001 -------- -------- 00539ED0 hulc_flash init
026A56EC 0000000044 026A56A8 026A5740 001 -------- -------- 00E4C48C hulc_flash init
...
Customer Visible

Was the description about this Bug Helpful?
(0)
Last Modified:
Mar 17,2017
Status:
Other
Severity:
2 Severe
Product:
(104)
Cisco IOS
Cisco Catalyst 3560E-12D-E Switch
Cisco Catalyst 3560E-24TD-S Switch
Cisco Catalyst 2960S-24PD-L Switch
Cisco Catalyst 2960-48TT-S Switch
Cisco Catalyst 3750V2-24PS Switch
Cisco Catalyst 2960-24-S Switch
Cisco Catalyst 3750V2-24FS Switch
Cisco Catalyst 3560E-48PD-SF Switch
Cisco Catalyst 2960-24TC-L Switch
Cisco Catalyst 3560E-48PD-E Switch
Cisco Catalyst 3560X-48P-S Switch
Cisco Catalyst 3560-12PC-S Compact Switch
Cisco Catalyst 3560G-48PS Switch
Cisco Catalyst 2960G-24TC-L Switch
Cisco Catalyst 3560X-48PF-S Switch
Cisco Catalyst 2960S-48TD-L Switch
Cisco Catalyst 3750X-24T-S Switch
Cisco Catalyst 3750X-48PF-L Switch
Cisco Catalyst 3750-24FS Switch
Cisco Catalyst 3560G-24TS Switch
Cisco Catalyst 2960S-48LPS-L Switch
Cisco Catalyst 2960-24TC-S Switch
Cisco Catalyst 3750G-12S-SD Switch
Cisco Catalyst 3560X-48PF-L Switch
Cisco Catalyst 2960-24PC-L Switch
Cisco Catalyst 2960-48PST-S Switch
Cisco Catalyst 2960PD-8TT-L Compact Switch
Cisco Catalyst 2960G-48TC-L Switch
Cisco Catalyst 3750E-24TD-E Switch
Cisco Catalyst 3560X-48T-L Switch
Cisco Catalyst 3750E-48PD-E Switch
Cisco Catalyst 3750X-48PF-S Switch
Cisco Catalyst 3750X-24S-S Switch
Cisco Catalyst 2960S-48FPS-L Switch
Cisco Catalyst 2960-48PST-L Switch
Cisco Catalyst 3750G-24TS-1U Switch
Cisco Catalyst 3560E-24PD-S Switch
Cisco Catalyst 2960-48TC-L Switch
Cisco Catalyst 3750X-48T-S Switch
Cisco Catalyst 2960S-24TD-L Switch
Cisco Catalyst 3750E-48PD-EF Switch
Cisco Catalyst 3750V2-48TS Switch
Cisco Catalyst 3560E-12D-S Switch
Cisco Catalyst 3560E-48PD-EF Switch
Cisco Catalyst 3560G-24PS Switch
Cisco Catalyst 2960S-48LPD-L Switch
Cisco Catalyst 2960S-24TS-L Switch
Cisco Catalyst 3750G-48TS Switch
Cisco Catalyst 2960S-48TS-S Switch
Cisco Catalyst 3750X-24T-L Switch
Cisco Catalyst 3560G-48TS Switch
Cisco Catalyst 3560X-24P-L Switch
Cisco Catalyst 3750V2-48PS Switch
Cisco Catalyst 3560V2-24PS Switch
Cisco Catalyst 3750X-48T-L Switch
Cisco Catalyst 3560E-24PD-E Switch
Cisco Catalyst 2960G-8TC-L Compact Switch
Cisco Enhanced Layer 2/3 EtherSwitch Service Module
Cisco Catalyst 3750E-24TD-S Switch
Cisco Catalyst 2960S-48TS-L Switch
Cisco Catalyst 3560X-48P-L Switch
Cisco Catalyst 3750V2-24TS Switch
Cisco Catalyst 3560E-24TD-E Switch
Cisco Catalyst 2960S-24PS-L Switch
Cisco Catalyst 3560V2-24TS Switch
Cisco Catalyst 2960-8TC-S Compact Switch
Cisco Catalyst 3750X-24S-E Switch
Cisco Catalyst 3750E-48TD-S Switch
Cisco Catalyst 3560V2-24DC Switch
Cisco Catalyst 3750E-24PD-E Switch
Cisco Catalyst 2960-8TC-L Compact Switch
Cisco Catalyst 3750X-48P-L Switch
Cisco Catalyst 3560X-48T-S Switch
Cisco Catalyst 3750E-48PD-SF Switch
Cisco Catalyst 3750E-24PD-S Switch
Cisco Catalyst 3560E-12SD-S Switch
Cisco Catalyst 3560V2-48TS Switch
Cisco Catalyst 3750X-24P-L Switch
Cisco Catalyst 3750X-12S-E Switch
Cisco Catalyst 2960-48TT-L Switch
Cisco Catalyst 2960-48TC-S Switch
Cisco Catalyst 3560X-24T-S Switch
Cisco Catalyst 3750X-48P-S Switch
Cisco Catalyst 3560-8PC Compact Switch
Cisco Catalyst 2960S-48FPD-L Switch
Cisco Catalyst 3560X-24T-L Switch
Cisco Catalyst 2960-24LC-S Switch
Cisco Catalyst 3560V2-48PS Switch
Cisco Catalyst 3750G-48PS Switch
Cisco Catalyst 3750X-12S-S Switch
Cisco Catalyst 2960-24PC-S Switch
Cisco Catalyst 2960-24LT-L Switch
Cisco Catalyst 3560E-48PD-S Switch
Cisco Catalyst 3560E-12SD-E Switch
Cisco Catalyst 3750G-24PS Switch
Cisco Catalyst 3560E-48TD-S Switch
Cisco Catalyst 2960-24TT-L Switch
Cisco Catalyst 3560X-24P-S Switch
Cisco Catalyst 3750X-24P-S Switch
Cisco Catalyst 3750E-48PD-S Switch
Cisco Catalyst 3560E-48TD-E Swtch
Cisco Catalyst 3750E-48TD-E Switch
Cisco Catalyst 2960S-24TS-S Switch
Support Cases:
10



4.3 CSCua52463 - C2960 - Memory leak in dead memory - not able to telnet/ssh to switch

hulc_flash init and init memory leaks seen in 3750 during dot1x testing
CSCuw93176
Symptom:
hulc_flash init and init memory leaks seen during dot1x testing on 3750G

Conditions:
This issue can be seen with or without any dot1x configured on the 3750G switch. Observed this even with the default config.

Workaround:
No workaround exists for this.

Further Problem Description:
Customer Visible

Was the description about this Bug Helpful?
(0)
Last Modified:
Feb 1,2017
Status:
Other
Severity:
3 Moderate
Product:
(143)
Cisco Catalyst 3750 Series Switches
Cisco Catalyst 2960-48TT-S Switch
Cisco Catalyst 2960-24-S Switch
Cisco Catalyst 3560E-12D-E Switch
Cisco Catalyst 2960S-24PD-L Switch
Cisco Catalyst 3560E-48PD-E Switch
Cisco Catalyst 3560X-48T-E Switch
Cisco Catalyst 3560E-48PD-SF Switch
Cisco Catalyst 3560X-48P-S Switch
Cisco ME 3400-24TS-A Switch
Cisco ME 3400EG-2CS-A Switch
Cisco Catalyst 2960G-24TC-L Switch
Cisco Catalyst 3560-12PC-S Compact Switch
Cisco Catalyst 3750G-12S Switch
Cisco Catalyst 3750V2-24FS Switch
Cisco Catalyst 2960-24TC-L Switch
Cisco Catalyst 3560E-24TD-S Switch
Cisco Catalyst 3750V2-24PS Switch
Cisco Catalyst 3560G-48PS Switch
Cisco Catalyst 3560G-24TS Switch
Cisco Catalyst 3560X-48PF-S Switch
Cisco Catalyst 2960S-48LPS-L Switch
Cisco ME 3400E-24TS-M Switch
Cisco IE 3000-8TC Industrial Ethernet Switch
Cisco Catalyst 3750X-48P-E Switch
Cisco Catalyst 3750G-16TD Switch
Cisco Catalyst 3750 Metro 24-AC Switch
Cisco Catalyst 2960S-48TD-L Switch
Cisco Catalyst 2960-24TC-S Switch
Cisco Catalyst 3750-48PS Switch
Cisco Catalyst 3750-24FS Switch
Cisco Catalyst 3750X-48PF-L Switch
Cisco Catalyst 3750X-24T-S Switch
Cisco Catalyst 3560-24TS Switch
Cisco Catalyst 3750X-24S-S Switch
Cisco Catalyst 3560X-48PF-L Switch
Cisco Catalyst 2960G-48TC-L Switch
Cisco Catalyst 2960-48PST-S Switch
Cisco Catalyst 3750X-48PF-S Switch
Cisco Catalyst 3750X-48PF-E Switch
Cisco Catalyst 3750G-24TS-1U Switch
Cisco Catalyst 2960S-48FPS-L Switch
Cisco Catalyst 3750E-24TD-E Switch
Cisco Catalyst 2960-48PST-L Switch
Cisco Catalyst 3750-48TS Switch
Cisco Catalyst 2960PD-8TT-L Compact Switch
Cisco Catalyst 3750E-48PD-E Switch
Cisco Catalyst 2960-24PC-L Switch
Cisco Catalyst 3750 Metro 24-DC Switch
Cisco ME 3400G-12CS-A Switch
Cisco Catalyst 3750G-12S-SD Switch
Cisco Catalyst 3560X-48T-L Switch
Cisco Catalyst 3560E-24PD-S Switch
Cisco ME 3400G-2CS-A Switch
Cisco Catalyst 2960S-24TD-L Switch
Cisco Catalyst 3750G-48TS Switch
Cisco Catalyst 2960S-48LPD-L Switch
Cisco Catalyst 3750V2-48PS Switch
Cisco Catalyst 2350-48TD-SD Switch
Cisco Catalyst 3560-48TS Switch
Cisco Catalyst 3750X-48T-S Switch
Cisco Catalyst 2350-48TD-S Switch
Cisco Catalyst 2960-48TC-L Switch
Cisco Catalyst 3560E-48PD-EF Switch
Cisco ME 3400EG-12CS-M Switch
Cisco Catalyst 3750X-24T-L Switch
Cisco Catalyst 2960S-24TS-L Switch
Cisco Catalyst 3560X-24T-E Switch
Cisco Catalyst 3750V2-48TS Switch
Cisco Catalyst 3750E-48PD-EF Switch
Cisco Catalyst 3560E-12D-S Switch
Cisco Catalyst 3560G-24PS Switch
Cisco Catalyst 3560X-24P-L Switch
Cisco Catalyst 3560G-48TS Switch
Cisco Catalyst 3750G-24TS Switch
Cisco Catalyst 2960S-48TS-S Switch
Cisco Catalyst 2960S-24PS-L Switch
Cisco Catalyst 3750V2-24TS Switch
Cisco Catalyst 3560X-24P-E Switch
Cisco Catalyst 3750X-24P-E Switch
Cisco Catalyst 3560-24PS Switch
Cisco Catalyst 3560E-24PD-E Switch
Cisco Catalyst 3560E-24TD-E Switch
Cisco Catalyst 3750X-48T-L Switch
Cisco Enhanced Layer 2/3 EtherSwitch Service Module
Cisco Catalyst 3560V2-24PS Switch
Cisco Catalyst 2960S-48TS-L Switch
Cisco Catalyst 3560X-48P-L Switch
Cisco Catalyst 3750E-24TD-S Switch
Cisco Catalyst 2960G-8TC-L Compact Switch
Cisco Catalyst 3750X-24T-E Switch
Cisco Catalyst 3750X-48P-L Switch
Cisco Catalyst 2960-8TC-L Compact Switch
Cisco Catalyst 3560E-12SD-S Switch
Cisco ME 3400-24FS-A Switch
Cisco Catalyst 3560X-48T-S Switch
Cisco Catalyst 3750E-48TD-S Switch
Cisco Catalyst 3750E-48PD-SF Switch
Cisco Catalyst 3560-48PS Switch
Cisco Catalyst 3560V2-24DC Switch
Cisco Catalyst 3560V2-24TS Switch
Cisco Catalyst 3750E-24PD-E Switch
Cisco Catalyst 3750G-24T Switch
Cisco Catalyst 3750E-24PD-S Switch
Cisco Catalyst 3750X-24S-E Switch
Cisco Catalyst 2960-8TC-S Compact Switch
Cisco Catalyst 3560X-48P-E Switch
Cisco Catalyst 2975 Switch
Cisco Catalyst 3750X-48T-E Switch
Cisco Catalyst 3750-24TS Switch
Cisco Catalyst 3560V2-48PS Switch
Cisco Catalyst 2960-24LC-S Switch
Cisco Catalyst 3750X-48P-S Switch
Cisco Catalyst 2960-48TT-L Switch
Cisco Enhanced Layer 2 EtherSwitch Service Module
Cisco Catalyst 3750X-12S-E Switch
Cisco Catalyst 3560V2-48TS Switch
Cisco Catalyst 3560X-24T-S Switch
Cisco Catalyst 2960S-48FPD-L Switch
Cisco Catalyst 3750X-24P-L Switch
Cisco Catalyst 3560-8PC Compact Switch
Cisco Catalyst 2960-48TC-S Switch
Cisco Catalyst 3560X-24T-L Switch
Cisco Catalyst 3750X-12S-S Switch
Cisco Catalyst 3560X-24P-S Switch
Cisco Catalyst 3750-24PS Switch
Cisco Catalyst 3560E-48TD-S Switch
Cisco Catalyst 3560E-12SD-E Switch
Cisco Catalyst 2960-24TT-L Switch
Cisco Catalyst 2960S-24TS-S Switch
Cisco Catalyst 3750E-48PD-S Switch
Cisco ME 3400G-12CS-D Switch
Cisco Catalyst 3750E-48TD-E Switch
Cisco Catalyst 3560E-48PD-S Switch
Cisco IE 3000-4TC Industrial Ethernet Switch
Cisco Catalyst 3750X-24P-S Switch
Cisco Catalyst 2960-24LT-L Switch
Cisco ME 3400-24TS-D Switch
Cisco Catalyst 3750G-24PS Switch
Cisco Catalyst 2960-24PC-S Switch
Cisco Catalyst 3560X-48PF-E Switch
Cisco Catalyst 3560E-48TD-E Swtch
Cisco Catalyst 3750G-48PS Switch
Support Cases:
1

Solution


Sadly although we found on bug CSCua52463 seems perfectly matching our case, Cisco does not post any solution and fix for it.


Downgrading to 12.2 or upgrading to 15.2(2)E3  will be our last step to try at this moment. Actually upgrading to 15.2(2)E3 resolved this issue.

Here is the monitoring information from NetMRI. After upgraded IOS to 15.2(2)E3, the free memory is getting stable. It was reducing slowly every minute until upgraded to 15.2(2)E3.
Type:Switch (99%)Vendor:Cisco
O/S Version:15.2(2)E3Model:catWsC2960x48tsLL
Up Time:1d 00h 17m 42sSNMP Status:Enabled
Last Communication:2017-03-27 10:13:30MAC Address:5C:A4:8A:5C:70:C1
Discovery Blackout:N/AChange Blackout:N/A

Free Switch Memory Status in Monitoring Tools





Reference:





No comments:

Post a Comment