Interrupt Utilisation on Cisco 2921

Hi all,

​

I was wondering if anyone has experience troubleshooting interrupt utilisation on a Cisco 2921.

Our Total utilsation is sitting around 96 percent however the interrupt utilisation is 91. We thought the number of NAT translations may have been causing the issue initially however after putting NAT timeouts in place and putting an EEM script in t periodically clear NAT there has been no improvement.

I read the following article which is for a 3750 platform however some of the commands are not applicable to a 2921

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Here is the output I get when I run a show platform interrupt command. I note that IO Controller network counters are high and continue to rise however so does POW interrupt, fpga interrupt & CIU Timer 0 ISR so I am not sure where the issue lies. I suspect that the router is just getting too much traffic pushed through it but can't see how I pin point that.

There is no throttle counters incremented on any of the interfaces. Any ideas what to do next ? Do not want to upgrade this router to a 4000 series as it will be getting removed soon anyway.

TIA

​

CIU Interrupt info:

CIU Int Handler Count Context Level Name

0 0x300C3958 932498927 0x0 IP2 Pow Interrupt

18 0x300CAACC 447376 0x2 IP3 fpga interrupt

30 0x300C9204 0 0xE IP2 USB 0 Over Current i

31 0x300C9204 0 0xF IP2 USB 1 Over Current g

32 0x304951CC 0 0x0 IP2 MBOX0 interrupt

33 0x304951CC 0 0x0 IP2 MBOX1 interrupt

34 0x304BC314 1 0x1 IP4 Aux Port interrupt

35 0x304BC314 1 0x0 IP4 Console interrupt

40 0x300C9620 0 0x0 IP2 MSI 0 interrupt

41 0x300C9620 0 0x1 IP2 MSI 1 interrupt

42 0x300C9478 2361007398 0x2 IP2 IO-Controller Networ

43 0x300C92D0 0 0x3 IP2 IO-Controller Mgmt/E

46 0x300C9D98 1 0x0 IP2 NPEI RML interrupt

52 0x304D6D64 66552628 0x3DE2CC18 IP3 CIU Timer 0 ISR

53 0x304D6D64 0 0x21472E98 IP2 CIU Timer 1 ISR

54 0x304D6D64 2600 0x3DE2CC8C IP2 CIU Timer 2 ISR

56 0x3051FAA4 1 0x0 IP2 USB 0 interrupt

​

IP2 Int En: 0x01604F03 C0000001

IP3 Int En: 0x00100000 00040000

IP4 Int En: 0x0000000C 00000000

​

Interrupt Counters:

Int Level Counter: L1 (netio) 930600569, L2 0, L3 (mgmt) 66996541, L4 (console) 2, L5 0, L6 0, l7 (4ms timer) 114485138

CIU IP level int counter: IP2 2758185883, IP3 66996541, IP4 2

​

Debug counters/Info:

ciu_int2_count 2758185883, ciu_int3_count 66996541, ciu_int4_count 2

unknown_ip4_int_cnt 0, unreg_int_cnt 0, wdog_int_cnt 0

unreg_int_sum1 0x0, unreg_int_sum2 0x0

spur_timer_int 0, unreg_msi_int_cnt 0

rml_int 1, npei_int 0, usb1_int 1

npei_pcie0_err 0, npei_pcie1_err 0, npei_pcie_timeout 0

fpa_err 0, pip_err 0, ipd_err 0, mio_err 1

l2c_err 0, l2c_int_sum 0x00000000, lmc_err 0

l2d_faddr 0x00000000, l2t_fddr 0x00000000, lmc_faddr 0x00000000

usb 0/1 over current 0/0

pcie0_cfg030 0x0000583F, pcie1_cfg030 0x00000000

pcie0_cfg065 0x00000000, pcie1_cfg065 0x00000000

pcie0_cfg068 0x00000000, pcie1_cfg068 0x00000000

pcie0_cfg071 0x00000000, pcie1_cfg071 0x00000000

pcie0_cfg072 0x00000000, pcie1_cfg072 0x00000000

pcie0_cfg073 0x00000000, pcie1_cfg073 0x00000000

pcie0_cfg074 0x00000000, pcie1_cfg074 0x00000000

pcie0_cfg076 0x00000001, pcie1_cfg076 0x00000000

pcie0_dbg_info 0x00000022, pcie1_dbg_info 0x00000000

fpa_int_sum 0x00000000, pip_err_reg 0x00000000, ipd_int_sum 0x00000000

​

​

MSI Interrupt info:

MSI 128, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 0

MSI 129, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x1, slot 0, IO_Controller, NETIO, number 1

MSI 130, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x2, slot 0, IO_Controller, NETIO, number 2

MSI 131, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x3, slot 0, IO_Controller, NETIO, number 3

MSI 132, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x4, slot 0, IO_Controller, NETIO, number 4

MSI 133, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x5, slot 0, IO_Controller, NETIO, number 5

MSI 134, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x6, slot 0, IO_Controller, NETIO, number 6

MSI 135, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x7, slot 0, IO_Controller, NETIO, number 7

MSI 136, name: net_int_wrapper, hdlr 0x3077D944, cnt 1290182450, ctx 0x21E02234, slot 0, IO_Controller, NETIO, number 8

MSI 137, name: net_int_wrapper, hdlr 0x3077D944, cnt 2361007907, ctx 0x3E7636D8, slot 0, IO_Controller, NETIO, number 9

MSI 138, name: net_int_wrapper, hdlr 0x3077D944, cnt 0, ctx 0x21E02340, slot 0, IO_Controller, NETIO, number 10

MSI 139, name: net_int_wrapper, hdlr 0x3077D944, cnt 457893, ctx 0x3E7637E4, slot 0, IO_Controller, NETIO, number 11

MSI 155, name: i2c intr handler, hdlr 0x30A26630, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 27

MSI 156, name: wic_mbrd_scc_netio_, hdlr 0x3079DDE4, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 28

MSI 192, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x0, slot 0, IO_Controller, ERR, number 0

MSI 193, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x1, slot 0, IO_Controller, ERR, number 1

MSI 194, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x2, slot 0, IO_Controller, ERR, number 2

MSI 195, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x3, slot 0, IO_Controller, ERR, number 3

MSI 196, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x4, slot 0, IO_Controller, ERR, number 4

MSI 197, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x5, slot 0, IO_Controller, ERR, number 5

MSI 198, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x6, slot 0, IO_Controller, ERR, number 6

MSI 199, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x7, slot 0, IO_Controller, ERR, number 7

MSI 203, name: error_int_wrapper, hdlr 0x3077D954, cnt 0, ctx 0x3E7637E4, slot 0, IO_Controller, ERR, number 11

MSI 212, name: Hsib Error Monitor , hdlr 0x30A2C5C0, cnt 0, ctx 0x0, slot 0, IO_Controller, ERR, number 20

MSI 220, name: wic_mbrd_scc_mgmt_i, hdlr 0x3079D708, cnt 0, ctx 0x0, slot 0, IO_Controller, MGMT, number 28

MSI 221, name: tdm_exception_handl, hdlr 0x30E6207C, cnt 0, ctx 0x108A0000, slot 0, IO_Controller, ERR, number 29

MSI 222, name: GPIO interrupt hand, hdlr 0x30A1BDC0, cnt 0, ctx 0x10900200, slot 0, IO_Controller, ERR, number 30

​

MSI Enable Reg 0: 0x00000000 00000000

MSI Enable Reg 1: 0x00000000 00000000

MSI Enable Reg 2: 0x00000000 18000FFF

MSI Enable Reg 3: 0x00000000 701008FF