Hi all,
I was wondering if anyone has experience troubleshooting interrupt utilisation on a Cisco 2921.
Our Total utilsation is sitting around 96 percent however the interrupt utilisation is 91. We thought the number of NAT translations may have been causing the issue initially however after putting NAT timeouts in place and putting an EEM script in t periodically clear NAT there has been no improvement.
I read the following article which is for a 3750 platform however some of the commands are not applicable to a 2921
https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html
Here is the output I get when I run a show platform interrupt command. I note that IO Controller network counters are high and continue to rise however so does POW interrupt, fpga interrupt & CIU Timer 0 ISR so I am not sure where the issue lies. I suspect that the router is just getting too much traffic pushed through it but can't see how I pin point that.
There is no throttle counters incremented on any of the interfaces. Any ideas what to do next ? Do not want to upgrade this router to a 4000 series as it will be getting removed soon anyway.
TIA
CIU Interrupt info:
CIU Int Handler Count Context Level Name
0 0x300C3958 932498927 0x0 IP2 Pow Interrupt
18 0x300CAACC 447376 0x2 IP3 fpga interrupt
30 0x300C9204 0 0xE IP2 USB 0 Over Current i
31 0x300C9204 0 0xF IP2 USB 1 Over Current g
32 0x304951CC 0 0x0 IP2 MBOX0 interrupt
33 0x304951CC 0 0x0 IP2 MBOX1 interrupt
34 0x304BC314 1 0x1 IP4 Aux Port interrupt
35 0x304BC314 1 0x0 IP4 Console interrupt
40 0x300C9620 0 0x0 IP2 MSI 0 interrupt
41 0x300C9620 0 0x1 IP2 MSI 1 interrupt
42 0x300C9478 2361007398 0x2 IP2 IO-Controller Networ
43 0x300C92D0 0 0x3 IP2 IO-Controller Mgmt/E
46 0x300C9D98 1 0x0 IP2 NPEI RML interrupt
52 0x304D6D64 66552628 0x3DE2CC18 IP3 CIU Timer 0 ISR
53 0x304D6D64 0 0x21472E98 IP2 CIU Timer 1 ISR
54 0x304D6D64 2600 0x3DE2CC8C IP2 CIU Timer 2 ISR
56 0x3051FAA4 1 0x0 IP2 USB 0 interrupt
IP2 Int En: 0x01604F03 C0000001
IP3 Int En: 0x00100000 00040000
IP4 Int En: 0x0000000C 00000000
Interrupt Counters:
Int Level Counter: L1 (netio) 930600569, L2 0, L3 (mgmt) 66996541, L4 (console) 2, L5 0, L6 0, l7 (4ms timer) 114485138
CIU IP level int counter: IP2 2758185883, IP3 66996541, IP4 2
Debug counters/Info:
ciu_int2_count 2758185883, ciu_int3_count 66996541, ciu_int4_count 2
unknown_ip4_int_cnt 0, unreg_int_cnt 0, wdog_int_cnt 0
unreg_int_sum1 0x0, unreg_int_sum2 0x0
spur_timer_int 0, unreg_msi_int_cnt 0
rml_int 1, npei_int 0, usb1_int 1
npei_pcie0_err 0, npei_pcie1_err 0, npei_pcie_timeout 0
fpa_err 0, pip_err 0, ipd_err 0, mio_err 1
l2c_err 0, l2c_int_sum 0x00000000, lmc_err 0
l2d_faddr 0x00000000, l2t_fddr 0x00000000, lmc_faddr 0x00000000
usb 0/1 over current 0/0
pcie0_cfg030 0x0000583F, pcie1_cfg030 0x00000000
pcie0_cfg065 0x00000000, pcie1_cfg065 0x00000000
pcie0_cfg068 0x00000000, pcie1_cfg068 0x00000000
pcie0_cfg071 0x00000000, pcie1_cfg071 0x00000000
pcie0_cfg072 0x00000000, pcie1_cfg072 0x00000000
pcie0_cfg073 0x00000000, pcie1_cfg073 0x00000000
pcie0_cfg074 0x00000000, pcie1_cfg074 0x00000000
pcie0_cfg076 0x00000001, pcie1_cfg076 0x00000000
pcie0_dbg_info 0x00000022, pcie1_dbg_info 0x00000000
fpa_int_sum 0x00000000, pip_err_reg 0x00000000, ipd_int_sum 0x00000000
MSI Interrupt info:
MSI 128, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 0
MSI 129, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x1, slot 0, IO_Controller, NETIO, number 1
MSI 130, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x2, slot 0, IO_Controller, NETIO, number 2
MSI 131, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x3, slot 0, IO_Controller, NETIO, number 3
MSI 132, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x4, slot 0, IO_Controller, NETIO, number 4
MSI 133, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x5, slot 0, IO_Controller, NETIO, number 5
MSI 134, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x6, slot 0, IO_Controller, NETIO, number 6
MSI 135, name: wic_mbrd_hdlc_intr, hdlr 0x3079E1C0, cnt 0, ctx 0x7, slot 0, IO_Controller, NETIO, number 7
MSI 136, name: net_int_wrapper, hdlr 0x3077D944, cnt 1290182450, ctx 0x21E02234, slot 0, IO_Controller, NETIO, number 8
MSI 137, name: net_int_wrapper, hdlr 0x3077D944, cnt 2361007907, ctx 0x3E7636D8, slot 0, IO_Controller, NETIO, number 9
MSI 138, name: net_int_wrapper, hdlr 0x3077D944, cnt 0, ctx 0x21E02340, slot 0, IO_Controller, NETIO, number 10
MSI 139, name: net_int_wrapper, hdlr 0x3077D944, cnt 457893, ctx 0x3E7637E4, slot 0, IO_Controller, NETIO, number 11
MSI 155, name: i2c intr handler, hdlr 0x30A26630, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 27
MSI 156, name: wic_mbrd_scc_netio_, hdlr 0x3079DDE4, cnt 0, ctx 0x0, slot 0, IO_Controller, NETIO, number 28
MSI 192, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x0, slot 0, IO_Controller, ERR, number 0
MSI 193, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x1, slot 0, IO_Controller, ERR, number 1
MSI 194, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x2, slot 0, IO_Controller, ERR, number 2
MSI 195, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x3, slot 0, IO_Controller, ERR, number 3
MSI 196, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x4, slot 0, IO_Controller, ERR, number 4
MSI 197, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x5, slot 0, IO_Controller, ERR, number 5
MSI 198, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x6, slot 0, IO_Controller, ERR, number 6
MSI 199, name: wic_mbrd_hdlc_error, hdlr 0x3079DE64, cnt 0, ctx 0x7, slot 0, IO_Controller, ERR, number 7
MSI 203, name: error_int_wrapper, hdlr 0x3077D954, cnt 0, ctx 0x3E7637E4, slot 0, IO_Controller, ERR, number 11
MSI 212, name: Hsib Error Monitor , hdlr 0x30A2C5C0, cnt 0, ctx 0x0, slot 0, IO_Controller, ERR, number 20
MSI 220, name: wic_mbrd_scc_mgmt_i, hdlr 0x3079D708, cnt 0, ctx 0x0, slot 0, IO_Controller, MGMT, number 28
MSI 221, name: tdm_exception_handl, hdlr 0x30E6207C, cnt 0, ctx 0x108A0000, slot 0, IO_Controller, ERR, number 29
MSI 222, name: GPIO interrupt hand, hdlr 0x30A1BDC0, cnt 0, ctx 0x10900200, slot 0, IO_Controller, ERR, number 30
MSI Enable Reg 0: 0x00000000 00000000
MSI Enable Reg 1: 0x00000000 00000000
MSI Enable Reg 2: 0x00000000 18000FFF
MSI Enable Reg 3: 0x00000000 701008FF