Article ID: 121971, created on Jun 10, 2014, last review on Jun 10, 2014

  • Applies to:
  • Virtuozzo 6.0
  • Virtuozzo containers for Linux 4.7
  • Virtuozzo hypervisor

Symptoms

Server gets Kernel Panic due to NETDEV watchdog with message NETDEV WATCHDOG: eth0 (bnx2): transmit queue 2 timed out:

Jun  9 09:47:48 poa1 kernel: [920820.473510] do_IRQ: 2.113 No irq handler for vector (irq -1)
Jun  9 09:47:48 poa1 rgmanager[8437]: [script] Executing /etc/init.d/vziscsi status
Jun  9 09:47:57 poa1 kernel: [920829.489248] ------------[ cut here ]------------
Jun  9 09:47:57 poa1 kernel: [920829.489261] WARNING: at net/sched/sch_generic.c:267 dev_watchdog+0x26b/0x280() (Tainted: P           ---------------   )
Jun  9 09:47:57 poa1 kernel: [920829.489266] Hardware name: System x3550 M3 -[7944C2G]-
Jun  9 09:47:57 poa1 kernel: [920829.489269] NETDEV WATCHDOG: eth0 (bnx2): transmit queue 2 timed out
Jun  9 09:47:57 poa1 kernel: [920829.489272] Modules linked in: des_generic ecb md4 nls_utf8 cifs cls_fw simfs sch_cbq ip_vzredir(P)(U) vzredir(P)(U) vzcompat(P)(U) vzrst vznetdev ip6_vzprivnet(P)(U) ip6_vzredir(P)(U) ip6_vznetstat(P)(U) ip_vzprivnet(P)(U) vziolimit vzsnap(P)(U) vzfs(P)(U) vzdquota vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc ext4 jbd2 vzlinkdev(P)(U) vzethdev vzevent vzlist(P)(U) vzstat(P)(U) vzmon ip_vznetstat(P)(U) vznetstat(P)(U) vzdev iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_LOG xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_mangle iptable_filter xt_multiport xt_limit xt_dscp ipt_REJECT ip_tables autofs4 dlm configfs cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables be2iscsi iscsi_boot_sysfs bnx2i cnic uio cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport _iscsi vfat fat bonding ipv6 8021q
Jun  9 09:47:57 poa1 kernel: garp stp llc tpm_tis tpm tpm_bios microcode iTCO_wdt iTCO_vendor_support igb i2c_algo_bit ptp pps_core bnx2 serio_raw i2c_i801 i2c_core lpc_ich mfd_core sg ioatdma dca i7core_edac edac_core shpchp ext3 jbd mbcache sd_mod crc_t10dif pata_acpi ata_generic ata_piix megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jun  9 09:47:57 poa1 kernel: [920829.489407] Pid: 0, comm: swapper veid: 0 Tainted: P           ---------------    2.6.32-042stab090.2 #1
Jun  9 09:47:57 poa1 kernel: [920829.489412] Call Trace:
Jun  9 09:47:57 poa1 kernel: [920829.489414]  <IRQ>  [<ffffffff81074f97>] ? warn_slowpath_common+0x87/0xc0
Jun  9 09:47:57 poa1 kernel: [920829.489428]  [<ffffffff81075086>] ? warn_slowpath_fmt+0x46/0x50
Jun  9 09:47:57 poa1 kernel: [920829.489433]  [<ffffffff8148819b>] ? dev_watchdog+0x26b/0x280
Jun  9 09:47:57 poa1 kernel: [920829.489440]  [<ffffffff8109a11d>] ? insert_work+0x6d/0xb0
Jun  9 09:47:57 poa1 kernel: [920829.489448]  [<ffffffff81088a9c>] ? run_timer_softirq+0x1bc/0x380
Jun  9 09:47:57 poa1 kernel: [920829.489453]  [<ffffffff81487f30>] ? dev_watchdog+0x0/0x280
Jun  9 09:47:57 poa1 kernel: [920829.489459]  [<ffffffff810321ad>] ? lapic_next_event+0x1d/0x30
Jun  9 09:47:57 poa1 kernel: [920829.489466]  [<ffffffff8107e36d>] ? __do_softirq+0x10d/0x250
Jun  9 09:47:57 poa1 kernel: [920829.489473]  [<ffffffff8100c44c>] ? call_softirq+0x1c/0x30
Jun  9 09:47:57 poa1 kernel: [920829.489479]  [<ffffffff81010195>] ? do_softirq+0x65/0xa0
Jun  9 09:47:57 poa1 kernel: [920829.489483]  [<ffffffff8107e18d>] ? irq_exit+0xcd/0xd0
Jun  9 09:47:57 poa1 kernel: [920829.489491]  [<ffffffff815375fa>] ? smp_apic_timer_interrupt+0x4a/0x60
Jun  9 09:47:57 poa1 kernel: [920829.489496]  [<ffffffff8100bc53>] ? apic_timer_interrupt+0x13/0x20
Jun  9 09:47:57 poa1 kernel: [920829.489499]  <EOI>  [<ffffffff812eb6be>] ? intel_idle+0xde/0x170
Jun  9 09:47:57 poa1 kernel: [920829.489508]  [<ffffffff812eb6a1>] ? intel_idle+0xc1/0x170
Jun  9 09:47:57 poa1 kernel: [920829.489515]  [<ffffffff814318c7>] ? cpuidle_idle_call+0xa7/0x140
Jun  9 09:47:57 poa1 kernel: [920829.489522]  [<ffffffff8100a026>] ? cpu_idle+0xb6/0x110
Jun  9 09:47:57 poa1 kernel: [920829.489528]  [<ffffffff81513095>] ? rest_init+0x85/0x90
Jun  9 09:47:57 poa1 kernel: [920829.489535]  [<ffffffff81c31f76>] ? start_kernel+0x406/0x412
Jun  9 09:47:57 poa1 kernel: [920829.489541]  [<ffffffff81c3133a>] ? x86_64_start_reservations+0x125/0x129
Jun  9 09:47:57 poa1 kernel: [920829.489546]  [<ffffffff81c31453>] ? x86_64_start_kernel+0x115/0x124

Cause

Due to a bug in firmware on 55XX Intel chipsets interrupt remapping is broken. This often causes loosing network card, and netdev watchdog might trigger a panic. This situation is described in CentOS bugtracker.

There is a high chance you'll be affected by this bug if you are using 55XX Intel chipset. You can check if you are on 55XX chipset by querying lspci:

# lspci | grep 55
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:10.0 PIC: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 (rev 22)
00:10.1 PIC: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 (rev 22)
00:11.0 PIC: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 (rev 22)
00:11.1 PIC: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:15.0 PIC: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)

Resolution

In order to workaround flaw in chipset firmware, you should append intremap=off option to the kernel line in grub.conf. E.g.:

title Parallels (2.6.32-042stab090.2)
        root (hd0,2)
        kernel /boot/vmlinuz-2.6.32-042stab090.2 ro root=UUID=2b7b0e76-2383-4423-805a-376893fadcb4 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM LANG=es_ES.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=es crashkernel=auto rhgb quiet intremap=off
        initrd /boot/initramfs-2.6.32-042stab090.2.img

Search Words

NETDEV WATCHDOG: eth0 (bnx2): transmit queue 2 timed out

a26b38f94253cdfbf1028d72cf3a498b 2897d76d56d2010f4e3a28f864d69223 e8e50b42231236b82df27684e7ec0beb d02f9caf3e11b191a38179103495106f 0dd5b9380c7d4884d77587f3eb0fa8ef 0c05f0c76fec3dd785e9feafce1099a9 c62e8726973f80975db0531f1ed5c6a2

Email subscription for changes to this article
Save as PDF