Article ID: 118833, created on Nov 24, 2013, last review on May 8, 2014

  • Applies to:
  • Virtuozzo 6.0
  • Virtuozzo containers for Linux
  • Virtuozzo hypervisor

Symptoms

Node crashes with the following messages in dmesg:

<1>[5929170.450104] BUG: unable to handle kernel NULL pointer dereference at (null)
<1>[5929170.464418] IP: [<ffffffff814a0bd0>] tcp_sacktag_walk+0x30/0x520
<4>[5929170.476816] PGD 6260aa067 PUD 36942b067 PMD 0
<4>[5929170.486109] Oops: 0000 [#1] SMP
<4>[5929170.492964] last sysfs file: /sys/devices/system/cpu/cpu23/cache/index2/shared_cpu_map
<4>[5929170.509204] CPU 18
<4>[5929170.513379] Modules linked in: mpt2sas scsi_transport_sas raid_class mptctl mptbase dell_rbu ipmi_devintf simfs 8021q garp stp llc snapapi26(P)(U) cls_fw sch_cbq ip_vzredir(U) vzredir(U) vzcompat(U) vzrst vznetdev ip6_vzprivnet(U) ip6_vzredir(U) ip6_vznetstat(U) ip_vzprivnet(U) vziolimit(U) vzsnap(U) vzfs(U) vzdquota vzcpt nfs lockd fscache auth_rpcgss nfs_acl sunrpc vzlinkdev(U) vzethdev vzevent vzlist(U) vzstat(U) vzmon ip_vznetstat(U) vznetstat(U) vzdev ipv6 xt_mark xt_connlimit ipt_MASQUERADE xt_length xt_hl xt_tcpmss xt_TCPMSS iptable_nat iptable_mangle xt_dscp xt_recent xt_owner nf_conntrack_ftp xt_conntrack ipt_REDIRECT nf_nat ipt_REJECT xt_multiport ipt_LOG xt_limit nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables xt_MARK arc4 ecb ppp_mppe ip_gre ppp_async crc_ccitt ppp_deflate zlib_deflate ppp_generic slhc fuse tun power_meter ses enclosure sg bnx2 dcdbas microcode serio_raw iTCO_wdt iTCO_vendor_support i7core_edac edac_core ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: edd]
<4>[5929170.714360]
<4>[5929170.717724] Pid: 65093, comm: java veid: 18530 Tainted: P         C ---------------    2.6.32-042stab079.5 #1 042stab079_5 Dell Inc. PowerEdge R410/01V648
<4>[5929170.745781] RIP: 0010:[<ffffffff814a0bd0>]  [<ffffffff814a0bd0>] tcp_sacktag_walk+0x30/0x520
<4>[5929170.763078] RSP: 0000:ffff880850843940  EFLAGS: 00010206
<4>[5929170.774061] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
<4>[5929170.788730] RDX: ffff88102341d000 RSI: ffff88086cb3b840 RDI: ffff880958156038
<4>[5929170.803409] RBP: ffff8808508439c0 R08: 0000000000000000 R09: 0000000000000002
<4>[5929170.818084] R10: ffff88086cb3b908 R11: 00000000a6443c16 R12: ffff88086cb3b840
<4>[5929170.832756] R13: ffff880958156038 R14: ffff880850843a40 R15: 0000000000000000
<4>[5929170.847436] FS:  00007fc27b52f700(0000) GS:ffff880850840000(0000) knlGS:0000000000000000
<4>[5929170.864016] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[5929170.875867] CR2: 0000000000000000 CR3: 000000029edce000 CR4: 00000000000007e0
<4>[5929170.890541] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>[5929170.905221] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>[5929170.919900] Process java (pid: 65093, veid: 18530, threadinfo ffff880988546000, task ffff880863048540)
<4>[5929170.938903] Stack:
<4>[5929170.943306]  ffffe8ffffc4d840 0000000000000000 0000000000000048 ffff88102c7d1c00
<4>[5929170.958143] <d> ffff8808a6443c16 ffff88086cb3b908 0000000000000002 0000000000000000
<4>[5929170.973925] <d> a644366d26afe0d8 ffff880850843990 31c1f9581ded1400 ffff88086cb3be60
<4>[5929170.990119] Call Trace:
<4>[5929170.995388]  <IRQ>
<4>[5929170.999984]  [<ffffffff814a14e2>] tcp_sacktag_write_queue+0x422/0xb80
<4>[5929171.013219]  [<ffffffff814a773e>] tcp_ack+0xa4e/0x1350
<4>[5929171.023862]  [<ffffffff814a466b>] ? tcp_validate_incoming+0x30b/0x3a0
<4>[5929171.037098]  [<ffffffff814a8a68>] tcp_rcv_state_process+0x1c8/0xa50
<4>[5929171.049990]  [<ffffffff814b0a7e>] tcp_v4_do_rcv+0x1fe/0x470
<4>[5929171.061511]  [<ffffffffa02055ca>] ? ipv4_confirm+0x8a/0x1e0 [nf_conntrack_ipv4]
<4>[5929171.076533]  [<ffffffff814b327e>] tcp_v4_rcv+0x50e/0x8e0
<4>[5929171.087518]  [<ffffffff8148ee20>] ? ip_local_deliver_finish+0x0/0x310
<4>[5929171.100753]  [<ffffffff8148ef3c>] ip_local_deliver_finish+0x11c/0x310
<4>[5929171.113990]  [<ffffffff8148f1c8>] ip_local_deliver+0x98/0xa0
<4>[5929171.125672]  [<ffffffff8148e63d>] ip_rcv_finish+0x12d/0x440
<4>[5929171.137175]  [<ffffffff8148ebd4>] ip_rcv+0x284/0x360
<4>[5929171.147473]  [<ffffffff8145626b>] __netif_receive_skb+0x34b/0x7b0
<4>[5929171.160016]  [<ffffffff8145676a>] process_backlog+0x9a/0x100
<4>[5929171.171702]  [<ffffffff8145ace0>] net_rx_action+0x110/0x300
<4>[5929171.183214]  [<ffffffff8107ab9d>] __do_softirq+0x10d/0x250
<4>[5929171.194548]  [<ffffffff8100c2ac>] call_softirq+0x1c/0x30
<4>[5929171.205531]  [<ffffffff8100df05>] do_softirq+0x65/0xa0
<4>[5929171.216167]  [<ffffffff8107a9bd>] irq_exit+0xcd/0xd0
<4>[5929171.226459]  [<ffffffff81520095>] do_IRQ+0x75/0xf0
<4>[5929171.236403]  [<ffffffff8100ba93>] ret_from_intr+0x0/0x11
<4>[5929171.247381]  <EOI>
<4>[5929171.251553] Code: 57 41 56 41 55 41 54 53 48 83 ec 58 0f 1f 44 00 00 4c 8d 96 c8 00 00 00 48 89 fb 49 89 f4 49 89 d7 49 89 ce 44 89 45 c4 45 89 cb <48> 8b 03 49 39 da 0f 18 08 0f 84 3c 01 00 00 49 3b 9c 24 00 02
<1>[5929171.291119] RIP  [<ffffffff814a0bd0>] tcp_sacktag_walk+0x30/0x520
<4>[5929171.303682]  RSP <ffff880850843940>
<4>[5929171.311027] CR2: 0000000000000000

Cause

The problem is recognized as a product bug PCLIN-32187

Resolution

The fix will be implemented in one of the kernel updates.

Search Words

PCLIN-32187

kernel panic

unable to handle kernel NULL pointer dereference at (null)

tcp_sacktag_write_queue

tcp_sacktag_walk

a26b38f94253cdfbf1028d72cf3a498b 2897d76d56d2010f4e3a28f864d69223 e8e50b42231236b82df27684e7ec0beb d02f9caf3e11b191a38179103495106f c62e8726973f80975db0531f1ed5c6a2 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF