Article ID: 117010, created on Aug 29, 2013, last review on May 7, 2014

  • Applies to:
  • Virtuozzo 6.0
  • Virtuozzo containers for Linux
  • Virtuozzo hypervisor

Symptoms

The hardware server running with kernels 2.6.32-042stab076.x or 2.6.32-042stab078.x might crash if a container in routed mode generates high network traffic. This can happen if Broadcom NIC is used and GSO/GRO is enabled for the corresponding network interface.

The extracted kernel ring buffer from the crash file contains the following calltrace in the end:

<1>[177359.960011] BUG: unable to handle kernel NULL pointer dereference at 0000000000000200
<1>[177359.960157] IP: [<ffffffffa0499019>] venet_ext_lookup+0x9/0x50 [vznetdev]
<4>[177359.960237] PGD 0 
<4>[177359.960306] Oops: 0000 [#1] SMP 
...
<4>[177359.961891] Pid: 0, comm: swapper veid: 0 Tainted: P           ---------------    2.6.32-042stab079.5 #1 042stab079_5 HP ProLiant DL380p Gen8
<4>[177359.962037] RIP: 0010:[<ffffffffa0499019>]  [<ffffffffa0499019>] venet_ext_lookup+0x9/0x50 [vznetdev]
<4>[177359.962177] RSP: 0018:ffff880069b83a00  EFLAGS: 00010246
<4>[177359.962260] RAX: 0000000000000000 RBX: ffff881028fc9480 RCX: 0000000000000000
...
<4>[177359.964180] Call Trace:
<4>[177359.964246]  <IRQ> 
<4>[177359.964317]  [<ffffffffa04f61f3>] veip_lookup_redirect+0x173/0x230 [vzredir]
<4>[177359.964393]  [<ffffffffa049884a>] venet_xmit+0xca/0x330 [vznetdev]
<4>[177359.964470]  [<ffffffff8145bc5d>] dev_hard_start_xmit+0x2ad/0x650
<4>[177359.964543]  [<ffffffff8145c208>] dev_queue_xmit+0x208/0x560
<4>[177359.964617]  [<ffffffff81494df7>] ip_finish_output+0x237/0x2f0
<4>[177359.964688]  [<ffffffff81494f68>] ip_output+0xb8/0xc0
<4>[177359.964759]  [<ffffffff8149032f>] ip_forward_finish+0x4f/0x60
<4>[177359.964831]  [<ffffffff814904d8>] ip_forward+0x198/0x430
<4>[177359.964904]  [<ffffffff8148e63d>] ip_rcv_finish+0x12d/0x440
<4>[177359.964975]  [<ffffffff8148ebd4>] ip_rcv+0x284/0x360
<4>[177359.965048]  [<ffffffff8145626b>] __netif_receive_skb+0x34b/0x7b0
<4>[177359.965121]  [<ffffffff81458848>] netif_receive_skb+0x58/0x60
<4>[177359.965193]  [<ffffffff814589f4>] napi_gro_complete+0x84/0xe0
<4>[177359.965266]  [<ffffffff81458edf>] napi_gro_flush+0x2f/0x50
<4>[177359.965339]  [<ffffffff81458f24>] napi_complete+0x24/0x60
<4>[177359.965421]  [<ffffffffa015e002>] bnx2x_poll+0x1f2/0x2e0 [bnx2x]
<4>[177359.965494]  [<ffffffff8145ace0>] net_rx_action+0x110/0x300
<4>[177359.965568]  [<ffffffff8107ab9d>] __do_softirq+0x10d/0x250
<4>[177359.965641]  [<ffffffff810138b3>] ? native_sched_clock+0x13/0x80
<4>[177359.965715]  [<ffffffff8100c2ac>] call_softirq+0x1c/0x30
<4>[177359.965787]  [<ffffffff8100df05>] do_softirq+0x65/0xa0
<4>[177359.965858]  [<ffffffff8107a9bd>] irq_exit+0xcd/0xd0
<4>[177359.965932]  [<ffffffff81520095>] do_IRQ+0x75/0xf0
<4>[177359.966004]  [<ffffffff8100ba93>] ret_from_intr+0x0/0x11
<4>[177359.966075]  <EOI> 
<4>[177359.966148]  [<ffffffff812e000e>] ? intel_idle+0xde/0x170
<4>[177359.966220]  [<ffffffff812dfff1>] ? intel_idle+0xc1/0x170
<4>[177359.966293]  [<ffffffff81422af7>] cpuidle_idle_call+0xa7/0x140
<4>[177359.966366]  [<ffffffff8100a026>] cpu_idle+0xb6/0x110
<4>[177359.966436]  [<ffffffff8150f72d>] start_secondary+0x2ac/0x2ef
<4>[177359.966507] Code: 4a 08 48 8b 56 20 48 83 c7 38 48 83 c6 20 e8 8f c5 df e0 c9 c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 <48> 8b 87 00 02 00 00 48 89 f2 48 85 c0 75 08 31 c0 c9 c3 0f 1f 
<1>[177359.966949] RIP  [<ffffffffa0499019>] venet_ext_lookup+0x9/0x50 [vznetdev]
<4>[177359.967027]  RSP <ffff880069b83a00>
<4>[177359.967095] CR2: 0000000000000200

Cause

Redhat added the support of build_skb() function to the kernel 2.6.32-241.el6, and updated the driver bnx2x in the kernel 2.6.32-358.6.1.el6, so that the module started to use build_skb(). The latter function in the affected Virtuozzo kernels does not initialize the pointer to the corresponding virtual environment and this leads to the kernel crash on accessing the kernel structure.

Resolution

Update the kernel to the version 2.6.32-042stab079.6 or higher and reboot the server. The following command can be used to update the installation:

  1. PVC 4.7 and PSBM 5.0:

    ~# vzup2date -m batch install --core --tools --self-update --loader-autoconfig
    
  2. PCS 6.0:

    ~# yum update
    

Search Words

venet_ext_lookup

NULL pointer dereference

a26b38f94253cdfbf1028d72cf3a498b 2897d76d56d2010f4e3a28f864d69223 e8e50b42231236b82df27684e7ec0beb d02f9caf3e11b191a38179103495106f c62e8726973f80975db0531f1ed5c6a2 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF