Article ID: 116115, created on May 17, 2013, last review on Aug 12, 2014

  • Applies to:
  • Virtuozzo 6.0
  • Virtuozzo containers for Linux

Symptoms

The Hardware Node is very slow, and sometimes the server crashes.   From the top utility, you can explore the high load average. In /var/log/messages, you can explore the following:
23:35:56 mypvc kernel: [55130.900367] INFO: task syslogd:15957 blocked for more than 120 seconds.
Apr  3 23:35:56 mypvc kernel: [55130.907172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr  3 23:35:56 mypvc kernel: [55130.915350] syslogd       D ffff881fec22ed60     0 15957   3764 1025 0x00000004
Apr  3 23:35:56 mypvc kernel: [55130.923175]  ffff882014d07948 0000000000000086 ffff881fec22f328 000000000001e9c0
Apr  3 23:35:56 mypvc kernel: [55130.931244]  000000000000fd88 ffff881fec22f328 ffff8800aec0ad60 ffff881fec22ed60
Apr  3 23:35:56 mypvc kernel: [55130.939325]  ffff882014d078f8 ffff882011632024 ffff881fec22f328 000000000001e9c0
Apr  3 23:35:56 mypvc kernel: [55130.947449] Call Trace:

Cause

The kernel detects the number of NUMA (Non-Unified Memory Access) nodes incorrectly, and the kernel is unable to handle memory from multiple NUMA nodes efficiently. NUMA is a memory design for multi-processor hosts where memory access time depends on the memory location relative to processor. Memory can be local and non-local to processor. Accessing local memory is faster than accessing non-local memory. On a hosts with enabled and configured NUMA processors will work with local memory, thus providing proper performance.

You can learn more about NUMA here:

Resolution

Ensure that NUMA support is enabled in the BIOS of your server. Here you can see an example of dmesg logs with properly configured NUMA:
[root@mypvc ~]# dmesg | grep -i numa [ 0.000000] NUMA: Using 30 for the hash shift.
You can also install numactl utility to control NUMA policy for processes or shared memory:
[root@benderbrau ~]# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 12 13 14 15 16 17
node 0 size: 24566 MB
node 0 free: 1465 MB
node 1 cpus: 6 7 8 9 10 11 18 19 20 21 22 23
node 1 size: 24576 MB
node 1 free: 2043 MB
node distances:
node   0   1
  0:  10  20
  1:  20  10
If you still experience this issue after NUMA was enabled, check also following articles:
  • Physical memory on your Hardware Node is not less than the entire amount provided to containers. For more information, refer to this article: 112740
  • High load average numbers on server with hardware RAID 5/10: 113694

d02f9caf3e11b191a38179103495106f 2897d76d56d2010f4e3a28f864d69223 e8e50b42231236b82df27684e7ec0beb 0dd5b9380c7d4884d77587f3eb0fa8ef c62e8726973f80975db0531f1ed5c6a2

Email subscription for changes to this article
Save as PDF