Article ID: 121901, created on Jun 4, 2014, last review on Jun 17, 2016

  • Applies to:
  • Virtuozzo 6.0
  • Virtuozzo containers for Linux 4.7

Symptoms

A hardware node experiences severe performance degradation. vzstat shows very high CPU and Mem latencies:

  6:40am, up 28 days,  7:30, 23 users, load average: 14.07, 16.87, 16.00
CTNum 1, procs 1374: R   9, S 1357, D   8, Z   0, T   0, X   0
CPU [CRIT]: CTs   0%, CT0 100%, user 100%, sys   0%, idle   0%, lat(ms) 3576/0
Mem [CRIT]: total 386783MB, free 226134MB, lat(ms) 2274/1155

At the same time, load average is within normal bounds and doesn't look abnormal.

Memory is highly fragmented

~# vzstat -lv | sed '/^Mem/,/Slab/!d'
Mem [ OK ]: total 386783MB, free 51085MB, lat(ms) 1/0
  ZONE0 (DMA): size 14MB, act 0MB, inact 0MB, free 15MB (0/0/0)
    fragm 1*1 1*2 2*4 1*8 2*16 1*32 0*64 0*128 1*256 1*512 3*1024 
  ZONE1 (DMA32): size 2947MB, act 0MB, inact 0MB, free 362MB (0/0/0)
    fragm 68*1 49*2 124*4 82*8 84*16 22*32 15*64 18*128 19*256 11*512 74*1024 
  ZONE2 (Normal): size 384048MB, act 79195MB, inact 149896MB, free 50706MB (63/79/95)
    fragm 1100266*1 1937200*2 554480*4 310228*8 115114*16 8459*32 4082*64 3078*128 407*256 100*512 374*1024 
  Mem lat (ms): A0 0, K0 1, U0 1, K1 0, U1 0
  Slab pages: 1680MB/1677MB (ino 279MB, de 0MB, bh 277MB, pb 0MB)
~#

Cause

A flaw in 'vzstat' kernel module: it scans the full list of memory pages, with putting a global lock, so all processes are waiting for the lock to be released. With the increased amount of pages, such behavior causes very high CPU and mem latencies.

Resolution

The issue has been fixed in kernel 2.6.32-042stab092.1 as a part of PSBM-26984

Workaround for vzkernel before version 2.6.32-042stab092.1

As a temporary measure, the module vzstat can be unloaded:

~# rmmod vzstat

Search Words

mem latency

i/o performance

73,58% [kernel] [k] _spin_lock

slow performance

high latency

i/o performance lun

cpu latency

High load average

hdd slow

c62e8726973f80975db0531f1ed5c6a2 2897d76d56d2010f4e3a28f864d69223 0dd5b9380c7d4884d77587f3eb0fa8ef e8e50b42231236b82df27684e7ec0beb d02f9caf3e11b191a38179103495106f 0c05f0c76fec3dd785e9feafce1099a9

Email subscription for changes to this article
Save as PDF