Article ID: 126074, created on Jul 3, 2015, last review on Jul 3, 2015

  • Applies to:
  • Virtuozzo 6.0

Symptoms

Virtual machines in pstorage cluster become temporarily PAUSED:

# grep PAUSED /var/log/parallels.log
02-05 11:52:40.004 F /disp:5382:5777/ Vm state was changed from VMS_RUNNING to VMS_PAUSED for vm {59c74828-5372-49e8-80ae-662fa26e6a06} (name='test'), to powerState 2(vpsPausedByVmFrozen)

IO latency is detected in pstorage-mount.log.gz:

# zcat /var/log/pstorage/cluster/pstorage-mount.log.gz | grep "IO latency on CS"
5-02-15 11:50:37.128 pcs_cs_deaccount: IO latency on CS1128 is 9816585, cwnd 131072, clamp 1048576
05-02-15 11:50:37.128 pcs_cs_deaccount: IO latency on CS1128 is 9480722, cwnd 131072, clamp 1048576
05-02-15 11:50:37.128 pcs_cs_deaccount: IO latency on CS1128 is 9276631, cwnd 131072, clamp 1048576
05-02-15 11:50:37.128 pcs_cs_deaccount: IO latency on CS1128 is 9276618, cwnd 131072, clamp 1048576

Swapiness spike detected in the problematic CS log:

05-02-15 11:50:36.518 pcs process is inactive for 1000 msecs (0)
05-02-15 11:50:36.518 Last PC: /lib64/libc.so.6(+0x797c2) [0x7f6a77be67c2]
05-02-15 11:50:36.518 [<ffffffff8113af2d>] sync_page+0x3d/0x50
05-02-15 11:50:36.518 [<ffffffff8113af4e>] sync_page_killable+0xe/0x40
05-02-15 11:50:36.518 [<ffffffff8113b0e5>] wait_on_page_bit_killable+0x75/0x80
05-02-15 11:50:36.518 [<ffffffff8113b27a>] __lock_page_or_retry+0xaa/0xc0
05-02-15 11:50:36.518 [<ffffffff81171d57>] handle_pte_fault+0xb77/0x1040
05-02-15 11:50:36.518 [<ffffffff8117244a>] handle_mm_fault+0x22a/0x300
05-02-15 11:50:36.518 [<ffffffff8104dfa8>] __do_page_fault+0x138/0x480
05-02-15 11:50:36.518 [<ffffffff815386be>] do_page_fault+0x3e/0xa0
05-02-15 11:50:36.518 [<ffffffff81535a25>] page_fault+0x25/0x30
05-02-15 11:50:36.518 [<ffffffffffffffff>] 0xffffffffffffffff
05-02-15 11:50:36.518 Version: 2.5
05-02-15 11:50:36.518
05-02-15 11:50:36.518 Latencies:
05-02-15 11:50:36.518 Type                         Lat            Total_lat                Calls
05-02-15 11:50:36.518 scheduling:              4760151      241951757989077           8359562584
05-02-15 11:50:36.518 alocatomic:                    0          11879000000             27270010
05-02-15 11:50:36.518 aloclow:                 1000000        1075159000000           2614663125
05-02-15 11:50:36.518 alochigh:                1000000        1969669000000           6471834566
05-02-15 11:50:36.518 aloclowmp:                     0           9107000000             14237335
05-02-15 11:50:36.518 alochighmp:                    0                    0                    0
05-02-15 11:50:36.518 swap_in:                   15791        1384629900114             34142069
05-02-15 11:50:36.518 page_in:                  148259        1372544557940           2230369346
05-02-15 11:50:36.518
05-02-15 11:50:36.518 Averages:
05-02-15 11:50:36.518 Type                        Avg1                 Avg5                Avg15
05-02-15 11:50:36.518 scheduling:             11652705              9906010              9196886
05-02-15 11:50:36.518 alocatomic:                56471                58231                68041
05-02-15 11:50:36.518 aloclow:                  998592               988329               987341
05-02-15 11:50:36.518 alochigh:               12162119              4448454              2185089
05-02-15 11:50:36.518 aloclowmp:                 59774                46725                45506
05-02-15 11:50:36.518 alochighmp:                    0                    0                    0
05-02-15 11:50:36.518 swap_in:                84483686             35886362             14958517
05-02-15 11:50:36.518 page_in:                   82575                68330                62024

Cause

The node does not have enough memory to serve the CS request.

Resolution

  • Increase the amount of RAM on the server.
  • Unload the node to free memory resources.

Search Words

pcs_cs_deaccount: IO latency on CS

freeze

backup

c62e8726973f80975db0531f1ed5c6a2 2897d76d56d2010f4e3a28f864d69223 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF