Article ID: 116671, created on Aug 8, 2013, last review on May 9, 2014

  • Applies to:
  • Virtuozzo 6.0


  1. /var/log/messages on the node contains records like this:

    __ext4_get_inode_loc: unable to read inode block - inode=399314, block=1573277
  2. There are Chunk Servers (CS) in failed state and chunks of the container's disk image are blocked, offline or in replication state constantly:

    ~#  pstorage -c pcs-storage-cluster file-info /vz/private/123456/root.hdd/root.hds | grep [RBX]
    flags: X - offline, B - blocked, D - degraded, O - overcommitted,
    0x8000000           4   X       #1032
    0xc000000           4   X       #1032
    0x18000000          0   XR      #1037   #1032
    0x28000000          4   X       #1032
    0xb8000000          0   XR      #1037   #1032
    0xd4000000          0   XR      #1037   #1032


The problem occurs if Chunk Server (CS) goes to failed state - the write operations to such Chunk Server (CS) is not possible.

Replication to these servers is hanging, blocking operations for writing if number of replicas is too low - among other triggers, wrong configuration with replication 1:1 for file(s) can be noted.


Fix pstorage status:

  1. Find problematic clients using events:

      07-08-13 13:59:29  MDS WRN Integrity failed accessing 'root.hds' by the client at
  2. Stop or suspend virtual environments (CTs, VMs) on such clients, umount pstorage share;

    ~# prlctl list --vmtype vm -Ho uuid | xargs -rn1 prlctl suspend
    ~# vzlist -Ho veid | xargs -rn1 vzctl suspend
    ~# umount /pstorage/pcs-storage-cluster

    If umount fails due to busy files, resolve this conflict and umount the share.

  3. Stop pstorage-mdsd service on the Master MDS, wait for few minutes (3-5), start it back - the relocation of Master MDS role should succeed;

    ~# service pstorage-mdsd stop; sleep 300; service pstorage-mdsd start
  4. Mount pstorage on the client(s), start or resume VMs and CTs.

    ~# mount /pstorage/pcs-storage-cluster
    ~# prlctl list --vmtype vm -Hao uuid,status | awk '$2=="suspended"{print $1}' | xargs -rn1 prlctl resume
    ~# vzlist -Hao veid,status | awk '$2=="suspended"{print $1}' | xargs -rn1 vzctl resume
  5. On all nodes of the cluster check for PCS updates and install all updates if there are any.

    ~# yum install

    If there is any kernel update, schedule the server's restart.

  6. If CSes are in failed state still, remove CS(es) without force, wait till the servers are removed in the output of pstorage top and add them back. After that replication should be completed.

    For each server:

    ~# pstorage -c pcs-storage-cluster rm-cs 1033

    After completing removal of CS:

    ~# pstorage -c pcs-storage-cluster add-cs -r /pstorage/pcs-storage-cluster-cs
    ~# service pstorage-csd restart
  7. Check records with R flag and restart CS(es) from which the replication is done.


    ~#  pstorage -c pcs-storage-cluster file-info /vz/private/123456/root.hdd/root.hds | grep ^0.*R
    0x18000000          0   XR      #1037   #1032
    0xb8000000          0   XR      #1037   #1032
    0xd4000000          0   XR      #1037   #1032
    0x218000000         0   XR      #1037   #1032

    Here, chunk is being replicated from CS #1037 to CS #1032 and pstorage-csd should be restarted on CS #1037, using this command:

    ~# service pstorage-csd restart
  8. Repeat the operation for all files on the cluster which have chunks blocked in this way. The following command is useful for determining CS(es) for restarting CSD:

    ~# find /pstorage/pcs-storage-cluster/ -type f | while read f; do rep=$(pstorage -c pcs-storage-cluster file-info "$f" 2>/dev/null| grep ^0.*R); [ "$rep" ] && echo $f:: $rep; done


    ~# find /pstorage/pcs-storage-cluster/ -type f | while read f; do rep=$(pstorage -c pcs-storage-cluster file-info "$f" 2>/dev/null| grep ^0.*R); [ "$rep" ] && echo $f:: $rep; done
    /pstorage/pcs-storage-cluster/vmprivate/W2008R2.pvm/harddisk.hdd/harddisk.hdd.0.{5fbaabe3-6958-40ff-92a7-860e329aab41}.hds:: 0xa8c000000 0 BR #1030 #1032 0xb30000000 0 BR #1030 #1032

Search Words

filesystem errors


replication hangs

c62e8726973f80975db0531f1ed5c6a2 2897d76d56d2010f4e3a28f864d69223 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF