Article ID: 119129, created on Dec 12, 2013, last review on Jun 17, 2016

  • Applies to:
  • Virtuozzo


This article assumes that all virtual environments in the cluster can be offline during the cluster recreation procedure. If it's necessary to keep them running, please follow this article.

On each node, follow these steps:

  1. Move the containers and virtual machines, registered on the node, to some local partition. Note: the partition should be large enough to hold all the data. Here's a real life example:

    • Learn how much disk space VMs occupy overall:

      [root@pcs ~]# s=0; for vmhome in $(awk -F"[<>]" '/VmHome.*pvm/{gsub("/config.pvs","");print$3}' /etc/parallels/vmdirectorylist.xml); do s=$(($s+$(du -s $vmhome|awk '{print$1}'))); done; echo $((s/1024/1024))GB
    • Learn how much disk space containers occupy overall (assuming all CTs are located in /vz/private/ directory):

      [root@pcs ~]# s=0; for veid in $(vzlist -Hao veid | sed 's/ //g' | grep -v "^1$") ; do s=$(($s+$(du -s /vz/private/$veid|awk '{print$1}'))); done; echo $((s/1024/1024))GB
    • Find the local partition with enough free space:

      [root@pcs ~]# df -h /pstorage
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/sdb1             4.5T  3.0T  1.5T  66% /pstorage
    • Stop vz and parallels-server services:

      # service parallels-server stop
      # service vz stop
    • Move VMs, VM templates and containers to the local partition:

      # mkdir /pstorage/cts /pstorage/vms
      # for vmhome in $(awk -F"[<>]" '/VmHome.*pvm/{gsub("/config.pvs","");print$3}' /etc/parallels/vmdirectorylist.xml); do mv "$vmhome" /pstorage/vms/; done
      # for veid in $(vzlist -Hao veid | sed 's/ //g' | grep -v "^1$") ; do mv /vz/private/$veid /pstorage/cts/; done
  2. Stop pstorage-fs service:

    # service pstorage-fs stop 
  3. When pstorage-fs service is stopped on all nodes, stop pstorage-csd and pstorage-mdsd services on all nodes:

    # service pstorage-csd stop
    # service pstorage-mdsd stop
  4. Install all available updates:

    # yum -y update
  5. Reboot to the newest kernel, if available:

    # vzreboot

    If vzreboot is not allowed by the license, perform a plain reboot.

  6. Clean up the cluster:

    Note: replace "pcs1" with your real cluster name.

    # clustername="pcs1"
    # rm -rf /pstorage/$clustername-mds*
    # rm -rf /pstorage/$clustername-cs*
    # rm -rf /etc/pstorage/clusters/$clustername
    # rm -rf /root/.pstorage/*
    # rm -rf /var/lib/pstorage/.pstorage/*

    If it's not necessary to save the log files, delete them as well to release the disk space:

    # rm -rf /var/log/pstorage/*

    Remove the pstorage-mount entry in /etc/fstab:

    # sed -i 's~pstorage://.*$~~' /etc/fstab
  7. On one of the servers, initiate a new cluster:

    Note: replace "pcs1" with your real cluster name.

    # pstorage -c pcs1 make-mds -I -a -r /pstorage/pcs1-mds -p
    # service pstorage-mdsd start

    Create additional MDS instances on the rest of the servers:

    # pstorage -c pcs1 auth-node
    # pstorage -c pcs1 make-mds -a -r /pstorage/pcs1-mds
    # service pstorage-mdsd start
  8. Create a CS server:

    # pstorage -c pcs1 make-cs -r /pstorage/pcs1-cs
    # service pstorage-csd start
  9. Load a valid Cloud Storage license:

    # pstorage -c pcs1 load-license -p <activation_code>
  10. Add a new entry to /etc/fstab:

    # mkdir /pstorage/pcs1
    # echo "pstorage://pcs1 /pstorage/pcs1 fuse.pstorage rw,nosuid,nodev 0 0" >> /etc/fstab
    # service pstorage-fs start
    # mkdir /pstorage/pcs1/vmprivate /pstorage/pcs1/private

    If the cluster name changed during the cluster recreation, the links for the default CTs and VMs directories should be recreated:

    # rm -rf /var/parallels /vz/private /vz/vmprivate
    # ln -s /pstorage/pcs1/private /vz/private
    # ln -s /pstorage/pcs1/vmprivate /var/parallels
    # ln -s /pstorage/pcs1/vmprivate /vz/vmprivate
  11. Verify that the cluster is healthy:

    # pstorage -c pcs1 top
  12. Move all VMs to the new storage:

    # mv /pstorage/vms/*.pvm /pstorage/pcs1/vmprivate/
    # mv /pstorage/cts/* /pstorage/pcs1/private/
  13. Start vz and parallels-server services:

    # service parallels-server start
    # service vz start
  14. Set the required replication level:

    # pstorage -c pcs1 set-attr -R -p / replicas=3 

The operation of cluster recreation is completed at this point. If any issues arise during any step of the whole process, and the reason is not clear, contact Parallels Technical Support for assistance.

Search Words


Unable connect to cluster, timeout (30 sec) expired.

restore cloud storage

from scratch



0dd5b9380c7d4884d77587f3eb0fa8ef 2897d76d56d2010f4e3a28f864d69223

Email subscription for changes to this article
Save as PDF