Article ID: 123227, created on Oct 23, 2014, last review on Feb 17, 2016

  • Applies to:
  • Virtuozzo 6.0

Symptoms

There is a notable performance degradation in Virtuozzo cluster with virtual environments stored in Virtuozzo storage.

All chunk servers on a single node show 100% IOWAIT values:

CSID STATUS      SPACE  AVAIL REPLICAS   UNIQUE IOWAIT IOLAT(ms) QDEPTH HOST BUILD_VERSION 
1050 active     439.3G 96.3GB    13863        0   100%       0/0   25.0 192.168.1.7
1052 active     548.9G 121.9G    19243        0    99%       0/0   31.9 192.168.1.7
1148 active     548.9G 122.1G    15060        0    99%       0/0   25.9 192.168.1.7

Chunk servers use SSD for storing write journals. The performance of the SSD is degraded, it serves I/O slowly.

Cause

The used SSD is not healthy and requires maintenance or replacement.

Conditions

Among all possible situations, this article provides resolution when SSD is alive - the first two options from the list below:

  • SSD is only worn out without read/write errors.
  • Reading from SSD hangs or times out.
  • System does not recognize SSD at all.

IMPORTANT: For the resolution below, the SSD should still be alive, it should be possible to read/write to it. In case the SSD is completely unresponsive, use the following article for resolution:
Failed Write Journalling SSDs

Resolution

Resolution path depends on existence of a spare SSD drive for replacement and disk hot-swap support.

For compatibility:


There is no spare SSD drive for immediate replacement

In this case, the services are to be reconfigured:

  • Detach read cache from the mount point(s)
  • Remove write journals from chunk server(s)
  • Delete MDS service(s) running on this host[1]

The attached script can be used to automate removal of CS journals and MDS services from a single host.

Once a new SSD is available, proceed with adding back SSD-related configuration.

Let us suppose that there are two SSD disks in the system and both are to be replaced. Mount points are /mnt/ssd1 and /mnt/ssd2, they are in use by CS and MDS services of two clusters, as well as to store read caches.

The following command should print what is to be done:

~# ./pstorage-ssd_prepare_removal.sh /mnt/ssd1 /mnt/ssd2

[INFO] Looking for cluster services depending on: /mnt/ssd1 /mnt/ssd2...
[TODO] Cluster 'vz-vm': unlink journal from CS repository '/pstorage/vzvm-cs1'.
[TODO] Cluster 'vz-vm': unlink journal from CS repository '/pstorage/vzvm-cs2'.
[TODO] Cluster 'vz-vm': unlink journal from CS repository '/pstorage/vzvm-cs3'.
[TODO] Cluster 'vz-vm': unlink journal from CS repository '/pstorage/vzvm-cs4'.
[TODO] Cluster 'vz-vm': unlink journal from CS repository '/pstorage/vzvm-cs5'.
[TODO] Cluster 'vz-vm': remove MDS server #17 with path '/mnt/ssd1/vzvm-mds'.
[TODO] Cluster 'vz-vm': remove cache '/mnt/ssd1/vzvm-readcache' from mount options.
[TODO] Cluster 'vz-ct': unlink journal from CS repository '/pstorage/vzct-cs1'.
[TODO] Cluster 'vz-ct': unlink journal from CS repository '/pstorage/vzct-cs2'.
[TODO] Cluster 'vz-ct': unlink journal from CS repository '/pstorage/vzct-cs3'.
[TODO] Cluster 'vz-ct': remove MDS server #5 with path '/mnt/ssd2/vzct-mds'.
[TODO] Cluster 'vz-ct': remove cache '/mnt/ssd2/vzct-readcache' from mount options.

[INFO] Configuration check completed, ready to proceed.

If the above is correct, type in the phrase 'Yes, do as I say!'> 

Entering phrase Yes, do as I say! should show exact commands to perform, which will not yet be executed (this is explained later):

If the above is correct, type in the phrase 'Yes, do as I say!'> Yes, do as I say!

[INFO] Proceeding, as instructed.

[INFO] Cluster 'vz-vm', reconfiguring and restarting CS instances.
# pstorage -c vz-vm configure-cs -r /pstorage/vzvm-cs1 -u
# service pstorage-csd restart /pstorage/vzvm-cs1
# pstorage -c vz-vm configure-cs -r /pstorage/vzvm-cs2 -u
# service pstorage-csd restart /pstorage/vzvm-cs2
# pstorage -c vz-vm configure-cs -r /pstorage/vzvm-cs3 -u
# service pstorage-csd restart /pstorage/vzvm-cs3
# pstorage -c vz-vm configure-cs -r /pstorage/vzvm-cs4 -u
# service pstorage-csd restart /pstorage/vzvm-cs4
# pstorage -c vz-vm configure-cs -r /pstorage/vzvm-cs5 -u
# service pstorage-csd restart /pstorage/vzvm-cs5

[INFO] Cluster 'vz-ct', reconfiguring and restarting CS instances.
# pstorage -c vz-ct configure-cs -r /pstorage/vzct-cs1 -u
# service pstorage-csd restart /pstorage/vzct-cs1
# pstorage -c vz-ct configure-cs -r /pstorage/vzct-cs2 -u
# service pstorage-csd restart /pstorage/vzct-cs2
# pstorage -c vz-ct configure-cs -r /pstorage/vzct-cs3 -u
# service pstorage-csd restart /pstorage/vzct-cs3

[INFO] Cluster 'vz-vm', removing MDS instances.
# pstorage -c vz-vm rm-mds 17

[INFO] Cluster 'vz-ct', removing MDS instances.
# pstorage -c vz-ct rm-mds 5

[WARN] Do not forget to create MDS service(s) on other host(s)!

[INFO] Changing mount options for Virtuozzo storage.
[WARN] Please remove 'cache=' and 'cachesize=' options for 'pstorage://vz-vm'.
[WARN] Please remove 'cache=' and 'cachesize=' options for 'pstorage://vz-ct'.

[INFO] All done.

To execute commands, define DRYRUN variable:

~# DRYRUN=no ./pstorage-ssd_prepare_removal.sh /mnt/ssd1 /mnt/ssd2

Mount options are not removed automatically, this is to be done manually (steps 1 and 2) to maintain formatting in the file /etc/fstab, before shutting down the server for SSD removal.

If the server cannot be shut down right now, remove mount options anyway and restart services using these commands. All virtual environments and iSCSI targets will be restarted!

  1. Stop services:

    ~# for svc in pvapp pvaagentd prl_mounterd parallels-server vz pstorage-iscsid pstorage-fs pstorage-csd pstorage-mdsd; do service $svc stop; done
    
  2. Ensure that there is no process holding SSD mount points:

    ~# lsof 2>/dev/null | grep -e /mnt/ssd1 -e /mnt/ssd2
    

    If there is any process, check what this process does and terminate it.

    There can be manual Ploop mounts too, check if there is anything:

    ~# ploop list
    
  3. Start services in reversed order

    ~# for svc in pstorage-mdsd pstorage-csd pstorage-fs pstorage-iscsid vz parallels-server prl_mounterd pvaagentd pvapp; do service $svc start; done
    

IMPORTANT[1]: Create MDS server on a different host or on a disk on this host to maintain the same number of MDS servers.


Disk hot-swap is supported and there is a new SSD disk for replacement

With disk hot-swap supported, just plug in a new SSD disk, create a partition and format it as Ext4.

  1. If it is not recognized automatically, force scanning SCSI bus:

    ~# echo "- - -" > /sys/class/scsi_host/hostN/scan
    

    Where N is the host bus ID of the SATA/SAS/SCSI connection.

  2. A new drive should appear, e.g. its name can be seen in the output of dmesg, hereafter - /dev/sdX.

  3. Create partition, format it and mount to some directory:

    ~# /usr/libexec/pstorage/prepare_pstorage_drive /dev/sdX --ssd
    ~# mkdir /mnt/new-ssd
    ~# mount /dev/sdX1 /mnt/new-ssd
    

    /mnt/new-ssd can be replaced with the appropriate path.

  4. Stop services:

    • if read cache is configured for Pstorage mount and it is saved on SSD, then stop all services. All running virtual environments and iSCSI targets will be stopped. If some environment should be running, migrate it to other host.

      ~# for svc in pvapp pvaagentd prl_mounterd parallels-server vz pstorage-iscsid pstorage-fs pstorage-csd pstorage-mdsd; do service $svc stop; done
      
    • without Pstorage mount cache, only CS and MDS instances should be stopped.

      ~# for svc in pstorage-csd pstorage-mdsd; do service $svc stop; done
      
  5. Use rsync to copy data from old SSD to a new one:

    ~# rsync -a /path-to/old-ssd/ /mnt/new-ssd/
    
  6. Remount drives:

    ~# umount /path-to/old-ssd/
    ~# umount /mnt/new-ssd/
    ~# mount /dev/sdX1 /path-to/old-ssd/
    
  7. Update /etc/fstab to mount new disk automatically.
  8. Start all services back.

    • if Pstorage mount has its read cache on SSD:

      ~# for svc in pstorage-mdsd pstorage-csd pstorage-fs pstorage-iscsid vz parallels-server prl_mounterd pvaagentd pvapp; do service $svc start; done
      
    • without read cache on SSD:

      ~# for svc in pstorage-mdsd pstorage-csd; do service $svc start; done
      


Disk hot-swap is not supported and there is a new SSD disk

  1. Shutdown the server, insert new drive.
  2. Start the server, boot into the single mode.
  3. A new drive should appear, hereafter - /dev/sdX.
  4. Create partition, format it and mount to some directory:

    ~# /usr/libexec/pstorage/prepare_pstorage_drive /dev/sdX --ssd
    ~# mkdir /mnt/new-ssd
    ~# mount /dev/sdx /mnt/new-ssd
    

    /mnt/new-ssd can be replaced with the appropriate path.

  5. Use rsync to copy data from old SSD to a new one:

    ~# rsync -a /path-to/old-ssd/ /mnt/new-ssd/
    
  6. Update /etc/fstab to mount new disk automatically.
  7. Shutdown the server, remove old SSD drive.
  8. Start and boot the server in normal way.


Manual replacement

For those who wants to do it manually, the instructions are below.

Removing SSD-related configuration

To start services it's necessary to remove any SSD-related configuration:

Chunk Services

For Chunk Services it is necessary to unlink the Write Journal

Note!: Replace <CLUSTER_NAME> with actual cluster name and <PATH_TO_CS> with actual path to CS

  1. Locate Chunk Services running on the host:

    # pstorage -c <CLUSTER_NAME> list-services
    
  2. For each Chunk Service execute following command:

    # pstorage -c <CLUSTER_NAME> configure-cs -r <PATH_TO_CS> -u
    

    Solution can be automated, following command will unlink journal for all CS-es running on the current host:

    # CLUSTER_NAME=<CLUSTER_NAME>
    # pstorage -c $CLUSTER_NAME list-services | awk '/^CS/{print $NF}' | while read cs; do pstorage -c $CLUSTER_NAME configure-cs -r "$cs" -u; done
    
  3. Once Journal was removed CS should start automatically. In case it didn't happen, just restart the corresponding service:

    # service pstorage-csd restart
    

Pstorage Mount

For Read Cache it will be necessary to correct the record in /etc/fstab file, remove path to the cache:

  1. Locate line containing pstorage record

    # grep ^pstorage /etc/fstab
    pstorage://mycluster /pstorage/mycluster fuse.pstorage rw,nosuid,nodev,cache=/ssd/pstorage-mycluster-cache 0 0
    
  2. Edit the file /etc/fstab and remove cache configuration

    # grep ^pstorage /etc/fstab
    pstorage://mycluster /pstorage/mycluster fuse.pstorage rw,nosuid,nodev 0 0
    
  3. Stop parallels-server service to avoid any I/O for the pstorage:

    # service parallels-server stop
    

    Note!: Be aware that this step will stop all Virtual Environments.

  4. Restart pstorage-fs service

    # service pstorage-fs restart
    
  5. Start parallels-server service:

    # service parallels-server start
    

Adding back SSD-related configuration

Chunk Services

Once SSD is replaced, in order to regain the benefits of SSD journalling, it is required to recreate all chunk servers on the node.

To remove a chunk server, follow these steps:
Removing Chunk Servers

To create a chunk server with SSD journalling, follow these steps:
Setting Up a Chunk Server with a Journal on SSD

Pstorage Mount

  1. Add back the configuration of the mount cache file:

    # grep ^pstorage /etc/fstab
    pstorage://mycluster /pstorage/mycluster fuse.pstorage rw,nosuid,nodev,cache=/ssd/pstorage-mycluster-cache 0 0
    
  2. Stop parallels-server service to avoid any I\O for the pstorage:

    # service parallels-server stop
    

    Note!: Be aware that this step will stop all Virtual Environments.

  3. Restart pstorage-fs service

    # service pstorage-fs restart
    
  4. Start parallels-server service:

    # service parallels-server start
    

Search Words

replace ssd

slow

some container cannot access

Failed to resume the VM

iowait

IO Performance Problem

cant setup metadata server

create chunk storage

c62e8726973f80975db0531f1ed5c6a2 2897d76d56d2010f4e3a28f864d69223 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF