Cloud storage hangs completely, it is impossible to connect to any MDS server:
~# pstorage -c CLUSTER stat ^C
Creating of a new one also hangs:
~# pstorage -c CLUSTER make-mds -a 172.16.0.7 -r /pstorage/CLUSTER-mds -b 172.16.0.2 20-09-13 10:22:13.078 Unable connect to cluster, timeout (30 sec) expired.
The problem is identified as a product bug PSBM-22373 and is connected with Cloud Storage license update operation.
To solve the issue after it happened:
Kill all MDS processes forcibly and start back the services on all MDS nodes:
~# service pstorage-mdsd stop Shutting down pstorage metadata server CLUSTER:/pstorage/CLUSTER-md[ OK ] ~# ps auxfwww | grep mdsd pstorage 282551 2.1 14.3 2403660 561276 ? Sl Sep20 958:26 /usr/bin/mdsd -r /pstorage/CLUSTER-mds/data -l /var/log/pstorage/CLUSTER/mds-1/mds.log.gz -u pstorage ~# kill -9 282551 ~# service pstorage-mdsd start Starting pstorage metadata server CLUSTER:/pstorage/CLUSTER-mds/dat[ OK ]
Update license manually:
~# pstorage -c CLUSTER update-license