Article ID: 126043, created on Jun 27, 2015, last review on Jun 27, 2015

  • Applies to:
  • Virtuozzo

Symptoms

A chunk server fails unexpectedly with the following message in pstorage top:

27-06-15 14:44:41  MON ERR CS#1099 died unexpectedly (122): csd: could not lock repository

In the corresponding log file /var/log/cs-xxxx/cs.log.gz:

27-06-15 15:10:25.911 pcs process is inactive for 2000 msecs (0)
27-06-15 15:10:25.911 [<ffffffffa02fa9b4>] rpc_wait_bit_killable+0x24/0x40 [sunrpc]
27-06-15 15:10:25.911 [<ffffffffa02fa98d>] __rpc_wait_for_completion_task+0x2d/0x30 [sunrpc]
27-06-15 15:10:25.911 [<ffffffffa03a51c2>] nfs_initiate_commit+0x152/0x180 [nfs]
27-06-15 15:10:25.911 [<ffffffffa03a6a2c>] nfs_generic_commit_list+0x8c/0x100 [nfs]
27-06-15 15:10:25.911 [<ffffffffa03a6b8c>] nfs_commit_inode+0xec/0x150 [nfs]
...

The pstorage mount is stuck, attempts to access it result in a hang-up in the terminal. df, ps, netstat commands also hang. In the corresponding log file /var/log/pstorage/pstorage-mount.log.gz:

27-06-15 14:17:10.716 pcs process is inactive for 1000 msecs (0)
27-06-15 15:01:07.904 [<ffffffffa0396384>] nfs_wait_bit_killable+0x24/0x40 [nfs]
27-06-15 15:01:07.904 [<ffffffffa03a6b4a>] nfs_commit_inode+0xaa/0x150 [nfs]
27-06-15 15:01:07.904 [<ffffffffa039353e>] nfs_release_page+0x8e/0xb0 [nfs]
27-06-15 15:01:07.904 [<ffffffff811343d0>] try_to_release_page+0x30/0x60
27-06-15 15:01:07.904 [<ffffffff81153134>] shrink_page_list.clone.0+0x80/0xae0

The MDS server on the same host may appear unavailable in pstorage top.

There's an NFS share mounted on the server:

[root@vz ~]# grep nfs /proc/mounts
172.16.1.34:/export /backups_nfs nfs rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,hard,nolock,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.1.2,mountvers=3,mountport=40230,mountproto=udp,local_lock=all,addr=172.16.1.2 0 0

Cause

NFS server is unresponsive. The NFS share is mounted with 'hard' option, so any attempt to reach the inacessible share will result in a complete lock-up.

Resolution

Restore the availability of the NFS server.

Search Words

pcs process is inactive

ps

hang

stuck

lock

nfs

0dd5b9380c7d4884d77587f3eb0fa8ef 2897d76d56d2010f4e3a28f864d69223

Email subscription for changes to this article
Save as PDF