Article ID: 116349, created on Jun 27, 2013, last review on Dec 12, 2014

  • Applies to:
  • Virtuozzo

Symptoms

After creating a new MDS server, it gets marked as 'stale' or 'unavailable' in the storage monitoring utility:

[root@pcs1 ~]# pstorage -c cluster top
...
MDSID STATUS   %CTIME   COMMITS   %CPU    MEM   UPTIME HOST
    1 avail      0.0%       0/s   0.0%    60m  20h 39m 10.254.254.10:2510
M   2 avail      0.0%       0/s   0.3%    60m   1d  2h 10.254.254.11:2510
    3 stale      0.0%       0/s   0.0%    33m   1d  1h 10.254.254.12:2510

At the same time, the following errors may be observed in the event log of the storage:

[root@pcs1 ~]# pstorage -c cluster get-event
...
28-04-14 23:49:52  JRN ERR MDS#3 (10.254.254.12:2510) didn't accept commits for 120 sec

The cluster network interface is configured with "jumbo frames" enabled, i.e. MTU is larger than standard length (1500 bytes):

[root@mds-2 ~]# ip link list dev eth1<br>
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP qlen 1000

Cause

The neighbour switch is not capable of processing jumbo frames, which breaks proper network communication between the master MDS server and the slave.

Resolution

Resolve the issues with the network switch: make sure it supports jumbo frames and this functionality is enabled.

NOTE: It is not recommended to increase MTU for 1Gb network adapters. On the contrary, for 10Gb adapters such setting is strongly advised in order to reach the full performance. Check the following article for more information:
Using 1 GbE and 10 GbE Networks

As a temporary workaround, MTU can be decreased to 1500.

# ip link set mtu 1500 dev eth1

Troubleshooting

In order to prove the issue on the switch side, it's needed to capture the traffic on two ends: the master MDS server and the slave MDS server that experiences issues. Below is a real life example, demonstrating the problem:

  • on the stale MDS server

    the capture shows a 9000-byte packet that comes between two small ones:

    # tcpdump -i eth1 -nnvvv 'host 172.16.8.2'
    
    00:45:05.928973 IP (tos 0x0, ttl 64, id 59656, offset 0, flags [DF], proto TCP (6), length 52)
        172.16.8.3.51147 > 172.16.8.2.2511: Flags [.], cksum 0x4ee0 (correct), seq 122, ack 50, win 140, options [nop,nop,TS val 100015076 ecr 102498724], length 0
    
    00:45:06.449886 IP (tos 0x0, ttl 64, id 55535, offset 0, flags [DF], proto TCP (6), length 9000)
        172.16.8.3.40550 > 172.16.8.2.2510: Flags [.], cksum 0x8b40 (incorrect -> 0x5652), seq 1332715211:1332724159, ack 1369301722, win 157, options [nop,nop,TS val 100015597 ecr 102448190], length 8948
    
    00:45:09.608852 IP (tos 0x0, ttl 64, id 45299, offset 0, flags [DF], proto TCP (6), length 60)
        172.16.8.2.38612 > 172.16.8.3.2511: Flags [S], cksum 0x8530 (correct), seq 2678759664, win 17920, options [mss 8960,sackOK,TS val 102502403 ecr 0,nop,wscale 7], length 0
    
  • at the same time on the master MDS server

    the capture shows the two small packets getting received, but the large packet is not delivered:

    # tcpdump -i eth1 -nnvvv 'host 172.16.8.3'
    
    00:45:05.925757 IP (tos 0x0, ttl 64, id 59656, offset 0, flags [DF], proto TCP (6), length 52)
        172.16.8.3.51147 > 172.16.8.2.2511: Flags [.], cksum 0x4ee0 (correct), seq 122, ack 50, win 140, options [nop,nop,TS val 100015076 ecr 102498724], length 0
    
    00:45:09.605574 IP (tos 0x0, ttl 64, id 45299, offset 0, flags [DF], proto TCP (6), length 60)
        172.16.8.2.38612 > 172.16.8.3.2511: Flags [S], cksum 0x8530 (correct), seq 2678759664, win 17920, options [mss 8960,sackOK,TS val 102502403 ecr 0,nop,wscale 7], length 0
    

Search Words

stale mds

unavailable

didn't accept commits

stale

mds

0dd5b9380c7d4884d77587f3eb0fa8ef 2897d76d56d2010f4e3a28f864d69223

Email subscription for changes to this article
Save as PDF