Article ID: 124534, created on Feb 24, 2015, last review on Feb 24, 2015

  • Applies to:
  • Virtuozzo
  • Virtuozzo containers for Linux
  • Virtuozzo hypervisor

Symptoms

A backup to a remote server fails with the following error:

-1 : Pipe read error: 110 (Connection timed out)

Cause

The backup session was terminated due to a hit TCP connection timeout. This may be caused by either a Network Hardware Firewall (like Cisco ASA or similar) configuration or insufficient TCPKeepAlive configuration on the nodes.

  • Scenario 1: A backup session is waiting for some local operation to complete and not sending any data. After the TCP timeout is reached, the Hardware Firewall closes communication for that session.
  • Scenario 2: A backup session is waiting for some local operation to complete and not sending any data. After tcp_keepalive_time + tcp_keepalive_probes * tcp_keepalive_intvl, the neighbor node closes the session.

Resolution

TCP Keepalive settings should be tuned on both source and destination nodes.

The default values are:

# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9

tcp_keepalive_time should be the same on both nodes, and it should be less than TCP timeout configured on the Hardware Firewall. This will instruct the nodes to start KeepAlive communication before Hardware Firewall timeout.

For example, if the Hardware Firewall is configured with 10 minutes TCP timeout:

[root@pcs ~]# echo 540 > /proc/sys/net/ipv4/tcp_keepalive_time

The above command will instruct the kernel to start sending KeepAlive probes after 9 minutes.

To increase the Keepalive duration:

[root@pcs ~]# echo 100 > /proc/sys/net/ipv4/tcp_keepalive_probes
[root@pcs ~]# echo 100 > /proc/sys/net/ipv4/tcp_keepalive_intvl

Please note, tcp_keepalive_intvl should be the same on both nodes, and it should be less then TCP timeout configured on the Hardware Firewall

The value of tcp_keepalive_probes should be chosen depending on the application needs. 100 KeepAlive packets with 100 seconds interval will result in more than two and a half hours of a live connection, which should be sufficient for most cases.

Note: correct /etc/sysctl.conf to make these changes permanent.

net.ipv4.tcp_keepalive_time = 540
net.ipv4.tcp_keepalive_probes = 100
net.ipv4.tcp_keepalive_intvl = 100

Search Words

backups fail

Pipe read error: 110

Connection timed out

a26b38f94253cdfbf1028d72cf3a498b 2897d76d56d2010f4e3a28f864d69223 d02f9caf3e11b191a38179103495106f e8e50b42231236b82df27684e7ec0beb 0dd5b9380c7d4884d77587f3eb0fa8ef

Email subscription for changes to this article
Save as PDF