Applications that handle a huge number of TCP connections, either as a server or as a client, will often see a large number of sockets in TIME_WAIT state. The sockets in TIME_WAIT can be seen with the “netstat -epn –tcp ” command. If the number of TIME_WAIT sockets gets too large, your address space will be exhausted, causing a disruption of TCP traffic.
The possible state values for TCP sockets are as follows:
|BOUND||Bound, ready to connect or listen.|
|CLOSED||Closed. The socket is not being used.|
|CLOSING||Closed, then remote shutdown; awaiting acknowledgment.|
|CLOSE_WAIT||Remote shutdown; waiting for the socket to close.|
|ESTABLISHED||Connection has been established.|
|FIN_WAIT_1||Socket closed; shutting down connection.|
|FIN_WAIT_2||Socket closed; waiting for shutdown from remote.|
|IDLE||Idle, opened but not bound.|
|LAST_ACK||Remote shutdown, then closed; awaiting acknowledgment.|
|LISTEN||Listening for incoming connections.|
|SYN_RECEIVED||Active/initiate synchronization received and the connection under way|
|SYN_SENT||Actively trying to establish connection.|
|TIME_WAIT||Wait after close for remote shutdown retransmission.|
‘CLOSE_WAIT’ state means the other end of the connection has been closed while the local end is still waiting for the application to close.
More Details on CLOSE_WAIT
‘CLOSE_WAIT‘ state on TCP connections occurs if the system has not received a close system call from the application, after having received notification (‘FIN‘ packet) from the other system that it has closed its endpoint. In other words, it means that the local end of the connection has received ‘FIN’ from the other end, but the OS is waiting for the program at the local end to actually close its connection.
The problem is that a program running on the local machine is not closing the socket. It is not a TCP tuning issue. A connection can stay in ‘CLOSE_WAIT’ forever while the program holds the connection open. So mostly this issue happens due to an application bug. However, if TCP/IP parameters are not properly set, closed TCP/IP connections will stay for a very long time in various ‘CLOSE’ states which take file descriptors from the process. In order to solve this problem, sometimes it is necessary to tune TCP/IP parameters in /etc/sysctl.conf (such as those net.ipv4.tcp_xxx parameters), so that TCP/IP connections are closed in a short amount of time, please contact Oracle customer support to troubleshoot the issue.
Reducing CLOSE_WAIT connection
1. TCP_FIN_TIMEOUT This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state:
# echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout (default: 60 seconds, recommended 15-30 seconds)
2. TCP_KEEPALIVE_INTERVAL determines the wait time between isAlive interval probes. To set:
# echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl (default: 75 seconds, recommended: 15-30 seconds)
3. TCP_KEEPALIVE_PROBES determines the number of probes before timing out. To set:
# echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes (default: 9, recommended 5)
4. TCP_TW_RECYCLE enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution.
# echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle (boolean, default: 0)
5. TCP_TW_REUSE allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle
# echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse (boolean, default: 0)
For persistently setting above parameters, use the /etc/sysctl.conf configuration file.