Kernel Logs Warning Messages “kernel: Possible SYN flooding on port X. Sending cookies” is logged”

The Problem

Sometimes Linux kernel logs warning messages as followings:

Mar  7 09:17:14 hostname kernel: TCP: Possible SYN flooding on port 26450. Sending cookies.

or

Mar  7 09:17:14 hostname kernel: TCP: Possible SYN flooding on port 26450. Dropping request.

The Solution

This is a warning message, which indicates that the server is frequently attempted to connect to the specific port, and the kernel warns that this might possibly be an SYN flood attack(=DoS(Denial of Service) attack).

When this message is logged, the kernel returns a syn cookie to the client or just drops the packet for self-guard, which is controlled by /proc/sys/net/ipv4/tcp_syncookies.

What is a SYN Cookie?

SYN cookies are a method by which TCP connections can continue to be established when a socket’s listen backlog fills up. SYN cookies allow connections to continue establishing at times when a socket faces a temporary SYN flood, or when the application does not accept new connections fast enough or at all.

If the system’s valid workload is such that SYN cookies are being logged regularly, the system and application should be tuned to avoid them. An SYN cookie is created by crafting a special SYN+ACK where the TCP Sequence Number is a function of the time, the Maximum Segment Size, and the client and server’s IP address and port numbers.

SYN cookies are sent because the functionality is compiled into the RHEL kernel, and enabled by default. SYN cookies are controlled by the kernel tunable:

# sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1

Determine whether the traffic is valid or malicious

Use application debugging, network monitoring tools, or work with your network team or service provider. This requires an understanding of:

• the application’s expected workload
• expected client IP addresses
• expected client behaviour

Use the netstat or ss commands to inspect TCP socket states as follows, where X is the port number reported in the Possible SYN flooding on port X message:

# netstat -nta | egrep "State|X"
# ss -nta '( dport = :X )'

Having many sockets in the SYN-RECV state could mean a malicious “SYN flood” attack, though this is not the only type of malicious attack. You may also wish to inspect the source IP addresses of traffic to the port in question to confirm if client IPs are expected or unexpected.

Use the tcpdump command to capture network traffic.

A packet capture which displays many SYNs, which the server responds to with SYN+ACK, but the client never replies with the final ACK could mean a traditional “SYN flood” attack, though this is not the only type of malicious attack.

If the traffic is malicious

Work with your network team or service provider to block the traffic before it reaches the listening system or your network. You may also use the iptables firewall to block traffic.

Changing the frequency of logging the message

Please check the port and network traffic whether it is certainly DoS attack. If no attack is confirmed, this message can be ignored. The frequency of logging the message can be controled by 2 kernel parameters below:

• /proc/sys/net/core/message_cost(def=5)
• /proc/sys/net/core/message_burst(def=10)

“message_cost” is “the interval(jiffies) how long the kernel decides it might be SYN flood attack”. “message_burst” is “how frequently the message logs during message_cost”. Reducing the number can reduce the frequency of logging the message.

These can be set by sysctl even on the running production system. For example, adding lines in /etc/sysctl.conf as:

# vi /etc/sysctl.conf
net.core.message_cost = 10
net.core.message_burst = 20

and run the following command after that:

# sysctl -p

This does not affect any system availability.