ssh, scp or sftp to a node that doesn’t appear in DNS or /etc/hosts file is slow to make an initial connection. After the connection is established, the speed is as expected. There are two cases to consider, see below. Note that in most environments this issue will not occur since IPs will be in /etc/hosts or DNS.
The time is spent while resolving the IP address of the target/source hostname or finding the hostname for the target/source IP address and / or doing authentication based on the very same data. There are two different verbose symptoms when we use -vvv with ssh constituting two different cases.
Case 1
# ssh -vvv 10.10.10.205 ... ... debug1: Next authentication method: gssapi-with-mic debug3: Trying to reverse map address 10.10.10.205. debug1: Unspecified GSS failure. Minor code may provide more information No credentials cache found
In CentOS/RHEL systems, SSH daemon is configured to use Generic Security Services Application Program Interface (GSSAPI) authentication by default. The GSSAPI by default does a lookup (via DNS or other means based on /etc/resolv.conf) for the requested IP. Note that the lookup happens even if you use IP addresses. That is required for a secure/reliable authentication.
If you check the SSH configuration manpage:
# man ssh_config
...
GSSAPIAuthentication
Specifies whether user authentication based on GSSAPI is allowed. The default is no
...
If the IP / host does not appear in /etc/hosts or in DNS database, the lookup query times out, as it re-tries, eventually it gives up and continues to open the connection. The time is spent during the lookup (50 seconds or more), until the password prompt is displayed. This stall only happens to hosts that are on the local LAN and not in DNS or /etc/hosts file. Notice the authentication method is GSSAPI.
Case 2
# ssh -vvv 10.10.10.12 . . . debug1: Offering public key: /root/.ssh/id_dsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply --stalls for 8 seconds--
Here, the remote host is trying to do a DNS lookup on the client node IP. It might be a brand new physical or virtual machine installation that’s not updated in /etc/hosts or DNS and hence stalls until the DNS query times out. Again, the DNS / hosts lookup is done (based on the /etc/resolv.conf on client / server).
The Solution
The supported and proper solution to this problem is to have the hostname / IP lookups succeed. To do that, you may:
- Check your configuration in /etc/resolv.conf for name resolution and DNS server configuration.
- If not defined, define the newly introduced host / IP to be included in the DNS server
- If it is a problem or unworthy to do DNS definitions (most likely the assignment is temporary – i.e. a test system, a temporary virtual machine etc.) make sure that the hostname / IP address pair is included in the source and target environments’ /etc/hosts file (this is way more practical)
Workarounds
Without a proper DNS / hosts (or /etc/resolv.conf config for that matter), the SSH connections are meant to be initially slow. That is how it is expected to be in OEL. Alternatively, the wait-up situation may be worked around by some configuration changes.
For Case 1 the GSSAPIAuthentication could have been disabled. Although that is possible it is strongly discouraged as the use of GSSAPI is one of the fundamental security feature provided by SSH. SSH not only enables the communication contents to be secure but also can make sure that that target is the one that is intended. That is because the GSSAPIAuthentication is set to yes in OEL (although the man page says it is “No” by default – but it is the default of the core ssh packages – not the specific distribution). Still, it is possible to disable it, if you are perfectly sure that you are on a closed network and aware that you know what you are doing:
Temporary setup – just to see if it works on the command line:
# ssh -o "GSSAPIAuthentication no" 10.10.10.205
Password prompt should appear instantly if Case 1, is your case. To set this up permanently (despite any recommendation otherwise) you can change GSSAPIAuthentication to no in one of these ways on the client:
1. Add line below to user’s home directory ssh config file (~/.ssh/config):
GSSAPIAuthentication no
2. Add it to the system SSH configuration file at the client. i.e. – /etc/ssh/ssh_config and on the server side. Edit server side (i.e. system you are connecting to, in this case it is 10.10.10.205) configuration file – /etc/ssh/sshd_config and restart sshd:
# service sshd restart
this can be done while there are already established SSH connections.
For Case 2, if you somehow cannot/do not want to alter the DNS configuration or /etc/hosts files (despite any recommendation otherwise), you can work around the situation by disabling DNS lookups done for the SSH protocol. You can do it by setting
UseDNS no
in the server SSH configuration file – /etc/ssh/sshd_config
and restart sshd on the server
# service sshd restart
this can be done while connections are active.