• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • VCS
  • Interview Questions
  • Database
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

“Connection reset by peer” – error while ssh into a CentOS/RHEL system with a specific user only

by admin

The Problem

When trying to open an ssh connection to a system with a specific account, it failed with ‘Connection reset by peer’. Other users can successfully connect with ssh to this system.

Below is an example showing the failed login to node [NODE2] with the account ‘oracle’.

[oracle@NODE1]$ ssh oracle@[NODE2]
oracle@[NODE2]'s password:
Read from remote host [NODE2]: Connection reset by peer
Connection to [NODE2] closed.
[oracle@NODE1]$ ssh root@[NODE2]
root@[NODE2]'s password:
Last login: Fri Mar 6 02:30:55 2009 from [NODE1]

The error information can be found in /var/log/messages log on node [NODE2]:

Feb 29 11:11:11 [NODE2] sshd[7194]: Accepted password for oracle from ::ffff:xx.xx.xx.xx port 24318 ssh2
Feb 29 11:11:11 [NODE2] sshd[7202]: fatal: setresuid 501: Resource temporarily unavailable

The Solution

More users have been added to this system. The limit value of ‘soft nofile‘ or ‘soft nproc‘ in file /etc/security/limits.conf is in effect. The /etc/security/limits.conf file sets limits on system resources for each user.

For example, the value for opened files returned by lsof is higher than the limit value of ‘soft nofile’ on node NODE2 for account ‘oracle’.

1. Check /etc/security/limits.conf:

[oracle@ ~]$cat /etc/security/limits.conf
# /etc/security/limits.conf
#
#Each line describes a limit for a user in the form:
#            
#Where:
# can be:
#        - an user name
#        - a group name, with @group syntax
#        - the wildcard *, for default entry
#        - the wildcard %, can be also used with %group syntax,
#                 for maxlogin limit
# can have the two values:
#        - "soft" for enforcing the soft limits
#        - "hard" for enforcing hard limits
# can be one of the following:
#        - core - limits the core file size (KB)
#        - data - max data size (KB)
#        - fsize - maximum filesize (KB)
#        - memlock - max locked-in-memory address space (KB)
#        - nofile - max number of open files
#        - rss - max resident set size (KB)
#        - stack - max stack size (KB)
#        - cpu - max CPU time (MIN)
#        - nproc - max number of processes
#        - as - address space limit
#        - maxlogins - max number of logins for this user
#        - maxsyslogins - max number of logins on the system
#        - priority - the priority to run user process with
#        - locks - max number of file locks the user can hold
#        - sigpending - max number of pending signals
#        - msgqueue - max memory used by POSIX message queues (bytes)
#[domain]      [type]  [item]         [value]
oracle hard nofile 65535
oracle soft nofile 4096
oracle hard nproc 20480
oracle soft nproc 2047

2. Check the processes run by user ‘oracle’:

[oracle@NODE2 ~]$ ps -u oracle|wc -l
489

3. Check the files opened by user ‘oracle’:

[oracle@[NODE2] ~]$ /usr/sbin/lsof -u oracle | wc -l
62490

Once you have identified the parameters limits, follow the steps outlined below to resolve the issue:

1. Modify /etc/security/limits.conf manually. Increase the value of ‘soft nofile‘ until it is equal to ‘hard nofile‘ value. Increase the value of ‘soft nproc‘ until it is equal to ‘hard nproc‘ value.

[oracle@NODE2 ~]$cat /etc/security/limits.conf
oracle hard nofile 65535
oracle soft nofile 65535
oracle hard nproc 20480
oracle soft nproc 20480

2. Check if the issue still exists:

[oracle@NODE1 ~]$ssh oracle@NODE2
oracle@NODE2's password:
Last login: Fri Mar 6 02:33:01 2009 from NODE1

Different errors when other limit settings are reached

The error is different when it is reaching the limit ‘open files’ and ‘max user processes’ in /etc/profile.

1. Error on reaching the limit ‘open files’:

[oracle@NODE1~]$ssh NODE2
oracle@NODE2's password:
-bash: ulimit: max user processes: cannot modify limit: Operation not permitted
-bash: /home/oracle/.bash_profile: Too many open files

2. Error on reaching the limit ‘max user processes’:

[oracle@NODE1 ~]$ssh oracle@NODE2
oracle@NODE2's password:
-bash: ulimit: open files: cannot modify limit: Operation not permitted
-bash: fork: Resource temporarily unavailable

Filed Under: CentOS/RHEL, CentOS/RHEL 5, CentOS/RHEL 6, CentOS/RHEL 7

Some more articles you might also be interested in …

  1. How to Remove virbr0 and lxcbr0 Interfaces on CentOS/RHEL 6,7
  2. Beginners Guide to DHCP – Install and configure DHCP server and client
  3. Time goes out of sync on a node running CentOS/RHEL 7
  4. How To Store iptables Rules To Survive Across Reboots in CentOS/RHEL
  5. How to scan newly Assigned LUNs in Multipathd under CentOS / RHEL
  6. How to Re-Create the Yum Cache and/or Force a Fetch of the Package List of the Enabled Repositories
  7. How to Change Kernel Semaphore Limits in CentOS/RHEL
  8. fprintd Logs Messages into /var/log/messages Even If USEFPRINTD=no in /etc/sysconfig/authconfig (CentOS/RHEL 7)
  9. How to Configure Automatic Package Updates on the Server in CentOS/RHEL 8
  10. Bind to port 22 on 0.0.0.0 failed: address already in use – error while starting sshd service CentOS/RHEL

You May Also Like

Primary Sidebar

Recent Posts

  • gml2gv Command Examples
  • glow Command Examples
  • glib-compile-resources Command Examples
  • glances Command Examples

© 2023 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright