The Problem
On an CentOS/RHEL 6 server, all logins started to fail to the system with the following errors:
Server refused to allocate pty /etc/profile: line 48: /dev/null: Permission denied /etc/profile: line 76: /dev/n ull: Permission denied /etc/profile: line 76: /dev/null: Permission denied /etc/pr ofile: line 76: /dev/null: Permission denied /etc/profile: line 76: /dev/null: Permission denied /etc/profile: line 76: /dev/null: Permission denied /etc/profile: line 76: /dev/null: Permission denied /etc/profile: line 76: /dev/null: Permission denied
Applications and database instancea on the server also started to fail.
The Solution
This can happen if the file under directory /dev/shm are removed. When trying to free up space by finding large files to remove in /root and /tmp we might also remove files under /tmp/shm unintentionally. /dev/shm (also known as tmpfs) is a file system used for shared memory between systems or user processes. tmpfs uses virtual memory, rather than accessing disk storage, giving significant performance improvements, and is managed by the Linux kernel.
If tmpfs has files deleted manually from it, it can impact on the running system or user processes and services that can no longer access the contents of shared memory they had been using, and it can thus cause system or application failures.
tmpfs is not intended for long-term storage, and it shouldn’t need to be manually changed or altered by users. In this case, the removal of files from tmpfs caused the allocation of terminals to users trying to log into the system to fail, so it became impossible for any users to log in to the system. Some files that were also being used by databases instances were removed, which causes the databases and applications to fail.
As tmpfs uses virtual memory for file storage, the contents of tmpfs don’t persist between reboots, and the contents of tmpfs are freshly generated whenever a system is rebooted. To resolve this issue, please reboot/reset the system.