• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • VCS
  • Interview Questions
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

SLES 12: Database Startup Error with ORA-27300 ORA-27301 ORA-27303 While Starting using Srvctl

by admin

The Problem

12.1 RAC database error while starting via srvctl, alert log showing database startup and further goes down:

alert_ORCL2.log inside [oracle base]/diag/rdbms/[db name]/[SID name]/trace:

Fri Nov 03 15:06:25 2017
Adjusting the default value of parameter parallel_max_servers
from 960 to 486 due to the value of parameter processes (600)
Starting ORACLE instance (normal) (OS id: 19684)
.
.
Fri Nov 03 15:10:45 2017
Process startup failed, error stack:
Fri Nov 03 15:10:45 2017
Errors in file /u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_psp0_19706.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Fri Nov 03 15:10:46 2017
Shutting down instance (abort)
License high water mark = 2
Fri Nov 03 15:10:46 2017
USER (ospid: 22067): terminating the instance
Fri Nov 03 15:10:47 2017
Instance terminated by USER, pid = 22067
Fri Nov 03 15:10:47 2017
Instance shutdown complete
Errors in file //u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_mmon_19853.trc (incident=14681):
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Incident details in: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

ORCL2_psp0_19706.trc:

*** 2017-11-03 15:10:44.989
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

*** 2017-11-03 15:10:45.993
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

ORCL2_mmon_19853.trc:

*** 2017-11-03 15:09:57.908
***KELR Apply Log: unable to schedule MMON Slave, error 3

*** 2017-11-03 15:10:47.847
Incident 14681 created, dump file: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1

KEBM: MMON action policy violation. 'PQ: Adjust Slave Pool' viol=0; err=600
error 0 detected in background process
kgxgnsdr: clssgsshdereg: warning: return status 26 (-558242808 )
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
kgxgnsdr: clssgsshdereg: warning: return status 26 (-813323584 )

Following error reported in OS logs

messages:

2017-11-03T15:10:11.690393+01:00 rachost1 Oracle Audit[7888]: LENGTH : '200' ACTION :[52] 'ALTER DATABASE MOUNT /* db agent *//* {2:39656:2} */' DATABASE USER:[1] '/' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[0] '' STATUS:[1] '0' DBID:[10] '2949004148'
2017-11-03T15:10:47.700094+01:00 rachost1 kernel: [ 173.788859] cgroup: fork rejected by pids controller in /system.slice/ohasd.service <<<<<<<<<<<<<<<<<<<<<

Database able to startup with sqlplus.

The Solution

This is due the PIDs cgroup controller introduced with SUSE12.

From the "Support for PIDs cgroup Controller" section on https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358

"To control the default TasksMax= setting for services and scopes running on the system, use the system.conf setting DefaultTasksMax=. This setting defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum.

For thread- or process-heavy services, you may need to set a higher TasksMax value. In such cases, set TasksMax directly in the specific unit files. Either choose a numeric value or even infinity."

From SLES12 onwards, systemd is used instead of initd and the OHASD server is only allowed to open a maximum of 512 tasks. Configure the value of DefaultTasksMax to 65535 in the file /etc/systemd/system.conf or set the TasksMax value properly for the ohasd systemd service.

For example:

# cat /etc/systemd/system/ohasd.service.d/lunar.conf
[Service]
TasksMax=16384
# systemctl status ohasd
● ohasd.service - LSB: Start and Stop Oracle High Availability Service
Loaded: loaded (/etc/init.d/ohasd; bad; vendor preset: disabled)
Drop-In: /etc/systemd/system/ohasd.service.d
└─lunar.conf
Active: active (exited) since Mon 2017-11-13 14:29:23 CET; 3h 5min ago
Docs: man:systemd-sysv-generator(8)
Process: 4876 ExecStart=/etc/init.d/ohasd start (code=exited, status=0/SUCCESS)
Tasks: 612 (limit: 16384) <<<<<<=================== here was the limit with 512

Filed Under: oracle, oracle 12c, SuSE Linux Enterprise

Some more articles you might also be interested in …

  1. How to solve ORA-28000: the account is locked
  2. How To Disable the Oracle WebLogic Server Default Welcome Page
  3. How to Create Interval-Reference Partitioned Tables in Oracle 12c
  4. How to Startup/Shutdown PDB’s in Oracle Database 12c
  5. Oracle Data Guard 12c – Creating a Physical Standby from Primary
  6. What are the .lok files used in WebLogic
  7. How to Change Timezone Settings on Exadata
  8. How to install and configure Oracle ASMLIB on Native Linux multipath mapper devices
  9. How to Recover From Lost or Missing Database Parameter Files (PFILE or SPFILE)
  10. How to connect sqlplus without tnsnames.ora

You May Also Like

Primary Sidebar

Recent Posts

  • What are /dev/zero and /dev/null files in Linux
  • grpck command – Remove corrupt or duplicate entries in the /etc/group and /etc/gshadow files.
  • xxd command – Expressed in hexadecimal form
  • sesearch: command not found

© 2022 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright