• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • VCS
  • Interview Questions
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

SLES 12: Database Startup Error with ORA-27300 ORA-27301 ORA-27303 While Starting using Srvctl

by admin

The Problem

12.1 RAC database error while starting via srvctl, alert log showing database startup and further goes down:

alert_ORCL2.log inside [oracle base]/diag/rdbms/[db name]/[SID name]/trace:

Fri Nov 03 15:06:25 2017
Adjusting the default value of parameter parallel_max_servers
from 960 to 486 due to the value of parameter processes (600)
Starting ORACLE instance (normal) (OS id: 19684)
.
.
Fri Nov 03 15:10:45 2017
Process startup failed, error stack:
Fri Nov 03 15:10:45 2017
Errors in file /u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_psp0_19706.trc:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3
Fri Nov 03 15:10:46 2017
Shutting down instance (abort)
License high water mark = 2
Fri Nov 03 15:10:46 2017
USER (ospid: 22067): terminating the instance
Fri Nov 03 15:10:47 2017
Instance terminated by USER, pid = 22067
Fri Nov 03 15:10:47 2017
Instance shutdown complete
Errors in file //u01/app/oracle/diag/rdbms/ORCL/ORCL2/trace/ORCL2_mmon_19853.trc (incident=14681):
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
Incident details in: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

ORCL2_psp0_19706.trc:

*** 2017-11-03 15:10:44.989
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

*** 2017-11-03 15:10:45.993
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 11
ORA-27301: OS failure message: Resource temporarily unavailable
ORA-27302: failure occurred at: skgpspawn3

ORCL2_mmon_19853.trc:

*** 2017-11-03 15:09:57.908
***KELR Apply Log: unable to schedule MMON Slave, error 3

*** 2017-11-03 15:10:47.847
Incident 14681 created, dump file: /u01/app/oracle/diag/rdbms/ORCL/ORCL2/incident/incdir_14681/ORCL2_mmon_19853_i14681.trc
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1

KEBM: MMON action policy violation. 'PQ: Adjust Slave Pool' viol=0; err=600
error 0 detected in background process
kgxgnsdr: clssgsshdereg: warning: return status 26 (-558242808 )
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
kgxgnsdr: clssgsshdereg: warning: return status 26 (-813323584 )

Following error reported in OS logs

messages:

2017-11-03T15:10:11.690393+01:00 rachost1 Oracle Audit[7888]: LENGTH : '200' ACTION :[52] 'ALTER DATABASE MOUNT /* db agent *//* {2:39656:2} */' DATABASE USER:[1] '/' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[0] '' STATUS:[1] '0' DBID:[10] '2949004148'
2017-11-03T15:10:47.700094+01:00 rachost1 kernel: [ 173.788859] cgroup: fork rejected by pids controller in /system.slice/ohasd.service <<<<<<<<<<<<<<<<<<<<<

Database able to startup with sqlplus.

The Solution

This is due the PIDs cgroup controller introduced with SUSE12.

From the "Support for PIDs cgroup Controller" section on https://www.suse.com/releasenotes/x86_64/SUSE-SLES/12-SP2/#fate-320358

"To control the default TasksMax= setting for services and scopes running on the system, use the system.conf setting DefaultTasksMax=. This setting defaults to 512, which means services that are not explicitly configured otherwise will only be able to create 512 processes or threads at maximum.

For thread- or process-heavy services, you may need to set a higher TasksMax value. In such cases, set TasksMax directly in the specific unit files. Either choose a numeric value or even infinity."

From SLES12 onwards, systemd is used instead of initd and the OHASD server is only allowed to open a maximum of 512 tasks. Configure the value of DefaultTasksMax to 65535 in the file /etc/systemd/system.conf or set the TasksMax value properly for the ohasd systemd service.

For example:

# cat /etc/systemd/system/ohasd.service.d/lunar.conf
[Service]
TasksMax=16384
# systemctl status ohasd
● ohasd.service - LSB: Start and Stop Oracle High Availability Service
Loaded: loaded (/etc/init.d/ohasd; bad; vendor preset: disabled)
Drop-In: /etc/systemd/system/ohasd.service.d
└─lunar.conf
Active: active (exited) since Mon 2017-11-13 14:29:23 CET; 3h 5min ago
Docs: man:systemd-sysv-generator(8)
Process: 4876 ExecStart=/etc/init.d/ohasd start (code=exited, status=0/SUCCESS)
Tasks: 612 (limit: 16384) <<<<<<=================== here was the limit with 512

Filed Under: oracle, oracle 12c, SuSE Linux Enterprise

Some more articles you might also be interested in …

  1. WITH Clause in ORACLE
  2. How to change the ASM rebalance power of an ongoing operation
  3. Create Password file for database in ASM Diskgroup – Oracle 12c RAC only
  4. How to Migrate ASM Disk Groups to another Storage Online [When ASMLIB Devices Are Involved]
  5. How to Start and Stop OSWatcher
  6. Oracle Database 18c : How to Merge Partitions And Subpartitions Online
  7. How to Roll Forward a standby database using RMAN incremental backup in 11g
  8. Oracle Database 18c New Feature – Memoptimized Rowstore
  9. Oracle Database: Redo log operations (Add/Drop/Change Location)
  10. How to resize an OCFS2 filesystem on Linux

You May Also Like

Primary Sidebar

Recent Posts

  • aws ec2: CLI for AWS EC2 (Command Examples)
  • aws cur – Create, query, and delete AWS usage report definitions (Command Examples)
  • aws configure – Manage configuration for the AWS CLI (Command Examples)
  • aws cognito-idp: Manage Amazon Cognito user pool and its users and groups using the CLI

© 2023 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright