In a Kubernetes architecture context, I am dealing with a problem that I cannot explain.
Here I have my configuration of a statefulset oracle :
containers:
- name: oracle
image: xx.xx/oracle:**19.13.0**
command:
- /bin/bash
- '-c'
- >
sed '/\/network\/admin\/sqlnet.ora/a echo \"SQLNET.EXPIRE_TIME=1\"
>> \"\$ORACLE_HOME\"\/network\/admin\/sqlnet.ora' -i
/opt/oracle/createDB.sh
/opt/oracle/runOracle.sh
env:
- name: ORACLE_CHARACTERSET
value: WE8MSWIN1252
- name: ORACLE_PWD
value: gOrwell1984
- name: INIT_SGA_SIZE
value: '7500'
- name: INIT_PGA_SIZE
value: '3000'
resources:
limits:
cpu: '4'
memory: 8Gi
requests:
cpu: '2'
memory: 4Gi
volumeMounts:
- name: data
mountPath: /opt/oracle/oradata
subPath: oracle
readinessProbe:
exec:
command:
- /bin/sh
- '-c'
- >-
if [ -f $ORACLE_BASE/checkDBLockStatus.sh ]; then
$ORACLE_BASE/checkDBLockStatus.sh ; else
$ORACLE_BASE/checkDBStatus.sh; fi
I set the scale on 1 replicas. This creates a pod, at the level of the oracle container logs, when initializing the database I get this error message:
/opt/oracle/createDB.sh: line 181: 54 Killed dbca -silent -createDatabase -enableArchive "$ENABLE_ARCHIVELOG" -archiveLogDest "$ARCHIVELOG_DIR" ${DBCA_CRED_OPTIONS} -responseFile "$ORACLE_BASE"/dbca.rsp
I’m trying to find a container level log to make this clearer but I can’t find anything. Something else at the node level on the syslog logs I see an oom message :
Mar 12 10:54:44 k8s containerd[927]: time="2024-03-12T10:54:44.438718600Z" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
Mar 12 10:54:44 k8s containerd[927]: time="2024-03-12T10:54:44.438790200Z" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
Mar 12 10:54:44 k8s containerd[927]: time="2024-03-12T10:54:44.438808400Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
Mar 12 10:54:44 k8s containerd[927]: time="2024-03-12T10:54:44.439782697Z" level=info msg="starting signal l 887 ora_svcb_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746853] [3718373] 54321 3718373 2036865 16120 708608 0 887 ora_pman_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746854] [3718377] 54321 3718377 2037952 19212 724992 0 887 ora_dia0_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746856] [3718379] 54321 3718379 2039889 19312 917504 0 887 ora_dbw0_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746857] [3718383] 54321 3718383 2037137 18807 757760 0 887 ora_lgwr_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746858] [3718387] 54321 3718387 2036880 17835 864256 0 887 ora_ckpt_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746859] [3718391] 54321 3718391 2036815 16162 729088 0 887 ora_lg00_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746860] [3718393] 54321 3718393 2036900 19850 921600 0 887 ora_smon_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746861] [3718397] 54321 3718397 2036814 15985 712704 0 887 ora_lg01_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746862] [3718399] 54321 3718399 2036863 16272 704512 0 887 ora_smco_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746863] [3718401] 54321 3718401 2037171 20274 942080 0 887 ora_reco_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746864] [3718403] 54321 3718403 2036799 15821 696320 0 887 ora_w000_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746865] [3718405] 54321 3718405 2038595 23059 995328 0 887 ora_lreg_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746867] [3718409] 54321 3718409 2036798 15811 700416 0 887 ora_w001_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746868] [3718411] 54321 3718411 2036864 15787 688128 0 887 ora_pxmn_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746869] [3718421] 54321 3718421 2037363 15050 659456 0 887 ora_d000_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746871] [3718423] 54321 3718423 2037186 14197 643072 0 887 ora_s000_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746872] [3718425] 54321 3718425 2036863 16125 708608 0 887 ora_tmon_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746873] [3718496] 54321 3718496 2036863 15773 688128 0 887 ora_mmnl_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746874] [3718498] 54321 3718498 2040737 21902 790528 0 887 ora_mmon_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746875] [3718504] 54321 3718504 2037951 17616 712704 0 887 ora_m000_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746876] [3718564] 54321 3718564 2042464 16233 753664 0 887 ora_tt00_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746877] [3718579] 54321 3718579 2036798 15670 696320 0 887 ora_tt01_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746878] [3718581] 54321 3718581 2036799 15646 700416 0 887 ora_tt02_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746880] [3718667] 54321 3718667 2036863 15688 720896 0 887 ora_cjq0_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746881] [3718668] 54321 3718668 2037725 26727 1126400 0 887 oracle_963_orcl
Mar 12 11:22:00 k8s kernel: [2759986.746882] [3718669] 54321 3718669 2927 667 69632 0 887 datapatch
Mar 12 11:22:00 k8s kernel: [2759986.746883] [3718671] 54321 3718671 2927 680 69632 0 887 sqlpatch
Mar 12 11:22:00 k8s kernel: [2759986.746884] [3718676] 54321 3718676 43697 12673 327680 0 887 perl
Mar 12 11:22:00 k8s kernel: [2759986.746885] [3718678] 54321 3718678 2063528 44426 1421312 0 887 oracle_973_orcl
Mar 12 11:22:00 k8s kernel: [2759986.746886] [3718744] 54321 3718744 2036799 15502 720896 0 887 ora_w002_orclcd
Mar 12 11:22:00 k8s kernel: [2759986.746887] [3718758] 54321 3718758 27466 5638 196608 0 887 sqlplus
Mar 12 11:22:00 k8s kernel: [2759986.746888] [3718759] 54321 3718759 27341 5517 172032 0 887 sqlplus
Mar 12 11:22:00 k8s kernel: [2759986.746890] [3718760] 54321 3718760 27341 5465 188416 0 887 sqlplus
Mar 12 11:22:00 k8s kernel: [2759986.746891] [3718761] 54321 3718761 27341 5503 180224 0 887 sqlplus
Mar 12 11:22:00 k8s kernel: [2759986.746892] [3718763] 54321 3718763 2038431 41224 1335296 0 887 oracle_1029_orc
Mar 12 11:22:00 k8s kernel: [2759986.746893] [3718765] 54321 3718765 2037607 22399 905216 0 887 oracle_1031_orc
Mar 12 11:22:00 k8s kernel: [2759986.746894] [3718767] 54321 3718767 2037609 22411 897024 0 887 oracle_1033_orc
Mar 12 11:22:00 k8s kernel: [2759986.746895] [3718769] 54321 3718769 2037608 22370 892928 0 887 oracle_1035_orc
Mar 12 11:22:00 k8s kernel: [2759986.746896] oom-kill:constraint=CONSTRAoop" namespace=moby path=/run/containerd/io.containerd.runtime.v2.task/moby/a8856c2174a8f8d4d9e190c893e8e09b2d8c952f86a6003cbed76b9da4a7948e pid=3666469 runtime=io.containerd.runc.v2
But even if I increase the memory, nothing works.
If you can help me please
The entire container log :
ssh-keygen: generating new host keys: RSA1 RSA DSA ECDSA ED25519
ORACLE EDITION: STANDARD
LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 12-MAR-2024 11:19:36
Copyright (c) 1991, 2021, Oracle. All rights reserved.
Starting /opt/oracle/product/19c/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/oracle-0/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date 12-MAR-2024 11:19:36
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/oracle-0/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
The listener supports no services
The command completed successfully
Prepare for db operation
8% complete
Copying database files
31% complete
Creating and starting Oracle instance
32% complete
36% complete
/opt/oracle/createDB.sh: line 181: 54 Killed dbca -silent -createDatabase -enableArchive "$ENABLE_ARCHIVELOG" -archiveLogDest "$ARCHIVELOG_DIR" ${DBCA_CRED_OPTIONS} -responseFile "$ORACLE_BASE"/dbca.rsp
[ 2024-03-12 11:19:44.320 UTC ] Prepare for db operation
DBCA_PROGRESS : 8%
[ 2024-03-12 11:19:44.419 UTC ] Copying database files
DBCA_PROGRESS : 31%
[ 2024-03-12 11:21:10.994 UTC ] Creating and starting Oracle instance
DBCA_PROGRESS : 32%
DBCA_PROGRESS : 36%
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Mar 12 11:22:00 2024
Version 19.13.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.13.0.0.0
SQL> ALTER SYSTEM SET control_files='/opt/oracle/oradata/ORCLCDB/control01.ctl' scope=spfile
*
ERROR at line 1:
ORA-32001: write to SPFILE requested but no SPFILE is in use
SQL>
System altered.
SQL> ALTER PLUGGABLE DATABASE ORCLPDB1 SAVE STATE
*
ERROR at line 1:
ORA-65011: Pluggable database ORCLPDB1 does not exist.
SQL>
PL/SQL procedure successfully completed.
SQL> SQL>
Session altered.
SQL>
User created.
SQL>
Grant succeeded.
SQL>
Grant succeeded.
SQL>
User altered.
SQL> SQL> Disconnected from Oracle Database 19c Standard Edition 2 Release 19.0.0.0.0 - Production
Version 19.13.0.0.0
The Oracle base remains unchanged with value /opt/oracle
mv: cannot stat '/opt/oracle/product/19c/dbhome_1/dbs/spfileORCLCDB.ora': No such file or directory
Executing user defined scripts
/opt/oracle/runUserScripts.sh: running /opt/oracle/scripts/extensions/setup/savePatchSummary.sh
DONE: Executing user defined scripts
Executing user defined scripts
/opt/oracle/runUserScripts.sh: running /opt/oracle/scripts/setup/setNLS.sh
The Oracle base remains unchanged with value /opt/oracle
SQL*Plus: Release 19.0.0.0.0 - Production on Tue Mar 12 11:22:06 2024
Version 19.13.0.0.0
Copyright (c) 1982, 2021, Oracle. All rights reserved.
Connected to an idle instance.
SQL> SQL> ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4475
Additional information: 1135834575
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/opt/oracle/product/19c/dbhome_1/dbs/initORCLCDB.ora'
SQL> SQL> alter system set nls_length_semantics=CHAR scope=both
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4475
Additional information: 1135834575
Process ID: 0
Session ID: 0 Serial number: 0
SQL> SQL> ORA-01078: failure in processing system parameters
LRM-00109: could not open parameter file '/opt/oracle/product/19c/dbhome_1/dbs/initORCLCDB.ora'
SQL> SELECT value from V$PARAMETER WHERE NAME LIKE '%nls_length_semantics%'
*
ERROR at line 1:
ORA-01034: ORACLE not available
Process ID: 0
Session ID: 0 Serial number: 0
SQL> Disconnected
DONE: Executing user defined scripts
The Oracle base remains unchanged with value /opt/oracle
#####################################
########### E R R O R ###############
DATABASE SETUP WAS NOT SUCCESSFUL!
Please check output for further info!
########### E R R O R ###############
#####################################
The following output is now a tail of the alert.log:
PMON (ospid: 837): terminating the instance due to ORA error 822
2024-03-12T11:22:03.323054+00:00
Cause - 'Instance is being terminated due to fatal process death (pid: 7, ospid: 859, MMAN)'
2024-03-12T11:22:03.324781+00:00
System state dump requested by (instance=1, osid=837 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /opt/oracle/diag/rdbms/orclcdb/ORCLCDB/trace/ORCLCDB_diag_868.trc
2024-03-12T11:22:04.975778+00:00
Dumping diagnostic data in directory=[cdmp_20240312112203], requested by (instance=1, osid=837 (PMON)), summary=[abnormal instance termination].
2024-03-12T11:22:06.184927+00:00
Instance terminated by PMON, pid = 837