Hello folks
I am getting below error while start up Apache in oracle Ebs R12.1.3
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/admin/scripts$ ./adopmnctl.sh status -l
You are running adopmnctl.sh version 120.6.12010000.5
Checking status of OPMN managed processes...
Processes in Instance: InstanceName.xxx
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
ias-component | process-type | pid | status | uid | memused | uptime | ports
---------------------------------+--------------------+---------+----------+------------+----------+-----------+------
OC4JGroup:default_group | OC4J:oafm | N/A | Down | N/A | N/A | N/A | N/A
OC4JGroup:default_group | OC4J:forms | 6168 | Alive | 556809033 | 142480 | 0:51:19 | rmi:20600,jms:23600,ajp:22100
OC4JGroup:default_group | OC4J:oacore | 6033 | Alive | 556809032 | 175064 | 0:51:50 | rmi:20102,jms:23102,ajp:21600
OC4JGroup:default_group | OC4J:oacore | 6035 | Alive | 556809031 | 175072 | 0:51:50 | rmi:20100,jms:23100,ajp:21601
OC4JGroup:default_group | OC4J:oacore | 6034 | Alive | 556809030 | 175072 | 0:51:50 | rmi:20101,jms:23101,ajp:21602
HTTP_Server | HTTP_Server | 5934 | Alive | 556809029 | 2485992 | 0:51:28 | https1:4463,http1:8020
adopmnctl.sh: exiting with status 0
adopmnctl.sh: check the logfile /opt/oraappl/`/inst/apps/InstanceName/logs/appl/admin/log/adopmnctl.txt for more information ...
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/admin/scripts$ ./adapcctl.sh status -l
You are running adapcctl.sh version 120.7.12010000.2
Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...
Processes in Instance: InstanceName.xxx
---------------------------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group | OC4J:oafm | N/A | Down
OC4JGroup:default_group | OC4J:forms | 6168 | Alive
OC4JGroup:default_group | OC4J:oacore | 6033 | Alive
OC4JGroup:default_group | OC4J:oacore | 6035 | Alive
OC4JGroup:default_group | OC4J:oacore | 6034 | Alive
HTTP_Server | HTTP_Server | 5934 | Alive
adapcctl.sh: exiting with status 0
adapcctl.sh: check the logfile /opt/oraappl/Instance/inst/apps/InstanceName/logs/appl/admin/log/adapcctl.txt for more information ...
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/admin/scripts$
-------------
ccc:/opt/oraappl/InstanceName/inst/apps/InstanceName/admin/scripts$ ps -ef | grep Apache
applmgr 13914 3078 0 13:38:37 pts/1 0:00 grep Apache
applmgr 6008 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 6002 5934 0 12:53:35 ? 0:01 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 5998 5934 0 12:53:34 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/rotatelogs /opt/oraa
applmgr 6005 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 6004 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 6080 5934 0 12:53:55 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 6001 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/rotatelogs /opt/oraa
applmgr 6000 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/rotatelogs /opt/oraa
applmgr 5999 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/rotatelogs /opt/oraa
applmgr 6003 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 5934 5799 0 12:53:31 ? 0:01 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
applmgr 6006 5934 0 12:53:35 ? 0:00 /opt/oraappl/InstanceName/apps/tech_st/10.1.3/Apache/Apache/bin/httpd -d /opt/oraapp
----
#1. Checking Oracle EBS OPMN Log
$LOG_HOME/ora/10.1.3/opmn/opmn.log
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/logs/ora/10.1.3/opmn$ pwd
/opt/oraappl/Instance/inst/apps/InstanceName/logs/ora/10.1.3/opmn
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/logs/ora/10.1.3/opmn$ vi opmn.log
15/12/25 12:53:28 [ons-internal] ONS server initiated
15/12/25 12:53:28 [pm-internal] PM state directory exists: /opt/oraappl/Instance/inst/apps/Instance_Name/ora/10.1.3/opmn/logs/states
15/12/25 12:53:28 [pm-internal] OPMN server ready. Request handling enabled.
15/12/25 12:53:30 [pm-requests] Request 4 Started. Command: /start?process-type=HTTP_Server
15/12/25 12:53:30 [pm-process] Starting Process: HTTP_Server~HTTP_Server~HTTP_Server~1 (556809029:0)
15/12/25 12:53:34 [pm-process] Process Alive: HTTP_Server~HTTP_Server~HTTP_Server~1 (556809029:5934)
15/12/25 12:53:34 [pm-requests] Request 4 Completed. Command: /start?process-type=HTTP_Server
15/12/25 12:53:35 [pm-requests] Request 7 Started. Command: /start?process-type=oacore
15/12/25 12:53:35 [pm-process] Starting Process: default_group~oacore~default_group~1 (556809030:0)
15/12/25 12:53:35 [pm-process] Starting Process: default_group~oacore~default_group~2 (556809031:0)
15/12/25 12:53:35 [pm-process] Starting Process: default_group~oacore~default_group~3 (556809032:0)
15/12/25 12:53:54 [pm-process] Process Alive: default_group~oacore~default_group~3 (556809032:6033)
15/12/25 12:53:58 [pm-process] Process Alive: default_group~oacore~default_group~2 (556809031:6035)
15/12/25 12:54:02 [pm-process] Process Alive: default_group~oacore~default_group~1 (556809030:6034)
15/12/25 12:54:02 [pm-requests] Request 7 Completed. Command: /start?process-type=oacore
15/12/25 12:54:06 [pm-requests] Request a Started. Command: /start?process-type=forms
15/12/25 12:54:06 [pm-process] Starting Process: default_group~forms~default_group~1 (556809033:0)
15/12/25 12:54:14 [pm-process] Process Alive: default_group~forms~default_group~1 (556809033:6168)
15/12/25 12:54:14 [pm-requests] Request a Completed. Command: /start?process-type=forms
15/12/25 12:54:15 [pm-requests] Request d Started. Command: /start?process-type=oafm
15/12/25 12:54:15 [pm-process] Starting Process: default_group~oafm~default_group~1 (556809034:0)
15/12/25 13:04:16 [pm-requests] Request d Completed. Command: /start?process-type=oafm
15/12/25 13:04:16 [pm-process] Stopping Process: default_group~oafm~default_group~1 (556809034:6204)
15/12/25 13:04:16 [libopmnoc4j] Process Stop Error: default_group~oafm~default_group~1 (556809034:6204)
15/12/25 13:04:16 [libopmnoc4j] Failed to construct stop command for proc: 556809034
15/12/25 13:04:16 [libopmnoc4j] Forcefully Terminating Process: default_group~oafm~default_group~1 (556809034:6204)
15/12/25 13:04:18 [pm-process] Process Stopped: default_group~oafm~default_group~1 (556809034:6204)
#2. Checking Oracle EBS R12 Startup logfile
$INST_TOP/logs/appl/admin/log
Executing service control script:
/opt/oraappl/Instance/inst/apps/Instance_Instance/admin/scripts/adformsctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
You are running adformsctl.sh version 120.16.12010000.3
Environment file required to start forms server in servlet mode not found.
adformsctl.sh: exiting with status 10
.end std out.
.end err out.
****************************************************
Executing service control script:
/opt/oraappl/Instance/inst/apps/Instance_Name/admin/scripts/adapcctl.sh stop
Timeout specified in context file: 100 second(s)
script returned:
****************************************************
You are running adapcctl.sh version 120.7.12010000.2
Stopping OPMN managed Oracle HTTP Server (OHS) instance ...
adapcctl.sh: exiting with status 150
adapcctl.sh: check the logfile /opt/oraappl/Instance/inst/apps/Instance_Name/logs/appl/admin/log/adapcctl.txt for more information ...
.end std out.
.end err out.
****************************************************
#3 Check Oracle ebs r12 Apache and error logfile
$LOG_HOME/ ora/ 10.1.3/ Apache/ error_log & access_log
I found below message in error log.
[Fri Dec 25 12:53:55 2015] [info] mod_oc4j: Inserting instance=InstanceName, opmnid=556809032 into proc table
[Fri Dec 25 12:53:55 2015] [info] got an event: IAS/PM/PROC_ALIVE
[Fri Dec 25 12:53:58 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:53:58 2015] [info] mod_oc4j: Inserting instance=InstanceName, opmnid=556809031 into proc table
[Fri Dec 25 12:53:58 2015] [info] got an event: IAS/PM/PROC_ALIVE
[Fri Dec 25 12:54:02 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:02 2015] [info] mod_oc4j: Inserting instance=InstanceName, opmnid=556809030 into proc table
[Fri Dec 25 12:54:02 2015] [info] got an event: IAS/PM/PROC_ALIVE
[Fri Dec 25 12:54:06 2015] [info] got an event: IAS/PM/PROC_START
[Fri Dec 25 12:54:15 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:15 2015] [info] mod_oc4j: Inserting instance=InstanceName, opmnid=556809033 into proc table
[Fri Dec 25 12:54:15 2015] [info] got an event: IAS/PM/PROC_ALIVE
[Fri Dec 25 12:54:15 2015] [info] got an event: IAS/PM/PROC_START
[Fri Dec 25 12:54:24 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:28 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:32 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:45 2015] [info] got an event: IAS/PM/PROC_READY
[Fri Dec 25 12:54:54 2015] [info] got an event: IAS/PM/PROC_READY
-----
Solution :
Go to $INST_TOP/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1 on application tier and look for jms.state and *.lock files. If you find them in this location once all you services are stop. Delete or rename them. Once you are done try again. Now OACORE OC4J service will come up.
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1$ ls -l
total 20
-rw-r----- 1 applmgr dba 512 Dec 18 08:49 Oc4jJmsExceptionQueue
-rw-r----- 1 applmgr dba 4096 Dec 25 12:53 jms.state
drwxr-x--- 3 applmgr dba 3 Dec 18 08:49 persistence
-rw-r----- 1 applmgr dba 512 Dec 18 08:49 scheduler_events
-rw-r----- 1 applmgr dba 122 Dec 25 12:53 scheduler_events.lock
-rw-r----- 1 applmgr dba 512 Dec 18 08:49 scheduler_jobstore
-rw-r----- 1 applmgr dba 122 Dec 25 12:53 scheduler_jobstore.lock
-rw-r----- 1 applmgr dba 512 Dec 18 08:49 scheduler_notifications
-rw-r----- 1 applmgr dba 122 Dec 25 12:53 scheduler_notifications.lock
ccc:/opt/oraappl/Instance/inst/apps/InstanceName/ora/10.1.3/j2ee/oacore/persistence/oacore_default_group_1$
