BPEL Server hangs due to "java.lang.OutOfMemoryError"

-
Oracle(R) BPEL Process Manager - Version: 10.1.2.0.2
Linux x86
Symptoms
The BPEL server hangs and a java.lang.OutOfMemoryError is displayed in the domain.log file.

You will see the following stack trace in the domain.log

<ERROR> <default.collaxa.cube.engine.dispatch>
<DispatchHelper::handleMessage> failed to handle message
javax.ejb.EJBException: Transaction was rolled back: java.lang.OutOfMemoryError; nested
exception
is: java.rmi.RemoteException: ; nested exception is:
java.lang.OutOfMemoryError
java.rmi.RemoteException: ; nested exception is:
java.lang.OutOfMemoryError
at com.evermind.server.ejb.EJBUtils.makeException(EJBUtils.java:886)
at

ICubeDeliveryLocalBean_StatelessSessionBeanWrapper16.handleInvoke(ICubeDeliveryLocalBean_StatelessSe
ssionBeanWrapper16.java:1846)
at

com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessageHandler.handle(InvokeInstanceMe
ssageHandler.java:37)
...
.....
Message handle error.
An exception occurred while attempting to process the message
"com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessage"; the exception is:
Transaction was rolled back: java.lang.OutOfMemoryError; nested exception is:
java.rmi.RemoteException: ; nested exception is:
java.lang.OutOfMemoryError

ORABPEL-05002

Message handle error.
An exception occurred while attempting to process the message
"com.collaxa.cube.engine.dispatch.message.invoke.InvokeInstanceMessage"; the exception is:
Transaction was rolled back: java.lang.OutOfMemoryError; nested exception is:
java.rmi.RemoteException: ; nested exception is:
java.lang.OutOfMemoryError
Cause
The BPEL OC4J container uses 512 MB of memory by default 512MB, which is not enough if you are running BPEL in a high load environment.

You need to increase the JVM settings. Here is are some recommendations for a Linux Box with 4GB of memory and 2 CPUs. (If you are using a different platform from Linux, these parameters may not apply.)

-Xmn1228m -XX:+AggressiveHeap -Xms2048m -Xmx2048m


-Xmn1228m
Garbage collector (Oracle recommends sizing the Eden space to be 60 to 70 percent of the total
heap size.)

-XX:+AggressiveHeap
Recommended if you are using two or more CPUs.
The -XX:+AggressiveHeap option inspects the system resources (size of memory and number of processors) and attempts to set variousparameters to be optimal for long-running, memory allocation-intensive jobs.

-Xms2048m
Iniatial heapsize. The heap size controls the amount of memory the JVM uses

-Xmx2048m
Maximum heapsize

It is also neccesary to apply some patches related to the memory usage in BPEL:

5638122 -->MERGE LABEL REQUEST ON TOP OF 10.1.2.0.2 BUGS 5149866 5595347 5205630
5499891 -->CHANGES MADE FOR BETTER MEMORY USE TO AVOID OUTOFMEMORY


Solution
To implement the solution, please execute the following steps:

1. Shut down your application server
2. Backup the <integration_home>\opmn\conf\opmn.xml
3. Increase the BPEL JVM settings. Edit the file <integration_home>\opmn\conf\opmn.xml :

3.1 Look for the OC4J_BPEL process type section

Your original opmn.xml should look like the example below (just the OC4J_BPEL section)

<process-type id="OC4J_BPEL" module-id="OC4J">
<module-data>
<category id="start-parameters">
<data id="java-options" value="-server-Xrs-Djava.security.policy=D:\Oracle\OAS\portal\MT\j2ee\OC4J_BPEL\config\java2.policy-Djava.awt.headless=true
-Xbootclasspath^/p:D:\Oracle\OAS\portal\MT\integration\orabpel/lib/orabpel-boot.jar;D:\Oracle\OAS\portal\MT\integration\orabpel/lib/connector15.jar
-Doc4j.userThreads=true
-Dorabpel.home=D:\Oracle\OAS\portal\MT\integration\orabpel
-Dhttp.proxySet=true
-Dhttp.proxyHost=www-proxy.us.oracle.com -Dhttp.proxyPort=80
-Dhttp.nonProxyHosts=aalvarad-pc.us.oracle.com|localhost|*.us.oracle.com|aalvarad-pc
-Djava.net.preferIPv4Stack=true -server -Xms256M -Xmx512M -Doracle.mdb.fastUndeploy=60 "/>
<data id="oc4j-options" value="-properties"/>
</category>
<category id="stop-parameters">
<data id="java-options" value="-Djava.security.policy=D:\Oracle\OAS\portal\MT\j2ee\OC4J_BPEL\config\java2.policy-Djava.awt.headless=true"/>
<data id="oc4j-options" value="force "/>
</category>
</module-data>
<start timeout="900" retry="2"/>
<stop timeout="120"/>
<restart timeout="720" retry="2"/>
<port id="ajp" range="12501-12600"/>
<port id="rmi" range="12401-12500"/>
<port id="jms" range="12601-12700"/>
<process-set id="default_island" numprocs="1"/>
</process-type>


Look for the entry
-server -Xms256M -Xmx512M
and replace it with
-server -Xmn1228m -XX:+AggressiveHeap -Xms2048m -Xmx2048m

3.2. After the modifications, your opmn.xml file should look like this

New opmn.xml file (just the OC4J_BPEL section):

<process-type id="OC4J_BPEL" module-id="OC4J">
<module-data>

<category id="start-parameters">
<data id="java-options" value="-server
-Xrs -Ddatasource.verbose=true
-Djava.security.policy=D:\Oracle\OAS\portal\MT\j2ee\OC4J_BPEL\config\java2.policy

-Djava.awt.headless=true
-Xbootclasspath^/p:D:\Oracle\OAS\portal\MT\integration\orabpel/lib/orabpel-boot.jar;D:\Oracle\OAS\po

rtal\MT\integration\orabpel/lib/connector15.jar -Doc4j.userThreads=true
-Dorabpel.home=D:\Oracle\OAS\portal\MT\integration\orabpel
-Dhttp.proxySet=true
-Dhttp.proxyHost=www-proxy.us.oracle.com -Dhttp.proxyPort=80
-Dhttp.nonProxyHosts=aalvarad-pc.us.oracle.com|localhost|*.us.oracle.com|aalvarad-pc

-Djava.net.preferIPv4Stack=true -server -Xmn1228m -XX:+AggressiveHeap -Xms2048m -Xmx2048m

-Doracle.mdb.fastUndeploy=60 "/>
<data id="oc4j-options" value="-properties"/>

</category>
<category id="stop-parameters">
<data id="java-options"

value="-Djava.security.policy=D:\Oracle\OAS\portal\MT\j2ee\OC4J_BPEL\config\java2.policy

-Djava.awt.headless=true"/>
<data id="oc4j-options" value="force "/>

</category>
</module-data>
<start timeout="900" retry="2"/>

<stop timeout="120"/>
<restart timeout="720" retry="2"/>

<port id="ajp" range="12501-12600"/>
<port id="rmi" range="12401-12500"/>

<port id="jms" range="12601-12700"/>
<process-set id="default_island" numprocs="1"/>
</process-type>



4. Apply the following patches:

Bug 5638122 MERGE LABEL REQUEST ON TOP OF 10.1.2.0.2 BUGS 5149866 5595347 5205630
Bug 5499891 CHANGES MADE FOR BETTER MEMORY USE TO AVOID OUTOFMEMORY

4) Restart your application server.

References
Patch 5499891 - CHANGES MADE FOR BETTER MEMORY USE TO AVOID OUTOFMEMORY
Patch 5638122 - MERGE LABEL REQUEST ON TOP OF 10.1.2.0.2 FOR BUGS 5149866 5595347 5205630

Comments

Thulasiram said…
This post helped us to increase the heap size in our project.
September 1, 2014 at 3:54 AM
Post a Comment

Popular posts from this blog

How to mount a WD Book Live as NFS in OEL6U3

-
Scenario : I have a Windows 8 (Host) laptop on which I am running two OEL6U3 (Guest) machines in a Virtual Box. I want to have a shared mount between the two Linux machine which will allow me to read/write from both the machines concurrently. Its a very common scenario for a High Availability Environment. Virtual Box provides a facility of creating sharable VDI disks which can be mounted on multiple machines, but it doesn’t allow concurrent read/write access from multiple machines. If you are installing RAC database then its still achievable, because ASM will provide the feature of concurrent read/writes, but if you want to install a WebLogic cluster with High Availability the recommended approach would be to have a Shared Mount. Step to create a shared nfs mount is fairly simple. I am using a WD My Book Live, and will mount the Public folder as shared mount 1. Get the hostname/ip of the WD device. 2. On the Linux Machine 1 create the folder where you want to mount the nfs
Read more

ORA-44412: XE edition memory parameter invalid or not specified

-
SQL> connect sys as sysdba Enter password: Connected to an idle instance. SQL> startup ORA-44412: XE edition memory parameter invalid or not specified SQL> startup pfile=C:\oraclexe\app\oracle\product\10.2.0\server\config\scripts\init.ora ORACLE instance started. Total System Global Area 805306368 bytes Fixed Size 1289996 bytes Variable Size 209715444 bytes Database Buffers 591396864 bytes Redo Buffers 2904064 bytes Database mounted. Database opened. SQL> create spfile from pfile='C:\oraclexe\app\oracle\product\10.2.0\server\config\scripts\init.ora'; File created. SQL> shutdown Database closed. Database dismounted. ORACLE instance shut down. SQL> startup ORACLE instance started. Total System Global Area 805306368 bytes Fixed Size 1289996 bytes Variable Size 209715444 bytes Database Buffers 591396864 byt
Read more

Oracle SQL Developer 19.4 font too small

-
Image
I downloaded the latest version of Oracle SQL developer on my windows laptop and when I started it, the font on GUI was too small. To fix the font size scroll to the bottom of the  'Welcome Page' and uncheck 'show on startup'. Restart the SQL Developer and issue will be fixed.
Read more