Avoiding JVM delays caused by random number generation

The library used for random number generation in Oracle’s JVM relies on /dev/random by default for UNIX platforms. This can potentially block the WebLogic Server process because on some operating systems /dev/random waits for a certain amount of “noise” to be generated on the host machine before returning a result.

Although /dev/random is more secure, it’s recommended to use /dev/urandom if the default JVM configuration delays WebLogic Server startup. To determine if your operating system exhibits this behaviour, try displaying a portion of the file from a shell prompt: head -n 1 /dev/random

If the command returns immediately, you can use /dev/random as the default generator for JVM. If the command does not return immediately, use these steps to configure the JVM to use /dev/urandom:

  1. Open the $JAVA_HOME/jre/lib/security/java.security file in a text editor.
  2. Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
  3. Save your change and exit the text editor.

And because there’s a bug in JDK when you use /dev/urandom you have to set it up as /dev/./urandom

You can also set up system property “java.security.egd” which will override the securerandom.source setting.
-Djava.security.egd=file:/dev/./urandom

Disabling Derby in Oracle WebLogic 12c

Oracle WebLogic has some interesting traits to help developers frustrate. From Weblogic 10.3.4 and above the Apache Derby Database is included in the installation. That’s fine but from 12.1.2 release it also starts automatically which is usually unwanted, useless and waste of resources. Previous versions of WebLogic didn’t automatically start the Derby database.

Fortunately you can disable it as basically there is a simple IF statement in the “$WL_DOMAIN_HOME$\bin\setDomainEnv.cmd” file:

@REM Set DERBY_FLAG, if derby is available.
 
if exist %WL_HOME%\common\derby\lib\derby.jar (
    set DERBY_FLAG=true
)

If you want to prevent Derby form starting you have three options:

  • Rename “derby.jar” to something else
  • Delete the IF statement from start-up script
  • Set the DERBY_FLAG to false in the startWeblogic.cmd script

I couldn’t find Oracle’s documentation about Derby in Weblogic but those four options seems to work. I prefer the third option which is quite easy to configure. (via Oracle Community)

In my “$WL_DOMAIN_HOME$\bin\startWebLogic.cmd” I added

...
@REM Call setDomainEnv here.
 
@REM Disabling Derby
set DERBY_FLAG = false
...

Authentication with LDAP provider in WebLogic gets stuck

Lately we upgraded our Java EE applications to new platform and began seeing stuck threads and slow starting times. The platform was upgraded from OC4J to WebLogic 12c and also the underlying LDAP service was changed to Oracle Access Manager. Looking at the server logs the one possible reason for stuck threads was quite clear: LDAP requests.

Fortunately the stuck threads problem with LDAP was a known problem with Oracle Weblogic Server 10.3.2 and later and covered in Oracle Support doc 1436044.1. The LDAP provider fails to authenticate for some users and the server logs show Stuck Threads in LDAP requests:

<10.9.2014 11.43.46 EEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "607" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 607304 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
        java.lang.Object.wait(Native Method)
        java.lang.Object.wait(Object.java:503)
        netscape.ldap.LDAPMessageQueue.waitForMessage(Unknown Source)
        netscape.ldap.LDAPMessageQueue.waitFirstMessage(Unknown Source)
        netscape.ldap.LDAPConnection.sendRequest(Unknown Source)
        netscape.ldap.LDAPConnection.search(Unknown Source)
        weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3771)
        weblogic.security.providers.authentication.LDAPAtnDelegate.userExists(LDAPAtnDelegate.java:2384)
        weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:199)
        com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        java.security.AccessController.doPrivileged(Native Method)
        com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:606)
        javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        ...

The cause for this is that authentication requests are hanging whenever the LDAP server is slow. By default, connections and searches to the LDAP server do not time out, so if the LDAP server is slow, authentication requests may take a very long time to retry. This can be seen as many threads stuck doing LDAP searches.

The solution is to set a timeout on LDAP requests for example as below (described in Oracle Support doc 1436044.1):

  1. Log in to the WLS Administration Console.
  2. Navigate to Security Realms -> myrealm -> Providers -> “your_ldap_authenticator.”
  3. Select the following values:
    • Connect Timeout 30
    • Results Time Limit 5000
    • Uncheck “Keep Alive Enabled”
  4. Save and apply changes. Restart the required servers if prompted.

NOTE: The optimal values may differ from environment to environment. But we can try the values specified here as starting places, and they will help in most cases like this. Our original values for the LDAP authenticator settings were: Connect Timeout: 0; Results Time Limit: 0; Keep Alive Enabled unchecked.

This is still a partial solution as you should investigate why the LDAP is slow. For now this solves our problem but has some side effects with user authentication.

Weblogic 12c and Managed Server in incompatible state

Oracle WebLogic application server (WLS) is quite a behemoth and usually works much better than e.g. OC4J which we previously had to use. But sometimes things don’t go as they should and it gets stuck. Some time ago my managed Server was in “FORCE_SHUTTING_DOWN” state and when trying to restart it I got an error message.

“The server myServer is in an incompatible state.

Warning All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.”

Seems that this is a “common” problem with WebLogic Server and happens at least with 12c version which was also my case. The solution is to delete the “managed_server.state” file under your domain in “servers/managed_server/data/nodemanager/” where managed_server is your managed server’s name.

So, stop everything, delete managed_server.state from your managed server folder and start everything (go to console to start your managed sever as it is SHUTDOWN).

Using the WebLogic 12c Maven Plug-In for Deployment

Using the WebLogic 12c 12.1.2 Maven plug-In for deployment is much easier and quicker than going through the WebLogic Server’s AdminServer and Oracle Documentation provides good examples how to do it. The weblogic-maven-plugin provides enhanced functionality to install, start and stop servers, create domains, execute WLST scripts, and compile and deploy applications.

The weblogic-maven-plugin plug-in is provided as a pre-built JAR file and accompanying pom file. In short, installing and configuring the WebLogic 12c Maven sync plug-in contains following steps:

Open command prompt and run commands:

cd D:\oracle\wls12120\oracle_common\plugins\maven\com\oracle\maven\oracle-maven-sync\12.1.2
mvn install:install-file -DpomFile=oracle-maven-sync.12.1.2.pom -Dfile=oracle-maven-sync.12.1.2.jar
mvn com.oracle.maven:oracle-maven-sync:push -Doracle-maven-sync.oracleHome=D:\oracle\wls12120

You can validate whether you have successfully installed the plug-in using the Maven help:describe goal.

mvn help:describe -DgroupId=com.oracle.weblogic -DartifactId=weblogic-maven-plugin -Dversion=12.1.2-0-0

And you’re ready to deploy with Maven. A lot more easier than generating the WebLogic 11g R2 Maven Plug-in where you had to i.a. build the jar by yourself.

The Maven plug-in can be used e.g. from application’s POM file and be bound to some phase of the Maven life cycle. For example it can be bound to “install” phase and every time you run the “mvn install” command, the deployment plug-in is also called. In my opinion better way to use the plug-in is to add it as a profile so you can call it just when you want with command like “mvn clean install -Pdeploy-wls“.

Maven Project pom.xml File

<profile>
	<id>deploy-wls</id>
	<build>
		<plugins> 
			<plugin> 
				<groupId>com.oracle.weblogic</groupId>
				<artifactId>weblogic-maven-plugin</artifactId> 
				<version>12.1.2-0-0</version> 
				<configuration> 
					<adminurl>t3://localhost:7001</adminurl>
					<user>weblogic</user> 
					<password>weblogic123</password> 
					<upload>true</upload> 
					<targets>myServer</targets>
					<action>deploy</action> 
					<remote>false</remote> 
					<verbose>true</verbose> 
<source>${project.build.directory}/${project.build.finalName}.${project.packaging}</source> 
					<name>${project.build.finalName}</name> 
				</configuration> 
				<executions> 
					<execution> 
						<phase>install</phase> 
						<goals> 
							<goal>deploy</goal> 
						</goals> 
					</execution> 
				</executions> 
			</plugin> 
		</plugins> 
	</build>
</profile>

The user credentials in the POM file are provided as clear-text but for more security you can use secure configuration authentication mechanism which stores the user name and password in encrypted form in an external file, and then uses it to supply the user credentials with which to connect to the WebLogic Server domain, along with the key that was used to encrypt the file.

Weblogic Server Auto Restart with Node Manager as Linux service

Sometimes servers need to reboot and then it’s nice to have certain services to start automatically. Oracle Weblogic’s Node Manager is one of them and in order to have Node Manager start automatically it must be configured as a daemon. Unfortunately Oracle doesn’t provide init scripts to run it as a Linux service but it’s pretty simple to create your own startup scripts. Just create a new nodemgr script under /etc/init.d/, add it as a service and you’re done, as Oracle Fusion Middleware -blog writes.

For example on Red Hat Enterprise Linux Server 5.6 with Oracle Weblogic Server 10.3.5 the /etc/init.d/nodemgr looks like this (edit the script to reflect your Weblogic installation path):


#!/bin/sh
#
# nodemgr Oracle Weblogic NodeManager service
#
# chkconfig:   345 85 15
# description: Oracle Weblogic NodeManager service

### BEGIN INIT INFO
# Provides: nodemgr
# Required-Start: $network $local_fs
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Oracle Weblogic NodeManager service.
# Description: Starts and stops Oracle Weblogic NodeManager.
### END INIT INFO

. /etc/rc.d/init.d/functions

# Your WLS home directory (where wlserver_10.3 is)
export MW_HOME="/oracle/product/mw11g"
export JAVA_HOME="/oracle/java/jdk1.6.0_29"
DAEMON_USER="oracle"
PROCESS_STRING="^.*/oracle/product/mw11g/.*weblogic.NodeManager.*"

source $MW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh > /dev/null
export NodeManagerHome="$WL_HOME/common/nodemanager"
NodeManagerLockFile="$NodeManagerHome/nodemanager.log.lck"

PROGRAM="$MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh"
SERVICE_NAME=`/bin/basename $0`
LOCKFILE="/var/lock/subsys/$SERVICE_NAME"

RETVAL=0

start() {
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ ! -z "$OLDPID" ]; then
            echo "$SERVICE_NAME is already running (pid $OLDPID) !"
            exit
        fi

        echo -n $"Starting $SERVICE_NAME: "
        /bin/su $DAEMON_USER -c "$PROGRAM &"

        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKFILE
}

stop() {
        echo -n $"Stopping $SERVICE_NAME: "
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/kill -TERM $OLDPID
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        echo
        /bin/rm -f $NodeManagerLockFile
        [ $RETVAL -eq 0 ] && rm -f $LOCKFILE

}

restart() {
        stop
        sleep 10
        start
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart|force-reload|reload)
        restart
        ;;
  condrestart|try-restart)
        [ -f $LOCKFILE ] && restart
        ;;
  status)
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/echo "$SERVICE_NAME is running (pid: $OLDPID)"
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
        exit 1
esac

exit $RETVAL


Add the Node Manager to start after server reboot:

# chmod +x /etc/init.d/nodemgr
# chkconfig --add nodemgr
# chkconfig --list
nodemgr         0:off   1:off   2:off   3:on    4:on    5:on    6:off

Also now the Node Manager can be controlled via the service command (e.g. service nodemgr restart).

When you have the Node Manager restarting automatically after a system reboot, you can also have Weblogic managed servers automatically restarted by Node Manager. Managed servers will be restarted only if they were running at the time the shutdown was issued. Just activate the Auto Restart option in the Administration Console (Environment > Servers > selected server > Health Monitoring) and you might also need to set the CrashRecoveryEnabled to “true” in $WL_HOME/wlserver_10.3/common/nodemanager/nodemanager.properties.

With little scripting and configuration your sysadmin tasks have now become a little easier.

Using the WebLogic Maven Plug-In for Deployment

Using the WebLogic Maven plug-In for deployment is much easier and quicker than going through the WebLogic Server’s AdminServer and Oracle Documentation provides good examples how to do it.

In short, generating WebLogic Maven Plug-In contains following steps:

1. Build the plug-in JAR file using the WebLogic JarBuilder Tool (wljarbuilder) under MW_HOME/wlserver_10.3/server/lib/ with the following command:

java -jar wljarbuilder.jar -profile weblogic-maven-plugin

2. Extract the pom.xml file from weblogic-maven-plugin.jar under the MW_HOME/wlserver_10.3/server/lib directory, and then copy the pom.xml file to MW_HOME/wlserver_10.3/server/lib.

jar xvf MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml
 
in Linux:	
cp META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib
 
or in Windows:
copy META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib

3. Provision the weblogic-maven-plugin.jar in your local Maven repository with the following command.

mvn install:install-file -Dfile=MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar -DpomFile=pom.xml

4. Done.

The Maven plug-in can be used e.g. from application’s POM file and be bound to some phase of the Maven life cycle. For example it can be bound to “install” phase and every time you run the “mvn install” command, the deployment plug-in is also called. In my opinion better way to use the plug-in is to add it as a profile so you can call it just when you want with command like “mvn clean install -Pdeploy-wls”.

Maven Project pom.xml File

<profile>
	<id>deploy-wls</id>
	<build>
		<plugins> 
			<plugin> 
				<groupId>com.oracle.weblogic</groupId>
				<artifactId>weblogic-maven-plugin</artifactId> 
				<version>10.3.6.0</version> 
				<configuration> 
					<adminurl>t3://localhost:7001</adminurl>
					<user>weblogic</user> 
					<password>weblogic123</password> 
					<upload>true</upload> 
					<targets>myServer</targets>
					<action>deploy</action> 
					<remote>false</remote> 
					<verbose>true</verbose> 
					<source>${project.build.directory}/${project.build.finalName}.${project.packaging}</source> 
					<name>${project.build.finalName}</name> 
				</configuration> 
				<executions> 
					<execution> 
						<phase>install</phase> 
						<goals> 
							<goal>deploy</goal> 
						</goals> 
					</execution> 
				</executions> 
			</plugin> 
		</plugins> 
	</build>
</profile>

The user credentials in the POM file are provided as clear-text but for more security you can use secure configuration authentication mechanism which stores the user name and password in encrypted form in an external file, and then uses it to supply the user credentials with which to connect to the WebLogic Server domain, along with the key that was used to encrypt the file.