Avoiding JVM delays caused by random number generation

The library used for random number generation in Oracle’s JVM relies on /dev/random by default for UNIX platforms. This can potentially block the WebLogic Server process because on some operating systems /dev/random waits for a certain amount of “noise” to be generated on the host machine before returning a result.

Although /dev/random is more secure, it’s recommended to use /dev/urandom if the default JVM configuration delays WebLogic Server startup. To determine if your operating system exhibits this behaviour, try displaying a portion of the file from a shell prompt: head -n 1 /dev/random

If the command returns immediately, you can use /dev/random as the default generator for JVM. If the command does not return immediately, use these steps to configure the JVM to use /dev/urandom:

  1. Open the $JAVA_HOME/jre/lib/security/java.security file in a text editor.
  2. Change the line “securerandom.source=file:/dev/random” to read: securerandom.source=file:/dev/./urandom
  3. Save your change and exit the text editor.

And because there’s a bug in JDK when you use /dev/urandom you have to set it up as /dev/./urandom

You can also set up system property “java.security.egd” which will override the securerandom.source setting.
-Djava.security.egd=file:/dev/./urandom

Problems with installing Oracle DB 12c EE, ORA-12547: TNS: lost contact

For development purposes I wanted to install Oracle Database 12c Enterprise Edition to Vagrant box so that I could play with it. It should’ve gone quite straight forwardly but in my case things got complicated although I had Oracle Linux and the pre-requirements fulfilled. Everything went fine until it was time to run the DBCA and create the database.

The DBCA gave “ORA-12547: TNS: lost contact” error which is quite common. Google gave me couple of resources to debug the issue. Oracle DBA Blog explained common issues which cause ORA-12547 and solutions to fix it.

One of the suggested solutions was to check to ensure that the following two files are not 0 bytes:

ls -lt $ORACLE_HOME/bin/oracle
ls -lt $ORACLE_HOME/rdbms/lib/config.o

And true, my oracle binary was 0 bytes

-rwsr-s--x 1 oracle oinstall 0 Jul  7  2014 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

To fix the binary you need to relink it and to do that rename the following file:

$ cd $ORACLE_HOME/rdbms/lib
$ mv config.o config.o.bad

Then, shutdown the database and listener and then “relink all”

$ relink all

If just things were that easy. Unfortunately relinking ended on error:

[oracle@oradb12c lib]$ relink all
/u01/app/oracle/product/12.1.0/dbhome_1/bin/relink: line 168: 13794 Segmentation fault      $ORACLE_HOME/perl/bin/perl $ORACLE_HOME/install/modmakedeps.pl $ORACLE_HOME $ORACLE_HOME/inventory/make/makeorder.xml > $CURR_MAKEORDER
writing relink log to: /u01/app/oracle/product/12.1.0/dbhome_1/install/relink.log

After googling some more I found similar problem and solution: Relink the executables by running make install.

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk install
 
cd $ORACLE_HOME/network/lib
make -f ins_net_server.mk install
</re>
 
If needed you can also relink other executables:
<pre lang="shell">
make -kf ins_sqlplus.mk install (in $ORACLE_HOME/sqlplus/lib)
make -kf ins_reports60w.mk install (on CCMgr server)
make -kf ins_forms60w.install (on Forms/Web server)

But of course it didn’t work out of the box and failed to error:

/bin/ld: cannot find -ljavavm12
collect2: error: ld returned 1 exit status
make: *** [/u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle] Error 1

The solution is to copy the libjavavm12.a to under $ORACLE_HOME lib as explained:

cp $ORACLE_HOME/javavm/jdk/jdk6/lib/libjavavm12.a $ORACLE_HOME/lib/

Run the make install commands from above again and you should’ve working oracle binary:

-rwsr-s--x 1 oracle oinstall 323649826 Feb 17 16:27 /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle

After this I ran the relink again which worked and also the install of the database worked fine.

cd $ORACLE_HOME/bin
relink all

Start the listener:

lsnrctl start LISTENER

Create the database:

dbca -silent -responseFile $ORACLE_BASE/installation/dbca.rsp

The problems I encountered while installing Oracle Database 12c Enterprise Edition to Oracle Linux 7 although in Vagrant and with Ansible were surprising as you would think that on certified platform it should just work. If I would’ve been using CentOS or Ubuntu it would’ve been totally different issue.

You can see the Ansible tasks I did to get Oracle DB 12c EE installed on Oracle Linux 7 in my vagrant-experiments GitHub repo.

Oracle DB 12c EE Ansible Tasks
Oracle DB 12c EE Ansible Tasks

Using Let’s Encrypt SSL certificates on Centos 6

Let's Encrypt all the things

Let’s Encrypt is now in public beta, meaning, you can get valid, trusted SSL certificates for your domains for free. Free SSL certificates for everyone! As Let’s Encrypt is relatively easy to setup, there’s now no reason not to use HTTPS for your sites. The needed steps are described in the documentation and here’s short guide how to setup Let’s Encrypt in CentOS 6.x and automate the SSL certificate renewal.

Let’s Encrypt installation

The Let’s Encrypt Client is a fully-featured, extensible client for the Let’s Encrypt CA that can automate the tasks of obtaining certificates and configuring web servers to use them. The installation is simple but in my case on CentOS 6.x I first needed to update to Python 2.7 as Let’s Encrypt supports Python 2.7+ only.

Installing Python 2.7 in Centos 6.x

# Install Epel Repository
yum install epel-release
 
# Install IUS Repository
rpm -ivh https://rhel6.iuscommunity.org/ius-release.rpm
 
# Install Python 2.7 and Git
yum --enablerepo=ius install python27 python27-devel python27-pip python27-setuptools python27-virtualenv -y

Setting up Lets encrypt

Install Git if you don’t have it yet.

yum install git

If letsencrypt is packaged for your operating system, you can install it from there, and the other solution is to use the letsencrypt-auto wrapper script, which obtains some dependencies from your operating system and puts others in a python virtual environment:

# Get letsencrypt
git clone https://github.com/letsencrypt/letsencrypt
 
# See help
./letsencrypt/letsencrypt-auto --help

Running the client

You can either just run letsencrypt-auto or letsencrypt, and the client will guide you through the process of obtaining and installing certs interactively or you you can tell it exactly what you want it to do from the command line.

For example obtain a cert for your domain using the Apache plugin to both obtain and install the certs, you could do this:

./letsencrypt-auto --apache -d thing.com -d www.thing.com -d otherthing.net

(The first time you run the command, it will make an account, and ask for an email and agreement to the Let’s Encrypt Subscriber Agreement; you can automate those with –email and –agree-tos)

Although you can use the Apache plugin to obtain and install the certs it didn’t work for me. I got an error: “The apache plugin is not working; there may be problems with your existing configuration.” This seems to be an issue with Apache 2.2 and until it’s fixed you can use the webroot authentication method as explained in documentation.

./letsencrypt-auto certonly --webroot -w /var/www/example/ -d example.com

The webroot plugin works by creating a temporary file for each of your requested domains in ${webroot-path}/.well-known/acme-challenge. Then the Let’s Encrypt validation server makes HTTP requests to validate that the DNS for each requested domain resolves to the server running letsencrypt. Note that to use the webroot plugin, your server must be configured to serve files from hidden directories.

Now your certificate and chain have been saved at Let’s Encrypt configuration directory at “/etc/letsencrypt” and “/etc/letsencrypt/live/ contains symlinks to the latest certificates. Making regular backups of this folder is ideal.

All we have to do now is set it up in Apache.

Configure Apache to use Let’s Encrypt certs

In Let’s Encrypt configuration directory at “/etc/letsencrypt/live/ the .pem files are as follows (from the Letsencrypt documentation):

  • privkey.pem: Private key for the certificate.
    • This must be kept secret at all times! Never share it with anyone, including Let’s Encrypt developers. You cannot put it into a safe, however – your server still needs to access this file in order for SSL/TLS to work.
    • This is what Apache needs for SSLCertificateKeyFile
  • cert.pem: Server certificate only.
    • This is what Apache needs for SSLCertificateFile.
  • chain.pem: All certificates that need to be served by the browser excluding server certificate, i.e. root and intermediate certificates only.
    • This is what Apache needs for SSLCertificateChainFile.
  • fullchain.pem: All certificates, including server certificate. This is concatenation of chain.pem and cert.pem.

Now that we know which file is which we can configure our VirtualHost to use SSL with our new certs. Change the following lines in your Apache’s virtualhost’s SSL configuration:

...
SSLCertificateFile /etc/letsencrypt/live/<your-domain>/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/<your-domain>/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/<your-domain>/chain.pem
...

Finally, restart apache

You can test that your SSL is working with SSL Labs.

Automate updating Let’s Encrypt certs

As you surely noticed Let’s Encrypt CA issues short lived certificates (90 days) and you have to renew the certificates at least once in 3 months. Nice way to force sysadmins to automate the process.

To obtain a new version of the certificate you can simply run Let’s Encrypt again but doing that manually is not feasible. Let’s Encrypt is working hard on automating the renewal process but until that we have to do it by ourselves.

Fortunately we don’t need to invent our own scripts as there’s excellent article about automating Let’s Encrypt and script for crontab.

Get the autole.sh -script from GitHub that automates tasks like:

  • Check the expire date of the certificate and renew when the remaining days are below a value
  • Check that the directory for the challenge is well mapped
  • Alert the admin if it’s not possible to renew the certificate

Now you can renew certain domain’s certificates with

./autole.sh www.mydomain.com

And to renew all your certificates use

./autole.sh --renew-all

Now you can add this to the crontab, run weekly, and your certificates will be ready and renew automatically. This cron job will execute the command every Monday at 08:30.

30 8 * * 1 /usr/local/sbin/autole.sh <your-domain> >> /var/log/autole.log

Now before I switch my WordPress over to HTTPS I have to do some find & replace in the database and fix the URL’s of the images to be protocol relative.

Creating Vagrant Base Box with Veewee

Vagrant is a great tool for creating and configuring lightweight, reproducible, portable virtual machine environments but the first step for using Vagrant, downloading an existing “base box”, raises some questions. E.g. How are these unverified boxes built? So, you might end up building your own base box which is often time consuming and cumbersome. Fortunately there’s a tool called Veewee which aims to automate all the steps for building base boxes and to collect best practices in a transparent way.

Vagrant Base Box with Veewee

Veewee is a tool for easily (and repeatedly) building custom Vagrant base boxes, KVMs, and virtual machine images. You can use it to build a Vagrant box in Linux, Mac OS X and Windows but I found out that fulfilling the requirements on Windows is quite difficult (read Ruby and RVM) so just forget it.

To get you started there are some requirements you need to fulfill. First you’ll need to install at least one of the supported virtual machine providers like VirtualBox and second you need some development libraries.

On Ubuntu 15.04 Linux and using VirtualBox you need these packages:

$ apt-get install virtualbox git curl ruby ruby-dev libxslt1-dev libxml2-dev zlib1g-dev

Install RVM on Linux

For Ruby environment it’s recommended to use either rvm or rbenv. I chose the RVM and followed the RVM install documentation.

Install mpapis public key:

$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

if keyserver fails, you can use $ curl -sSL https://rvm.io/mpapis.asc | gpg --import -

Install RVM stable with ruby:

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

Installing Veewee with RVM

With RVM already installed, ensure a ruby version that’s supported by Veewee is available on your machine:

$ source /home/marko/.rvm/scripts/rvm
$ rvm install ruby

Clone the veewee project from source:

$ cd <path_to_workspace>
$ git clone https://github.com/jedi4ever/veewee.git
$ cd veewee

Set the local gemset and ruby version within the current directory:

$ rvm use ruby@veewee --create

Run bundle install to install Gemfile dependencies for your local gemset:

$ gem install bundler
$ bundle install

Bundle install will take some time.

Building Vagrant Box with Veewee

Veewee uses definitions to build new virtual machines and ‘definition’ is derived from a ‘template’ and preconfigured templates are found in templates/ folder. Veewee Basics explains how you can create your own customized definition.

For my customized Vagrant Box I decided to use Tommy Muehle’s definition as a template as it contained what I wanted. Simple CentOS 6.6. Box with Puppet. I just changed the localization to Finland and made it bigger for WebLogic use case in mind. My definition for Vagrant Box can be found in GitHub.

To use my definition just clone the repository for CentOS 6.6 Box, copy the “centos-6.6-x86_64_puppet” folder to definitions/ folder under Veewee and make your own changes if needed. After you’re done run:

$ bundle exec veewee vbox build centos-6.6-x86_64_puppet

The build command runs Veewee scripts and automates the manual steps needed while installing a new Linux distribution.

Installing CentOS to Vagrant Box with Veewee

To export the Box for further use with Vagrant, run:

$ bundle exec veewee vbox export centos-6.6-x86_64_puppet

The above command is actually calling “vagrant package –base ‘centos-6.6-x86_64_puppet’ –output ‘boxes/centos-6.6-x86_64_puppet'”. The machine gets shut down, exported and will be packed in a centos-6.6-x86_64_puppet.box file inside the current directory.

And you’re all done. Now you can use your just created base box for Vagrant boxes. Import it into Vagrant’s box repository and use it to initialize a fresh project:

$ vagrant box add 'centos-6.6-x86_64_puppet' 'centos-6.6-x86_64_puppet.box'
$ vagrant init 'centos-6.6-x86_64_puppet'

Using Veewee to build a Vagrant Box is simple and what’s more important it’s automated and reproducible. Using Ruby and RVM on Windows 7 turned out to be practically impossible but old ThinkPad W510 with Ubuntu 15.04 worked nicely. Of course you could create a base box with Vagrant way which means installing and configuring your Linux manually. But why would you want to that if you can just automate it?

Disabling Derby in Oracle WebLogic 12c

Oracle WebLogic has some interesting traits to help developers frustrate. From Weblogic 10.3.4 and above the Apache Derby Database is included in the installation. That’s fine but from 12.1.2 release it also starts automatically which is usually unwanted, useless and waste of resources. Previous versions of WebLogic didn’t automatically start the Derby database.

Fortunately you can disable it as basically there is a simple IF statement in the “$WL_DOMAIN_HOME$\bin\setDomainEnv.cmd” file:

@REM Set DERBY_FLAG, if derby is available.
 
if exist %WL_HOME%\common\derby\lib\derby.jar (
    set DERBY_FLAG=true
)

If you want to prevent Derby form starting you have three options:

  • Rename “derby.jar” to something else
  • Delete the IF statement from start-up script
  • Set the DERBY_FLAG to false in the startWeblogic.cmd script

I couldn’t find Oracle’s documentation about Derby in Weblogic but those four options seems to work. I prefer the third option which is quite easy to configure. (via Oracle Community)

In my “$WL_DOMAIN_HOME$\bin\startWebLogic.cmd” I added

...
@REM Call setDomainEnv here.
 
@REM Disabling Derby
set DERBY_FLAG = false
...

Authentication with LDAP provider in WebLogic gets stuck

Lately we upgraded our Java EE applications to new platform and began seeing stuck threads and slow starting times. The platform was upgraded from OC4J to WebLogic 12c and also the underlying LDAP service was changed to Oracle Access Manager. Looking at the server logs the one possible reason for stuck threads was quite clear: LDAP requests.

Fortunately the stuck threads problem with LDAP was a known problem with Oracle Weblogic Server 10.3.2 and later and covered in Oracle Support doc 1436044.1. The LDAP provider fails to authenticate for some users and the server logs show Stuck Threads in LDAP requests:

<10.9.2014 11.43.46 EEST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "607" seconds working on the request "Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 607304 ms
", which is more than the configured time (StuckThreadMaxTime) of "600" seconds in "server-failure-trigger". Stack trace:
        java.lang.Object.wait(Native Method)
        java.lang.Object.wait(Object.java:503)
        netscape.ldap.LDAPMessageQueue.waitForMessage(Unknown Source)
        netscape.ldap.LDAPMessageQueue.waitFirstMessage(Unknown Source)
        netscape.ldap.LDAPConnection.sendRequest(Unknown Source)
        netscape.ldap.LDAPConnection.search(Unknown Source)
        weblogic.security.providers.authentication.LDAPAtnDelegate.getDNForUser(LDAPAtnDelegate.java:3771)
        weblogic.security.providers.authentication.LDAPAtnDelegate.userExists(LDAPAtnDelegate.java:2384)
        weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:199)
        com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
        java.security.AccessController.doPrivileged(Native Method)
        com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
        sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        java.lang.reflect.Method.invoke(Method.java:606)
        javax.security.auth.login.LoginContext.invoke(LoginContext.java:762)
        javax.security.auth.login.LoginContext.access$000(LoginContext.java:203)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:690)
        javax.security.auth.login.LoginContext$4.run(LoginContext.java:688)
        java.security.AccessController.doPrivileged(Native Method)
        javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:687)
        javax.security.auth.login.LoginContext.login(LoginContext.java:595)
        ...

The cause for this is that authentication requests are hanging whenever the LDAP server is slow. By default, connections and searches to the LDAP server do not time out, so if the LDAP server is slow, authentication requests may take a very long time to retry. This can be seen as many threads stuck doing LDAP searches.

The solution is to set a timeout on LDAP requests for example as below (described in Oracle Support doc 1436044.1):

  1. Log in to the WLS Administration Console.
  2. Navigate to Security Realms -> myrealm -> Providers -> “your_ldap_authenticator.”
  3. Select the following values:
    • Connect Timeout 30
    • Results Time Limit 5000
    • Uncheck “Keep Alive Enabled”
  4. Save and apply changes. Restart the required servers if prompted.

NOTE: The optimal values may differ from environment to environment. But we can try the values specified here as starting places, and they will help in most cases like this. Our original values for the LDAP authenticator settings were: Connect Timeout: 0; Results Time Limit: 0; Keep Alive Enabled unchecked.

This is still a partial solution as you should investigate why the LDAP is slow. For now this solves our problem but has some side effects with user authentication.

Weblogic 12c and Managed Server in incompatible state

Oracle WebLogic application server (WLS) is quite a behemoth and usually works much better than e.g. OC4J which we previously had to use. But sometimes things don’t go as they should and it gets stuck. Some time ago my managed Server was in “FORCE_SHUTTING_DOWN” state and when trying to restart it I got an error message.

“The server myServer is in an incompatible state.

Warning All of the servers selected are currently in a state which is incompatible with this operation or are not associated with a running Node Manager or you are not authorized to perform the action requested. No action will be performed.”

Seems that this is a “common” problem with WebLogic Server and happens at least with 12c version which was also my case. The solution is to delete the “managed_server.state” file under your domain in “servers/managed_server/data/nodemanager/” where managed_server is your managed server’s name.

So, stop everything, delete managed_server.state from your managed server folder and start everything (go to console to start your managed sever as it is SHUTDOWN).

Windows and X11 forwarding with Xming

Usually I prefer to do thing directly from the Linux terminal but sometimes there is a need for remote graphical tools and X11 forwarding. Linux supports X Forwarding with no extra software, on OS X you need e.g. XQuartz and on Windows you need two pieces of software: a secure shell program (ssh) to establish the remote connection and an X Server to handle the local display. Here we concentrate on Windows and use Xming and PuTTY for those tasks.

Using Xming for X11 forwarding on Windows

There are different tools to do X11 forwarding on Windows but Xming is small, fast and simple software for X Window System Server for Microsoft Windows. You also need program like PuTTY or FuTTY to create the remote connection.

After you have Xming installed and started check what the icon says on the taskbar. Mine says “Xming Server:0.0” so for PuTTY settings we get “0.0”. It might also be “0:0”.

Next open Putty and in the Category list go to

  • Connection > SSH > X11
  • Check X11 forwarding: [x] Enable X11 forwarding
  • Add to the “X display location” field: localhost:0.0

Choose Session in the Category list and add your Linux hostname and select SSH. Click Save and then Open the SSH connection to your remote host.

On the remote machine check that the X11 forwarding works with e.g. -bash-3.2$ xeyes &

X11 forwarding as other user

That’s the typical use case but if you change to another user and try the same command you get error “MIT-MAGIC-COOKIE-1 data did not match”. Go back to previous user and check what the xauth list says. Take note of that line as you need it later.

-bash-3.2$ xauth list
your.linuxmachine.foo/unix:10  MIT-MAGIC-COOKIE-1  38dcf9f9fd5b587f3d5fa8d77281f66c

Login again as another user e.g. -bash-3.2$ sudo su - oracle and add the original users MIT-MAGIC-COOKIE to xauth.

-bash-3.2$ xauth add your.linuxmachine.foo/unix:10  MIT-MAGIC-COOKIE-1  38dcf9f9fd5b587f3d5fa8d77281f66c

After that the command -bash-3.2$ xeyes & as oracle should work.

If you get error, -bash-3.2$ Error: Can't open display:, correct the DISPLAY variable like:

-bash-3.2$ DISPLAY=localhost:10.0; export DISPLAY

Now you can run e.g. Oracle’s Database Configuration Assistant (DBCA) to setup a new database remotely with X11 forwarding:

Runnign DBCA remotely with X11 forwarding

Using PHP-FPM with Apache 2 on CentOS

Running Apache 2 and PHP is simple with mod_php but there are more efficient alternatives like using PHP-FPM (FastCGI Process Manager) which is an alternative PHP FastCGI implementation. With it the PHP process runs standalone without the need for a web server and listens for incoming requests on either a TCP or a Unix socket. Web servers can connect the PHP process and send requests using the FastCGI protocol. It solves mod_php’s problem of spinning up and destroying PHP instances with every request and thus is more memory efficient and provides better performance.

These instructions are for CentOS 6.4 but the process should however work similarly with other Linux distributions.

Setting up the PHP-FPM

Install the FPM-CGI binary for PHP and add it to start after server reboot:

# yum install php-fpm
# chkconfig --levels 235 php-fpm on

Configure the PHP-FPM pool in /etc/php-fpm.d/www.conf to use sockets and enable some status information for e.g. Munit:

;listen = 127.0.0.1:9000
listen = /tmp/php5-fpm.sock
pm.status_path = /status
ping.path = /ping

Start the service with:

service php-fpm start

Setting up Apache and mod_fastcgi

Apache can be configured to run FastCGI with two modules: mod_fastcgi and mod_fcgid. The difference is explained at Debian bug report #504132: “mod_fcgid passes just one request to the FCGI server at a time while mod_fastcgi passes several requests at once, the latter is usually better for PHP, as PHP can manage several request using several threads and opcode caches like APC usually work only with threads and not with processes. This means that using mod_fcgid you end up having many PHP processes which all have their very own opcode cache.”

In short: mod_fastcgi is better.

Install mod_fastcgi

So we need to get mod_fastcgi which isn’t at the time found from CentOS default or EPEL repos but from RPMForge or by building it from sources.

Getting mod_fastcgi from RPMForge

Install the RPMForge repo:

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Add some priorities which repo to use:

# yum install yum-priorities
 
# vi /etc/yum.repos.d/epel.repo 
... add the line priority=10 to the [epel] section

Install mod_fastcgi

# yum install mod_fastcgi

Or building mod_fastcgi from sources

You can build the mod_fastcgi from sources. Make sure required packages are installed (httpd-devel and apr-devel required to compile mod_fastcgi):

# yum install libtool httpd-devel apr-devel apr

Get the latest mod_fastcgi source code:

# cd /opt
# wget http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz

Untar tar ball:

# tar -zxvf mod_fastcgi-current.tar.gz
# cd mod_fastcgi-2.4.6/

As we are using Apache 2, we make a copy of Makefile.AP2: cp Makefile.AP2 Makefile

Compile and install mod_fastcgi for 64 bit system:

# make top_dir=/usr/lib64/httpd
# make install top_dir=/usr/lib64/httpd

Configure mod_fastcgi

If you have php enabled disable it

# mv /etc/httpd/conf.d/{php.conf,php.conf.disable}

Set up a (non-existent) directory that Apache can route the requests through. That directory must be available to Apache and it might be /usr/lib/cgi-bin/ so the routed file is then e.g. /usr/lib/cgi-bin/php5-fcgi.

# mkdir /usr/lib/cgi-bin/

Configure mod_fastcgi settings in /etc/httpd/conf.d/mod_fastcgi.conf to be:

LoadModule fastcgi_module modules/mod_fastcgi.so
 
<IfModule mod_fastcgi.c>
	DirectoryIndex index.php index.html index.shtml index.cgi
	AddHandler php5-fcgi .php
	Action php5-fcgi /php5-fcgi
	Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
	FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization
 
	# For monitoring status with e.g. Munin
	<LocationMatch "/(ping|status)">
		SetHandler php5-fcgi-virt
		Action php5-fcgi-virt /php5-fcgi virtual
	</LocationMatch>
</IfModule>

We add handler and action which sends all requests of PHP to the virtual URL created above, which is in turn then sent to the external FastCGI server. We also add configuration to have some status information about our PHP-FPM.

Start Apache:

# service httpd start

PHP should now work.

Weblogic Server Auto Restart with Node Manager as Linux service

Sometimes servers need to reboot and then it’s nice to have certain services to start automatically. Oracle Weblogic’s Node Manager is one of them and in order to have Node Manager start automatically it must be configured as a daemon. Unfortunately Oracle doesn’t provide init scripts to run it as a Linux service but it’s pretty simple to create your own startup scripts. Just create a new nodemgr script under /etc/init.d/, add it as a service and you’re done, as Oracle Fusion Middleware -blog writes.

For example on Red Hat Enterprise Linux Server 5.6 with Oracle Weblogic Server 10.3.5 the /etc/init.d/nodemgr looks like this (edit the script to reflect your Weblogic installation path):


#!/bin/sh
#
# nodemgr Oracle Weblogic NodeManager service
#
# chkconfig:   345 85 15
# description: Oracle Weblogic NodeManager service

### BEGIN INIT INFO
# Provides: nodemgr
# Required-Start: $network $local_fs
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Oracle Weblogic NodeManager service.
# Description: Starts and stops Oracle Weblogic NodeManager.
### END INIT INFO

. /etc/rc.d/init.d/functions

# Your WLS home directory (where wlserver_10.3 is)
export MW_HOME="/oracle/product/mw11g"
export JAVA_HOME="/oracle/java/jdk1.6.0_29"
DAEMON_USER="oracle"
PROCESS_STRING="^.*/oracle/product/mw11g/.*weblogic.NodeManager.*"

source $MW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh > /dev/null
export NodeManagerHome="$WL_HOME/common/nodemanager"
NodeManagerLockFile="$NodeManagerHome/nodemanager.log.lck"

PROGRAM="$MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh"
SERVICE_NAME=`/bin/basename $0`
LOCKFILE="/var/lock/subsys/$SERVICE_NAME"

RETVAL=0

start() {
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ ! -z "$OLDPID" ]; then
            echo "$SERVICE_NAME is already running (pid $OLDPID) !"
            exit
        fi

        echo -n $"Starting $SERVICE_NAME: "
        /bin/su $DAEMON_USER -c "$PROGRAM &"

        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKFILE
}

stop() {
        echo -n $"Stopping $SERVICE_NAME: "
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/kill -TERM $OLDPID
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        echo
        /bin/rm -f $NodeManagerLockFile
        [ $RETVAL -eq 0 ] && rm -f $LOCKFILE

}

restart() {
        stop
        sleep 10
        start
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart|force-reload|reload)
        restart
        ;;
  condrestart|try-restart)
        [ -f $LOCKFILE ] && restart
        ;;
  status)
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/echo "$SERVICE_NAME is running (pid: $OLDPID)"
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
        exit 1
esac

exit $RETVAL


Add the Node Manager to start after server reboot:

# chmod +x /etc/init.d/nodemgr
# chkconfig --add nodemgr
# chkconfig --list
nodemgr         0:off   1:off   2:off   3:on    4:on    5:on    6:off

Also now the Node Manager can be controlled via the service command (e.g. service nodemgr restart).

When you have the Node Manager restarting automatically after a system reboot, you can also have Weblogic managed servers automatically restarted by Node Manager. Managed servers will be restarted only if they were running at the time the shutdown was issued. Just activate the Auto Restart option in the Administration Console (Environment > Servers > selected server > Health Monitoring) and you might also need to set the CrashRecoveryEnabled to “true” in $WL_HOME/wlserver_10.3/common/nodemanager/nodemanager.properties.

With little scripting and configuration your sysadmin tasks have now become a little easier.