Using PHP-FPM with Apache 2 on CentOS

Running Apache 2 and PHP is simple with mod_php but there are more efficient alternatives like using PHP-FPM (FastCGI Process Manager) which is an alternative PHP FastCGI implementation. With it the PHP process runs standalone without the need for a web server and listens for incoming requests on either a TCP or a Unix socket. Web servers can connect the PHP process and send requests using the FastCGI protocol. It solves mod_php’s problem of spinning up and destroying PHP instances with every request and thus is more memory efficient and provides better performance.

These instructions are for CentOS 6.4 but the process should however work similarly with other Linux distributions.

Setting up the PHP-FPM

Install the FPM-CGI binary for PHP and add it to start after server reboot:

# yum install php-fpm
# chkconfig --levels 235 php-fpm on

Configure the PHP-FPM pool in /etc/php-fpm.d/www.conf to use sockets and enable some status information for e.g. Munit:

;listen = 127.0.0.1:9000
listen = /tmp/php5-fpm.sock
pm.status_path = /status
ping.path = /ping

Start the service with:

service php-fpm start

Setting up Apache and mod_fastcgi

Apache can be configured to run FastCGI with two modules: mod_fastcgi and mod_fcgid. The difference is explained at Debian bug report #504132: “mod_fcgid passes just one request to the FCGI server at a time while mod_fastcgi passes several requests at once, the latter is usually better for PHP, as PHP can manage several request using several threads and opcode caches like APC usually work only with threads and not with processes. This means that using mod_fcgid you end up having many PHP processes which all have their very own opcode cache.”

In short: mod_fastcgi is better.

Install mod_fastcgi

So we need to get mod_fastcgi which isn’t at the time found from CentOS default or EPEL repos but from RPMForge or by building it from sources.

Getting mod_fastcgi from RPMForge

Install the RPMForge repo:

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Add some priorities which repo to use:

# yum install yum-priorities
 
# vi /etc/yum.repos.d/epel.repo 
... add the line priority=10 to the [epel] section

Install mod_fastcgi

# yum install mod_fastcgi

Or building mod_fastcgi from sources

You can build the mod_fastcgi from sources. Make sure required packages are installed (httpd-devel and apr-devel required to compile mod_fastcgi):

# yum install libtool httpd-devel apr-devel apr

Get the latest mod_fastcgi source code:

# cd /opt
# wget http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz

Untar tar ball:

# tar -zxvf mod_fastcgi-current.tar.gz
# cd mod_fastcgi-2.4.6/

As we are using Apache 2, we make a copy of Makefile.AP2: cp Makefile.AP2 Makefile

Compile and install mod_fastcgi for 64 bit system:

# make top_dir=/usr/lib64/httpd
# make install top_dir=/usr/lib64/httpd

Configure mod_fastcgi

If you have php enabled disable it

# mv /etc/httpd/conf.d/{php.conf,php.conf.disable}

Set up a (non-existent) directory that Apache can route the requests through. That directory must be available to Apache and it might be /usr/lib/cgi-bin/ so the routed file is then e.g. /usr/lib/cgi-bin/php5-fcgi.

# mkdir /usr/lib/cgi-bin/

Configure mod_fastcgi settings in /etc/httpd/conf.d/mod_fastcgi.conf to be:

LoadModule fastcgi_module modules/mod_fastcgi.so
 
<IfModule mod_fastcgi.c>
	DirectoryIndex index.php index.html index.shtml index.cgi
	AddHandler php5-fcgi .php
	Action php5-fcgi /php5-fcgi
	Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
	FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization
 
	# For monitoring status with e.g. Munin
	<LocationMatch "/(ping|status)">
		SetHandler php5-fcgi-virt
		Action php5-fcgi-virt /php5-fcgi virtual
	</LocationMatch>
</IfModule>

We add handler and action which sends all requests of PHP to the virtual URL created above, which is in turn then sent to the external FastCGI server. We also add configuration to have some status information about our PHP-FPM.

Start Apache:

# service httpd start

PHP should now work.

Maven plugin to generate HTML from Markdown

Writing documentation or user guide isn’t the most interesting part of the software project but it’s still needed to be done and important for the end users. There are many ways to achieve documentation but using Markdown is good way to make it easier. After that you just need to convert it to HTML and as I couldn’t find a suitable Maven plugin to do that nicely I wrote one myself: markdown-page-generator-plugin.

Software developers are a bit lazy to write good and up to date documentation and as I needed to write the API documentation for the web application as separate HTML pages I got to know the issue first hand. I like writing documents but typing the API documentation as static HTML wasn’t very effective or fun. Fortunately it was quite easy to do it with Markdown which allows you to write documentation using an easy-to-read, easy-to-write plain text format which you can then convert to HTML. After that the only problem was to enable easy HTML generation directly from Maven Lifecycle and thus was the “Markdown to HTML Page Generator Maven Plugin” developed.

Markdown to HTML Page Generator Maven Plugin

“Markdown to HTML Page Generator” is a simple Maven Plugin which reads the input directory for Markdown files with .md suffix and outputs HTML files to target directory. There are couple of configuration options to set the input/output directories and if you want to add header and footer files to make it structurally valid HTML. The plugin uses pegdown Markdown processor.

Source code

The plugin code can be found from GitHub: markdown-page-generator-plugin.

Usage

Check out the plugin’s GitHub page for configuration options.

Using the plugin could be easier but as it isn’t in Maven repository you have to e.g. add it as Maven project to same Eclipse workspace as your project and build it. After that add it as a plugin to your build and it’s run automatically when processing sources. You can also run it from command line with “mvn com.ruleoftech:markdown-page-generator-plugin:generate”.

Setting up LAMP stack on OS X

Setting up LAMP stack for web development on OS X can be done with 3rd party software like MAMP but as Mac OS X comes with pre-installed Apache and PHP it’s easy to use the native setup. You just need to configure Apache, PHP and install MySQL.

Setup Apache2

Set up the Server Name to localhost to suppress the warning about fully qualified domain name and enable PHP module.

$ sudo vim /etc/apache2/httpd.conf
ServerName localhost:80
LoadModule php5_module libexec/apache2/libphp5.so

Create “virtual hosts” under your Sites. Change the username to your account’s username.

~$ sudo vim /etc/apache2/users/username.conf
<VirtualHost *:80>
  ServerName dev
  DocumentRoot /Users/username/Sites
  VirtualDocumentRoot /Users/username/Sites/%-2/htdocs
  UseCanonicalName Off
 
  <Directory "/Users/username/Sites/*/htdocs">
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

Now Apache serves your projects from your home directory’s Sites folder. Apache will serve files from the htdocs folder like “~/Sites/projectname/htdocs”.

Now just restart Apache and check that it’s running.

$ sudo apachectl restart
$ ps aux | grep httpd

Setup PHP

$ sudo cp /etc/php.ini.default /etc/php.ini

Edit php.ini for easier debugging:

error_reporting  =  E_ALL | E_STRICT
display_errors = On
html_errors = On

Setup MySQL

MySQL can be installed directly from Oracle’s MySQL packages or by using Homebrew.

Install Homebrew

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install MySQL using Homebrew

$ brew install mysql

Install the MySQL system tables and have it run as your system user:

$ unset TMPDIR
$ mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

Start MySQL and check that it’s running

$ mysql.server start
$ ps aux | grep mysql

Reset the root password. Change the “5.5.27” to your installed version number.

$ /usr/local/Cellar/mysql/5.5.27/bin/mysqladmin -u root password 'YOUR_NEW_PASSWORD'

As we are using the Homebrew package for MySQL and the default php.ini file then PHP is trying to connect to MySQL through the default_socket at /var/mysql/mysql.sock which doesn’t exist as MySQL is using /tmp/mysql.sock. Just change all instances of /var/mysql/mysql.sock to /tmp/mysql.sock.

$ sudo sed -i "" "s:/var/mysql/mysql.sock:/tmp/mysql.sock:g" /etc/php.ini

And you’re done.

Monitoring Java EE application with JavaMelody

Software development is much more than just coding application by requirements and deploying it to production as the real work really starts after it has been shipped: maintenance, improvements and problem solving. And for that it’s good to have some data. It’s said “if you can’t measure it, you can’t improve it” and in ideal situations you plan for measuring your app but often you have to do it in production to detect and diagnose performance problems. There are different ways to measure a Java EE application and one good tool to achieve performance and monitoring statistics is Javamelody which has low overhead, is non-intrusive, informative and simple to setup but still very capable.

“If you can not measure it, you can not improve it.” – Lord Kelvin

Application performance can be measured by two main methods: computational resources used by the application and the performance as seen by a user of the application. By measuring these quantities we get an empirical performance baseline of the application which then can used to detect changes in performance. By using performance monitoring, which is an act of non-intrusively collect or observe performance data from running application, we get measurements to identify or isolate potential issues from real operation of an application without having a severe impact on runtime responsiveness or throughput.

There are different ways to achieve performance and monitoring statistics and it’s useful to plan for measuring application’s performance while still in development as later on you surely want to know how it is performing. You can do it by leveraging different JMX-based tools like Metrics and Servo but if you didn’t really plan for it you still have choices. You can use tools like standard JDK tools (jconsole, jstat, jmap, jstack, hprof) for low level JVM monitoring or some monitoring application like Javamelody. And as time is always scarce in development it’s useful to deploy a tool like JavaMelody which provide quick and easy access to performance monitoring. But if you are interested of some narrow focus measurement you’re better of with some custom tool.

Monitoring with JavaMelody

JavaMelody is an open source (LGPL) application to monitor Java or Java EE application servers in QA and production environments. It is a tool to measure and calculate statistics on real operation of an application depending on the usage of the application by users and is mainly based on statistics of requests and on evolution charts which can be viewed on the current day, week, month, year or custom period. The statistics can be viewed on a HTML page and sent as PDF reports by email.

JavaMelody lists the following use cases:

  • It allows to improve applications in QA and production
  • Give facts about the average response times and number of executions
  • Make decisions when trends are bad, before problems become too serious
  • Optimize based on the more limiting response times
  • Find the root causes of response times
  • Verify the real improvement after optimization

Setting up

Setting up JavaMelody is quite easy and the needed steps are covered in User guide. Integrating JavaMelody to your application can be done in less than 10 minutes, by automatic discovery of environment: it only requires to copy 2 jar files and to add 10 lines in a xml file. But for more detailed and wider measurements you need couple more lines to xml files and here are the changes I made for our Wicket, Spring, JPA, Hibernate -Java EE project.

JavaMelody and dependencies

Getting JavaMelody and it’s dependencies is easy with Maven2 and you just need to add javamelody-core and if you want to have PDF reports also iText in your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!-- javamelody-core -->
<dependency>
	<groupId>net.bull.javamelody</groupId>
	<artifactId>javamelody-core</artifactId>
	<version>1.45.0</version>
</dependency>
<!-- itext, option to add PDF export -->
<dependency>
	<groupId>com.lowagie</groupId>
	<artifactId>itext</artifactId>
	<version>2.1.7</version>
	<exclusions>
		<exclusion>
			<artifactId>bcmail-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bcprov-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bctsp-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
	</exclusions>
</dependency>

Monitoring application

JavaMelody needs just a monitoring filter before the description of webapp’s servlet in WEB-INF/web.xml or not even that as described in the User guide. But usually you want a little bit more than the minimum 10 additional lines in web.xml.

By adding the customResourceFilter you get customized look & feel, you can exclude some urls from statistics with url-exclude-pattern regular expression pattern, eamil-parameters are for weekly reports and by including monitoring-spring.xml you can monitor DataSource with a Spring post-processor and objects initialized with Spring.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
...
<context-param>
	<param-name>contextConfigLocation</param-name>
	<param-value>
		classpath:net/bull/javamelody/monitoring-spring.xml
		classpath:applicationContext-service.xml
		classpath:applicationContext-persist.xml
		classpath:applicationContext-web.xml
		classpath:applicationContext-security.xml
    </param-value>
</context-param>
 
<!-- Must be defined before javamelody filters -->
<filter>
	<filter-name>springSecurityFilterChain</filter-name>
	<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
	<filter-name>springSecurityFilterChain</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
 
<!--====================== Monitoring ===================================-->
<!-- Custom CSS -->
<filter>
	<filter-name>customResourceFilter</filter-name>
	<filter-class>net.bull.javamelody.CustomResourceFilter</filter-class>
	<init-param>
		<param-name>monitoring.css</param-name>
		<param-value>/styles/monitoring.css</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>customResourceFilter</filter-name>
	<url-pattern>/monitoring</url-pattern>
</filter-mapping>
<!-- Monitor filter -->
<filter>
	<filter-name>monitoring</filter-name>
	<filter-class>net.bull.javamelody.MonitoringFilter</filter-class>
	<init-param>
		<param-name>storage-directory</param-name>
		<param-value>logs/monitoring</param-value>
	</init-param>
	<init-param>
		<param-name>url-exclude-pattern</param-name>
		<param-value>(/images/.*|/js/.*|/styles/.*)</param-value>
	</init-param>
	<init-param>
		<param-name>admin-emails</param-name>
		<param-value>first.last@example.com</param-value>
	</init-param>
	<init-param>
		<param-name>mail-session</param-name>
		<param-value>MailSession</param-value>
	</init-param>
	<init-param>
		<param-name>mail-periods</param-name>
		<param-value>week,month</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>monitoring</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
<listener>
	<listener-class>net.bull.javamelody.SessionListener</listener-class>
</listener>
...

Spring Security config

For restricting the access to monitoring statistics I added the following to the Spring Security’s applicationContext-security.xml. There are also other options for security. The key point here is to make sure, that the monitoring-filter in web.xml (above) is defined after the Spring Security filter chain.

1
2
3
4
5
6
<http auto-config="true" use-expressions="true">
    ...
    <intercept-url pattern="/monitoring/**" access="hasRole('ROLE_ADMIN')" />
	<intercept-url pattern="/**"
			access="hasAnyRole('ROLE_ADMIN, 'ROLE_USER'')" />
</http>

Monitoring SQL and datasources

For monitoring datasources and SQL I just added jndi-lookup to applicationContext-persist.xml. Other options to enable JDBC monitoring are described in the User guide.

1
2
3
4
5
6
7
...
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myAppDS"/>
 
<bean id="entityManagerFactory"
	class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
	<property name="dataSource" ref="dataSource" />
...

Business facades (Spring)

If the application to monitor contains some objects initialized by Spring, EJB or Guice their methods execution can also be added to statistics. As described in the User guide for monitoring Spring Business facades there are couple of options and for example with JdkRegexpMethodPointcut in applicationContext-web.xml you can catch objects with regular expression like “all that have Service in their names”.

1
2
3
4
5
6
7
8
9
...
<bean id="facadeMonitoringAdvisor" class="net.bull.javamelody.MonitoringSpringAdvisor">
	<property name="pointcut">
		<bean class="org.springframework.aop.support.JdkRegexpMethodPointcut">
			<property name="pattern" value=".*Service.*" />
		</bean>
	</property>
</bean>
...

EHCache statistics

If you want to see also EHCache statistics add statistics=”true” to ehcache.xml config file.

1
2
3
4
5
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
    <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" statistics="true" />
    <cache name="fooCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
    <cache name="barCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
</ehcache>

Database information and statistics

JavaMelody already shows datasource and SQL information but it’s also possible to display information and statistics on the database like the longest requests in cumulative time with display of the cpu time and of the elementary cost. For showing that information the user in the database used by the monitored application must have the necessary rights to read those information and statistics.

In Oracle database the request practically means select * from v$session and you can grant access as ‘system’ user with GRANT SELECT any dictionary TO myapplicationuser. Strangely granting just the select for v$session (GRANT SELECT ON sys.v_$session TO myapplicationuser;) wasn’t enough.

Reports

There is also a possibility to get weekly, daily or monthly report in pdf format sent by email to one or several people. It needs iText library for webapp and Java’s JavaMail and Activation libraries in your server for the mail session.

With WebLogic you add some email parameters in webapp’s web.xml (above) and configure a Mail Session in WebLogic AdminServer: Services > Mail Sessions with JNDI Name and JavaMail properties. The report provides the same information you can find in monitoring web page with high and detailed level information.

Statistics’ storage

Javamelody data is stored in files on disk and there are 2 types of files: *.rrd files for values in graphics in RRD format (using jrobin library) and *.ser.gz for values in statistics.

The performance statistics are stored to temp/javamelody directory by default which means /tmp/javamelody in Linux and in Windows when running webapp in Eclipse something like Users/developer/AppData/Local/Temp/javamelody. If you want to reset the counters just delete all the files. The location can be changed with storage-directory parameter and if the name of the directory starts with ‘/’, it is considered as an absolute path, otherwise it is considered as relative to the temporary directory.

There is also option to use centralized collect server to store statistics and for monitoring of several applications.

Viewing performance monitoring

After you have set up your webapp to have the monitoring you can see statistics at URL like http:////monitoring depending your configuration.

Some example JavaMelody statistics from development:

Overhead

Monitoring and filtering doesn’t come without costs and there has been discussions on JavaMelody wiki about the overhead the monitoring does. It is said that the overhead is so low that it can be enabled continuously in Quality Assuarance environment and if no problem arises in QA, also continuously in production environment. And with little overhead you will be able to know what needs optimizing in the QA or production servers so that the overhead of JavaMelody will practically be negative.

The discussion contained some notes:

  • Architecture of JavaMelody is lightweight so it has a lower overhead as compared it to other available solutions
  • It is only statistics and not events so the overhead of memory is quite minimal.
  • It does monitoring not profiling: there is no instrumentation of classes and instead “interceptors” for http, jdbc, spring or ejb3.
  • No database and no recording of each events even in a file or over the wire: only statistics of requests are kept. The overhead of cpu is minimal with no I/O on the wire and minimal I/O on disk (just to take a backup of statistics at a regular interval). The overhead of some other good monitoring solutions is in the recording of each event in a database or in a master server.
  • You have the choice to use centralized collect server which unloads the memory, the backup storage and the generation of reports to another server while adding I/O on the wire for sending deltas of the statistics.

Summary

“If you can not measure it, you can not improve it.” It doesn’t matter how you do it but if you want to use real user data and environment just do it non-intrusively with performance monitoring tools like Javamelody.

Weblogic Server Auto Restart with Node Manager as Linux service

Sometimes servers need to reboot and then it’s nice to have certain services to start automatically. Oracle Weblogic’s Node Manager is one of them and in order to have Node Manager start automatically it must be configured as a daemon. Unfortunately Oracle doesn’t provide init scripts to run it as a Linux service but it’s pretty simple to create your own startup scripts. Just create a new nodemgr script under /etc/init.d/, add it as a service and you’re done, as Oracle Fusion Middleware -blog writes.

For example on Red Hat Enterprise Linux Server 5.6 with Oracle Weblogic Server 10.3.5 the /etc/init.d/nodemgr looks like this (edit the script to reflect your Weblogic installation path):


#!/bin/sh
#
# nodemgr Oracle Weblogic NodeManager service
#
# chkconfig:   345 85 15
# description: Oracle Weblogic NodeManager service

### BEGIN INIT INFO
# Provides: nodemgr
# Required-Start: $network $local_fs
# Required-Stop:
# Should-Start:
# Should-Stop:
# Default-Start: 3 4 5
# Default-Stop: 0 1 2 6
# Short-Description: Oracle Weblogic NodeManager service.
# Description: Starts and stops Oracle Weblogic NodeManager.
### END INIT INFO

. /etc/rc.d/init.d/functions

# Your WLS home directory (where wlserver_10.3 is)
export MW_HOME="/oracle/product/mw11g"
export JAVA_HOME="/oracle/java/jdk1.6.0_29"
DAEMON_USER="oracle"
PROCESS_STRING="^.*/oracle/product/mw11g/.*weblogic.NodeManager.*"

source $MW_HOME/wlserver_10.3/server/bin/setWLSEnv.sh > /dev/null
export NodeManagerHome="$WL_HOME/common/nodemanager"
NodeManagerLockFile="$NodeManagerHome/nodemanager.log.lck"

PROGRAM="$MW_HOME/wlserver_10.3/server/bin/startNodeManager.sh"
SERVICE_NAME=`/bin/basename $0`
LOCKFILE="/var/lock/subsys/$SERVICE_NAME"

RETVAL=0

start() {
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ ! -z "$OLDPID" ]; then
            echo "$SERVICE_NAME is already running (pid $OLDPID) !"
            exit
        fi

        echo -n $"Starting $SERVICE_NAME: "
        /bin/su $DAEMON_USER -c "$PROGRAM &"

        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] && touch $LOCKFILE
}

stop() {
        echo -n $"Stopping $SERVICE_NAME: "
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/kill -TERM $OLDPID
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        echo
        /bin/rm -f $NodeManagerLockFile
        [ $RETVAL -eq 0 ] && rm -f $LOCKFILE

}

restart() {
        stop
        sleep 10
        start
}

case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart|force-reload|reload)
        restart
        ;;
  condrestart|try-restart)
        [ -f $LOCKFILE ] && restart
        ;;
  status)
        OLDPID=`/usr/bin/pgrep -f $PROCESS_STRING`
        if [ "$OLDPID" != "" ]; then
            /bin/echo "$SERVICE_NAME is running (pid: $OLDPID)"
        else
            /bin/echo "$SERVICE_NAME is stopped"
        fi
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload|condrestart}"
        exit 1
esac

exit $RETVAL


Add the Node Manager to start after server reboot:

# chmod +x /etc/init.d/nodemgr
# chkconfig --add nodemgr
# chkconfig --list
nodemgr         0:off   1:off   2:off   3:on    4:on    5:on    6:off

Also now the Node Manager can be controlled via the service command (e.g. service nodemgr restart).

When you have the Node Manager restarting automatically after a system reboot, you can also have Weblogic managed servers automatically restarted by Node Manager. Managed servers will be restarted only if they were running at the time the shutdown was issued. Just activate the Auto Restart option in the Administration Console (Environment > Servers > selected server > Health Monitoring) and you might also need to set the CrashRecoveryEnabled to “true” in $WL_HOME/wlserver_10.3/common/nodemanager/nodemanager.properties.

With little scripting and configuration your sysadmin tasks have now become a little easier.

Web application test automation with Robot Framework

Software quality has always been important but seems that lately it has become more generally acknowledged fact that quality assurance and testing aren’t things to be left behind. With Java EE Web applications you have different ways to achieve test coverage and test that your application works with tools like JUnit, Mockito and DBUnit. But what about testing your web application with different browsers? One great way is to use Robot Framework which is a generic test automation framework and when combined with Selenium 2 it makes both writing your tests and running them quite intuitive.

Contents

Introduction

Robot Framework which is a generic test automation framework for acceptance testing and its tabular test data syntax is almost plain English and easy to understand. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new higher-level keywords from existing ones using the same syntax that is used for creating test cases. Robot Framework itself is open source and released under Apache License 2.0, and most of the libraries and tools in the ecosystem are also open source. The development of the core framework is supported by Nokia Siemens Networks.

Robot Framework doesn’t do any specific testing activity but instead it acts as a front end for libraries like Selenium2Library. Selenium2Library is a web testing library for Robot Framework that leverages the Selenium 2 (WebDriver) libraries from the Selenium project. In practice it starts the browser (eg. IE, Firefox, Chrome) and runs the tests against it natively as a user would. There’s no need to manually click through the user interface.

Robot Framework has good documentation and by going through the “Web testing with Robot Framework and Selenium2Library” demo you see how it’s used in web testing, get introduction to test data syntax, how tests are executed, and how logs and reports look like. For more detailed view about Robot Framework’s features you can read the User Guide.

Installing test tools

The “Web testing with Robot Framework and Selenium2Library” demo is good starting point for getting to know Robot Framework but it more or less skips the details of setting up the system and as the installation instructions are a bit too verbose here is an example how to install and use Robot Framework and Selenium 2 in 64-bit Windows 7.

Python installation

First we need Python as a precondition to run Robot Framework and we install Python version 2.7.x as Robot Framework is currently not compatible with Python 3.x. From the Python download page select Python 2.7.9 Windows X86-64 Installer.

For using the RIDE editor we also need wxPython. From the download page select wxPython2.8-win64-unicode-py27 for 64-bit Python 2.7.

Next we need to set up the PATH environment variable in Windows if you didn’t setup it when you installed Python.

Open Start > Settings > Control Panel > System > Advanced > Environment Variables
Select System variables > PATH > Edit and add e.g. ;\Python27;C:\Python27\Scripts at the end of the value.
Exit the dialog with OK to save the changes.

Starting from Python 2.7.9, the standard Windows installer by default installs and activates pip.

Robot Framework and Selenium2Library installation

In practice it is easiest to install Robot Framework and Selenium2Library along with its dependencies using pip package manager. Once you have pip installed, all you need to do is running these commands in your Command Prompt:

1. pip install robotframework
2. pip install robotframework-selenium2library

It’s good to notice that pip has a “feature” that unless a specific version is given, they install the latest possible version even if that is an alpha or beta release. A workaround is giving the version explicitly. like pip install robotframework==2.7.7

RIDE installation

RIDE is a light-weight and intuitive editor for Robot Framework test case files. It can be installed by using Windows installer (select robotframework-ride-1.1.win-amd64.exe) or with pip using:

pip install robotframework-ride

The Windows installer does a shortcut to the desktop and you can start it from Command Prompt with command ride.py.

Now you have everything you need to create and execute Robot Framework tests.

Executing Robot Framework tests

As described in WebDemo running tests requires the demo application located under demoapp directory to be running. It can be started by executing it from the command line:

python demoapp/server.py

After the demo application is started, it is be available at http://localhost:7272 and it needs to be running while executing the automated tests. It can be shut down by using Ctrl-C.

In Robot Framework each file contains one or more tests and is treated as a test suite. Every directory that contains a test suite file or directory is also a test suite. When Robot Framework is executed on a directory it will go through all files and directories of the correct kind except those that start with an underscore character.

WebDemo’s test cases are located in login_tests directory and to execute them all type in your Command Prompt:

pybot login_tests

Running the tests opens a browser window which Selenium 2 is driving natively as a user would and you can see the interactions.
When the test is finished executing four files will have been generated: report.html, log.html and output.xml. On failed tests selenium takes screenshots which are named like selenium-screenshot-1.png. The browser can also be run on a remote machine using the Selenium Server.

You can also run an individual test case file and use various command line options (see pybot –help) supported by Robot Framework:

pybot login_tests/valid_login.txt
pybot --test InvalidUserName --loglevel DEBUG login_tests

If you selected Firefox as your browser and get an error like “Type Error: environment can only contain strings” that’s a bug in Selenium’s Firefox profile. You can fix it with a “monkey patch” to C:\Python27\Lib\site-packages\selenium\webdriver\firefox\firefox_profile.py.

Using different browsers

The browser that is used is controlled by ${BROWSER} variable defined in resource.txt resource file. Firefox browser is used by default, but that can be easily overridden from the command line.

pybot --variable BROWSER:Chrome login_tests
pybot --variable BROWSER:IE login_tests

Browsers like Chrome and Internet Explorer require separate Internet Explorer Driver and Chrome Driver to be installed before they can be used. InternetExplorerDriver can be downloaded from Selenium project and ChromeDriver from Chromium project. Just place them both somewhere in your PATH.

With Internet Explorer Driver you can get an error like “‘Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones.'”. As it reads in the driver’s configuration you must set the Protected Mode settings for each zone to be the same value. To set the Protected Mode settings in Internet Explorer, choose “Internet Options…” from the Tools menu, and click on the Security tab. For each zone, there will be a check box at the bottom of the tab labeled “Enable Protected Mode”.

Reading the results

After the tests have run there are couple of result files to read: report.html and log.html.

The report.html shows the results of your tests and its background is green when all tests have passed and red if any have failed. It also shows “Test Statistics” for how many tests have passed and failed. “Test Details” shows how long the test took to run and, if it failed, what the fail message was.

The log.html gives you more detailed information about why some test fails if the fail message doesn’t make it obvious. It also gives a detailed view of the execution of each of the tests.

Summary

From the short experience I have played with Robot Framework it seems to be powerful tool for designing and executing tests and good way to improve your application’s overall quality.

Next it’s time to get to know the Robot Framework syntax better, write some tests and run Selenium Server. Also the Maven plugin and RobotFramework-EclipseIDE plugin looks interesting.

References

Robot Framework documentation
Robot Framework User Guide
Web testing with Robot Framework and Selenium2Library demo
RIDE: light-weight and intuitive editor for Robot Framework test case files

Patching RichFaces 3.3.3 AJAX.js for IE9

There are always some problems when working with 3rd party frameworks when the world moves forward but the framework you’re using doesn’t. After JSF 2 was released the RichFaces development moved to 4.x version and they dropped support for the older versions although many users are still using the older versions as it’s not trivial or free to update to JSF 2 & RichFaces 4. Now if users have problems with the older versions they just have to patch it themselves as the RichFaces team isn’t going to fix e.g. the IE9 issues.

The problem with Internet Explorer 9 is that with RichFaces 3.3.3.Final the Ajax components doesn’t work and there are errors shown in the JavaScript Console. For example when testing RichFaces Ajax demos the JavaScript is assigning A.outerHTML = new XMLSerializer().serializeToString(C) which gives the following error:

SCRIPT16386: No such interface supported

Fortunately it’s relatively easy to fix it for IE9 by patching the AJAX.js JavaScript file in RichFaces. The process is discussed in RichFaces’s RF-9485 -JIRA ticket and JBoss forum message. In short the patching for AJAX.js contains following steps.

Update 2013-06-13: If you want quick solutions and aren’t interested about how it’s done, you can get the patched AJAX.js from the RF-9485 ticket as Ken Clubok pointed out in the comments.

Upgrading Sarissa Framework and patching RichFaces AJAX.js file

Resources:

1) Instead of patching Sarissa as suggested in the ticket it’s better to upgrade Sarissa to 0.9.9.6 as it has some additional IE9 issues fixed.

Download Sarissa:
http://sourceforge.net/projects/sarissa/files/sarissa/Sarissa%200.9.9.6/sarissa-0.9.9.6.jar/download

Extract sarissa.js from the jar:
sarissa-0.9.9.6.jar/gr/abiss/js/sarissa/sarissa.js

Replace first part of AJAX.js (till EOF comment) or replace sarissa.js in richfaces souce resources.
richfaces-impl-3.3.3.Final.jar/org/ajax4jsf/javascript/scripts/AJAX.js or richfaces-ui-3.3.3.Final\framework\impl\src\main\javascript\ajaxjsf\

2) For richfaces-impl work in non-compatibility mode in IE9, A4J.AJAX.replacePage method should be fixed in RichFaces sources in AJAX.js or JSFAJAX.js because custom document.open method does not work in IE9 document mode.

Replace (line 2125 in AJAX.js, line 1121 in JFSAJAX.js)
	if(isIE) {  
With
	if(isIE&&!Sarissa._SARISSA_IS_IE9) {

3) You will also need to change or comment the next line in AJAX.js or JSFAJAX.js (richfaces-ui-3.3.3.Final\framework\impl\src\main\javascript\ajaxjsf\)

Replace (line 2119 in AJAX.js, line 1044 in JFSAJAX.js)
	LOG.debug("Hidden JSF state fields: "+idsSpan);
to
	LOG.debug("Hidden JSF state fields: ");//+idsSpan);

Otherwise you get the following error in IE9:

	SCRIPT438: Object doesn't support this property or method
	For this line 2648 in framework pack:
	LOG.debug("Hidden JSF state fields: "+Q);

4) Now you have patched AJAX.js and you can either build a new RichFaces Jar, replace the original AJAX.js in RichFaces Jar with the patched one or just override it in your html.

The easiest way is just include the patched AJAX.js file as the last script-include within HEAD and it will take control over the bundled version of RichFaces.

In your template.jspx or similar:

<html>
	<head>
	<a4j:loadScript src="/js/AJAX_IE9fix.js" />
	...
	</head>
	...

5) Done.

Other ways to fix IE9 compatibility issues has been discussed in tickets RF-10774 as forcing IE9 to use another document mode. Also suggested solutions are to implement a browser sniffer and tell the Sarissa library to use its own XMLSerializer when Internet Explorer 9 is detected and forcing IE9 to run in an emulated mode but neither of those worked for me.

Using the WebLogic Maven Plug-In for Deployment

Using the WebLogic Maven plug-In for deployment is much easier and quicker than going through the WebLogic Server’s AdminServer and Oracle Documentation provides good examples how to do it.

In short, generating WebLogic Maven Plug-In contains following steps:

1. Build the plug-in JAR file using the WebLogic JarBuilder Tool (wljarbuilder) under MW_HOME/wlserver_10.3/server/lib/ with the following command:

java -jar wljarbuilder.jar -profile weblogic-maven-plugin

2. Extract the pom.xml file from weblogic-maven-plugin.jar under the MW_HOME/wlserver_10.3/server/lib directory, and then copy the pom.xml file to MW_HOME/wlserver_10.3/server/lib.

jar xvf MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml
 
in Linux:	
cp META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib
 
or in Windows:
copy META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib

3. Provision the weblogic-maven-plugin.jar in your local Maven repository with the following command.

mvn install:install-file -Dfile=MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar -DpomFile=pom.xml

4. Done.

The Maven plug-in can be used e.g. from application’s POM file and be bound to some phase of the Maven life cycle. For example it can be bound to “install” phase and every time you run the “mvn install” command, the deployment plug-in is also called. In my opinion better way to use the plug-in is to add it as a profile so you can call it just when you want with command like “mvn clean install -Pdeploy-wls”.

Maven Project pom.xml File

<profile>
	<id>deploy-wls</id>
	<build>
		<plugins> 
			<plugin> 
				<groupId>com.oracle.weblogic</groupId>
				<artifactId>weblogic-maven-plugin</artifactId> 
				<version>10.3.6.0</version> 
				<configuration> 
					<adminurl>t3://localhost:7001</adminurl>
					<user>weblogic</user> 
					<password>weblogic123</password> 
					<upload>true</upload> 
					<targets>myServer</targets>
					<action>deploy</action> 
					<remote>false</remote> 
					<verbose>true</verbose> 
					<source>${project.build.directory}/${project.build.finalName}.${project.packaging}</source> 
					<name>${project.build.finalName}</name> 
				</configuration> 
				<executions> 
					<execution> 
						<phase>install</phase> 
						<goals> 
							<goal>deploy</goal> 
						</goals> 
					</execution> 
				</executions> 
			</plugin> 
		</plugins> 
	</build>
</profile>

The user credentials in the POM file are provided as clear-text but for more security you can use secure configuration authentication mechanism which stores the user name and password in encrypted form in an external file, and then uses it to supply the user credentials with which to connect to the WebLogic Server domain, along with the key that was used to encrypt the file.

Looking back, planning ahead

Again a year has passed and it’s time to take a short look back and plan for the next year. A year ago I wrote that in the year 2012 the blog would be more active and I partly managed to keep up that promise with 9 articles. Not quite at least an article per month as I planned.

The articles in 2012 started with technical issues like running Crucible as a service, JSF 1.2 and value from dropdown menu and using CAcert.org signed certificates for TLS. In the Autumn as a Fujitsu LIFEBOOK4Life Insider I reviewed Stylistic M532 Android tablet and wrote about my visit to Fujitsu Forum 2012. There were more technical issues to write about software development but again the time ran out. It’s so much easier to say couple of words in Twitter or Google+ than write a coherent and insightful post about it :)

The year 2013 is almost here so it’s time to plan ahead. There are already some articles about Apache Wicket and running different development tools on Linux in the backlog waiting to be finished and some technical tasks like upgrading the theme and maybe changing the VPS provider are to be done. Will see if I manage to achieve one article per month so subscribe to the RSS feed and stay tuned.

Happy New Year!

Insights to future workplace from Fujitsu Forum 2012

This year my Autumn holiday was a bit different as I was one of the four bloggers who were invited by Fujitsu’s LIFEBOOK4Life campaign to visit Fujitsu Forum 2012 in Munich to hear about how the future workplace might look like and to experience new technologies to support that. We also got a tour at Fujitsu’s factory in Augsburg which provided some views how computers are made from mainboards to final product. The three days in Germany were insightful and fun. How can you not like to hear about new ideas, see innovative things, meet new people and of course experience the excellent German food and beer.

Fujitsu Forum is the largest IT-event in Europe which is visited by professional users and IT decision-makers, as well as Fujitsu channel and technology partners. In 2012 more than 12,000 IT experts from around the world attended the Fujitsu Forum in Munich. And I was one of them invited by Fujitsu LIFEBOOK4Life campaign with three other bloggers (called Insiders). The chance I wrote last time came true. Our trip to Fujitsu Forum 2012 was scheduled to contain breakout sessions and keynotes on Wednesday and a tour at the Fujitsu factory in Augsburg on Thursday.

This year the Forum’s motto was “Reshaping ICT – Reshaping Business”: How to combine business processes and IT to form a stable basis that will ensure growth and success in the future. Kind of redefinition of last year’s “Reshaping IT” theme. In short some buzzwords I heard most were: consumerization, BYOD, tablets, virtualization, in-sync, cloud and win 8. “One workplace – on any device”.

First day, Wednesday 7.11


photo by Kim Ekman, Vision 51

The first day at the Forum was full of exciting breakout sessions and keynotes and the exhibition area provided some innovative ideas and services from Fujitsu and technology partners. And of course you got to test new devices like Windows 8 enabled STYLISTIC Q572 -tablet with AMD Hondo platform and STYLISTIC Q702 hybrid tablet with Intel Core i3/i5. There were also some prototypes of future Fujitsu products.

The breakout sessions and keynotes provided some good insights about how the future workplace might look like, which decisions and technologies are needed to support that and what kind of tools are we using in the future. In short the three breakout sessions I listened “Has the Post-Pc Era Begun – What will be on your desktop tomorrow”, “The War Between Enduser Devices” and “Tablets & More – Cool scenarios for hot devices” can be summed to words: consumerization, byod, tablets, virtualization, in-sync, cloud and win 8. “One workplace – on any device”. The materials for all sessions can be found at Fujitsu Forum 2012 documentation page.

The three breakout sessions’ ideas about the future of workplace and tools were similar what I think about the issue: the work and workplace is changing to support mobility and working anywhere anytime. Some of the driving actors for this is that the borders between business and private life are blurring, consumerization (same devices home and work) and BYOD (bring your own device) are coming more common. We should think more about user-oriented workplace. Although the mobile computing is big the traditional desktop computing model isn’t going anywhere as there is always need for more computing power, larger screen, data protection and security.

In the future the devices we use will be more diverse and we have more of them. Thus there will be need for virtualization, syncing data and support for “One workplace – on any device” ideology. Different tasks have different needs for the device so we need a dynamic desktop experience. The sessions also provided some views about the tablets’ platforms and in Western Europe Windows, iOS and Android will be about equals in strength. It will be seen. It was said about Windows 8 that there will be a challenge with transition and touch monitors are needed to deliver significant impact on Win 8.

There were also some suggestions about the devices which can support our future workplace needs. At the desktop we could find the ESPRIMO X -series which has integrated desktop computer at the base of the monitor and the same form factor works in multiple use cases. And with the integrated Web camera with presence sensor you can lock the screen when leaving and login with face detection. Crafty. For the mobility aspects there are STYLISTIC Q572 and Q702 tablets and for laptop needs you have Ultrabooks LIFEBOOK U772 and more traditional LIFEBOOK T902.

After the day at the sessions and exhibition area it was time for lunch. A nice buffet with currywurst, burger and Weissbier. The final event for the day was Celebration Night with some comedians with guitars and then the stage was open for Amy McDonald. It was a nice gig by Amy but last year’s Anastacia was better :)

Second day, Thursday 8.11

The second day for us at the Fujitsu Forum 2012 was dedicated to for a tour at Fujitsu’s factory in Augsburg. It was kind of surprising to see how many workers there are although the automation percent was something like 85-90. Some mainboard components and checking is more easy to do by humans. Also the storage and putting things together were done manually.


After the visit to the factory it was more or less free time to roam the exhibition area because we didn’t have time for anymore sessions or keynotes. At the exhibition we found about Fraunhofer’s nLightened workplace with touch, rotate, scan, change light, adjust hight features and which runs on top of Linux. Again we saw the Made4You customization service which engraves with laser any product you want. Like my Twitter name and Fujitsu Forum 2012 hashtag to a metal cased pen.

The second day was interesting and it looked like there was some party starting when the Insiders and moderators had to head back home. It would have been fun to spend some more days in Munich but as usual the work calls.

Summary

Fujitsu Forum 2012 provided insights and ideas about how the future workplace might look like and we got to experience new technologies to support that. The three days in Munich went (too) fast and it was again great to visit Fujitsu Forum and to see other Insiders and LIFEBOOK4Life crew. Thanks LIFEBOOK4Life and Fujitsu.

If you want to read more check out what Antonia wrote about our trip to campaign site and Aba from FujitsuFans.com has made some previews and reports from the Forum.