Monitoring Java EE application with JavaMelody

Software development is much more than just coding application by requirements and deploying it to production as the real work really starts after it has been shipped: maintenance, improvements and problem solving. And for that it's good to have some data. It's said "if you can't measure it, you can't improve it" and in ideal situations you plan for measuring your app but often you have to do it in production to detect and diagnose performance problems. There are different ways to measure a Java EE application and one good tool to achieve performance and monitoring statistics is Javamelody which has low overhead, is non-intrusive, informative and simple to setup but still very capable.

"If you can not measure it, you can not improve it." - Lord Kelvin

Application performance can be measured by two main methods: computational resources used by the application and the performance as seen by a user of the application. By measuring these quantities we get an empirical performance baseline of the application which then can used to detect changes in performance. By using performance monitoring, which is an act of non-intrusively collect or observe performance data from running application, we get measurements to identify or isolate potential issues from real operation of an application without having a severe impact on runtime responsiveness or throughput.

There are different ways to achieve performance and monitoring statistics and it's useful to plan for measuring application's performance while still in development as later on you surely want to know how it is performing. You can do it by leveraging different JMX-based tools like Metrics and Servo but if you didn't really plan for it you still have choices. You can use tools like standard JDK tools (jconsole, jstat, jmap, jstack, hprof) for low level JVM monitoring or some monitoring application like Javamelody. And as time is always scarce in development it's useful to deploy a tool like JavaMelody which provide quick and easy access to performance monitoring. But if you are interested of some narrow focus measurement you're better of with some custom tool.

Monitoring with JavaMelody

JavaMelody is an open source (LGPL) application to monitor Java or Java EE application servers in QA and production environments. It is a tool to measure and calculate statistics on real operation of an application depending on the usage of the application by users and is mainly based on statistics of requests and on evolution charts which can be viewed on the current day, week, month, year or custom period. The statistics can be viewed on a HTML page and sent as PDF reports by email.

JavaMelody lists the following use cases:

  • It allows to improve applications in QA and production
  • Give facts about the average response times and number of executions
  • Make decisions when trends are bad, before problems become too serious
  • Optimize based on the more limiting response times
  • Find the root causes of response times
  • Verify the real improvement after optimization

Setting up

Setting up JavaMelody is quite easy and the needed steps are covered in User guide. Integrating JavaMelody to your application can be done in less than 10 minutes, by automatic discovery of environment: it only requires to copy 2 jar files and to add 10 lines in a xml file. But for more detailed and wider measurements you need couple more lines to xml files and here are the changes I made for our Wicket, Spring, JPA, Hibernate -Java EE project.

JavaMelody and dependencies

Getting JavaMelody and it's dependencies is easy with Maven2 and you just need to add javamelody-core and if you want to have PDF reports also iText in your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!-- javamelody-core -->
<dependency>
	<groupId>net.bull.javamelody</groupId>
	<artifactId>javamelody-core</artifactId>
	<version>1.45.0</version>
</dependency>
<!-- itext, option to add PDF export -->
<dependency>
	<groupId>com.lowagie</groupId>
	<artifactId>itext</artifactId>
	<version>2.1.7</version>
	<exclusions>
		<exclusion>
			<artifactId>bcmail-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bcprov-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bctsp-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
	</exclusions>
</dependency>

Monitoring application

JavaMelody needs just a monitoring filter before the description of webapp's servlet in WEB-INF/web.xml or not even that as described in the User guide. But usually you want a little bit more than the minimum 10 additional lines in web.xml.

By adding the customResourceFilter you get customized look & feel, you can exclude some urls from statistics with url-exclude-pattern regular expression pattern, eamil-parameters are for weekly reports and by including monitoring-spring.xml you can monitor DataSource with a Spring post-processor and objects initialized with Spring.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
...
<context-param>
	<param-name>contextConfigLocation</param-name>
	<param-value>
		classpath:net/bull/javamelody/monitoring-spring.xml
		classpath:applicationContext-service.xml
		classpath:applicationContext-persist.xml
		classpath:applicationContext-web.xml
		classpath:applicationContext-security.xml
    </param-value>
</context-param>
 
<!-- Must be defined before javamelody filters -->
<filter>
	<filter-name>springSecurityFilterChain</filter-name>
	<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
	<filter-name>springSecurityFilterChain</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
 
<!--====================== Monitoring ===================================-->
<!-- Custom CSS -->
<filter>
	<filter-name>customResourceFilter</filter-name>
	<filter-class>net.bull.javamelody.CustomResourceFilter</filter-class>
	<init-param>
		<param-name>monitoring.css</param-name>
		<param-value>/styles/monitoring.css</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>customResourceFilter</filter-name>
	<url-pattern>/monitoring</url-pattern>
</filter-mapping>
<!-- Monitor filter -->
<filter>
	<filter-name>monitoring</filter-name>
	<filter-class>net.bull.javamelody.MonitoringFilter</filter-class>
	<init-param>
		<param-name>storage-directory</param-name>
		<param-value>logs/monitoring</param-value>
	</init-param>
	<init-param>
		<param-name>url-exclude-pattern</param-name>
		<param-value>(/images/.*|/js/.*|/styles/.*)</param-value>
	</init-param>
	<init-param>
		<param-name>admin-emails</param-name>
		<param-value>first.last@example.com</param-value>
	</init-param>
	<init-param>
		<param-name>mail-session</param-name>
		<param-value>MailSession</param-value>
	</init-param>
	<init-param>
		<param-name>mail-periods</param-name>
		<param-value>week,month</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>monitoring</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
<listener>
	<listener-class>net.bull.javamelody.SessionListener</listener-class>
</listener>
...

Spring Security config

For restricting the access to monitoring statistics I added the following to the Spring Security's applicationContext-security.xml. There are also other options for security. The key point here is to make sure, that the monitoring-filter in web.xml (above) is defined after the Spring Security filter chain.

1
2
3
4
5
6
<http auto-config="true" use-expressions="true">
    ...
    <intercept-url pattern="/monitoring/**" access="hasRole('ROLE_ADMIN')" />
	<intercept-url pattern="/**"
			access="hasAnyRole('ROLE_ADMIN, 'ROLE_USER'')" />
</http>

Monitoring SQL and datasources

For monitoring datasources and SQL I just added jndi-lookup to applicationContext-persist.xml. Other options to enable JDBC monitoring are described in the User guide.

1
2
3
4
5
6
7
...
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myAppDS"/>
 
<bean id="entityManagerFactory"
	class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
	<property name="dataSource" ref="dataSource" />
...

Business facades (Spring)

If the application to monitor contains some objects initialized by Spring, EJB or Guice their methods execution can also be added to statistics. As described in the User guide for monitoring Spring Business facades there are couple of options and for example with JdkRegexpMethodPointcut in applicationContext-web.xml you can catch objects with regular expression like "all that have Service in their names".

1
2
3
4
5
6
7
8
9
...
<bean id="facadeMonitoringAdvisor" class="net.bull.javamelody.MonitoringSpringAdvisor">
	<property name="pointcut">
		<bean class="org.springframework.aop.support.JdkRegexpMethodPointcut">
			<property name="pattern" value=".*Service.*" />
		</bean>
	</property>
</bean>
...

EHCache statistics

If you want to see also EHCache statistics add statistics="true" to ehcache.xml config file.

1
2
3
4
5
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
    <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" statistics="true" />
    <cache name="fooCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
    <cache name="barCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
</ehcache>

Database information and statistics

JavaMelody already shows datasource and SQL information but it's also possible to display information and statistics on the database like the longest requests in cumulative time with display of the cpu time and of the elementary cost. For showing that information the user in the database used by the monitored application must have the necessary rights to read those information and statistics.

In Oracle database the request practically means select * from v$session and you can grant access as 'system' user with GRANT SELECT any dictionary TO myapplicationuser. Strangely granting just the select for v$session (GRANT SELECT ON sys.v_$session TO myapplicationuser;) wasn't enough.

Reports

There is also a possibility to get weekly, daily or monthly report in pdf format sent by email to one or several people. It needs iText library for webapp and Java's JavaMail and Activation libraries in your server for the mail session.

With WebLogic you add some email parameters in webapp's web.xml (above) and configure a Mail Session in WebLogic AdminServer: Services > Mail Sessions with JNDI Name and JavaMail properties. The report provides the same information you can find in monitoring web page with high and detailed level information.

Statistics' storage

Javamelody data is stored in files on disk and there are 2 types of files: *.rrd files for values in graphics in RRD format (using jrobin library) and *.ser.gz for values in statistics.

The performance statistics are stored to temp/javamelody directory by default which means /tmp/javamelody in Linux and in Windows when running webapp in Eclipse something like Users/developer/AppData/Local/Temp/javamelody. If you want to reset the counters just delete all the files. The location can be changed with storage-directory parameter and if the name of the directory starts with '/', it is considered as an absolute path, otherwise it is considered as relative to the temporary directory.

There is also option to use centralized collect server to store statistics and for monitoring of several applications.

Viewing performance monitoring

After you have set up your webapp to have the monitoring you can see statistics at URL like http:////monitoring depending your configuration.

Some example JavaMelody statistics from development:

Overhead

Monitoring and filtering doesn't come without costs and there has been discussions on JavaMelody wiki about the overhead the monitoring does. It is said that the overhead is so low that it can be enabled continuously in Quality Assuarance environment and if no problem arises in QA, also continuously in production environment. And with little overhead you will be able to know what needs optimizing in the QA or production servers so that the overhead of JavaMelody will practically be negative.

The discussion contained some notes:

  • Architecture of JavaMelody is lightweight so it has a lower overhead as compared it to other available solutions
  • It is only statistics and not events so the overhead of memory is quite minimal.
  • It does monitoring not profiling: there is no instrumentation of classes and instead "interceptors" for http, jdbc, spring or ejb3.
  • No database and no recording of each events even in a file or over the wire: only statistics of requests are kept. The overhead of cpu is minimal with no I/O on the wire and minimal I/O on disk (just to take a backup of statistics at a regular interval). The overhead of some other good monitoring solutions is in the recording of each event in a database or in a master server.
  • You have the choice to use centralized collect server which unloads the memory, the backup storage and the generation of reports to another server while adding I/O on the wire for sending deltas of the statistics.

Summary

"If you can not measure it, you can not improve it." It doesn't matter how you do it but if you want to use real user data and environment just do it non-intrusively with performance monitoring tools like Javamelody.

Web application test automation with Robot Framework

Software quality has always been important but seems that lately it has become more generally acknowledged fact that quality assurance and testing aren't things to be left behind. With Java EE Web applications you have different ways to achieve test coverage and test that your application works with tools like JUnit, Mockito and DBUnit. But what about testing your web application with different browsers? One great way is to use Robot Framework which is a generic test automation framework and when combined with Selenium 2 it makes both writing your tests and running them quite intuitive.

Contents

Introduction

Robot Framework which is a generic test automation framework for acceptance testing and its tabular test data syntax is almost plain English and easy to understand. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new higher-level keywords from existing ones using the same syntax that is used for creating test cases. Robot Framework itself is open source and released under Apache License 2.0, and most of the libraries and tools in the ecosystem are also open source. The development of the core framework is supported by Nokia Siemens Networks.

Robot Framework doesn't do any specific testing activity but instead it acts as a front end for libraries like Selenium2Library. Selenium2Library is a web testing library for Robot Framework that leverages the Selenium 2 (WebDriver) libraries from the Selenium project. In practice it starts the browser (eg. IE, Firefox, Chrome) and runs the tests against it natively as a user would. There's no need to manually click through the user interface.

Robot Framework has good documentation and by going through the "Web testing with Robot Framework and Selenium2Library" demo you see how it's used in web testing, get introduction to test data syntax, how tests are executed, and how logs and reports look like. For more detailed view about Robot Framework's features you can read the User Guide.

Installing test tools

The "Web testing with Robot Framework and Selenium2Library" demo is good starting point for getting to know Robot Framework but it more or less skips the details of setting up the system and as the installation instructions are a bit too verbose here is an example how to install and use Robot Framework and Selenium 2 in 64-bit Windows 7.

Python installation

First we need Python as a precondition to run Robot Framework and we install Python version 2.7.x as Robot Framework is currently not compatible with Python 3.x. From the Python download page select Python 2.7.9 Windows X86-64 Installer.

For using the RIDE editor we also need wxPython. From the download page select wxPython2.8-win64-unicode-py27 for 64-bit Python 2.7.

Next we need to set up the PATH environment variable in Windows if you didn't setup it when you installed Python.

Open Start > Settings > Control Panel > System > Advanced > Environment Variables
Select System variables > PATH > Edit and add e.g. ;\Python27;C:\Python27\Scripts at the end of the value.
Exit the dialog with OK to save the changes.

Starting from Python 2.7.9, the standard Windows installer by default installs and activates pip.

Robot Framework and Selenium2Library installation

In practice it is easiest to install Robot Framework and Selenium2Library along with its dependencies using pip package manager. Once you have pip installed, all you need to do is running these commands in your Command Prompt:

1. pip install robotframework
2. pip install robotframework-selenium2library

It's good to notice that pip has a "feature" that unless a specific version is given, they install the latest possible version even if that is an alpha or beta release. A workaround is giving the version explicitly. like pip install robotframework==2.7.7

RIDE installation

RIDE is a light-weight and intuitive editor for Robot Framework test case files. It can be installed by using Windows installer (select robotframework-ride-1.1.win-amd64.exe) or with pip using:

pip install robotframework-ride

The Windows installer does a shortcut to the desktop and you can start it from Command Prompt with command ride.py.

Now you have everything you need to create and execute Robot Framework tests.

Executing Robot Framework tests

As described in WebDemo running tests requires the demo application located under demoapp directory to be running. It can be started by executing it from the command line:

python demoapp/server.py

After the demo application is started, it is be available at http://localhost:7272 and it needs to be running while executing the automated tests. It can be shut down by using Ctrl-C.

In Robot Framework each file contains one or more tests and is treated as a test suite. Every directory that contains a test suite file or directory is also a test suite. When Robot Framework is executed on a directory it will go through all files and directories of the correct kind except those that start with an underscore character.

WebDemo's test cases are located in login_tests directory and to execute them all type in your Command Prompt:

pybot login_tests

Running the tests opens a browser window which Selenium 2 is driving natively as a user would and you can see the interactions.
When the test is finished executing four files will have been generated: report.html, log.html and output.xml. On failed tests selenium takes screenshots which are named like selenium-screenshot-1.png. The browser can also be run on a remote machine using the Selenium Server.

You can also run an individual test case file and use various command line options (see pybot --help) supported by Robot Framework:

pybot login_tests/valid_login.txt
pybot --test InvalidUserName --loglevel DEBUG login_tests

If you selected Firefox as your browser and get an error like "Type Error: environment can only contain strings" that's a bug in Selenium's Firefox profile. You can fix it with a "monkey patch" to C:\Python27\Lib\site-packages\selenium\webdriver\firefox\firefox_profile.py.

Using different browsers

The browser that is used is controlled by ${BROWSER} variable defined in resource.txt resource file. Firefox browser is used by default, but that can be easily overridden from the command line.

pybot --variable BROWSER:Chrome login_tests
pybot --variable BROWSER:IE login_tests

Browsers like Chrome and Internet Explorer require separate Internet Explorer Driver and Chrome Driver to be installed before they can be used. InternetExplorerDriver can be downloaded from Selenium project and ChromeDriver from Chromium project. Just place them both somewhere in your PATH.

With Internet Explorer Driver you can get an error like "'Unexpected error launching Internet Explorer. Protected Mode settings are not the same for all zones. Enable Protected Mode must be set to the same value (enabled or disabled) for all zones.'". As it reads in the driver's configuration you must set the Protected Mode settings for each zone to be the same value. To set the Protected Mode settings in Internet Explorer, choose "Internet Options..." from the Tools menu, and click on the Security tab. For each zone, there will be a check box at the bottom of the tab labeled "Enable Protected Mode".

Reading the results

After the tests have run there are couple of result files to read: report.html and log.html.

The report.html shows the results of your tests and its background is green when all tests have passed and red if any have failed. It also shows "Test Statistics" for how many tests have passed and failed. "Test Details" shows how long the test took to run and, if it failed, what the fail message was.

The log.html gives you more detailed information about why some test fails if the fail message doesn't make it obvious. It also gives a detailed view of the execution of each of the tests.

Summary

From the short experience I have played with Robot Framework it seems to be powerful tool for designing and executing tests and good way to improve your application's overall quality.

Next it's time to get to know the Robot Framework syntax better, write some tests and run Selenium Server. Also the Maven plugin and RobotFramework-EclipseIDE plugin looks interesting.

References

Robot Framework documentation
Robot Framework User Guide
Web testing with Robot Framework and Selenium2Library demo
RIDE: light-weight and intuitive editor for Robot Framework test case files

Using the WebLogic Maven Plug-In for Deployment

Using the WebLogic Maven plug-In for deployment is much easier and quicker than going through the WebLogic Server's AdminServer and Oracle Documentation provides good examples how to do it.

In short, generating WebLogic Maven Plug-In contains following steps:

1. Build the plug-in JAR file using the WebLogic JarBuilder Tool (wljarbuilder) under MW_HOME/wlserver_10.3/server/lib/ with the following command:

java -jar wljarbuilder.jar -profile weblogic-maven-plugin

2. Extract the pom.xml file from weblogic-maven-plugin.jar under the MW_HOME/wlserver_10.3/server/lib directory, and then copy the pom.xml file to MW_HOME/wlserver_10.3/server/lib.

jar xvf MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml
 
in Linux:	
cp META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib
 
or in Windows:
copy META-INF/maven/com.oracle.weblogic/weblogic-maven-plugin/pom.xml MW_HOME/wlserver_10.3/server/lib

3. Provision the weblogic-maven-plugin.jar in your local Maven repository with the following command.

mvn install:install-file -Dfile=MW_HOME/wlserver_10.3/server/lib/weblogic-maven-plugin.jar -DpomFile=pom.xml

4. Done.

The Maven plug-in can be used e.g. from application's POM file and be bound to some phase of the Maven life cycle. For example it can be bound to "install" phase and every time you run the "mvn install" command, the deployment plug-in is also called. In my opinion better way to use the plug-in is to add it as a profile so you can call it just when you want with command like "mvn clean install -Pdeploy-wls".

Maven Project pom.xml File

<profile>
	<id>deploy-wls</id>
	<build>
		<plugins> 
			<plugin> 
				<groupId>com.oracle.weblogic</groupId>
				<artifactId>weblogic-maven-plugin</artifactId> 
				<version>10.3.6.0</version> 
				<configuration> 
					<adminurl>t3://localhost:7001</adminurl>
					<user>weblogic</user> 
					<password>weblogic123</password> 
					<upload>true</upload> 
					<targets>myServer</targets>
					<action>deploy</action> 
					<remote>false</remote> 
					<verbose>true</verbose> 
					<source>${project.build.directory}/${project.build.finalName}.${project.packaging}</source> 
					<name>${project.build.finalName}</name> 
				</configuration> 
				<executions> 
					<execution> 
						<phase>install</phase> 
						<goals> 
							<goal>deploy</goal> 
						</goals> 
					</execution> 
				</executions> 
			</plugin> 
		</plugins> 
	</build>
</profile>

The user credentials in the POM file are provided as clear-text but for more security you can use secure configuration authentication mechanism which stores the user name and password in encrypted form in an external file, and then uses it to supply the user credentials with which to connect to the WebLogic Server domain, along with the key that was used to encrypt the file.