Web analytics with Piwik: keeping control over your own data

Web analytics is one the essential tools for a website and including measuring web traffic and getting information about the number of visitors it can be also used as a tool to assess and improve the effectiveness of a website. The most common way to collect data is to use on-site web analytics, measure a visitor’s behavior once on your website, with page tagging technology like on Google Analytics which is widely used web analytics service. But what would you use if you want to keep control over your own data?

You don’t have to look far as the only open source web analytics application is Piwik which aims to be the ultimate open alternative to Google Analytics. Here’s a short overview to Piwik Analytics and how to get started with it.

“Web analytics is the measurement, collection, analysis and reporting of web data for purposes of understanding and optimizing web usage.” – Wikipedia

Piwik Open Analytics Platform

Piwik is web analytics application which tracks online visits to one or more websites and displays reports on these visits for analysis. In short it aims to be the ultimate open source alternative to Google Analytics. The code is GPL v3 licensed and available in GitHub. In technical side Piwik is written in PHP, uses MySQL database and you can host it by yourself. And if you don’t want to setup or host Piwik yourself you can also get commercial services.

Piwik provides the usual features you would expect from a web analytics application. You get reports regarding the geographic location of visits, the source of visits, the technical capabilities of visitors, what the visitors did and the time of visits. Piwik also provides features for analysis of the data it accumulates such as saving notes to data, goals for actions, transitions for seeing how visitors navigate, overlaying analytics data on top of a website and displaying how metrics change over time. The easiest way to see what it has to offer is to check the Piwik online demo.

Feature highlights

You might ask how Piwik differs from other web analytics applications such as Google Analytics? One principle advantage of using Piwik is that you are in control. You can host Piwik on your own server and the data is tracked inside your MySQL database: you’ve full control over your data. Software as a service analytics applications on the other hand, have full access to the data users collect. Data privacy is essential for public sector and enterprises who can’t or don’t want to share it for example with Google. You ensure that your visitors behavior on your website is not shared with advertising companies.

Other interesting feature is that it provides advanced privacy options: ability to anonymize IP addresses, purge tracking data regularly (but not report data), opt-out support and Do Not Track support. Your website visitors can decide if they want to be tracked.

You can also do scheduled reports which are sent by e-mail, import data from web server logs, use the API for accessing reports and administrative functions and Piwik also has mobile app to access the analytics data. Piwik is also customizable with plugins and you can integrate it with WordPress and other applications.

Piwik’s User Interface

Piwik has clean and simple user interface as seen in the following screenshots (taken from the online demo).

Piwik main view
Piwik main view

Piwik visitors overview
Piwik visitors overview

Setting up Piwik

Setting up Piwik is easy and there’s good documention available for running Piwik web analytics. All you need is web server like Nginx, PHP 5.5 and MySQL or MariaDB. You can setup it manually but the most easiest way to start with it is to use the provided Docker image and docker-compose. The docker-compose file setups four containers (MySQL, Piwik, Nginx and Cron) and with compose you can start it up. The Piwik image is available from official docker-library.

The alternative is to do your own Docker image for Piwik and related services. In my opinion it makes sense to have just two containers: one for Piwik related web stuff and other for MySQL. The Piwik container runs Piwik, Nginx and Cron script with e.g. supervisor. The official image uses Debian (from PHP) but Piwik runs nicely also on Alpine Linux. One thing to tinker with when using Docker is to get MySQL access to Piwik’s assets for LOAD DATA INFILE which will greatly speed Piwik’s archiving process.

If you’re setting up Piwik manually you can watch a video of installation and after that a video of configuring settings. After you’re done with the 5 minute installation you get the JavaScript tag which you add to the bottom of each page of your website. If you’re using React there’s Piwik analytics component for React Router. Piwik will then record the activity across your website within your database.

And that’s about all there is to starting with Piwik. Simple setup with Docker or doing it manually, adding the JavaScript tag, configuring some options if needed and then just wait for the data from visitors.

Summary

Piwik is good and feature rich alternative for web analytics application. Setting it up isn’t as straightforward as using some hosted service as Google Analytics but that’s the way self-hosted services always are. If you need web analytics and want to keep control of your own data and don’t mind hosting it yourseld and paying for the server then Piwik is a good choice.

Starting with WeeChat

I’ve been using IRC for some time and although Irssi has served me well, it’s time to try something different. WeeChat is a modular chat client with support for IRC and the interesting part is that it’s possible to use other interfaces like glowing-bear web frontend. WeeChat is similar to Irssi so switching over shouldn’t be an issue. But to get the configuration right and what you had on Irssi needs some effort. Here are my notes about starting with WeeChat and how I like my chat client to look.

Compiling Weechat on CentOS 6

I have my shell on CentOS and although you can find WeeChat from the repositories, it’s quite old (0.4.3 when 1.4. is the newest). So you might want to compile WeeChat by yourself. Compiling WeeChat is explained on the User Guide. You need to install some libraries before you can try using make.

yum install gettext make libgcrypt-devel libcurl-devel python-devel v8-devel gnutls-devel ruby cppunit perl-ExtUtils-Embed

After you’ve installed the needed packages download WeeChat sources and extract the weechat-1.4.tar.gz package to directory you want.

Go to the directory you extracted WeeChat and run the following commands:

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/directory
$ make
$ make install

Configuring WeeChat

Quick Start guide helps you to get started so I don’t duplicate that here. If you’re familiar with Irssi you should feel more or less at home. As I didn’t use much scripts in Irssi the most difficult part for me was to create as good theme as I had with Irssi. Otherwise the switch went better than expected.

Start WeeChat with weechat so we can start configuration.

When I started with WeeChat of course I googled how others had configured it and thus my configuration is based on this and some other snippets.

Scripts to improve functionality

/script install buffer_autoclose.py iset.pl colorize_nicks.py urlbuf.py screen_away.py
  • buffer_autoclose.py: Automatically close inactive private message buffers.
  • iset.pl: Interactive Set for configuration options
  • colorize_nicks.py: Use the weechat nick colors in the chat area and command line.
  • urlbuf.py: Common buffer for received URLs.
  • irssi_awaylog.py: Log highlights/private messages when you are away.
  • screen_away.py: Set away status when detaching and attaching from screen or tmux.

To see and set the options for the plugins you can use /set with wildcard “*”

/set *urlbuf*
/set *away*

You can install scripts also by typing “script search iset” and a selection of available plugins will appear. To leave it type “q” then press enter, if you want to install the script type “i” then press enter.

Adjust layout and colors

Make the title bar and the status bar using dark colors.

/set weechat.bar.status.color_bg 0
/set weechat.bar.title.color_bg 0
/set buffers.color.current_bg 0
/set weechat.color.chat_nick_colors 1,2,3,4,5,6
/set buffers.color.hotlist_message_fg yellow

Place a list of all buffers at the top of the screen

/set weechat.bar.buffers.position top

Merge the server buffers in to one to take up less of our precious screen space (you can switch between them with ctrl-x).

/set buffers.look.hide_merged_buffers server

Hide any channels from the buffer list that haven’t had activity in 10 minutes or less to keep the clutter down.

/set buffers.look.detach 600

Hide joins and parts don’t count as ‘activity’ on a channel.
some filter tweaks to make IRC less annoying.

/set weechat.look.buffer_notify_default message

Hide joins/parts from users that haven’t spoken recently (default is 5 minutes). You can toggle this with ctrl-=.

/set irc.look.smart_filter on
/filter add irc_smart * irc_smart_filter *

When joining a channel, a lot of information is spit out, most of which is redundant. Hide all this with the following command.

/filter add irc_join_names * irc_366,irc_332,irc_333,irc_329,irc_324 *

Some visual bits to make Weechat a bit nicer. Like replace the nickname for a message with a little arrow if the previous message in your buffer was from the same user.

/set weechat.look.prefix_same_nick "⤷"
/set weechat.look.prefix_error "⚠"
/set weechat.look.prefix_network "ℹ "
/set weechat.look.prefix_action "⚡"
/set weechat.look.bar_more_down "▼▼"
/set weechat.look.bar_more_left "◀◀"
/set weechat.look.bar_more_right "▶▶"
/set weechat.look.bar_more_up "▲▲"
/set weechat.look.prefix_suffix "|"
/set weechat.look.separator_horizontal "—"

Prettier timeformat

/set weechat.look.buffer_time_format "${color:252}%H${color:245}:%M${color:240}:%S"

Restrict the length of nicks in the sidebar to 15 characters

/set weechat.look.prefix_align_max 15

Bind alt-n to toggle the nicklist.

/key bind meta-n /bar toggle nicklist

More colors

/set weechat.color.chat_delimiters 29
/set weechat.color.chat_host 24
/set weechat.color.chat_prefix_suffix 24
/set weechat.color.nicklist_away 244
/set weechat.color.separator 60
/set weechat.color.chat_highlight_bg 058
/set irc.color.reason_quit  244
/set irc.color.topic_new  36
/set irc.color.topic_old 244

Get rid of the title bar and just add the buffer name to the input bar

/bar hide title
/bar set input items [time]+ buffer_name+ [input_prompt]+(away),[input_search],[input_paste],input_text

Adjust the input bar

/bar set input items [input_prompt]+(away)::,[input_search],[input_paste],input_text

Save your configuration with

/save

Save layout and buffer order

/layout store

My WeeChat theme is like this

Simple WeeChat setup
Simple WeeChat setup

Relay

You can use WeeChat also with other interfaces than ncurses like web frontends which make using IRC easier on mobile devices.

Add a relay server using the weechat protocol, using port 40900 and setting a password.

/relay add weechat 40900
/set relay.network.password YOURPASSWORD

Now you can connect to your WeeChat with web frontend by using the server where your WeeChat is running and providing the port and password.

Note: Connections with weechat are not encrypted. So it’s better to e.g. setting up WeeChat relay behind nginx.

Clicking on long URLs

By default, WeeChat displays time and prefix for each line and optional bars around chat area which makes clicking long URLs impossible.

To make easier URL click, you have to options as explained in FAQ:

1. Enable option “eat_newline_glitch”, so that new line char is not added at the end of each line displayed (it will not break URL selection):

/set weechat.look.eat_newline_glitch on

2. Move nicklist to top and remove alignment on nick:

/set weechat.bar.nicklist.position top
/set weechat.look.prefix_align none
/set weechat.look.align_end_of_lines time

I found the first option to be better as it lets you to have the prefix after nick and before text. It’s not as good as Irssi has it but it works. I hope they’ll merge this pull request which should make it better.

You can also use the bare display (default key: Alt+l).

Other notes

Joining channels with channel names on different character set, like ISO8859-1 and umlauts. You need to input the octal code for the letter, for example latin small letter a and o with diaeresis:

/eval /join #${\0344}${\0344}kk${\0366}set

Search text in buffer

The default key is Ctrl+r (command is: /input search_text_here). And jump to highlights: Alt+p / Alt+n.

Log all messages on IRC buffers but not join/part/quit messages:

  • All IRC buffers: /set logger.level.irc 3
  • Server and its channels: /set logger.level.irc.freenode 3
  • Specific channel: /set logger.level.irc.freenode.#weechat 3

Configuration with iset

Now you should have basic setup quite right and to continue configurations it’s nice to use iset plugin. Just type /iset to enter the iset screen. You will now see a list of all the parameters which can be modified. If you type something in the input bar, it will look for the pattern in the list of variables. If you want to search through the values, put an = before the pattern.

To change the value, press Alt + Enter then enter the new value (it is possible to navigate through values depending on variable type by pressing the Tab key).

Summary

After short use WeeChat works as well as Irssi and vice versa. It will be seen if I stuck with it or get back to Irssi. I’m not quite satisfied with the configuration but it works well enough.

Essential IntelliJ IDEA keyboard shortcuts

Recently I switched from using Eclipse to IntelliJ IDEA as our Java EE application’s front-end was done with JavaScript and the support for front-end technologies in Eclipse is more or less non-existent. The switch for long time Eclipse user wasn’t easy as IDEA works a bit differently but the change was worth it. The biggest difference in daily work with IDE is the shortcuts which are quite different in IDEA. In theory you can use Eclipse keymap for shortcuts but it just doesn’t work like it should and in practice you have to learn the IDEA way. There are many posts in the Internet about keyboard shortcuts in IDEA but there’s always place for more :) So, here’s my list of shortcuts to keep in your finger memory.

Learn keymap with Key Promoter

To learn your way around IntelliJ IDEA’s keyboard shortcuts there’s nice “Key Promoter” plugin to train yourself. It prompts whenever you use the mouse when you could’ve used the keyboard instead (similar to Eclipse’s Mousefeed).

To install the plugin:

  1. Ctrl+Alt+S to pull up the Settings screen
  2. Filter on “plugin”. Click “Plugins”, then “Browse Repositories” at the bottom
  3. Filter on “key promoter”
  4. Double click to install
  5. Essential IntelliJ IDEA keyboard shortcuts

    IntelliJ IDEA keymap

    You may be tempted to just go with the Eclipse keymap but it’s better to learn the IDEA way although it’s quite irritating at start. You also should change some default IDEA keyboard shortcuts to better ones like “closing editor window” with Ctrl+F4 which is too cumbersome compared to the de facto Ctrl+W. And changing “comment current line or selection” with Ctrl+/ which is impossible with Finnish keyboards to Ctrl+7.

    If you want to know how Eclipse shortcuts map to IDEA there’s nice post about IntelliJ IDEA shortcuts for Eclipse users and I added some in my list.

    Recent Viewed or edited Files: CTLR + E / CTRL + SHIFT + E
    Shows you a popup with all the recent files that you have opened or actually changed in the IDE. If you start typing, you can filter the files.

    Go to Class or file: CTRL + N and CTRL + Shift + N
    Allows you to search by name for a Java file in your project. If you combine it with SHIFT, it searches any file. Adding ALT on top of that it searches for symbols. (Eclipse: Ctrl+Shift+T and Ctrl+Shift+R)

    Find and Replace in Path: CTRL + SHIFT + F / CTRL + SHIFT + R
    Allows you to find in files or replace in files. (Eclipse: Ctrl+H)

    Incremental Find: F3 / CTRL + L
    When using CTLR-F to find in current file the F3 lets you to loop through the results. (Eclipse: Ctrl+K)

    Delete line: CTRL + Y
    Delete current line under cursor. Yank it. (Eclipse: Ctrl+D)

    Goto line: CTRL + G
    Go to given line number. (Eclipse: Ctrl-L)

    Syntax Aware Selection: CTRL + W
    Allows you to select code with context. Awesome when you need to select large blocks or just specific parts of a piece of code.

    Complete Statement: CTRL + SHIFT + ENTER
    This will try to complete your current statement. How? By adding curly braces, or semicolon and line change.

    Smart Type Completion: CTRL + SHIFT + SPACE
    Like auto complete (CTRL + SPACE) but if you add a SHIFT you get the smart completion. This means that the IDE will try to match expected types that suit the current context and filter all the other options.

    Reformat source code and optimize imports: CTRL + ALT + L
    Allows you to reformat source code to meet the requirements of your code style. Lays out spacing, indents, keywords etc. Reformatting can apply to the selected text, entire file, or entire project.

    Quick Fix: Alt+Enter
    (Eclipse: Ctrl+1)
    Gives you a list of intentions applicable to the code at the caret.

    Paste one of the previous values from clipboard: CTRL + SHIFT + V
    Shows you a dialog to select previous value from the clipboard to be pasted.

    Comment or uncomment line or block: Ctrl+7 / Ctrl+Shift+7
    Allows you to comment or uncomment the current line or selected block of source code. This is originally Ctrl + / (Slash) which is impossible with Finnish keyboard layouts.

    Show Diff (in Changes): CTRL + D
    In Changes tab compares the file with latest repository version.

    Highlight Usages: CTRL + SHIFT + F7
    Place the cursor in a element and after pressing the cursor the IDE will highlight all the occurrences of the selected element.

    Go to Declaration: CTRL + B
    If you place the cursor in a class, method or variable and use the shortcut you will immediately jump to the declaration of the element.

    Navigate Between Methods: ALT + UP/DOWN Arrows
    Jump between methods.

    Refactoring String Fragments: CTRL + ALT + V
    Refactor hardcoded string into variable/field/constant. Select the section of the String you want to extract, and use the normal “Extract…” shortcuts to extract it into a variable.

    Other useful keyboard shortcuts

    There are many useful keyboard shortcuts and you can print them from Help > Default Keymap Reference. Here are some more shortcuts which are also handy.

    Update application: CTRL + F10
    Current file structure: CTRL + F12
    Bookmarks: F11 and SHIFT + F11
    Generate Getters & Setters (in editor): ALT + INSERT
    Create New _* (in project navigator): ALT + INSERT
    Refactor – Rename: SHIFT + F6
    Open Settings CTRL + Alt + S
    Duplicate line: CTRL + D
    Move line: CTRL + ALT + UP / DOWN
    Find Command: CTRL + SHIFT + A
    Show usages in a pop-up list: CTRL + Alt + F7
    Extract Variable/Method/Constant/Field: CTRL + ALT + V/M/C/F
    Quick JavaDoc Popup: CTRL + Q
    Tab switcher: CTRL + TAB
    Jump to Project Navigator: ALT + 1
    Jump back to last tool window/panel: F12
    Jump to beginning/end of block (e.g., method start/end): CTRL + [ and CTRL + ]
    Toggle uppercase/lowercase of selection: CTRL + SHIFT + U
    Toggle collapse/expand: CTRL + .
    Toggle full screen editor (hide other tool windows): CTRL + SHIFT + F12

    Not a keyboard shortcut exactly but the “iter” smart template is great. If you want to iterate though something using a for loop type “iter” then TAB to use the live template. It will figure out the most likely variable you want to iterate over and generate a for loop for it. In Eclipse it worked more logically with just typing for and then autocomplete.

Transferring Linux install media to USB thumb drive

Optical drives are nowadays less common on laptops which makes making bootable install medias a little easier. Yes, easier by requiring you to use USB thumb drives. There are different ways to achieve what you want and here is couple of examples how to transfer Linux install media to USB thumb drive on Windows and Mac OS X.

First you need to download the install media of your choice as ISO image which you want to transfer to your USB thumb drive. In theory you need a media which is meant for USB thumb drive or is a Hybrid ISO image but practically all modern distributions’ media are Hybrid ISO images.

Windows

Plug in the USB thumb drive and format it with FAT32 file system

Download UNetbootin and transfer ISO content to USB thumb drive using UNetbootin.

  1. Select Diskimage and click the “. . .” button to find and open the ISO file. Make sure Type is USB drive and the Drive letter is what your Windows detect and click OK. Your ISO content will be copied to USB thumb drive.
  2. Next you also need to copy the ISO file that you used with UNetbootin to the root of the USB thumb drive, e.g. the ISO file should be F:\linuxmint-16-cinnamon-dvd-64bit.iso

Now you can proceed to boot up the machine that you want to install with e.g. Linux Mint 16.

Linux

Creating bootable USB thumb drive in Linux is much simpler. You just need install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

You just need to have the ISO image and one command:

$ sudo dd bs=4M if=linux-image.iso of=/dev/sdX oflag=direct

Where sdX is the device where your USB thumb drive is, e.g. sdb. If your system doesn’t support ‘oflag=direct’, you can just leave it out as it is simply intended to speed up the process a bit. If you leave the bs-option off the default will be 512 bytes and takes forever.

Mac OS X

OS X’s DiskUtil can’t burn ISO images to USB and gives “Invalid argument” but fortunately just like with Linux also Mac OS X makes things easy for you. You just need the install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

$ dd if=linux-image.iso of=/dev/rdiskX bs=8192

Where rdiskX is the device where your USB thumb drive is, e.g. rdisk1. You can see the device (disk identifier) from DiskUtility.app by selecting your USB stick and checking the Device section.

It’s useful to notice that on OS X you can use the ‘/dev/rdiskX’ device which stands for raw disk and is much faster than the block level device ‘/dev/diskX’ which is buffered and where data undergoes extra processing. For example rdiskX gives “823132160 bytes transferred in 625.470137 secs (1316022 bytes/sec)” and diskX gives “823132160 bytes transferred in 1296.675998 secs (634802 bytes/sec)”.

If you get an error e.g. “dd: /dev/rdisk2: Resource busy” then you have to make sure the Mac OS X isn’t using it and thus unmount it’s partitions. Open the DiskUtility.app, and on your USB hard drive, unmount any of it’s partitions. Do not eject the USB hard drive.

Monitoring Java EE application with JavaMelody

Software development is much more than just coding application by requirements and deploying it to production as the real work really starts after it has been shipped: maintenance, improvements and problem solving. And for that it’s good to have some data. It’s said “if you can’t measure it, you can’t improve it” and in ideal situations you plan for measuring your app but often you have to do it in production to detect and diagnose performance problems. There are different ways to measure a Java EE application and one good tool to achieve performance and monitoring statistics is Javamelody which has low overhead, is non-intrusive, informative and simple to setup but still very capable.

“If you can not measure it, you can not improve it.” – Lord Kelvin

Application performance can be measured by two main methods: computational resources used by the application and the performance as seen by a user of the application. By measuring these quantities we get an empirical performance baseline of the application which then can used to detect changes in performance. By using performance monitoring, which is an act of non-intrusively collect or observe performance data from running application, we get measurements to identify or isolate potential issues from real operation of an application without having a severe impact on runtime responsiveness or throughput.

There are different ways to achieve performance and monitoring statistics and it’s useful to plan for measuring application’s performance while still in development as later on you surely want to know how it is performing. You can do it by leveraging different JMX-based tools like Metrics and Servo but if you didn’t really plan for it you still have choices. You can use tools like standard JDK tools (jconsole, jstat, jmap, jstack, hprof) for low level JVM monitoring or some monitoring application like Javamelody. And as time is always scarce in development it’s useful to deploy a tool like JavaMelody which provide quick and easy access to performance monitoring. But if you are interested of some narrow focus measurement you’re better of with some custom tool.

Monitoring with JavaMelody

JavaMelody is an open source (LGPL) application to monitor Java or Java EE application servers in QA and production environments. It is a tool to measure and calculate statistics on real operation of an application depending on the usage of the application by users and is mainly based on statistics of requests and on evolution charts which can be viewed on the current day, week, month, year or custom period. The statistics can be viewed on a HTML page and sent as PDF reports by email.

JavaMelody lists the following use cases:

  • It allows to improve applications in QA and production
  • Give facts about the average response times and number of executions
  • Make decisions when trends are bad, before problems become too serious
  • Optimize based on the more limiting response times
  • Find the root causes of response times
  • Verify the real improvement after optimization

Setting up

Setting up JavaMelody is quite easy and the needed steps are covered in User guide. Integrating JavaMelody to your application can be done in less than 10 minutes, by automatic discovery of environment: it only requires to copy 2 jar files and to add 10 lines in a xml file. But for more detailed and wider measurements you need couple more lines to xml files and here are the changes I made for our Wicket, Spring, JPA, Hibernate -Java EE project.

JavaMelody and dependencies

Getting JavaMelody and it’s dependencies is easy with Maven2 and you just need to add javamelody-core and if you want to have PDF reports also iText in your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!-- javamelody-core -->
<dependency>
	<groupId>net.bull.javamelody</groupId>
	<artifactId>javamelody-core</artifactId>
	<version>1.45.0</version>
</dependency>
<!-- itext, option to add PDF export -->
<dependency>
	<groupId>com.lowagie</groupId>
	<artifactId>itext</artifactId>
	<version>2.1.7</version>
	<exclusions>
		<exclusion>
			<artifactId>bcmail-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bcprov-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bctsp-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
	</exclusions>
</dependency>

Monitoring application

JavaMelody needs just a monitoring filter before the description of webapp’s servlet in WEB-INF/web.xml or not even that as described in the User guide. But usually you want a little bit more than the minimum 10 additional lines in web.xml.

By adding the customResourceFilter you get customized look & feel, you can exclude some urls from statistics with url-exclude-pattern regular expression pattern, eamil-parameters are for weekly reports and by including monitoring-spring.xml you can monitor DataSource with a Spring post-processor and objects initialized with Spring.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
...
<context-param>
	<param-name>contextConfigLocation</param-name>
	<param-value>
		classpath:net/bull/javamelody/monitoring-spring.xml
		classpath:applicationContext-service.xml
		classpath:applicationContext-persist.xml
		classpath:applicationContext-web.xml
		classpath:applicationContext-security.xml
    </param-value>
</context-param>
 
<!-- Must be defined before javamelody filters -->
<filter>
	<filter-name>springSecurityFilterChain</filter-name>
	<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
	<filter-name>springSecurityFilterChain</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
 
<!--====================== Monitoring ===================================-->
<!-- Custom CSS -->
<filter>
	<filter-name>customResourceFilter</filter-name>
	<filter-class>net.bull.javamelody.CustomResourceFilter</filter-class>
	<init-param>
		<param-name>monitoring.css</param-name>
		<param-value>/styles/monitoring.css</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>customResourceFilter</filter-name>
	<url-pattern>/monitoring</url-pattern>
</filter-mapping>
<!-- Monitor filter -->
<filter>
	<filter-name>monitoring</filter-name>
	<filter-class>net.bull.javamelody.MonitoringFilter</filter-class>
	<init-param>
		<param-name>storage-directory</param-name>
		<param-value>logs/monitoring</param-value>
	</init-param>
	<init-param>
		<param-name>url-exclude-pattern</param-name>
		<param-value>(/images/.*|/js/.*|/styles/.*)</param-value>
	</init-param>
	<init-param>
		<param-name>admin-emails</param-name>
		<param-value>first.last@example.com</param-value>
	</init-param>
	<init-param>
		<param-name>mail-session</param-name>
		<param-value>MailSession</param-value>
	</init-param>
	<init-param>
		<param-name>mail-periods</param-name>
		<param-value>week,month</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>monitoring</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
<listener>
	<listener-class>net.bull.javamelody.SessionListener</listener-class>
</listener>
...

Spring Security config

For restricting the access to monitoring statistics I added the following to the Spring Security’s applicationContext-security.xml. There are also other options for security. The key point here is to make sure, that the monitoring-filter in web.xml (above) is defined after the Spring Security filter chain.

1
2
3
4
5
6
<http auto-config="true" use-expressions="true">
    ...
    <intercept-url pattern="/monitoring/**" access="hasRole('ROLE_ADMIN')" />
	<intercept-url pattern="/**"
			access="hasAnyRole('ROLE_ADMIN, 'ROLE_USER'')" />
</http>

Monitoring SQL and datasources

For monitoring datasources and SQL I just added jndi-lookup to applicationContext-persist.xml. Other options to enable JDBC monitoring are described in the User guide.

1
2
3
4
5
6
7
...
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myAppDS"/>
 
<bean id="entityManagerFactory"
	class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
	<property name="dataSource" ref="dataSource" />
...

Business facades (Spring)

If the application to monitor contains some objects initialized by Spring, EJB or Guice their methods execution can also be added to statistics. As described in the User guide for monitoring Spring Business facades there are couple of options and for example with JdkRegexpMethodPointcut in applicationContext-web.xml you can catch objects with regular expression like “all that have Service in their names”.

1
2
3
4
5
6
7
8
9
...
<bean id="facadeMonitoringAdvisor" class="net.bull.javamelody.MonitoringSpringAdvisor">
	<property name="pointcut">
		<bean class="org.springframework.aop.support.JdkRegexpMethodPointcut">
			<property name="pattern" value=".*Service.*" />
		</bean>
	</property>
</bean>
...

EHCache statistics

If you want to see also EHCache statistics add statistics=”true” to ehcache.xml config file.

1
2
3
4
5
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
    <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" statistics="true" />
    <cache name="fooCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
    <cache name="barCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
</ehcache>

Database information and statistics

JavaMelody already shows datasource and SQL information but it’s also possible to display information and statistics on the database like the longest requests in cumulative time with display of the cpu time and of the elementary cost. For showing that information the user in the database used by the monitored application must have the necessary rights to read those information and statistics.

In Oracle database the request practically means select * from v$session and you can grant access as ‘system’ user with GRANT SELECT any dictionary TO myapplicationuser. Strangely granting just the select for v$session (GRANT SELECT ON sys.v_$session TO myapplicationuser;) wasn’t enough.

Reports

There is also a possibility to get weekly, daily or monthly report in pdf format sent by email to one or several people. It needs iText library for webapp and Java’s JavaMail and Activation libraries in your server for the mail session.

With WebLogic you add some email parameters in webapp’s web.xml (above) and configure a Mail Session in WebLogic AdminServer: Services > Mail Sessions with JNDI Name and JavaMail properties. The report provides the same information you can find in monitoring web page with high and detailed level information.

Statistics’ storage

Javamelody data is stored in files on disk and there are 2 types of files: *.rrd files for values in graphics in RRD format (using jrobin library) and *.ser.gz for values in statistics.

The performance statistics are stored to temp/javamelody directory by default which means /tmp/javamelody in Linux and in Windows when running webapp in Eclipse something like Users/developer/AppData/Local/Temp/javamelody. If you want to reset the counters just delete all the files. The location can be changed with storage-directory parameter and if the name of the directory starts with ‘/’, it is considered as an absolute path, otherwise it is considered as relative to the temporary directory.

There is also option to use centralized collect server to store statistics and for monitoring of several applications.

Viewing performance monitoring

After you have set up your webapp to have the monitoring you can see statistics at URL like http:////monitoring depending your configuration.

Some example JavaMelody statistics from development:

Overhead

Monitoring and filtering doesn’t come without costs and there has been discussions on JavaMelody wiki about the overhead the monitoring does. It is said that the overhead is so low that it can be enabled continuously in Quality Assuarance environment and if no problem arises in QA, also continuously in production environment. And with little overhead you will be able to know what needs optimizing in the QA or production servers so that the overhead of JavaMelody will practically be negative.

The discussion contained some notes:

  • Architecture of JavaMelody is lightweight so it has a lower overhead as compared it to other available solutions
  • It is only statistics and not events so the overhead of memory is quite minimal.
  • It does monitoring not profiling: there is no instrumentation of classes and instead “interceptors” for http, jdbc, spring or ejb3.
  • No database and no recording of each events even in a file or over the wire: only statistics of requests are kept. The overhead of cpu is minimal with no I/O on the wire and minimal I/O on disk (just to take a backup of statistics at a regular interval). The overhead of some other good monitoring solutions is in the recording of each event in a database or in a master server.
  • You have the choice to use centralized collect server which unloads the memory, the backup storage and the generation of reports to another server while adding I/O on the wire for sending deltas of the statistics.

Summary

“If you can not measure it, you can not improve it.” It doesn’t matter how you do it but if you want to use real user data and environment just do it non-intrusively with performance monitoring tools like Javamelody.

Eclipse and Maven Console

Eclipse 3.7 Indigo has integrated Maven m2e plugin but is missing some expected functionality which was previously present in Sonatype releases by default. If you want your Maven Console to show something you must also install the optional “m2e – slf4j over logback logging” plugin.

When installing the m2e plugin there is an optional feature “m2e – slf4j over logback logging” which is needed for the Maven Console to work. Without it the plugin produces no output to Eclipse’s Maven Console view so that it is impossible to track plugin’s activity (background maven builds, source and javadoc downloads, etc).

There is a bug filed about it but it is resolved with comment “As a tool, m2e is not in the position to impose any specific slf4j logging backend on the host Eclipse installation.” That seems kinda strange as without the optional component the plugin is missing useful parts.

Just “Install new sofware > Indigo > Collaboration > “m2e – slf4j over logback logging (Optional)” and your Maven Console is back in business.