Web analytics with Piwik: keeping control over your own data

Web analytics is one the essential tools for a website and including measuring web traffic and getting information about the number of visitors it can be also used as a tool to assess and improve the effectiveness of a website. The most common way to collect data is to use on-site web analytics, measure a visitor’s behavior once on your website, with page tagging technology like on Google Analytics which is widely used web analytics service. But what would you use if you want to keep control over your own data?

You don’t have to look far as the only open source web analytics application is Piwik which aims to be the ultimate open alternative to Google Analytics. Here’s a short overview to Piwik Analytics and how to get started with it.

“Web analytics is the measurement, collection, analysis and reporting of web data for purposes of understanding and optimizing web usage.” – Wikipedia

Piwik Open Analytics Platform

Piwik is web analytics application which tracks online visits to one or more websites and displays reports on these visits for analysis. In short it aims to be the ultimate open source alternative to Google Analytics. The code is GPL v3 licensed and available in GitHub. In technical side Piwik is written in PHP, uses MySQL database and you can host it by yourself. And if you don’t want to setup or host Piwik yourself you can also get commercial services.

Piwik provides the usual features you would expect from a web analytics application. You get reports regarding the geographic location of visits, the source of visits, the technical capabilities of visitors, what the visitors did and the time of visits. Piwik also provides features for analysis of the data it accumulates such as saving notes to data, goals for actions, transitions for seeing how visitors navigate, overlaying analytics data on top of a website and displaying how metrics change over time. The easiest way to see what it has to offer is to check the Piwik online demo.

Feature highlights

You might ask how Piwik differs from other web analytics applications such as Google Analytics? One principle advantage of using Piwik is that you are in control. You can host Piwik on your own server and the data is tracked inside your MySQL database: you’ve full control over your data. Software as a service analytics applications on the other hand, have full access to the data users collect. Data privacy is essential for public sector and enterprises who can’t or don’t want to share it for example with Google. You ensure that your visitors behavior on your website is not shared with advertising companies.

Other interesting feature is that it provides advanced privacy options: ability to anonymize IP addresses, purge tracking data regularly (but not report data), opt-out support and Do Not Track support. Your website visitors can decide if they want to be tracked.

You can also do scheduled reports which are sent by e-mail, import data from web server logs, use the API for accessing reports and administrative functions and Piwik also has mobile app to access the analytics data. Piwik is also customizable with plugins and you can integrate it with WordPress and other applications.

Piwik’s User Interface

Piwik has clean and simple user interface as seen in the following screenshots (taken from the online demo).

Piwik main view
Piwik main view

Piwik visitors overview
Piwik visitors overview

Setting up Piwik

Setting up Piwik is easy and there’s good documention available for running Piwik web analytics. All you need is web server like Nginx, PHP 5.5 and MySQL or MariaDB. You can setup it manually but the most easiest way to start with it is to use the provided Docker image and docker-compose. The docker-compose file setups four containers (MySQL, Piwik, Nginx and Cron) and with compose you can start it up. The Piwik image is available from official docker-library.

The alternative is to do your own Docker image for Piwik and related services. In my opinion it makes sense to have just two containers: one for Piwik related web stuff and other for MySQL. The Piwik container runs Piwik, Nginx and Cron script with e.g. supervisor. The official image uses Debian (from PHP) but Piwik runs nicely also on Alpine Linux. One thing to tinker with when using Docker is to get MySQL access to Piwik’s assets for LOAD DATA INFILE which will greatly speed Piwik’s archiving process.

If you’re setting up Piwik manually you can watch a video of installation and after that a video of configuring settings. After you’re done with the 5 minute installation you get the JavaScript tag which you add to the bottom of each page of your website. If you’re using React there’s Piwik analytics component for React Router. Piwik will then record the activity across your website within your database.

And that’s about all there is to starting with Piwik. Simple setup with Docker or doing it manually, adding the JavaScript tag, configuring some options if needed and then just wait for the data from visitors.


Piwik is good and feature rich alternative for web analytics application. Setting it up isn’t as straightforward as using some hosted service as Google Analytics but that’s the way self-hosted services always are. If you need web analytics and want to keep control of your own data and don’t mind hosting it yourseld and paying for the server then Piwik is a good choice.

Starting with WeeChat

I’ve been using IRC for some time and although Irssi has served me well, it’s time to try something different. WeeChat is a modular chat client with support for IRC and the interesting part is that it’s possible to use other interfaces like glowing-bear web frontend. WeeChat is similar to Irssi so switching over shouldn’t be an issue. But to get the configuration right and what you had on Irssi needs some effort. Here are my notes about starting with WeeChat and how I like my chat client to look.

Compiling Weechat on CentOS 6

I have my shell on CentOS and although you can find WeeChat from the repositories, it’s quite old (0.4.3 when 1.4. is the newest). So you might want to compile WeeChat by yourself. Compiling WeeChat is explained on the User Guide. You need to install some libraries before you can try using make.

yum install gettext make libgcrypt-devel libcurl-devel python-devel v8-devel gnutls-devel ruby cppunit perl-ExtUtils-Embed

After you’ve installed the needed packages download WeeChat sources and extract the weechat-1.4.tar.gz package to directory you want.

Go to the directory you extracted WeeChat and run the following commands:

$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/path/to/directory
$ make
$ make install

Configuring WeeChat

Quick Start guide helps you to get started so I don’t duplicate that here. If you’re familiar with Irssi you should feel more or less at home. As I didn’t use much scripts in Irssi the most difficult part for me was to create as good theme as I had with Irssi. Otherwise the switch went better than expected.

Start WeeChat with weechat so we can start configuration.

When I started with WeeChat of course I googled how others had configured it and thus my configuration is based on this and some other snippets.

Scripts to improve functionality

/script install buffer_autoclose.py iset.pl colorize_nicks.py urlbuf.py screen_away.py
  • buffer_autoclose.py: Automatically close inactive private message buffers.
  • iset.pl: Interactive Set for configuration options
  • colorize_nicks.py: Use the weechat nick colors in the chat area and command line.
  • urlbuf.py: Common buffer for received URLs.
  • irssi_awaylog.py: Log highlights/private messages when you are away.
  • screen_away.py: Set away status when detaching and attaching from screen or tmux.

To see and set the options for the plugins you can use /set with wildcard “*”

/set *urlbuf*
/set *away*

You can install scripts also by typing “script search iset” and a selection of available plugins will appear. To leave it type “q” then press enter, if you want to install the script type “i” then press enter.

Adjust layout and colors

Make the title bar and the status bar using dark colors.

/set weechat.bar.status.color_bg 0
/set weechat.bar.title.color_bg 0
/set buffers.color.current_bg 0
/set weechat.color.chat_nick_colors 1,2,3,4,5,6
/set buffers.color.hotlist_message_fg yellow

Place a list of all buffers at the top of the screen

/set weechat.bar.buffers.position top

Merge the server buffers in to one to take up less of our precious screen space (you can switch between them with ctrl-x).

/set buffers.look.hide_merged_buffers server

Hide any channels from the buffer list that haven’t had activity in 10 minutes or less to keep the clutter down.

/set buffers.look.detach 600

Hide joins and parts don’t count as ‘activity’ on a channel.
some filter tweaks to make IRC less annoying.

/set weechat.look.buffer_notify_default message

Hide joins/parts from users that haven’t spoken recently (default is 5 minutes). You can toggle this with ctrl-=.

/set irc.look.smart_filter on
/filter add irc_smart * irc_smart_filter *

When joining a channel, a lot of information is spit out, most of which is redundant. Hide all this with the following command.

/filter add irc_join_names * irc_366,irc_332,irc_333,irc_329,irc_324 *

Some visual bits to make Weechat a bit nicer. Like replace the nickname for a message with a little arrow if the previous message in your buffer was from the same user.

/set weechat.look.prefix_same_nick "⤷"
/set weechat.look.prefix_error "⚠"
/set weechat.look.prefix_network "ℹ "
/set weechat.look.prefix_action "⚡"
/set weechat.look.bar_more_down "▼▼"
/set weechat.look.bar_more_left "◀◀"
/set weechat.look.bar_more_right "▶▶"
/set weechat.look.bar_more_up "▲▲"
/set weechat.look.prefix_suffix "|"
/set weechat.look.separator_horizontal "—"

Prettier timeformat

/set weechat.look.buffer_time_format "${color:252}%H${color:245}:%M${color:240}:%S"

Restrict the length of nicks in the sidebar to 15 characters

/set weechat.look.prefix_align_max 15

Bind alt-n to toggle the nicklist.

/key bind meta-n /bar toggle nicklist

More colors

/set weechat.color.chat_delimiters 29
/set weechat.color.chat_host 24
/set weechat.color.chat_prefix_suffix 24
/set weechat.color.nicklist_away 244
/set weechat.color.separator 60
/set weechat.color.chat_highlight_bg 058
/set irc.color.reason_quit  244
/set irc.color.topic_new  36
/set irc.color.topic_old 244

Get rid of the title bar and just add the buffer name to the input bar

/bar hide title
/bar set input items [time]+ buffer_name+ [input_prompt]+(away),[input_search],[input_paste],input_text

Adjust the input bar

/bar set input items [input_prompt]+(away)::,[input_search],[input_paste],input_text

Save your configuration with


Save layout and buffer order

/layout store

My WeeChat theme is like this

Simple WeeChat setup
Simple WeeChat setup


You can use WeeChat also with other interfaces than ncurses like web frontends which make using IRC easier on mobile devices.

Add a relay server using the weechat protocol, using port 40900 and setting a password.

/relay add weechat 40900
/set relay.network.password YOURPASSWORD

Now you can connect to your WeeChat with web frontend by using the server where your WeeChat is running and providing the port and password.

Note: Connections with weechat are not encrypted. So it’s better to e.g. setting up WeeChat relay behind nginx.

Clicking on long URLs

By default, WeeChat displays time and prefix for each line and optional bars around chat area which makes clicking long URLs impossible.

To make easier URL click, you have to options as explained in FAQ:

1. Enable option “eat_newline_glitch”, so that new line char is not added at the end of each line displayed (it will not break URL selection):

/set weechat.look.eat_newline_glitch on

2. Move nicklist to top and remove alignment on nick:

/set weechat.bar.nicklist.position top
/set weechat.look.prefix_align none
/set weechat.look.align_end_of_lines time

I found the first option to be better as it lets you to have the prefix after nick and before text. It’s not as good as Irssi has it but it works. I hope they’ll merge this pull request which should make it better.

You can also use the bare display (default key: Alt+l).

Other notes

Joining channels with channel names on different character set, like ISO8859-1 and umlauts. You need to input the octal code for the letter, for example latin small letter a and o with diaeresis:

/eval /join #${\0344}${\0344}kk${\0366}set

Search text in buffer

The default key is Ctrl+r (command is: /input search_text_here). And jump to highlights: Alt+p / Alt+n.

Log all messages on IRC buffers but not join/part/quit messages:

  • All IRC buffers: /set logger.level.irc 3
  • Server and its channels: /set logger.level.irc.freenode 3
  • Specific channel: /set logger.level.irc.freenode.#weechat 3

Configuration with iset

Now you should have basic setup quite right and to continue configurations it’s nice to use iset plugin. Just type /iset to enter the iset screen. You will now see a list of all the parameters which can be modified. If you type something in the input bar, it will look for the pattern in the list of variables. If you want to search through the values, put an = before the pattern.

To change the value, press Alt + Enter then enter the new value (it is possible to navigate through values depending on variable type by pressing the Tab key).


After short use WeeChat works as well as Irssi and vice versa. It will be seen if I stuck with it or get back to Irssi. I’m not quite satisfied with the configuration but it works well enough.

Creating Vagrant Base Box with Veewee

Vagrant is a great tool for creating and configuring lightweight, reproducible, portable virtual machine environments but the first step for using Vagrant, downloading an existing “base box”, raises some questions. E.g. How are these unverified boxes built? So, you might end up building your own base box which is often time consuming and cumbersome. Fortunately there’s a tool called Veewee which aims to automate all the steps for building base boxes and to collect best practices in a transparent way.

Vagrant Base Box with Veewee

Veewee is a tool for easily (and repeatedly) building custom Vagrant base boxes, KVMs, and virtual machine images. You can use it to build a Vagrant box in Linux, Mac OS X and Windows but I found out that fulfilling the requirements on Windows is quite difficult (read Ruby and RVM) so just forget it.

To get you started there are some requirements you need to fulfill. First you’ll need to install at least one of the supported virtual machine providers like VirtualBox and second you need some development libraries.

On Ubuntu 15.04 Linux and using VirtualBox you need these packages:

$ apt-get install virtualbox git curl ruby ruby-dev libxslt1-dev libxml2-dev zlib1g-dev

Install RVM on Linux

For Ruby environment it’s recommended to use either rvm or rbenv. I chose the RVM and followed the RVM install documentation.

Install mpapis public key:

$ gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3

if keyserver fails, you can use $ curl -sSL https://rvm.io/mpapis.asc | gpg --import -

Install RVM stable with ruby:

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

Installing Veewee with RVM

With RVM already installed, ensure a ruby version that’s supported by Veewee is available on your machine:

$ source /home/marko/.rvm/scripts/rvm
$ rvm install ruby

Clone the veewee project from source:

$ cd <path_to_workspace>
$ git clone https://github.com/jedi4ever/veewee.git
$ cd veewee

Set the local gemset and ruby version within the current directory:

$ rvm use ruby@veewee --create

Run bundle install to install Gemfile dependencies for your local gemset:

$ gem install bundler
$ bundle install

Bundle install will take some time.

Building Vagrant Box with Veewee

Veewee uses definitions to build new virtual machines and ‘definition’ is derived from a ‘template’ and preconfigured templates are found in templates/ folder. Veewee Basics explains how you can create your own customized definition.

For my customized Vagrant Box I decided to use Tommy Muehle’s definition as a template as it contained what I wanted. Simple CentOS 6.6. Box with Puppet. I just changed the localization to Finland and made it bigger for WebLogic use case in mind. My definition for Vagrant Box can be found in GitHub.

To use my definition just clone the repository for CentOS 6.6 Box, copy the “centos-6.6-x86_64_puppet” folder to definitions/ folder under Veewee and make your own changes if needed. After you’re done run:

$ bundle exec veewee vbox build centos-6.6-x86_64_puppet

The build command runs Veewee scripts and automates the manual steps needed while installing a new Linux distribution.

Installing CentOS to Vagrant Box with Veewee

To export the Box for further use with Vagrant, run:

$ bundle exec veewee vbox export centos-6.6-x86_64_puppet

The above command is actually calling “vagrant package –base ‘centos-6.6-x86_64_puppet’ –output ‘boxes/centos-6.6-x86_64_puppet'”. The machine gets shut down, exported and will be packed in a centos-6.6-x86_64_puppet.box file inside the current directory.

And you’re all done. Now you can use your just created base box for Vagrant boxes. Import it into Vagrant’s box repository and use it to initialize a fresh project:

$ vagrant box add 'centos-6.6-x86_64_puppet' 'centos-6.6-x86_64_puppet.box'
$ vagrant init 'centos-6.6-x86_64_puppet'

Using Veewee to build a Vagrant Box is simple and what’s more important it’s automated and reproducible. Using Ruby and RVM on Windows 7 turned out to be practically impossible but old ThinkPad W510 with Ubuntu 15.04 worked nicely. Of course you could create a base box with Vagrant way which means installing and configuring your Linux manually. But why would you want to that if you can just automate it?

Setting up Bower and Gulp in Windows

Doing things manually is fine once but if you can automate things it’s better. With little tools you can speed up development and reduce recursive mundane tasks such as starting a project or setting up boilerplate code. I recently came across Bower which is a package manager for the web. With Bower you can fetch and install packages from all over, and it takes care of finding, downloading, and saving the stuff you’re looking for. The other interesting tool to help you get going is Gulp which enables you to automate and enhance your workflow. Let’s see how to put things together on Windows, nothing special just steps to get you started.

Gulp tasks

Install Git

Bower needs Git to work so first install Git if you don’t have it. I chose Git for Windows which gives you BASH emulation used to run Git from the command line, graphical user interface for using Git and Shell integration.

Just click through the installation.

Install Node.js

Bower depends on Node.js and NPM so you need to get Node.js. Just download the installation package from Node.js site and click through it.

Install Bower

After you have Node.js installed we can install Bower with npm. You might need to restart your Windows to get all the path variables setup so Npm can find them.

Open up the Git Bash or Command Prompt and Bower is installed globally by running the following command.

$ npm install -g bower

Once you have Bower installed you then can install packages and dependencies using these commands:

# Using a local or remote package
bower install <package>
# Using a specific version of a package
bower install <package>#<version>
# Search packages
$ bower search <package>

By default packages will be put in the bower_components directory which can be changed if you prefer. If you want your packages downloaded into js/libs you can achieve this by creating a .bowerrc file


    "directory": "js/libs"

You can also create a bower.json file which allows you to define the packages needed along with dependencies and then simply run bower install to download packages. In our example we setup a simple Backbone.js application which uses Bootstrap.


    "name": "Foobar",
    "version": "0.1.0",
    "dependencies": {
          "jquery": "~2.0.3",
          "underscore": "~1.5.0",
          "bootstrap": "~3.3.2",
          "backbone": "~1.1.2"

Our bower.json describes that we want some JavaScript libraries and as we have defined the version with ~ it can have bigger minor versions, e.g. jquery version can be between 2.0.3 < 2.1.0. Read more about semantic versioner for npm.

Now after creating that file inside the app directory you can run the following command:

$ bower install

After that you should see all your JavaScript packages under bower_components folder.

Install Gulp

To automate and enhance your workflow you can use Gulp for example to copy the files where you want them. There are nice recipes to show how to benefit of Gulp.

Install Gulp globally with npm:

$ npm install --global gulp

Install Gulp also in your project devDependencies:

$ npm install --save-dev gulp

Now we can setup our Gulp dependencies which pull from npm. Create a new package.json file in your project root and just add an empty object, {} and save it.

Next we install gulp-bower plugin which we can use to install Bower packages.

$ npm install --save-dev gulp-bower

This will install all the needed dependencies in a node_modules folder and also automatically update our package.json file with these dependencies.

Finally we need to setup the gulpfile.js which defines our tasks we want to perform. First we define what we installed in npm step above and create a config object to hold various settings. The bowerDir is just the path to the bower_components. Finally we add task for running bower and default task. Our bower tasks basically runs bower install but by including in the gulpfile other contributors only have to run gulp bower and have them all setup and ready.


var gulp = require('gulp'),
    bower = require('gulp-bower');
var config = {
     bowerDir: './bower_components'
gulp.task('bower', function() {
    return bower()
$ gulp.task('default', ['bower']);

The default task runs the bower task and all the user has to do to setup the needed packages is to run

$ gulp

In our case running gulp just runs our bower task which downloads the JavaScript packages we need. Pretty simple.

Gulp is powerful tool and has many use cases but also needs some to get all things working like you want and even then you might need to make compromises. One crafty task for Gulp and Bower is to customize your Bootstrap theme. Also Mark Goodyear has written good article about Getting started with gulp which shows some typical use cases.

Essential IntelliJ IDEA keyboard shortcuts

Recently I switched from using Eclipse to IntelliJ IDEA as our Java EE application’s front-end was done with JavaScript and the support for front-end technologies in Eclipse is more or less non-existent. The switch for long time Eclipse user wasn’t easy as IDEA works a bit differently but the change was worth it. The biggest difference in daily work with IDE is the shortcuts which are quite different in IDEA. In theory you can use Eclipse keymap for shortcuts but it just doesn’t work like it should and in practice you have to learn the IDEA way. There are many posts in the Internet about keyboard shortcuts in IDEA but there’s always place for more :) So, here’s my list of shortcuts to keep in your finger memory.

Learn keymap with Key Promoter

To learn your way around IntelliJ IDEA’s keyboard shortcuts there’s nice “Key Promoter” plugin to train yourself. It prompts whenever you use the mouse when you could’ve used the keyboard instead (similar to Eclipse’s Mousefeed).

To install the plugin:

  1. Ctrl+Alt+S to pull up the Settings screen
  2. Filter on “plugin”. Click “Plugins”, then “Browse Repositories” at the bottom
  3. Filter on “key promoter”
  4. Double click to install
  5. Essential IntelliJ IDEA keyboard shortcuts

    IntelliJ IDEA keymap

    You may be tempted to just go with the Eclipse keymap but it’s better to learn the IDEA way although it’s quite irritating at start. You also should change some default IDEA keyboard shortcuts to better ones like “closing editor window” with Ctrl+F4 which is too cumbersome compared to the de facto Ctrl+W. And changing “comment current line or selection” with Ctrl+/ which is impossible with Finnish keyboards to Ctrl+7.

    If you want to know how Eclipse shortcuts map to IDEA there’s nice post about IntelliJ IDEA shortcuts for Eclipse users and I added some in my list.

    Recent Viewed or edited Files: CTLR + E / CTRL + SHIFT + E
    Shows you a popup with all the recent files that you have opened or actually changed in the IDE. If you start typing, you can filter the files.

    Go to Class or file: CTRL + N and CTRL + Shift + N
    Allows you to search by name for a Java file in your project. If you combine it with SHIFT, it searches any file. Adding ALT on top of that it searches for symbols. (Eclipse: Ctrl+Shift+T and Ctrl+Shift+R)

    Find and Replace in Path: CTRL + SHIFT + F / CTRL + SHIFT + R
    Allows you to find in files or replace in files. (Eclipse: Ctrl+H)

    Incremental Find: F3 / CTRL + L
    When using CTLR-F to find in current file the F3 lets you to loop through the results. (Eclipse: Ctrl+K)

    Delete line: CTRL + Y
    Delete current line under cursor. Yank it. (Eclipse: Ctrl+D)

    Goto line: CTRL + G
    Go to given line number. (Eclipse: Ctrl-L)

    Syntax Aware Selection: CTRL + W
    Allows you to select code with context. Awesome when you need to select large blocks or just specific parts of a piece of code.

    Complete Statement: CTRL + SHIFT + ENTER
    This will try to complete your current statement. How? By adding curly braces, or semicolon and line change.

    Smart Type Completion: CTRL + SHIFT + SPACE
    Like auto complete (CTRL + SPACE) but if you add a SHIFT you get the smart completion. This means that the IDE will try to match expected types that suit the current context and filter all the other options.

    Reformat source code and optimize imports: CTRL + ALT + L
    Allows you to reformat source code to meet the requirements of your code style. Lays out spacing, indents, keywords etc. Reformatting can apply to the selected text, entire file, or entire project.

    Quick Fix: Alt+Enter
    (Eclipse: Ctrl+1)
    Gives you a list of intentions applicable to the code at the caret.

    Paste one of the previous values from clipboard: CTRL + SHIFT + V
    Shows you a dialog to select previous value from the clipboard to be pasted.

    Comment or uncomment line or block: Ctrl+7 / Ctrl+Shift+7
    Allows you to comment or uncomment the current line or selected block of source code. This is originally Ctrl + / (Slash) which is impossible with Finnish keyboard layouts.

    Show Diff (in Changes): CTRL + D
    In Changes tab compares the file with latest repository version.

    Highlight Usages: CTRL + SHIFT + F7
    Place the cursor in a element and after pressing the cursor the IDE will highlight all the occurrences of the selected element.

    Go to Declaration: CTRL + B
    If you place the cursor in a class, method or variable and use the shortcut you will immediately jump to the declaration of the element.

    Navigate Between Methods: ALT + UP/DOWN Arrows
    Jump between methods.

    Refactoring String Fragments: CTRL + ALT + V
    Refactor hardcoded string into variable/field/constant. Select the section of the String you want to extract, and use the normal “Extract…” shortcuts to extract it into a variable.

    Other useful keyboard shortcuts

    There are many useful keyboard shortcuts and you can print them from Help > Default Keymap Reference. Here are some more shortcuts which are also handy.

    Update application: CTRL + F10
    Current file structure: CTRL + F12
    Bookmarks: F11 and SHIFT + F11
    Generate Getters & Setters (in editor): ALT + INSERT
    Create New _* (in project navigator): ALT + INSERT
    Refactor – Rename: SHIFT + F6
    Open Settings CTRL + Alt + S
    Duplicate line: CTRL + D
    Move line: CTRL + ALT + UP / DOWN
    Find Command: CTRL + SHIFT + A
    Show usages in a pop-up list: CTRL + Alt + F7
    Extract Variable/Method/Constant/Field: CTRL + ALT + V/M/C/F
    Quick JavaDoc Popup: CTRL + Q
    Tab switcher: CTRL + TAB
    Jump to Project Navigator: ALT + 1
    Jump back to last tool window/panel: F12
    Jump to beginning/end of block (e.g., method start/end): CTRL + [ and CTRL + ]
    Toggle uppercase/lowercase of selection: CTRL + SHIFT + U
    Toggle collapse/expand: CTRL + .
    Toggle full screen editor (hide other tool windows): CTRL + SHIFT + F12

    Not a keyboard shortcut exactly but the “iter” smart template is great. If you want to iterate though something using a for loop type “iter” then TAB to use the live template. It will figure out the most likely variable you want to iterate over and generate a for loop for it. In Eclipse it worked more logically with just typing for and then autocomplete.

Getting Git Right in Helsinki

Software development is fun if you have tools which work great and support what you’re doing. So it was finally great to get hear Sven Peters talking about better software development in teams as Atlassian’s Getting Git Right landed to Helsinki (24.11.2014). Event about Git and of course about Atlassian’s tools.

Getting Git right by svenpet and durdn

Getting Git Right’s main theme was about happy developers, productive teams and how Git and Atlassian’s tools help to achieve that. Sven Peters and Nicola Paolucci presented how to be a happier developer with Git, and how to ship software faster and smarter. It’s good to remember that developing software is after all a social challenge, not a technical one. And Git helps you with it. The presentation slides are available at SlideShare and you can also watch it on Youtube (different event).

Git: You can rewrite history. Timemachine without paradoxes

Nicola Paolucci gave a nice and 5 minute talk about Git and it’s internals. Lot’s of technical details. The main points were “Fast and compact”, “Freedom and safety”, “Explore and understand”, “Control and assemble”. With Git you can rewrite the history safely to e.g. clean commits. Paolucci showed also some tools to help working with Git on the command line like hooks they use and using “better” prompt like liquid-prompt. For GUI you can use Atlassian’s SourceTree.

Git datamodel


Interesting part of the event was talk about what is efficient and the best Git workflow? The answer is “we don’t know”. It depends as there are different cultures, different products and different teams. There’s no right way but there are some good workflows which might work for you.

One is to use branch per issue, e.g. hot-fix/jira-30-user-avatars, feature/jira-27-user-sign. The simplest workflow is to use feature branches with develop branch. Then the master is very stable. If you have multiple product versions then release branches are good and bug fixes are done to separate branch and merged to other branches.

They also presented how Atlassian’s Stash can help you to work with Git and branches. Like merging changes to branches can be done automatically with hooks or by using Stash. Stash looked nice for controlling and managing your repository with visual interface.

Code reviews: do they feel like this?

Git also helps you to improve code quality with e.g. code reviews. Code reviews shouldn’t be painful as it’s about team ownership, shared knowledge and aim for better code but often there’s developer guilt. It can be made easier by making code reviews part of your daily work by doing it in small patches like pull requests.

Development is also about communication and for that Atlassian presented HipChat. It looked quite nice tool for following what’s happening in a project with aggregating team chat and information from different tools. Following commits and continuous information brings you clear view what’s happening. There are also alternatives to HipChat like Slack or just basic IRC.

But why should you use Git? Benefits like more time to code, better collaboration, dev productivity and it’s the future doesn’t convince everyone. Like pointy haired bosses. So it’s good to remember that Git is also about economics. Delivering software faster, having less bugs and thus having happy customers. Shipping software faster and smarter.

Why Git? from Peters’ slides:

Why Git?

It’s also about economics

In Questions and Answers session there was talk about Atlassian’s strategy with Bitbucket and Stash. They said that both are going strong as they have different use cases. Stash has more enterprise features and you can have the repository on your own premises. Bitbucket is about hosting the repository in Atlassian’s platform and for small and medium team. What I have used Bitbucket it’s nice service but not as user friendly as GitHub. Another interesting question was about storing binary files in Git. There’s no optimal solution yet but just some workarounds like git-annex and git-media which allows managing files with git, without checking the file contents into git. In practice you shouldn’t store binary files in Git and you should separate product to code and assets.


Atlassians Getting Git Right was nice event and gave good overview about Git and how to use it in software development team. It would have been nice to hear something about the alternatives to Atlassian’s tools (BitBucket, Stash, SourceTree and HipChat) which helps you to do better software development. I can’t deny that Atlassian’s tools work nicely together but sometimes the price is just too high.

Now it’s time to start using Git also on work projects and as all participants got “Just do Git” T-shirt it’s easier :) Thanks to Atlassian and Ambientia for arranging this event.

Sailfish OS User Interface design practices

The operating system running on Jolla has different and refreshing approach to user experience than the mainstream mobile operating systems. Sailfish OS is quite new project which also shows in applications’ user interfaces as common practices are not quite established. There are some guidelines and component usage examples how the applications should be built but they don’t cover every aspect especially with complex applications. Here are some examples how different apps solve common design issues in Sailfish OS.

Sailfish OS user interface design practices

Sailfish OS UI design practices

Sailfish OS guides you to make your app to work in certain ways and component gallery in development environment shows some practices but at the end how the app’s flow and user experience really works is up to the developer. There really isn’t comprehensive guide how elements and menu structures should work like there’s for Windows Phone and iOS. And thus Sailfish OS applications’ look and feel are sometimes quite different.

Here are some examples how different user interface aspects are designed in couple of Jolla’s own and third party developers’ apps. The presented examples and comments are just to point out different ways to build the user interface in Sailfish OS applications and doesn’t take a stance on which way is better or how it should be done.

Menu for different sections

There are couple of practices to present a menu for app’s different pages. The default is to use the pull down and push up menus but with more complex apps it’s not enough as the recommendation is to have only couple of items in that menu.

Tweetian (Twitter) uses bottom tab panel and you can flick left or right or press to change the page. In HalfTrail (map + GPS app) different sections are also as icons in bottom tab panel but activate only when pressed. In Haikala (high.fi news) settings page’s can be flicked or pressed with textual buttons.

Tab panel for pages
Action buttons on bottom panel
Bottom tab panel

Haikala uses side panel for actions to switch category. IRC uses the side panel for actions to select different IRC channels. Previously also Sailimgur (imgur) used left side panel for navigating to different pages.

Side panel for pages
Side panel for info & actions
Side panel for info & actions

Cargo Dock (file manager) is based on having two different sides and you can copy files between them. Both sides are identical but separate pages. Jolla Store shows links to apps in grid view and also arrows for selecting certain category of apps.

Two sides to work with
Arrows for sections
Grid view for selecting items

Settings menu

Settings are usually shown in separate page like in Jolla’s Settings and Hunger Meter (battery usage). Terminal app on the other hand uses hamburger menu which opens side panel.

Settings page
Settings in own page
Hamburger-menu settings

Showing controls and actions

Jolla’s Media app uses docked panel to show controls. The panel is shown only when a song is playing. You can also have docked panel with pull up menu for showing more controls icons. Tweetian has different actions in pull up and pull down menus.

Controls in docked panel
Docked panel with pull up menu
Actions in Pull up menu

Other common way is to show controls in bottom or top panel with tool icons. This can be seen in HalfTrail with action buttons in bottom toolbar. In Paint there’s a top toolbar with icons. Neither provide legend for icons although Paint has help in About page. MitaKuuluu (WhatsApp) also uses bottom toolbar for chat related actions. In Sailimgur you can choose to have to toolbar at the bottom or at the top. The bottom toolbar has better reachability if used with one hand.

Action buttons on bottom panel
Tools in top panel
Actions in bottom toolbar
Toolbar at the bottom
Top toolbar

Jolla Settings app uses switches to toggle different features. Terminal app on the contrary takes quite different approach to showing actions and settings. Terminal uses so called Hamburger menu which works quite nicely although isn’t “Sailfish OS” style.

Actions as switches
Hamburger-menu settings

Sailimgur shows actions for the photo when long pressed in a context menu. Actions related to a comment are shown as bottom context menu or icon buttons. Quickddit (Reddit) shows actions as icon buttons.

Actions in context menu
Actions in context menu
Actions as icon buttons

Back navigation

For the back navigation there isn’t a physical button and you either flick left, click page stack icon or use some button and action in pull down menu.

In Jolla’s Gallery app you scroll through photos in slideshow view and you get actions for back navigation only when pressing the photo and opening the Docked panel where you can flick back to the gallery overview. Alternative way for back navigation from photo slideshow is just press the photo as seen in e.g. Jolla Store.

Jolla Opas (Reittiopas) uses simple icon button in toolbar for navigating the route on the map and navigating back. And of course you can have the back navigation in pull down menu like in Sailimgur when navigating back from Web view.

Back navigation in docked panel
Back navigation with button
Closing view in pull down menu

Navigating between items

Navigating between items in a list is normally done by flicking left or right. But sometimes that’s not usable or it’s better to provide also alternative way.

Sailimgur has previous and next buttons at the bottom, Tidings (RSS feeds) provides them in pull down and pull up menus and Browser has them in bottom tool bar as icons.

Next & Previous as buttons
Next in pull down menu
Prev/next in toolbar


Providing search is usually done as a separate page but it can be put on pull down menu or as normal element in page.

Warehouse (OpenRepos) application has search link in pull down menu which navigates you to separate search page which also shows the results. Sailimgur has the search element in pull down menu and it closes after user presses enter to show the results in main view. Jolla Opas is a route planning app so the search elements are in main page. The search results are shown (if found more than one destination) as blue circle with results number and can be selected in separate dialog page.

Search in own page
Search in menu
Search as main function
Inline search in main view

Showing gallery of images

Showing list of images is pretty simple but there are differences how applications do it. Gallery application uses one big grid of cropped thumbnails, Warehouse shows full thumbnail and selection list in side and Jolla Store shows limited list of cropped thumbnails which opens in own view.

Only thumbnails, press to open own view
Grid view of images
Large image, side preview list

Notifications about new features

Mobile applications update quite often but users don’t usually check the changelog what’s changed or if there’s new features. So it’s nice to provide a simple changelog in first start like Flashlight, Paketti (post packages) and Haikala.

New features notification
Showing changelog
Changelog dialog

Transferring Linux install media to USB thumb drive

Optical drives are nowadays less common on laptops which makes making bootable install medias a little easier. Yes, easier by requiring you to use USB thumb drives. There are different ways to achieve what you want and here is couple of examples how to transfer Linux install media to USB thumb drive on Windows and Mac OS X.

First you need to download the install media of your choice as ISO image which you want to transfer to your USB thumb drive. In theory you need a media which is meant for USB thumb drive or is a Hybrid ISO image but practically all modern distributions’ media are Hybrid ISO images.


Plug in the USB thumb drive and format it with FAT32 file system

Download UNetbootin and transfer ISO content to USB thumb drive using UNetbootin.

  1. Select Diskimage and click the “. . .” button to find and open the ISO file. Make sure Type is USB drive and the Drive letter is what your Windows detect and click OK. Your ISO content will be copied to USB thumb drive.
  2. Next you also need to copy the ISO file that you used with UNetbootin to the root of the USB thumb drive, e.g. the ISO file should be F:\linuxmint-16-cinnamon-dvd-64bit.iso

Now you can proceed to boot up the machine that you want to install with e.g. Linux Mint 16.


Creating bootable USB thumb drive in Linux is much simpler. You just need install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

You just need to have the ISO image and one command:

$ sudo dd bs=4M if=linux-image.iso of=/dev/sdX oflag=direct

Where sdX is the device where your USB thumb drive is, e.g. sdb. If your system doesn’t support ‘oflag=direct’, you can just leave it out as it is simply intended to speed up the process a bit. If you leave the bs-option off the default will be 512 bytes and takes forever.

Mac OS X

OS X’s DiskUtil can’t burn ISO images to USB and gives “Invalid argument” but fortunately just like with Linux also Mac OS X makes things easy for you. You just need the install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

$ dd if=linux-image.iso of=/dev/rdiskX bs=8192

Where rdiskX is the device where your USB thumb drive is, e.g. rdisk1. You can see the device (disk identifier) from DiskUtility.app by selecting your USB stick and checking the Device section.

It’s useful to notice that on OS X you can use the ‘/dev/rdiskX’ device which stands for raw disk and is much faster than the block level device ‘/dev/diskX’ which is buffered and where data undergoes extra processing. For example rdiskX gives “823132160 bytes transferred in 625.470137 secs (1316022 bytes/sec)” and diskX gives “823132160 bytes transferred in 1296.675998 secs (634802 bytes/sec)”.

If you get an error e.g. “dd: /dev/rdisk2: Resource busy” then you have to make sure the Mac OS X isn’t using it and thus unmount it’s partitions. Open the DiskUtility.app, and on your USB hard drive, unmount any of it’s partitions. Do not eject the USB hard drive.

Do a clean install of Windows 8 with an upgrade key

There are times when you have to do a clean install of your Windows 8 but if you have just an upgrade key you need to make couple of extra hoops before you can activate the new install. The upgrade key doesn’t prevent you installing to a clean disk but when you try to activate, you get an error 0x8007007B, saying your product key can only be used for upgrading. Another fine example how Microsoft makes things complicated for legitimate users.

Fortunately there’s a way to fix that issue as Lifehacker’s article tells:

  • Open the Registry Editor (Win + R, type regedit).
  • Navigate to “HKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE/” and change the MediabootInstall key’s value from 1 to 0.
  • Open the Command Prompt (Win + R, type cmd). Right-click on the Command Prompt icon and run it as an administrator.
  • Type slmgr -rearm and press Enter.
  • Reboot Windows.

After that is done and you get back into Windows, you should be able to run the Activation utility and activate Windows as normal, without getting an error.

Apparently you can also call Microsoft Support and they will walk you through the proper way doing this because they understand you may have bought a new drive. There is a dialog you can get to in system tools where it will ask you for a numerical code. The support personnel will give you the number, you click “OK” after typing it in, then go the activation again and it works. This process is likely doing the work around mention above, but through an approved administrative process.

By the way, restarting Windows 8 is most easily done by left-clicking once on an empty spot on the desktop and holding Alt + F4).

Monitoring Java EE application with JavaMelody

Software development is much more than just coding application by requirements and deploying it to production as the real work really starts after it has been shipped: maintenance, improvements and problem solving. And for that it’s good to have some data. It’s said “if you can’t measure it, you can’t improve it” and in ideal situations you plan for measuring your app but often you have to do it in production to detect and diagnose performance problems. There are different ways to measure a Java EE application and one good tool to achieve performance and monitoring statistics is Javamelody which has low overhead, is non-intrusive, informative and simple to setup but still very capable.

“If you can not measure it, you can not improve it.” – Lord Kelvin

Application performance can be measured by two main methods: computational resources used by the application and the performance as seen by a user of the application. By measuring these quantities we get an empirical performance baseline of the application which then can used to detect changes in performance. By using performance monitoring, which is an act of non-intrusively collect or observe performance data from running application, we get measurements to identify or isolate potential issues from real operation of an application without having a severe impact on runtime responsiveness or throughput.

There are different ways to achieve performance and monitoring statistics and it’s useful to plan for measuring application’s performance while still in development as later on you surely want to know how it is performing. You can do it by leveraging different JMX-based tools like Metrics and Servo but if you didn’t really plan for it you still have choices. You can use tools like standard JDK tools (jconsole, jstat, jmap, jstack, hprof) for low level JVM monitoring or some monitoring application like Javamelody. And as time is always scarce in development it’s useful to deploy a tool like JavaMelody which provide quick and easy access to performance monitoring. But if you are interested of some narrow focus measurement you’re better of with some custom tool.

Monitoring with JavaMelody

JavaMelody is an open source (LGPL) application to monitor Java or Java EE application servers in QA and production environments. It is a tool to measure and calculate statistics on real operation of an application depending on the usage of the application by users and is mainly based on statistics of requests and on evolution charts which can be viewed on the current day, week, month, year or custom period. The statistics can be viewed on a HTML page and sent as PDF reports by email.

JavaMelody lists the following use cases:

  • It allows to improve applications in QA and production
  • Give facts about the average response times and number of executions
  • Make decisions when trends are bad, before problems become too serious
  • Optimize based on the more limiting response times
  • Find the root causes of response times
  • Verify the real improvement after optimization

Setting up

Setting up JavaMelody is quite easy and the needed steps are covered in User guide. Integrating JavaMelody to your application can be done in less than 10 minutes, by automatic discovery of environment: it only requires to copy 2 jar files and to add 10 lines in a xml file. But for more detailed and wider measurements you need couple more lines to xml files and here are the changes I made for our Wicket, Spring, JPA, Hibernate -Java EE project.

JavaMelody and dependencies

Getting JavaMelody and it’s dependencies is easy with Maven2 and you just need to add javamelody-core and if you want to have PDF reports also iText in your pom.xml:

<!-- javamelody-core -->
<!-- itext, option to add PDF export -->

Monitoring application

JavaMelody needs just a monitoring filter before the description of webapp’s servlet in WEB-INF/web.xml or not even that as described in the User guide. But usually you want a little bit more than the minimum 10 additional lines in web.xml.

By adding the customResourceFilter you get customized look & feel, you can exclude some urls from statistics with url-exclude-pattern regular expression pattern, eamil-parameters are for weekly reports and by including monitoring-spring.xml you can monitor DataSource with a Spring post-processor and objects initialized with Spring.

<!-- Must be defined before javamelody filters -->
<!--====================== Monitoring ===================================-->
<!-- Custom CSS -->
<!-- Monitor filter -->

Spring Security config

For restricting the access to monitoring statistics I added the following to the Spring Security’s applicationContext-security.xml. There are also other options for security. The key point here is to make sure, that the monitoring-filter in web.xml (above) is defined after the Spring Security filter chain.

<http auto-config="true" use-expressions="true">
    <intercept-url pattern="/monitoring/**" access="hasRole('ROLE_ADMIN')" />
	<intercept-url pattern="/**"
			access="hasAnyRole('ROLE_ADMIN, 'ROLE_USER'')" />

Monitoring SQL and datasources

For monitoring datasources and SQL I just added jndi-lookup to applicationContext-persist.xml. Other options to enable JDBC monitoring are described in the User guide.

<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myAppDS"/>
<bean id="entityManagerFactory"
	<property name="dataSource" ref="dataSource" />

Business facades (Spring)

If the application to monitor contains some objects initialized by Spring, EJB or Guice their methods execution can also be added to statistics. As described in the User guide for monitoring Spring Business facades there are couple of options and for example with JdkRegexpMethodPointcut in applicationContext-web.xml you can catch objects with regular expression like “all that have Service in their names”.

<bean id="facadeMonitoringAdvisor" class="net.bull.javamelody.MonitoringSpringAdvisor">
	<property name="pointcut">
		<bean class="org.springframework.aop.support.JdkRegexpMethodPointcut">
			<property name="pattern" value=".*Service.*" />

EHCache statistics

If you want to see also EHCache statistics add statistics=”true” to ehcache.xml config file.

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
    <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" statistics="true" />
    <cache name="fooCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
    <cache name="barCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />

Database information and statistics

JavaMelody already shows datasource and SQL information but it’s also possible to display information and statistics on the database like the longest requests in cumulative time with display of the cpu time and of the elementary cost. For showing that information the user in the database used by the monitored application must have the necessary rights to read those information and statistics.

In Oracle database the request practically means select * from v$session and you can grant access as ‘system’ user with GRANT SELECT any dictionary TO myapplicationuser. Strangely granting just the select for v$session (GRANT SELECT ON sys.v_$session TO myapplicationuser;) wasn’t enough.


There is also a possibility to get weekly, daily or monthly report in pdf format sent by email to one or several people. It needs iText library for webapp and Java’s JavaMail and Activation libraries in your server for the mail session.

With WebLogic you add some email parameters in webapp’s web.xml (above) and configure a Mail Session in WebLogic AdminServer: Services > Mail Sessions with JNDI Name and JavaMail properties. The report provides the same information you can find in monitoring web page with high and detailed level information.

Statistics’ storage

Javamelody data is stored in files on disk and there are 2 types of files: *.rrd files for values in graphics in RRD format (using jrobin library) and *.ser.gz for values in statistics.

The performance statistics are stored to temp/javamelody directory by default which means /tmp/javamelody in Linux and in Windows when running webapp in Eclipse something like Users/developer/AppData/Local/Temp/javamelody. If you want to reset the counters just delete all the files. The location can be changed with storage-directory parameter and if the name of the directory starts with ‘/’, it is considered as an absolute path, otherwise it is considered as relative to the temporary directory.

There is also option to use centralized collect server to store statistics and for monitoring of several applications.

Viewing performance monitoring

After you have set up your webapp to have the monitoring you can see statistics at URL like http:////monitoring depending your configuration.

Some example JavaMelody statistics from development:


Monitoring and filtering doesn’t come without costs and there has been discussions on JavaMelody wiki about the overhead the monitoring does. It is said that the overhead is so low that it can be enabled continuously in Quality Assuarance environment and if no problem arises in QA, also continuously in production environment. And with little overhead you will be able to know what needs optimizing in the QA or production servers so that the overhead of JavaMelody will practically be negative.

The discussion contained some notes:

  • Architecture of JavaMelody is lightweight so it has a lower overhead as compared it to other available solutions
  • It is only statistics and not events so the overhead of memory is quite minimal.
  • It does monitoring not profiling: there is no instrumentation of classes and instead “interceptors” for http, jdbc, spring or ejb3.
  • No database and no recording of each events even in a file or over the wire: only statistics of requests are kept. The overhead of cpu is minimal with no I/O on the wire and minimal I/O on disk (just to take a backup of statistics at a regular interval). The overhead of some other good monitoring solutions is in the recording of each event in a database or in a master server.
  • You have the choice to use centralized collect server which unloads the memory, the backup storage and the generation of reports to another server while adding I/O on the wire for sending deltas of the statistics.


“If you can not measure it, you can not improve it.” It doesn’t matter how you do it but if you want to use real user data and environment just do it non-intrusively with performance monitoring tools like Javamelody.