Looking back at 2013, planning ahead for 2014

The year 2014 is almost here so it’s time to take a look back and make some plans for the coming year. In 2013 the blog was a bit more active than previous years and I managed to put together 13 articles as I last year promised. Not one article per month but on average :) So let’s have a quick look what I wrote.

Looking back

The year 2013 in Rule of Tech was about programming and sysadmin stuff with a short visit to Fujitsu Forum 2013.

This year it was my third visit to Munich and Fujitsu Forum. With keynotes and breakout sessions the forum provided a view to human centric intelligent society and how the future might look like with technology trends like Internet of things. It was again nice to see other bloggers and the Fujitsu team, get some insights to ICT and not to forget the hands-ons with new LIFEBOOKS like U904 and T904 and new STYLISTIC tablets.

On software development I wrote on three main topics: testing, monitoring and documentation. In software development the code is one part but also it’s quality and testing it are also essential aspects to make it great. I learned some new tricks with testing Java EE applications and wrote how to start on web application test automation with Robot Framework. I planned to write about how to actually write some test cases but didn’t have the time.

It’s also good to remember that “If you can not measure it, you can not improve it.” (Lord Kelvin) so I covered it with article about monitoring Java EE application with JavaMelody. JavaMelody proved to be very handy tool to do monitoring in different levels and was quite easy to set up. It can show information about e.g. CPU and memory, HTTP requests, SQL queries, Spring beans and REST calls. And it is said that the overhead is so low that it can be enabled continuously even in production environments.

To get things easily documented I made my first Maven plugin which generates HTML pages from Markdown syntax. It isn’t yet on the central repository but I’m thinking on it. The source code is on GitHub.

As I have always wrote my blogs like they were my notebooks, I made short guides how to do stuff like transferring Linux install media to USB thumb drive and do a clean install of Windows 8 with an upgrade key. Simple things but things you often can’t remember how to do. And although I hope I don’t need to use JSF 1.2 anywhere anymore I wrote about patching RichFaces 3.3.3 AJAX.js for IE9.

On the sysadmin side I finally wrote about using PHP-FPM with Apache 2 on CentOS and howto set up LAMP stack on OS X. And as everything doesn’t run on PHP I also played with WebLogic Server and wrote how to auto restart Weblogic Server with Node Manager as Linux service and how to use WebLogic Maven Plug-In for deployment.

I’m quite satisfied about the articles I managed to make and especially about Robot Framework and JavaMelody which have been helpful on many occasions. During 2013 I also moved the blog to new VPS provider and although there was also an idea to make a new custom theme, it didn’t materialize. Maybe next year.

Planning ahead

For the coming year 2014 the plan, as usual, is to write approximately one article per month. Blog’s draft folder has some items about JavaScript and charts, Munin plugins and general software development and some sysadmin posts but it will be seen if they mature enough for full posts. I will maybe also write something about Sailfish OS development and Sailimgur.

So subscribe to the RSS feed and stay tuned. And if you can read Finnish check also my other blog. Will the new year be better than the previous? Who knows, but at least you can try your best to make that so.

Happy New Year!

Transferring Linux install media to USB thumb drive

Optical drives are nowadays less common on laptops which makes making bootable install medias a little easier. Yes, easier by requiring you to use USB thumb drives. There are different ways to achieve what you want and here is couple of examples how to transfer Linux install media to USB thumb drive on Windows and Mac OS X.

First you need to download the install media of your choice as ISO image which you want to transfer to your USB thumb drive. In theory you need a media which is meant for USB thumb drive or is a Hybrid ISO image but practically all modern distributions’ media are Hybrid ISO images.

Windows

Plug in the USB thumb drive and format it with FAT32 file system

Download UNetbootin and transfer ISO content to USB thumb drive using UNetbootin.

  1. Select Diskimage and click the “. . .” button to find and open the ISO file. Make sure Type is USB drive and the Drive letter is what your Windows detect and click OK. Your ISO content will be copied to USB thumb drive.
  2. Next you also need to copy the ISO file that you used with UNetbootin to the root of the USB thumb drive, e.g. the ISO file should be F:\linuxmint-16-cinnamon-dvd-64bit.iso

Now you can proceed to boot up the machine that you want to install with e.g. Linux Mint 16.

Linux

Creating bootable USB thumb drive in Linux is much simpler. You just need install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

You just need to have the ISO image and one command:

$ sudo dd bs=4M if=linux-image.iso of=/dev/sdX oflag=direct

Where sdX is the device where your USB thumb drive is, e.g. sdb. If your system doesn’t support ‘oflag=direct’, you can just leave it out as it is simply intended to speed up the process a bit. If you leave the bs-option off the default will be 512 bytes and takes forever.

Mac OS X

OS X’s DiskUtil can’t burn ISO images to USB and gives “Invalid argument” but fortunately just like with Linux also Mac OS X makes things easy for you. You just need the install media which is meant for USB thumb drive or is a Hybrid ISO image. With hybrid images, you can simply write the content of the ISO and make a bootable USB stick which can install Linux by using the ‘dd’ command.

$ dd if=linux-image.iso of=/dev/rdiskX bs=8192

Where rdiskX is the device where your USB thumb drive is, e.g. rdisk1. You can see the device (disk identifier) from DiskUtility.app by selecting your USB stick and checking the Device section.

It’s useful to notice that on OS X you can use the ‘/dev/rdiskX’ device which stands for raw disk and is much faster than the block level device ‘/dev/diskX’ which is buffered and where data undergoes extra processing. For example rdiskX gives “823132160 bytes transferred in 625.470137 secs (1316022 bytes/sec)” and diskX gives “823132160 bytes transferred in 1296.675998 secs (634802 bytes/sec)”.

If you get an error e.g. “dd: /dev/rdisk2: Resource busy” then you have to make sure the Mac OS X isn’t using it and thus unmount it’s partitions. Open the DiskUtility.app, and on your USB hard drive, unmount any of it’s partitions. Do not eject the USB hard drive.

Fujitsu Forum 2013: view to human centric intelligent society

Munich in Autumn isn’t maybe the most predictable travel target but for many decision-makers, experts, project managers, IT architects, consultants and users it’s the place to go when Fujitsu’s largest IT-event in Europe took place on 6th and 7th November. I was one of the invited bloggers to visit Fujitsu Forum 2013 and again it was very insightful and exciting two days. With the motto “Reshaping ICT, Reshaping Business and Society” there were over 11,000 attendees around the world getting new information and looking for strategic and operative options for modernising and developing their IT.

Fujitsu Forum 2013

Fujitsu Forum 2013 in Munich is Fujitsu’s largest IT-event in Europe and was this year attended by more than 11,000 IT experts from around the world. With keynotes, breakout sessions, expert talks, exhibition and evening event the Fujitsu Forum is good place to get new information and looki for strategic and operative options for modernising and developing IT. Just like last year, I was invited by Fujitsu LIFEBOOK4Life campaign with three other insiders aka bloggers to spend couple of days at the Forum and to get insight what the future of IT might look like.

This year the Forum was held with the motto “Reshaping ICT, Reshaping Business and Society” and the focus was more about how the technology affects us and how we can make it work for us. Or in business words: “How to combine business processes and IT to form a stable basis that will ensure growth and success in the future.” The main topic was reworked from last years’ “Reshaping ICT – Reshaping Business” and the ICT trend could be easily seen from the buzzwords I heard also this year. Consumerization, BYOD, tablets, virtualization, in-sync, cloud and “One workplace – on any device” were still hot topics but maybe not so strong than last year as most of the hype has passed.

Fujitsu Forum 2013 starts
L4L team and insiders at press lounge

Sessions

Our great LIFEBOOK4LIFE team had planned two keynotes and one breakout session which meant the days wouldn’t be so packed and we would have time to share our insights and to stroll at the Exhibition area.

We started with Defining Your Workplace Strategy: Our Recommendation to IT breakout session by Meinolf Althaus (Fujitsu) who told us how to manage the different users’ needs for the workplace. The three points were: Analyze your users and create user profiles (one size does not fit all); Accommodate multiple devices, operating systems, delivery technologies; Rely on a partner which offers an end-to-end portfolio.

Last year we missed the opening keynote but this year we were wiser. The opening keynote provided a view to Fujitsu’s technology and services vision: human centric intelligent society. The keynote presented the idea behind the “human centric intelligent society” vision and also the exhibition was also built on the three dimensions presented on the keynote: Technology and infrastructure and integration (optimize ict systems from end to end); process and information (power business and society with information); people and innovation (create innovation through people). The main point was that we are in a global village and virtually everything and everybody are connected in one extensive network. But with global village come some challenges both in business and society sense. For example social media is more pervasive and the volume of information is astounding. The question is, how do you mine, analyze and act upon that data? For more detailed information check Marco Rossi’s short recap of the session and how Fujitsu’s services address this topic.

Our last session was on Thursday and I have always liked Dr. Reger’s presentations which are both informative, clever and witty. The CIO Session: “All things considered” – How the internet of things changes our world gave us good ideas what are the technology trends that are going to affect our lives and businesses in the next few years and how the world could look like when every possible machine has it’s own IP address.

Internet of things = there is no people and cat has IP address – Dr. Joseph Reger

In short “All things considered human centric intelligent society is the internet done right. The job is done by the things for the people. Internet of things will make our lives more complex but with multiple opportunities.”. The road to Internet of things isn’t yet paved and there are still many problems to solve like starting with IPv6 becoming mainstream, standards to control the devices and security issues. Reger gave some Sci-Fi like examples what kind of possibilities the Internet of things will present to help our everyday life for example by sensing different things and acting based on that information. It’s worth to check out the video recording of the session from the Documentation or read the written recap by Fujitsu’s Marco Rossi.

Virtually everything and everybody are connected
Everything will have IP address, like light bulb

Overall, I have to admit I didn’t get a good grasp about the message of the Fujitsu Forum 2013 which had a bit different aspect to cover than last year. The same ideas were on the background but from my Fujitsu Forum 2012 insights the message is easier to pinpoint: “The work and workplace is changing to support mobility and working anywhere anytime. The borders between business and private life are blurring, consumerization (same devices home and work) and BYOD (bring your own device) are coming more common. We should think more about user-oriented workplace. In the future the devices we use will be more diverse and we have more of them. Thus there will be need for virtualization, syncing data and support for “One workplace – on any device” ideology. Different tasks have different needs for the device so we need a dynamic desktop experience.” Maybe the topic felt this year more vague as I attended two keynotes which had more abstract than concrete ideas how the workplace might look like compared to last years’ three breakout sessions.

If you weren’t able to attend the Forum or some particular session you can re-experience the Fujitsu Forum 2013 and watch or download the presentations from Fujitsu Forum 2013 documentation page.

Exhibition

Between the breakout sessions and keynotes we had time to check out the exhibition are with 250 highlights from the portfolio offered by Fujitsu and its partners. I especially liked innovative ideas like The Office 21: Intelligent Desk which showed how to make the physical desk better and the unified communication & collaboration setup (article by FujitsuFans.com) which presented how video conferencing and sharing ideas can work more seamlessly. I also didn’t know that Fujitsu also makes ATMs.

Some highlight from the exhibition in 15 seconds: ATM, retail solution, unified communications, U904, T904, mainboards, displays, Eternus parts, intelligent desk, cad.

And of course as a technology geek getting my hands on to Fujitsu’s new LIFEBOOKs Ultrabooks with IGZO display was nice. Both the versatile T904 convertible ultrabook/tablet and the ultra-slim U904 were beautiful devices and I especially liked how the RJ-45 port was designed. Also the semi-rugged STYLISTIC Q704 tablet had nice use cases and accessories to support it.

The Office 21: Intelligent Desk
360 degree video conferences and collaboration

For more detailed information about the different topics presented at the Exhibition check the Fujitsu Fans -blog with the forum tag and the YouTube playlist. I just don’t understand how Abassin who keeps the Fujitsu Fans blog (MixBlogeu in YouTube) had time to do so many hands-ons and interviews at the Forum :) If you’re interested to know more about the new LIFEBOOK Ultrabooks U904 and T904 check the hands-on articles from the Fujitsu Fan blog as they present nicely what kind of Ultrabooks they are.

LIFEBOOK Ultrabooks with innovative RJ45 port
Hologram showcase

For visual view and to get the overall feeling of the Fujitsu Forum 2013, Fujitsu DE has made day 1 impressions and day 2 impressions in pictures on Flickr.

Networking and entertainment

And lest the whole two days be just about technical topics there was an Oktoberfest themed evening event on Wednesday. This year the event provided something different and was more social than previous years’ concerts with Anastacia (2011) and Amy McDonald (2012) which were also nice. At the “Oktoberfest” there were special Bavarian dishes, such as crispy roast chicken (Hendl), spiced strips of pork belly (Hüttenspeck) and of course pretzels (Obatzda und Brezn) and excellent beer. And it was not just food and drinks as there were playful games like sawing a log, throwing a pint, hammering nails and throwing balls. If the games weren’t for you, you could take a ride in a carousel, ferris wheel or swing (schauke).

Oktoberfest by Fujitsu Forum 2013. G’suffa.
Accidentally half a chicken. And weissbeer.

The Oktoberfest atmosphere was something like this:

[vine url=”https://vine.co/v/hIqPEj5HTWV” width=300 height=300] [vine url=”https://vine.co/v/hIq13WmJ7uZ” width=300 height=300]

Summary

Fujitsu Forum 2013 is over but the new insights are just starting to form up to new ideas how to do things better. This was my third visit to the Forum and every year has been interesting and a bit different and great on their own kind of way. This year the sessions, keynotes, I listened were more about ideas than doing and they gave a good view how the future might look like with trends like Internet of things and how Fujitsu sees the future and business through their “human centric intelligent society” vision. I also got to see great looking Ultrabooks, interesting new concepts and products and I really liked the Oktoberfest themed evening event which was more social than previous years’ concerts.

The three days in Munich went fast and it was again great to visit Fujitsu Forum and to see other Insiders and the campaign crew. Thanks LIFEBOOK4Life and Fujitsu.

Fujitsu Forum 2013

Getting ready for Fujitsu Forum 2013

In two weeks time it’s again that time of the year when business- and IT decision-makers, experts and specialists, project managers and IT architects, consultants and users travel to Germany to visit Fujitsu’s largest IT-event in Europe: Fujitsu Forum 2013. And that’s also the destination where I’m going as I’m one of the bloggers invited by Fujitsu’s Lifebook4Life campaign to visit the forum. This year the event is held 6th and 7th November at the ICM in Munich and there will probably be over 12,000 attendees around the world getting insights and looking for strategic and operative options for modernizing and developing their IT.

The motto this year, “Reshaping ICT, Reshaping Business and Society”, highlights the key theme that is addressed by all aspects of the event: How to combine business processes and IT to form a stable basis that will ensure growth and success in the future. So what can you expect from Fujitsu Forum 2013? The event overview gives you some idea about the different topics covered but let’s look which events I’m looking forward to the most.

The presentations I picked from Keynotes and the list of 52 Breakout Sessions for my Fujitsu Forum 2013 agenda span all of the three conference tracks, “Create innovation through people”, “Power Business and Society with information” and “Optimize ICT systems from end to end”, and cover topics like cloud services, creating value from enabling innovative IT models with mobile technologies and information security and data protection. Overall I selected 10 sessions to my agenda but pruned it to one Keynote and four Breakouts as you don’t have time for every interesting topic.

Defining Your Workplace Strategy: Our Recommendation to IT by Meinolf Althaus (Fujitsu), 6 November, 10:00, Room 13b

Adopting new computing devices and delivering concepts to provide optimized workplaces for the different user roles within a company present a new challenge. An optimized end-to-end selection, from back-end to workplace devices and a suitable combination of desktop delivery methods is essential to guarantee high availability, flexibility and user satisfaction. This session looks at the various ways to identify the ideal workplace strategy.

Leveraging Advanced ICT to Support Fujitsu Technology and Service Vision by Tatsuo Tomita (Fujitsu), 6 November, 13:00, Room 4

Leveraging ICT to generate New Value for society is essential to enable sustainable growth and advancement of societies. This session highlights how Fujitsu’s R&D is being leveraged toward realizing a Human Centric Intelligent Society envisioned by Fujitsu, by supporting the Fujitsu Technology and Service Vision, to be achieved through Three Actions: Create Innovation through People’s Activities; Power Business and Society with Information; and Optimize ICT Systems from End to End. We introduce how it will contribute to societal growth and realization of a prosperous society in which people can thrive.

Intelligent Workplace. Greater Efficiency. Here is the answer! by Anand Srivatsa (Intel) and Dieter Heiss (Fujitsu), 6 November, 16:00, Room 13a

There are fundamental changes taking place in society, new technology and changing demands are redesigning the workplace. The enterprise workplace will dramatically improve interactions across applications and between workers, driving productivity and improvements in work quality. This session will show you how new technologies support you at your workplace and how to ensure efficiency in and outside the office.

Keynote: The CIO Session by Dr. Joseph Reger (Fujitsu) and Ed Goldman (Intel), 7th November, 10:00

“All things considered” – How the internet of things changes our world

Powering Healthcare Information – improve and secure the healthcare data value chain by Lester Russel (Fujitsu), 7 November, 12:00, Room 5

Healthcare costs are rising and the burden of managing long term conditions threatens to destabilize whole economies. Clinical information has to flow smoothly to ensure a seamless patient journey through the healthcare system and the safe practice of medicine has special requirements in terms of the data value chain. The right information, in the right place, at the right time. This session will give examples of how IT can power healthcare in new and exciting ways and underpin the radically different processes we need to adopt, if we are to avoid meltdown in our healthcare systems.

And of course you should not forget the Exhibition with 250 highlights from the portfolio offered by Fujitsu and its partners which has always been interesting. I’m looking forward to get hands-on with Fujitsu’s new LIFEBOOKs with IGZO display like the versatile T904 and the ultra-slim U904 and check out the semi-rugged STYLISTIC Q704 tablet.

And lest the whole two days be just about technical topics there’s the Oktoberfest themed Evening Event on Wednesday. Excellent entertainment and networking opportunity with drinks together with special Bavarian dishes, such as crispy roast chicken (Hendl), spiced strips of pork belly (Hüttenspeck), bread and dripping (Schmalzbrot), chive bread, savory cheese spread with pretzels (Obatzda und Brezn). Hmmm, and now I’m hungry :)

Until then, read about my insights from last year’s Fujitsu Forum 2012 where I was invited by Fujitsu’s Lifebook4Life campaign and follow me on Twitter (@walokra) for upcoming insights from Fujitsu Forum 2013. See you all on 6th and 7th November at the ICM in Munich.

Do a clean install of Windows 8 with an upgrade key

There are times when you have to do a clean install of your Windows 8 but if you have just an upgrade key you need to make couple of extra hoops before you can activate the new install. The upgrade key doesn’t prevent you installing to a clean disk but when you try to activate, you get an error 0x8007007B, saying your product key can only be used for upgrading. Another fine example how Microsoft makes things complicated for legitimate users.

Fortunately there’s a way to fix that issue as Lifehacker’s article tells:

  • Open the Registry Editor (Win + R, type regedit).
  • Navigate to “HKEY_LOCAL_MACHINE/Software/Microsoft/Windows/CurrentVersion/Setup/OOBE/” and change the MediabootInstall key’s value from 1 to 0.
  • Open the Command Prompt (Win + R, type cmd). Right-click on the Command Prompt icon and run it as an administrator.
  • Type slmgr -rearm and press Enter.
  • Reboot Windows.

After that is done and you get back into Windows, you should be able to run the Activation utility and activate Windows as normal, without getting an error.

Apparently you can also call Microsoft Support and they will walk you through the proper way doing this because they understand you may have bought a new drive. There is a dialog you can get to in system tools where it will ask you for a numerical code. The support personnel will give you the number, you click “OK” after typing it in, then go the activation again and it works. This process is likely doing the work around mention above, but through an approved administrative process.

By the way, restarting Windows 8 is most easily done by left-clicking once on an empty spot on the desktop and holding Alt + F4).

Auto-expand WP-Syntax code blocks

Hilighting code blocks in Worpdress blog posts can be achieved with different plugins and one of them is the WP-Syntax plugin which uses GeSHi source code highlighting and supports a wide range of popular languages, highlighting with or without line numbers and maintains formatting while copying snippets of code from the browser. The plugin highlights code using HTML’s <pre> tag, e.g. <pre lang=”java” line=”1″> for Java and line numbers starting at 1.

Source code highlighting makes it easier to read examples but sometimes if your layout isn’t 900+ pixels wide, you may end up with horizontal scrollbars for wider code blocks as you can’t always make it 80 characters. But fortunately you can make the plugin auto-expand the code block when hovered over. You just have to edit the plugins code a bit as Derek MacDonald explains.

For some reason the code to auto-expand the block when hovered over has been commented out in the plugin and to enable it you’ll have to edit three files:

wp-syntax/wp-syntax.php
In method enqueueScripts(), uncomment the line that invokes wp_enqueue_script() so wp-syntax.js will be loaded.

wp-syntax/css/wp-syntax.css
Ensure the overflow, overflow-x, and overflow-y properties of class .wp_syntax are set to hidden.

wp-syntax/js/wp-syntax.js
It might be better to remove the box-shadow CSS property added during the mouseover event because it doesn’t look good in most designs.

The mouseover/mouseout expand/collapse events will be immediate with no delay.

Using PHP-FPM with Apache 2 on CentOS

Running Apache 2 and PHP is simple with mod_php but there are more efficient alternatives like using PHP-FPM (FastCGI Process Manager) which is an alternative PHP FastCGI implementation. With it the PHP process runs standalone without the need for a web server and listens for incoming requests on either a TCP or a Unix socket. Web servers can connect the PHP process and send requests using the FastCGI protocol. It solves mod_php’s problem of spinning up and destroying PHP instances with every request and thus is more memory efficient and provides better performance.

These instructions are for CentOS 6.4 but the process should however work similarly with other Linux distributions.

Setting up the PHP-FPM

Install the FPM-CGI binary for PHP and add it to start after server reboot:

# yum install php-fpm
# chkconfig --levels 235 php-fpm on

Configure the PHP-FPM pool in /etc/php-fpm.d/www.conf to use sockets and enable some status information for e.g. Munit:

;listen = 127.0.0.1:9000
listen = /tmp/php5-fpm.sock
pm.status_path = /status
ping.path = /ping

Start the service with:

service php-fpm start

Setting up Apache and mod_fastcgi

Apache can be configured to run FastCGI with two modules: mod_fastcgi and mod_fcgid. The difference is explained at Debian bug report #504132: “mod_fcgid passes just one request to the FCGI server at a time while mod_fastcgi passes several requests at once, the latter is usually better for PHP, as PHP can manage several request using several threads and opcode caches like APC usually work only with threads and not with processes. This means that using mod_fcgid you end up having many PHP processes which all have their very own opcode cache.”

In short: mod_fastcgi is better.

Install mod_fastcgi

So we need to get mod_fastcgi which isn’t at the time found from CentOS default or EPEL repos but from RPMForge or by building it from sources.

Getting mod_fastcgi from RPMForge

Install the RPMForge repo:

# wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
# rpm -ivh rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Add some priorities which repo to use:

# yum install yum-priorities
 
# vi /etc/yum.repos.d/epel.repo 
... add the line priority=10 to the [epel] section

Install mod_fastcgi

# yum install mod_fastcgi

Or building mod_fastcgi from sources

You can build the mod_fastcgi from sources. Make sure required packages are installed (httpd-devel and apr-devel required to compile mod_fastcgi):

# yum install libtool httpd-devel apr-devel apr

Get the latest mod_fastcgi source code:

# cd /opt
# wget http://www.fastcgi.com/dist/mod_fastcgi-current.tar.gz

Untar tar ball:

# tar -zxvf mod_fastcgi-current.tar.gz
# cd mod_fastcgi-2.4.6/

As we are using Apache 2, we make a copy of Makefile.AP2: cp Makefile.AP2 Makefile

Compile and install mod_fastcgi for 64 bit system:

# make top_dir=/usr/lib64/httpd
# make install top_dir=/usr/lib64/httpd

Configure mod_fastcgi

If you have php enabled disable it

# mv /etc/httpd/conf.d/{php.conf,php.conf.disable}

Set up a (non-existent) directory that Apache can route the requests through. That directory must be available to Apache and it might be /usr/lib/cgi-bin/ so the routed file is then e.g. /usr/lib/cgi-bin/php5-fcgi.

# mkdir /usr/lib/cgi-bin/

Configure mod_fastcgi settings in /etc/httpd/conf.d/mod_fastcgi.conf to be:

LoadModule fastcgi_module modules/mod_fastcgi.so
 
<IfModule mod_fastcgi.c>
	DirectoryIndex index.php index.html index.shtml index.cgi
	AddHandler php5-fcgi .php
	Action php5-fcgi /php5-fcgi
	Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
	FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /tmp/php5-fpm.sock -pass-header Authorization
 
	# For monitoring status with e.g. Munin
	<LocationMatch "/(ping|status)">
		SetHandler php5-fcgi-virt
		Action php5-fcgi-virt /php5-fcgi virtual
	</LocationMatch>
</IfModule>

We add handler and action which sends all requests of PHP to the virtual URL created above, which is in turn then sent to the external FastCGI server. We also add configuration to have some status information about our PHP-FPM.

Start Apache:

# service httpd start

PHP should now work.

Maven plugin to generate HTML from Markdown

Writing documentation or user guide isn’t the most interesting part of the software project but it’s still needed to be done and important for the end users. There are many ways to achieve documentation but using Markdown is good way to make it easier. After that you just need to convert it to HTML and as I couldn’t find a suitable Maven plugin to do that nicely I wrote one myself: markdown-page-generator-plugin.

Software developers are a bit lazy to write good and up to date documentation and as I needed to write the API documentation for the web application as separate HTML pages I got to know the issue first hand. I like writing documents but typing the API documentation as static HTML wasn’t very effective or fun. Fortunately it was quite easy to do it with Markdown which allows you to write documentation using an easy-to-read, easy-to-write plain text format which you can then convert to HTML. After that the only problem was to enable easy HTML generation directly from Maven Lifecycle and thus was the “Markdown to HTML Page Generator Maven Plugin” developed.

Markdown to HTML Page Generator Maven Plugin

“Markdown to HTML Page Generator” is a simple Maven Plugin which reads the input directory for Markdown files with .md suffix and outputs HTML files to target directory. There are couple of configuration options to set the input/output directories and if you want to add header and footer files to make it structurally valid HTML. The plugin uses pegdown Markdown processor.

Source code

The plugin code can be found from GitHub: markdown-page-generator-plugin.

Usage

Check out the plugin’s GitHub page for configuration options.

Using the plugin could be easier but as it isn’t in Maven repository you have to e.g. add it as Maven project to same Eclipse workspace as your project and build it. After that add it as a plugin to your build and it’s run automatically when processing sources. You can also run it from command line with “mvn com.ruleoftech:markdown-page-generator-plugin:generate”.

Setting up LAMP stack on OS X

Setting up LAMP stack for web development on OS X can be done with 3rd party software like MAMP but as Mac OS X comes with pre-installed Apache and PHP it’s easy to use the native setup. You just need to configure Apache, PHP and install MySQL.

Setup Apache2

Set up the Server Name to localhost to suppress the warning about fully qualified domain name and enable PHP module.

$ sudo vim /etc/apache2/httpd.conf
ServerName localhost:80
LoadModule php5_module libexec/apache2/libphp5.so

Create “virtual hosts” under your Sites. Change the username to your account’s username.

~$ sudo vim /etc/apache2/users/username.conf
<VirtualHost *:80>
  ServerName dev
  DocumentRoot /Users/username/Sites
  VirtualDocumentRoot /Users/username/Sites/%-2/htdocs
  UseCanonicalName Off
 
  <Directory "/Users/username/Sites/*/htdocs">
    AllowOverride All
    Order allow,deny
    Allow from all
  </Directory>
</VirtualHost>

Now Apache serves your projects from your home directory’s Sites folder. Apache will serve files from the htdocs folder like “~/Sites/projectname/htdocs”.

Now just restart Apache and check that it’s running.

$ sudo apachectl restart
$ ps aux | grep httpd

Setup PHP

$ sudo cp /etc/php.ini.default /etc/php.ini

Edit php.ini for easier debugging:

error_reporting  =  E_ALL | E_STRICT
display_errors = On
html_errors = On

Setup MySQL

MySQL can be installed directly from Oracle’s MySQL packages or by using Homebrew.

Install Homebrew

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install MySQL using Homebrew

$ brew install mysql

Install the MySQL system tables and have it run as your system user:

$ unset TMPDIR
$ mysql_install_db --verbose --user=`whoami` --basedir="$(brew --prefix mysql)" --datadir=/usr/local/var/mysql --tmpdir=/tmp

Start MySQL and check that it’s running

$ mysql.server start
$ ps aux | grep mysql

Reset the root password. Change the “5.5.27” to your installed version number.

$ /usr/local/Cellar/mysql/5.5.27/bin/mysqladmin -u root password 'YOUR_NEW_PASSWORD'

As we are using the Homebrew package for MySQL and the default php.ini file then PHP is trying to connect to MySQL through the default_socket at /var/mysql/mysql.sock which doesn’t exist as MySQL is using /tmp/mysql.sock. Just change all instances of /var/mysql/mysql.sock to /tmp/mysql.sock.

$ sudo sed -i "" "s:/var/mysql/mysql.sock:/tmp/mysql.sock:g" /etc/php.ini

And you’re done.

Monitoring Java EE application with JavaMelody

Software development is much more than just coding application by requirements and deploying it to production as the real work really starts after it has been shipped: maintenance, improvements and problem solving. And for that it’s good to have some data. It’s said “if you can’t measure it, you can’t improve it” and in ideal situations you plan for measuring your app but often you have to do it in production to detect and diagnose performance problems. There are different ways to measure a Java EE application and one good tool to achieve performance and monitoring statistics is Javamelody which has low overhead, is non-intrusive, informative and simple to setup but still very capable.

“If you can not measure it, you can not improve it.” – Lord Kelvin

Application performance can be measured by two main methods: computational resources used by the application and the performance as seen by a user of the application. By measuring these quantities we get an empirical performance baseline of the application which then can used to detect changes in performance. By using performance monitoring, which is an act of non-intrusively collect or observe performance data from running application, we get measurements to identify or isolate potential issues from real operation of an application without having a severe impact on runtime responsiveness or throughput.

There are different ways to achieve performance and monitoring statistics and it’s useful to plan for measuring application’s performance while still in development as later on you surely want to know how it is performing. You can do it by leveraging different JMX-based tools like Metrics and Servo but if you didn’t really plan for it you still have choices. You can use tools like standard JDK tools (jconsole, jstat, jmap, jstack, hprof) for low level JVM monitoring or some monitoring application like Javamelody. And as time is always scarce in development it’s useful to deploy a tool like JavaMelody which provide quick and easy access to performance monitoring. But if you are interested of some narrow focus measurement you’re better of with some custom tool.

Monitoring with JavaMelody

JavaMelody is an open source (LGPL) application to monitor Java or Java EE application servers in QA and production environments. It is a tool to measure and calculate statistics on real operation of an application depending on the usage of the application by users and is mainly based on statistics of requests and on evolution charts which can be viewed on the current day, week, month, year or custom period. The statistics can be viewed on a HTML page and sent as PDF reports by email.

JavaMelody lists the following use cases:

  • It allows to improve applications in QA and production
  • Give facts about the average response times and number of executions
  • Make decisions when trends are bad, before problems become too serious
  • Optimize based on the more limiting response times
  • Find the root causes of response times
  • Verify the real improvement after optimization

Setting up

Setting up JavaMelody is quite easy and the needed steps are covered in User guide. Integrating JavaMelody to your application can be done in less than 10 minutes, by automatic discovery of environment: it only requires to copy 2 jar files and to add 10 lines in a xml file. But for more detailed and wider measurements you need couple more lines to xml files and here are the changes I made for our Wicket, Spring, JPA, Hibernate -Java EE project.

JavaMelody and dependencies

Getting JavaMelody and it’s dependencies is easy with Maven2 and you just need to add javamelody-core and if you want to have PDF reports also iText in your pom.xml:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<!-- javamelody-core -->
<dependency>
	<groupId>net.bull.javamelody</groupId>
	<artifactId>javamelody-core</artifactId>
	<version>1.45.0</version>
</dependency>
<!-- itext, option to add PDF export -->
<dependency>
	<groupId>com.lowagie</groupId>
	<artifactId>itext</artifactId>
	<version>2.1.7</version>
	<exclusions>
		<exclusion>
			<artifactId>bcmail-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bcprov-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
		<exclusion>
			<artifactId>bctsp-jdk14</artifactId>
			<groupId>bouncycastle</groupId>
		</exclusion>
	</exclusions>
</dependency>

Monitoring application

JavaMelody needs just a monitoring filter before the description of webapp’s servlet in WEB-INF/web.xml or not even that as described in the User guide. But usually you want a little bit more than the minimum 10 additional lines in web.xml.

By adding the customResourceFilter you get customized look & feel, you can exclude some urls from statistics with url-exclude-pattern regular expression pattern, eamil-parameters are for weekly reports and by including monitoring-spring.xml you can monitor DataSource with a Spring post-processor and objects initialized with Spring.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
...
<context-param>
	<param-name>contextConfigLocation</param-name>
	<param-value>
		classpath:net/bull/javamelody/monitoring-spring.xml
		classpath:applicationContext-service.xml
		classpath:applicationContext-persist.xml
		classpath:applicationContext-web.xml
		classpath:applicationContext-security.xml
    </param-value>
</context-param>
 
<!-- Must be defined before javamelody filters -->
<filter>
	<filter-name>springSecurityFilterChain</filter-name>
	<filter-class>org.springframework.web.filter.DelegatingFilterProxy</filter-class>
</filter>
<filter-mapping>
	<filter-name>springSecurityFilterChain</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
 
<!--====================== Monitoring ===================================-->
<!-- Custom CSS -->
<filter>
	<filter-name>customResourceFilter</filter-name>
	<filter-class>net.bull.javamelody.CustomResourceFilter</filter-class>
	<init-param>
		<param-name>monitoring.css</param-name>
		<param-value>/styles/monitoring.css</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>customResourceFilter</filter-name>
	<url-pattern>/monitoring</url-pattern>
</filter-mapping>
<!-- Monitor filter -->
<filter>
	<filter-name>monitoring</filter-name>
	<filter-class>net.bull.javamelody.MonitoringFilter</filter-class>
	<init-param>
		<param-name>storage-directory</param-name>
		<param-value>logs/monitoring</param-value>
	</init-param>
	<init-param>
		<param-name>url-exclude-pattern</param-name>
		<param-value>(/images/.*|/js/.*|/styles/.*)</param-value>
	</init-param>
	<init-param>
		<param-name>admin-emails</param-name>
		<param-value>first.last@example.com</param-value>
	</init-param>
	<init-param>
		<param-name>mail-session</param-name>
		<param-value>MailSession</param-value>
	</init-param>
	<init-param>
		<param-name>mail-periods</param-name>
		<param-value>week,month</param-value>
	</init-param>
</filter>
<filter-mapping>
	<filter-name>monitoring</filter-name>
	<url-pattern>/*</url-pattern>
</filter-mapping>
<listener>
	<listener-class>net.bull.javamelody.SessionListener</listener-class>
</listener>
...

Spring Security config

For restricting the access to monitoring statistics I added the following to the Spring Security’s applicationContext-security.xml. There are also other options for security. The key point here is to make sure, that the monitoring-filter in web.xml (above) is defined after the Spring Security filter chain.

1
2
3
4
5
6
<http auto-config="true" use-expressions="true">
    ...
    <intercept-url pattern="/monitoring/**" access="hasRole('ROLE_ADMIN')" />
	<intercept-url pattern="/**"
			access="hasAnyRole('ROLE_ADMIN, 'ROLE_USER'')" />
</http>

Monitoring SQL and datasources

For monitoring datasources and SQL I just added jndi-lookup to applicationContext-persist.xml. Other options to enable JDBC monitoring are described in the User guide.

1
2
3
4
5
6
7
...
<jee:jndi-lookup id="dataSource" jndi-name="jdbc/myAppDS"/>
 
<bean id="entityManagerFactory"
	class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
	<property name="dataSource" ref="dataSource" />
...

Business facades (Spring)

If the application to monitor contains some objects initialized by Spring, EJB or Guice their methods execution can also be added to statistics. As described in the User guide for monitoring Spring Business facades there are couple of options and for example with JdkRegexpMethodPointcut in applicationContext-web.xml you can catch objects with regular expression like “all that have Service in their names”.

1
2
3
4
5
6
7
8
9
...
<bean id="facadeMonitoringAdvisor" class="net.bull.javamelody.MonitoringSpringAdvisor">
	<property name="pointcut">
		<bean class="org.springframework.aop.support.JdkRegexpMethodPointcut">
			<property name="pattern" value=".*Service.*" />
		</bean>
	</property>
</bean>
...

EHCache statistics

If you want to see also EHCache statistics add statistics=”true” to ehcache.xml config file.

1
2
3
4
5
<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://ehcache.org/ehcache.xsd">
    <defaultCache eternal="true" maxElementsInMemory="100" overflowToDisk="false" statistics="true" />
    <cache name="fooCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
    <cache name="barCache" maxElementsInMemory="1000" eternal="false" overflowToDisk="false" />
</ehcache>

Database information and statistics

JavaMelody already shows datasource and SQL information but it’s also possible to display information and statistics on the database like the longest requests in cumulative time with display of the cpu time and of the elementary cost. For showing that information the user in the database used by the monitored application must have the necessary rights to read those information and statistics.

In Oracle database the request practically means select * from v$session and you can grant access as ‘system’ user with GRANT SELECT any dictionary TO myapplicationuser. Strangely granting just the select for v$session (GRANT SELECT ON sys.v_$session TO myapplicationuser;) wasn’t enough.

Reports

There is also a possibility to get weekly, daily or monthly report in pdf format sent by email to one or several people. It needs iText library for webapp and Java’s JavaMail and Activation libraries in your server for the mail session.

With WebLogic you add some email parameters in webapp’s web.xml (above) and configure a Mail Session in WebLogic AdminServer: Services > Mail Sessions with JNDI Name and JavaMail properties. The report provides the same information you can find in monitoring web page with high and detailed level information.

Statistics’ storage

Javamelody data is stored in files on disk and there are 2 types of files: *.rrd files for values in graphics in RRD format (using jrobin library) and *.ser.gz for values in statistics.

The performance statistics are stored to temp/javamelody directory by default which means /tmp/javamelody in Linux and in Windows when running webapp in Eclipse something like Users/developer/AppData/Local/Temp/javamelody. If you want to reset the counters just delete all the files. The location can be changed with storage-directory parameter and if the name of the directory starts with ‘/’, it is considered as an absolute path, otherwise it is considered as relative to the temporary directory.

There is also option to use centralized collect server to store statistics and for monitoring of several applications.

Viewing performance monitoring

After you have set up your webapp to have the monitoring you can see statistics at URL like http:////monitoring depending your configuration.

Some example JavaMelody statistics from development:

Overhead

Monitoring and filtering doesn’t come without costs and there has been discussions on JavaMelody wiki about the overhead the monitoring does. It is said that the overhead is so low that it can be enabled continuously in Quality Assuarance environment and if no problem arises in QA, also continuously in production environment. And with little overhead you will be able to know what needs optimizing in the QA or production servers so that the overhead of JavaMelody will practically be negative.

The discussion contained some notes:

  • Architecture of JavaMelody is lightweight so it has a lower overhead as compared it to other available solutions
  • It is only statistics and not events so the overhead of memory is quite minimal.
  • It does monitoring not profiling: there is no instrumentation of classes and instead “interceptors” for http, jdbc, spring or ejb3.
  • No database and no recording of each events even in a file or over the wire: only statistics of requests are kept. The overhead of cpu is minimal with no I/O on the wire and minimal I/O on disk (just to take a backup of statistics at a regular interval). The overhead of some other good monitoring solutions is in the recording of each event in a database or in a master server.
  • You have the choice to use centralized collect server which unloads the memory, the backup storage and the generation of reports to another server while adding I/O on the wire for sending deltas of the statistics.

Summary

“If you can not measure it, you can not improve it.” It doesn’t matter how you do it but if you want to use real user data and environment just do it non-intrusively with performance monitoring tools like Javamelody.