Docker containers and using Alpine Linux for minimal base images

After using Docker for a while, you quickly realize that you spend a lot of time downloading or distributing images. This is not necessarily a bad thing for some but for others that scale their infrastructure are required to store a copy of every image that’s running on each Docker host. One solution to make your images lean is to use Alpine Linux which is a security-oriented, lightweight Linux distribution.

Lately I’ve been working with our Docker images for Java and Node.js microservices and when our stack consist of over twenty services, one thing to consider is how we build our docker images and what distributions to use. Building images upon Debian based distributions like Ubuntu works nicely but it gives packages and services which we don’t need. And that’s why developers are aiming to create the thinnest most usable image possible either by stripping conventional distributions, or using minimal distributions like Alpine Linux.

Choosing your Linux distribution

What’s a good choice of Linux distribution to be used with Docker containers? There was a good discussion in Hacker News about small Docker images, which had good points in the comment section to consider when choosing container operating system.

For some, size is a tiny concern, and far more important concerns are, for example:

  • All the packages in the base system are well maintained and updated with security fixes.
  • It’s still maintained a few years from now.
  • It handles all the special corner cases with Docker.

In the end the choice depends on your needs and how you want to run your services. Some like to use the quite large Phusion Ubuntu base image which is modified for Docker-friendliness, whereas others like to keep things simple and minimal with Alpine Linux.

Divide and conquer?

One question to ask yourself is: do you need full operating system? If you dump an OS in a container you are treating it like a lightweight virtual machine and that might be fine in some cases. If you however restrict it to exactly what you need and its runtime dependencies plus absolutely nothing more then suddenly it’s something else entirely – it’s process isolation, or better yet, it’s portable process isolation.

Other thing to think about is if you should combine multiple processes in single container. For example if you care about logging you shouldn’t use a logger daemon or logrotate in a container, but you probably want to store them externally – in a volume or mounted host directory. SSH server in container could be useful for diagnosing problems in production, but if you have to log in to a container running in production – you’re doing something wrong (and there’s docker exec anyways). And for cron, run it in a separate container and give access to the exact things your cronjob needs.

There are a couple of different schools of thought about how to use docker containers: as a way to distribute and run a single process, or as a lighter form of a virtual machine. It depends on what you’re doing with docker and how you manage your containers/applications. It makes sense to combine some services, but on the other hand you should still separate everything. It’s preferred to isolate every single process and explicitly telling it how to communicate with other processes. It’s sane from many perspectives: security, maintainability, flexibility and speed. But again, where you draw the line is almost always a personal, aesthetic choice. In my opinion it could make sense to combine nginx and php-fpm in a single container.

Minimal approach

Lately, there has been some movement towards minimal distributions like Alpine Linux, and it has got a lot of positive attention from the Docker community. Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox using a grsecurity/PaX patched Linux kernel and OpenRC as its init system. In its x86_64 ISO flavor, it weighs in at an 82MB and a container requires no more than 8 MB. Alpine provides a wealth of possible packages via its apk package manager. As it uses musl, you may run into some issues with environments expecting glibc-like behaviour (for example Kubernetes or with compiling some npm modules), but for most use cases it should work just fine. And with minimal base images it’s more convenient to divide your processes to many small containers.

Some advantages for using Alpine Linux are:

  • Speed in which the image is downloaded, installed and running on your Docker host
  • Security is improved as the image has a smaller footprint thus making the attack surface also smaller
  • Faster migration between hosts which is especially helpful in high availability and disaster recovery configurations.
  • Your system admin won’t complain as much as you will use less disk space

For my purposes, I need to run Spring Boot and Node.js applications on Docker containers, and they were easily switched from Debian based images to Alpine Linux without any changes. There are official Docker images for OpenJDK/OpenJRE on Alpine and Dockerfiles for running Oracle Java on Alpine. Although there isn’t an official Node.js image built on Alpine, you can easily make your own Dockerfile or use community provided files. When official Java Docker image is 642 MB, Alpine Linux with OpenJDK 8 is 150 MB and with Oracle JDK 382 MB (can be stripped down to 172 MB). With official Node.js image it’s 651 MB (or if using slim 211 MB) and with Alpine Linux that’s 36 MB. That’s a quite a reduction in size.

Examples of using minimal container based on Alpine Linux:

For Node.js:

FROM alpine:edge
 
ENV NODE_ALPINE_VERSION=6.2.0-r0
 
RUN apk update && apk upgrade \
    && apk add nodejs="$NODE_ALPINE_VERSION"

For Java applications with OpenJDK:

FROM alpine:edge
ENV LANG C.UTF-8
 
RUN { \
      echo '#!/bin/sh'; \
      echo 'set -e'; \
      echo; \
      echo 'dirname "$(dirname "$(readlink -f "$(which javac || which java)")")"'; \
   } > /usr/local/bin/docker-java-home \
   && chmod +x /usr/local/bin/docker-java-home
 
ENV JAVA_HOME /usr/lib/jvm/java-1.8-openjdk
ENV PATH $PATH:$JAVA_HOME/bin
ENV JAVA_VERSION 8u92
ENV JAVA_ALPINE_VERSION 8.92.14-r0
 
RUN set -x \
    && apk update && apk upgrade \
    && apk add --no-cache bash \
    && apk add --no-cache \
      openjdk8="$JAVA_ALPINE_VERSION" \
    && [ "$JAVA_HOME" = "$(docker-java-home)" ]

If you want to read more about running services on Alpine Linux, check Atlassian’s Nicola Paolucci’s nice article about experiences of running Java apps on Alpine.

Go small or go home?

So, should you use Alpine Linux for running your application on Docker? As also Docker official images are moving to Alpine Linux then it seems to make perfect sense from both a performance and security perspectives to switch to Alpine. And if you don’t want to take the leap from Debian or Ubuntu or want support from the downstream vendor you should consider stripping it from unneeded files to make it smaller.

Notes from Tampere goes Agile 2015

What could be a better way to spend a beautiful Autumn Saturday than visiting Tampere goes Agile and being inspired beyong agile. Well, I can think couple of activities which beat waking up 5:30 to catch a train to Tampere but attending a conference and listening to thought provoking presentations is always refreshing. So, what did they tell about being “Inspired beyond agile” in Tampere goes Agile 2015?

Tampere goes Agile 2015

Tampere goes Agile is a free to attend event about agile and this year the theme of the conference was “inspired beyond agile”. The event was held at Sokos Hotel Ilves and there were roughly 140 attendees. Agile as a topic isn’t interesting as it’s practices are widely in use, so the event went past agile and concentrated on “being agile”. How the organizational level and our mindsets has to change to make agile work. Waterfall mindset eats your agile culture for breakfast. And that’s the problem many presentations addressed.

Juho Vepsäläinen wrote also a great blog post about afterthoughts of Tampere Goes Agile 2015. It was nice to read a recap of the sessions which were in the other room than I was.

Keynote: After Agile

Event started with a keynote by Bob Marshall who asked what’s after agile. He has introduced the concept of right shifting where the core idea is that a large amount of organizations are underperforming. We’re always more or less prisoners of our mindset and existing ways.

“It’s not enough to do your best; you must know what to do, and then do your best.” – W. Edwards Deming

Marshall showed his right shifting organizational effectiveness chart where mean is around 1 (0 to 5 scale) and organizations using agile sit around 1.25 to 2. So, what’s beyond that? Agile thinking isn’t getting us there. What differs the organizations in the chart is their mindset: adhoc, analytic, synergistic and finally chaordic.

The Marshall model:

The Marshall model

In order to improve the effectiveness and efficiency of our organizations, we’ll need to be able to imagine better ones. The question is, what does an ideal organization look like? What kind of society we would build if it was wiped out? Starting from clean slate. We should look at the organization as a whole and what Marshall suggest is to use therapy to understand organization health and changing the mindset of the organization to one that’s more conducive for high performance.

The ideal model for IT company is built around: people, relationships between people, collective mindset, cognitive function and motivation. And it’s good to remember the difference between effectiveness and efficiency: Doing the right thing or doing the thing right.

Doctor, please fix my Agile!

Ville Törmälä talked about how we have seen changes on the method level, organizations are still mostly functioning the same ways as before. Many have tried to become more agile but without much success as there’s a waterfall way of doing everything.

Törmälä presented his definition of agile: 1) Make the work better 2) Make the work work better 3) Make lives better. But waterfall mindset eats our agile culture for breakfast so it’s about time to broaden our thinking about what really constitutes a long-term success in organizations doing any kind of knowledge work. Agile gives you tools and ideas but organizations can’t change or improve by “doing agile” better. If you fail with one agile “method”, you probably fail with the rest of them. It’s a systemic problem. It’s all built deep in to the thinking and structures of the organizations. That is the challenge.

“Every system is perfectly designed to achieve the results it gets”. We should change from “project thinking” to “stable teams thinking”. To change the power and influence structure from managing people to empowering people and further to liberating people.

One way of doing this is to use KBIs, Key Behaviour Indicators, where you write down examples of behaviour you want to see, think in what kind of environmental it’s possible or could happen and then create the environment, write down concrete actions.

“The supreme art of agile is to subdue the waterfall thinking without fighting” – Sun Tzu, The Art of War

In summary, we need to look beyond methods and practices. Organizations change by changing how they think and become better by understanding better how work works, how to create value and how to learn better. We’ve to work with the system, aiming to understand and affect its thinking.

Pairing is sharing

Pair programming is a core agile technical practice but many people still have reluctance to pair and Maaret Pyhäjärvi talked about the deliberate practice in building up the skill of pairing to allow pairing to take one’s skills on other activities to a new level. Pyhäjärvi shared her different stages of pairing and lessons picked up as a testing specialist.

Again, pairing is also about mindset and effective pairing is far from trivial – but it is skill that can be practiced. Pyhäjärvi talked about growth patterns from pairing with peers to pairing and mobbing with developers, from traditional style and side-by-side work to strong-style pairing and to pairing on both testing and programming activities.

Mindset: fixed <> growth

Listeners also got to test specific style of pairing, Strong-style pairing, where for an idea to go from your head to the computer it must go through someone else’s hands. You really need to think the steps through for the other to manage the given task.

One presented point about pairing was that you must unlearn ownership of ideas and contributions. Co-creation vs. collaboration.

Pyhäjärvi also told that selling pairing to team (of introvert programmers) is hard but Mob Programming has been their gateway to pair programming. It feels safer. You can read more about it from Mob Programming guide book.

Beyond Continuous Deployment: Documentation Pipeline

Before lunch there was also nice lightning talk about documentation pipeline by Antti Virtanen. He told about Lessons learnt from creating a Documentation Pipeline for Continuous Deployment with Jenkins and other open source tools. His slides are available from SlideShare.

1 ???
2 Continuous Delivery DevOps magic
3 ???
4 Profit

The DevOps magic with Jenkins was more or less standard practice and it was configured to generate documentation from database schema, JavaDocs, test coverage reports, performance test results and API specification. Reminded me of all the work I should introduce to our continuous integration.

3 standard tricks were presented:

  • Jenkins is the Swiss knife.
  • Database documentation in database metadata and generating ER-diagrams with SchemaSpy.
  • API documentation with Swagger.

When quality is just a cost: Useful approaches to testing

Testing is also important part of successful projects so Jani Grönman talked about useful approaches to testing and software quality.

“Software quality is measured by your customer success, not development project metrics and quality processes.”

Grönman approached the topic with often surprisingly common attitude towards testing and quality:

  • “Quality is just a cost and like other costs, it should be avoided or minimized.”
  • “Testing is it just another buffer in project’s budget”
  • “testers are not skilled labor, it’s enough if they can read and write.”
  • “What automation? They can quickly click trough the app can’t they?”.

And as you know this is all wrong. It’s true that testing is expensive but so is development. Can you afford not to test? You should think it as an investment. The presentation went through the reasons and motivations behind the various attitudes and explored differences in views and how to best tackle them using the right technology and approach. He also talked about the schools of testing: Analytical, Standard, Quality, Context driven, Agile.

But overall you should know that testing is skilled activity and part of the development. Testing provides information to the project and you should use mix of techniques like exploratory and automatisation. And think about what testing would be most effective now. You need to choose the right set of QA tools for the job. One size fits no-one.

DevOps: Boosting the agile way of working

DevOps has been quite the buzzword for some time, so it was interesting to hear what Timo Stordell had to say how Devops is boosting the agile way of working. In short DevOps isn’t anything revolutionary and should be seen an incremental way to improve our development practices. And talking about revolution, Stordell’s slides had nice Soviet theme.

The presentation was more or less what you would expect from a topic covering DevOps and has nice touch to it. In short: Small bangs over a big bang, requirements management meet acceptance testing, standardize development environments, monitor to understand what to develop.

Stordell had nice demo of how they perform acceptance testing using physical devices and automation. They have built a rig of CNC mill run by Raspberry Pi to test payment system.

For those interested about DevOps movement and everything around it there’s DevOps Finland meetup group. You can also download Eficode’s DevOps Quick Guide to read more about it.

Keynote: Beyond projects

Event’s final keynote was by Allan Kelly who spoke about #noprojects. Why projects are wrong and what to do instead. The main point of the keynote was that the project model doesn’t match software development and outlined an alternative to the project model and what companies need to do to achieve it. The presentation slides are available from SlideShare. Kelly has also written a book about team centric agile software developmen,: Xanpan, which combines Kanban and XP.

Going beyond projects is an interesting idea as everything we do is somewhat tied to doing things in projects. So, what’s wrong with projects? Projects are temporary whereas software is forever. Projects have end dates which in turn is against the defining feature of successful software: it doesn’t end. Software which is useful is used and demands change, stop changing it and you kill it. At worst the project metaphor leads to dead software, higher costs and missed business opportunities.

We should think projects more like a continuous flow where it’s success isn’t determined by staying on schedule, on budget, and with quality. We should concentrate on the value delivered and put value in flexibility as requirements change. This goes against the fixed nature of projects. Also after project you often break a functioning team and start all over again. We should put emphasis on teams, treat team as an unit and push work through it.

The other thing is that software is not milk. It’s cheapest in small packages, not in big cartons. Software development has not economics of scale. Big projects are risk. Think small and make regular delivery which increases ROI. Fail fast, fail cheap. Quite basic agile thinking.

So, beyond projects: waterfall 2.0, continuous flow

Continuous flow of waterfall

Now we have #noestimates, #nomanagement and #noprojects. Profit?

Summary

It was my first time visiting Tampere goes Agile and it was nice conference. The topics provided something to think about and not just the same agile thinking. You could clearly see the theme “Inspired beyond agile” working through different presentations and the emphasis was about changing our mindsets.

Going beyond agile isn’t easy as it’s more about thinking than tools. Old habits die hard and changing the waterfall way of thinking isn’t trivial. We should start with understanding our organization’s health and changing the mindset of the organization to one that’s more conducive for high performance. Switch from “project thinking” to “stable teams thinking” and change the power and influence structure from managing people to empowering people and further to liberating people.

The after party was at Ruby & Fellas but after early morning and couple of nice beers it was time to take train back home. But before that I had to visit the Moro Sky Bar with nice scenery over Tampere.

Tampere from Moro Sky Bar

Newsletters for software developers

Software development is one of the professions where you just have to keep your knowledge up to date and follow what happens in the field. But as normal information overload is easily achieved so it’s beneficial to use for example curated newsletters for the subjects which intersects the stack you’re using and topics you’re interested at. Here are my selection of newsletters for software developers covering topics like web and mobile development, user experience and design and general topics. For more newsletters for developers you can check what for example Dzone wrote.

The power of newsletter lies in the fact that it can deliver condensed and digestible content which is harder to achieve with other good news sources like feed subscriptions and Twitter. Well curated newsletter to targeted audience is a pleasure to read and even if you forgot to check your newsletter folder, you can always get back to them later :)

General

Hacker Newsletter
Weekly newsletter of the best articles in Hacker News.

Status code
A language agnostic roundup of the latest ideas, releases, trends, events and must-read articles from the programming world (think C, UNIX, algorithms, editors, protocols)

Mobile development

iOS Dev Weekly
Hand picked round up of the best iOS development links published every Friday.

This Week In Swift
List of the best Swift resources of the week.

iOS Dev nuggets
Short iOS app development nugget every Friday/Saturday. Short and usually something you can read in a few minutes and improve your skills at iOS app development.

In depth Mac and iOS articles archives

Java

Java Web Weekly by Baeldung
Once-weekly email roundup of Java Web curated news by Eugen Baeldung.

The Java Specialists’ Newsletter
A monthly newsletter exploring the intricacies and depths of Java, curated Dr. Heinz Kabutz.

Java Performance Tuning News
A monthly newsletter focusing on Java performance issues, including the latest tips, articles, and news about Java Performance. Curated by Jack Shirazi and Kirk Pepperdine.

Database

DB Weekly
A weekly round-up of database technology news and articles covering new developments, SQL, NoSQL, document databases, graph databases, and more.

HTML and CSS

HTML5Weekly
Weekly HTML5 and Web Platform technology roundup. Curated by Peter Cooper.

CSS Weekly
Roundup of css articles, tutorials, experiments and tools. Curated by Zoran Jambor.

Web development

Web Development Reading List
Weekly roundup of web development–related sources, selected by Anselm Hannemann.

Versioning
SitePoint’s daily newsletter, which features the latest web development news.

Hacking UI
Newsletter for designers, front-end developers and product managers.

Scott Hanselman
Includes interesting and useful stuff Scott has found over the last few weeks and other wonderful things.

The Modern Web Observer
Biweekly email newsletter about current issues and trends in front-end web development. It is much like a commentory on the important current news and articles related to front end development.

Web Design Weekly
Links to the best news and articles to hit the interweb during the week.

MergeLinks
Weekly email of curated links to articles, resources, freebies and inspiration for web designers and developers.

For front-end developers there’s “How to keep up to date on
Front-End Technologies”
page which lists newsletters, blogs and people to follow.

JavaScript

JavaScript Weekly
Weekly e-mail round-up of JavaScript news and articles. Curated by Peter Cooper.

A Drip of JavaScript
“One quick JavaScript tip”, delivered every other Tuesday and written by Joshua Clanton.

SuperHero.js
Collection of the best articles, videos, and presentations on creating, testing, and maintaining a JavaScript code base.

Node Weekly
Once–weekly e-mail round-up of Node.js news and articles.

User experience and design

UX weekly
Five links each week with the best UX writing, process, analysis, and critique from around the web. Its content lies at the intersection of user experience design, game design, and tech industry critique.

GoodUI
Monthly newsletter where the author will share ideas on how to improve customer conversion and ease of use.

Sidebar.io
To satisfy your web aesthetics with list of the 5 best design links of the day. The content is manually curated by a couple great editors.

Userfocus
Updates you monthly about the happenings in the UX/usability arena.

UX Design Weekly
Best user experience design links every week, published every Friday.

Ops

DevOps Weekly
Weekly slice of devops news.

Web Operations Weekly
Weekly newsletter on Web operations, infrastructure, performance, and tooling, from the browser down to the metal.

Microservice Weekly
Weekly newsletter of articles regarding microservices.

Build secure Web applications by reading Iron-Clad Java

Building secure Web applications isn’t easy and contains many aspects that the development team has to consider and take into account. “Iron-Clad Java: Building Secure Web Applications” book is good starting point to learn concepts, tactics, patterns and anti-patterns to develop, deploy and maintain secure Java applications. With 304 pages the book is more about getting an overview and pointers for further reading and research but works quite nicely in that regard.

“Iron-Clad Java: Building secure Web applications”

As the name suggests, “Iron-Clad Java: Building Secure Web Applications” by Jim Manico and August Detlefsen, is targeted for Java developers and is suitable reading also for less technical people in the team like project managers and managers as it doesn’t go too deeply to technical aspects or code. After reading the book even the managers should get an appreciation and the vocabulary to discuss security with their staff. The reader should get a solid understanding of what is wrong with many web apps in general and what corrective measures to take to mitigate the issues. The book was published September 2014 and has 304 pages (ISBN-13: 978-0071835886).

The book covers topics like secure authentication and session management processes, access control design, defending against cross-site scripting (XSS) and cross-site request forgery (CSRF), protecting sensitive data while stored or in transit, preventing SQL injection, ensuring safe file I/O and upload, using effective logging, error handling, and intrusion detection methods and also guide for secure software development lifecycle (secure-SDLC). The topics are written with theory and practice and so that they are approachable for developers new to security, for those that might be a little inexperienced but still providing useful nuggets for experienced developers.

In good and bad the book gives somewhat opinionated answers what technics and tools you can use to address security issues but overall the advice is solid and un-biased and more or less framework agnostic. So, the lessons learned should apply to any project. For me, writing examples with e.g. JSP and Struts makes me question if also the other tools the book suggest (which I wasn’t familiar with) are suitable for modern Java EE application development. Something to look into after reading the book. Also OWASP seemed to have answer to almost every security aspect.

Anyways, the book’s advice isn’t about using certain technologies but giving you something to think about and educating about security aspects in your Java Web application. What matters is that the book gives explanations of why you need to implement a specific control for a given issue, how you could do it and what the impacts are. This is what many security professionals miss when speaking to developers. The book tells you what the security problem is and then why and how you should fix that so it makes sense for developers.

Taking care for Web application security isn’t just for architects and developers but it’s also a topic which importance whole team should know and understand. The “Iron-Clad Java: Building secure Web applications” gives good overview to security and is suitable for the whole development team to read.

Book: Real World Java EE Night Hacks

Reading software development related books can be said to be unnecessary as all the information can be also found from the Internet but sometimes it’s easier to read all the related topics from one place. Adam Bien’s “Real World Java EE Night Hacks: Dissecting the Business Tier” is a book which walks through best practices and patterns used to create a Java EE 6 application and covers several important topics from architecture to performance and monitoring to testing. The book has 167 pages with source code so the topics are more about getting the idea than explaining them thoroughly. So if you’re new to Java EE 6 and patterns this book is for you. It gets you started and gives you topics to research more.

Real World Java EE Night Hacks

“Real World Java EE Night Hacks” walks through best practices and patterns used to create a real world Java EE application called “X-ray.” It’s a high-performance blog statistics application add-on for Apache Roller which is built with “vanilla” Java EE 6. It tells you about the core principles of Java EE like EJB 3.1, CDI, JPA, JTA, JAX-RS, Dependency Injection, Convention over Configuration, interceptors, transactions and binds them in “X-ray” application with source code to follow. The book is also more than just Java EE as it covers concepts like unit and integration testing, performance measuring and monitoring, continuous integration, real-time monitoring and timers and batch processing.

The book is easy to read although it isn’t for beginners as it requires you to know the Java jargon and main topics of Java EE. The book covers all the important topics regarding what you would need to know when building Java EE application but doesn’t explain or cover them thoroughly. It’s understandable as you would need more than one book to go them all through in sufficient detail. It’s more about telling you that there are this kinds of things to consider and how to apply them with Java EE application. It’s a starting point for your own research. It would’ve been also nice to have more pictures and diagrams in it.

In overall, the “Real World Java EE Night Hacks” is a decent book about implementing Java EE concepts and application architecture with best practices and patterns but it still feels a bit meager especially as the example isn’t an application you would first think of Java EE application to be.

Setting up Bower and Gulp in Windows

Doing things manually is fine once but if you can automate things it’s better. With little tools you can speed up development and reduce recursive mundane tasks such as starting a project or setting up boilerplate code. I recently came across Bower which is a package manager for the web. With Bower you can fetch and install packages from all over, and it takes care of finding, downloading, and saving the stuff you’re looking for. The other interesting tool to help you get going is Gulp which enables you to automate and enhance your workflow. Let’s see how to put things together on Windows, nothing special just steps to get you started.

Gulp tasks

Install Git

Bower needs Git to work so first install Git if you don’t have it. I chose Git for Windows which gives you BASH emulation used to run Git from the command line, graphical user interface for using Git and Shell integration.

Just click through the installation.

Install Node.js

Bower depends on Node.js and NPM so you need to get Node.js. Just download the installation package from Node.js site and click through it.

Install Bower

After you have Node.js installed we can install Bower with npm. You might need to restart your Windows to get all the path variables setup so Npm can find them.

Open up the Git Bash or Command Prompt and Bower is installed globally by running the following command.

$ npm install -g bower

Once you have Bower installed you then can install packages and dependencies using these commands:

# Using a local or remote package
bower install <package>
 
# Using a specific version of a package
bower install <package>#<version>
 
# Search packages
$ bower search <package>

By default packages will be put in the bower_components directory which can be changed if you prefer. If you want your packages downloaded into js/libs you can achieve this by creating a .bowerrc file

.bowerrc

{
    "directory": "js/libs"
}

You can also create a bower.json file which allows you to define the packages needed along with dependencies and then simply run bower install to download packages. In our example we setup a simple Backbone.js application which uses Bootstrap.

bower.json

{
    "name": "Foobar",
    "version": "0.1.0",
    "dependencies": {
          "jquery": "~2.0.3",
          "underscore": "~1.5.0",
          "bootstrap": "~3.3.2",
          "backbone": "~1.1.2"
    }
}

Our bower.json describes that we want some JavaScript libraries and as we have defined the version with ~ it can have bigger minor versions, e.g. jquery version can be between 2.0.3 < 2.1.0. Read more about semantic versioner for npm.

Now after creating that file inside the app directory you can run the following command:

$ bower install

After that you should see all your JavaScript packages under bower_components folder.

Install Gulp

To automate and enhance your workflow you can use Gulp for example to copy the files where you want them. There are nice recipes to show how to benefit of Gulp.

Install Gulp globally with npm:

$ npm install --global gulp

Install Gulp also in your project devDependencies:

$ npm install --save-dev gulp

Now we can setup our Gulp dependencies which pull from npm. Create a new package.json file in your project root and just add an empty object, {} and save it.

Next we install gulp-bower plugin which we can use to install Bower packages.

$ npm install --save-dev gulp-bower

This will install all the needed dependencies in a node_modules folder and also automatically update our package.json file with these dependencies.

Finally we need to setup the gulpfile.js which defines our tasks we want to perform. First we define what we installed in npm step above and create a config object to hold various settings. The bowerDir is just the path to the bower_components. Finally we add task for running bower and default task. Our bower tasks basically runs bower install but by including in the gulpfile other contributors only have to run gulp bower and have them all setup and ready.

gulp.js

var gulp = require('gulp'),
    bower = require('gulp-bower');
 
var config = {
     bowerDir: './bower_components'
}
 
gulp.task('bower', function() {
    return bower()
        .pipe(gulp.dest(config.bowerDir))
});
 
$ gulp.task('default', ['bower']);

The default task runs the bower task and all the user has to do to setup the needed packages is to run

$ gulp

In our case running gulp just runs our bower task which downloads the JavaScript packages we need. Pretty simple.

Gulp is powerful tool and has many use cases but also needs some to get all things working like you want and even then you might need to make compromises. One crafty task for Gulp and Bower is to customize your Bootstrap theme. Also Mark Goodyear has written good article about Getting started with gulp which shows some typical use cases.

Essential IntelliJ IDEA keyboard shortcuts

Recently I switched from using Eclipse to IntelliJ IDEA as our Java EE application’s front-end was done with JavaScript and the support for front-end technologies in Eclipse is more or less non-existent. The switch for long time Eclipse user wasn’t easy as IDEA works a bit differently but the change was worth it. The biggest difference in daily work with IDE is the shortcuts which are quite different in IDEA. In theory you can use Eclipse keymap for shortcuts but it just doesn’t work like it should and in practice you have to learn the IDEA way. There are many posts in the Internet about keyboard shortcuts in IDEA but there’s always place for more :) So, here’s my list of shortcuts to keep in your finger memory.

Learn keymap with Key Promoter

To learn your way around IntelliJ IDEA’s keyboard shortcuts there’s nice “Key Promoter” plugin to train yourself. It prompts whenever you use the mouse when you could’ve used the keyboard instead (similar to Eclipse’s Mousefeed).

To install the plugin:

  1. Ctrl+Alt+S to pull up the Settings screen
  2. Filter on “plugin”. Click “Plugins”, then “Browse Repositories” at the bottom
  3. Filter on “key promoter”
  4. Double click to install
  5. Essential IntelliJ IDEA keyboard shortcuts

    IntelliJ IDEA keymap

    You may be tempted to just go with the Eclipse keymap but it’s better to learn the IDEA way although it’s quite irritating at start. You also should change some default IDEA keyboard shortcuts to better ones like “closing editor window” with Ctrl+F4 which is too cumbersome compared to the de facto Ctrl+W. And changing “comment current line or selection” with Ctrl+/ which is impossible with Finnish keyboards to Ctrl+7.

    If you want to know how Eclipse shortcuts map to IDEA there’s nice post about IntelliJ IDEA shortcuts for Eclipse users and I added some in my list.

    Recent Viewed or edited Files: CTLR + E / CTRL + SHIFT + E
    Shows you a popup with all the recent files that you have opened or actually changed in the IDE. If you start typing, you can filter the files.

    Go to Class or file: CTRL + N and CTRL + Shift + N
    Allows you to search by name for a Java file in your project. If you combine it with SHIFT, it searches any file. Adding ALT on top of that it searches for symbols. (Eclipse: Ctrl+Shift+T and Ctrl+Shift+R)

    Find and Replace in Path: CTRL + SHIFT + F / CTRL + SHIFT + R
    Allows you to find in files or replace in files. (Eclipse: Ctrl+H)

    Incremental Find: F3 / CTRL + L
    When using CTLR-F to find in current file the F3 lets you to loop through the results. (Eclipse: Ctrl+K)

    Delete line: CTRL + Y
    Delete current line under cursor. Yank it. (Eclipse: Ctrl+D)

    Goto line: CTRL + G
    Go to given line number. (Eclipse: Ctrl-L)

    Syntax Aware Selection: CTRL + W
    Allows you to select code with context. Awesome when you need to select large blocks or just specific parts of a piece of code.

    Complete Statement: CTRL + SHIFT + ENTER
    This will try to complete your current statement. How? By adding curly braces, or semicolon and line change.

    Smart Type Completion: CTRL + SHIFT + SPACE
    Like auto complete (CTRL + SPACE) but if you add a SHIFT you get the smart completion. This means that the IDE will try to match expected types that suit the current context and filter all the other options.

    Reformat source code and optimize imports: CTRL + ALT + L
    Allows you to reformat source code to meet the requirements of your code style. Lays out spacing, indents, keywords etc. Reformatting can apply to the selected text, entire file, or entire project.

    Quick Fix: Alt+Enter
    (Eclipse: Ctrl+1)
    Gives you a list of intentions applicable to the code at the caret.

    Paste one of the previous values from clipboard: CTRL + SHIFT + V
    Shows you a dialog to select previous value from the clipboard to be pasted.

    Comment or uncomment line or block: Ctrl+7 / Ctrl+Shift+7
    Allows you to comment or uncomment the current line or selected block of source code. This is originally Ctrl + / (Slash) which is impossible with Finnish keyboard layouts.

    Show Diff (in Changes): CTRL + D
    In Changes tab compares the file with latest repository version.

    Highlight Usages: CTRL + SHIFT + F7
    Place the cursor in a element and after pressing the cursor the IDE will highlight all the occurrences of the selected element.

    Go to Declaration: CTRL + B
    If you place the cursor in a class, method or variable and use the shortcut you will immediately jump to the declaration of the element.

    Navigate Between Methods: ALT + UP/DOWN Arrows
    Jump between methods.

    Refactoring String Fragments: CTRL + ALT + V
    Refactor hardcoded string into variable/field/constant. Select the section of the String you want to extract, and use the normal “Extract…” shortcuts to extract it into a variable.

    Other useful keyboard shortcuts

    There are many useful keyboard shortcuts and you can print them from Help > Default Keymap Reference. Here are some more shortcuts which are also handy.

    Update application: CTRL + F10
    Current file structure: CTRL + F12
    Bookmarks: F11 and SHIFT + F11
    Generate Getters & Setters (in editor): ALT + INSERT
    Create New _* (in project navigator): ALT + INSERT
    Refactor – Rename: SHIFT + F6
    Open Settings CTRL + Alt + S
    Duplicate line: CTRL + D
    Move line: CTRL + ALT + UP / DOWN
    Find Command: CTRL + SHIFT + A
    Show usages in a pop-up list: CTRL + Alt + F7
    Extract Variable/Method/Constant/Field: CTRL + ALT + V/M/C/F
    Quick JavaDoc Popup: CTRL + Q
    Tab switcher: CTRL + TAB
    Jump to Project Navigator: ALT + 1
    Jump back to last tool window/panel: F12
    Jump to beginning/end of block (e.g., method start/end): CTRL + [ and CTRL + ]
    Toggle uppercase/lowercase of selection: CTRL + SHIFT + U
    Toggle collapse/expand: CTRL + .
    Toggle full screen editor (hide other tool windows): CTRL + SHIFT + F12

    Not a keyboard shortcut exactly but the “iter” smart template is great. If you want to iterate though something using a for loop type “iter” then TAB to use the live template. It will figure out the most likely variable you want to iterate over and generate a for loop for it. In Eclipse it worked more logically with just typing for and then autocomplete.

Distribute project’s artifacts in Maven Central with OSSRH

You have developed some crafty Java library or Maven plugin and now you want to distribute it to other users through Maven Central repository? Using Sonatype Open Source Software Repository Hosting Service is a nice way for open source projects to achieve that and there’s two options to get your artifacts in it: 1. Release with Sonatype’s process or 2. Use Bintray’s process. In short, both options end up with the same results, your project’s artifacts are in Maven Central repository, but using Sonatype’s process and Maven release plugin makes it easy to release new version after you have set it up.

Using Sonatype’s service for pushing artifacts to Maven Central

Sonatype Open Source Software Repository Hosting Service (OSSRH) provides Maven repository hosting service for open source projects. You can deploy snapshots, stage releases, and promote your releases so they will be synced to Maven Central.

The process with Sonatype has a bit more steps than Bintray but if you follow the user guide, quite easy. And now as they have made some changes to their release process it’s easier more or less the same than using Bintray. All you need to do is to sign up a Sonatype JIRA account, create a JIRA ticket and make some POM/settings configuration and then use Maven release plugin to perform release. The procedure is described in Sonatype OSS Maven Repository Usage Guide but here is a short recap.

Performing release deployment with the Maven release plugin

Using Maven to perform release deployment to OSSRH is described in Sonatype’s guide which tells you how to do it manually or through the Maven release plugin which I find convenient.

First start with Initial Setup and create your JIRA account and new project ticket. The new project ticket triggers creation of your repositories. While your repositories are taken care of you should review the requirements for components in Central.

The artifacts need to be GPG signed so if you yet haven’t got a GPG key follow this guide to create one. Then you need to have your GPG key in a keyserver so start by sending your GPG key to keyserver:

gpg --keyserver pool.sks-keyservers.net --send-keys <your KEYID>

If you have more than one GPG key in your keychain then it’s useful to set a default key to be used for signing. Edit your ~/.gnupg/gpg.conf and uncomment the “default-key KEYID” line and add your KEYID.

Your Maven project settings

Configure your Maven settings.xml to have Sonatype OSSRH credentials (in .m2/settings.xml).

<settings>
     ...
    <servers>
    <server>
      <id>ossrh</id>
      <username>your-jira-id</username>
      <password>your-jira-pwd</password>
    </server>
  </servers>
     ...
</settings>

Edit your’project’s pom.xml for Maven release

Add the distributionManagement section to point to OSSRH.

<distributionManagement>
	<snapshotRepository>
		<id>ossrh</id>
		<url>https://oss.sonatype.org/content/repositories/snapshots</url>
	</snapshotRepository>
	<repository>
		<id>ossrh</id>
		<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
	</repository>
</distributionManagement>

The configuration for the Maven release plugin should include disabling the release profile that is part of the Maven Super POM, since we are using our own profile, and specify the deploy goal together with the activation of our release profile

<pluginManagement>
          <plugins>
                <plugin>
                     <groupId>org.apache.maven.plugins</groupId>
                     <artifactId>maven-release-plugin</artifactId>
                     <version>2.5</version>
                     <configuration>
                          <useReleaseProfile>false</useReleaseProfile>
                          <releaseProfiles>release</releaseProfiles>
                          <goals>deploy</goals>
                     </configuration>
                </plugin>
          </plugins>
</pluginManagement>

Add the nexus-staging plugin to define where’s Nexus and if we release it right after closing.

<plugin>
	<groupId>org.sonatype.plugins</groupId>
	<artifactId>nexus-staging-maven-plugin</artifactId>
	<version>1.6.3</version>
	<extensions>true</extensions>
	<configuration>
		<serverId>ossrh</serverId>
		<nexusUrl>https://oss.sonatype.org/</nexusUrl>
		<autoReleaseAfterClose>true</autoReleaseAfterClose>
	</configuration>
</plugin>

Add the release profile for signing with GPG & Javadoc and Sources Attachments

<profile>
	<id>release</id>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-source-plugin</artifactId>
				<version>2.2.1</version>
				<executions>
					<execution>
						<id>attach-sources</id>
						<goals>
							<goal>jar-no-fork</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-javadoc-plugin</artifactId>
				<version>2.9.1</version>
				<executions>
					<execution>
						<id>attach-javadocs</id>
						<goals>
							<goal>jar</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-gpg-plugin</artifactId>
				<version>1.5</version>
				<executions>
					<execution>
						<id>sign-artifacts</id>
						<phase>verify</phase>
						<goals>
							<goal>sign</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
</profile>

You can see the full pom.xml e.g from my markdown-page-generator-plugin’s pom.xml.

Publishing Snapshots

Snapshot deployment are performed when your version ends in -SNAPSHOT. You do not need to fulfil the requirements when performing snapshot deployments and can simply run

$ mvn clean deploy

Successfully deployed SNAPSHOT versions will be found in Snapshot repository. If you need to delete your artifacts, you can log in to Sonatype’s Nexus using the same credentials you use to access to the Sonatype JIRA.

Performing a Release Deployment

The Maven Release Plugin can be used to automate the changes to the Maven POM files, sanity checks, the SCM operations required, the actual deployment execution and you can perform a release deployment to OSSRH with the following steps.

Prepare the release by answering the prompts for versions and tags

$ mvn release:clean release:prepare

If prepare fails use

$ mvn release:rollback
$ mvn release:clean

The prepare will create a new tag in SCM, even in GitHub, automatically checking in on your behalf. For it to work you need to have working public key in GitHub for git-push.

Perform the release. The perform process will ask for your gpg.passphrase if you don’t give it with passphrase argument.

$ mvn release:perform

For some reason I can’t provide the GPG passphrase when Maven GPG plugin asks it and I have to use the -Dgpg.passphrase argument.

$ mvn -Darguments="-Dgpg.passphrase=&lt;your GPG key passphrase&gt; release:perform

This execution will deploy to OSSRH and release to the Central Repository in one go, thanks to the usage of the Nexus Staging Maven Plugin with autoReleaseAfterClose set to true.

Now your artifacts are in the Releases repository. The updates to The Central Repository search can take up to two hours. Once your artifacts are released you will not be allowed to update or delete them.

The first time you promote a release, you need to comment on the OSSRH JIRA ticket you created so OSSRH can know you are ready to be synced.

Manually Releasing the Deployment to the Central Repository

If you don’t want to release artifacts automatically you can also do it manually. Set the autoReleaseAfterClose set to false so you can inspect and potentially release the deployed artifacts manually. Then to release your artifacts is done through Nexus as described in the guide.

Using Bintray for pushing artifacts to Maven Central

Bintray offers developers the fastest way to publish and consume OSS software releases. Whether you are distributing software packages or downloading ones”. In short it provides i.a. an alternative way to release artifacts to Maven Central and to Sonatype Open Source Software Repository.

I found out about Bintray after I had already used the Sonatype’s way to release artifacts so I haven’t tested this myself. Bintray’s process is more or less similar than Sonatype’s but if I hadn’t read the blog post about how to do it I wouldn’t have know where to start. In short the process is the following.

  1. Register to Bintray and set up auto-signing: Generate yourself a keypair, if you don’t have one. Add it to your profile, and setup your default Maven repo (or a new one) for signing with your GPG key: Bintray can then sign your jars automatically.
  2. Add your Sonatype account under “accounts”. If you don’t have one, follow this procedure
  3. Create and link your package: Import from a GitHub repo or create a new package for your Maven project.
  4. You can link your package to JCenter by clicking “Add to JCenter”.
  5. Set up Maven up to deploy to Bintray by copy-pasting the pom.xml snippets from “Set me up!” guide.

Then for each release:

  1. Deploy: Deploy files by running your build tool.
  2. Publish: Review the build artifacts and publish the version files if satisfied.
  3. Sync: On the version page go to the Maven Central tab, enter your Sonatype password and click “Sync” and you’re done!
  4. Your package is now in https://oss.sonatype.org/content/repositories/releases and will be synced to Maven Central (usually takes time). In case of a sync problem, Bintray will automatically take care of any needed cleanup.

Summary

For distributing your open source projects’ artifacts through Maven Central repository and OSSRH you have two options: 1. Release with Sonatype’s process or 2. Use Bintray’s process. Both options end up with the same results: your project’s artifacts are in Maven Central repository. In my opinion if you’re using Maven then it doesn’t really matter which option you choose.

Before Sonatype simplified their process for making releases to OSSRH the Bintray’s process was the only almost pain-free gateway to Maven Central. But with the new Sonatype way the two alternative ways are more or less the same and I could say that Sonatype’s new process feels a bit easier when using the Maven release plugin. In practice there’s not much differences, e.g. both supports SNAPSHOTs, Sonatype in OSSRH and Bintray in oss.jfrog.org. Bintray is more flexible especially if not using Maven so if you’re using other tools like Gradle then it’s the easier way to go.

Maven plugin to generate HTML from Markdown now in Maven Central

Generating documents is always not the most pleasant job but it can be made easier by using easier markup language like Markdown. For that purpose I made last year a simple Markdown to HTML Page Generator Maven Plugin and now it’s also available in Maven Central.

“Markdown to HTML Page Generator” is a simple Maven Plugin which reads the input directory for Markdown files with .md suffix and outputs HTML files to target directory. There are couple of configuration options to set the input/output directories and if you want to add header and footer files to make it structurally valid HTML. The plugin uses pegdown Markdown processor.

Maven plugin artifacts can be found in Sonatypes’ OSS Maven repository:

Read more about the plugin and how to use it from markdown-page-generator-plugin’s GitHub page.

Debugging Sailfish OS application power consumption

Lately my simple news feed reader, Ampiaiskala, got rejected from Jolla Store in Harbour Quality Assurance as it seemed that it was too power hungry while minimized. I was a bit surprised as 0.1 version was accepted and the functionality hadn’t changed. So, it was time to get familiarized with PowerTOP and how to debug problems with power consumption.

First we need to install PowerTOP to Jolla and then just ran it via SSH over USB or WLAN while the app is minimized and screen is closed. After that it’s about what all those numbers mean and where they come from.

Installing PowerTOP on Jolla

Steps to install PowerTOP from command line:

1. Add mer-tools repository:

$ ssu ar mer-tools

2. Refresh repository cache:

$ pkcon refresh

3. Install tool:

$ pkcon install powertop

4. Run tool:

$ devel-su powertop

Debugging power consumption

PowerTOP reports how much power each process uses and provides a list of all running processes which are waking up and how many times in a second (how many times each one sends a CPU interrupt). Only one process is monitored so it doesn’t tell all about the system, e.g. when the process has interaction with audio or network.

After installing PowerTOP tool, usage is easy via SSH over USB or WLAN connection. Difference between these connections is that device does not suspend with PC USB connection enabled and with WLAN device goes to suspend if there’s not any network traffic and display is turned off.

Connect to your phone e.g. with ssh nemo@192.168.1.58 and ran the PowerTOP as root: devel-su powertop. The command prompts you for the root password which you have given in Settings > System > Developer mode > SSH and root password.

Before starting to debug your app’s power consumption it’s useful to create a baseline by running PowerTOP after a cold startup, without opening any applications, and then run it a few more times throughout the day to get a comparison of different workloads.

Generate HTML page of PowerTOP results for baseline:

$ devel-su powertop --html=<filename>

After getting some baseline results it’s just reading the numbers and figuring out if your application eats power or not. And if it is power hungry beast then figuring out why it does that.

Complementary to PowerTOP, there’s Strace which lets you find out how a program is interacting with the operating system by monitoring system calls and signals. Just install it with $ pkcon install strace and ran it with strace -p or strace -ttT -p . It shows what your application does while it’s expected to be idle. For example:

$ ps aux | grep ampiaiskala
$ devel-su strace -p 1301

The result looks something like this:

Don’t ask me what all that means :)

Power consumption and Harbour QA testing

The reason I’m debugging my application for power usage is Jolla Harbour Quality Assurance which rejected my app with “It seems that app doesn’t go to idle state when device does”. The Quality Assurance procedure for power consumption is explained in Together being the following.

“Jolla’s Harbour QA testing is mostly done with PC USB cable connection, application minimized to home screen and display is turned off. In this state application should not block suspend and wake ups should be minimal (< ~60, this is a reference value). A good practice is to take a "system state" before opening application and compare those results when application is minimized to home screen." "The phone suspend is blocked by wake locks and easy test for that is catting /sys/power/wake_lock file. The file should be empty normally if nothing from user space is not blocking suspend. If application is using e.g. audio, then file should include “mce_cpu_keepalive” string. Of course applications are allowed to do things in the background but that should be minimal for its own functionality. E.g are internet streaming.”

After another Harbour QA rejection (2014-06-05) I asked in IRC from Jolla Sailor, nazanin, about the power consumption issue and what steps they are using so I could repeat them. The steps are the following:

  • First there are no apps open, lock the device, run powertop command
  • Wait for about 1 minute until the device status becomes stable
  • Unlock it, open the application you want to test, play around with it for a bit, until the number of wake ups reaches 300 or 400
  • Check if you already see the app in wakeup list in this status
  • Lock the device and observe how powertop updates
  • If after 1 minute or 2, the application drops down from the list, it’s okay
  • If it doesn’t, then wait more, for about 5-10 minutes
  • If you see the application is still there, that is when you can assume the test case is failed and application is causing power consumption

Possible reasons for excess power consumption

The procedure to figure out if your application is consuming power more than it should is relatively easy but finding out why it does that is a bit harder. Things like timers, animations, WebViews and using animated images are possible culprits and how the app behaves with none or bad network connectivity. I don’t know there to be any good tips how to debug these issues other than going through your app, knowing what it does and figuring out possible issues.

My own cases for Harbour QA rejections due power consumption have been with using SilicaWebview (QtWebKit) and keeping it open when the app is minimized and thus eating too much CPU and preventing the device go to sleep. The solution would be that you suspend the WebKit rendering or close the WebView. But that’s easier said than done. After discussing this issue in IRC with couple of Sailors (nazanin, kontio) for now there’s no good solution to suspend the WebKit as it doesn’t provide an easy API to suspend rendering. If it’s reasonable for user experience you can pop the WebView page when app is minimized. You can bind the action e.g. to CoverPage and use onStatusChanged signal and check for “status == PageStatus.Activating.

I came across this WebKit issue with Haikala and Ampiaiskala as they open WebKit view for reading the news feed item’s web page. For those I modified my code to allow the user to close the web view from the pull down menu when user goes back to the feed list. After that they passed the Harbour QA. Couple of months ago there was a similar case on SailfishDevel mailing list.

With my imgur app, Sailimgur, it was again using WebKit for viewing imgur’s gallery page but also there’s animated images running while showing the gallery or image page in the app. The app was seen under wake-ups when it was showing an animated image like .gif. If the app wasn’t showing some funny gifs from imgur it dropped down from the list after a while as it should. Understandable effect as the UI is running the gif which consumes power and prevents the device go to sleep.

The partial solution is to manually stop the animated images when the app is minimized but in my opinion that should be something that Sailfish OS should take care automatically. The other thing is the animated image running in the SilicaWebView where QtWebKit keeps running in the background and doesn’t suspend automatically. You could send SIGSTOP to suspend it when minimized and then continue with SIGCONT from C++ but it’s a bit hassle as you would need a monitoring process to manage that. And it’s not known if that works or not.

In the end after considering the options the Sailors said that they accept my app in the current state, since WebKit doesn’t provide an easy API to suspend rendering. But have to think about what I could do to limit wake-ups.