Automate validating code changes with Git hooks

What could be more annoying than committing code changes to repository and noticing afterwards that formatting isn’t right or tests are failing? Your automated tests on Continuous Integration shows rain clouds and you need to get back to the code and fix minor issues with extra commits polluting the git history? Fortunately with small enhancements to your development workflow you can automatically prevent all the hassle and check your changes before committing them. The answer is to use Git hooks for example on pre-commit for running linters and tests.

Git Hooks

Git hooks are scripts that Git executes before or after events such as: commit, push, and receive. They’re a built-in feature and run locally. Hook scripts are only limited by a developer’s imagination. Some example hook scripts include:

  • pre-commit: Check the commit for linting errors.
  • pre-receive: Enforce project coding standards.
  • post-commit: Email team members of a new commit.
  • post-receive: Push the code to production.

Every Git repository has a .git/hooks folder with a script for each hook you can bind to. You’re free to change or update these scripts as necessary, and Git will execute them when those events occur.

Git hooks can greatly increase your productivity as a developer as you can automate tasks and ensure that your code is ready for commit or pushing to remote repository.

For more reading about Git hooks you can check missing Git hooks documentation, read the basics and check tutorial how to use Git hooks on local Git clients and Git servers.

Pre-commit

One productive way to use Git hooks is pre-commit framework for managing and maintaining multi-language pre-commit hooks. Read tips for using a pre-commit hook.

Pre-commit is nice for example running linters to ensure that your changes conform to coding standards. All you need is to install pre-commit and then add hooks.

Installing pre-commit, ktlint and pre-commit-hook on MacOS with Homebrew:

$ brew install pre-commit
$ brew install ktlint
$ ktlint --install-git-pre-commit-hook

For example the pre-commit hook to run ktlint with auto-correct option looks like the following in projects .git/hooks/pre-commit. The “export PATH=/usr/local/bin:$PATH” is for SourceTree to find git on MacOS.

#!/bin/sh
export PATH=/usr/local/bin:$PATH
# https://github.com/shyiko/ktlint pre-commit hook
git diff --name-only --cached --relative | grep '\.kt[s"]\?$' | xargs ktlint -F --relative .
if [ $? -ne 0 ]; then exit 1; else git add .; fi

The main disadvantage is using pre-commit and local git hooks is that hooks are kept within .git directory and it never comes to the remote repository. Each contributor will have to install them manually in his local repository which may be overlooked.

Maven projects

Githook Maven plugin deals with the problem of providing hook configuration to the repository and automates their installation. It binds to Maven projects build process and configures and installs local git hooks.

It keeps a mapping between the hook name and the script by creating a respective file in .git/hooks for each hook containing given script in Maven project’s initial lifecycle phase. It’s good to notice that the plugin rewrites hooks.

Usage Example:

<build>
    <plugins>
	<plugin>
	    <groupId>org.sandbox</groupId>
	    <artifactId>githook-maven-plugin</artifactId>
	    <version>1.0.0</version>
	    <executions>
	        <execution>
	            <goals>
	                <goal>install</goal>
	            </goals>
	            <configuration>
	                <hooks>
	                    <pre-commit>
	                         echo running validation build
	                         exec mvn clean install
	                    </pre-commit>
	                </hooks>
	            </configuration>
	        </execution>
	    </executions>
	</plugin>
    </plugins>
</build>

Git hooks for Node.js projects

On Node.js projects you can define scripts in package.json and run them with npm which enables an another approach to running Git hooks.

🐶 Husky is Git hooks made easy for Node.js projects. It keeps existing user hooks, supports GUI Git clients and all Git hooks.

Installing Husky is like any other npm library

npm install husky --save-dev

The following configuration on your package.json runs lint (e.g. eslint with –fix) command when you try to commit and runs lint and tests (e.g. mocha, jest) when you try to push to remote repository.

"husky": {
   "hooks": {
     "pre-commit": "npm run lint",
     "pre-push": "npm run lint && npm run test"
   }
}

Another useful tool is lint-staged which utilizes husky and runs linters against staged git files.

Summary

Make your development workflow easier by automating all the things. Check your changes before committing them with pre-commit, husky or Githook Maven plugin. You get better code and commit quality for free and your team is happier.

This article was originally published at 15.7.2019 on Gofore’s blog.

Ignoring files and folders in Subversion with propset

Before committing code to the Subversion repository we always set the svn:ignore property on the directory to prevent some files and directories to be checked in. You would usually want to exclude the IDE project files and the target/ directory.

It’s useful to put all the ignored files and directories into a file: .svnignore. Your .svnignore could look like:

*.iml
target/*

Put the .svnignore file in the project folder and commit it to your repository so the ignored files are shared between committers.

Now reference the file with the -F option:
$ svn propset svn:ignore -F .svnignore.

Of course I hope everyone has by now moved to git and uses .gitignore for this same purpose.

Best Practices for Version Control in 8 steps

Using version control is an essential part of modern software development and using it efficiently should be part of every developer’s tool kit. Knowing the basic rules makes it even more useful. Here are some best practices that help you on your way.

tl; dr;

  1. Commit logical changesets (atomic commits)
  2. Commit Early, Commit Often
  3. Write Reasonable Commit Messages
  4. Don’t Commit Generated Sources
  5. Don’t Commit Half-Done Work
  6. Test Before You Commit
  7. Use Branches
  8. Agree on a Workflow
Simplified Git Flow
Simplified Git Flow (source: buildazure)

Commit logical changesets (atomic commits)

A commit should be a wrapper for related changes. Make sure your change reflects a single purpose: the fixing of a specific bug, the addition of a new feature, or some particular task. Small commits make it easier for other developers to understand the changes and roll them back if something went wrong.

Your commit will create a new revision number which can forever be used as a “name” for the change. You can mention this revision number in bug databases, or use it as an argument to merge should you want to undo the change or port it to another branch. Git makes it easy to create very granular commits.

So if you do many changes to multiple logical components at the same time, commit them in separate parts. That way it’s easier to follow changes and their history. So working with features A, B and C and fixing bugs 1, 2 and 3 should make at least 6 commits.

Commit Early , Commit Often

It is recommended to commit code to version control often which keeps your commits small and, again, helps you commit only related changes. It also allows you to share your code more frequently with others.

It’s easier for everyone to integrate changes regularly and avoid having merge conflicts. Having few large commits and sharing them rarely, in contrast, makes it hard to solve conflicts.

“If the code isn’t checked into source control, it doesn’t exist.”

Coding Horror

Write Reasonable Commit Messages

Always write some reasonable comment on your commit. It should be short and descriptive and tell what was changed and why.

Begin your message with a short summary of your changes (up to 50 characters as a guideline). Separate it from the following body by including a blank line.

It is also useful to add some prefix to your message like Fix or Add, depending on what kind of changes you did. Use the imperative, present tense (“change”, not “changed” or “changes”) to be consistent with generated messages from commands like git merge.

If fixing some bug or making some feature and it has a JIRA ticket, add the ticket identifier as a prefix.

For example: “Fix a few bugs in the interface. Added an ID field. Removed a couple unnecessary functions. Refactored the context check.” or “Fix bad allocations in image processing routines”.

Not like this: “Fixed some bugs.”

The body of your message should provide detailed answers to the following questions: What was the motivation for the change? How does it differ from the previous implementation?

“If the changes you made are not important enough to comment on, they probably are not worth committing either.”

loop label

Don’t Commit Generated Sources

Don’t commit files which are generated dynamically or which are user dependent. Like target folder or IDEA’s .iml files or Eclipse’s .settings and .project files. They change depending what the user likes and don’t relate to project’s code.

Also project’s binary files and Javadocs are files that don’t belong to version control.

Don’t Commit Half-Done Work

You should only commit code when it’s completed. Split the feature’s implementation into logical chunks and remember to commit early and often. Use branches or consider using Git’s Stash feature if you need a clean working copy (to check out a branch, pull in changes, etc.).

On the other hand you should never leave the office without commiting your changes.

“It’s better to have a broken build in your working repository than a working build on your broken hard drive.”

loop label

Test Before You Commit

You should only commit code which is tested and passes tests. And this includes code formatting with linters. Write tests and run tests to make sure the feature or bug fix really is completed and has no side effects (as far as one can tell).

Having your code tested is even more important when it comes to pushing / sharing your code with others.

Use Branches

Branching is one of Git’s most powerful features – and this is not by accident: quick and easy branching was a central requirement from day one. Branches are the perfect tool to help you avoid mixing up different lines of development.

You should use branches extensively in your development workflows: for new features, bug fixes and ideas.

Agree on a Workflow

Git lets you pick from a lot of different workflows: long-running branches, topic branches, merge or rebase, git-flow.

Which one you choose depends on a couple of factors: your project, your overall development and deployment workflows and (maybe most importantly) on your and your teammates’ personal preferences. However you choose to work, just make sure to agree on a common workflow that everyone follows.

Atlassian has done good article of comparing workflows to suit your needs and covers centralized, feature Branch, gitflow and forking workflows.

Summary

Using version control is usually and fortunately an acknowledged best practice and part of software development. By using even couple of the above practices makes working with the code much more pleasant. Adopting at least “Commit logical changesets” and “Reasonable Commit Messages” helps a lot.

Code quality metrics for Kotlin project on SonarQube

Code quality in software development projects is important and a good metric to follow. Code coverage, technical debt, vulnerabilities in dependencies and conforming to code style rules are couple of things you should follow. There are some de facto tools you can use to visualize things and one of them is SonarQube. Here’s a short technical note of how to setup it on Kotlin project and visualize metrics from different tools.

Including what analysis SonarQube’s default plugins provide we are also using Detekt for static source code analysis and OWASP Dependency-Check to detect publicly disclosed vulnerabilities contained within project dependencies.

Visualizing Kotlin project metrics on SonarQube

SonarQube is nice graphical tool to visualize different metrics of your project. Lately it has started to support also Kotlin with SonarKotlin plugin and sonar-kotlin plugin. From typical Java project you need some extra settings to get things working. It’s also good to notice that the support for Kotlin isn’t quite yet there and sonar-kotlin provides better information i.e. what comes to code coverage

Steps to integrate reporting to Sonar with maven build:

  • Add configuration in project pom.xml: Surefire, Failsafe, jaCoCo, Detekt, Dependency-Check
  • Run Sonar in Docker
  • Maven build with sonar:sonar option
  • Check Sonar dashboard
SonarQube overview
SonarQube project overview

Configure Kotlin project

Configure your Kotlin project built with Maven to have test reporting and static analysis. We are using Surefire to run unit tests, Failsafe for integration tests and JaCoCo generates reports for e.g. SonarQube. See the full pom.xml from example project (coming soon).

Test results reporting

pom.xml

<properties> 
<sonar.coverage.jacoco.xmlReportPaths>${project.build.directory}/site/jacoco/jacoco.xml</sonar.coverage.jacoco.xmlReportPaths> 
</properties> 

<build> 
    <plugins>
        <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>default-prepare-agent</id>
                    <goals>
                        <goal>prepare-agent</goal>
                    </goals>
                </execution>
                <execution>
                    <id>pre-integration-test</id>
                    <goals>
                        <goal>prepare-agent-integration</goal>
                    </goals>
                </execution>
                <execution>
                    <id>jacoco-site</id>
                    <phase>verify</phase>
                    <goals>
                        <goal>report</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <skipTests>${unit-tests.skip}</skipTests>
                <excludes>
                    <exclude>**/*IT.java</exclude>
                    <exclude>**/*IT.kt</exclude>
                    <exclude>**/*IT.class</exclude>
                </excludes>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                        <goal>verify</goal>
                    </goals>
                </execution>
            </executions>
            <configuration>
                <skipTests>${integration-tests.skip}</skipTests>
                <includes>
                    <include>**/*IT.class</include>
                </includes>
                <runOrder>alphabetical</runOrder>
            </configuration>
        </plugin>
    </plugins> 

    <pluginManagement>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>2.22.1</version>
            </plugin>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.3</version>
            </plugin>
        </plugins>
    </pluginManagement>

... 
</build> 

Static code analysis with Detekt

Detekt static code analysis configuration as AntRun. There’s also unofficial Maven plugin for Detekt. It’s good to notice that there are some “false positive” findings on Detekt and you can either customize detekt rules or suppress findings if they are intentional such as @Suppress(“MagicNumber”).

Detekt code smells
Detekt code smells

pom.xml

<properties> 
    <sonar.kotlin.detekt.reportPaths>${project.build.directory}/detekt.xml</sonar.kotlin.detekt.reportPaths> 
</properties> 

<build> 
... 
<plugins> 
<plugin> 
    <groupId>org.apache.maven.plugins</groupId> 
    <artifactId>maven-antrun-plugin</artifactId> 
    <version>1.8</version> 
    <executions> 
        <execution> 
            <!-- This can be run separately with mvn antrun:run@detekt --> 
            <id>detekt</id> 
            <phase>verify</phase> 
            <configuration> 
                <target name="detekt"> 
                    <java taskname="detekt" dir="${basedir}" 
                          fork="true" 
                          failonerror="false" 
                          classname="io.gitlab.arturbosch.detekt.cli.Main" 
                          classpathref="maven.plugin.classpath"> 
                        <arg value="--input"/> 
                        <arg value="${basedir}/src"/> 
                        <arg value="--filters"/> 
                        <arg value=".*/target/.*,.*/resources/.*"/> 
                        <arg value="--report"/> 
                        <arg value="xml:${project.build.directory}/detekt.xml"/> 
                    </java> 
                </target> 
            </configuration> 
            <goals> 
                <goal>run</goal> 
            </goals> 
        </execution> 
    </executions> 
    <dependencies> 
        <dependency> 
            <groupId>io.gitlab.arturbosch.detekt</groupId> 
            <artifactId>detekt-cli</artifactId> 
            <version>1.0.0-RC14</version> 
        </dependency> 
    </dependencies> 
</plugin> 
</plugins> 
... 
</build> 

Dependency checks

Dependency check with OWASP Dependency-Check Maven plugin

OWASP Dependency-Check
OWASP Dependency-Check

pom.xml

<properties> 
    <dependency.check.report.dir>${project.build.directory}/dependency-check</dependency.check.report.dir> 
    <sonar.host.url>http://localhost:9000/</sonar.host.url> 
    <sonar.dependencyCheck.reportPath>${dependency.check.report.dir}/dependency-check-report.xml</sonar.dependencyCheck.reportPath>
    <sonar.dependencyCheck.htmlReportPath>${dependency.check.report.dir}/dependency-check-report.html</sonar.dependencyCheck.htmlReportPath>
</properties> 

<build> 
... 
<plugins> 
<plugin> 
    <groupId>org.owasp</groupId> 
    <artifactId>dependency-check-maven</artifactId> 
    <version>4.0.2</version> 
    <configuration> 
        <format>ALL</format> 
        <skipProvidedScope>true</skipProvidedScope> 
        <skipRuntimeScope>true</skipRuntimeScope> 
        <outputDirectory>${dependency.check.report.dir}</outputDirectory> 
    </configuration> 
    <executions> 
        <execution> 
            <goals> 
                <goal>check</goal> 
            </goals> 
        </execution> 
    </executions> 
</plugin> 
</plugins> 
... 
</build>

Sonar scanner to run with Maven

pom.xml

<build> 
... 
    <pluginManagement> 
        <plugins> 
            <plugin> 
                <groupId>org.sonarsource.scanner.maven</groupId> 
                <artifactId>sonar-maven-plugin</artifactId> 
                <version>3.6.0.1398</version> 
            </plugin> 
        </plugins> 
    </pluginManagement> 
... 
</build> 

Running Sonar with Kotlin plugin

Create a SonarQube server with Docker

$ docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube 

There’s also OWASP docker image for SonarQube which adds several community plugins to enable SAST. But for our purposes the “plain” SonarQube works nicely.

Use the Kotlin plugin which comes with SonarQube (SonarKotlin) or install the sonar-kotlin plugin which shows information differently. If you want to use sonar-kotlin and are using the official Docker image for SonarQube then you’ve to first remove the SonarKotlin plugin.

Using sonar-kotlin

$ git clone https://github.com/arturbosch/sonar-kotlin 
$ cd sonar-kotlin 
$ mvn package  
$ docker exec -it sonarqube sh -c "ls /opt/sonarqube/extensions/plugins" 
$ docker exec -it sonarqube sh -c "rm /opt/sonarqube/extensions/plugins/sonar-kotlin-plugin-1.5.0.315.jar" 
$ docker cp target/sonar-kotlin-0.5.2.jar sonarqube:/opt/sonarqube/extensions/plugins 
$ docker stop sonarqube 
$ docker start sonarqube 

Adding dependency-check-sonar-plugin to SonarQube

$ curl -JLO https://github.com/SonarSecurityCommunity/dependency-check-sonar-plugin/releases/download/1.2.1/sonar-dependency-check-plugin-1.2.1.jar 
$ docker cp sonar-dependency-check-plugin-1.2.1.jar sonarqube:/opt/sonarqube/extensions/plugins 
$ docker stop sonarqube 
$ docker start sonarqube 

Run test on project and scan with Sonar

The verify phase runs your tests and should generate i.a. jacoco.xml under target/site/jacoco and detekt.xml.

$ mvn clean verify sonar:sonar

Access Sonar via http://localhost:9000/

Code quality metrics? So what?

You now have metrics on Sonar to show to stakeholders but what should you do with those numbers?

One use case is to set quality gates on SonarQube to check that a set of conditions must be met before project can be released into production. Ensuring code quality of “new” code while fixing existing ones is one good way to maintain a good codebase over time. The Quality Gate facilitates setting up rules for validating every new code added to the codebase on subsequent analysis. By default the rules are: coverage on new code < 80%; percentage of duplicated lines on new code > 3; maintainability, reliability or security rating is worse than A.

Best Practices of forking git repository and continuing development

Sometimes there’s a need to fork a git repository and continue development with your own additions. It’s recommended to make pull request to upstream so that everyone could benefit of your changes but in some situations it’s not possible or feasible. When continuing development in forked repo there’s some questions which come to mind when starting. Here’s some questions and answers I found useful when we forked a repository in Github and continued to develop it with our specific changes.

Repository name: new or fork?

If you’re releasing your own package (to e.g. npm or mvn) from the forked repository with your additions then it’s logical to also rename the repository to that package name.

If it’s a npm package and you’re using scoped packages then you could also keep the original repository name.

Keeping master and continuing developing on branch?

Using master is the sane thing to do. You can always sync your fork with an upstream repository. See: syncing a fork

Generally you want to keep your local master branch as a close mirror of the upstream master and execute any work in feature branches (that might become pull requests later).

How you should do versioning?

Suppose that the original repository (origin) is still in active development and does new releases. How should you do versioning in your forked repository as you probably want to bring the changes done in the origin to your fork? And still maintain semantic versioning.

In short, semver doesn’t support prepending or appending strings to version. So adding your tag to the version number from the origin which your version is following breaks the versioning. So, you can’t use something like “1.0.0@your-org.0.1” or “1.0.0-your-org.1”. This has been discussed i.a. semver #287. The suggestion was to use a build meta tag to encode the other version as shown in semver spec item-10. But the downside is that “Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence.”

If you want to keep relation the original package version and follow semver then your options are short. The only option is to use build meta tag: e.g. “1.0.0+your-org.1”.

It seems that when following semantic versioning your only option is to differ from origin version and continue as you go.

If you don’t need to or want to follow semver you can track upstream version and mark your changes using similar markings as semver pre-releases: e.g. “1.0.0-your-org.1”.

npm package: scoped or unscoped?

Using scoped packages is a good way to signal official packages for organizations. Example of using scoped packages can be seen from Storybook.

It’s more of a preference and naming conventions of your packages. If you’re using something like your-org-awesome-times-ahead-package and your-org-patch-the-world-package then using scoped packages seems redundant.

Who should be the author?

At least add yourself to contributors in package.json.

Forking only for patching npm library?

Don’t fork, use patch-package which lets app authors instantly make and keep fixes to npm dependencies. Patches created by patch-package are automatically and gracefully applied when you use npm(>=5) or yarn. Now you don’t need to wait around for pull requests to be merged and published. No more forking repos just to fix that one tiny thing preventing your app from working.

This post was originally published on Gofore Group blog at 11.2.2019.

Learning and Staying Current in Software Development

Software development is one of the professions where you have to keep your knowledge up to date and follow what happens in the field. Staying current in the field and expanding your horizons can be achieved with different ways and one good way I have used is to follow different news sources, newsletters, listening podcasts and attending meetups. Here is my opinionated selection of resources to learn, share ideas, newsletters, meetups and other things for software developers. Meetups and some things are Finnish related.

News

There are some good sites to follow what happens in technology. They provide community powered links and discussions.

Podcasts

Podcasts provide nice resource for gathering experiences and new information how things can be done and what’s happening and coming up in software development. I commute daily about an hour and time flies when you find good episodes to listen. Here’s my selection of podcast relating to software development.

General

  • Software Engineering Daily: “The world through the lens of software” (iTunes)
  • Software Engineering Radio: “Targeted at the professional software developer. The goal is to be a lasting educational resource, not a newscast”. (feed)
  • ShopTalk: “An internet radio show about the internet starring Dave Rupert and Chris Coyier.” (iTunes)
  • Full Stack Radio: “Every episode, Adam Wathan is joined by a guest to talk about everything from product design and user experience to unit testing and system administration.” (feed)

Front-end

  • Syntax: “A Tasty Treats Podcast for Web Developers.” (iTunes)
  • The Changelog: “Conversations with the hackers, leaders, and innovators of software development.”
  • React Podcast: “Conversations about React with your favorite developers.”
  • Brainfork: “A podcast about mental health & tech”

In Finnish

  • ATK-hetki: “Vesa Vänskä ja Antti Akonniemi keskustelevat teknologiasta, bisneksestä ja itsensä kehittämisestä.”
  • Webbidevaus: “Puheradiota webbikehityksestä suomeksi! Juontajina Antti Mattila ja Riku Rouvila.”

Newsletters

Normal information overload is easily achieved so it’s beneficial to use for example curated newsletters for the subjects which intersects the stack you’re using and topics you’re interested at.

The power of newsletter lies in the fact that it can deliver condensed and digestible content which is harder to achieve with other good news sources like feed subscriptions and Twitter. Well curated newsletter to targeted audience is a pleasure to read and even if you forgot to check your newsletter folder, you can always get back to them later.

General

Mobile development

  • iOS Dev Weekly: Hand picked round up of the best iOS development links published every Friday
  • This Week In Swift: List of the best Swift resources of the week.
  • iOS Dev nuggets: Short iOS app development nugget every Friday/Saturday. Short and usually something you can read in a few minutes and improve your skills at iOS app development.

Java

Database

  • DB Weekly: A weekly round-up of database technology news and articles covering new developments, SQL, NoSQL, document databases, graph databases, and more.

HTML and CSS

  • HTML5Weekly: Weekly HTML5 and Web Platform technology roundup. Curated by Peter Cooper.
  • CSS Weekly: Roundup of css articles, tutorials, experiments and tools. Curated by Zoran Jambor.

Web development

  • Status code: “Keeping developers informed.” weekly email newsletters on a range of programming niches (links to JavaScript weekly, DevOps weekly etc.)
  • Web Development Reading List: Weekly roundup of web development–related sources, selected by Anselm Hannemann.
  • Versioning: “Daily knowledge devs and designers need to get ahead of the game.” SitePoint’s daily newsletter, which features the latest web development news.
  • Hacking UI: A weekly email with our favorite articles about design, front-end development, technology, startups, productivity and the occasional inspirational life lesson.
  • Scott Hanselman: Newsletter of Wonderful Things. Includes interesting and useful stuff Scott has found over the last few weeks and other wonderful things.
  • MergeLinks: Weekly email of curated links to articles, resources, freebies and inspiration for web designers and developers.
  • “How to keep up to date on: Front-End Technologies” page lists newsletters, blogs and people to follow.

JavaScript

  • JavaScript Weekly: Weekly e-mail round-up of JavaScript news and articles. Curated by Peter Cooper.
  • Node Weekly: Once–weekly e-mail round-up of Node.js news and articles.
    A Drip of JavaScript: “One quick JavaScript tip”, delivered every other Tuesday and written by Joshua Clanton.
  • SuperHero.js: Collection of the best articles, videos, and presentations on creating, testing, and maintaining a JavaScript code base.
  • State of JS: Results of yearly JavaScript surveys

User experience and design

  • UX Design Weekly: Hand picked list of the best user experience design links every week. Curated by Kenny Chen & published every Monday.
  • Sidebar.io: To satisfy your web aesthetics with list of the 5 best design links of the day. The content is manually curated by a couple great editors.
  • Userfocus: Monthly newsletter which shares an in-depth article on user experience.

Ops

  • DevOps Weekly: Weekly slice of devops news.
  • Web Operations Weekly: Weekly newsletter on Web operations, infrastructure, performance, and tooling, from the browser down to the metal.
  • Microservice Weekly: A hand-curated weekly newsletter with the best articles on microservices.

Twitter

Following fellow developers and other people and accounts on Twitter is good way to know what’s happening right now. Here’s a selection of accounts I (@walokra) follow. In no particular order.

Development

  • @ThePracticalDev: Great posts from the amazing dev.to community, with some opinion and humor mixed in.
  • @CommitStrip: The blog relating the daily life of developers. Official english account.
  • @baeldung: Author of restwithspring.com and learnspringsecurity.com, passionate about REST, Security, TDD and everything in between.
  • @martinfowler: Author and international public speaker on software development, specializing in object-oriented analysis and design, UML, patterns, and agile software development methodologies.

Infosec

  • @troyhunt: Pluralsight author. Microsoft Regional Director and MVP for Developer Security. Online security, technology and “The Cloud”. Creator of @haveibeenpwned.
  • @briankrebs: Independent investigative journalist. Writes about cybercrime. Author of ‘Spam Nation’, a NYT bestseller. Wrote for The Washington Post ’95-’09
  • @mikko: CRO at F-Secure ● TED Speaker ● Revɘrse Engineer ● Supervillain
  • @TinkerSec Infosec Hacker things
  • @Anakondantti: Mostly software security related, but occasionally other things too. I’m a white hat hacker at team ROT.
  • @SunTzuCyber: If Sun Tzu had written “The Art of Cyber War”, these would be his quotes.
  • @lennyzeltser: Advances information security. Grows tech businesses. Fights malware. // VP of Products @MinervaLabs. Author and Instructor @SANSInstitute.

React scene

  • @jevakallio: @FormidableLabs, React/Native engineer, comedian, speaker, writer, improviser, Twitter Developer Expert™. Artisanal small batch free range shitposting.
  • @bebraw: Award winning founder of @survivejs and @jsterlibs. I also organize @ReactFinland.
  • @ReactJSNews: The latest React news and articles.

Design / UX

  • @steveschoger: Designer for @TightenCo and @taylorotwell ❯ Maker of heropatterns , heroicons  and zondicons  ❯  ? Design Tips
  • @UX_Grant: ? Senior Designer @ booking.com . ? Creating, Learning, Sharing ? Maker: MakersMusic.co  ?
  • @jonikorpi: Making multiplayer games using the web platform, as @vuorodesign. Previously web design at @kiskolabs.
  • @lukew: Humanizing technology. Founded: Polar (Google acquired) Bagcheck (Twitter acquired) Wrote: Mobile First, Web Form Design, Site Seeing. Worked: Yahoo, eBay, NCSA.
  • @autiomaa: Helping people, with design & technology. Front-end development, visual design, photography. Learning something new every day.
  • @skrug: Best known as the guy who wrote Don’t Make Me Think (now in its 3rd edition!) and Rocket Surgery Made Easy.
  • @jnd1er: Don Norman. Design thinker, company advisor, professor, columnist, author, … Latest book: Design of Everyday Things, Revised and Expanded.
  • @mpietila: User experience etc. Occasional smart-assery & besserwisserism. I have a history of seeing what they did there. Head of design at @qvik.

Database

Miscellanous

Java

  • @mreinhold: Chief Architect, Java Platform Group, Oracle.
  • @jodastephen: Java Champion. Developer at OpenGamma. Occasional blogger and speaker. Best known for Joda projects and JSR-310

Technology News

Meetups

You can learn much from others and to broaden your horizon it’s beneficial to attend different meetups and listen how others have done things and watch war stories. Also free food and drinks.

Mostly Helsinki based

Tampere based

Community chats

Generating documentation as code with mermaid and PlantUML

Writing documentation is always a task which isn’t much liked and especially with diagrams and flowcharts there’s the problem of which tools to use. One crafty tool is Draw.io with web and desktop editors but what to use if you want to write documentation as a code and see the changes clearly in text format and maintain source-controlled diagrams? One of the tools for drawing diagrams with human readable text are mermaid and PlantUML.

mermaid

“Generation of diagrams and flowcharts from text in a similar manner as markdown.”

mermaid is a simple markdown-like script language for generating charts from text via javascript. You can try it in live editor.

mermaid sequence diagram

You can write mermaid diagrams in text editor but it’s better to use some editor with plugins to preview your work. Markdown Preview Enhanced for Atom and VS Code can render mermaid and PlantUML. There’s also dedicated preview plugins for VS Code and Atom.

To preview mermaid definition in VS Code with Markdown Preview Enhanced press Cmd-P to open Command palette and select Markdown Preview Enhanced: Open Preview.

mermaid in VS Code with Markdown Preview Enhanced

To preview mermaid definition in VS Code with Mermaid Preview press Cmd-P to open Command palette and select Preview Mermaid Diagram.

mermaid in VS Code with Mermaid preview
mermaid in VS Code with Mermaid preview

Generating PNG images from mermaid definitions

To use mermaid diagrams it’s useful to export them to PNGs. You can use mermaid.cli tool which takes a mermaid definition file as input and generates svg/png/pdf file as output.

Install mermaid.cli locally:

npm install mermaid.cli

Generate PNG:

./node_modules/.bin/mmdc -i input.mmd -o output.png

If you have plenty of defition files you can use the following script to generate PNGs:

#!/usr/bin/env bash
 
for mmd in ./docs/*.mmd
do
    filename="${mmd##*/}"
    echo "Generating $mmd"
    ./node_modules/.bin/mmdc -i $mmd -o ${filename%%.*}.png
done

Alternatively you can use node_modules/mermaid/bin/mermaid.js $mmd where mmd is the mermaid file.

PlantUML diagrams

PlantUML is used to draw UML diagrams, using a simple and human readable text description.

PlantUML is used to draw UML diagrams, using a simple and human readable text description. Diagrams are defined using a simple and intuitive language (pdf) and images can be generated in PNG, in SVG or in LaTeX format.

You can use PlantUML to write e.g. sequence diagrams, usecase diagrams, class diagrams, component diagrams, state diagrams and deployment diagrams.

PlantUML example diagram:

PlantUML diagram

Simple way to create and view PlantUML diagrams is to use Visual Studio Code and Markdown Preview Enhanced plugin which renders both PlantUML and mermaid diagrams. Alternative option is to use plantuml plugin.

To preview PlantUML diagram in VS Code with Markdown Preview Enhanced press Cmd-P to open Command palette and select Markdown Preview Enhanced: Open Preview.

PlantUML in VS Code with Markdown Preview Enhanced

There’s an online demo server which you can use to view PlantUML diagrams. The whole diagram is compressed into the URL itself and diagram data is stored in PNG metadata, so you can fetch it even from a downloaded image. For example this link opens the PlantUML Server with a simple Authentication activity diagram.

Running PlantUML server locally

Although you can render PlantUML diagrams online it’s better for usability and security reasons to install a local server. And this approach is important if you plan to generate diagrams with sensitive information. The easiest path is to run PlantUML Server Docker container and configure localhost as server.

docker run -d -p 8080:8080 plantuml/plantuml-server:jetty

In VS Code config, open user setting, and configure like:

"plantuml.server": "http://localhost:8080",
"plantuml.render": "PlantUMLServer",

Now to preview diagram in VS Code press alt-D to start PlantUML preview.

PlantUML preview in VS Code and local server

You can also generate diagrams from the command line. First download PlantUML compiled Jar and run the following command which will look for @startXXX into file1, file2 and file3. For each diagram, a .png file will be created.

java -jar plantuml.jar file1 file2 file3

The plantuml.jar needs Graphviz for dot (graph description language) and on macOS you can install it from Homebrew: brew install graphviz.

For processing a whole directory, you can use the following command which will search for @startXXX and @endXXX in .c, .h, .cpp, .txt, .pu, .tex, .html, .htm or .java files of the given directories:

java -jar plantuml.jar "directory1" "directory2"

Maintain source-controlled diagrams as a code

Documentation and drawing diagrams can be simple and maintaining source-controlled diagrams with tools like PlantUML and mermaid is achievable. These tools are not like the behemoth of Sparx Enterprise Architect but provide light and easy way to draw different diagrams for software development. You don’t have to draw lines and position labels manually as they are magically added where they fit and you even get as crude boxes and squares as thousands of dollars more expensive tools. Now the question is which tool to choose: PlantUML or mermaid?

Two days of React Finland 2018: Day two with React and React Native

React Finland 2018 conference was held last week and I had the opportunity to attend it and listen what’s hot in the React world. The conference started with workshops and after that there was two days of talks of React, React Native, React VR and all things that go with developing web applications with them. The two conference days were packed with great talks and new information. This is the second part of my recap of the talks and my notes which I posted to Twitter. Check out also the first part of my notes from the first day’s talks.

React Finland 2018, Day 2

How React changed everything — Ken Wheeler

Second day started with keynote by Ken Wheeler. He examined how React changed the front end landscape as we know it and started it with nice time travel to the 90s with i.a. Flash, JavaScript and AngularJS. Most importantly the talk took a look at the core idea of React, and why it transcends language or rendering target and posit on what that means going forward. And last we heard about what React async: suspense and time slicing.

“Best part of React is the community”

How React changed everything

Get started with Reason — Nik Graf

The keynote also touched Reason ML and Nik Graf went into details kicking off with the basics and going into how to leverage features like variant types and pattern matching to make impossible states impossible.

Get started with Reason ML

Making Unreasonable States Impossible — Patrick Stapfer

Based on “Get started with Reason” Patrick Stapfer’s talk went deeper into the world of variant types and pattern matching and put them into a practical context. The talk was nice learning by doing TicTacToe live coding. It showed how Reason ML helps you design solid APIs, which are impossible to misuse by consumers. We also got more insights into practical ReasonReact code. Presentation is available on the Internet.

Conclusion about ReasonReact:

  • More rigid design
  • More KISS (keep it simple, stupid) than DRY (don’t repeat yourself)
  • Forces edge-cases to be handled

Learning Reason by doing TicTacToe

Reactive State Machines and Statecharts — David Khourshid

David Khourshid’s talk about state machines and statecharts was interesting. Functional + reactive approach to state machines can make it much easier to understand, visualize, implement and automatically create tests for complex user interfaces and flows. Model the code and automatically generate exhaustive tests for every possible permutation of the code. Things mentioned: React automata, xstate. Slides are available on the Internet.

“Model once, implement anywhere” – David Khourshid

The talk was surprisingly interesting especially for use cases as anything to make testing better is good. This might be something to look into.

ReactVR — Shay Keinan

After theory heavy presentations we got into more visual stuff: React VR. Shay Keinan presented the core concepts behind VR, showed different demonstrations, and how to get started with React VR and how to add new features from the Three.js library. React VR: Three.js + React Native = 360 and VR content. On the VR device side it was mentioned that Oculus Go, HTC Vive Focus are the big step to Virtual Reality.

“Virtual Reality’s possibilities are endless. Compares to lucid dreaming.” – Shay Keinan

WebVR enables web developers to create frictionless, immersive experiences and we got to see Solar demo and Three VR demo which were lit ?.

React VR

World Class experience with React Native — Michał Chudziak

I’ve shortly experimented with React Native so it was nice to listen Michał Chudziak’s talk how to set up a friendly React Native development environment with the best DX, spot bugs in early stage and deliver continuous builds to QA. Again Redux was dropped in favour of apollo-link-state.

Work close to your team – Napoleon Hill

What makes a good Developer eXperience?

  • stability
  • function
  • clarity
  • easiness

GraphQL was mentioned to be the holy grail of frontend development and perfect with React Native. Tools for better developer experience: Haul, CircleCI, Fastlane, ESLint, Flow, Jest, Danger, Detox. Other tips were i.a to use native IDEs (XCode, Android Studio) as it helps debugging. XCode Instruments helps debug performance (check iTunes for video) and there’s also Android Profiler.

World Class experience with React Native

React Finland App – Lessons learned — Toni Ristola

Every conference has to have an app and React Finland of course did a React Native app. Toni Ristola lightning talked about lessons learned. Technologies used with React Native was Ignite, GraphQL and Apollo Client ? App’s source code is available on GitHub.

Lessons learned:

  • Have a designer in the team
  • Reserve enough time — doing and testing a good app takes time
  • Test with enough devices — publish alpha early

React Finland App – Lessons learned

React Native Ignite — Gant Laborde

80% of mobile app development is the same old song which can be cut short with Ignite CLI. Using Ignite, you can jump into React Native development with a popular combination of technologies, OR brew your own. Gant Laborde talked about the new Bowser version which makes things even better with Storybook, Typescript, Solidarity, mobx-state-tree and lint-staged. Slides can be found on the Internet.

Ignite
Ignite

How to use React, webpack and other buzzwords if there is no need — Varya Stepanova

Varya Stepanova’s lightning talk suggested to start a side-project other than ToDo app to study new development approaches and showed what it can be in React. The example was how to generate a multilingual static website using Metalsmith, React and other modern technologies and tools which she uses to build her personal blog. Slides can be found on the Internet.

Doing meaningful side-projects is a great idea to study new things and I’ve used that for i.a. learning Swift with Highkara newsreader, did couple of apps for Sailfish OS and played with GraphQL and microservices while developing app with largish vehicle dataset.

After party

Summary

Two days full of talks of React, React Native, React VR and all the things that go with developing web applications with them was great experience. Days were packed with great talks, new information and everything went smoothly. The conference was nicely organized, food was good and participants got soft hoodies to go with the Allas Sea Pool ticket. The talks were all great but especially “World Class experience with React Native” and “React Native Ignite” gave new inspiration to write some app. Also “ReactVR” seemed interesting although I think Augmented Reality will be bigger thing than Virtual Reality. It was nice to hear from “The New Best Practices” talk that there really is no new best practices as the old ones still work. Just use them!

Something to try and even to take into production will be Immer, styled components and Next.js. One thing which is easy to implement is to start using lint-staged although we are linting all the things already.

One of the conference organizers and speaker, Juho Vepsäläinen, wrote Lessons Learned from the conference and many of the points he mentions are to the point. The food was nice but “there wasn’t anything substantial for the afternoon break”. There wasn’t anything to eat after lunch but luckily I had own snacks. Vepsäläinen also mentions that “there was sometimes too much time between the presentations” but I think the longer breaks between some presentations were nice for having a quick stroll outside and have some fresh air. The venue was quite warm and the air wasn’t so good in the afternoon.

The Afterparty at Sea Life Helsinki was interesting choice and it worked nicely although there wasn’t so many people there. The aquarium was fishy experience and provided also some other content than refreshments. Too bad I hadn’t have time to go and check the Allas Sea Pool which we got a free ticket. Maybe next time.

Thanks to the conference crew for such a good event and of course to my fellow Goforeans which attended it and had a great time!

Two days of React Finland 2018: Day one topics of React

React Finland 2018 conference was held last week and I had the opportunity to attend it and listen what’s hot in the React world. The conference started with workshops and after that there was two days of talks of React, React Native, React VR and all things that go with developing web applications with them. The two conference days were packed with great talks and new information. Here’s the first part of my notes from the talks which I posted to Twitter. Read also the second part with more of React Native.

React Finland 2018: Day 1

React Finland combined the Finnish React community with international flavor from Jani Eväkallio to Ken Wheeler and other leading talents of the community. The event was the first of its kind in Finland and consisted of a workshop day and two days of talks around the topic. It was nice that the event was single track so you didn’t need to choose between interesting talks.

At work I’ve been developing with React couple of years and tried my hands with React Native so the topics were familiar. The conference provided crafty new knowledge to learn from and maybe even put to production. Overall the conference was great experience and everything went smoothly. Nice work from the React Finland conference team! And of course thanks to Gofore which sponsored the conference and got me a ticket.

I tweeted my notes from almost every presentation and here’s a recap of the talks. I heard that the videos from the conference will be available shortly.

The New Best Practices — Jani Eväkallio

First day’s keynote was by Jani Eväkallio who talked about “The New Best Practices”. As the talk description wrote “When React was first introduced, it was ridiculed for going against established web development best practices as we knew them. Five years later, React is the gold standard for how we create user interfaces. Along the way, we’ve discovered a new set of tools, design patterns and programming techniques.”

The new best practices were:

  • Build big things from small things
  • Write code for humans first: flow, Typescript, storybook
  • Stay close to the language:
    • helps i.a. linters
  • Always prefer simplicity
  • Don’t break things:
    • Facebook makes React API changes easy to upgrade, depreciation well in advance, migration, documentation. it’s a flow, not versions. Use codemod.
  • Keep an open mind

You ask “what new best practices”? Yep, that’s the thing. We don’t need new best practices as the same concepts like Model-View-Controller and separation of concerns are still valid. We should use best practices which have been proved good before as they also work nicely with React philosophy. Eväkallio also talked why React will be around for a long time. It’s because components and interoperable components are an innovation primitive.

The New Best Practices

Declarative state and side effects — Christian Alfoni

After the keynote it was time to get more practical and Christian Alfoni talked about how we can get help writing our business logic in a declarative manner and see what benefits it gives us. He talked about lessons learned refactoring Codesandbox.io from Redux to Cerebral and about Cerebral which provides a declarative state and side effects management for popular JavaScript frameworks. Talks slides are available on the Internet.

Alfoni also pointed to Turning the database inside out with Apache Samza. Also that Cerebral had time travel before Dan Abramov presented Live React in his talk Hot Reloading with Time Travel at react-europe 2015

Immer: Immutability made easy — Michel Weststrate

Immutable data structures are a good thing and Michel Weststrate showed Immer which is a tiny package that allows you to work with immutable data structures with unprecedented ease. Managing the state of React app is a huge deal with Redux and any help is welcome. “Immer doesn’t require learning new data structures or update APIs, but instead creates a temporarily shadow tree which can be modified using the standard JavaScript APIs. The shadow tree will be used to generate your next immutable state tree.”

The talk showed how to write your reducers in a much more readable way, with half the code and without requiring additional large libraries. The talk slides are available on the Internet.

Get Rich Quick With React Context — Patrick Hund

“Get Rich Quick With React Context” lightning talk by Patrick Hund didn’t tell how good job opportunities you have when doing React But how with React 16.3 the context API has been completely revamped and demonstrated a good use case: Putting ad placements on your web page to get rich quick! Other use cases are localizations. Check out the slides which will tell you how easy it is to use context now and how to migrate your old context code to the new API.

There’s always a better way to handle localization — Eemeli Aro

“There’s always a better way to handle localization” lightning talk by Eemeli Aro told about how localization is a ridiculously difficult problem in the general case, but in the specific you can get away with really simple solutions, especially if you understand the compromises you’re making.

I must have been dozing as all I got was there are also other options to store localizations than JSON like YAML and JavaScript property format especially when dealing with non-developers like translators. The talk was quite general and on abstract level and mentioned solutions to localization were react-intl, react-i18next and react-message-context.

Styled Components, SSR, and Theming — Kasia Jastrzębska

Web applications need to be styled and Kasia Jastrzębska talked about CSS-in-JS with styled-components by going through the new API, performance improvements, server side rendering with Next.js. She also showed the theming manager available with v2 of styled-components. Talk slides are available on the Internet.

Takeaways from this talk was that CSS in React app can be written as you always have or by using CSS-in-JS solutions. There are several benefits of using styled-components but I’m still thinking how styles get scattered all over components.

Universal React Apps Using Next.js — Sia Karamalegos

53% of mobile site visits are abandoned if pages take longer than 3 seconds to load.
DoubleClick by Google, 2016

Every user’s hardware is different and processing speed can hinder user experience on client-side rendered React applications and so Sia Karamalegos talked how server-side rendering and code-splitting can drastically improve user experience. By minimizing the work that the client has to do. Performance and shipping your code matters. The talk showed how to easily build an universal React apps using the Next.js framework and walked through the concepts and code examples. Talk slides are available on the Internet.

There are lots of old (mobile) devices which especially benefit from Server Side Rendering. Next.js is a minimalistic framework for universal, server-rendered (or statically pre-rendered) React applications which enables faster page loads. Pages are server-rendered by default for initial load, you can enable prefetching future routes and there’s automatic code splitting. It’s also customizable so you can use own Babel and Webpack configurations and customize the server API with e.g. Express. And if you don’t want to use a server Next.js can also build static web apps that you can then host on Github pages or AWS S3.

Universal React apps using Next.js

State Management in React Apps with Apollo Client — Sara Vieira

Apollo Client was one of the most mentioned framework in the conference along with Reason ML and Sara Vieira gave energetic talk how to use it for state management in React Apps. If you haven’t come across Apollo Client it’s caching GraphQL client and helps you to manage data coming from the server. Virieira showed how to manage local state with apollo-link-state.

The talk was fast paced and I somewhat missed the why part but at least it’s easy to setup: yarn add apollo-boost graphql react-apollo. Have to see slides and demo later. Maybe the talk can be wrapped up to: “GQL all the things” and “I don’t like Redux” :D

State Management with Apollo

Detox: A year in. Building it, Testing with it — Rotem Mizrachi-Meidan

Detox testing framework for React Native talk by Rotem Mizrachi-Meidan was the other talk I dozed along. Mizrachi-Meidan talked what developing and using Detox in production has taught and how Detox works and what makes it deterministic. The talk showed how mobile apps could be tested. There’s a video of earlier talk on the Internet.

Detox

Make linting great again! — Andrey Okonetchnikov

One thing in software development which always gets developers to argue over stupid things is code formatting and linting. Andrey Okonetchnikov talked how “with a wrong workflow linting can be really a pain and will slow you and your team down but with a proper setup it can save you hours of manual work reformatting the code and reducing the code-review overhead.”

The talk was a quick introduction how ?? lint-staged a node.js library can improve developer experience. Small tool coupled with tools that analyze and improve the code like ESLint, Stylelint, Prettier and Jest can make a big difference.

Missed talks

There was also two talks I missed: “Understanding the differences is accepting” by Sven Sauleau and “Why I YAML” by Eemeli Aro. Sauleau showed “interesting” twists of Javascript language.

Read also the second part with more of React Native.

Build Monitor with Raspberry Pi and Touch Screen

Information is a great tool in software development and it’s useful to have easy access to it. The more obvious you make your problems, the harder you make them to ignore. The more attention they get, the quicker they get solved. One thing developers like to monitor in software development is continuous integration status and metrics from running services. And what better way to achieve visibility and visualize to those metrics than building an information radiator.

I didn’t want to invent the wheel again so I got Raspberry Pi 3 Model B with accessories and 7″ touch screen to base my project. Using a Raspberry Pi as an information radiator isn’t a new idea and the Internet is full of examples of different adaptations with screens, lights, bells and whistles. For the start we just visualized our Jenkins builds and Grafana dashboard but later on we will propably do a custom dashboard.

Setting up the base

The information radiator is easy to get running as you only need a computer which preferably runs Linux. You can use an old laptop and attach it to external screen or if you’re like me and want to tinker you can get e.g. Raspberry Pi 3 and couple it with small external screen for portability. Nice and low cost solution which gets you some hacker value. I got the Rpi from our local hardware store and unfortunately the Model B+ was just released on the same day. The extra 15% power, 5 GHz Wifi and less heat and throttling would’ve been nice.

Raspberry Pi 3 Model B and accessories

I got the Raspberry Pi starting package with the official case, power supply, HDMI cable and a MicroSD card with preloaded NOOBS. So I just needed to connect the cables, put SD Card in and click to install Raspbian. Other interesting operating systems would’ve been Fedberry which is Fedora ‘Minimal, XFCE and LXQt’ Remixes.

For the screen I used 7″ IPS 5 point touch screen for Raspberry Pi with 1024×600 resolution and HDMI from joy-it codename RB-LCD-7-2. Initially I thought I could install the whole system with this display but as it turned out Rpi doesn’t understand it out of the box. It just showed some white noise and interference . Luckily some one had already solved this and I got the right config after I had installed Raspbian with real monitor.

Joy-it touch screen with default settings

Edit your /boot/config.txt:

# uncomment to force a specific HDMI mode (this will force VGA)
hdmi_group=2
hdmi_mode=87

# Add line:
hdmi_cvt=1024 600 60 3 0 0 0

And reboot your Raspberry Pi after those changes.

You should also run $ sudo raspi-config to setup for example WiFi country to allow channels 12 and 13 and your current Timezone.

I also updated Raspbian which bumps it to rpi-4.14.y linux tree:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade
$ sudo rpi-update

To connect to Rpi with SSH enable it with raspi-config > Interfacing Options or just:

$ sudo systemctl enable ssh
$ sudo systemctl start ssh 

For the note, by default the user pi has password raspberry. You should change it but if you want to remove the nagging of default password do the following:

$ sudo apt-get -y purge pprompt
$ sudo rm /etc/profile.d/sshpwd.sh
$ sudo rm /etc/xdg/lxsession/LXDE-pi/sshpwd.sh

Problems with WiFi connection

I set up the Raspberry Pi at our local office and at home and there were no problems with WiFi connection. But when I brought it to customer premises the WiFi connection was weak and practically couldn’t move a bit. My MacBook worked fine but it was connected to 5 GHz network which isn’t an option with my Rpi 3 Model B. The WiFi on Rpi 3 was using channel 11 on 802.11i with WPA2 as shown with iwlist wlan0 scan.

There is a thread on Raspberry Pi forum about Very poor wifi performance which suggest to set up WiFi internalisation correctly to allow channels 12 and 13. At one point the issue was that only channels 1-11 are available on the Rpi 3 but checking out the ‘next’ branch of firmware/kernel (sudo BRANCH=next rpi-update) apparently fixed channels 12/13. I was on kernel 4.9.80 so it wasn’t a problem for me. The other suggested problem is with Atheros chipset based router which doesn’t like Broadcom WiFi on Rpi 3.

For some disabling power management solves the connection issues. For RPi built-in Broadcom (Cypress) WiFi there’s no control for power management and it’s disabled by the kernel. In iw / iwlist / iwconfig you see bug with “Power Management:on”.

But nevertheless testing switching it off made my WiFi connection better but it’s strength didn’t of course change.

$ sudo iwconfig wlan0 power off

To make it permanent you can add something like this in your interfaces file:

$ sudo touch /etc/network/if-up.d/wlan0
$ sudo chmod +x /etc/network/if-up.d/wlan0
$ sudo echo -e '#!/bin/bash\niwconfig wlan0 power off' > /etc/network/if-up.d/wlan0

Accessing Raspberry Pi remotely

The information radiator is usually connected to a TV with no keyboard or mouse attached so accessing it remotely is useful. You can use x11vnc which allows you to VNC into a headless Pi with a VNC client like Apple Remote Desktop, RealVNC’s vncviewer or homebrew’s tiger-vnc.

$ sudo apt-get install ttf-mscorefonts-installer
$ sudo apt-get install x11vnc

To start x11vnc automatically create new or edit existing ~/.xsessionrc file:

$ cat ~/.xsessionrc
/usr/bin/x11vnc -noxrecord -noxdamage -forever -bg -rfbport 5900

Getting interesting things on the screen

To test our setup and quickly show some data I just added a Build Monitor view in Jenkins and other view with Dashboard view. I also configured the Rpi to automatically start Chromium browser in kiosk mode after reboots and directed it to Jenkins website so there would be no need for interactions to get things on the screen. To show several sources of data and get things running quickly without customized information radiator we used Revolver – Tabs Chromium extension to rotate between multiple browser tabs: one showed Jenkins Build Monitor other Grafana Dashboard and third Twitter feed.

To automatically start the chromium-browser after Raspbian desktop starts, edit the following lxsession file:

$ cp /etc/xdg/lxsession/LXDE-pi/autostart /home/pi/.config/lxsession/LXDE-pi/autostart
$ vim /home/pi/.config/lxsession/LXDE-pi/autostart

#@xscreensaver -no-splash  # comment this line out to disable screensaver
# Disable Xsession from blanking
@xset s off
@xset -dpms
@xset s noblank

@sh ./autostart.sh
# load chromium after boot and point to the localhost webserver in full screen mode
@chromium-browser --kiosk --no-default-browser-check --no-first-run --disable-infobars "http://localhost/"

Chromium has a feature to show “Restore pages” nagging popup when not grafefully shutdown and you can try the following Stack Overflow suggestion. What was also suggested was doing “chmod 001 ~/.config/chromium/Default/Preferences” but it results to another nagging window.

$ cat ./autostart.sh
#!/bin/sh
sed -i 's/"exited_cleanly":false/"exited_cleanly":true/' ~/.config/chromium/'Local State'
sed -i 's/"exited_cleanly":false/"exited_cleanly":true/; s/"exit_type":"[^"]\+"/"exit_type":"Normal"/' ~/.config/chromium/Default/Preferences

You could use “–restore-last-session” or “–incognito” parameter which also works but it has several disadvantages, such as disabling the cache and login information. Or maybe I should just use Firefox.

Raspberry Pi and Jenkins Build Monitor

It might be also useful to set Chromium to restart every night. When running Chromium for longer periods it may fill Rpi’s memory with garbage and after it must be hard rebooted.

Turning the monitor on and off automatically

When running Rpi as a wall monitor it’s useful to save energy and extend the life of your monitor by turning the monitor on/off on a daily schedule. You can do this by running a cron script. Get this script and put it in /home/pi/rpi-hdmi.sh and make it executable: chmod +x /home/pi/rpi-hdmi.sh. Call the script at the desired time with cron entry:

$ crontab -e

# Turn HDMI Off (22:00)
0 17 * * * /home/pi/rpi-hdmi.sh off

# Turn HDMI On (7:00)
30 7 * * * /home/pi/rpi-hdmi.sh on 

If you’ve problems with the above script try this one which is the original but change “curr_vt=`fgconsole`” to be “curr_vt=`sudo fgconsole`” as fgconsole needs sudo privileges and otherwise you get an error “Couldn’t get a file descriptor referring to the console”.

Information Radiator in use

From simple dashboard to real information Radiator

Showing just Jenkins Build Monitor or Grafana dashboards is simple but to get more information of things running you could show things like the success rate of the builds, build health, latest open pull request and project Twitter Messages. One nice example of information radiator is Panic’s Status board.

There are different ways to create customizable dashboard and one way is to use Dashing which is a dashboard framework and to get headstart you can see Project Dashboard is which shows “build health” for the latest build, the latest open pull request in Github, the success rate of your builds, some free-form text and your project’s or company’s logo. It uses the Jenkins API to get the ration of successful / non-successful builds as well as the latest build state. It also calls the Github API to get the latest pull request to display the name and picture of the author and the title of the pull request on the dashboard.

For more leisure use you can set up the Raspberry Pi as a wall display to show information like calendar, weather, photos and RSS feeds. One option is to use Dakboard which is a web interface used to display information and is quite configurable with different services. At first Dakboard seems nice but is quite limited on what data it can show and some useful features are premium. Another open source option is MagicMirror² which seems to be more modular and extensible (as you can create your own modules) but needs more tinkering.