Keep Maven dependencies up to date

Software development projects come usually with lots of dependencies and keeping them up to date can be burdensome if done manually. Fortunately there are tools to help you. For Node.js projects there are e.g. npm-check and npm-check-updates and for Maven projects there are OWASP/Dependency-Check and Versions Maven plugins. Here's a short introduction how to setup your Maven project to automatically check dependencies for vulnerabilities and if there's outdated dependencies.

OWASP/Dependency-Check

OWASP dependency-check is an open source solution the OWASP Top 10 2013 entry: "A9 - Using Components with Known Vulnerabilities".

Dependency-check can currently be used to scan Java and .NET applications to identify the use of known vulnerable components. The dependency-check plugin is, by default, tied to the verify or site phase depending on if it is configured as a build or reporting plugin.

The example below can be executed using mvn verify:

<project>
     ...
     <build>
         ...
         <plugins>
             ...
<plugin> 
    <groupId>org.owasp</groupId> 
    <artifactId>dependency-check-maven</artifactId> 
    <version>5.0.0-M3</version> 
    <configuration>
        <failBuildOnCVSS>8</failBuildOnCVSS>
        <skipProvidedScope>true</skipProvidedScope> 
        <skipRuntimeScope>true</skipRuntimeScope> 
    </configuration> 
    <executions> 
        <execution> 
            <goals> 
                <goal>check</goal> 
            </goals> 
        </execution> 
    </executions> 
</plugin>
            ...
         </plugins>
         ...
     </build>
     ...
</project>

The example fails the build for CVSS greater than or equal to 8 and skips scanning the provided and runtime scoped dependencies.

Versions Maven Plugin

The Versions Maven Plugin is the de facto standard way to manage versions of artifacts in a project's POM. From high-level comparisons between remote repositories up to low-level timestamp-locking for SNAPSHOT versions, its massive list of goals allows us to take care of every aspect of our projects involving dependencies.

The example configuration of versions-maven-plugin:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>versions-maven-plugin</artifactId>
    <version>2.7</version>
    <configuration>
        <allowAnyUpdates>false</allowAnyUpdates>
        <allowMajorUpdates>false</allowMajorUpdates>
        <allowMinorUpdates>false</allowMinorUpdates>
        <processDependencyManagement>false</processDependencyManagement>
    </configuration>
</plugin>

You could use goals that modify the pom.xml as described in the usage documentation but often it's easier to check versions manually  as you might not be able to update all of the suggested dependencies.

The display-dependency-updates goal will check all the dependencies used in your project and display a list of those dependencies with newer versions available.

Check new dependencies with:

mvn versions:display-dependency-updates

Check new plugin versions with:

mvn versions:display-plugin-updates

Summary

Using OWASP/Dependency-Check in your Continuous Integration build flow to automatically check dependencies for vulnerabilities and running periodically Versions Maven Plugin to check if there are outdated dependencies helps you to keep your project up to date and secure. Small but important things to remember while developing and maintaining a software project.

Tracking vulnerabilities and keeping Node.js packages up to date

Software evolves quickly and new versions of libraries are released but how do you keep track of updated dependencies and vulnerable libraries? Managing dependencies has always been somewhat a pain point but an important part of software development as it's better to be tracking vulnerabilities and running fresh packages than being pwned.

There are couple of tools for JavaScript projects which use npm to manage dependencies to check new versions and some tools to track vulnerabilities. Here's a short introduction to npm audit, depcheck, npm-check-updates and npm-check to help you on your way.

If your project is using yarn adjust your workflow accordingly. There's for example yarn audit and yarn-check to match tools for npm. And it goes without saying that don't use npm if your project uses yarn.

Running security audit with npm audit

From version 6 onwards npm comes build with audit command which checks for vulnerabilities in your dependencies and runs automatically when you install a package with npm install. You can also run npm audit manually on your locally installed packages to conduct a security audit of the package and produce a report of dependency vulnerabilities and suggested patches.

The npm audit command submits a description of the dependencies configured in your package to your default registry and asks for a report of known vulnerabilities. It checks direct dependencies, devDependencies, bundledDependencies, and optionalDependencies, but does not check peerDependencies.

If your npm registry doesn't support npm audit, like Artifactory, you can pass in the --registry flag to point to public npm. The downside is that now you can't audit private packages that are on the Artifactory registry.

$ npm audit --registry=https://registry.npmjs.org

"Running npm audit will produce a report of security vulnerabilities with the affected package name, vulnerability severity and description, path, and other information, and, if available, commands to apply patches to resolve vulnerabilities."

Example: partial output of npm audit run

Using npm audit is useful also in Continuous Integration as it will return a non-zero response code if security vulnerabilities are found.

For more information read npm's Auditing dependencies for security vulnerabilities.

Updating packages with npm outdated

It's recommended to regularly update the local packages your project depends on to improve your code as improvements to its dependencies are made. In your project root directory, run the update command and then outdated. There should not be any output.

$ npm update
$ npm outdated 
Example of results from npm outdated

You can also update globally-installed packages. To see which global packages need to be updated run outdated first with --depth=0.

$ npm outdated -g --depth=0
$ npm outdated -g

For more information read updating packages downloaded from the registry.

Check updates with npm-check-updates

Package.json contains dependencies with semantic versioning policy and to find newer versions of package dependencies than what your package.json allows you need tools like npm-check-updates. It can upgrade your package.json dependencies to the latest versions, ignoring specified versions while maintaining your existing semantic versioning policies.

Install npm-check-updates globally with:

$ npm install -g npm-check-updates 

And run it with:

$ ncu

The result shows any new dependencies for the project in the current directory. See documentation for i.a. configuration files for filtering and excluding dependencies.

Example of results from ncu

And finally you can run ncu -u to upgrade the package.json.

Check updates with npm-check

Similar tool to npm-check-updates is npm-check which additionally gives more information about the version changes available and also lets you interactively pick which packages to update instead of an all or nothing approach. It checks for outdated, incorrect, and unused dependencies.

Install npm-check globally with:

$ npm i -g npm-check

Now you can run the command inside your project directory:

$ npm-check
Or
$ npm-check --registry=https://registry.npmjs.org

It will display all possible updates with information about the type of update, project URL, commands, and will attempt to check if the package is still in use. You can easily parse through the results and see what packages might be safe to update. When updates are required it will return a non-zero response code that you can use in your CI tools.

The check for unused dependencies uses depcheck and isn't able to foresee all ways dependencies can be used so be vary with careless removing of packages.

To see an interactive UI for choosing which modules to update run:

$ npm-check –u

Analyze dependencies with depcheck

Your package.json is filled with dependencies and some of them might be useless or even missing from package.json. Depcheck is a tool for analyzing the dependencies in a project to see how each dependency is used, which dependencies are useless, and which dependencies are missing. It does not only recognizes the dependencies in JavaScript files, but also supports i.a. React JSX and Typescript.

Install depcheck with:

$ npm install -g depcheck
And with additional syntax support for Typescript
$ npm install -g depcheck typescript

Run depcheck with:

$ depcheck [directory]
Example of results from depcheck

Summary

tl;dr;

  1. Use npm audit in your CI pipeline
  2. Update dependencies with npm outdated
  3. Check new versions of dependencies with either npm-check-updates or npm-check
  4. Analyze dependencies with depcheck

Notes of Best Practices for writing Cypress tests

Cypress is a nice tool for end-to-end tests and it has good documentation also for Best Practices including "Cypress Best Practices" talk by Brian Mann at Assert(JS) 2018. Here are my notes from the talk combined with the Cypress documentation. This article assumes you know and have Cypress running.

In short:

  • Set state programmatically, don't use the UI to build up state.
  • Write specs in isolation, avoid coupling.
  • Don't limit yourself trying to act like a user.
  • Tests should always be able to be run independently and still pass.
  • Only test what you control.
  • Use data-* attributes to provide context to your selectors.
  • Clean up state before tests run (not after).

Organizing tests

- Don't use page objects to share UI knowledge
+ Write specs in isolation, avoid coupling

"Writing and Organizing tests" documentation just tells you the basics how you should organize your tests. You should organize tests by pages and by components as you should test components individually if possible. So the folder structure for tests might look like.

├ articles
├── article_details_spec.js
├── article_new_spec.js
├── article_list_spec.js
├ author
├── author_details_spec.js
├ shared
├── header_spec.js
├ user
├── login_spec.js
├── register_spec.js
└── settings_spec.js

Selecting Elements

- Dont' use highly brittle selectors that are subject to change.
+ Use data-* attributes to provide context to your selectors and insulate them from CSS or JS changes.

Add data-* attributes to make it easier to target elements.

For example:

<button id="main" class="btn btn-large" name="submit"
  role="button" data-cy="submit">Submit</button>

Writing Tests

- Don't couple multiple tests together.
+ Tests should always be able to be run independently and still pass.

Best practice when writing tests on Cypress is to iterate on a single one at a time, i.a.

describe('/login', () => {

  beforeEach() => {
    // Wipe out state from the previous tests
    cy.visit('/#/login')
  }

  it('requires email', () =>
    cy.get('form').contains('Sign in').click()
    cy.get('.error-messages')
    .should('contain', 'email can\'t be blank')
  })

  it('requires password', () => {
    cy.get('[data-test=email]').type('joe@example.com{enter}')
    cy.get('.error-messages')
    .should('contain', 'password can\'t be blank')
  })

  it('navigates to #/ on successful login', () => {
    cy.get('[data-test=email]').type('joe@example.com')
    cy.get('[data-test=password]').type('joe{enter}')
    cy.hash().should('eq', '#/')
  })

})

Note that we don't add assertions about the home page because we're on the login spec, that's not our responsibility. We'll leave that for the home page which is the article spec.

Controlling State

"abstraction, reusability and decoupling"

- Don't use the UI to build up state
+ Set state directly / programmatically

Now you have the login spec done and it's the cornerstone for every single test you will do. So how do you use it in e.g. settings spec? For not to copy & paste login steps to each of your tests and duplicating code you could use custom command: cy.login(). But using custom command for login fails at testing in isolation, adds 0% more confidence and accounts for 75% of the test duration. You need to log in without using the UI. And to do that depends of how your app works. For example you can check for JWT token in the App and in Cypress make a silent (HTTP) request.

So your custom login command becomes:

Cypress.Commands.add('login', () => {
  cy.request({
    method: 'POST',
    url: 'http://localhost:3000/api/users/login',
    body: {
      user: {
        email: 'joe@example.com',
        password: 'joe',
      }
    }
  })
  .then((resp) => {
    window.localStorage.setItem('jwt', resp.body.user.token)
  })
})

Setting state programmatically isn't always as easy as making requests to endpoint. You might need to manually dispatch e.g. Vue actions to set desired values for the application state in the store. Cypress documentation has good example of how you can test Vue web applications with Vuex data store & REST backend.

Visiting external sites

- Don't try to visit or interact with sites or servers you do not control.
+ Only test what you control.

Try to avoid requiring a 3rd party server. When necessary, always use cy.request() to talk to 3rd party servers via their APIs like testing log in when your app uses another provider via OAuth. Or you could try stub out the OAuth provider. Cypress has recipes for different approaches.

Add multiple assertions

- Don't create "tiny" tests with a single assertion and acting like you’re writing unit tests.
+ Add multiple assertions and don’t worry about it

Cypress runs a series of async lifecycle events that reset state between tests. Resetting tests is much slower than adding more assertions.

it('validates and formats first name', function () {
    cy.get('#first')
      .type('johnny')
      .should('have.attr', 'data-validation', 'required')
      .and('have.class', 'active')
      .and('have.value', 'Johnny')
  })

Clean up state before tests run

- Don't use after or afterEach hooks to clean up state.
+ Clean up state before tests run.

When your tests end - you are left with your working application at the exact point where your test finished. If you remove your application's state after each test, then you lose the ability to use your application in this mode or debug your application or write a partial tests.

Unnecessary Waiting

- Don't wait for arbitrary time periods using cy.wait(Number).
+ Use route aliases or assertions to guard Cypress from proceeding until an explicit condition is met.

For example waiting explicitly for an aliased route:

cy.server()
cy.route('GET', /users/, [{ 'name': 'Maggy' }, { 'name': 'Joan' }]).as('getUsers')
cy.get('#fetch').click()
cy.wait('@getUsers')     // <--- wait explicitly for this route to finish
cy.get('table tr').should('have.length', 2)

No constraints

You've native access to everything so don't limit yourself trying to act like a user. You can e.g.

  • Control Time: cy.clock(), e.g. control how your app responds to system time, force set timeouts and set intervals to fire when you want them to.
  • Stub Objects: cy.stub(), force callbacks to fire, assert things are called with right arguments.
  • Modify Stores: cy.window(), e.g. dispatch events, like logout.

Set global baseUrl

+ Set a baseUrl in your configuration file.

Adding a baseUrl in your configuration allows you to omit passing the baseUrl to commands like cy.visit() and cy.request().

Without baseUrl set, Cypress loads main window in localhost + random port. As soon as it encounters a cy.visit(), it then switches to the url of the main window to the url specified in your visit. This can result in a ‘flash’ or ‘reload’ when your tests first start. By setting the baseUrl, you can avoid this reload altogether.

Assertions should be obvious

"A good practice is to force an assertion to fail and see if the error message and the output is enough to know why. It is easiest to put a .only on the it block you're evaluating. This way the application will stop where a screenshot is normally taken and you're left to debug as if you were debugging a real failure. Thinking about the failure case will help the person who has to work on a failing test." (Best practices for maintainable tests)

<code>
it.only('check for tab descendants', () => {
  cy
    .get('body')
    .should('have.descendants', '[data-testid=Tab]') // expected '' to have descendants '[data-testid=Tab]'
    .find('[data-testid=Tab]')
    .should('have.length', 2) // expected '[ <div[data-testid=tab]>, 4 more... ]' to have a length of 2 but got 5
});
</code>

Explore the environment

You can pause the test execution by using debugger keyword. Make sure the DevTools are open.

it('bar', function () {
   debugger
   // explore "this" context
 })

Running in CI

If you're running in Cypress in CI and need to start and stop your web server there's recipes showing you that.

Try the start-server-and-test module. It's good to note that when using e2e-cypress plugin for vue-cli it starts the app automatically for Cypress.

If your videos taken during cypress run freeze when running on CI then increase the CPU resources, see: #4722

Adjust the compression level on cypress.json to minimal with "videoCompression": 0 or disable it with "videoCompression": false. Disable recording with "video": false.

Record success and failure videos

Cypress captures videos from test runs and whenever a test fails you can watch the failure video side by side with the video from the last successful test run. The differences in the subject under test are quickly obvious as Bahtumov's tips suggests.

If you're using e.g. GitLab CI you can configure it to keep artifacts from failed test runs for 1 week, while keeping videos from successful test runs only for a 3 days.

artifacts:
    when: on_failure
    expire_in: '1 week'
    untracked: true
    paths:
      - cypress/videos
      - cypress/screenshots
  artifacts:
    when: on_success
    expire_in: '3 days'
    untracked: true
    paths:
      - cypress/screenshots

Helpful practices

Disable ServiceWorker

ServiceWorkers are great but they can really affect your end-to-end tests by introducing caching and coupling tests. If you want to disable the service worker caching you need to remove or delete navigator.serviceWorker when visiting the page with cy.visit.

it('disable serviceWorker', function () {
  cy.visit('index.html', {
    onBeforeLoad (win) {
      delete win.navigator.__proto__.serviceWorker
    }
  })
})

Note: once deleted, the SW stays deleted in the window, even if the application navigates to another URL.

Get command log on failure

In the headless CI mode, you can get a JSON file for each failed test with the log of all commands. All you need is cypress-failed-log project and include it from your cypress/support/index.js file.

Conditional logic

Sometimes you might need to interact with a page element that does not always exist. For example there might a modal dialog the first time you use the website. You want to close the modal dialog. But the modal is not shown the second time around and the above code will fail.

In order to check if an element exists without asserting it, use the proxied jQuery function Cypress.$:

const $el = Cypress.$('.greeting')
if ($el.length) {
  cy.log('Closing greeting')
  cy.get('.greeting')
    .contains('Close')
    .click()
}
cy.get('.greeting')
  .should('not.be.visible')

Summary

- Don't use the UI to build up state
+ Set state directly / programmatically

- Don't use page objects to share UI knowledge
+ Write specs in isolation, avoid coupling

- Don't limit yourself trying to act like a user
+ You have native access to everything

- Don't couple multiple tests together.
+ Tests should always be able to be run independently and still pass.

- Don't try to visit or interact with sites or servers you do not control.
+ Only test what you control.

- Dont' use highly brittle selectors that are subject to change.
+ Use data-* attributes to provide context to your selectors

- Don't create tests with a single assertion
+ Add multiple assertions and don’t worry about it

- Don't use after or afterEach hooks to clean up state.
+ Clean up state before tests run.

+ Set a baseUrl in your configuration file.

More to read

Use cypress-testing-library which encourage good testing practices through simple and complete custom Cypress commands and utilities.

Set up intelligent code completion for Cypress commands and assertions by adding a triple-slash directive to the head of your JavaScript or TypeScript testing spec file. This will turn the IntelliSense on a per file basis.

/// <reference types="Cypress" />

Read What I’ve Learned Using Cypress.io for the Past Three Weeks if you need a temporary workaround for iframes and testing file uploads as for now Cypress does not natively support those.

And of course Gleb Bahmutov's blog is useful resource for practical things like Tips and tricks post.

Automate validating code changes with Git hooks

What could be more annoying than committing code changes to repository and noticing afterwards that formatting isn't right or tests are failing? Your automated tests on Continuous Integration shows rain clouds and you need to get back to the code and fix minor issues with extra commits polluting the git history? Fortunately with small enhancements to your development workflow you can automatically prevent all the hassle and check your changes before committing them. The answer is to use Git hooks for example on pre-commit for running linters and tests.

Git Hooks

Git hooks are scripts that Git executes before or after events such as: commit, push, and receive. They're a built-in feature and run locally. Hook scripts are only limited by a developer's imagination. Some example hook scripts include:

  • pre-commit: Check the commit for linting errors.
  • pre-receive: Enforce project coding standards.
  • post-commit: Email team members of a new commit.
  • post-receive: Push the code to production.

Every Git repository has a .git/hooks folder with a script for each hook you can bind to. You're free to change or update these scripts as necessary, and Git will execute them when those events occur.

Git hooks can greatly increase your productivity as a developer as you can automate tasks and ensure that your code is ready for commit or pushing to remote repository.

For more reading about Git hooks you can check missing Git hooks documentation, read the basics and check tutorial how to use Git hooks on local Git clients and Git servers.

Pre-commit

One productive way to use Git hooks is pre-commit framework for managing and maintaining multi-language pre-commit hooks. Read tips for using a pre-commit hook.

Pre-commit is nice for example running linters to ensure that your changes conform to coding standards. All you need is to install pre-commit and then add hooks.

Installing pre-commit, ktlint and pre-commit-hook on MacOS with Homebrew:

$ brew install pre-commit
$ brew install ktlint
$ ktlint --install-git-pre-commit-hook

For example the pre-commit hook to run ktlint with auto-correct option looks like the following in projects .git/hooks/pre-commit. The "export PATH=/usr/local/bin:$PATH" is for SourceTree to find git on MacOS.

#!/bin/sh
export PATH=/usr/local/bin:$PATH
# https://github.com/shyiko/ktlint pre-commit hook
git diff --name-only --cached --relative | grep '\.kt[s"]\?$' | xargs ktlint -F --relative .
if [ $? -ne 0 ]; then exit 1; else git add .; fi

The main disadvantage is using pre-commit and local git hooks is that hooks are kept within .git directory and it never comes to the remote repository. Each contributor will have to install them manually in his local repository which may be overlooked.

Maven projects

Githook Maven plugin deals with the problem of providing hook configuration to the repository and automates their installation. It binds to Maven projects build process and configures and installs local git hooks.

It keeps a mapping between the hook name and the script by creating a respective file in .git/hooks for each hook containing given script in Maven project's initial lifecycle phase. It's good to notice that the plugin rewrites hooks.

Usage Example:

<build>
    <plugins>
	<plugin>
	    <groupId>org.sandbox</groupId>
	    <artifactId>githook-maven-plugin</artifactId>
	    <version>1.0.0</version>
	    <executions>
	        <execution>
	            <goals>
	                <goal>install</goal>
	            </goals>
	            <configuration>
	                <hooks>
	                    <pre-commit>
	                         echo running validation build
	                         exec mvn clean install
	                    </pre-commit>
	                </hooks>
	            </configuration>
	        </execution>
	    </executions>
	</plugin>
    </plugins>
</build>

Git hooks for Node.js projects

On Node.js projects you can define scripts in package.json and run them with npm which enables an another approach to running Git hooks.

🐶 Husky is Git hooks made easy for Node.js projects. It keeps existing user hooks, supports GUI Git clients and all Git hooks.

Installing Husky is like any other npm library

npm install husky --save-dev

The following configuration on your package.json runs lint (e.g. eslint with --fix) command when you try to commit and runs lint and tests (e.g. mocha, jest) when you try to push to remote repository.

"husky": {
   "hooks": {
     "pre-commit": "npm run lint",
     "pre-push": "npm run lint && npm run test"
   }
}

Another useful tool is lint-staged which utilizes husky and runs linters against staged git files.

Summary

Make your development workflow easier by automating all the things. Check your changes before committing them with pre-commit, husky or Githook Maven plugin. You get better code and commit quality for free and your team is happier.

This article was originally published at 15.7.2019 on Gofore's blog.

Ignoring files and folders in Subversion with propset

Before committing code to the Subversion repository we always set the svn:ignore property on the directory to prevent some files and directories to be checked in. You would usually want to exclude the IDE project files and the target/ directory.

It's useful to put all the ignored files and directories into a file: .svnignore. Your .svnignore could look like:

*.iml
target/*

Put the .svnignore file in the project folder and commit it to your repository so the ignored files are shared between committers.

Now reference the file with the -F option:
$ svn propset svn:ignore -F .svnignore.

Of course I hope everyone has by now moved to git and uses .gitignore for this same purpose.

Best Practices of forking git repository and continuing development

Sometimes there's a need to fork a git repository and continue development with your own additions. It's recommended to make pull request to upstream so that everyone could benefit of your changes but in some situations it's not possible or feasible. When continuing development in forked repo there's some questions which come to mind when starting. Here's some questions and answers I found useful when we forked a repository in Github and continued to develop it with our specific changes.

Repository name: new or fork?

If you're releasing your own package (to e.g. npm or mvn) from the forked repository with your additions then it's logical to also rename the repository to that package name.

If it's a npm package and you're using scoped packages then you could also keep the original repository name.

Keeping master and continuing developing on branch?

Using master is the sane thing to do. You can always sync your fork with an upstream repository. See: syncing a fork

Generally you want to keep your local master branch as a close mirror of the upstream master and execute any work in feature branches (that might become pull requests later).

How you should do versioning?

Suppose that the original repository (origin) is still in active development and does new releases. How should you do versioning in your forked repository as you probably want to bring the changes done in the origin to your fork? And still maintain semantic versioning.

In short, semver doesn't support prepending or appending strings to version. So adding your tag to the version number from the origin which your version is following breaks the versioning. So, you can't use something like "1.0.0@your-org.0.1" or "1.0.0-your-org.1". This has been discussed i.a. semver #287. The suggestion was to use a build meta tag to encode the other version as shown in semver spec item-10. But the downside is that "Build metadata SHOULD be ignored when determining version precedence. Thus two versions that differ only in the build metadata, have the same precedence."

If you want to keep relation the original package version and follow semver then your options are short. The only option is to use build meta tag: e.g. "1.0.0+your-org.1".

It seems that when following semantic versioning your only option is to differ from origin version and continue as you go.

If you don't need to or want to follow semver you can track upstream version and mark your changes using similar markings as semver pre-releases: e.g. "1.0.0-your-org.1".

npm package: scoped or unscoped?

Using scoped packages is a good way to signal official packages for organizations. Example of using scoped packages can be seen from Storybook.

It's more of a preference and naming conventions of your packages. If you're using something like your-org-awesome-times-ahead-package and your-org-patch-the-world-package then using scoped packages seems redundant.

Who should be the author?

At least add yourself to contributors in package.json.

Forking only for patching npm library?

Don't fork, use patch-package which lets app authors instantly make and keep fixes to npm dependencies. Patches created by patch-package are automatically and gracefully applied when you use npm(>=5) or yarn. Now you don't need to wait around for pull requests to be merged and published. No more forking repos just to fix that one tiny thing preventing your app from working.

This post was originally published on Gofore Group blog at 11.2.2019.

Expanding your horizons on JVM with Kotlin

The power of Java ecosystem lies in the Java Virtual Machine (JVM) which runs variety of programming languages which are better suitable for some tasks than Java. One relatively new JVM language is Kotlin which is statically typed programming language that targets the JVM and JavaScript. You can use it with Java, Android and the browser and it's 100% interoperable with Java. Kotlin is open source (Apache 2 License) and developed by a team at JetBrains. The name comes from the Kotlin Island, near St. Petersburg. The first officially considered stable release of Kotlin v1.0 was released on February 15, 2016.

Why Kotlin?

"Kotlin is designed to be an industrial-strength object-oriented language, and to be a better language than Java but still be fully interoperable with Java code, allowing companies to make a gradual migration from Java to Kotlin." - Kotlin, Wikipedia

Kotlin's page summaries the question "Why Kotlin?" to:

  • Concise: Reduce the amount of boilerplate code you need to write.
  • Safe: Avoid entire classes of errors such as null pointer exceptions.
  • Versatile: Build server-side applications, Android apps or frontend code running in the browser. You can write code in Kotlin and target JavaScript to run on Node.js or in browser.
  • Interoperable: Leverage existing frameworks and libraries of the JVM with 100% Java Interoperability.

"You can write code that’s more expressive and more concise than even a scripting language, but with way fewer bugs and with way better performance." - Why Kotlin is my next programming language

One of the obvious applications of Kotlin is Android development as the platform uses Java 6 although it can use most of Java 7 and some backported Java 8 features. Only the recent Android N which changes to use OpenJDK introduces support for Java 8 language features.

For Java developers one significant feature in Kotlin is Higher-Order Functions, function that takes functions as parameters, which makes functional programming more convenient than in Java. But in general, I'm not so sure if using Kotlin compared to Java 8 is as much beneficial. It smooths off a lot of Java’s rough edges, makes code leaner and costs nothing to adopt (other than using IntelliJ IDEA) so it's at least worth trying. But if you're stuck with legacy code and can't upgrade from Java 6, I would jump right in.

Learning Kotlin

Coming from Java background Kotlin at first glance looks a lot leaner, elegant, simpler and the syntax is familiar if you've written Swift. To get to know the language it's useful to do some Kotlin Examples and Koans which get you through how it works. They also have "Convert from Java" tool which is useful to see how Java classes translate to Kotlin. For mode detailed information you can read the complete reference to the Kotlin language and the standard library.

If you compare Kotlin to Java you see that null references are controlled by the type system, there's no raw types, arrays are invariant (can't assign an Array to an Array) and there's no checked exceptions. Also semicolons are not required, there's no static members, non-private fields or wildcard types.

And what Kotlin has that Java doesn't have? For starters there's null safety, smart casts, extension functions and lots of things Java just got in recent versions like Null safety, streams, lambdas ( although which are "expensive"). On the other hand Kotlin targets Java 6 bytecode and doesn't use some of the improvements in Java 8 like invoke-dynamic or lambda support. Some of JDK7/8 features are going to be included in Standard Library in 1.1 and in the mean time you can use small kotlinx-support library. It provides extension and top-level functions to use JDK7/JDK8 features such as calling default methods of collection interfaces and use extension for AutoCloseable.

And you can also call Java code from Kotlin which makes it easier to write it alongside Java if you want to utilize it in existing project and write some part of your codebase with Kotlin.

The Kotlin Discuss is also nice forum to read experiences of using Kotlin.

Tooling: in practice IntelliJ IDEA

You can use simple text editors and compile your code from the command line or use build tools such as Ant, Gradle and Maven but good IDEs make the development more convenient. In practice, using Kotlin is easiest with JetBrains IntelliJ IDEA and you can use their open source Community edition for free. There's also Eclipse plugin for Kotlin but naturally it's much less sophisticated than the IntelliJ support.

Example project

The simplest way to start with Kotlin application is to use Spring Boot's project generator, add your dependencies, choose Gradle or Maven and click on "Generate Project".

There are some gotchas with using Spring and Kotling together which can be seen from Spring + Kotlin FAQ. For example by default, classes are final and you have to mark them as "open" if you want the standard Java behaviour. This is useful to know with @Configuration classes and @Bean methods. There's also Kotlin Primavera which is a set of libraries to support Spring portfolio projects.

For example Spring Boot + Kotlin application you should look at Spring.io writeup where they do a geospatial messenger with Kotlin, Spring Boot and PostgreSQL

What does Kotlin look like compared to Java?
Simple example of using Java 6, Java 8 and Kotlin to filter a Map and return a String. Notice that Kotlin and Java 8 are quite similar.

# Java 6
String result = "";
for (Map.Entry<Integer, String> entry : someMap.entrySet()) {
	if("something".equals(entry.getValue())){
		result += entry.getValue();
}
 
# Java 8
String result = someMap.entrySet().stream()
		.filter(map -> "something".equals(map.getValue()))
		.map(map->map.getValue())
		.collect(Collectors.joining());
 
# Kotlin
val result = someMap
  .values
  .filter { it == "something" }
  .joinToString("")
 
# Kotlin, shorter 
val str = "something"
val result = str.repeat(someMap.count { it.value == str })
 
# Kotlin, more efficient with large maps where only some matching.
val result = someMap
  .asSequence()
  .map { it.value }
  .filter { it == "something" }
  .joinToString("")

The last Kotlin example makes the evaluation lazy by changing the map to sequence. In Kotlin collections map/filter methods aren't lazy by default but create always a new collection. So if we call filter after values method then it's not as efficient with large maps where only some elements are matching the predicate.

Using Java and Kotlin in same project

To start with Kotlin it's easiest to mix it existing Java project and write some classes with Kotlin. Using Kotlin in Maven project is explained in the Reference and to compile mixed code applications Kotlin compiler should be invoked before Java compiler. In maven terms that means kotlin-maven-plugin should be run before maven-compiler-plugin.

Just add the kotlin and kotlin-maven-plugin to your pom.xml as following

<dependencies>
    <dependency>
        <groupId>org.jetbrains.kotlin</groupId>
        <artifactId>kotlin-stdlib</artifactId>
        <version>1.0.3</version>
    </dependency>
</dependencies>
 
<plugin>
    <artifactId>kotlin-maven-plugin</artifactId>
    <groupId>org.jetbrains.kotlin</groupId>
    <version>1.0.3</version>
    <executions>
        <execution>
            <id>compile</id>
            <phase>process-sources</phase>
            <goals> <goal>compile</goal> </goals>
        </execution>
        <execution>
            <id>test-compile</id>
            <phase>process-test-sources</phase>
            <goals> <goal>test-compile</goal> </goals>
        </execution>
    </executions>
</plugin>

Notes on testing

Almost everything is final in Kotlin by default (classes, methods, etc) which is good as it forces immutability, less bugs. In most cases you use interfaces which you can easily mock and in integration and functional tests you're likely to use real classes, so even then final is not an obstacle. For using Mockito there's Mockito-Kotlin library https://github.com/nhaarman/mockito-kotlin which provides helper functions.

You can also do better than just tests by using Spek which is a specification framework for Kotlin. It allows you to easily define specifications in a clear, understandable, human readable way.

There's yet no static analyzers for Kotlin. Java has: FindBugs, PMD, Checkstyle, Sonarqube, Error Prone, FB infer. Kotlin has kotlinc and IntelliJ itself comes with static analysis engine called the Inspector. Findbugs works with Kotlin but detects some issues that are already covered by the programming language itself and are impossible in Kotlin.

To use Kotlin or not?

After writing some classes with Kotlin and testing converting existing Java classes to Kotlin it makes the code leaner and easier to read especially with data classes like DTOs. Less (boilerplate) code is better. You can call Java code from Kotlin and Kotlin code can be used from Java rather smoothly as well although there are some things to remember.

So, to use Kotlin or not? It looks a good statically-typed alternative to Java if you want to expand your horizons. It's pragmatic evolution to Java that respects the need for good Java integration and doesn’t introduce anything that’s terribly hard to understand and includes a whole bunch of features you might like. The downsides what I've come across are that tooling support is kind of limited, meaning in practice only IntelliJ IDEA. Also documentation isn't always up to date or updated when the language evolves and that's also an issue when searching for examples and issues. But hey, everything is fun with Kotlin :)

Patching RichFaces 3.3.3 AJAX.js for IE11

Couple of years ago I wrote about patching RichFaces 3.3.3 AJAX.js for IE9 and as the browser world has moved on, it's now time to patch RichFaces 3.3.3 AJAX.js for Internet Explorer 11. Of course you could update your web application to JSF 2 and RichFaces 4 or PrimeFaces but it's neither trivial nor free. The issue with RichFaces 3.3.3 still stands, development has moved to 4.x version and they've dropped support for the older versions although at least IE issues could be easily fixed. Fortunately patching RichFaces AJAX.js is relatively easy.

The problem with Internet Explorer 11 is that although you upgraded Sarissa Framework and patched RichFaces AJAX.js file for IE 9 it just isn't enough anymore as IE 11 uses different User-agent string and disguises itself as "like Gecko". The User-agent string in Win 8.1 for IE11 is "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv:11.0) like Gecko". The issue is also discussed on JBoss Forums and on Stack Overflow.

The different User-agent string breakes the "rerender" after Ajax Request. For example using a4j:function in combination with "rerender" is not working on IE 11 and the problem is that after Ajax request the result XML can't be correctly append to the body. Also the rerendering is abnormal for h:inputTextarea, rich:modalPanel, h:inputTextarea and rich:calendar.

Fortunately the fix is simple as RicFaces issue RF-13443 tells.

Patching RichFaces 3.3.3 AJAX.js for IE 11

Step 1: Do the "Upgrading Sarissa Framework and patching RichFaces AJAX.js file" steps in my Patching RichFaces 3.3.3 AJAX.js for IE9 article.

Step 2: Edit the AJAX_IE9fix.js file you just created and add IE 11 checking to Sarissa._SARISSA_IS_IE and Sarissa._SARISSA_IS_IE9. The changed lines should look like the following:

Sarissa._SARISSA_IS_IE = (document.all && window.ActiveXObject && navigator.userAgent.toLowerCase().indexOf("msie") > -1 && navigator.userAgent.toLowerCase().indexOf("opera") == -1) || (navigator.userAgent.toLowerCase().indexOf("like gecko") > -1 && navigator.userAgent.toLowerCase().indexOf("11.") > -1);
 
Sarissa._SARISSA_IS_IE9 = Sarissa._SARISSA_IS_IE && (parseFloat(navigator.appVersion.substring(navigator.appVersion.indexOf("MSIE")+5))) >= 9 || (navigator.userAgent.toLowerCase().indexOf("like gecko") > -1 && navigator.userAgent.toLowerCase().indexOf("11.") > -1);

If all fails you can try to tell IE 11 to emulate IE 10 with adding X-UA-Compatible meta tag to head.

<meta http-equiv="X-UA-Compatible" content="IE=EmulateIE10"/>

The other thing I noticed with RichFaces 3.3.3 and modern browsers is that if you use ui:fragment which contains rich:suggestionbox and rerender it, the suggestionbox doesn't work correctly. It gives an error: SCRIPT5007: Unable to get property 'parentNode' of undefined or null reference. For now I didn't have time to figure out the issue and just changed my page structure and function.

Although it's 2015 using JSF 1.2 and RichFaces 3.3.3 is still working quite nicely :)

Using Bacon2D with Sailfish OS

Sailfish OS development for Jolla is quite versatile and you can use QML plugins to enhance your application. But if you've used a QML module which isn't approved in Harbour and want to distribute your application via Jolla Store, you've to do some extra steps with it. Lately I ported Ubuntu Phone's Falldown game for Sailfish OS and it uses Bacon2D framework to ease 2D game development. Bacon2D isn't allowed in Harbour so I had to patch it to accept custom namespace and either package the dynamically linked library with my app's RPM or use statically linked Bacon2D with my app. Here are my notes about it.

The examples will be with my Falldown game project and the code can be found on GitHub. The steps shown below are not the optimal way of doing things as you could streamline the process by including Bacon2D build to your project build and in RPM spec patch the needed files. Will see if I get around doing that and automate things.

Building Bacon2D for Sailfish OS

Bacon2D is a framework to ease 2D game development, providing ready-to-use QML elements representing basic game entities needed by most of games. Bacon2D uses Box2D, a 2D rigid body simulation engine for games. Box2D lets you make objects move in realistic ways and make the game world more interactive.

You can use Bacon2D in your Sailfish OS project by using it as dynamically or statically linked library. The easiest way is to just use the statically linked version so it's statically compiled into your project. But it's almost as easy to build the dynamically linked library.

The other issue is getting the Bacon2D built such way that you can use it in your project and submit your application to Jolla Store. As Bacon2D plugin isn't allowed in Harbour you've to patch it to your own namespace, e.g harbour.falldown.bacon2d.

The next chapters cover the three different ways to build and use Bacon2D with Sailfish OS.

Get Bacon2D framework

Start with adding the Bacon2D repository to your project's Git repository as submodule.

$ git submodule add https://github.com/Bacon2D/Bacon2D.git

Now the Bacon2D project files are in e.g. falldown/Bacon2D and falldown directory will be our base.

Now you're ready to build and use Bacon2D with three optional ways:

After I got the needed things to work, I used the Option 3 way to get Bacon2D included in my application and hopefully it gets approved to Jolla Store.

Option 1: Using statically linked Bacon2D

In your project's project file e.g. harbour-falldown.pro add to the top of the file include clause for including Bacon2D statically

include(./Bacon2D/src/Bacon2D-static.pri)

In your project's main.cpp include plugins.h and register types before loading QML resources.

#include <plugins.h>
 
Plugins plugins;
plugins.registerTypes("Bacon2D");

Build your project in Qt Creator by selecting Build->Release, Deploy->Deploy as RPM Package and then running Build -> Clean, Build, Deploy.

Your Bacon2D powered app should now launch in your phone.

Option 2: Using dynamically linked Bacon2D

Open Bacon2D/Bacon2D.pro with Qt Creator and disable the test and examples subdirectories.

SUBDIRS += src

Build Bacon2D for Armv7: Select Kit - MerSDK-SailfishOS-armv7hl - Release - Deploy by copying binaries. The build will fail with "Error 127" but the libbacon2dplugin.so is built so you can use it now.

Copy the "build-Bacon2D-MerSDK_SailfishOS_armv7hl-Release/src/imports" directory which contains QML files and libbacon2dplugin.so to your project's lib directory, e.g. falldown/lib.

$ cp -r build-Bacon2D-MerSDK_SailfishOS_armv7hl-Release/src/imports/Bacon2D lib

In e.g. harbour-falldown.pro add the lib directory to be added inside RPM.

DEPLOYMENT_PATH = /usr/share/$${TARGET}
lib.files += lib
lib.path = $${DEPLOYMENT_PATH}
INSTALLS += lib

In your main.cpp add import path to Bacon2D files.

view->engine()->addImportPath(SailfishApp::pathTo("qml/imports/").toLocalFile());

In Qt Creator select Build->Release, Deploy->Deploy as RPM Package. Run Build -> Clean, Build, Deploy.

Your Bacon2D powered app for Sailfish OS should now launch in Jolla.

Option 3: Using statically linked Bacon2D the Harbour way

Including statically linked Bacon2D to be built with you Sailfish OS application can be done different ways and I chose not to write my own plugins.cpp to register QML types and just patched the plugins.cpp which Bacon2D provides.

In harbour-falldown.pro add to the top of the file include clause for including Bacon2D statically

include(./Bacon2D/src/Bacon2D-static.pri)

Patch Bacon2D Bacon2D/src/plugins.cpp to accept e.g. harbour.falldown.bacon2d namespace. Change every qmlRegisterType line from "Bacon2D" to your namespace.

qmlRegisterType<Layer>("harbour.falldown.bacon2d", 1, 0, "Layer");

In your project's src/main.cpp include plugins.h and register types before loading QMLresources.

#include <plugins.h>
 
Plugins plugins;
plugins.registerTypes("harbour.falldown.bacon2d");

Put Bacon2D QML files under e.g. qml/components and change them to import e.g. harbour.falldown.bacon2d 1.0. Somehow I didn't get the Bacon2D type to work which defines enums.

Build your project in Qt Creator by selecting Build->Release, Deploy->Deploy as RPM Package and then running Build -> Clean, Build, Deploy.

Your Bacon2D powered app should now launch in your phone.

Option 4: Using dynamically linked Bacon2D the Harbour way

You can link against shared libraries which ships with your Sailfish OS app in the RPM but because of Harbour rules you have to install the library under e.g. /usr/share/harbour-falldown/lib.

But before you can use the dynamically linked Bacon2D for Sailfish OS app intended to be distributed via Jolla Harbour you have to patch Bacon2D to accept e.g. harbour.falldown.bacon2d namespace and build the library.

In Bacon2D project change the following files.

Bacon2D/Bacon2D.pro

PROJECT_NAME = harbour.falldown.bacon2d

Bacon2D/src/src.pro

TARGETPATH = harbour/falldown/bacon2d
DESTDIR = $$OUT_PWD/imports/harbour/falldown/bacon2d/

Bacon2D/src/plugins.cpp

qmlRegisterType<Layer>("harbour.falldown.bacon2d", 1, 0, "Layer");

Bacon2D/src/qmldir

module harbour.falldown.bacon2d

In Bacon2D/src directory change every QML file to import e.g. harbour.falldown.bacon2d 1.0

Build Bacon2D with Qt Creator for Armv7: Kit - MerSDK-SailfishOS-armv7hl - Release - Deploy by copying binaries

Copy Bacon2D QML files and libbacon2dplugin.so under lib directory and to the structure which match you import namespace, e.g.: lib/harbour/falldown/bacon2d/.

$ cp -r build-Bacon2D-MerSDK_SailfishOS_armv7hl-Release/src/imports/harbour lib

In your project’s project file, e.g. harbour-falldown.pro add the lib directory to be added inside RPM

DEPLOYMENT_PATH = /usr/share/$${TARGET}
lib.files += lib
lib.path = $${DEPLOYMENT_PATH}
INSTALLS += lib

In your spec file, e.g. rpm/harbour-falldown.spec add following exclude as Harbour RPM packages should not provide anything.

# >> macros
%define __provides_exclude_from ^%{_datadir}/.*$
# << macros

Also to use QML modules which ships together with the application but you have to prefix the name of the imports with e.g. "harbour.falldown.bacon2d". And you have to install them under /usr/share/harbour-falldown (loadable QML plugins or the QML files).

In src/main.cpp add import path to Bacon2D QML files and plugin before loading qml resources.

view->engine()->addImportPath(SailfishApp::pathTo("lib/").toLocalFile());

Build your project in Qt Creator by selecting Kit – MerSDK-SailfishOS-armv7hl – Release - Deploy - Deploy as RPM Package and then running Clean, Build, Deploy.

Your Bacon2D powered app should now launch in your phone.

Starting iOS development with Swift

Mobile application development differs between platforms and after doing couple of applications for the Sailfish OS powering Jolla phone it was finally time to see what other platforms have to provide. I develop Java applications at work so it was logical to look into iOS and learn some Swift. The Internet is full of resources of how to start developing for iOS and here’s my take to the topic. Now I just need an iPhone to run my app on a real device :)

Getting started

Coming from the Java EE and Web application world it's good to read some documentation about mobile application development for iPhone and iOS before starting to code. You need to learn the basics concepts about iOS platform and Swift language and good starting point is to check Apple's resources for developers and iOS developer library and read the guide how to start developing iOS apps (although it's with Objective-C). To learn Swift you can read guide to Swift language or if you like books there’s also The Swift Programming Language book.

You can also start learning iOS development with several free or paid online courses. Coding with Chris "how to make an iPhone app" series of videos is a good starting point although it's designed for people who have no programming experience. It provides nice overview to the tools and how to start developing. You can also follow the App Development: Teaching Swift by Apple Education with code examples or if you've the money and need diploma see the Udemy course for iOS developers or Udacity's iOS developer Nanodegree.

It's also good to read Human Interface Guidelines for iOS-based devices although the guidelines don't provide any practical examples. It's a good resource to learn how iOS applications should work, tells you how your app should look and behave and how to use the UI elements that UIKit provides. As I have done apps for Sailfish OS it was good to adapt my thinking to see things in the iOS way.

In practice the best way to learn is to just write code and experiment. Getting to know XCode and Interface Builder takes some time. After using Eclipse and IntelliJ IDEA for Java development both XCode and Apple’s graphical UI editor are somewhat confusing at first.

There's also couple of iOS development newsletters to follow: iOS Dev Weekly, This Week In Swift. Also the In depth Mac and iOS articles archives is a good resource.

And if you're using IRC there's #cocoa-init channel on irc.freenode.net focused on asking and answering questions for beginning developers on iOS and OS X. For more general iOS development you can join #iphonedev channel on irc.freenode.net. To join them you need to register your nick.

Development in practice

Getting started with iOS can be challenging as Swift and Objective-C are used mainly only on Apple's platform and it has its own names for almost everything. I haven't yet gathered my own good practices so it's great that you can read about iOS good practices from Futurice.

Basic tools

For iOS development your options with tools are somewhat limited as you need Mac computer running OS X (10.9.4 or later) for being able to run Apple Xcode and iOS SDK. The other option is to use JetBrains Appcode (99e) which is better (what I've understood). Xcode can be installed from Mac App Store and it comes with iOS SDK. Also although you can run your applications in iOS Simulator it helps to have a real device which helps you to understand how apps interact with users and what the look and feel should be. The documentation and examples gets you far but nothing beats to have first hand experience of the platform.

I found it beneficial to watch e.g. the Coding with Chris "how to make an iPhone app" series of videos for getting around XCode development environment.

Xcode and UI builder

Developing for iOS

iPhone applications can be written with Objective-C or with newer Swift programming language. Objective-C is built on top of the C programming language and provides object-oriented capabilities and a dynamic runtime. Swift in the other hand can be described as Objective without the C and is a replacement for the Objective-C language and works side-by-side with it. Wikipedia has good short description of Swift.

Although Swift is relatively new and what I've read isn't quite as robust as Objective-C it’s good starting point for developing iOS apps. Having used some Objective-C for OS X apps I wouldn't recommend it to anyone if you don't actually need it.

iOS platform

Apple's operating system for iPhone, iOS, provides many frameworks for developers and as a developer you've to decide which version to target as it affects your application and capabilities. Apple's iOS developer page provides short overview of what it has to offer. The current version is iOS 8 and i.a. Shinobicontrols has written iOS 8 Day by Day eBook which consists of 39 blog posts covering the most significant features available to developers in iOS 8.

As a developer you have to choose which version of iOS you target and whether your app is universal or only for iPhone or iPad as it affects your code and potential users. Do you need the new features in iOS 8 or is iOS 7 enough and is it worth to make solutions to fit both versions? And do you need different user interface for bigger screen in iPad or is universal version enough? iOS 8 is supported on iPhone 4s or newer and newer iPads and what I read about 72% of devices are using iOS 8 and 25% are on iOS 7. So, I think it's enough to target iOS 8 as it provides you more options how to implement your app.

Developing with Swift

The best way to get to know Swift is just read some code, watch tutorials and of course write code. To learn Swift you can read guide to Swift language and you can watch from Youtube e.g. rm2kdev series about Swift starting from the basics and example of doing a To Do List app.

One nice element of the Swift system which helps you to get hang of it, is its ability to be cleanly debugged and run within the development environment, using a read–eval–print loop (REPL). It gives you interactive properties more in common with the scripting capabilities of Python than traditional systems programming languages. It's useful to play with Swift in Xcode Playground and see what your code does.

Knowing Swift language is just one part and the bigger part in my opinion is to know how to use the UI elements that UIKit provides to create good user experience. When I started developing my app with Swift majority of time went to figuring out how to construct the user interface with Xcode UI builder and what the elements should be, not actually writing much code in Swift. User experience section in iOS Developer Library and UI Elements in iOS Human Interface Guidelines provide good starting points for reading about user experience but doesn't help you much with coding especially when the sample codes are in Objective-C. Basic UI elements like "pull to refresh", "swipe between views", "split view" and "slide-out navigation" are more easily found by googling.

We all have our own ways to learn new platform and programming language and I find it beneficial to just create small application which experiment with different concepts and interactions. I'm gathering my own experiments with iOS technologies to GitHub and building a news reader application for High.fi on the way.

So, what are you waiting for? Download Xcode and start developing your own app for iOS :)

Summary

Documentation

iOS 8

Other resources

Courses and guides