Automate your dependency management using update tool

Software often consists of not just your own code but also is dependent of third party libraries and other software which has their own update cycle and new versions are released now and then with fixes to vulnerabilities and with new features. Now the question is what is your dependency management strategy and how do you automate it?

Fortunately automated dependency updates for multiple languages is a solved problem as there are several update tools to help you: Renovate, Dependabot (GitHub), Greenkeeper ($), Depfu ($) and Dependencies.io ($) to name some alternatives. In this blog post I will concentrate on using Renovate and integrate it with GitLab CI.

Renovate your dependencies

Renovate is open source tool which works with most git hosting platforms (public or self-hosted) and it's possible to host Renovate Bot yourself. It’s installable via npm/yarn or Docker Hub.

In short, the idea and workflow of dependency update tools are following:

  1. Checks for updates: pulls down your dependency files and looks for any outdated or insecure requirements.
  2. Opens pull requests: If any of your dependencies are out-of-date, tool opens individual pull requests to update each one.
  3. Review and merge: You check that your tests pass, scan the included changelog and release notes, then hit merge with confidence.

Now you just run the depedency update tool on regular basis on your continuous integration, watch how the pull requests fly and you get to keep your dependencies secure and up-to-date.

The manual chore of checking for updates, looking for changelogs, making changes, running tests, writing pull requests and more is now moved to reviewing pull requests with better confidence of what has changed.

Self-hosted in GitLab CI

Renovate Bot is a node.js application so you’ve couple of alternative ways to run it on your CI/CD environment. You can use a node docker image which installs and runs renovate, or you can use Renovate Bot's own Docker image as I chose to do. We are using docker-in-docker approach of running the Renovate docker container. That means you can start Docker containers from within an other Docker container.

First create an  account for the bot on the Gitlab instance (the best choice) or use your own account. Then generate a personal access token with the api scope for renovate to access the repositories and create the branches and merge requests containing the dependency updates.

Then create a repository for the configuration and renovate will use that repo’s CI pipelines. Paste your Gitlab token under CI / CD > Variables as a new variable and give it the name RENOVATE_TOKEN. Set it to protected and masked to hide the token from the CI logs and to only use it for Pipelines starting on protected branches (your master branch is protected by default).

You'll also need a Github access token with the repos scope for renovate to read sources and changelogs of dependencies hosted on Github. It’s not important what Github account is used as it's just needed because Github's rate-limiting would block your bot making unauthenticated requests. Paste it as an other variable with the name GITHUB_COM_TOKEN.

To configure Renovate we need to add three files to our repository:

config.js for configuring renovate:

module.exports = {
  platform: ‘gitlab’,
  endpoint: ‘https://gitlab.com/api/v4/',
  assignees: [‘your-username’],
  baseBranches: [‘master’],
  labels: ['renovate', 'dependencies', 'automated'],
  onboarding: true,
  onboardingConfig: {
    extends: ['config:base'],
  },
};

repositories.txt for repositories we want to check:

openpatch/ui-core
openpatch/template

.gitlab-ci.yml to run renovate:

default:
  image: docker:19
  services:
    - docker:19-dind

# Because our GitLab runner doesn’t have TLS certs mounted and runs on K8s
variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ‘'

renovate:
  stage: build
  script:
    - docker run -e RENOVATE_TOKEN="$RENOVATE_TOKEN" -e GITHUB_COM_TOKEN="$GITHUB_COM_TOKEN" -v $PWD/config.js:/usr/src/app/config.js renovate/renovate:13 $(cat repositories.txt | xargs)
  only:
    - master

Now everything is finished and when you run the pipeline renovate will check the repositories in repositories.txt and create merge request if a dependency needs to be updated.

The first merge request to repository is Configure Renovate which helps you to understand and configure settings before regular Merge Requests begin.

As a last step create a Pipeline Schedule to run the pipeline every x hours or x day or whatever you like. You can do this in the bot's config project / repository under CI / CD > Schedules by creating a new schedule and chosing the frequency to run your bot.

You can also reduce noise by Package Grouping and Automerging. Here’s an example of grouping eslint themed packages and automerging them if tests pass.

Project’s renovate.json:

{
    "packageRules": [
        {
          "packagePatterns": [ "eslint" ],
          "groupName": "eslint",
          "automerge": true,
          "automergeType": "branch"
        }
      ]
}

Summary

Congratulations! You’ve now automated the dependency updating with GitLab CI. Just keep waiting for the merge requests and see if your test suites are successful.  If you are really trusting your test suite, you can even let renovate auto-merge the request, if the pipeline succeeds.

Automate versioning and changelog with release-it on GitLab CI/CD

It’s said that you should automate all the things and one of the things could be versioning your software. Incrementing the version number in your e.g. package.json is easy but it’s easier when you bundle it to your continuous integration and continuous deployment process. There are different tools you can use to achieve your needs and in this article we are using release-it. Other options are for example standard-version and semantic-release.

🚀 Automate versioning and package publishing

Using release-it with CI/CD pipeline

Release It is a generic CLI tool to automate versioning and package publishing related tasks. It’s installation requires npm but package.json is not needed. With it you can i.a. bump version (in e.g. package.json), create git commit, tag and push, create release at GitHub or GitLab, generate changelog and make a release from any CI/CD environment.

Here is an example setup how to use release-it on Node.js project with Gitlab CI/CD.

Install and configure release-it

Install release-it with npm init release-it which ask you questions or manually with npm install --save-dev release-it .

For example the package.json can look the following where commit message has been customized to have "v" before version number and npm publish is disabled (although private: true should be enough for that). You could add [skip ci] to "commitMessage" for i.a. GitLab CI/CD to skip running pipeline on release commit or use Git Push option ci.skip.

package.json
{
  "name": “example-frontend",
  "version": "0.1.2",
  "private": true,
  "scripts": {
    ...
    "release": "release-it"
  },
  "dependencies": {
    …
  },
  "devDependencies": {
    ...
    "release-it": "^12.4.3”
},
"release-it": {
    "git": {
      "tagName": "v${version}",
      "requireCleanWorkingDir": false,
      "requireUpstream": false,
      "commitMessage": "Release v%s"
    },
    "npm": {
      "publish": false
    }
  }
}

Now you can run npm run release from the command line:

npm run release
npm run release -- patch --ci

In the latter command things are run without prompts (--ci) and patch increases the 0.0.x number.

Using release-it with GitLab CI/CD

Now it’s time to combine release-it with GitLab CI/CD. Adding release-it stage is quite straigthforward but you need to do couple of things. First in order to push the release commit and tag back to the remote, we need the CI/CD environment to be authenticated with the original host and we use SSH and public key for that. You could also use private token with HTTPS.

  1. Create SSH keys as we are using the Docker executorssh-keygen -t ed25519.
  2. Create a new SSH_PRIVATE_KEY variable in "project > repository > CI / CD Settings" where and paste the content of your private key that you created to the Value field.
  3. In your "project > repository > Repository" add new deploy key where Title is something describing and Key is the content of your public key that you created.
  4. Tap "Write access allowed".

Now you’re ready for git activity for your repository in CI/CD pipeline. Your .gitlab-ci.yaml release stage could look following.

Update 2021-07-30: release-it has updated documentation for running it on CI and doing releases which differs a bit from the below.

image: docker:19.03.1

stages:
  - release

Release:
  stage: release
  image: node:12-alpine
  only:
    - master
  before_script:
    - apk add --update openssh-client git
    # Using Deploy keys and ssh for pushing to git
    # Run ssh-agent (inside the build environment)
    - eval $(ssh-agent -s)
    # Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    # Create the SSH directory and give it the right permissions
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    # Don't verify Host key
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - git config user.email "gitlab-runner@your-domain.com"
    - git config user.name "Gitlab Runner"
  script:
    # See https://gist.github.com/serdroid/7bd7e171681aa17109e3f350abe97817
    # Set remote push URL
    # We need to extract the ssh/git URL as the runner uses a tokenized URL
    # Replace start of the string up to '@'  with git@' and append a ':' before first '/'
    - export CI_PUSH_REPO=$(echo "$CI_REPOSITORY_URL" | sed -e "s|.*@\(.*\)|git@\1|" -e "s|/|:/|" )
    - git remote set-url --push origin "ssh://${CI_PUSH_REPO}"
    # runner runs on a detached HEAD, checkout current branch for editing
    - git checkout $CI_COMMIT_REF_NAME
    - git pull origin $CI_COMMIT_REF_NAME
    # Run release-it to bump version and tag
    - npm ci
    - npm run release -- patch --ci --verbose

We are running release-it here with patch increment. If you want to skip CI pipeline on release-it commit you can either use the ci.skip Git Push option package.json git.pushArgs which tells GitLab CI/CD to not create a CI pipeline for the latest push. This way we don't need to add [skip ci] to commit message.

And now you're ready to run the pipeline with release stage and enjoy of automated patch updates to your application's version number. And you also get GitLab Releases if you want.

Setting up the script step was not so clear but fortunately people in the Internet had done it earlier and Google found a working gist and comment on GitLab issue. Interacting with git in the GitLab CI/CD could be easier and there are some feature requests for that like allowing runners to push via their CI token.

Customizing when pipelines are run

There are some more options for GitLab CI/CD pipelines if you want to run pipelines after you've tagged your version. Here's snippet of running "release" stage on commits to master branch and skipping it if commit message is for release.

Release:
  stage: release
  image: node:12-alpine
  only:
    refs:
      - master
    variables:
      # Run only on master and commit message doesn't start with "Release v"
      - $CI_COMMIT_MESSAGE !~ /^Release v.*/
  before_script:
    ...
  script:
    ...

Now we can build a new container for the deployment of our application after it has been tagged and version bumped. Also we are reading the package.json version for tagging the image.

variables:
  PACKAGE_VERSION: $(cat package.json | grep version | head -1 | awk -F= "{ print $2 }" | sed 's/[version:,\",]//g' | tr -d '[[:space:]]')

Build dev:
  before_script:
    - export VERSION=`eval $PACKAGE_VERSION`
  stage: build
  script:
    - >
      docker build
      --pull
      --tag your-docker-image:latest
      --tag your-docker-image:$VERSION.dev
      .
    - docker push your-docker-image:latest
    - docker push your-docker-image:$VERSION.dev
  only:
    refs:
      - master
    variables:
      # Run only on master and commit message starts with "Release v"
      - $CI_COMMIT_MESSAGE =~ /^Release v.*/

Using release-it on detached HEAD

In the previous example we made a checkout to current branch for editing as the runner runs on detached HEAD. You can use the detached HEAD as shown below but the downside is that you can't create GitLab Releases from the pipeline as there's no git tag for the release and thus it fails to "ERROR Response code 422 (Unprocessable Entity)" (see also issue #533).

Then the .gitlab-ci.yml is following:

...
script:
    - export CI_PUSH_REPO=$(echo "$CI_REPOSITORY_URL" | sed -e "s|.*@\(.*\)|git@\1|" -e "s|/|:/|" )
    - git remote set-url --push origin "ssh://${CI_PUSH_REPO}"
    # gitlab-runner runs on a detached HEAD, create a temporary local branch for editing
    - git checkout -B ci_processing
    # Run release-it to bump version and tag
    - npm ci
    - DEBUG=release-it:* npm run release -- patch --ci --verbose --no-git.push
    # Push changes to originating branch
    # Always return true so that the build does not fail if there are no changes
    - git push --follow-tags origin ci_processing:${CI_COMMIT_REF_NAME} || true

Using release-it and doing GitLab releases

Using release-it to tag and release versions on GitLab CI is quite straightforward when using the detached HEAD approach but if you want to do GitLab release for the version you need to set the release step separately and have a changelog generation set up in package.json. Like below:

package.json

"release-it": {
    ...
    "hooks": {
      "before:init": "git fetch --prune --prune-tags origin",
      "after:bump": "./generate-release-notes.sh"
    },
    "gitlab": {
      "release": false,
      "releaseName": "v${version}",
      "releaseNotes": "./generate-release-notes.sh ${latestVersion} v${version}"
    }
    ...
  }

generate-release-notes.sh

#!/usr/bin/env sh
node_modules/conventional-changelog-cli/cli.js -p angular -r 1

.gitlab-ci.yml

...
script:
    - git checkout -B ci_processing
    # Install packages
    - npm ci --legacy-peer-deps
    # Run release-it to bump version and tag
    - DEBUG=release-it:* npm run release -- --ci --preRelease --verbose --no-git.push --disable-metrics
    # Push changes to originating branch
    # Always return true so that the build does not fail if there are no changes
    - git push --follow-tags origin ci_processing:${CI_COMMIT_REF_NAME} || true
    # Do GitLab Release
    - npm run release -- --ci --no-increment --no-git --gitlab.release