Automate your dependency management using update tool

Software often consists of not just your own code but also is dependent of third party libraries and other software which has their own update cycle and new versions are released now and then with fixes to vulnerabilities and with new features. Now the question is what is your dependency management strategy and how do you automate it?

Fortunately automated dependency updates for multiple languages is a solved problem as there are several update tools to help you: Renovate, Dependabot (GitHub), Greenkeeper ($), Depfu ($) and Dependencies.io ($) to name some alternatives. In this blog post I will concentrate on using Renovate and integrate it with GitLab CI.

Renovate your dependencies

Renovate is open source tool which works with most git hosting platforms (public or self-hosted) and it's possible to host Renovate Bot yourself. It’s installable via npm/yarn or Docker Hub.

In short, the idea and workflow of dependency update tools are following:

  1. Checks for updates: pulls down your dependency files and looks for any outdated or insecure requirements.
  2. Opens pull requests: If any of your dependencies are out-of-date, tool opens individual pull requests to update each one.
  3. Review and merge: You check that your tests pass, scan the included changelog and release notes, then hit merge with confidence.

Now you just run the depedency update tool on regular basis on your continuous integration, watch how the pull requests fly and you get to keep your dependencies secure and up-to-date.

The manual chore of checking for updates, looking for changelogs, making changes, running tests, writing pull requests and more is now moved to reviewing pull requests with better confidence of what has changed.

Self-hosted in GitLab CI

Renovate Bot is a node.js application so you’ve couple of alternative ways to run it on your CI/CD environment. You can use a node docker image which installs and runs renovate, or you can use Renovate Bot's own Docker image as I chose to do. We are using docker-in-docker approach of running the Renovate docker container. That means you can start Docker containers from within an other Docker container.

First create an  account for the bot on the Gitlab instance (the best choice) or use your own account. Then generate a personal access token with the api scope for renovate to access the repositories and create the branches and merge requests containing the dependency updates.

Then create a repository for the configuration and renovate will use that repo’s CI pipelines. Paste your Gitlab token under CI / CD > Variables as a new variable and give it the name RENOVATE_TOKEN. Set it to protected and masked to hide the token from the CI logs and to only use it for Pipelines starting on protected branches (your master branch is protected by default).

You'll also need a Github access token with the repos scope for renovate to read sources and changelogs of dependencies hosted on Github. It’s not important what Github account is used as it's just needed because Github's rate-limiting would block your bot making unauthenticated requests. Paste it as an other variable with the name GITHUB_COM_TOKEN.

To configure Renovate we need to add three files to our repository:

config.js for configuring renovate:

module.exports = {
  platform: ‘gitlab’,
  endpoint: ‘https://gitlab.com/api/v4/',
  assignees: [‘your-username’],
  baseBranches: [‘master’],
  labels: ['renovate', 'dependencies', 'automated'],
  onboarding: true,
  onboardingConfig: {
    extends: ['config:base'],
  },
};

repositories.txt for repositories we want to check:

openpatch/ui-core
openpatch/template

.gitlab-ci.yml to run renovate:

default:
  image: docker:19
  services:
    - docker:19-dind

# Because our GitLab runner doesn’t have TLS certs mounted and runs on K8s
variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ‘'

renovate:
  stage: build
  script:
    - docker run -e RENOVATE_TOKEN="$RENOVATE_TOKEN" -e GITHUB_COM_TOKEN="$GITHUB_COM_TOKEN" -v $PWD/config.js:/usr/src/app/config.js renovate/renovate:13 $(cat repositories.txt | xargs)
  only:
    - master

Now everything is finished and when you run the pipeline renovate will check the repositories in repositories.txt and create merge request if a dependency needs to be updated.

The first merge request to repository is Configure Renovate which helps you to understand and configure settings before regular Merge Requests begin.

As a last step create a Pipeline Schedule to run the pipeline every x hours or x day or whatever you like. You can do this in the bot's config project / repository under CI / CD > Schedules by creating a new schedule and chosing the frequency to run your bot.

You can also reduce noise by Package Grouping and Automerging. Here’s an example of grouping eslint themed packages and automerging them if tests pass.

Project’s renovate.json:

{
    "packageRules": [
        {
          "packagePatterns": [ "eslint" ],
          "groupName": "eslint",
          "automerge": true,
          "automergeType": "branch"
        }
      ]
}

Summary

Congratulations! You’ve now automated the dependency updating with GitLab CI. Just keep waiting for the merge requests and see if your test suites are successful.  If you are really trusting your test suite, you can even let renovate auto-merge the request, if the pipeline succeeds.

Short notes on tech 11/2021

Week 11 of 2021

Work life

Handbook for Remote employees
Read how Remote works as a company and take notes.

How to Deal with Difficult People on Software Projects

Your Thinking Rate Is Fixed
"You can’t force yourself to think faster. If you try, you’re likely to end up making much worse decisions. Here’s how to improve the actual quality of your decisions instead of chasing hacks to speed them up." (from Hackernewsletter)

Generalists vs specialists - who has a greater chance of success?
(from Hackernewsletter)

Learning

The Front-End Developer Learning Roadmap

AWS Cloud Development Kit Workshop
Learn CDK.

Software development

Best practices for REST API design
"In this article, we'll look at how to design REST APIs to be easy to understand for anyone consuming them, future-proof, and secure and fast since they serve data to clients that may be confidential." (from Hackernewsletter)

You really should use dev containers
tl;dr; "use development containers, indicating the act of running and compiling code inside a container, not on the host machine OS."

The case for and against Amazon Cognito

Information Security

How often should I rotate my ssh keys?
"More often than never! As often as you can. And make sure you retire old keys when you’re done with them." (from Hackernewsletter)

Tools of the trade

Helppo
"Instant admin UI for your database" Supports PostgreSQL and MySQL.

TSDX
Zero-config CLI for TypeScript package development

Colorful VSCode titlebars for better productivity
tl;dr;
"workbench.colorCustomizations": {
"titleBar.activeBackground": "#553955" // change this color!
}

Short notes on tech 10/2021

Week 10/2021

iOS development

Analyse, Design & Code the ClubHouse app with SwiftUI
This video from Franck Ndame on recreating the Clubhouse UI from scratch is incredibly well-produced and enjoyable to watch. Follow it up by checking out the Figma design and source code. (from iOS Dev Weekly)

Cloud

A Quick Look at GKE Autopilot (in 15 minutes)
"I was curious to take a look at GKE Autopilot, so if you don't have time to play with it, I did it for you. (from Cloud Security Reading List)"

Software development

Ethical Design Guide
Resources for creating ethical products that don't cause harm. (from Weekend Reading List)

Why is it so hard to see code from 5 minutes ago?
"Do you undo recent changes to remember what the code looked like a few minutes ago? Why can’t IDEs do that for us?" Waiting for "Yestercode" to be a VS Code extension. (from Weekend Reading List)

Modules, monoliths, and microservices
(from Hacker News comments)

Worklife

Things your manager might not know
If you never managed people before, it's not obvious how to manage your manager. Julia explains how to do that. (from Weekend Reading List)

How to be more productive without forcing yourself

Something different

Internet Archive Infrastructure
tl;dr; Lots of hardware.

Short notes on tech 9/2021

Week 9/2021

Tools of the trade

Uizard
"The design tool for everybody. Scan a paper scribble and you get a wireframe. Upload images and it will extract color and style to generate a matching theme." (from Weekend Reading)

Camo
Camo makes you look great on Zoom by using your iPhone as a webcam. You can control lighting, cropping, focus, Bokeh effect, stream 1080p with no stutter (needs cable), use front-facing, telephoto or wide lens. (from Weekend Reading)

Free for developers
List of free and free tier resources.

Cloud

Kubernetes README
"A collection of useful resources to read to learn more about Kubernetes." (from Cloud Security List)

Work

Unpacking Interview Questions
"Series sharing some of the questions the writer uses when he interviews for technical roles. He’ll unpack the question, when to ask it, and how to evaluate answers."

Something different

Scientists break through the wall of sleep to the untapped world of dreams
"Researchers at Paller’s lab at Northwestern University in Illinois, along with researchers in France, Germany and the Netherlands, have independently demonstrated two-way communication with people as they are lucidly dreaming during REM (rapid eye movement) sleep." (from Weekend Reading)

Monthly notes 56

Issue 56, 26.2.2021

Work life

Researchers identify four causes of "Zoom fatigue" and their simple fixes
"Those video calls are likely tiring you out." tl;dr;

  • Excessive amounts of close-up eye contact is highly intense.
  • Seeing yourself during video chats constantly in real-time is fatiguing.
  • Video chats dramatically reduce our usual mobility.
  • The cognitive load is much higher in video chats.

Maximizing Developer Effectiveness
"It’s all about tight feedback loops." (from Weekend Reading)

Information security

OWASP Top 10 for Web
"Inspired by real-world vulnerabilities and case studies, we have created a series of interactive application security training modules to help developers understand, identify and mitigate security vulnerabilities in their applications."

3 Ways to Mitigate Risk When Using Private Package Feeds
"Microsoft whitepaper on best practices to follow to reduce risks against substitution attacks." (from Cloud Security List)

How to use Docker Security Scan Locally
"Docker and Snyk recently entered into a partnership to provide container vulnerability scanning to official images on Docker Hub. Additionally, Docker has integrated scanning directly into Docker for Desktop clients." (from Cloud Security List)

Cloud

Introducing GKE Autopilot: a revolution in managed Kubernetes
"Autopilot is a new mode of operation in Google Kubernetes Engine (GKE). Autopilot clusters are pre-configured with an optimized cluster configuration that is ready for production workloads. This streamlined configuration follows GKE best practices and recommendations for cluster and workload setup and security." You can achieve "the same" by manually ticking the right options.

AWS Account Setup Guide
A guide for configuring new AWS accounts with an emphasis on security, including customizable templates. (from Cloud Security List)

Microservices

A Practical Guide to Writing Secure Dockerfiles
How to write secure Dockerfiles, and how to automate security checks as codified policies and validate them against the Dockerfiles to identify potential security risks before deploying them into production. (from Cloud Security List)

Learning

Tackling TypeScript: Upgrading from JavaScript
"For JavaScript developers looking to learn TypeScript." (from Weekend Reading)

How they SRE
A curated collection of publicly available resources on how technology and tech-savvy organizations around the world practice Site Reliability Engineering (SRE) (from Hacker Newsletter)

Tools of the trade

skan
"sKan is a Kubernetes configuration files and resources scanner that enables developers and devops team members to check whether their work is compliant with security & ops best practices." (from Cloud Security List)

Something different

Calvin and Hobbes search engine
(from Hacker Newsletter, comments)

Short notes on tech 7/2021

Week 7/2021

Microservices

A Practical Guide to Writing Secure Dockerfiles
How to write secure Dockerfiles, and how to automate security checks as codified policies and validate them against the Dockerfiles to identify potential security risks before deploying them into production. (from Cloud Security List)

Tools of the trade

Dockle
Container Image Linter for Security, Helping build the Best-Practice Docker Image.

Cloud

AWS Account Setup Guide
A guide for configuring new AWS accounts with an emphasis on security, including customizable templates. (from Cloud Security List)

Software development

Visual Studio Code Extensions for better programming

Visual Studio Code has become "The Editor" for many in software development and it has many extensions which you can use to extend the functionality for your needs and customize it. Here’s a short list of the extensions I use for frontend (React, JavaScript, Node.js), backend (GraphQL, Python, Node.js, Java, PHP, Docker) and database (PostgreSQL, MongoDB) development.

General

editorconfig
Attempts to override user/workspace settings with settings found in .editorconfig files.

Visual Studio IntelliCode
Provides AI-assisted development features for Python, TypeScript/JavaScript and Java developers in Visual Studio Code, with insights based on understanding your code context combined with machine learning.

GitLens
Visualize code authorship at a glance via Git blame annotations and code lens, seamlessly navigate and explore Git repositories, gain valuable insights via powerful comparison commands, and so much more.
Git Blame
See Git Blame information in the status bar for the currently selected line.

Local History
Plugin for maintaining local history of files.

Language and technology specific

ESlint
Integrates ESLint into VS Code.

Prettier
Opinionated code formatter which enforces a consistent style by parsing your code and re-printing it with its own rules that take the maximum line length into account, wrapping code when necessary.

Python
Linting, Debugging (multi-threaded, remote), Intellisense, Jupyter Notebooks, code formatting, refactoring, unit tests, and more.

PHP Intelephense
PHP code intelligence for Visual Studio Code is a high performance PHP language server packed full of essential features for productive PHP development.

Java Extension Pack
Popular extensions for Java development and more.

Docker
Makes it easy to build, manage, and deploy containerized applications from Visual Studio Code. It also provides one-click debugging of Node.js, Python, and .NET Core inside a container.

Markdown All in One
All you need to write Markdown (keyboard shortcuts, table of contents, auto preview and more)
Markdownlint
Includes a library of rules to encourage standards and consistency for Markdown files.
Markdown Preview Enhanced
Provides you with many useful functionalities such as automatic scroll sync, math typesetting, mermaid, PlantUML, pandoc, PDF export, code chunk, presentation writer, etc.

Prettify JSON
Prettify ugly JSON inside VSCode.

PlantUML
Rich PlantUML support for Visual Studio Code.

HashiCorp Terraform
Syntax highlighting and autocompletion for Terraform

Database

PostgreSQL
Query tool for PostgreSQL databases. While there is a database explorer it is NOT meant for creating/dropping databases or tables. The explorer is a visual aid for helping to craft your queries.

MongoDB
Makes it easy to work with MongoDB.

GraphQL
Adds syntax highlighting, validation, and language features like go to definition, hover information and autocompletion for graphql projects. This extension also works with queries annotated with gql tag.
GraphQL for VSCode
VSCode extension for GraphQL schema authoring & consumption.
Apollo GraphQL for VS Code
Rich editor support for GraphQL client and server development that seamlessly integrates with the Apollo platform.

Javascript

Babel
JavaScript syntax highlighting for ES201x, React JSX, Flow and GraphQL.

Jest
Use Facebook's Jest with pleasure.

npm
Supports running npm scripts defined in the package.json file and validating the installed modules against the dependencies defined in the package.json.

User Interface specific

indent-rainbow
Simple extension to make indentation more readable

Rainbow Brackets
Rainbow colors for the round brackets, the square brackets and the squiggly brackets.

vscode-icons
Icons for filetypes in file browser.

Other tips

VScode Show Full Path in Title Bar
With Code open, hit: Command+ , “window.title”: “{activeEditorLong}activeEditorLong{separator}${rootName}”

Slow integrated terminal in macOS
codesign --remove-signature /Applications/Visual\ Studio\ Code.app/Contents/Frameworks/Code\ Helper\ (Renderer).app

Short notes on tech 5/2021

Week 5/2021

Worklife

Why Working from Home Will Stick
Or will it? Hacker News comments provide a good pointers why it won't stick for the broader society.

Software development

Maximizing Developer Effectiveness
"It’s all about tight feedback loops." (from Weekend Reading)

Google Engineering Practices Documentation
"Google has many generalized engineering practices that cover all languages and all projects. These documents represent our collective experience of various best practices that we have developed over time." Unfortunately it currently contains only "Google's Code Review Guidelines".

Awesome Software and Architectural Design Patterns
"A curated list of software and architecture related design patterns."

Tools of the trade

deep-email-validator
"Library that handles all the email validation strategies: regex, common typos, disposable email blacklists, MX record lookup, and SMTP to check the inbox exists." (from Weekend Reading)

Mock Service Worker
Seamless API mocking library for browser and Node. (from Weekend Reading)

Running static analysis tools for PHP

We all write bug free code but analyzing your code is still important part of software development if for some reason there could've been some mishap with typing. Here's a short introduction how to run static analysis for PHP code.

Static analysis tools for PHP

The curated list of static analysis tools for PHP show you many options for doing analysis. Too much you say? Yes but fortunately you can start with some tools and continue with the specific needs you have.

You can run different analysis tools by installing them with composer or you can use the Toolbox which helps to discover and install tools. You can use it as a Docker container.

First, fetch the docker image with static analysis tools for PHP:

$ docker pull jakzal/phpqa:<your php version>
e.g.
$ docker pull jakzal/phpqa:php7.4-alpine

PHPMD: PHP Mess Detector

One of the tools provided in the image is PHPMD which aims to be a PHP equivalent of the well known Java tool PMD. PHPMD can be seen as an user friendly and easy to configure frontend for the raw metrics measured by PHP Depend.

It looks for several potential problems within that source like:

  • Possible bugs
  • Suboptimal code
  • Overcomplicated expressions
  • Unused parameters, methods, properties

You can install the phpmd with composer: composer require phpmd/phpmd. Then run it with e.g. ./vendor/bin/phpmd src html unusedcode --reportfile phpmd.html

Or run the command below which runs phpmd in a docker container and mounts the current working directory as a /project.

docker run -it --rm -v $(pwd):/project -w /project jakzal/phpqa:php7.4-alpine \
    phpmd src html cleancode,codesize,controversial,design,naming,unusedcode --reportfile phpmd.html

You can also make your custom rules to reduce false positives: phpmd.test.xml

<?xml version="1.0"?>
<ruleset name="VV PHPMD rule set"
         xmlns="http://pmd.sf.net/ruleset/1.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://pmd.sf.net/ruleset/1.0.0
                     http://pmd.sf.net/ruleset_xml_schema.xsd"
         xsi:noNamespaceSchemaLocation="
                     http://pmd.sf.net/ruleset_xml_schema.xsd">
    <description>
        Custom rule set that checks my code.
    </description>
<rule ref="rulesets/codesize.xml">
    <exclude name="CyclomaticComplexity"/>
    <exclude name="ExcessiveMethodLength"/>
    <exclude name="NPathComplexity"/>
    <exclude name="TooManyMethods"/>
    <exclude name="ExcessiveClassComplexity"/>
    <exclude name="ExcessivePublicCount"/>
    <exclude name="TooManyPublicMethods"/>
    <exclude name="TooManyFields"/>
</rule>
<rule ref="rulesets/codesize.xml/TooManyFields">
    <properties>
        <property name="maxfields" value="21"/>
    </properties>
</rule>
<rule ref="rulesets/cleancode.xml">
    <exclude name="StaticAccess"/>
    <exclude name="ElseExpression"/>
    <exclude name="MissingImport" />
</rule>
<rule ref="rulesets/controversial.xml">
    <exclude name="CamelCaseParameterName" />
    <exclude name="CamelCaseVariableName" />
    <exclude name="Superglobals" />
</rule>
<rule ref="rulesets/design.xml">
    <exclude name="CouplingBetweenObjects" />
    <exclude name="NumberOfChildren" />
</rule>
<rule ref="rulesets/design.xml/NumberOfChildren">
    <properties>
        <property name="minimum" value="20"/>
    </properties>
</rule>
<rule ref="rulesets/naming.xml">
    <exclude name="ShortVariable"/>
    <exclude name="LongVariable"/>
</rule>
<rule ref="rulesets/unusedcode.xml">
    <exclude name="UnusedFormalParameter"/>
</rule>
<rule ref="rulesets/codesize.xml/ExcessiveClassLength">
    <properties>
        <property name="minimum" value="1500"/>
    </properties>
</rule>
</ruleset>

Then run your analysis with:

docker run -it --rm -v $(pwd):/project -w /project jakzal/phpqa:php7.4-alpine phpmd src html phpmd.test.xml unusedcode --reportfile phpmd.html

You get a list of found issues formatted to a HTML file

PHPMD Report

PHPStan - PHP Static Analysis Tool

"PHPstan focuses on finding errors in your code without actually running it. It catches whole classes of bugs even before you write tests for the code. It moves PHP closer to compiled languages in the sense that the correctness of each line of the code can be checked before you run the actual line."

Installing with composer: composer require --dev phpstan/phpstan

Or run on Docker container:

docker run -it --rm -v $(pwd):/project -w /project jakzal/phpqa:php7.4-alpine phpstan analyse --level 1 src

By default you will get a report to console formatted to a table and grouped errors by file, colorized. For human consumption.

PHPStan report

By default PHPStan is performing only the most basic checks and you can pass a higher rule level through the --level option (0 is the loosest and 8 is the strictest) to analyse code more thoroughly. Start with 0 and increase the level as you go fixing possible issues.

PHPStan found some more issues which PHPMD didn't find but the output of the PHPStan could be better. There's a Web UI for browsing found errors and you can click and open your editor of choice on the offending line but you've to pay for it. PHPStan Pro costs 7 EUR for individuals monthly, 70 EUR for teams.

VS Code extension for PHP

If you're using Visual Studio Code for PHP programming there are some extensions to help you.

PHP Intelephense
PHP code intelligence for Visual Studio Code provides better intellisense then VS Code builtin and also does some signature checking etc. The extension has also premium version for some additional features.

Short notes on tech 4/2021

It's already week 4 of 2021 😱 This week the short notes is a bit bigger edition recapping the three first weeks of 2021.

Week 4, 2021

Web-End

Progressive Web Apps in 2021
So far it's been a slow start for PWA. The thing is, they're pretty hard to deploy or retrofit into existing websites. But I expect new stacks will ship with PWA support and at some point they'll become the default choice, the tide will turn (from Weekend Reading)

User stories that should never exist
Twitter account with 😂 user stories.

Choosing a stack of low-code solutions by Jason Lengstorf
"This is a really good thread about choosing a stack of low-code solutions" (from Weekend Reading)

Exploring Rootless Docker
"With the release of Docker 20.10, the rootless containers feature has left experimental status. This post explores setup and usability of rootless Docker." (from Cloud Security Reading List)

Programming

naming-cheatsheet
"A cheetsheet for naming variable and function names. The styling here is JavaScript, but you can adapt these rules to any other language." (from Weekend Reading)

Worklife

No Meetings, No Deadlines, No Full-Time Employees
"What if work was like open source?"

HR is not your friend, and other things I think you should know
"I think people go into HR with the ideal of helping, and in the beginning it's all fun and office parties. By the time they realize that HR is “The Department for Mitigating Legal Risk”, it's too late." (from Weekend Reading) (Hacker News comments)

Cloud

How to Enable Logging on Every AWS Service in Existence (Circa 2021)
"Cloud security best practices, as well as most compliance programs, require that logging be enabled for all in-scope services. However, that simple requirement - enable logging - comes with many follow-up questions. Is CloudTrail enough? How do I turn on logging for all these services? Aren't logs collected by default?" (from Cloud Security Reading List)

What You Need to Know About AWS Security Monitoring, Logging, and Alerting
"Post laying out the different AWS security monitoring and logging sources, how to collect logs from them, and how to select the most appropriate collection technique." (from Cloud Security Reading List)

AWS announces forks of Elasticsearch and Kibana
"Elastic will change their software licensing strategy from the Apache License, Version 2.0 (ALv2) to the Elastic License (which limits how it can be used) or the Server Side Public License (which has requirements that make it unacceptable to many in the open source community). This means that Elasticsearch and Kibana will no longer be open source software. In order to ensure open source versions of both packages remain available and well supported we are announcing that AWS will step up to create and maintain a ALv2-licensed fork of open source Elasticsearch and Kibana."

Tools of the trade

Hush
"Noiseless browsing". This is a tiny app that blocks nags to accept cookies and privacy invasive tracking. Safari only, macOS/iOS, open source, so maybe you can port it to Android/Chrome. Free. (from Weekend Reading)

Scott Hanselman's 2021 Ultimate Developer and Power Users Tool List for Windows

Signal, Telegram, WhatsApp and other apps, what’s the difference?
Ola Bini's Twitter thread of giving an overview about perspective on the security of different applications.

Altair
"GraphQL client app with tons of features."

cloudfour/lighthouse-parade
"Command line tool that crawls a domain and gathers lighthouse performance data for every page." (from Weekend Reading)

OpenScan – open-source document scanner app

Upptime – GitHub-powered uptime monitor and status page

Something different

I logged my activities at 15-minute intervals for the whole year
"Where does the time go?" Log it and find out. (Hacker News comments)