Two days of React Finland 2018: Day one topics of React

React Finland 2018 conference was held last week and I had the opportunity to attend it and listen what’s hot in the React world. The conference started with workshops and after that there was two days of talks of React, React Native, React VR and all things that go with developing web applications with them. The two conference days were packed with great talks and new information. Here’s the first part of my notes from the talks which I posted to Twitter. Read also the second part with more of React Native.

React Finland 2018: Day 1

React Finland combined the Finnish React community with international flavor from Jani Eväkallio to Ken Wheeler and other leading talents of the community. The event was the first of its kind in Finland and consisted of a workshop day and two days of talks around the topic. It was nice that the event was single track so you didn’t need to choose between interesting talks.

At work I’ve been developing with React couple of years and tried my hands with React Native so the topics were familiar. The conference provided crafty new knowledge to learn from and maybe even put to production. Overall the conference was great experience and everything went smoothly. Nice work from the React Finland conference team! And of course thanks to Gofore which sponsored the conference and got me a ticket.

I tweeted my notes from almost every presentation and here’s a recap of the talks. I heard that the videos from the conference will be available shortly.

The New Best Practices — Jani Eväkallio

First day’s keynote was by Jani Eväkallio who talked about “The New Best Practices”. As the talk description wrote “When React was first introduced, it was ridiculed for going against established web development best practices as we knew them. Five years later, React is the gold standard for how we create user interfaces. Along the way, we’ve discovered a new set of tools, design patterns and programming techniques.”

The new best practices were:

  • Build big things from small things
  • Write code for humans first: flow, Typescript, storybook
  • Stay close to the language:
    • helps i.a. linters
  • Always prefer simplicity
  • Don’t break things:
    • Facebook makes React API changes easy to upgrade, depreciation well in advance, migration, documentation. it’s a flow, not versions. Use codemod.
  • Keep an open mind

You ask “what new best practices”? Yep, that’s the thing. We don’t need new best practices as the same concepts like Model-View-Controller and separation of concerns are still valid. We should use best practices which have been proved good before as they also work nicely with React philosophy. Eväkallio also talked why React will be around for a long time. It’s because components and interoperable components are an innovation primitive.

The New Best Practices

Declarative state and side effects — Christian Alfoni

After the keynote it was time to get more practical and Christian Alfoni talked about how we can get help writing our business logic in a declarative manner and see what benefits it gives us. He talked about lessons learned refactoring Codesandbox.io from Redux to Cerebral and about Cerebral which provides a declarative state and side effects management for popular JavaScript frameworks. Talks slides are available on the Internet.

Alfoni also pointed to Turning the database inside out with Apache Samza. Also that Cerebral had time travel before Dan Abramov presented Live React in his talk Hot Reloading with Time Travel at react-europe 2015

Immer: Immutability made easy — Michel Weststrate

Immutable data structures are a good thing and Michel Weststrate showed Immer which is a tiny package that allows you to work with immutable data structures with unprecedented ease. Managing the state of React app is a huge deal with Redux and any help is welcome. “Immer doesn’t require learning new data structures or update APIs, but instead creates a temporarily shadow tree which can be modified using the standard JavaScript APIs. The shadow tree will be used to generate your next immutable state tree.”

The talk showed how to write your reducers in a much more readable way, with half the code and without requiring additional large libraries. The talk slides are available on the Internet.

Get Rich Quick With React Context — Patrick Hund

“Get Rich Quick With React Context” lightning talk by Patrick Hund didn’t tell how good job opportunities you have when doing React But how with React 16.3 the context API has been completely revamped and demonstrated a good use case: Putting ad placements on your web page to get rich quick! Other use cases are localizations. Check out the slides which will tell you how easy it is to use context now and how to migrate your old context code to the new API.

There’s always a better way to handle localization — Eemeli Aro

“There’s always a better way to handle localization” lightning talk by Eemeli Aro told about how localization is a ridiculously difficult problem in the general case, but in the specific you can get away with really simple solutions, especially if you understand the compromises you’re making.

I must have been dozing as all I got was there are also other options to store localizations than JSON like YAML and JavaScript property format especially when dealing with non-developers like translators. The talk was quite general and on abstract level and mentioned solutions to localization were react-intl, react-i18next and react-message-context.

Styled Components, SSR, and Theming — Kasia Jastrzębska

Web applications need to be styled and Kasia Jastrzębska talked about CSS-in-JS with styled-components by going through the new API, performance improvements, server side rendering with Next.js. She also showed the theming manager available with v2 of styled-components. Talk slides are available on the Internet.

Takeaways from this talk was that CSS in React app can be written as you always have or by using CSS-in-JS solutions. There are several benefits of using styled-components but I’m still thinking how styles get scattered all over components.

Universal React Apps Using Next.js — Sia Karamalegos

53% of mobile site visits are abandoned if pages take longer than 3 seconds to load.
– DoubleClick by Google, 2016

Every user’s hardware is different and processing speed can hinder user experience on client-side rendered React applications and so Sia Karamalegos talked how server-side rendering and code-splitting can drastically improve user experience. By minimizing the work that the client has to do. Performance and shipping your code matters. The talk showed how to easily build an universal React apps using the Next.js framework and walked through the concepts and code examples. Talk slides are available on the Internet.

There are lots of old (mobile) devices which especially benefit from Server Side Rendering. Next.js is a minimalistic framework for universal, server-rendered (or statically pre-rendered) React applications which enables faster page loads. Pages are server-rendered by default for initial load, you can enable prefetching future routes and there’s automatic code splitting. It’s also customizable so you can use own Babel and Webpack configurations and customize the server API with e.g. Express. And if you don’t want to use a server Next.js can also build static web apps that you can then host on Github pages or AWS S3.

Universal React apps using Next.js

State Management in React Apps with Apollo Client — Sara Vieira

Apollo Client was one of the most mentioned framework in the conference along with Reason ML and Sara Vieira gave energetic talk how to use it for state management in React Apps. If you haven’t come across Apollo Client it’s caching GraphQL client and helps you to manage data coming from the server. Virieira showed how to manage local state with apollo-link-state.

The talk was fast paced and I somewhat missed the why part but at least it’s easy to setup: yarn add apollo-boost graphql react-apollo. Have to see slides and demo later. Maybe the talk can be wrapped up to: “GQL all the things” and “I don’t like Redux” :D

State Management with Apollo

Detox: A year in. Building it, Testing with it — Rotem Mizrachi-Meidan

Detox testing framework for React Native talk by Rotem Mizrachi-Meidan was the other talk I dozed along. Mizrachi-Meidan talked what developing and using Detox in production has taught and how Detox works and what makes it deterministic. The talk showed how mobile apps could be tested. There’s a video of earlier talk on the Internet.

Detox

Make linting great again! — Andrey Okonetchnikov

One thing in software development which always gets developers to argue over stupid things is code formatting and linting. Andrey Okonetchnikov talked how “with a wrong workflow linting can be really a pain and will slow you and your team down but with a proper setup it can save you hours of manual work reformatting the code and reducing the code-review overhead.”

The talk was a quick introduction how ?? lint-staged a node.js library can improve developer experience. Small tool coupled with tools that analyze and improve the code like ESLint, Stylelint, Prettier and Jest can make a big difference.

Missed talks

There was also two talks I missed: “Understanding the differences is accepting” by Sven Sauleau and “Why I YAML” by Eemeli Aro. Sauleau showed “interesting” twists of Javascript language.

Read also the second part with more of React Native.

Monthly notes 29

This month’s notes are about front-end technologies: Sneak peek beyound React 16 and videos from Vue and Angular conferences. Also CSS Blocks + OptiCSS is great and for us in EU it’s nice that Fargate is finally available in Ireland. Check also list of important podcasts for software engineers.

Issue 29, 29.4.2018

Security

Computer security principles
One should keep in mind that there’s no such thing as perfect security. To put it another way, a 100% hack-safe systems do not exist. It’s all about the resources attacker(s) have, whether it is money, brain power, or equipment. Security standards and best practices changes quickly and therefore a system built five years ago is not inevitably conforming to current standards. So let’s look at some proactive measures that can be done to harden a system or code.

Cloud

AWS DevDay Munich 2018: Deep Dive into AWS Fargate (video)
“The session covers the state of containerized application development and deployment trends, new container capabilities on AWS that are available now, options for running containerized applications on AWS, and how AWS customers successfully run container workloads in production.” And for us in the EU Fargate is now available in Ireland.

Deploying FARGATE services using CloudFormation
“TL;DR — Deploying Fargate services is not as straight forward as you may think, especially if you’re used to the current EC2 configuration and are now trying to migrate running services. I had to go through a couple of days and few dozens of CloudFormation deployment iterations to figure out my missing / wrong settings before I made it through my first running Fargate container.”

CloudFormation Templates for AWS Fargate deployments
Sample CloudFormation templates for how to run Docker containers in AWS Fargate with various networking configurations.

Front-End

CSS at Scale: LinkedIn’s New Open Source Projects Take on Stylesheet Performance
“TL;DR: CSS Blocks + OptiCSS = :fire: So you get to write component-scoped CSS but end up with globally scoped, browser-friendly and compressed CSS classes (think atomic CSS). CSS Blocks does its magic with statically analyzing your markup and updating it with the new classes. It runs the OptiCSS as well, so you get tree-shaking and dead-code elimination also. Not 100% of the terms here, but basically unused code gets wiped.”

Sneak Peek: Beyond React 16
Intriguing ~30min talk with demos of what the future of React might look like showing off the new capabilities that async rendering unlocks for your components. Time Slicing lets you render and update large React component trees without blocking the user interactions. Suspense lets you render a component tree “in background” while components are fetching data, and display them only after the whole tree is ready. (from Twitter)

Talks from VueConf US 2018
The first ever VueConf US took place in New Orleans on March 26-28, 2018.

Talks from ng-conf 2018
Collection of all lectures that were presented during the conference. Each session includes a concise description and relevant slides.

To listen

Awesome list of Important Podcasts for software engineers
List of podcasts which are helpful for software engineers/programmers. Can’t pick any favorite from that list as I’m mostly listening to Finnish podcasts like ATK-hetki and Webbidevaus.fi.

Something different

“Never mad over what I can’t change. Never stressed over what I can’t control.”
– from Rhay1991

Designer Challenges Himself To Create Logos With Hidden Meanings For A Year, And Result Is Amazing
Clever.

Build Monitor with Raspberry Pi and Touch Screen

Information is a great tool in software development and it’s useful to have easy access to it. The more obvious you make your problems, the harder you make them to ignore. The more attention they get, the quicker they get solved. One thing developers like to monitor in software development is continuous integration status and metrics from running services. And what better way to achieve visibility and visualize to those metrics than building an information radiator.

I didn’t want to invent the wheel again so I got Raspberry Pi 3 Model B with accessories and 7″ touch screen to base my project. Using a Raspberry Pi as an information radiator isn’t a new idea and the Internet is full of examples of different adaptations with screens, lights, bells and whistles. For the start we just visualized our Jenkins builds and Grafana dashboard but later on we will propably do a custom dashboard.

Setting up the base

The information radiator is easy to get running as you only need a computer which preferably runs Linux. You can use an old laptop and attach it to external screen or if you’re like me and want to tinker you can get e.g. Raspberry Pi 3 and couple it with small external screen for portability. Nice and low cost solution which gets you some hacker value. I got the Rpi from our local hardware store and unfortunately the Model B+ was just released on the same day. The extra 15% power, 5 GHz Wifi and less heat and throttling would’ve been nice.

Raspberry Pi 3 Model B and accessories

I got the Raspberry Pi starting package with the official case, power supply, HDMI cable and a MicroSD card with preloaded NOOBS. So I just needed to connect the cables, put SD Card in and click to install Raspbian. Other interesting operating systems would’ve been Fedberry which is Fedora ‘Minimal, XFCE and LXQt’ Remixes.

For the screen I used 7″ IPS 5 point touch screen for Raspberry Pi with 1024×600 resolution and HDMI from joy-it codename RB-LCD-7-2. Initially I thought I could install the whole system with this display but as it turned out Rpi doesn’t understand it out of the box. It just showed some white noise and interference . Luckily some one had already solved this and I got the right config after I had installed Raspbian with real monitor.

Joy-it touch screen with default settings

Edit your /boot/config.txt:

# uncomment to force a specific HDMI mode (this will force VGA)
hdmi_group=2
hdmi_mode=87
 
# Add line:
hdmi_cvt=1024 600 60 3 0 0 0

And reboot your Raspberry Pi after those changes.

You should also run $ sudo raspi-config to setup for example WiFi country to allow channels 12 and 13 and your current Timezone.

I also updated Raspbian which bumps it to rpi-4.14.y linux tree:

$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get dist-upgrade
$ sudo rpi-update

To connect to Rpi with SSH enable it with raspi-config > Interfacing Options or just:

$ sudo systemctl enable ssh
$ sudo systemctl start ssh

For the note, by default the user pi has password raspberry. You should change it but if you want to remove the nagging of default password do the following:

$ sudo apt-get -y purge pprompt
$ sudo rm /etc/profile.d/sshpwd.sh
$ sudo rm /etc/xdg/lxsession/LXDE-pi/sshpwd.sh

Problems with WiFi connection

I set up the Raspberry Pi at our local office and at home and there were no problems with WiFi connection. But when I brought it to customer premises the WiFi connection was weak and practically couldn’t move a bit. My MacBook worked fine but it was connected to 5 GHz network which isn’t an option with my Rpi 3 Model B. The WiFi on Rpi 3 was using channel 11 on 802.11i with WPA2 as shown with iwlist wlan0 scan.

There is a thread on Raspberry Pi forum about Very poor wifi performance which suggest to set up WiFi internalisation correctly to allow channels 12 and 13. At one point the issue was that only channels 1-11 are available on the Rpi 3 but checking out the ‘next’ branch of firmware/kernel (sudo BRANCH=next rpi-update) apparently fixed channels 12/13. I was on kernel 4.9.80 so it wasn’t a problem for me. The other suggested problem is with Atheros chipset based router which doesn’t like Broadcom WiFi on Rpi 3.

For some disabling power management solves the connection issues. For RPi built-in Broadcom (Cypress) WiFi there’s no control for power management and it’s disabled by the kernel. In iw / iwlist / iwconfig you see bug with “Power Management:on”.

But nevertheless testing switching it off made my WiFi connection better but it’s strength didn’t of course change.

$ sudo iwconfig wlan0 power off

To make it permanent you can add something like this in your interfaces file:

$ sudo touch /etc/network/if-up.d/wlan0
$ sudo chmod +x /etc/network/if-up.d/wlan0
$ sudo echo -e '#!/bin/bash\niwconfig wlan0 power off' > /etc/network/if-up.d/wlan0

Accessing Raspberry Pi remotely

The information radiator is usually connected to a TV with no keyboard or mouse attached so accessing it remotely is useful. You can use x11vnc which allows you to VNC into a headless Pi with a VNC client like Apple Remote Desktop, RealVNC’s vncviewer or homebrew’s tiger-vnc.

$ sudo apt-get install ttf-mscorefonts-installer
$ sudo apt-get install x11vnc

To start x11vnc automatically create new or edit existing ~/.xsessionrc file:

$ cat ~/.xsessionrc
/usr/bin/x11vnc -noxrecord -noxdamage -forever -bg -rfbport 5900

Getting interesting things on the screen

To test our setup and quickly show some data I just added a Build Monitor view in Jenkins and other view with Dashboard view. I also configured the Rpi to automatically start Chromium browser in kiosk mode after reboots and directed it to Jenkins website so there would be no need for interactions to get things on the screen. To show several sources of data and get things running quickly without customized information radiator we used Revolver – Tabs Chromium extension to rotate between multiple browser tabs: one showed Jenkins Build Monitor other Grafana Dashboard and third Twitter feed.

To automatically start the chromium-browser after Raspbian desktop starts, edit the following lxsession file:

$ cp /etc/xdg/lxsession/LXDE-pi/autostart /home/pi/.config/lxsession/LXDE-pi/autostart
$ vim /home/pi/.config/lxsession/LXDE-pi/autostart
 
#@xscreensaver -no-splash  # comment this line out to disable screensaver
# Disable Xsession from blanking
@xset s off
@xset -dpms
@xset s noblank
 
@sh ./autostart.sh
# load chromium after boot and point to the localhost webserver in full screen mode
@chromium-browser --kiosk --no-default-browser-check --no-first-run --disable-infobars "http://localhost/"

Chromium has a feature to show “Restore pages” nagging popup when not grafefully shutdown and you can try the following Stack Overflow suggestion. What was also suggested was doing “chmod 001 ~/.config/chromium/Default/Preferences” but it results to another nagging window.

$ cat ./autostart.sh
#!/bin/sh
sed -i 's/"exited_cleanly":false/"exited_cleanly":true/' ~/.config/chromium/'Local State'
sed -i 's/"exited_cleanly":false/"exited_cleanly":true/; s/"exit_type":"[^"]\+"/"exit_type":"Normal"/' ~/.config/chromium/Default/Preferences

You could use “–restore-last-session” or “–incognito” parameter which also works but it has several disadvantages, such as disabling the cache and login information. Or maybe I should just use Firefox.

Raspberry Pi and Jenkins Build Monitor

It might be also useful to set Chromium to restart every night. When running Chromium for longer periods it may fill Rpi’s memory with garbage and after it must be hard rebooted.

Turning the monitor on and off automatically

When running Rpi as a wall monitor it’s useful to save energy and extend the life of your monitor by turning the monitor on/off on a daily schedule. You can do this by running a cron script. Get this script and put it in /home/pi/rpi-hdmi.sh and make it executable: chmod +x /home/pi/rpi-hdmi.sh. Call the script at the desired time with cron entry:

$ crontab -e
 
# Turn HDMI Off (22:00)
0 17 * * * /home/pi/rpi-hdmi.sh off
 
# Turn HDMI On (7:00)
30 7 * * * /home/pi/rpi-hdmi.sh on

If you’ve problems with the above script try this which is the original but change “curr_vt=`fgconsole`” to be “curr_vt=`sudo fgconsole`” as fgconsole needs sudo privileges and otherwise you get an error “Couldn’t get a file descriptor referring to the console”.

From simple dashboard to real information Radiator

Showing just Jenkins Build Monitor or Grafana dashboards is simple but to get more information of things running you could show things like the success rate of the builds, build health, latest open pull request and project Twitter Messages. One nice example of information radiator is Panic’s Status board.

There are different ways to create customizable dashboard and one way is to use Dashing which is a dashboard framework and to get headstart you can see Project Dashboard https://github.com/martin-naumann/project-cockpit is which shows “build health” for the latest build, the latest open pull request in Github, the success rate of your builds, some free-form text and your project’s or company’s logo. It uses the Jenkins API to get the ration of successful / non-successful builds as well as the latest build state. It also calls the Github API to get the latest pull request to display the name and picture of the author and the title of the pull request on the dashboard.

For more leisure use you can set up the Raspberry Pi as a wall display to show information like calendar, weather, photos and RSS feeds. One option is to use Dakboard which is a web interface used to display information and is quite configurable with different services. At first Dakboard seems nice but is quite limited on what data it can show and some useful features are premium. Another open source option is MagicMirror² which seems to be more modular and extensible (as you can create your own modules) but needs more tinkering.

Monthly notes 28

Winter refuses to make way for the Spring and March in Southern Finland has been quite cold despite warm and rainy week which melted away some of the already scarce snow. So, skiing mainly on artificial snow and mountainbiking on icy paths which is nice. But this also leaves time to read what has happened on software development field. This month it’s about tools and working methods.

Issue 28: 17.3.2018

Tools of the trade

Must have extensions for VS Code (according to me)
tl;dr; Auto import, Beautif, Clipboard History, Git History, Code Runner, Docker, Material Icon Theme, Path Intellisense. (from @ThePracticalDev)

Reclaim your abandonware
Super cool post about how to get the abandoned mac Twitter client to support 280 character tweets by modifying its assembly. (from @b0rk)

Keep calm and code on: Productivity tools for developers
Suggestion of tools for developers for different tasks. I didn’t agree all of them so my suggestions are in brackets. tl;dr; 1. actiTIME (or toggl) 2. Cold Turkey (or other pomodoro) 3. Strict Workflow 4. Habitica 5. Oh My Zsh (or Bash-it) 6. The Silver Searcher 7. UltraEdit (or Atom, VS Code etc.) 8. Homebrew 9. GitHub Changelog Generator. (from @ThePracticalDev)

Git aliases
If you use Git command-line a lot, you will probably grow your own list of Git aliases sooner or later. After simple standard aliases (ci -> commit, co -> checkout) you might want to see some advanced tricks you may find useful.

CTFR
Get subdomains of an HTTPS website abusing Certificate Transparency logs. (from @KitPloit). Apparently also curl "https://crt.sh/?q=%.starbucks.com&output=json" -sS | jq .name_value | uniq | tee output works.

Front-end

Front-End Performance Checklist 2018
Unbiased and objective front-end performance checklist for 2018 — an overview of the issues you might need to consider to ensure that your response times are fast, user interaction is smooth and your sites don’t drain user’s bandwidth. (from @igrigorik)

Working methdods

Tim Ottinger: what once was thought impossibility is now commonplace in software development
TDD, pairing, mobbing, evolutionary design, self organizing, lean startup, commenting code, interpreted languages, beta, noestimates. “You have to ask, what impossible thing is going to be done next? We change how we think, and new vistas open up.”

Branching Is Easy. So? Git-flow Is Not Agile.
I wrote this blog post ages ago and I still stand firmly behind it trunk-based development 4ever. (from @skamille)

Getting Things Done – A Programmer Productivity Guide
“Everybody has some sort of system—even not having a system and trying to remember everything is technically a system. I wanted to share mine because it seems to work pretty well.” (from @ThePracticalDev)

Herding cats is easy compared to managing developers (video)
A short and sharp 10 minute guide to managing developers by Dom Millar at NDC Conference Sydney 2017.

Something different

10 x weekend brunches in Helsinki
Il Birricifio, Ipi Kulmakuppila, Sandro, Gastro Café Kallio, Yes Yes Yes!, Loop, Moko Market, Sue Ellen, Paulig Kulma and Krog Roba. My addition to the list is Rupla. (from @VisitHelsinki)

Monthly notes 27

For cold winter evenings here’s something to read. Monthly notes for February are about relearning and thinking.

Issue 27: 23.2.2018

Relearning

Computer Science and why it’s necessary even for web developers
“Computer Science and why it’s necessary even for web developers I know that in some countries a degree in CS is expensive or unattainable, and that some companies do unnecessary algorithm interviews. This thread is not about degrees or interviews, it’s about CS itself.”

Free Intro to Web Development slides (with demos)
Slides of the Web Dev Intro labs for the “6.813 User Interface Design and Implementation” at MIT
(from Twitter)

The Four Rules of Simple Design (in order of importance)

  • Passes the tests
  • Reveals intention
  • No duplication
  • Fewest elements

And, yes, “fewest elements” is last, which means you only minimize classes and methods if everything else satisfied

Tools

shuttle
When openssh port forwarding doesn’t cut it, use sshuttle: “Transparent proxy meets VPN meets ssh.”

Microservices

The Death of Microservice Madness in 2018
There are many cases where great efforts have been made to adopt microservice patterns without necessarily understanding how the costs and benefits will apply to the specifics of the problem at hand. The post describes in detail what microservices are, why the pattern is so appealing, and also some of the key challenges that they present.

Should that be a Microservice? Keep These Six Factors in Mind
These days, you can’t swing a dry erase marker without hitting someone talking about microservices but few have spent any appreciable time asking if a given application should be a microservice. tl;dr; “1. Multiple Rates of Change; 2. Independent Life Cycles; 3. Independent Scalability; 4. Isolated Failure; 5. Simplify Interactions with External Dependencies; 6. The Freedom to Choose the Right Tech for the Job”.

JavaScript

A Guide to Web Performance Optimization with Webpack
This guide walks through how to effectively optimize site resources using webpack. This can help users load and interact with your sites more quickly. (from JavaScript Weekly 373)

Security

face-verify.js: Monitoring who is physically looking at a website for additional security
Demo project showing how Machine Box tech can be integrated into JavaScript applications. Facebox takes an image and tells you how many faces it sees, as well as who those faces belong to provided you have shown it a single example previously. You can use this capability to build additional security into web apps so you can see how many people are watching the screen and who they are. Using the webcam with some JavaScript and Facebox, you can periodically check to ensure only authorised people can see the information that users consider sensitive.

Mac Privacy: Sandboxed Mac apps can record your screen at any time without you knowing
TL;DR Any Mac app can take screenshots of your Mac silently, and use basic OCR software to read all text on the screen. (from Weekend Reading)

To think about

Nick Stenning on Twitter
“Flat organisational structures do not exist. There are only organisations with visible structure and organisations with invisible structure”. (from Weekend Reading)

Developers On Call
Quite self-explanatory ideas for how to manage on-call rotations without burn out but maybe it’s not always that way. The linked Twitter thread is worth reading. (from Weekend Reading)

Something different

2017: The Year in Charts
These are the charts and themes that tell the story of 2017. I. The Year Volatility Died; II. Records Are Made to Be Broken; III. The World is Flattening; IV. Still Easy After All These Years; V. A Good Old-Fashioned Mania; VI. King Dollar Dethroned; VII. Wrapping Up: 1991-99 Redux?

Extracting JSON value from command line with jq and Python

Developing modern web applications you often come to around checking REST API responses and parsing JSON values. You can do it with a combination of Unix tools like sed, cut and awk but if you’re allowed to install extra tools or use Python then things get easier. This post shows you couple of options for extracting JSON values with Unix tools.

There are a number of tools specifically designed for the purpose of manipulating JSON from the command line, and will be a lot easier and more reliable than doing it with awk. One of those tools is jq as shown Stack Overflow. You can install it in macOS from Homebrew: brew install jq.

$ curl -s 'https://api.github.com/users/walokra' | jq -r '.name'

If you’re limited to tools that are likely installed on your system such as Python, using the json module gives you the benefit of a proper JSON parser and avoiding any extra dependencies.

Python:

$ curl -s 'https://api.github.com/users/walokra' | \
    python -c "import sys, json; print(json.load(sys.stdin)['name'])"

Stack Overflow answers to the question of “Parsing JSON with Unix tools” shows you other options with standard tools like sed, cut and Awk and more exotic options with Perl, Node.js and PHP.

Monthly notes 26

January finally brought snow also to Southern Finland and darkness is retreating slowly when the day becomes longer. This time monthly notes tells you about different JavaScript frameworks, making webpack perform better and looks into bootstrapping microservices and running docker securely. On programming side there are articles for best practices with Kotlin and about Kotlin stdlib. If you haven’t stumbled upon Kotlin, it’s good to check it out as it’s a nice language for building services targeting the Java Virtual Machine.

Issue 26: 23.1.2018

Web development

An Extensive Guide to JS Frameworks
The world is full of JavaScript frameworks and this roundup post goest through 52 of them and touches on their pros, cons, and distinctive features. (from JavaScript Weekly 369)

2017’s JavaScript Rising Stars
A look at what JS projects were hot or not in 2017 based on their GitHub star counts. (from JavaScript Weekly 369)

Keep webpack Fast: A Guide for Better Build Perf
webpack is a great tool for bundling frontend assets but it’s worth knowing what to do when it starts to get bogged down. (from JavaScript Weekly 369)

Short

webpack: Plugin to Remove Unused Moment.js Locales

Microservices

Bootstrapping a microservice architecture (screencast)
Screencasts to present an open source bootstrap project to help you with your next microservice architecture using Java. The repository addresses some common challenges that everyone faces when starting with microservices.

Top tips to keep Docker running securely in production (video)
Gianluca Arbezzano gave important tips on setting up a production environment, immutability, and security concepts for dockers in his session at the DevOpsCon 2017.

Kotlin

Idiomatic Kotlin. Best Practices.
“In order to take full advantage of Kotlin, we have to revisit some best practices we got used to in Java. Many of them can be replaced with better alternatives that are provided by Kotlin.”

Make your life easier with Kotlin stdlib
“Kotlin is not about big killer features but about a bunch of small improvements that have deep impact. Most of them are not built-in into the language, but are functions offered as part of the Kotlin standard library.” The post goes through a limited set of them, and describes how they can be used to improve the code.

Something different

The best science fiction, fantasy, and horror novels of 2017
The Verge lists great books of 2017 in science fiction, fantasy, and horror category which shined a light in the darkness. You newer know if a book is interesting by reading it’s description but these took my eye: Meg Howrey’s The Wanderers, Kameron Hurley’s The Stars are Legion, N.K. Jemisin’s Broken Earth trilogy, Zachary Mason’s Void Star, Joe M. McDermott’s The Fortress at the End of Time, Ian McDonald Luna: New Moon and Linda Nagata’s The Last Good Man.

2017 Retrospective

It’s January 2018 and while I’m gathering my notes for the year’s first post its’ good to look back what I wrote in 2017 and make plans for the new year. In 2017 I managed to write as leisurely as usual and put together 17 articles of which 6 are something other than monthly notes. On average I wrote 1.4 post per month. I visited some meetups, did software development and tested technology stuff. Business as usual and I presume that it’s going to continue this way also this year.

Monthly notes

Writing Monthly notes series about interesting articles I’ve come across has proved to be good way to ensure that I keep reading what happens in software development and also think about it. Collecting articles to monthly post have worked better than publishing weekly. In July I was mostly mountain biking and away from the computer so there was no Monthly notes.

Meetups

Meetup scene in Helsinki has grown and there are several interesting events you can attend almost monthly. But that said, it’s also starting to get growded and events with good topics tend to fill up quickly. I usually find myself going to events to hear war stories of Amazon Web Services, Docker, DevOps, Frontend and Mobile. It’s useful to hear how other’s do things and get new ideas. Meetups and conferences are also nice way to both freshen your thinking and get to know people working in the same field.

In Nebula Tech Thursday – Beer & DevOps we heard stories about “Cloud Analytics – Providing Insight on Application Health and Performance” and “Building a Full Devops Pipeline with Open Source Tools”. OWASP Helsinki chapter meeting #31 presented topics like “DevSec – Developers are the key to security”, “Docker Security” and “Leaking credentials – a security malpractice more common than expected”. Both events where nice and as usual Nebula Tech Thursday with great food and drinks. If you follow me on Twitter you might have noticed that I went to more meetups than I wrote about, like Solita Core and Slush.D.

Software development as usual

Microservices and Docker has evolved the way we do things and to Dockerize all the things you can run Ansible inside Docker container. You might ask why and that’s easy to answer: isolating all of the required dependencies from the host machine and to get the Ansible version we want.

For making software development more reliable I introduced git pre-commit and pre-receive hooks for validating YAML to our continuous integration process. Validating YAML can be done by using a yamllint and hooking it to pre-commit or pre-receive helps you to automate the check for syntax validity, for weirdnesses like key repetition and cosmetic problems such as lines length, trailing spaces and indentation.

Other things

As an engineer I’m interested of technology and gadgets and sometimes I get things to test. In July I wrote about keeping data secured with iStorage datAshur Personal2 USB flash drive. It is an USB flash drive with combination of hardware encryption, physical keypad and tamper-proofing. Small external devices are easy to lose and can leave your data vulnerable if not encrypted. The hardware encrypted USB flash drive seemed to be quite crafty.

Awesome times ahead

New year, old me. Or something like that. Plans are to continue as before, write about technology, collect interesting articles, learn new things about software development and of course ride mountain bike. The training for the Enduro racing season has already started.

So, stay tuned by subscribing to the RSS feed or follow me on Twitter. Check also my other blog in Finnish.

Monthly notes 25

December has gone fast and this time the monthly notes are more about pointers to tools and resources. Especially for accessibility which is important aspect of web development. If you don’t follow front-end development actively check out the recap of it’s development in 2017. And to learn more about security it’s good to read the updated OWASP Top-10 list. Happy reading and have nice holidays!

Issue 25, 20.12.2017

Web development

A recap of front-end development in 2017
tl;dr; PWA, yarn, serverless, vue.js, css-in-js, GraphQL, React Router 4, types in JavaScript.

Pointers for better accessibility

Inclusive Components
Blog which writes about designing inclusive web interfaces, piece by piece. Trying to be a pattern library.

Web Accessibility In Mind
Resources for reading about Web accessibility.

Web Accessibility Checklist
A beginner’s guide to web accessibility.

aXe
Nice open-source tool for accessibility testing. Runs right in your web browser.

NonVisual Desktop Access
Developing for better accessibility is easier when you can test how end users “see” things. NVDA (NonVisual Desktop Access) is a free “screen reader” for Windows which enables blind and vision impaired people to use computers. It reads the text on the screen in a computerised voice.

Security

OWASP Top 10 – 2017
The Ten Most Critical Web Application Security Risks. Read the PDF.

Internet Chemotherapy
Internet Chemotherapy was a 13 month project between Nov 2016 – Dec 2017. It has been known under names such as ‘BrickerBot’, ‘bad firmware upgrade’.

Testing tools

Cypress
“Cypress is the new standard in front-end testing that every developer and QA engineer needs. No more Selenium. Lots more power.”

TestCafe
“A Node.js tool to automate end-to-end web testing. Write tests in JS or TypeScript, run them and view results.”

mountebank
Provides cross-platform, multi-protocol test doubles over the wire. Simply point your application under test to mountebank instead of the real dependency, and test like you would with traditional stubs and mocks.

Something different

The 10 Best Mountain Biking Videos of the Year

Git pre-commit and pre-receive hooks: validating YAML

Software development has many steps which you can automate and one useful thing to automate is to add Git commit hooks to validate your commits to version control. Firing off custom client-side and server-side scripts when certain important actions occur. Validating commited files’ contents is important for syntax validity and even more when providing Spring Cloud Config configurations in YAML for microservices as otherwise things fail.

Validating YAML can be done by using a yamllint and hooking it to pre-commit or pre-receive. It does not only check for syntax validity, but for weirdnesses like key repetition and cosmetic problems such as lines length, trailing spaces and indentation. Here’s a short overview to get started with yamllint on Git commit hooks.

Quickstart for yamllint

Installing yamllint

On Fedora / CentOS:
$ sudo dnf install yamllint
 
using pip, the Python package manager:
$ sudo pip install yamllint
 
or as in macOS
$ sudo -H python -m pip install yamllint

You can also install yamllint from sources when e.g. network connectivity is limited. The linter depends on pathspec >=0.5.3 and pyyaml >= 3.12.

Custom config

Yamllint is quite strict with validation and you might want to make it a bit more relax with custom configuration. For example I need to allow long lines. You can also disable checks for a specific line with a comment.

$ cat yamllint-config.yml
 
extends: default
 
rules:
  line-length: disable
  comments:
    require-starting-space: false

Usage

$ yamllint file.yml other-file.yaml

Usage with custom config:

$ yamllint -c yamllint-config.yml .

Or with custom config without config file:

$ yamllint -d "{extends: relaxed, rules: {line-length: {max: 120}}}" file.yaml

Or more specific case like running yamllint in Jenkins job’s workspace and validating files with specific suffix:

$ find . -type f -iname '*.j2' -exec yamllint -s -c yamllint-config.yaml {} \;

Pre-commit hook and yamllint

Better way to use yamllint is to integrate it with e.g. git and pre-commit-hook or pre-receive-hook. Adding yamllint to pre-commit-hook is easy with pre-commit which is a framework for managing and maintaining multi-language pre-commit hooks.

Installing pre-commit:

Using pip:
$ pip install pre-commit
 
Or on macOS:
$ brew install pre-commit

To enable yamllint pre-commit plugin you just add a file called .pre-commit-config.yaml to the root of your project and add following snippet to it

$ cat .pre-commit-config.yaml
---
- repo: https://github.com/adrienverge/yamllint.git
  sha: v1.10.0
  hooks:
    - id: yamllint

With custom config and strict mode:

$ cat .pre-commit-config.yaml
---
repos:
 - repo: https://github.com/adrienverge/yamllint.git
   sha: v1.10.0
   hooks:
     - id: yamllint
       args: ['-d {extends: relaxed, rules: {line-length: disable}}', '-s']

You can also use repository-local hooks when e.g. it makes sense to distribute the hook scripts with the repository. Install yamllint locally and configure yamllint to your project’s root directory’s .pre-commit-config.yaml as repository local hook. As you can see, I’m using custom config for yamllint.

$ cat .pre-commit-config.yaml
---
- repo: local
  hooks: 
  - id: yamllint
    name: yamllint
    entry: yamllint -c yamllint-config.yml .
    language: python
    types: [file, yaml]

Note: If you’re linting files with other suffix than yaml/yml like ansible template files with .j2 suffix then use types: [file]

Pre-receive hook and yamllint

Using pre-commit-hooks to process commits is easy but often doing checks in server side with pre-receive hooks is better. Pre-receive hooks are useful for satisfying business rules, enforce regulatory compliance, and prevent certain common mistakes. Common use cases are to require commit messages to follow a specific pattern or format, lock a branch or repository by rejecting all pushes, prevent sensitive data from being added to the repository by blocking keywords, patterns or filetypes and prevent a PR author from merging their own changes.

One example of pre-receive hooks is to run a linter like yamllint to ensure that business critical file is valid. In practice the hook works similarly as pre-commit hook but files you check in to repository are not kept there “just like that”. Some of them are stored as deltas to others, or their the contents are compressed. There is no place where these files are guaranteed to exist in their “ready-to-consume” state. So you must take some extra hoops to get your files available for opening them and running checks.

There are different approaches to make files available for pre-receive hook’s script as StackOverflow describes. One way is to check out the files in a temporary location or if you’re on linux you can just point /dev/stdin as input file and put the files through pipe. Both ways have the same principle: checking modified files between new and the old revision and if files are present in new revision, runs the validation script with custom config.

Using /dev/stdin trick in Linux:

#!/usr/bin/env bash
 
set -e
 
ENV_PYTHON='/usr/bin/python'
 
if ((
       (ENV_PYTHON_RETV != 0) && (YAMLLINT != 0)
)); then
    echo '`python` or `yamllint` not found.'
    exit 1
fi
 
oldrev=$1
newrev=$2
refname=$3
 
while read oldrev newrev refname; do
    # Get a list of all objects in the new revision
    objects=`git ls-tree --full-name -r ${newrev}`
 
    # Get the file names, without directory, of the files that have been modified
    # between the new revision and the old revision
    git diff --name-only $oldrev $newrev | while read file; do
        # Search for the file name in the list of all objects
        object=`echo -e "${objects}" | egrep "(\s)${file}\$" | egrep '\.yml$' | awk '{ print $3 }'`
        # If it's not present, then continue to the the next itteration
        if [ -z ${object} ]; 
        then 
            continue; 
        fi
 
        # Get file in commit and point /dev/stdin as input file 
        # and put the files through pipe for syntax validation
        echo $file
        git show $newrev:$file | /usr/bin/yamllint -d "{extends: relaxed, rules: {line-length: disable, comments: disable, trailing-spaces: disable, empty-lines: disable}}" /dev/stdin || exit 1
    done
done

Alternative way: copy changed files to temporary location

#!/usr/bin/env bash
 
set -e
 
EXIT_CODE=0
ENV_PYTHON='/usr/bin/python'
COMMAND='/usr/bin/yamllint'
TEMPDIR=`mktemp -d`
 
if ((
        (ENV_PYTHON_RETV != 0) &&
        (YAMLLINT != 0)
)); then
    echo '`python` or `yamllint` not found.'
    exit 1
fi
 
oldrev=$1
newrev=$2
refname=$3
 
while read oldrev newrev refname; do
 
    # Get the file names, without directory, of the files that have been modified
    # between the new revision and the old revision
    files=`git diff --name-only ${oldrev} ${newrev}`
 
    # Get a list of all objects in the new revision
    objects=`git ls-tree --full-name -r ${newrev}`
 
    # Iterate over each of these files
    for file in ${files}; do
 
        # Search for the file name in the list of all objects
        object=`echo -e "${objects}" | egrep "(\s)${file}\$" | awk '{ print $3 }'`
 
        # If it's not present, then continue to the the next itteration
        if [ -z ${object} ]; 
        then 
            continue; 
        fi
 
        # Otherwise, create all the necessary sub directories in the new temp directory
        mkdir -p "${TEMPDIR}/`dirname ${file}`" &>/dev/null
        # and output the object content into it's original file name
        git cat-file blob ${object} > ${TEMPDIR}/${file}
 
    done;
done
 
# Now loop over each file in the temp dir to parse them for valid syntax
files_found=`find ${TEMPDIR} -name '*.yml'`
for fname in ${files_found}; do
    ${COMMAND} ${fname}
    if [[ $? -ne 0 ]];
    then
      echo "ERROR: parser failed on ${fname}"
      BAD_FILE=1
    fi
done;
 
rm -rf ${TEMPDIR} &> /dev/null
 
if [[ $BAD_FILE -eq 1 ]]
then
  exit 1
fi
 
exit 0

Testing pre-receive hook locally is a bit more difficult than pre-commit-hook as you need to get the environment where you have to remote repository. Fortunately you can use the process which is described for GitHub Enterprise pre-receive hooks. You create a local Docker environment to act as a remote repository that can execute the pre-receive hook.