Extracting JSON value from command line with jq and Python

Developing modern web applications you often come to around checking REST API responses and parsing JSON values. You can do it with a combination of Unix tools like sed, cut and awk but if you’re allowed to install extra tools or use Python then things get easier. This post shows you couple of options for extracting JSON values with Unix tools.

There are a number of tools specifically designed for the purpose of manipulating JSON from the command line, and will be a lot easier and more reliable than doing it with awk. One of those tools is jq as shown Stack Overflow. You can install it in macOS from Homebrew: brew install jq.

$ curl -s 'https://api.github.com/users/walokra' | jq -r '.name'

If you’re limited to tools that are likely installed on your system such as Python, using the json module gives you the benefit of a proper JSON parser and avoiding any extra dependencies.

Python:

$ curl -s 'https://api.github.com/users/walokra' | \
    python -c "import sys, json; print(json.load(sys.stdin)['name'])"

Stack Overflow answers to the question of “Parsing JSON with Unix tools” shows you other options with standard tools like sed, cut and Awk and more exotic options with Perl, Node.js and PHP.

Monthly notes 26

January finally brought snow also to Southern Finland and darkness is retreating slowly when the day becomes longer. This time monthly notes tells you about different JavaScript frameworks, making webpack perform better and looks into bootstrapping microservices and running docker securely. On programming side there are articles for best practices with Kotlin and about Kotlin stdlib. If you haven’t stumbled upon Kotlin, it’s good to check it out as it’s a nice language for building services targeting the Java Virtual Machine.

Issue 26: 23.1.2018

Web development

An Extensive Guide to JS Frameworks
The world is full of JavaScript frameworks and this roundup post goest through 52 of them and touches on their pros, cons, and distinctive features. (from JavaScript Weekly 369)

2017’s JavaScript Rising Stars
A look at what JS projects were hot or not in 2017 based on their GitHub star counts. (from JavaScript Weekly 369)

Keep webpack Fast: A Guide for Better Build Perf
webpack is a great tool for bundling frontend assets but it’s worth knowing what to do when it starts to get bogged down. (from JavaScript Weekly 369)

Short

webpack: Plugin to Remove Unused Moment.js Locales

Microservices

Bootstrapping a microservice architecture (screencast)
Screencasts to present an open source bootstrap project to help you with your next microservice architecture using Java. The repository addresses some common challenges that everyone faces when starting with microservices.

Top tips to keep Docker running securely in production (video)
Gianluca Arbezzano gave important tips on setting up a production environment, immutability, and security concepts for dockers in his session at the DevOpsCon 2017.

Kotlin

Idiomatic Kotlin. Best Practices.
“In order to take full advantage of Kotlin, we have to revisit some best practices we got used to in Java. Many of them can be replaced with better alternatives that are provided by Kotlin.”

Make your life easier with Kotlin stdlib
“Kotlin is not about big killer features but about a bunch of small improvements that have deep impact. Most of them are not built-in into the language, but are functions offered as part of the Kotlin standard library.” The post goes through a limited set of them, and describes how they can be used to improve the code.

Something different

The best science fiction, fantasy, and horror novels of 2017
The Verge lists great books of 2017 in science fiction, fantasy, and horror category which shined a light in the darkness. You newer know if a book is interesting by reading it’s description but these took my eye: Meg Howrey’s The Wanderers, Kameron Hurley’s The Stars are Legion, N.K. Jemisin’s Broken Earth trilogy, Zachary Mason’s Void Star, Joe M. McDermott’s The Fortress at the End of Time, Ian McDonald Luna: New Moon and Linda Nagata’s The Last Good Man.

2017 Retrospective

It’s January 2018 and while I’m gathering my notes for the year’s first post its’ good to look back what I wrote in 2017 and make plans for the new year. In 2017 I managed to write as leisurely as usual and put together 17 articles of which 6 are something other than monthly notes. On average I wrote 1.4 post per month. I visited some meetups, did software development and tested technology stuff. Business as usual and I presume that it’s going to continue this way also this year.

Monthly notes

Writing Monthly notes series about interesting articles I’ve come across has proved to be good way to ensure that I keep reading what happens in software development and also think about it. Collecting articles to monthly post have worked better than publishing weekly. In July I was mostly mountain biking and away from the computer so there was no Monthly notes.

Meetups

Meetup scene in Helsinki has grown and there are several interesting events you can attend almost monthly. But that said, it’s also starting to get growded and events with good topics tend to fill up quickly. I usually find myself going to events to hear war stories of Amazon Web Services, Docker, DevOps, Frontend and Mobile. It’s useful to hear how other’s do things and get new ideas. Meetups and conferences are also nice way to both freshen your thinking and get to know people working in the same field.

In Nebula Tech Thursday – Beer & DevOps we heard stories about “Cloud Analytics – Providing Insight on Application Health and Performance” and “Building a Full Devops Pipeline with Open Source Tools”. OWASP Helsinki chapter meeting #31 presented topics like “DevSec – Developers are the key to security”, “Docker Security” and “Leaking credentials – a security malpractice more common than expected”. Both events where nice and as usual Nebula Tech Thursday with great food and drinks. If you follow me on Twitter you might have noticed that I went to more meetups than I wrote about, like Solita Core and Slush.D.

Software development as usual

Microservices and Docker has evolved the way we do things and to Dockerize all the things you can run Ansible inside Docker container. You might ask why and that’s easy to answer: isolating all of the required dependencies from the host machine and to get the Ansible version we want.

For making software development more reliable I introduced git pre-commit and pre-receive hooks for validating YAML to our continuous integration process. Validating YAML can be done by using a yamllint and hooking it to pre-commit or pre-receive helps you to automate the check for syntax validity, for weirdnesses like key repetition and cosmetic problems such as lines length, trailing spaces and indentation.

Other things

As an engineer I’m interested of technology and gadgets and sometimes I get things to test. In July I wrote about keeping data secured with iStorage datAshur Personal2 USB flash drive. It is an USB flash drive with combination of hardware encryption, physical keypad and tamper-proofing. Small external devices are easy to lose and can leave your data vulnerable if not encrypted. The hardware encrypted USB flash drive seemed to be quite crafty.

Awesome times ahead

New year, old me. Or something like that. Plans are to continue as before, write about technology, collect interesting articles, learn new things about software development and of course ride mountain bike. The training for the Enduro racing season has already started.

So, stay tuned by subscribing to the RSS feed or follow me on Twitter. Check also my other blog in Finnish.