Monthly notes 47

Issue 47: 30.1.2020

War Stories

#Y2038 problem. "It's *already here*. Fix your stuff."
In many systems time is represented as number of seconds passed since 00:00:00 UTC on 1 Jan 1970 and stored as signed 32-bit integer. Such implementations can't encode times after 03:14:07 UTC on 19 January 2038. (from @walokra)

Ops Lessons We All Learn The Hard Way
Good Twitter thread of lessons learned on Ops.

Web

Front-End Performance Checklist 2020
Great resource to read for better front-end performance. Remember, if you don’t measure it, you can’t improve it 🚀 To get you started: use i.a. page speed and lighthouse to see where you stand.

JavaScript

20 ways to become better node.js developer in 2020
tl;dr; Sleep more; Use Jest & Ava; GraphQL!; Check Nest.js; Gradual deployment; Test in production? Learn Docker & Kubernetes; Read vulnerable code; Use monitoring; CI with quality tools;

Kubernetes

How Soon We Forget: Security in the Age of Docker & Kubernetes
Good starting point for hardening your containers and Kubernetes cluster. tl;dr; Running as non-root. Read-only file system. Not terminating TLS too soon. Setting resources limits (Denial of service). Use Kubernetes policies. (from nicolas_frankel)

Software Development

Goodbye, Clean Code
AHA! Avoid Hasty Abstractions. Prefer duplication over the wrong abstraction. Check also Dan’s tweet’s thread.

Remote working tips
tl;dr; the thread: 1) activity signal to start work day 2) frequent small breaks 3) dedicate a space for work (not sofa/bed) 4) take sick days 5) connect with other humans 6) non-project connection point with peers 7) block out distractions 8) go out for lunch.

Tools

tiny-helpers.dev
Collection of useful single-purpose online tools that are useful for web devs. (from @stefanjudis)

Insomnia
"Debug APIs like a human, not a robot". If you don't like Postman then try Insomnia which says to be powerful HTTP and GraphQL tool belt and open source. Seems that it doesn't have similar scripting and testing features as Postman though.

Something different

Finding the best bicycle chain
tl;dr; No major reason for not to use drivetrain manufacturer’s recommended chain. The lubricant you use will play the most critical role in drivetrain durability. Run a good lube and keep your drivetrain clean.

Tracking vulnerabilities and keeping Node.js packages up to date

Software evolves quickly and new versions of libraries are released but how do you keep track of updated dependencies and vulnerable libraries? Managing dependencies has always been somewhat a pain point but an important part of software development as it's better to be tracking vulnerabilities and running fresh packages than being pwned.

There are couple of tools for JavaScript projects which use npm to manage dependencies to check new versions and some tools to track vulnerabilities. Here's a short introduction to npm audit, depcheck, npm-check-updates and npm-check to help you on your way.

If your project is using yarn adjust your workflow accordingly. There's for example yarn audit and yarn-check to match tools for npm. And it goes without saying that don't use npm if your project uses yarn.

Running security audit with npm audit

From version 6 onwards npm comes build with audit command which checks for vulnerabilities in your dependencies and runs automatically when you install a package with npm install. You can also run npm audit manually on your locally installed packages to conduct a security audit of the package and produce a report of dependency vulnerabilities and suggested patches.

The npm audit command submits a description of the dependencies configured in your package to your default registry and asks for a report of known vulnerabilities. It checks direct dependencies, devDependencies, bundledDependencies, and optionalDependencies, but does not check peerDependencies.

If your npm registry doesn't support npm audit, like Artifactory, you can pass in the --registry flag to point to public npm. The downside is that now you can't audit private packages that are on the Artifactory registry.

$ npm audit --registry=https://registry.npmjs.org

"Running npm audit will produce a report of security vulnerabilities with the affected package name, vulnerability severity and description, path, and other information, and, if available, commands to apply patches to resolve vulnerabilities."

Example: partial output of npm audit run

Using npm audit is useful also in Continuous Integration as it will return a non-zero response code if security vulnerabilities are found.

For more information read npm's Auditing dependencies for security vulnerabilities.

Updating packages with npm outdated

It's recommended to regularly update the local packages your project depends on to improve your code as improvements to its dependencies are made. In your project root directory, run the update command and then outdated. There should not be any output.

$ npm update
$ npm outdated 
Example of results from npm outdated

You can also update globally-installed packages. To see which global packages need to be updated run outdated first with --depth=0.

$ npm outdated -g --depth=0
$ npm outdated -g

For more information read updating packages downloaded from the registry.

Check updates with npm-check-updates

Package.json contains dependencies with semantic versioning policy and to find newer versions of package dependencies than what your package.json allows you need tools like npm-check-updates. It can upgrade your package.json dependencies to the latest versions, ignoring specified versions while maintaining your existing semantic versioning policies.

Install npm-check-updates globally with:

$ npm install -g npm-check-updates 

And run it with:

$ ncu

The result shows any new dependencies for the project in the current directory. See documentation for i.a. configuration files for filtering and excluding dependencies.

Example of results from ncu

And finally you can run ncu -u to upgrade the package.json.

Check updates with npm-check

Similar tool to npm-check-updates is npm-check which additionally gives more information about the version changes available and also lets you interactively pick which packages to update instead of an all or nothing approach. It checks for outdated, incorrect, and unused dependencies.

Install npm-check globally with:

$ npm i -g npm-check

Now you can run the command inside your project directory:

$ npm-check
Or
$ npm-check --registry=https://registry.npmjs.org

It will display all possible updates with information about the type of update, project URL, commands, and will attempt to check if the package is still in use. You can easily parse through the results and see what packages might be safe to update. When updates are required it will return a non-zero response code that you can use in your CI tools.

The check for unused dependencies uses depcheck and isn't able to foresee all ways dependencies can be used so be vary with careless removing of packages.

To see an interactive UI for choosing which modules to update run:

$ npm-check –u

Analyze dependencies with depcheck

Your package.json is filled with dependencies and some of them might be useless or even missing from package.json. Depcheck is a tool for analyzing the dependencies in a project to see how each dependency is used, which dependencies are useless, and which dependencies are missing. It does not only recognizes the dependencies in JavaScript files, but also supports i.a. React JSX and Typescript.

Install depcheck with:

$ npm install -g depcheck
And with additional syntax support for Typescript
$ npm install -g depcheck typescript

Run depcheck with:

$ depcheck [directory]
Example of results from depcheck

Summary

tl;dr;

  1. Use npm audit in your CI pipeline
  2. Update dependencies with npm outdated
  3. Check new versions of dependencies with either npm-check-updates or npm-check
  4. Analyze dependencies with depcheck

Notes from security in the age of Docker & Kubernetes

Security is always the more obscure part of software development and while container runtimes provide good isolation from the host operating system when using Docker and running containers in Kubernetes, you should not assume to be free from exploits. Remember to use the best practices when you were not using containers.

Here is my notes from How Soon We Forget: Security in the Age of Docker & Kubernetes article which looked at some common regressions in security practices associated with the migration to Docker and Kubernetes and suggested ways to avoid them. And to continue the topic with notes from Taking the Scissors away: make your Kubernetes Cluster safe for DevOps talk which gives good advice and looks at some of the concepts of forcing security of the application workloads both from conceptual and practical points of view. Also Best Practices for Kubernetes deployment and Securing a cluster are worth reading.

These notes don't explain things so it's worth reading either the documents or the articles mentioned above.

Running as non-root

"One of the most common and easiest security lapses to address is running binaries as root."

Use non-root Docker images. It requires effort and is easier for greenfield projects.

In Kubernetes, you can enforce running containers as non-root using the pod and container security context.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
      name: app
    spec:
      containers:
      - image: my/app:1.0.0
        name: app
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
      securityContext:
        fsGroup: 2866
        runAsNonRoot: true
        runAsUser: 2866

Use read-only file system

"Do you really need to write files within a container?"

In Kubernetes, set the root file system to read-only using the pod security context and create an emptyDir volume to mount at /tmp.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
      name: app
    spec:
      containers:
      - env:
        - name: TMPDIR
          value: /tmp
        image: my/app:1.0.0
        name: app
        securityContext:
          readOnlyRootFilesystem: true
        volumeMounts:
        - mountPath: /tmp
          name: tmp
      volumes:
      - emptyDir: {}
        name: tmp

Protect against Denial of service

"Setting resources limits for your containers protects against a host of denial of service attacks."

With resource quotas you can limit a container to e.g. half a CPU and half a GiB of memory. Kubernetes deployment specification would look like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
      name: app
    spec:
      containers:
      - image: my/app:1.0.0
        name: app
        resources:
          limits:
            cpu: 500m
            memory: 512Mi

Health and readiness checks

"It's a good idea to make sure if your application is not healthy that it shuts down properly so it can be replaced. Kubernetes can help you with this if your application can respond to health and readiness checks and you configure them in your pod specification."

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: app
      name: app
    spec:
      containers:
      - image: my/app:1.0.0
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 3
        name: app
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: http
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 3

The liveness probe should indicate if the application is running and readiness probe should indicate if the application can service requests. Read more from Kubernetes documentation.

Use Kubernetes policies

"Kubernetes provides network and pod security policies that give you control over what pods can communicate with each other and what types of pods can be started, respectively."

Pod Security Policies allow you to control what capabilities pods can have. When pod security policies are enabled, Kubernetes will only start pods that satisfy the constraints of the pod security policies.

They say that Pod Security Policy is actually one of the most difficult things to configure properly in Kubernetes cluster. For example it's easy to completely cap your cluster: you can't create any pods.

An example of a pod security policy that enforces some of the best practices mentioned: non-privileged containers, allow only read-only filesystem, minimum set of allowed volumes and don't use host’s network, pid or ipc namespaces.

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: best-practices
spec:
  # non-privileged containers
  privileged: false
  allowPrivilegeEscalation: false
  requiredDropCapabilities:
    - ALL
  runAsUser:
    rule: MustRunAsNonRoot
  supplementalGroups:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  fsGroup:
    rule: MustRunAs
    ranges:
      - min: 1
        max: 65535
  # restrict file systems
  readOnlyRootFilesystem: true
  volumes:
    - configMap
    - emptyDir
    - projected
    - secret
    - downwardAPI
    - persistentVolumeClaim
  # limit interaction with host
  hostNetwork: false
  hostIPC: false
  hostPID: false

Network Policies

"Network policies allow you to define ingress and egress rules, i.e., firewall rules, for your pods using IP CIDR ranges and Kubernetes label selectors for pods and namespaces, similar to how Kubernetes service resources select pods."

For example you can create a network policy which will deny ingress from pods in other namespaces but allow pods within the namespace to communicate with each other.

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: deny-from-other-namespaces
  namespace: mine
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - podSelector: {}

There is a GitHub repository of common network policies to help you get started using network policies.

Namespaces

Use namespaces and ensure that you've set the following defaults:

Summary

"defense in depth" is still important even in the world of containers. The container is not safe. The operating system is not safe. The host is not safe. The network is not safe.

How Soon We Forget: Security in the Age of Docker & Kubernetes