Field Ops Guide "The Field Ops Guide (by Futurice) is a booklet that makes it possible to survive a software development project. It's a distillation of years of wisdom gathered working in client projects."
Threat matrix for Kubernetes "While Kubernetes has many advantages, it also brings new security challenges that should be considered. Therefore, it is crucial to understand the various security risks that exist in containerized environments, and specifically in Kubernetes."
Faster Builds and Smaller Images Using BuildKit and Multistage Builds "Multistage builds feature in Dockerfiles enables you to create smaller container images with better caching and smaller security footprint. In this blog post, I’ll show some more advanced patterns that go beyond copying files between a build and a runtime stage, allowing to get most out of the feature."
This year has been challenging for meetups and gatherings but one good side of the restrictions was that remote work has become more acceptable and also meetups and conferences have invested to streaming and virtual participation which is great for people living in an area where there's no meetups.
Fighting alert fatigue and visibility issues in SOC
Juuso Myllylä from OptimeSys talked about fighting alert fatigue in security operations center (stream from 41:41 onwards). The goal in the talk was to improve automated detection, introduce "detection logic killchain" framework he has worked on as his master's thesis and shift our minds from signature based detection and move towards intelligence based.
Threat detection framework based on design science research method:
Identify: What is a threat? What kind of things make up a threat?
Convert your idea into a security information and event management (SIEM) search query
Procedures: many APT (Advanced Persistent Threat) groups have used valid accounts as an entrypoint
Deploy the use case
Evaluate: evaluate detection logic
Analyze the SIEM logs once your SIEM use case has been deployed
e.g. check Azure AD audit logs, eliminate non-related data
Applies also to threat hunting
Communicate: Document your detection logic in Sigma form
Can be shared with others, try to be SIEM agnostic
iPhone BFU Acquisition and Analysis
The meetup continued with iPhone forensic from @1:19 by Timo Miettinen from Nixu. The presentation explained first how the iPhone iOS filesystem's two main partitions are protected: non-encrypted System and encrypted Data. Data partition is encrypted with burned-into-hardware UID key. The files have additionally 4 classes of Data Protection.
From forensics point of view access to data is protected with many layers: USB connectivity is restricted; Logical extraction is divided to iTunes backup + some media files, password protected backups contain more data, backup password can be reset but has deviations; Full file system extraction needs jailbraking the device; iCloud extraction (synced backup).
The case talked in the talk was a lost iPhone which was later returned by law enforcement. The question was: What was done with it while missing? Was it stolen or just inspected by friendly authorities? Phone was powered off and passcode was changed.
So they had BFU device in their hands for data extraction: device that has been powered off or rebooted and has never been subsequently unlocked. The amount of the data they can theoretically get is really limited.
In BFU the file encryption keys are wiped from the device RAM and only unencrypted class D protected files are available. Biometric authentication is not possible, USB restricted mode is enabled (need biometric authentication or passcode to activate data connections), lockdown records become useless (logical data acquisition impossible) and passcode recovery attack falls to BFU speeds.
Utilizing exploits and jailbreaks:
checkm8: unpatchable bootrom exploit released by axi0mx on September 2019 which enables jailbreaks, activation lock bypass etc.
checkra1n: jailbreak released on November 2019 which utilizes the checkm8 exploit to run unsigned code on an iOS device. Doesn't always pass the USB restricted mode, depends on the combinations of hardware and software versions.
Open source and free tools:
libimobiledevices has collection of useful tools:
SSH over USB using iproxy
ideviceinfo gives iOS and HW versions
idevicecrashreport gets crash logs from the device
In their use case the checkra1n jailbreak didn't work and USB restricted mode was activated. Also some of the commercial tools enabled to extract some data but wasn't able to read the archive format the software created. They decided to the analysis manually which is a good idea even if the tools are working.
Some open source or free tools for analysis:
APOLLO (Apple Pattern of Life Lazy Output'er): parses pattern of life data from databases and properties into human readable format.
For reading plists you can use: plutil -p <filename>
DB Browser for SQLite
Google's protobuf utilities (protoc)
When doing analysis you should look for plists, binary plists, plists in plists, blobs may contain binary plist files and SQLite databases (Shared Memory file .shm, write ahead log .wal). Some applications store data in protocol buffers (protobufs) in SQLite database blobs, plist files or just data files. Tools find most of the interesting data but you can make your own script to dump all text files, convert plist files to readable format, dumps data from every database, get all embedded binary plists from plist files and databases and convert them to readable format.
In their case they found out that the phone was reinstalled 12 hours after it was lost. Mobile banking, social media and instant messaging applications were installed. The device was used to communicate with several contacts and used around the city. The phone was stolen and reinstalled with intention to use it.
Still Fuzzing Faster (U Fool)
Joona Hoikkala talked about Web Fuzzing and using fuff tool for fuzzing directories, login, basic auth, virtual domain, content id:s and more. Follow the talk from the stream at 2:19:00 and the demo starts around @2:33:00. The slides are good starting point.
You can use fuzzing with different input contents to target i.a. GET parameters (names, values or both), headers (Host, authentication, cookies, proxy headers) and POST data (form data, JSON, files). What to look for (matching)? Response codes, content (regexes) response sizes (bytes, # of words).
Laura Kankaala, from Robocorp and Team Whack fame talked about the price of a digital identity starting at 3:20:40. Data is central, security - privacy: how companies view data and how data sellers view data.
What we are
What we have
What we produce
Laura also presented that ~90-99% of data collected is dark data which is collected but not really utilized. But we are just getting started. But it's good to remember that our data belongs to us. We give permission to collectors and controllers.
Do you know what's your data worth? Data is valuable and for example there are companies like doc.ai and datum which tries to monetize it so that also user gets part of it. But yet the data is used more for targeted ads, providing content just for us, increasing efficiency and creating better services. And of course every one remembers Cambridge Analytica and (trying) to affect electoral processes in the US.
The most valuable things being sold online are: credit cards, identity numbers, passports, credentials, phone numbers, home address. Passports are quite logical of value and e.g. France passport goes for $124, USA $115, Canada $103, UK $60 and so on, depending of the data included with it.
Kankaala talked about how the collecting of our data has sneak to our lives (e.g. social credit systems). Companies collect data and when our normal live becomes very tangled with our live online it becomes easier to monitor us, to see what we're up to and to moderate our behaviour. We need to be careful when we allow new type of access to our live, e.g. COVID-19 tracking.
Regulation, awareness and education is at least a patch to some of these issues. We are hackers and we should be the pathfinders and show people that it doesn't have to be that if something works the way it does today and although it works it doesn't mean it works right or ethically.
We are all vulnerable
Magnus Lundgren from Recorded Future told a tale of two databases, a panda and someone who was listening starting at 4:25:00.
There's a race of when vulnerability is found and assigned a CVE number until it's either patched or exploited. 12 517 CVEs were first published on NVD in 2016-2017 and it takes average 33 days until an initial assesment of the vulnerability is made available via NIST's NVD. For example Dirty Cow (CVE-2016-519) it took 21 days to initial release on NVD but it took only 8 days to create an exploit (Proof of Concept shared on Pastebin) for it and sold/shared on the deep and dark web.
A tale of two databases: NVD (NIST) and CNNVD (CNITSEC). In the Chinese CNNVD it takes on average only 13 days compared to 33 days on NVD for initial assesment. The difference comes from the detail that CNNCD is doing active collection while NVD is doing passive collection from vendors. But always it doesn't be that way like it was the case with some Android backdoor where it took 236 days from CNNVD and 60 days from NVD. It takes longer for CNNVD to publish high threat vulnerabilities than low threat ones and during the publication lag Chinese APT groups are exploiting those vulnerabilities.
When the Recorded Future published a blog post identifying 343 "outlier" CVEs (regarding the issue the of CNNVD lag) CNNVD backdated 338 of those CVEs. Someone was listening.
Deep / Dark web monitoring of activity is crucial for a good patching cadence.
Magic can be done with threat intel data that has been organiced for analysis.
Chinese intersection is particularly vicious for foreign companies: Ministry of State Security (China) runs multiple threat actors e.g. APT3, runs CNNVD and cherry picks CNNVD vulnerabilities for targeting.
Some important things to keep in mind when you work remotely Check the Twitter thread for 10 great tips for working remotely. They are also good tips also in general. I've also found the tip 8. be great. Writing notes and making (public) blog posts of them helps you to process new information better and also help other developers. Documentation is often undervalued and it takes time to do it correctly.
It's probably time to stop recommending Clean Code "There is a growing movement against Rob Martin's books (e.g., Clean Code). After reading the article, I have to agree with a lot of it, but I also hope that this movement doesn't push too far to the other side." (from @maybeFrederick) My take is that don't believe everything you read be it on a book or nowadays in the Internet. Use your own thinking and reasoning. "Clean Code" has good points and suggestions but also goes a bit overboard with how "clean" things should look.
Boop "Boop is a place to paste text, and transform it using basic operations. The goal is to allow quick experimentation and avoid using random websites to do that stuff. It's super useful when working with logs, JSON data, etc." (from @OKatBest). This is what I've always needed. No more searching for online tool for a specific task (or looking it from tiny-helpers.dev which is a great collection).
Git-bug Fully embedded bug-tracker in git: you only need your git repository to have a bug tracker.
I've used Google Cloud Platform for some time and got a opportunity to attend Codemen Cloud Academy's Google Workshop which concentrated to "Kubernetes in the Google Cloud" and "Google Cloud Run Serverless Workshop" topics using the Qwiklabs is a platform. Here's my (very) short notes from the workshop and using Qwiklabs. Most of the things I had used already by running our service on GKE but there's always something to learn from other's experiences.
Google Cloud Workshop with Qwiklabs
Qwiklabs is a platform for learning cloud technologies by following exercises and hands-on training. It gives temporary credentials to Google Cloud Platform and Amazon Web Services, so you can learn the cloud using the real thing.
The workshop used Cloud Study Jams 2020 session contents. After we completed the first lab, we were automatically granted 30-day pass to continue doing the rest of the labs. The quests in the labs are "priced" with credits which you can buy ($1 per credit) or get with workshop code.
Kubernetes in Google Cloud
The "Kubernetes in Google Cloud" quest in Qwiklabs is an advanced-level quest which gets you hands-on practice of configuring Docker images and containers, and deploying fully-fledged Kubernetes Engine applications. It teaches you the practical skills needed for integrating container orchestration into your own workflow.
There's nothing much to tell about the quests contents except bunch of docker, gcloud and kubectl commands so I'll not go through them here.
The Kubernetes in Google Cloud Quest in QwikLabs was as hands-on as it promised and the final quest "Challenge Lab" put all the things together with quite strict time limit. Although I had made notes from the previous quests I just and just managed to paste the commands, wait for the cloud to provision and especially for the Jenkins service to run continuous integration jobs.
Overall the "Kubernetes in Google Cloud" lab was excellent overview to Kubernetes and how things work in Google Cloud. It covered essential topics and showed how to do things in practice. It helped to have previous experience with Google Cloud but everything was explained and shown so you can learn by doing.
Qwiklabs Google Cloud quests
Qwiklabs has also other Google Cloud related labs as shown below but I didn't had time to go through them (I totally forgot :/) although the participants who completed the Kubernetes course got two month's free pass to the platform.
How should I answer a health check? Explains how to use liveness and readiness probes (on Kubernetes). Heard that liveness probe should be always off unless there’s a bug in app which it can’t recover. And long checks can be cached.
Managed Kubernetes Price Comparison (2020) "TL;DR: Azure and Digital Ocean don’t charge for the compute resources used for the control plane, making AKS and DO the cheapest for running many, smaller clusters. For running fewer, larger clusters GKE is the most affordable option. Also, running on spot/preemptible/low-priority nodes or long-term committed nodes makes a massive impact across all of the platforms."
Kubectl-debug kubectl-debug is an out-of-tree solution for troubleshooting running pods, which allows you to run a new container in running pods for debugging purpose (examples). The new container will join the pid, network, user and ipc namespaces of the target container, so you can use arbitrary trouble-shooting tools without pre-installing them in your production container image.
Securely transmitting information between services and authorization can be achieved with using JSON Web Tokens. JWTs are an open, industry standard RFC 7519 method for representing claims securely between two parties. Here's a short explanation and guide of what they are, their use and how to generate the needed things.
"JSON Web Token (JWT) is an open standard (RFC 7519) that defines a compact and self-contained way for securely transmitting information between parties as a JSON object. This information can be verified and trusted because it is digitally signed. JWTs can be signed using a secret (with the HMAC algorithm) or a public/private key pair using RSA or ECDSA."
In short, authorization and information exchange are some scenarios where JSON Web Tokens are useful. They essentially encode any sets of identity claims into a payload, provide some header data about how it is to be signed, then calculate a signature using one of several algorithms and append that signature to the header and claims. JWTs can also be encrypted to provide secrecy between parties. When a server receives a JWT, it can guarantee the data it contains can be trusted because it's signed by the source.
Usually two algorithms are supported for signing JSON Web Tokens: RS256 and HS256. RS256 generates an asymmetric signature, which means a private key must be used to sign the JWT and a different public key must be used to verify the signature.
JSON Web Key
JSON Web Key (JWK) provides a mechanism for distributing the public keys that can be used to verify JWTs. The specification is used to represent the cryptographic keys used for signing RS256 tokens. This specification defines two high level data structures: JSON Web Key (JWK) and JSON Web Key Set (JWKS):
JSON Web Key (JWK): A JSON object that represents a cryptographic key. The members of the object represent properties of the key, including its value.
JSON Web Key Set (JWKS): A JSON object that represents a set of JWKs. The JSON object MUST have a keys member, which is an array of JWKs. The JWKS is a set of keys containing the public keys that should be used to verify any JWT.
In short, the service signs JWT-tokens with it's private key (in this case PKCS12 format) and the receiving service checks the signature with the public key which is in JWK format.
Generating keys and certificate for JWT
In this example we are using JWTs for information exchange as they are a good way of securely transmitting information between parties. Because JWTs can be signed—for example, using public/private key pairs — you can be sure the senders are who they say they are. Additionally, as the signature is calculated using the header and the payload, you can also verify that the content hasn't been tampered with.
Generate the certificate for JWT with OpenSSL, in this case self-signed is enough:
$ openssl genrsa -out private.pem 4096
Generate public key from earlier generated private key for if pem-jwk needs it, it isn't needed otherwise
Now we finally get to the part where we generate the JWK. The final result is a JSON file which contains the public key from earlier created certificate in JWK-format so that the service can accept the signed tokens.
Convert the PEM to JWK format with e.g. pem-jwk or with pem_to_jwks.py. The key is in pkcs12 format. The values for public key's values n and e are extracted from private key with following commands. jq part extracts the public parts and excludes the private parts.
DevOps 2020 Online was held 21.4. and 22.4.2020 and the first day talked about Cloud & Transformation and the second was 5G DevOps Seminar. Here are some quick notes from the talks I found the most interesting. The talk recordings are available from the conference site.
How to improve your DevOps capability in 2020
Marko Klemetti from Eficode presented three actions you can take to improve your DevOps capabilities. It looked at current DevOps trends against organizations on different maturity levels and gave ideas how you can improve tooling, culture and processes.
Build the production pipeline around your business targets.
Automation build bridges until you have self-organized teams.
Adopt a DevOps platform. Aim for self-service.
Invest in a Design System and testing in natural language:
brings people in organization together.
Testing is the common language between stakeholders.
You can have discussion over the test cases: automated quality assurance from stakeholders.
Validate business hypothesis in production:
Enable canary releasing to lower the deployment barrier.
You cannot improve what you don't see. Make your pipeline data-driven.
Practical DevSecOps Using Security Instrumentation
Jeff Williams from Contrast Security talked about how we need a new approach to security that doesn't slow development or hamper innovation. He shows how you can ensure software security from the "inside out" by leveraging the power of software instrumentation. It establishes a safe and powerful way for development, security, and operations teams to collaborate.
Security testing with instrumentation:
Add matchers to catch potentially vulnerable code and report rule violations when it happens, like using unparameterized SQL. Similar what static code analysis does.
Making security observable with instrumentation:
Check for e.g. access control for methods
Preventing exploits with instrumentation:
Check that command isn't run outside of scope
The examples were written with Java but the security checks should be implementable also on other platforms.
Their AppSec platform's Community Edition is free to try out but only for Java and .Net.
Open Culture: The key to unlocking DevOps success
Chris Baynham-Hughes from RedHat talked how blockers for DevOps in most organisations are people and process based rather than a lack of tooling. Addressing issues relating to culture and practice are key to breaking down organisational silos, shortening feedback loops and reducing the time to market.
Three layers required for effective transformation:
People and culture
Scaling DevSecOps to integrate security tooling for 100+ deployments per day
Rasmus Selsmark from Unity talked how Unity integrates security tooling better into the deployment process. Best practices for securing your deployments involve running security scanning tools as early as possible during your CI/CD pipeline, not as an isolated step after service has been deployed to production. The session covered best security practices for securing build and deployment pipeline with examples and tooling.
Standardized CI/CD pipeline, used to deploy 200+ microservices to Kubernetes.
Make CI feedback actionable for teams, like generating notifications directly in PRs.
When to trigger deployment
PR with at least one approver.
No direct pushes to master branch.
Only CI/CD pipeline has staging and production deployment access.
Secrets management using Vault. Secrets separate from codebase, write-only for devs, only vault-fetcher can read. Values replaced during container startup, no environment variables passed outside to container.
Container runtime security with Falco: identify security issues in containers running in production.
Data-driven DevOps: The Key to Improving Speed & Scale
Kohsuke Kawaguchi, Creator of Jenkins, from Launchable talked how some organizations are more successful with DevOps than others and where those differences seem to be made. One is around data (insight) and another is around how they leverage "economy of scale".
CFO: why do we spend so much on AWS?
Visibility into cost at project level
Make developers aware of the trade-off they are making: Build time vs. Annual cost
Small: 15 mins / $1000; medium: 10 mins / $2000; large: 8 mins / $3000
Whose problem is it?
A build failed: Who should be notified first?
Regular expression pattern matching
Improving software delivery process isn't get prioritized:
Data (& story) helps your boss see the problem you see
Data helps you apply effort to the right place
Data helps you show the impact of your work
Cut the cost & time of the software delivery process
Predictive test selection
You wait 1 hour for CI to clear your pull request?
Your integration tests only run nightly?
Reordering tests: Reducing time to first failure (TTFF)
Creating an adaptive run: Run a subset of your tests?
Deployment risk prediction: Can we flag risky deployments beforehand?
Learn from previous deployments to train the model
Automation is table stake
Using data from automation to drive progress isn't
Lots of low hanging fruits there
Unicorns are using "big data" effectively
How can the rest of us get there?
Moving 100,000 engineers to DevOps on the public cloud
Sam Guckenheimer from Microsoft talked how Microsoft transformed to using Azure DevOps and GitHub with a globally distributed 24x7x365 service on the public cloud. The session covered organizational and engineering practices in five areas.
Connect our customers directly and measure:
Direct feedback in product, visible on public site, and captured in backlog
Develop personal Connection and cadence
For top customers, have a "Champ" which maintain: Regular personal contact, long-term relationship, understanding customer desires
Definition of done: live in production, collecting telemetry that examines the hypothesis which motivated the deployment
You Build It, You Love It
Live site incidents
Communicate externally and internally
Gather data for repair items & mitigate for customers
Record every action
Use repair items to prevent recurrence
Align outcomes, not outputs
You get what you measure (don't measure what you don't want)
Customer usage: acquisition, retention, engagement, etc.
Pipeline throughput: time to build, test, deploy, improve, failed and flaky automation, etc.
Service reliability: time to detect, communicate, mitigate; which customers affected, SLA per customer, etc.
"Don't" measure: original estimate, completed hours, lines of code, burndown, velocity, code coverage, bugs found, etc.
Good metrics are leading indicators
Trailing indicators: revenue, work accomplished, bugs found
Leading indicators: change in monthly growth rate of adoption, change in performance, change in time to learn, change in frequency of incidents
Measure outcomes not outputs
Get clean, stay clean
Progress follows a J-curve
Getting clean is highly manual
Staying clean requires dependable automation
Make technical debt visible on every team's dashboard
Your aim won't be perfect: Control the impact radius
Deploy one ring at a time: canary, data centers with small user counts, highest latency, th rest.
Feature flags control the access to new work: setting is per user within organization
If you've used Kubernetes you might have come across Ingress which manages external access to services in a cluster, typically HTTP. When running with GKE the "default" is GLBC which is a "load balancer controller that manages external loadbalancers configured through the Kubernetes Ingress API". It's easy to use but doesn't let you to to customize it much. The alternative is to use for example NGINX Ingress Controller which is more down to earth. Here are my notes of configuring ingress-nginx with cert manager on Google Cloud Kubernetes Engine.
Then if you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml.
Now you're ready to create mandatory resources, use kubectl apply and the -f flag to specify the manifest file hosted on GitHub:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml
$ kubectl apply -f ingress-nginx_mandatory.yaml
Create the LoadBalancer Service:
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/provider/cloud-generic.yaml
$ kubectl get svc --namespace=ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx LoadBalancer 10.10.10.1 184.108.40.206 80:30598/TCP,443:31334/TCP 40s
$ kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch
NAMESPACE NAME READY STATUS RESTARTS AGE
ingress-nginx nginx-ingress-controller-6cb75cf6dd-f4cx7 1/1 Running 0 2m17s
Configure proxy settings
In some situations the payload for ingress-nginx might be too large and you have to increase it. Add the "nginx.ingress.kubernetes.io/proxy-body-size" annotation to your ingress metadata with value you need. 0 to not limit the body size.
Create a production certificate ClusterIssuer, prod_issuer.yaml:
# The ACME server URL
# Email address used for ACME registration
# Name of a secret used to store the ACME account private key
# Enable the HTTP-01 challenge provider
Apply production issuer using kubectl:
kubectl create -f prod_issuer.yaml
Update ingress.yml to use "letsencrypt-prod" issuer:
Software development projects come usually with lots of dependencies and keeping them up to date can be burdensome if done manually. Fortunately there are tools to help you. For Node.js projects there are e.g. npm-check and npm-check-updates and for Maven projects there are OWASP/Dependency-Check and Versions Maven plugins. Here's a short introduction how to setup your Maven project to automatically check dependencies for vulnerabilities and if there's outdated dependencies.
OWASP dependency-check is an open source solution the OWASP Top 10 2013 entry: "A9 - Using Components with Known Vulnerabilities".
Dependency-check can currently be used to scan Java and .NET applications to identify the use of known vulnerable components. The dependency-check plugin is, by default, tied to the verify or site phase depending on if it is configured as a build or reporting plugin.
The example below can be executed using mvn verify:
The example fails the build for CVSS greater than or equal to 8 and skips scanning the provided and runtime scoped dependencies.
Versions Maven Plugin
The Versions Maven Plugin is the de facto standard way to manage versions of artifacts in a project's POM. From high-level comparisons between remote repositories up to low-level timestamp-locking for SNAPSHOT versions, its massive list of goals allows us to take care of every aspect of our projects involving dependencies.
The example configuration of versions-maven-plugin:
You could use goals that modify the pom.xml as described in the usage documentation but often it's easier to check versions manually as you might not be able to update all of the suggested dependencies.
The display-dependency-updates goal will check all the dependencies used in your project and display a list of those dependencies with newer versions available.
Check new dependencies with:
Check new plugin versions with:
Using OWASP/Dependency-Check in your Continuous Integration build
flow to automatically check dependencies for vulnerabilities and running
periodically Versions Maven Plugin to check if there are outdated
dependencies helps you to keep your project up to date and secure. Small
but important things to remember while developing and maintaining a