Running Detox end-to-end tests on CI

Now that you've added end-to-end tests with Detox to your React Native application from my previous blog post it's time to automate running the tests and add them to continuous integration (CI) pipeline. Detox has documentation for running it on some Continuous Integration services but it's somewhat lacking and outdated. This post will show how to automate E2E testing on GitHub Actions.

Detox E2E tests on GitHub Actions

GitHub Actions makes it easy to automate software workflows and basically provides some capabilities than any common CI/CD service like GitLab, CircleCI, Bitrise etc. At work I used GitLab CI/CD to run our e2e tests so adding automation for my personal Hailakka project on GitHub Actions was a nice exercise. You can see the workflow file from the repository: ios.yml

Creating the workflow for GitHub Actions was quite straightforward by modernizing the workflow shown in the article I found and there's more modern blog post also which covers also Azure Pipelines.

GitHub Actions provides a macOS runner so setting up things were as simple as writing the steps you'd do manually for running the Detox tests and translating those steps to workflow. Adding a workflow for iOS and getting tests to run was easy. The result was simple pipeline on repository "push" action in pull request and master branches.

GitHub Actions workflows

There is a workflow for Hailakka on Android but I managed it to get running only on API level 29 (or lower) on the GitHub Actions although locally it run on API level 31. Somehow the newer API didn't create any UI (empty images) for Detox to click through. Anyways the Android tests seemed more flaky even when running on local machine.

Detox tests for my simple newsreader application runs on iOS runner quickly after the app was build. The total runtime for the workflow run is around 30 minutes. As I've set the artifacts to be generated only when the tests fail the output is just a succeeded job. Maybe I should look into partial visual regression testing πŸ€”

Detox tests on GitHub Actions job

You can add a badge to your readme of the workflow status βœ…

Workflow run for iOS passes

Automated End-to-End testing React Native apps with Detox

Everyone knows the importance of testing in software development projects so lets jump directly to the topic of how to use Detox for end-to-end testing React Native applications. It's similar to how you would do end-to-end testing React applications with Cypress like I wrote previously. Testing React Native applications needs a bit more setup especially when running on CI/CD but the benefits of having comprehensive tests is great. This article focuses on using Detox and leaves the setup stuff for the official documentation.

Detox for end-to-end testing

End-to-end testing is widely performed in the web applications using frameworks like Cypress but with mobile applications it's not as common. There are essentially couple of frameworks for end-to-end testing mobile applications: Detox and Appium.

This article covers how to use Detox which uses grey-box-testing by monitoring the state of the app to tackle the problem of flaky tests. The main difference between Appium and Detox is that Appium uses black-box-testing, meaning that it doesn't monitor the internal state of the app. There's a good article of comparing Detox and Appium with example projects.

I'll use my personal Hailakka project as an example of setting up Detox for end-to-end testing and for visual regression testing. The app is based on my native Highkara news reader but done in React Native and is still missing features like localization.

Hailakka on Android and iOS

Setup Detox for React Native project

Starting with Detox e2e and React Native gets you through setting Detox up in your project, one step at a time both for Android and iOS.

Detox provides an example project for React Native which shows a starting point for your own e2e tests. It's good to note that support for Mocha is dropped on upcoming Detox 20.

Now you just need to write simple e2e tests for starters and build your app and run Detox tests. Detox has good documentation to get you through the steps and has some troubleshooting tips you might come across especially with Android.

Detox with Expo

Using Detox with Expo projects should work quite nicely without any helper libraries (although some older blog posts suggests so). You can even run e2e tests on EAS builds. For practice I added example of using Detox to my personal Hailakka project with managed workflow.

The changes compared to non-Expo project I followed the Detox on EAS build documentation and added the @config-plugins/detoxΒ for a managed project. Also to get the native projects for Detox I ran the npx expo prebuild script which practically configured both iOS and especially Android projects to work with Detox!. You can see my detox.config.js from my GitHub.

Running Detox end-to-end tests

With iOS related tests on simulator I got everything working pretty smoothly but on Android I had some problems to troubleshoot with the help of Detox documentation. Using the FORCE_BUNDLING=true environment variable helped to get rid of the Metro bundler when running Detox as Android emulator had problems connecting to it. Before that I got "unable to load script from assets index.android.bundle" error and I had to use adb reverse tcp:8081 tcp:8081 for proxying the connection.

For running the Detox tests I created the following npm scripts.

"e2e:ios": "npm run e2e:ios:build && npm run e2e:ios:run",
"e2e:ios:build": "detox build --configuration ios.development",
"e2e:ios:run": "detox test --configuration ios.development",
"e2e:android": "npm run e2e:android:build && npm run e2e:android:run",
"e2e:android:build": "detox build --configuration android.development",
"e2e:android:run": "detox test --configuration android.development",

Note on running on Android: Use system images without Google services (AOSP emulators). Detox "strongly recommend to strictly use this flavor of emulators for running automation/Detox tests.".

Also remember to change the Java SDK to 11. You can do it with e.g. asdf.

brew reinstall asdf
asdf plugin add java
asdf list-all java
asdf install java zulu-11.60.19
asdf global java zulu-11.60.19

If you need to debug your Detox tests on Android and run Metro bundler on the side you might need to start the metro builder (react-native start) manually. Also when using concurrently the arguments need to be passed through to desired script, e.g. "e2e:android:run": "concurrently --passthrough-arguments 'npx react-native start' 'npm run e2e:android:test -- {@}' --",.

Running same tests on iOS and Android have some differences and it helps to use "--loglevel=verbose" parameter to debug component hierarchy. For example opening the sidebar navigation from icon button failed on Android but worked on iOS. The issue was that I was using Android system image with Google's services and not the AOSP version.

Setting locale for Detox tests

Detox provides launchApp command which allows you to accept permissions and set language and locale.

For example in beforeAll you can set the locale for and accept the permissions for notifications as we can't accept it when the app runs and the modal is shown.

beforeAll(async () => {
    await device.launchApp({
        languageAndLocale: { language: 'en', locale: 'en-US' },
        newInstance: true,
        permissions: { notifications: 'YES' }, // This will cause the app to terminate before permissions are applied.
    });
});

But unfortunately setting the languageAndLocale works only on iOS. For Android you need a lot more steps to achieve that. Fortunately I'm not alone and "Changing Android 10 or higher device Locale programatically" covers just what I needed. You can use the Appium Settings application and get it to automatically installed to the emulator by using "utilBinaryPaths" as shown in Detox documentation

You can run the needed adb shell commands with execSync when setting up the tests:

import { execSync } from 'child_process';

// Set permissions for Settings app for changing locale
execSync('adb shell pm grant io.appium.settings android.permission.CHANGE_CONFIGURATION');
// Set lang and country
execSync(`adb shell am broadcast -a io.appium.settings.locale -n io.appium.settings/.receivers.LocaleSettingReceiver --es lang fi --es country FI`

Tips for writing Detox tests

People at Siili have written about Detox testing native mobile apps which provides good practices for testing.

  • Start each scenario from a predictable application state. Using beforeAll, beforeEach, await device.reloadReactNative() methods can help with that
  • Add testIDs to the elements you are going to use in the tests,
  • Keep all locators in one place (changing them in the future will not cost too much time)
  • Create helpers method to make your code more legible and easily maintainable

Visual regression testing React Native apps with Detox

Now we have achieved end-to-end tests with Detox but we can do more. By harnessing Detox for visual regression testing we can "verify the proper visual structure and layout of elements appearing on the device's screen". Detox supports taking screenshots, which can be used for visual regression testing purposes as they describe in their documentation.

  • Taking a screenshot, once, and manually verifying it, visually.
  • Storing it as an e2e-test asset (i.e. the snapshot).
  • Using it as the point-of-reference for comparison against screenshots taken in consequent tests, from that point on.

They write that "more practical way of doing this, is by utilizing more advanced 3rd-party image snapshotting & comparison tools such as Applitools." And here is were we come into the good article of how to do Visual regression testing with Detox. In short, we use Jest Image Snapshot and recommendations using SSIM Comparison.

jest-image-snapshot generates an image which shows thee baseline and how it differs with current state with red color.

jest-image-snapshot baseline diff with Detox

To aid writing tests, use some convenience methods by extending Jest expect and automatically taking a screenshot if the method is invoked; consider the platform and device name when doing the comparisons.

Helper methods in setup.ts extending Jest expect, adapted to TypeScript from Visual regression testing React Native apps with Detox and Jest:

import { device } from 'detox';
import fs from 'fs';
import { configureToMatchImageSnapshot } from 'jest-image-snapshot';
import path from 'path';
const jestExpect = (global as any).expect;

const kebabCase = (str: string) =>
    str.match(/[A-Z]{2,}(?=[A-Z][a-z]+[0-9]*|\b)|[A-Z]?[a-z]+[0-9]*|[A-Z]|[0-9]+/g)!.join('-').toLowerCase();

const toMatchImage = configureToMatchImageSnapshot({
    comparisonMethod: 'ssim',
    failureThreshold: 0.01, // fail if there is more than a 1% difference
    failureThresholdType: 'percent',
});

jestExpect.extend({ toMatchImage });

jestExpect.extend({
    async toMatchImageSnapshot(
        // a matcher is a method, it has access to Jest context on `this`
        this: jest.MatcherContext,
        screenName: string
    ) {
        const { name } = device;
        const deviceName = name.split(' ').slice(1).join('').replace('(','').replace(')', '');
        const language = languageAndLocale();
        const SNAPSHOTS_DIR = `__image_snapshots__/${language.code}/${deviceName}`;
        const { testPath } = this;
        const customSnapshotsDir = path.join(path.dirname(testPath || ''),
SNAPSHOTS_DIR);
        const customSnapshotIdentifier = kebabCase(`${screenName}`);
        const tempPath = await device.takeScreenshot(screenName);
        const image = fs.readFileSync(tempPath);
        jestExpect(image).toMatchImage({ customSnapshotIdentifier, customSnapshotsDir });
        return { message: () => 'screenshot matches', pass: true };
    },
});

Writing an expectation for an image comparison then becomes as simple as (HomeScreen.e2e.ts):

import { by, element, expect } from 'detox';

const jestExpect = (global as any).expect;

describe('Home Screen', () => {
  it('should show section header', async () => {
    await expect(element(by.id('navigation-container-view'))).toBeVisible();

    await jestExpect('Home Screen').toMatchImageSnapshot();
  });
});

You should also put the device into demo mode by freezing the irrelevant, volatile elements (like time, network information etc.). The helper.ts might look like:

import { execSync } from 'child_process';
import { device } from 'detox';

export const setDemoMode = async () => {
    if (device.getPlatform() === 'ios') {
        await (device as any).setStatusBar({
            batteryLevel: '100',
            batteryState: 'charged',
            cellularBars: '4',
            cellularMode: 'active',
            dataNetwork: 'wifi',
            time: '9:42',
            wifiBars: '3',
        });
    } else {
        // enter demo mode
        execSync('adb shell settings put global sysui_demo_allowed 1');
        // display time 12:00
        execSync('adb shell am broadcast -a com.android.systemui.demo -e command clock -e hhmm 1200');
        // Display full mobile data with 4g type and no wifi
        execSync(
            'adb shell am broadcast -a com.android.systemui.demo -e command network -e mobile show -e level 4 -e datatype 4g -e wifi true'
        );
        // Hide notifications
        execSync('adb shell am broadcast -a com.android.systemui.demo -e command notifications -e visible false');
        // Show full battery but not in charging state
        execSync(
            'adb shell am broadcast -a com.android.systemui.demo -e command battery -e plugged false -e level 100'
        );
    }
};

And as your application changes you may want to add some script to update the snapshots to you package.json

"e2e:ios:refresh": "npm run e2e:ios:build && npm run e2e:ios:run -- --updateSnapshot",
"e2e:android:refresh": "npm run e2e:android:build && npm run e2e:android:run -- --updateSnapshot",

Now you can just enjoy your end-to-end tests taking care that the application looks and works like you want it to after changes. And wait for React Native Owl to mature for getting out-of-the box visual regression testing.

The visual regression testing part of using Detox was for my case more of a theoretical practice than a real use case. As the visual regression testing compares current state to baseline images but with my personal is news reader with changing news it isn't feasible at it's current state.