End-to-end Testing Mobile Apps with Maestro

End-to-end testing (E2E testing) is a methodology that evaluates applications from start to finish, simulating real user interactions to ensure smooth, bug-free experiences. As a critical component of software development, testing ensures applications meet quality standards and user expectations.

Let's explore different testing approaches and why Maestro stands out for mobile apps.

Types of automated software testing

Different testing strategies serve different purposes in your development workflow:

Unit testing

Unit tests focus on individual modules, components, or functions—ensuring they work correctly in isolation. These low-level tests are relatively straightforward to automate since they don't require the context of an entire application.

Integration testing

Rather than testing elements in isolation, integration testing examines how components interact. For example, testing whether a product correctly appears in a cart component after adding it. These tests validate that individual units work properly when combined, though they introduce more complexity due to increased dependencies.

End-to-end testing

E2E testing simulates complete user flows to verify system behavior meets expectations. It complements unit and integration testing by covering potential gaps in coverage. E2E tests can validate critical paths like user registration, form submission, or password changes.

While E2E testing is the most complex approach due to its involvement with the entire application system, it provides the most realistic assessment of real-world usage.

It's important to note that automated testing doesn't replace manual QA—functional tests simply provide tools for faster, more accurate testing.

End-to-end Testing Frameworks

Testing frameworks establish guidelines for writing, organizing, and executing tests. They provide structure for test management, offer integration with CI/CD pipelines, and deliver detailed reporting to help identify failures.

Each framework has its own approach, with some specializing in specific application types, like mobile. Let's look at some popular options for mobile testing:

Appium

Appium is a well-established open-source framework for E2E mobile testing that supports both cross-platform and native applications. While powerful and versatile with support for multiple programming languages, it can be challenging to set up due to platform-specific requirements and dependencies.

Detox

Detox is a gray-box testing framework designed specifically for React Native applications. It requires less knowledge of system internals compared to white-box testing, making it more accessible. While it technically supports native apps, it was designed with React Native in mind.

Maestro

Maestro offers a more intuitive approach than its competitors. Tests are written in YAML rather than a programming language, making them easier to understand and implement. It also has minimal setup requirements, as we'll explore next.

There are also platform-specific frameworks like XCUITest (iOS) and Espresso (Android) that focus exclusively on their respective native platforms.

Why we choose Maestro for E2E Testing?

While you should evaluate all options based on your specific needs, Maestro offers compelling advantages:

1. Maestro is easy to learn also for non-developers
2. Maestro is easy to setup with little external dependencies
3. Maestro supports both iOS and Android and integrates easily with cross-platform frameworks such as React Native

Here is a quick overview of what a simple test of the Wikipedia app looks like:

For comparison, here's what a simple test looks like in Detox:

describe('Wikipedia Test', () => {
  beforeAll(async () => {
    await device.launchApp(); // Launch the app
  });

  it('Search for "Software testing', async () => {
    
    await element(by.id('search_bar')).tap();
    await element(by.id('search_input')).typeText('E2E testing');
    await expect(element(by.text('Software testing'))).toBeVisible();
    await element(by.text('Software testing')).tap();
  });
});

This Detox implementation is less intuitive and requires more code for the same result. Notice how we need to explicitly wait for elements to become visible before proceeding—something Maestro handles automatically with intelligent waiting.

Detox also requires initial setup, while Maestro only needs its CLI installed to get started.

Handling Permissions

Native permissions present another challenge in mobile testing. Detox requires configuration in its config file, which often isn't sufficient for simulating permission triggers. Solutions differ between iOS and Android, adding complexity.

Maestro, however, handles native permissions elegantly with a single line:

permissions: { all: allow }

Working with System UI

Maestro also excels at handling external elements like WebViews or native screens that can be troublesome in Detox. Since Maestro detects content on screen regardless of source, it maintains a consistent approach across different UI contexts.

Robust Debugging

When tests fail, Maestro provides clear diagnostics. For example, if a search for "E2E testing" in Wikipedia doesn't return an expected result for "HybridHeroes," Maestro automatically generates logs and captures a screenshot of the application state at failure time, making it easy to identify the issue:

Maestro Studio: Visual Test Building

Maestro Studio, a GUI application for building test flows, further simplifies the process. As you navigate your app in the simulator, Maestro Studio reflects your actions and allows you to generate commands by clicking on elements. You can test commands immediately, verifying each step in real-time instead of writing the entire test before debugging.

Once your flow is created, you can export it directly to your test file with a single click.

Integrate Maestro in your CI

Integrating Maestro into your CI/CD pipeline can be approached in several ways, each with its own benefits:

The most flexible approach is installing the Maestro CLI directly in your CI environment. This method works with any CI platform and gives you complete control over the configuration, but requires:

• Setting up Android emulators and/or iOS simulators in your CI environment
• Configuring build processes for your app
• A macOS runner if you need iOS testing capabilities

For teams wanting to avoid the additional complexity, Maestro Cloud offers a streamlined alternative.

Using their CLI you can run your tests on Maestro's cloud devices, eliminating the need for local emulator configuration. It requires a Maestro Cloud account and API key.

Maestro Cloud also provides dedicated integrations for popular CI platforms:

• Github Actions
• Bitrise
• Bitbucket
• CircleCI

The best approach depends on your specific needs, balancing factors like complexity, test volume, security requirements, and budget.

Conclusion

We selected Maestro for E2E testing in our mobile applications for several reasons:

• It has a significantly lower learning curve than alternatives like Detox, enabling faster onboarding and flow creation.
• Tools like Maestro Studio make test creation intuitive while still producing efficient, robust tests.
• It handles challenging aspects like native permissions seamlessly.
• It reduces the time and resources needed to implement E2E testing, making it more cost-effective.

Whether you choose Maestro or another framework, E2E testing remains a crucial component of application development. Properly implemented E2E tests ensure your users' most common scenarios work as expected, ultimately delivering a better product.