Home | Blog

Optimizing slow Unit Tests

Cover Image for Optimizing slow Unit Tests
Matija Kovacek
Matija Kovacek

Understanding the motivation behind optimizing slow unit tests is crucial. We'll explore the challenges faced by Client XYZ, why we wanted to fix them, and the good things that happened afterward. Expect insights into how faster tests can boost productivity and project success.


Why?

  • 90% reduced project build time
  • 11x faster project build speed
  • 13x faster unit test execution

Before & After

Imagine how this affects big development team where each team memeber builds the project several times per day.

Consequences:

  • Slower development
  • Developers locally skip tests execution
  • Failing deployments to higher environments

Some Facts Why I did it

  • I like unit/integration testing topics
  • I like to optimize things
  • I prefer Integration over Unit tests
  • Test Diamond > Test Pyramid
  • I don’t like mocks

Test Diamond


Project XYZ

  • AEM Multi-tenant Project
  • Some tenants already live, some in development
  • Joined at late stage of the project
  • Code coverage around 60%
  • Unit/Integration tests not in the best shape
  • 343 test classes
  • 1083 test methods

How to not write Unit Tests

  • Don’t write a test that doesn’t make sense
  • Don’t test if Mockito when() then() methods works correctly
  • Don’t overuse mocking
  • Don’t import and use not needed objects and their methods

Test Example 1

Test Example 1

  • Don’t test directly if POJO getters and setters work correctly
  • Don’t write tests just to achieve numbers

Test Example 2

  • Don’t call external endpoints in unit tests (mock their responses, e.g use Wiremock)

Test Example 3


Optimization Strategy

Since some projects are already live and multiple development teams are working with same codebase, optimization startegy was simple:

  • Only quick wins
  • Minimal code changes
  • Easy code changes
  • Don’t change test code logic
  • Don't change implementation code logic

1. Optimization - Parallel Test Execution for Junit 5

One of the first things that would come to everyone's minds is, let's introduce some concurrency and parallelism. This should speed up test, but not solve root cause, and this is just fine for our optimization strategy.

Parallel Test Execution for Junit 5

What happened:

  • Some tests were failing due not isolated test context between test cases
  • Test code change was required
  • Execution time didn’t decrease

Optimization

  • Failed

2. Optimization - Easy Code Changes

Changes:

  • Removed not needed AemContext object and AemContextExtension

Removed not needed AemContext object and AemContextExtension

  • Removed not needed MockitoExtension and not needed Mockito init(), open() initialization methods

Removed not needed MockitoExtension and not needed Mockito init(), open() initialization methods

  • Replaced @Mock annotations with mock(ClassName.class) method

Replaced @Mock annotations with mock(ClassName.class) method

  • Removed not need mocked objects and service registrations

Removed not need mocked objects and service registrations

  • Converted @BeforeEach to @BeforeAll

Converted @BeforeEach to @BeforeAll

  • Set appropriate ResourceResolverType in AemContext

Set appropriate ResourceResolverType in AemContext

  • Mock endpoint calls and responses

Mock endpoint calls and responses

Mock endpoint calls and responses 2

Optimization

  • Improved test speed execution from ~15 min to ~5 min
  • 3x faster unit test execution

3. Optimization – Maven Surefire Plugin & Test Logging Library

Each test was gradually taking more time to execute, meaning that we had potential memory leaks and big CPU activity. We found out that Test logging library is logging all logs (trace) in memory.

Maven Surefire Plugin & Test Logging Library

Changes:

  • Replaced uk.org.lidalia:slf4j-test with org.slf4j:slf4j-simple
  • Turned off Test Logging

Optimization

  • Improved test speed execution from ~5 min to ~1:30 min
  • 3.3x faster unit test execution

4. Optimization – Maven Surefire Plugin & Forked Test Execution

Since the first try with concurrency and parallelism optimization failed, we were a bit stubborn and started investigating further so we found out that we could execute tests concurrently by configuring the Maven Surefire Plugin.

Maven Surefire Plugin & Forked Test Execution

The parameter forkCount defines the maximum number of JVM processes that maven-surefire-plugin will spawn concurrently to execute the tests.

Changes:

  • Set forkCount=2
  • Note: bigger values didn’t bring bigger optimization, potentially can cause build crash on the machine with low resources

Optimization

  • Improved test speed execution from ~1:30 min to ~1 min
  • 1.5x faster unit test execution

5. Bonus tip and final optimization – Maven Daemon

Long story short, Maven Deamon builds maven modules in parallel. If you want to read more about Speeding up the Maven Build time with Maven Daemon, check my blog post Speed up the AEM Build Time.

Optimization

  • 90% reduced build time
  • 11x faster project build speed
  • 13x faster unit test execution

Before & After


Conclusion

  • Remove uk.org.lidalia:slf4j-test and turn off or decrease log level in maven-surefire-plugin
    • From ~15 min to ~ 2 min
  • Set forkCount in maven-surefire-plugin for Parallel Test execution
    • From ~15 min to ~ 6:40 min
  • Treat your tests and write them as an implementation code (clean, quality, performant)
    • From ~15 min to ~ 5 min
  • Use additional tools to speed up the development process
    • Maven Daemon
    • aemsync

If you want to read more about Unit/Integration Testing, check my blog post Test Beahviour not Implementation.