Optimizing slow Unit Tests
Understanding the motivation behind optimizing slow unit tests is crucial. We'll explore the challenges faced by Client XYZ, why we wanted to fix them, and the good things that happened afterward. Expect insights into how faster tests can boost productivity and project success.
Why?
- 90% reduced project build time
- 11x faster project build speed
- 13x faster unit test execution
Imagine how this affects a big development team where each team member builds the project several times per day.
Consequences:
- Slower development
- Developers locally skip test execution
- Failing deployments to higher environments
Some Facts Why I did it
- I like unit/integration testing topics
- I like to optimize things
- I prefer Integration over Unit tests
- Test Diamond > Test Pyramid
- I don’t like mocks
Project XYZ
- AEM Multi-tenant Project
- Some tenants already live, some in development
- Joined at late stage of the project
- Code coverage around 60%
- Unit/Integration tests not in the best shape
- 343 test classes
- 1083 test methods
How to not write Unit Tests
- Don’t write a test that doesn’t make sense
- Don’t test if Mockito when() then() methods works correctly
- Don’t overuse mocking
- Don’t import and use not needed objects and their methods
- Don’t test directly if POJO getters and setters work correctly
- Don’t write tests just to achieve numbers
- Don’t call external endpoints in unit tests (mock their responses, e.g use Wiremock)
Optimization Strategy
Since some projects are already live and multiple development teams are working with the same codebase, the optimization strategy was simple:
- Only quick wins
- Minimal code changes
- Easy code changes
- Don’t change test code logic
- Don't change implementation code logic
1. Optimization - Parallel Test Execution for Junit 5
One of the first things that would come to everyone's minds is, let's introduce some concurrency and parallelism. This should speed up the test, but not solve the root cause, and this is just fine for our optimization strategy.
What happened:
- Some tests were failing due to not isolated test context between test cases
- Test code change was required
- Execution time didn’t decrease
Optimization
- Failed
2. Optimization - Easy Code Changes
Changes:
- Removed not needed AemContext object and AemContextExtension
- Removed not needed MockitoExtension and not needed Mockito init(), open() initialization methods
- Replaced @Mock annotations with mock(ClassName.class) method
- Removed not need mocked objects and service registrations
- Converted @BeforeEach to @BeforeAll
- Set appropriate ResourceResolverType in AemContext
- Mock endpoint calls and responses
Optimization
- Improved test speed execution from ~15 min to ~5 min
- 3x faster unit test execution
3. Optimization – Maven Surefire Plugin & Test Logging Library
Each test was gradually taking more time to execute, meaning that we had potential memory leaks and big CPU activity. We found out that Test logging library is logging all logs (trace) in memory.
Changes:
- Replaced uk.org.lidalia:slf4j-test with org.slf4j:slf4j-simple
- Turned off Test Logging
Optimization
- Improved test speed execution from ~5 min to ~1:30 min
- 3.3x faster unit test execution
4. Optimization – Maven Surefire Plugin & Forked Test Execution
Since the first try with concurrency and parallelism optimization failed, we were a bit stubborn and started investigating further so we found out that we could execute tests concurrently by configuring the Maven Surefire Plugin.
The parameter forkCount defines the maximum number of JVM processes that maven-surefire-plugin will spawn concurrently to execute the tests.
Changes:
- Set forkCount=2
- Note: bigger values didn’t bring bigger optimization, potentially can cause build crash on the machine with low resources
Optimization
- Improved test speed execution from ~1:30 min to ~1 min
- 1.5x faster unit test execution
5. Bonus tip and final optimization – Maven Daemon
Long story short, Maven Deamon builds maven modules in parallel. If you want to read more about Speeding up the Maven Build time with Maven Daemon, check my blog post Speed up the AEM Build Time.
Optimization
- 90% reduced build time
- 11x faster project build speed
- 13x faster unit test execution
Conclusion
- Remove uk.org.lidalia:slf4j-test and turn off or decrease the log level in maven-surefire-plugin
- From ~15 min to ~ 2 min
- Set forkCount in maven-surefire-plugin for Parallel Test execution
- From ~15 min to ~ 6:40 min
- Treat your tests and write them as an implementation code (clean, quality, performant)
- From ~15 min to ~ 5 min
- Use additional tools to speed up the development process
- Maven Daemon
- aemsync
If you want to read more about Unit/Integration Testing, check my blog post Test Beahviour not Implementation.
