Nowadays, performance testingandcontinuous integrationare necessary in the software development process, and at first glance, the two concepts may seem unrelated.
This article examines the benefits of having performance testing in the continuous integration pipeline and see what improvements it creates in the software production process when we run our performance
Table of Contents
Why Performance Testing Matters?
In general, testing software before releasing it to the public is a standard that all people in the software industry agree about. Among all kinds of tests,
Performance testing is important because it guarantees that applications not only work correctly but also meet non-functional requirements such as speed, scalability, and reliability.
It has an important role in providing a good experience for our end users, optimizing costs, and meeting commitments. Here are some of the main reasons that show why performance testing matters:
- ResponsivenessandConsistency: Performance testing ensures that the application behaves consistently and is fast under various conditions, providing a smooth experience in different loads.
- Finding issues: Performance testing helps us to identify the application’s bottlenecks and provides valuable insights into its scalability.
- Resource Optimization: By identifying inefficiencies, performance testing helps us to optimize resource usage (e.g., CPU, memory, network).
- Compliance: Performance testing ensures thatSLAsare met, avoiding potential penalties and maintaining customer trust.
Where is performance testing in the test pyramid?
The traditionalTest Pyramidprimarily focused onfunctional testingat different levels:
- Unit Tests: These tests are designed to quickly assess individual components or functions in isolation.
- Integration Tests: The purpose of these tests is to check how different components or modules interact with each other and ensure that they are working as expected.
- UI/End-to-End Tests: These tests are more comprehensive, covering a wide range of scenarios, simulating user interactions, and testing the entire system.
Performance Testing is anon-functional testingtype focused on how the system behaves under specific conditions, such asload,stress, andscalabilityscenarios. Because of its nature, performance testing doesn’t fit into a specific layer of the test pyramid but is generally considered an“extra”layer outside of the traditional pyramid structure.
Similar toend-to-end tests, Performance tests usually operate at the system level, but they’re concerned with metrics likeresponse time,throughput,resource usage, andstabilityunder load rather than functionality.
Performance testing is often done after functional tests to ensure the system meets performance requirements rather than verifying correctness.
Importance of Continous Integration
Before discussing the conjunction of performance testing and CI, let’s learn more about the importance of continuous integration. CI is a fundamental part of modern development practices and brings significant benefits to software development teams. By leveraging the CI, developers frequently commit their code into a shared repository, often multiple times a day. Each commit is automatically verified by running tests to detect integration errors as quickly as possible.
CI offers numerous benefits, including early error detection, improved code quality, faster feedback, and enhanced collaboration. By automating the integration process, CI reduces risks, increases productivity, and enables faster, more reliable releases.
Benefits of integrating performance testing in CI
Integratingperformance testinginto theCI pipelineoffers numerous benefits, including:
- Early detection of performance issues
- Consistent Performance monitoring
- Faster feedback loop
- Increased confidence in code changes and avoiding regressions
- And more
Is CI a good place for performance testing?
Integrating performance testing into a Continuous Integration (CI) pipeline requires careful planning and execution to ensure that it runs smoothly without disrupting the flow of development.
In practice, performance tests need to run in environments that mimic production.
Since performance tests are inherently different from functional tests, for example, functional tests are generallyfasterthan performance tests, or we run functional tests more frequently (every code commit or pull request) than performance tests, we need to consider some factors while integrating performance testing in CI, If not managed carefully, performance tests can significantly slow down the CI pipeline due to their resource and time requirements:
- Isolate Performance Testing Environments: Performance tests typically require a more controlled andproduction-likeenvironment to yield meaningful results. They may needdedicated infrastructureor virtual machines to simulate real-world conditions accurately. This isolation ensures that the performance metrics we gather are reliable and not influenced by other activities in the CI environment.
- Automate Performance Testing: As I mentioned above, While some basic performance tests might be run frequently, more comprehensive performance tests are usually scheduled at intervals (e.g.,nightlyorbefore release candidates) due to their complexity and resource demands.
- Analyze and Report Results Automatically: Integrate reporting tools that can automatically analyze the results of performance tests and generate reports. These reports should be accessible to the entire team and should include comparisons with previous test runs to identify trends. Interpreting performance test results often involves analyzing graphs and response times and comparing them against baselines or SLAs.
- Set up Alerts for Performance Degradations: Configure our CI system to trigger alerts if performance metrics fall below predefined thresholds. This ensures that the team is immediately aware of any performance issues that need attention.
Dynamic Analysis using Digma in CI
One of the side advantages of having continuous performance testing in CI is that we can catch issues using tools likeDigmaearly and fix them at the early stage of the software development cycle.
We can also configure Digma in the CI to show us thecommit IDrelated to an issue. We need toadd Commit hashes to our observability datato do so. This information will be used to identify the issue. For example, here is the ticket details for an issue found by Digma:
We wrote a previous blog post and described how Dynamic Analysis relates to Performance Testing:
Digmacan collect tracing data and metrics locally and gives us a lot ofinsightswithout leaving the IDE, and we will be able to navigate between code and traces inside the IDE to follow the flow of a request to improve issues.
Moreover, It is possible toconnect our CI environment to Digma. In this way, by having load tests in the CI pipeline and also connecting it to Digma, we can complete theDynamic Analysis Phases, and in addition to having load test results and benefits, Digma will inform us when any issues are found during the performance test.
Using Digma in GitHub Actions
Digma has recently unveiledtwo new Github Actions stepsdesigned to assist us in setting up instrumentation for our application and verifying that there are no new issues during each test run.
When we create a new PR, Digma will run through the steps in CI. When there is an issue, Digma can find it by comparing test data with the baseline. The issue will then be visible to our IDE. If we configure Digma in the CI to show us thecommit IDrelated to an issue, the issue will be linked to the code that caused it, and we can easily find the problematic commit.
Final Thoughts
Performance Testing in Continuous Integration (CI) brings several advantages to the software development process, ensuring that applications not only function correctly but also perform optimally under various conditions. This approach allows development teams to continuously assess theresponsiveness,scalability, andstabilityof their applications throughout the development process. We saw that in addition to these benefits, we could also leverage tools likeDigmato complete theDynamic Analysis Phasesand find more issues in our codebase before deploying the application into production and avoiding regressions.
Download Digma: Here
