- The development and testing systems do not reflect the hardware specifications of the production system.
- Features tested on the testing systems are mainly tested in isolation.
- The load on testing systems is not comparable to that of a production system.
Moreover, test cases only cover the happy path, which is already covered even by unit tests but not the combination of several functionalities or even with the whole software suite, which is often not feasible anyway. Another aspect to consider is data. The amount of data available in development and testing systems is only a fraction of the data used in production systems, and therefore the behavior is not comparable.
With cloud software, the whole software lifecycle management and the provisioning of suitable hardware often become the responsibility of software vendors and their service providers. With this comes many new opportunities that are not possible in their entirety when providing on-premise software. We came up with one idea at Celonis to test our software directly in production. As a software user, you may think: “That is not an innovation, it seems almost all software vendors are doing this.” However, our approach doesn’t interfere with the user experience or even degrade performance. We are aiming for exactly the opposite by improving our CI/CD pipeline and at the same time improving quality and avoiding regressions.