Mirrord at Celonis

Mirrord at Celonis

Background

Most of the services that Celonis delivers are hosted on the Kubernetes (K8s) platform. This allows us a consistent platform across multiple service providers, as well as offering a self-hosted option. The K8s platform offers some great patterns, such as rolling deployments, re-starting failed containers, node scheduling, DNS, and more.

The other side of this power and consistency is a sizable increase in complexity. With VMs, you deploy your service artifacts, then run a command to start your service. For K8s, you need to put that service in a Pod, define the Deployment and Service objects, and publish an image that can be used in your K8s environment. In addition, your code needs to be written in such a way that it expects this kind of environment, and behaves accordingly.

Challenge

There is an obvious impedance mismatch between running code on your laptop, and running it inside of K8s. This can give rise to some hacks or code that try to deal with this mismatch, and those fixes usually become tribal knowledge. It becomes a time sink for new and existing employees alike, as the fixes need to be explained or justified.

Some coding practices can help alleviate this issue, but they won’t solve it. This begs the question of what the solved state would look like. How much work should we expect new folks to do to get set up? How much work is needed to maintain it? How much mental energy should we expend mapping between K8s environments and local development environments?

Let’s be bold: None, None, and None, respectively. The goal of almost any development experience effort is to make things faster, easier, more robust, or more powerful.

Solution

Service Talking to K8s

Since we run our services within K8s, the best match we can provide is a K8s environment. That can be a big challenge to run locally, especially as you have dozens of services that interact to form a whole application. So how do we run our service as though it’s running in K8s? That’s what Mirrord provides!

Mirrord is a solution from MetalBear that runs your service locally – with a couple changes. It intercepts all the network and file system traffic, and forwards it to K8s. It does this by intercepting certain kernel-level calls, and forwarding those calls to an agent in your K8s cluster. For example, when your application tries to resolve a DNS name, Mirrord resolves that name within the context of your cluster. This means that you can address dependent services the same way you would with a normal deployment. You can utilize the Kube DNS and use http://. – or just http:// within the same namespace. You can even use it to run ad-hoc requests to your services:

❯ curl ems-service.dev/ems-service/actuator/health -v * Could not resolve host: ems-service.dev * Closing connection 0 curl: (6) Could not resolve host: ems-service.dev ❯ mirrord exec — curl ems-service.dev/ems-service/actuator/health -v * Trying 99.171.25.108:80... * Connected to ems-service.dev (99.171.25.108) port 80 (#0) > GET /ems-service/actuator/health HTTP/1.1 > Host: ems-service.dev > User-Agent: curl/8.1.2 > Accept: */* > < HTTP/1.1 200 OK < {"status":"UP","groups":["liveness","readiness"]}

Notice how running the curl command on its own resulted in a failure to resolve the DNS. However, when running with Mirrord , it resolves our service. Where does that 99.171.25.108 address come from?

❯ kubectl get svc —namespace dev | grep ems-service ems-service ClusterIP 99.171.25.108 80/TCP

It’s the IP of our service within K8s. When you run with the Mirrord executor, your application really does behave as though it’s within the K8s cluster.

K8s Talking to Service

This is where the magic of Mirrord really comes into focus. So far, we have just focused on allowing your application to talk to the K8s cluster. This is great but for the most part, this could be accomplished with kubectl port-forward. What we really need is the ability to “replace” a service – so that when other services call to your service, that traffic gets redirected to the version running on your laptop. But we also want to make sure that these replacement versions don’t affect the environment for anyone who is not that developer.

Mirrord accomplishes this by adding an HTTP traffic interceptor. When it finds a request with the right headers, it uses low-level networking to drive the request to your locally-running service. This allows you to replace that service with your own version, without affecting other developers, and without permanently altering the environment. As you use the UI, your version of the service receives all the traffic, and runs your new code.

Looking back to our questions, let’s address them in the context of Mirrord:

1. How much work should we expect new folks to do to get set up? With Mirrord config files already created, and a properly set-up IDE, developers don’t have to do any work! They can simply run the service.
2. How much work is needed to maintain it? Developers likely won’t have to do any maintenance. Given that it’s a packaged offering, maintenance and upgrades to the tool are not our responsibility. So long as the dev environment remains stable, users can inject their services and generally ignore the tool. Deployment and management of the Mirrord tool can be done by a single individual, not left to individual developers. Compare this to the work of maintaining a parallel environment with Docker or VM configuration, dealing with version and configuration drift, and marvel at the time savings!
3. How much mental energy should we expend mapping between K8s environments and local development environments? When maintaining a Docker or VM-based solution, a lot of configuration options must be tweaked to manage the mismatch between environments. With the Mirrord-based solution, there is no impedance mismatch. It runs just like it’s in a K8s environment. It dramatically simplifies the pipeline from Laptop to Production.

How to Use

Developer

Mirrord has plugins for IntelliJ IDEA and Visual Studio Code, and that’s the easiest way to run it. It works with Go, Rust, Java, Node, Python, Ruby, and much, much more. As shown above, there is also a CLI app that will perform its magic on nearly any executable – like curl, Postman, and more.

Running your service with your IDE, you can choose a Mirrord configuration (and enable/disable the plugin), and it takes care of everything else for you.

Service

Each service needs a one-time effort to create a configuration file, but after that, no more changes are needed. We can create different files for different usage scenarios, but the out-of-box configuration should allow new developers to immediately jump into productive work.

Infrastructure

Metal Bear publishes a Helm chart that deals with the configuration and installation of all the necessary resources (such as the Operator). We have wrapped that up in an ArgoCD application, making updates and deployments more consistent. The Operator may require periodic updates, but that becomes a small maintenance task for an individual, instead of bringing the whole team to a halt.