At Celonis, we enable our users to make use of Realtime Connectors to get their source system data into the Celonis EMS -as the name suggests- in real-time.
This is an avant-garde alternative to the traditional batch extractions. A modern technology in view that once the extraction is configured, our real-time connectors monitor every update in the source system and apply the changes on the Celonis side automatically, in a matter of seconds.
We are going to walk through the benefits for real-time data ingestion, as well as the types of Realtime Connectors and how they function from a high-level manner.
In modern business, massive amounts of data are generated every second. Each update carries new information and users expect their data to be available the second it’s updated. The only way to do this is real-time connectivity. It is a necessity to keep up with modern business requirements.
This is where Celonis comes in. By extracting process data from on-prem and cloud-based ERPs, CRMs, data warehouses, productivity tools and desktop activities (i.e task mining), you can turn raw data into meaningful insights. Celonis automates data extraction so you can execute on the most up-to-date information available. With a stable real-time pipeline, it is possible to quickly identify process inefficiencies and take further, even automated, action to fix them. In this sense, real-time connectivity is a fundamental step for unlocking execution capacity.
Our goal was to make this initial step as user friendly as possible. The subset of benefits of Realtime Connectivity stated below serve the objective of better user experience:
Realtime Connectors are differentiated by the technology behind them. Let’s walk through two main Realtime Connector types and how they function in the Celonis EMS.
Real-time connectivity to JDBC (Java Database Connectivity) based source tables can be achieved using the Replication Cockpit in Celonis.
The Replication Cockpit executes a new extraction for the requested table every few minutes (the time interval is configurable) with a major twist of using change-log tables to detect changes in the source system. In technical terminology, the process of ensuring the consistency between two systems is called “replication”.
Real-time_Connectors_-_Graphic_1__1_.jpg
Let’s define 2 major components of a replication:
Real-time connectivity in JDBC can be achieved by creating changelog tables that would be populated when there is a change in the source table. The changes in the source table can be tracked with the help of triggers.
In short, the replication process can be summarized in 3 steps:
The processing of the changelog tables is done by joining the changelog tables with the source table over the primary keys. We always expect the primary keys of the source table to be part of the changelog tables in order to join the tables correctly.
For traditional source systems, the data is constantly present and can be requested at any time. For streaming based source systems this is not the case. The whole data is not available any time. The required data should be retrieved, processed and stored in realtime by communicating with the streaming service. In Celonis, for the supported streaming services Realtime Connectivity can be achieved using Streaming Cockpit.
In order to accommodate these streaming source systems, we introduced a different type of extraction architecture. Each connection to the streaming source system is called "Subscription" in the Streaming Cockpit. These subscriptions can be turned on and off using the Streaming Cockpit UI.
Celonis supports different streaming services like Azure Service Bus, Azure Event Hub, Salesforce Streaming. Since the streaming services are not maintained by Celonis, we are playing the role of a “user”. To retrieve and process data, we have different strategies for different streaming services. Let’s check an example to concretize.
Real-time_Connectors_-_Graphic_2.jpg
For example, Salesforce Streaming gives us a “replayID” which indicates the event number in order. Among all the present events Salesforce Streaming services provide, we know from where to process the data by the replayID. We can signal how many new messages we received and processed. In case of a temporary failure, extraction will correct itself and re-execute automatically by using the last successfully processed replayID.
Once we successfully process the data, we update our replayID, signal the changes and let the Salesforce Streaming decide whether to keep or delete the messages from their stream. We cannot enforce the streaming services to delete processed data because there might be other users watching the same stream.
Salesforce Streaming is just one example. Not all streaming services have the same structure, so we customize our solutions to the specific needs of the streaming services to have a better efficiency in every case.
Software Engineer in Extractors Suite team, Enes works at the intersection of customer data and the Celonis EMS. He specializes in extraction of valuable data from different sources. He is a drone pilot. He can throw-and-catch a frisbee like a boomerang. Interested in fragrances.
Companies are struggling to reach their environmental goals. Why? One word: Complexity. Tracking processes that affect carbon emissions is not an easy task.
Industry-Blog_Hero_2560x1440-Consumer-Goods.jpg
Quality Management is one of those processes that’s just always been manual. Whether that’s lab technicians working on formulations for a new food product with literal pen and paper…
Industry-Blog_Hero_2560x1440-Retail.jpg
For years the predictions claimed E-Com was going to eat brick-and-mortar’s breakfast, but the figures kept coming back, and brick-and-mortar still amounted to about 82% of all retail sales and the YoY shift remained small.
