Camel-K

Leaving aside complexities like the Enterprise Integration Patterns, we can consider most integrations as a form of advanced ETL: Extract, Transform, and Load. We extract data from a data store or service. Then we transform it from an input to an output format. And finally we push or load that transformed data into some output channel. It is the easiness to connect with the input and output channels what makes the ETL need a proper integration framework. Complex integrations will combine these three steps differently. But the outcome is always to move information from one place to another, connecting different systems. Where the information may be a full dataset or just a triggered event. I already tackled the issue of choosing the right integration tool from an engineer’s perspective and what variables to take into account. But when we are talking about data science and data analysis, there is a requirement that goes on top of all of the previous: the accessibility and easiness of usage of the tool.

On this article we will explore how to do no code and low code integrations based on Apache Camel. “Every paradigm including data flow, programming by example, and programming through analogies brings its own set of affordances and obstacles.” Alexander Repenning - DOI reference number: 10.18293/VLSS2017-010 We are going to use Kaoto, which just made its 1.0.0 release. On this release, the Kaoto team has focused on the no-code graphical canvas to make sure the user experience is as smooth as possible.

A year ago I started with a couple of friends a new project based on Quarkus to create a visual editor for integrations called Kaoto. As responsible of the backend side, I obviously chose Java to do it. Coming from the Java 8 world with shy traces of Java 11, I decided to jump directly to Quarkus on Java 17 (unstable at the time) with Reactive and explore the serverless possibilities while, at the same time, keep the over-engineering and the over-fanciness of new features as reasonable as possible. On this article I will discuss the good and the bad of this experience. I am not a Quarkus developer, I am a developer that used Quarkus. And as any average developer that starts with a new technology, I obviously skipped the documentation and just bungee jumped into the framework, blindfolded and without safe nets.

When we are integrating different components and services in our software architecture, the first step is to select a good orchestration framework. On this opinionated article I will present my criteria to decide which is the right framework. Riding the Enterprise Service Bus As you compose services, you will notice the need for an Enterprise Service Bus to communicate with each other. But an EBS can be useless if you don’t have good ETL (extract, transform, load) tools along with it to manipulate our data. The same way that an ETL without a proper routing system can leave us orphaned. We need to route messages and events and at the same time make sure data transformations take place so different endpoints with varied protocols and formats can interact with each other. That’s where integration frameworks come in. The Enterprise Integration Patterns can help developers on both tasks: by providing data transformations between outputs and inputs and offering different routing strategies.

I would like to present you with an ETL and integration editor Rachel and I have been working on for the past year with the initial help of Zineb: Kaoto. What is Kaoto? Kaoto is an integration editor to create and deploy integrations in a low-code way and no-code way based on Apache Camel. It combines a source code editor and a drag and drop graphical space synchronized with each other. It can be run both as standalone and as a service (SaaS). With the no-code mode, the user can build the entire integration orchestration with the drag and drop function. Kaoto has a step catalog with a list of all available building blocks that the users may want to transform data or integrate with services.