Kaoto: Integrate without limits

I would like to present you with an ETL and integration editor Rachel and I have been working on for the past year with the initial help of Zineb: Kaoto.

What is Kaoto?

Kaoto is an integration editor to create and deploy integrations in a low-code way and no-code way based on Apache Camel. It combines a source code editor and a drag and drop graphical space synchronized with each other. It can be run both as standalone and as a service (SaaS).

With the no-code mode, the user can build the entire integration orchestration with the drag and drop function. Kaoto has a step catalog with a list of all available building blocks that the users may want to transform data or integrate with services.

The source code will be available for users curious to know what they are going to deploy. But in no case they have to understand or even see that code.

Example of building block drag and drop

With the low code mode, users can learn how to create integrations and, at the same time, control what they are deploying. They can use both the drag and drop and the source code editor, that will autocomplete what the user won’t or don’t know how to write. Meanwhile the graphical space will show the integration being built and the drag and drop will still be available for them to complete or accelerate the development.

Example of low code integrations.

Kaoto can help users start with Apache Camel and slowly build their knowledge. All the source code generated is clean and viewable in any IDE.

Customizing Kaoto

Each building block type can have its own microfrontend. This is useful when you add your own building blocks to your Kaoto instance. But it can also help adapt Kaoto to different kinds of users, hiding or extending certain details important for each use-case. Extensions can be manuals and helpers for Kaoto.

When used as a service, the extensions and the list of available building blocks are settings that can be stored in the cloud. Therefore, administrator users can modify this configuration, which will refresh live in the users playgrounds. In addition, as this configuration can be in the cloud, different users can share the configuration. This can help organizations accommodate different groups of users, offering different configurations depending on the profile.

What is on the roadmap?

We started the development focused on Kamelet Bindings, but we soon realized we could add more languages. Edition of Apache Camel routes (in yaml form) and Kamelet definitions are next in development queue. We are also working on translating from other integration languages to Camel DSL. This can help users migrate to Apache Camel.

We will soon have one-click support for cloud-native Apache Camel deployments via Camel-K. Matej is close to having an operator for Kubernetes clusters which will simplify even more the installation of Kaoto in the cloud.

You can quickly test it via docker as described on the quickstart. Make sure your docker images have access to internet to be able to access the default remote configuration!

You can find more information on the webpage of the project.

What is GeoNetwork?

GeoNetwork is a server side application that allows you to maintain a geographic referenced metadata catalogue. This means: a search portal that allows to view metadata combined with maps.

GeoNetwork logo
The yoga man

Based on Free and Open Source Software, it strictly follows different standards for metadata, from Inspire to OGC. It implements the CSW interface to be able to interact with generic clients looking for data. It also has built-in harvesters to connect to other servers and populate data.

This has allowed GeoNetwork to expand to a lot of organizations. For example: the swiss geoportal or the brasilian one, not forgetting the New zealander. GeoNetwork is the most used open sourced spatial catalog in the world. You can find it in most of the public administrations that use free and open source software.

The catalogue deploys on a java application container (like tomcat or jetty). It works over the Jeeves framework. Jeeves is based on XSLT transformation server library. This allows a powerful development of interfaces, for humans (HTML) or machines (XML). Therefore, it makes metadata from GeoNetwork to be easily accessible by different platforms.

Recently half-refactored to Spring and AngularJS, GeoNetwork has a REST API and an event hook system to make extensions and customizations easier.

Annotations and Decorators in Java

Annotations on the code or decorators have become very common. They allow the programmer to add additional useful information about how to improve the code or change how to compile / run a particular class. They are a Java extension to allow aspect-oriented programming.

We have three types of annotations based on the moment of usage:

Information for the Compiler

These annotations allow the compiler to indicate whether or not to ignore errors and warnings or what to do with them. If you’ve worked with a Java IDE (like eclipse) probably you would have used this type of annotations. For example, you can use @Override on a function to indicate that you are overwriting a method defined on a parent class. 

This annotation is completely optional, but allows both the compiler and the developer to check that they are indeed overwriting existing hierarchical functionality.

For example:

public class Parent {     
    public void do(){
        System.out.println("Parent");
     }
}

public class Son extends Parent{     
    @Override
    public void do(){
        System.out.println("Son");
     }
}
Compiler-time and deployment-time processing

These annotations allow the compiler to add extra information about how to generate the code. By adding or modifying functionality from that in the source code you can alter how a class behaves. Also to create new classes (based on a file descriptor), etc …

These annotations will only be visible at this point. They are not compiled to the .class files. Therefore they are not available at runtime.

Runtime Annotations

You can use this annotations on runtime and they work on a very similar way as an interface.

Let’s see an example on how to create a Runtime Annotation and how can we use it. The annotation named MyAnnotation can be applied to elements oftype field:

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
     public @interface MyAnnotation {
}

Now we can create an annotated class by this annotation:

public class myObject
 {
 @MyAnnotation
 public String field;
 }

This way, we can check by reflection if an object has an annotated field on any part of the code:

Class<?> res = objeto.getClass();
for (Field f : res.getFields()) {
     if (f.isAnnotationPresent(MyAnnotation.class)) {
          System.out.println("OK");
      }
}

More Information:

en_GBEnglish (UK)