You may have heard any or all of these keywords before: middleware, integration, orchestration. And you may be wondering why and when to use them. Take a walk with me to understand when and how integration frameworks are useful.

Imagine you are in charge of solving a new need of your company. There is no complete software stack for what you need. You will have to involve your team to create something new. Even if you reuse some components, you have to make them interact and talk to each other.

You are an experienced software engineer and have solved previously many of the requirements with some components you are already familiar with. But now you have to orchestrate all these components together and make them work like a clock. Now you need a proper integration. You want all of them to cooperate smoothly in your architecture.

The first thing any good developer thinks about is building a custom software that acts as the glue between all these components. Maybe adding some fancy extra functionality. And, (why not?) as we are at the beginning of a new exciting project, probably we want to try all these new technologies you have been reading and hearing about. Whatever the fad buzzword is now, you are willing to try it.

Although this may be appealing, your inner experienced engineer tells you to stop. There’s something you also read about, these integration frameworks. Could they be useful here?

The Integration Paradigm

As much as we would like to start a new clean project from scratch and throw all our ingeniousness into it, we shouldn’t reinvent the wheel. Let’s take a look at what is this middleware or integration software.

Middleware, or integration software, can help us orchestrate and automate the interaction between different applications, APIs, third party services or any other software piece we may have to connect.

A proper integration tool should provide us with the following features: transformation, integration patterns and connectors to existing protocols and components.

Transformations

When we connect different components of an architecture, they rarely speak the same languages or, on this case, data formats. Some components will output an xml that has to be fed to the following component on a json form. Maybe we even need to add or remove some attributes on that json data.

We need some way to easily transform the data traveling from one component to the following so it fits properly.

If we want to do this with our own script, there are many libraries that can help us doing this like Jackson or the built-in xml libraries on Python . We even have the XSLT language to transform XML. But to use any of these properly, we would have to learn first how to use them. And any code we generate will have to be maintained and upgraded properly.

An integration framework allows us to define what is the mapping between the output of one component and the input of the following so we can forget about the explicit implementation. The less code we have to maintain, the better.

Enterprise Integration Patterns

Not all workflows in the architecture will be lineal. Some of the steps will require broadcasting, some of them will require conditional flowing. Some will require waiting the output of different components to conflate the data. These action patterns are something that have been studied for a long time. And as with software development patterns, you can classify them and study them to create better integrations.

You can find all of these patterns prettily explained in the classic Enterprise Integration Patterns book.

Connectors

All of the above is useless if we can’t connect to (and from) the specific component we need.

Our ideal integration framework should offer support for common protocols like ftp, http, jdbc,… Also it should offer support to connect to common components like a mail server, messaging services, atom,… We could claim even that no integration tool would be good if it doesn’t also support specific well known services like being able to send a message through a Telegram bot or store information on Elastic Search.

Being able to seamlessly connect from one component to the next without having to worry about the specifics of their interfaces is what distinguishes an average integration tool from a good integration tool.

Apache Camel

Let’s talk about something less abstract. At this point you may be wondering where you can find a good integration framework.

Apache Camel is not only one of the most active projects inside the Apache Software Foundation, it is also the lightest and most complete integration framework available. And on top of it, it is also Free and Open Source Software!

Camel is already an old actor on the integration world. It has support for hundreds of components, protocols and formats. Some of these protocols come very handy allowing the user, for example, to connect to any REST API that they need.

Camel uses its own DSL, a simplified language to define easily the workflows step by step.

Camel-K

Camel is also available in Knative. This means, we can use it on a serverless environment, making sure the orchestration between services runs and escalates properly.

Camel K Orchestration Example

This example demonstrates how to orchestrate integrations using Camel-K and Kafka as a messaging service. We are going to implement two integrations that interact through a database to simulate how cat adoptions work.

The full example can be found on Github.

One integration will store cats coming from Kafka to the database waiting for a person to adopt them. The second integration will receive people interested in adopting and will match cats with them.

Cat Input from Kafka to Database

First we are going to implement the storage of cat input messages to the database.

As you can see, the Camel DSL is very intuitive: this integration listens to the proper Kafka broker and for every message that arrives, it unmarshalls the json to extract the data and pushes it to the database. The Cat class is just a simple bean with getters and setters for the attributes.

// camel-k: language=java

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;
import model.Cat;

public class CatInput extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    //Listen to kafka cat broker
    from("kafka:cat?brokers=my-cluster-kafka-bootstrap:9092")
    .log("Message received from Kafka : ${body}")
    .unmarshal().json(JsonLibrary.Gson, Cat.class)
  
    //Store it on the database with a null person
    .setBody().simple("INSERT INTO cat (name, image) VALUES ('${body.name}', '${body.image}')")
    .to("jdbc:postgresBean?")
  
    //Write some log to know it finishes properly
    .log("Cat stored.");}
  }
}

Person Input from Kafka to Adopt

Now we are going to implement the reception of people wanting to adopt a cat.

This integration is a bit more complex, as we are going to introduce a conditional choice: if there is a cat available on the database, it will be assigned to the person. If there is no cat (otherwise), a message will be returned saying no cat is available.

// camel-k: language=java

import org.apache.camel.builder.RouteBuilder;

public class PersonInput extends RouteBuilder {
  @Override
  public void configure() throws Exception {
    //Listen to kafka person broker
    from("kafka:person?brokers=my-cluster-kafka-bootstrap:9092")
    .log("Message received from Kafka : ${body}")
    .log("${body} wants to adopt a cat")
    
    //Store the name of the person
    .setProperty("person", simple("${body}"))
    
    //Search for a lonely cat
    .log("...looking for available cats...")
    .setBody().simple("SELECT id, name, image FROM cat WHERE person is NULL LIMIT 1;")
    .to("jdbc:postgresBean?")
    
    .choice()
      .when(header("CamelJdbcRowCount").isGreaterThanOrEqualTo(1))
        .setProperty("catname", simple("${body[0][name]}"))
        .setProperty("catimage", simple("${body[0][image]}"))
        .setProperty("catid", simple("${body[0][id]}"))
        .log("Cat found called ${exchangeProperty.catname} with ID ${exchangeProperty.catid}")
        //There's a cat available, adopt it!
        .setBody().simple("UPDATE cat SET person='${exchangeProperty.person}' WHERE id=${exchangeProperty.catid}")
        .to("jdbc:postgresBean?")
  
        //Write some log to know it finishes properly
        .setBody().simple("Congratulations! ${exchangeProperty.catname} adopted ${exchangeProperty.person}. See how happy is on ${exchangeProperty.catimage}.")
        .to("log:info")
      .otherwise()
        //Write some log to know it finishes properly
        .setBody().simple("We are sorry, there's no cat looking for a family at this moment.")
        .to("log:info")
    .end();
  }
}

Feeding data automatically

As an extra step on this exercise, we are going to implement a final job that sends random new cat data to the Kafka “cat” topic with a timer.

The complexity on this class is not the Camel side, but the random generator of cat names.

// camel-k: language=java dependency=camel:gson

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;

public class AutoCat extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    // Preparing properties to build a GeoJSON Feature
    Processor processor = new Processor() {

      String[] title = new String[] { "", "Lady", "Princess", "Mighty", "Your Highness", "Little", "Purry", "Empress", "Doctor", "Professor" };
      String[] firstname = new String[] { "Dewey", "Butter", "Merlin", "Epiphany", "Blasfemy", "Metaphor", "Fuzzy",
          "Whity", "Astro", "Salty", "Smol", "Whiskers", "Scully" };
      String[] lastname = new String[] { "", "Luna", "Wild", "Dragonis", "Firefly", "Puff", "Purrcy", "Priss",
          "Catsie" };

      Random r = new Random();

      @Override
      public void process(Exchange exchange) throws Exception {
        Map<String, String> map = new HashMap<String, String>();
        map.put("image", exchange.getProperty("catimage").toString());

        StringBuilder name = new StringBuilder();
        name.append(title[r.nextInt(title.length)]);
        name.append(" ");
        name.append(firstname[r.nextInt(firstname.length)]);
        name.append(" ");
        name.append(lastname[r.nextInt(lastname.length)]);

        exchange.setProperty("catname", name.toString());
        map.put("name", name.toString().trim());

        exchange.getMessage().setBody(map);

      }

    };

    // Listen to kafka cat broker
    from("timer:java?period=10s")
      
      // Take a random image
      .to("https://api.thecatapi.com/v1/images/search")
      .unmarshal().json(JsonLibrary.Gson)
      .log("A new cat arrived today ${body[0][url]}")
      .setProperty("catimage", simple("${body[0][url]}"))

      // name cat and prepare json
      .process(processor)
      .log("${body}")
      .marshal().json(JsonLibrary.Gson)
      .log("We named them ${exchangeProperty.catname}")

      // Send it to Kafka cat broker
      .to("kafka:cat?brokers=my-cluster-kafka-bootstrap:9092")

      // Write some log to know it finishes properly
      .log("Cat is looking for a family.");

  }
}

Now you are ready to implement your own orchestrations with Kafka and Camel K.