What’s all this integration agitation, anyway?

Imagine you are in charge of creating a whole new complex architecture to solve a new need of your company. You are already an experienced engineer and have solved many of the requirements with some components you are already familiar with. But now, you have to orchestrate all these components together and make them work like a clock. Now we need a proper integration.

You may have heard any or all of these keywords before: middleware, integration, orchestration. And you may be wondering why and when to use them. Take a walk with me to understand when and how integration frameworks are useful.

Imagine you are in charge of solving a new need of your company. There is no complete software stack for what you need. You will have to involve your team to create something new. Even if you reuse some components, you have to make them interact and talk to each other.

You are an experienced software engineer and have solved previously many of the requirements with some components you are already familiar with. But now you have to orchestrate all these components together and make them work like a clock. Now you need a proper integration. You want all of them to cooperate smoothly in your architecture.

The first thing any good developer thinks about is building a custom software that acts as the glue between all these components. Maybe adding some fancy extra functionality. And, (why not?) as we are at the beginning of a new exciting project, probably we want to try all these new technologies you have been reading and hearing about. Whatever the fad buzzword is now, you are willing to try it.

Although this may be appealing, your inner experienced engineer tells you to stop. There’s something you also read about, these integration frameworks. Could they be useful here?

The Integration Paradigm

As much as we would like to start a new clean project from scratch and throw all our ingeniousness into it, we shouldn’t reinvent the wheel. Let’s take a look at what is this middleware or integration software.

Middleware, or integration software, can help us orchestrate and automate the interaction between different applications, APIs, third party services or any other software piece we may have to connect.

A proper integration tool should provide us with the following features: transformation, integration patterns and connectors to existing protocols and components.

Transformations

When we connect different components of an architecture, they rarely speak the same languages or, on this case, data formats. Some components will output an xml that has to be fed to the following component on a json form. Maybe we even need to add or remove some attributes on that json data.

We need some way to easily transform the data traveling from one component to the following so it fits properly.

If we want to do this with our own script, there are many libraries that can help us doing this like Jackson or the built-in xml libraries on Python . We even have the XSLT language to transform XML. But to use any of these properly, we would have to learn first how to use them. And any code we generate will have to be maintained and upgraded properly.

An integration framework allows us to define what is the mapping between the output of one component and the input of the following so we can forget about the explicit implementation. The less code we have to maintain, the better.

Enterprise Integration Patterns

Not all workflows in the architecture will be lineal. Some of the steps will require broadcasting, some of them will require conditional flowing. Some will require waiting the output of different components to conflate the data. These action patterns are something that have been studied for a long time. And as with software development patterns, you can classify them and study them to create better integrations.

You can find all of these patterns prettily explained in the classic Enterprise Integration Patterns book.

Connectors

All of the above is useless if we can’t connect to (and from) the specific component we need.

Our ideal integration framework should offer support for common protocols like ftp, http, jdbc,… Also it should offer support to connect to common components like a mail server, messaging services, atom,… We could claim even that no integration tool would be good if it doesn’t also support specific well known services like being able to send a message through a Telegram bot or store information on Elastic Search.

Integration Frameworks as Lego building blocks

Being able to seamlessly connect from one component to the next without having to worry about the specifics of their interfaces is what distinguishes an average integration tool from a good integration tool.

Apache Camel

Let’s talk about something less abstract. At this point you may be wondering where you can find a good integration framework.

Apache Camel is not only one of the most active projects inside the Apache Software Foundation, it is also the lightest and most complete integration framework available. And on top of it, it is also Free and Open Source Software!

Camel is already an old actor on the integration world. It has support for hundreds of components, protocols and formats. Some of these protocols come very handy allowing the user, for example, to connect to any REST API that they need.

Camel uses its own DSL, a simplified language to define easily the workflows step by step.

Camel-K

Camel is also available in Knative. This means, we can use it on a serverless environment, making sure the orchestration between services runs and escalates properly.

Camel K Orchestration Example

This example demonstrates how to orchestrate integrations using Camel-K and Kafka as a messaging service. We are going to implement two integrations that interact through a database to simulate how cat adoptions work.

The full example can be found on Github.

Flux diagram
Two integration workflows that simulate how cat adoptions work

One integration will store cats coming from Kafka to the database waiting for a person to adopt them. The second integration will receive people interested in adopting and will match cats with them.

Cat Input from Kafka to Database

First we are going to implement the storage of cat input messages to the database.

As you can see, the Camel DSL is very intuitive: this integration listens to the proper Kafka broker and for every message that arrives, it unmarshalls the json to extract the data and pushes it to the database. The Cat class is just a simple bean with getters and setters for the attributes.

// camel-k: language=java

import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;
import model.Cat;

public class CatInput extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    //Listen to kafka cat broker
    from("kafka:cat?brokers=my-cluster-kafka-bootstrap:9092")
    .log("Message received from Kafka : ${body}")
    .unmarshal().json(JsonLibrary.Gson, Cat.class)
  
    //Store it on the database with a null person
    .setBody().simple("INSERT INTO cat (name, image) VALUES ('${body.name}', '${body.image}')")
    .to("jdbc:postgresBean?")
  
    //Write some log to know it finishes properly
    .log("Cat stored.");}
  }
}

Person Input from Kafka to Adopt

Now we are going to implement the reception of people wanting to adopt a cat.

This integration is a bit more complex, as we are going to introduce a conditional choice: if there is a cat available on the database, it will be assigned to the person. If there is no cat (otherwise), a message will be returned saying no cat is available.

// camel-k: language=java

import org.apache.camel.builder.RouteBuilder;

public class PersonInput extends RouteBuilder {
  @Override
  public void configure() throws Exception {
    //Listen to kafka person broker
    from("kafka:person?brokers=my-cluster-kafka-bootstrap:9092")
    .log("Message received from Kafka : ${body}")
    .log("${body} wants to adopt a cat")
    
    //Store the name of the person
    .setProperty("person", simple("${body}"))
    
    //Search for a lonely cat
    .log("...looking for available cats...")
    .setBody().simple("SELECT id, name, image FROM cat WHERE person is NULL LIMIT 1;")
    .to("jdbc:postgresBean?")
    
    .choice()
      .when(header("CamelJdbcRowCount").isGreaterThanOrEqualTo(1))
        .setProperty("catname", simple("${body[0][name]}"))
        .setProperty("catimage", simple("${body[0][image]}"))
        .setProperty("catid", simple("${body[0][id]}"))
        .log("Cat found called ${exchangeProperty.catname} with ID ${exchangeProperty.catid}")
        //There's a cat available, adopt it!
        .setBody().simple("UPDATE cat SET person='${exchangeProperty.person}' WHERE id=${exchangeProperty.catid}")
        .to("jdbc:postgresBean?")
  
        //Write some log to know it finishes properly
        .setBody().simple("Congratulations! ${exchangeProperty.catname} adopted ${exchangeProperty.person}. See how happy is on ${exchangeProperty.catimage}.")
        .to("log:info")
      .otherwise()
        //Write some log to know it finishes properly
        .setBody().simple("We are sorry, there's no cat looking for a family at this moment.")
        .to("log:info")
    .end();
  }
}

Feeding data automatically

As an extra step on this exercise, we are going to implement a final job that sends random new cat data to the Kafka “cat” topic with a timer.

The complexity on this class is not the Camel side, but the random generator of cat names.

// camel-k: language=java dependency=camel:gson

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.model.dataformat.JsonLibrary;

public class AutoCat extends RouteBuilder {
  @Override
  public void configure() throws Exception {

    // Preparing properties to build a GeoJSON Feature
    Processor processor = new Processor() {

      String[] title = new String[] { "", "Lady", "Princess", "Mighty", "Your Highness", "Little", "Purry", "Empress", "Doctor", "Professor" };
      String[] firstname = new String[] { "Dewey", "Butter", "Merlin", "Epiphany", "Blasfemy", "Metaphor", "Fuzzy",
          "Whity", "Astro", "Salty", "Smol", "Whiskers", "Scully" };
      String[] lastname = new String[] { "", "Luna", "Wild", "Dragonis", "Firefly", "Puff", "Purrcy", "Priss",
          "Catsie" };

      Random r = new Random();

      @Override
      public void process(Exchange exchange) throws Exception {
        Map<String, String> map = new HashMap<String, String>();
        map.put("image", exchange.getProperty("catimage").toString());

        StringBuilder name = new StringBuilder();
        name.append(title[r.nextInt(title.length)]);
        name.append(" ");
        name.append(firstname[r.nextInt(firstname.length)]);
        name.append(" ");
        name.append(lastname[r.nextInt(lastname.length)]);

        exchange.setProperty("catname", name.toString());
        map.put("name", name.toString().trim());

        exchange.getMessage().setBody(map);

      }

    };

    // Listen to kafka cat broker
    from("timer:java?period=10s")
      
      // Take a random image
      .to("https://api.thecatapi.com/v1/images/search")
      .unmarshal().json(JsonLibrary.Gson)
      .log("A new cat arrived today ${body[0][url]}")
      .setProperty("catimage", simple("${body[0][url]}"))

      // name cat and prepare json
      .process(processor)
      .log("${body}")
      .marshal().json(JsonLibrary.Gson)
      .log("We named them ${exchangeProperty.catname}")

      // Send it to Kafka cat broker
      .to("kafka:cat?brokers=my-cluster-kafka-bootstrap:9092")

      // Write some log to know it finishes properly
      .log("Cat is looking for a family.");

  }
}

Now you are ready to implement your own orchestrations with Kafka and Camel K.

From Cat to Hat

On June I joined Red Hat as a Senior Software Engineer.

I always said that I preferred to work on a small company before a big company, because on a big company you can’t be anything more than a number. That you can’t really grow professionally if you are such a small piece on a big machine.

I have to say: I was completely wrong. The room for manoeuvre you have on a big company is not comparable to that on a smaller one.

Just a month after my latest job change and I am completely in love with Red Hat. I keep asking around me “where is the trick? where is the trap?” because there have to be. Tough it seems there isn’t. If this is not my dream job, it is only one “geospatial” label away.

Continue reading “From Cat to Hat”

I want to fix the diversity on my event, help me!

If someone linked you this post is probable you are organizing an event where diversity and inclusivity is an issue and they want to help you fix that. If you want, you can jump to the subsection that better adjust to your case. Remember: diversity is not a TL;DR, you probably need to read the full article to get a better grasp of what you need. As usual: I’m going to focus on the gender gap because it’s easier for me to talk in those terms, but similar strategies can be applied to any other under-represented group.

I was told I have a manel, what’s that?

A manel is a panel full of men (usually white and middle aged). Usually this manel is the main panel or the keynoters panel whose members are the most relevant/the most advertised speakers. They are the display case of your event and they may tell more about your event than you probably suspect.

Bonus track: Have you heard about the Techdel Test?

But I had no women speaker candidates for my event!

It doesn’t matter if it was a set of speakers chosen manually or if you sent a call for papers to the internet waiting for proposals. If you have few (or none) proposals that improve your diversity line-up, something went wrong. Because there are women (and PoC and functionally diverse speakers) out there. You just didn’t met them. But don’t worry, there’s many things you can do to improve it.

Continue reading “I want to fix the diversity on my event, help me!”
en_GBEnglish (UK)