Selecting your orchestration conductor

When we are integrating different components and services in our software architecture, the first step is to select a good orchestration framework. On this opinionated article I will present my criteria to decide which is the right framework.

Riding the Enterprise Service Bus

As you compose services, you will notice the need for an Enterprise Service Bus to communicate with each other. But an EBS can be useless if you don’t have good ETL (extract, transform, load) tools along with it to manipulate our data. The same way that an ETL without a proper routing system can leave us orphaned.

We need to route messages and events and at the same time make sure data transformations take place so different endpoints with varied protocols and formats can interact with each other. That’s where integration frameworks come in.

The Enterprise Integration Patterns can help developers on both tasks: by providing data transformations between outputs and inputs and offering different routing strategies.

Orchestration Functionality

Our framework should be able to support not only the Enterprise Integration Patterns, but also a wide range of protocols and data formats.

Usability

Easiness of use, specially if we have complex use cases to maintain, is relevant to keep our architecture clean. This usability should never be in detriment of the functionality or the extensibility of the framework. We want our framework to interact with a varied range of components and services.

Sacrificing features in favor of user experience will make you hit faster the ceiling of what you can do.

Multiple Orchestration Languages

Related to the easiness of use is the language we need to use to build the integration routes. We don’t want to be tied to a specific language like Java or Python. There are frameworks that allow you to build the integration route in different languages.

When our orchestration builders come from different backgrounds, or are not very tech savvy, being able to integrate on different languages may come in handy. We want them to feel comfortable in whatever language they need to orchestrate.

Tutorials and documentation

It was never the case when good usability made up for a bad documentation. We need tutorials, manuals, helpers,… Both for users and developers. And of course, we will want to have some reference to fall back when something doesn’t work as expected and we need to find why.

Customization

Sometimes we need to use some data format or connect to some service not currently supported by the framework. Whether if we do it ourselves or hire someone to do it, can the existing functionality be extended to support them?

Dependencies size

We don’t want to drag a heavy dependency to our architecture. But not only smaller in size, we also want a framework that will use a light footprint over our hardware, while at the same time, it doesn’t drag our performance.

A smaller dependency usually means less source code that can introduce bugs to your software too.

Technical Support

Maybe we don’t need help, but what if we have a problem? Can we hire someone for technical support? Are there developers to hire to implement our custom features?

Having a wide range of companies offering services around our framework will greatly improve our experience in the long run.

License

It doesn’t matter how many companies are supporting the framework if the license is restricted. Only a software with a FOSS license will warrant you will not be tied to the whims or misfortunes of any external force or private company.

If you have been following me, this will not come as a surprise. You know I already have a preferred choice that scores high on all these criteria: Apache Camel.

Kaoto: Integrate without limits

I would like to present you with an ETL and integration editor Rachel and I have been working on for the past year with the initial help of Zineb: Kaoto.

What is Kaoto?

Kaoto is an integration editor to create and deploy integrations in a low-code way and no-code way based on Apache Camel. It combines a source code editor and a drag and drop graphical space synchronized with each other. It can be run both as standalone and as a service (SaaS).

With the no-code mode, the user can build the entire integration orchestration with the drag and drop function. Kaoto has a step catalog with a list of all available building blocks that the users may want to transform data or integrate with services.

The source code will be available for users curious to know what they are going to deploy. But in no case they have to understand or even see that code.

Example of building block drag and drop

With the low code mode, users can learn how to create integrations and, at the same time, control what they are deploying. They can use both the drag and drop and the source code editor, that will autocomplete what the user won’t or don’t know how to write. Meanwhile the graphical space will show the integration being built and the drag and drop will still be available for them to complete or accelerate the development.

Example of low code integrations.

Kaoto can help users start with Apache Camel and slowly build their knowledge. All the source code generated is clean and viewable in any IDE.

Customizing Kaoto

Each building block type can have its own microfrontend. This is useful when you add your own building blocks to your Kaoto instance. But it can also help adapt Kaoto to different kinds of users, hiding or extending certain details important for each use-case. Extensions can be manuals and helpers for Kaoto.

When used as a service, the extensions and the list of available building blocks are settings that can be stored in the cloud. Therefore, administrator users can modify this configuration, which will refresh live in the users playgrounds. In addition, as this configuration can be in the cloud, different users can share the configuration. This can help organizations accommodate different groups of users, offering different configurations depending on the profile.

What is on the roadmap?

We started the development focused on Kamelet Bindings, but we soon realized we could add more languages. Edition of Apache Camel routes (in yaml form) and Kamelet definitions are next in development queue. We are also working on translating from other integration languages to Camel DSL. This can help users migrate to Apache Camel.

We will soon have one-click support for cloud-native Apache Camel deployments via Camel-K. Matej is close to having an operator for Kubernetes clusters which will simplify even more the installation of Kaoto in the cloud.

You can quickly test it via docker as described on the quickstart. Make sure your docker images have access to internet to be able to access the default remote configuration!

You can find more information on the webpage of the project.

Are online events the new normal?

Some people have just (re-)discovered that we don’t have to meet in person to run an event. They are very enthusiastic about it. They are even asking to replace all conferences as online events. But, are they right?

Some of you may know I am going to co-chair the next FOSS4G. (Come Hell or High Water, but we are going to have a FOSS4G next year!) I have also participated on the organization of the last ApacheCON and some other minor online events. I have also spoken on many online events by now.

And, let me tell you, I am not as optimistic as those people. I really hope this post ages badly and in the following years we get better options for online events.

But as of today, I don’t feel we are anywhere close to replace face to face events with online ones. Delivering the talks as streaming is something good but we will be losing most of what makes a conference an amazing experience.

Let me tell you why.

Online Events Drawbacks

Online events allow people from all over the world to attend without having to travel. People with hard conciliation and people that can’t afford or are not able to travel will be on your event. It’s true, you will get attendees you wouldn’t have if it was a face to face event. And that’s good!

But that’s where the advantages of an online event end. And where a lot of new issues are created.

Timezones

Have you ever tried to find a good time slot for a simple one hour meeting between an Australian and someone living in Los Angeles? Exactly.

Now imagine allocating at least six good hours for streaming and interactions and trying to have at least everybody awake, no matter if it is early or late. Spoiler: you can’t.

Of course you can always find some arbitrary schedule that fits your biggest group of potential attendees and force the rest of your attendees adapt to it. Just make them imagine they are traveling and live on a different timezone, waking up at 3am to watch your talks, having lunch instead of breakfast and sleeping during the day. Seriously: is your event worth it?

Do you remember those people that couldn’t travel to your face to face event but can attend your online event? A good percentage of them will not attend your online event if it doesn’t fit their timezone. It doesn’t matter if they have the means to do it.

Hacking timezones

Of course, there are patches for this. Like having a 24/7 event so at all times there’s something happening for the people that are awake at that time. Which, in summary, just means having like separated events that overlap a bit between regions of the world. You will end up interacting mostly with people on your time zone region.

Some events are replaying their talks at different time zones so everyone has the opportunity to attend all sessions. But what’s the fun of watching some past talk, when even the speaker is no longer available for questions? I can watch it next month and it will be the same.

Bandwidth

I just mentioned those people that couldn’t travel to your face to face event but can attend your online event. If only that was as truthful as it sounds.

Not everyone has a good internet connection at home. We can even say not everyone has an internet connection at hand to watch and participate on your event. Thinking otherwise just shows your privilege and how you forget not everyone has access to a good and stable bandwidth. Specially at home.

For some people it is easier to travel to your face to face event than to get a good bandwidth to watch the streamed talks.

So this group of people that wouldn’t travel to your event but would attend your online event just keep shrinking. Your real potential attendees are those in the right time zone and the right technological requirements.

Shadow Audience on Online Events

This one is for speakers. There’s no way your talk will be the same if you can’t see, hear and feel the audience as in face to face. Online events, no matter if the audience has their cameras on, are isolating. It is never the same.

Not having feedback on how well your talk is being received (or even worse: seeing a lot of cameras of distracted attendees doing something else) is a huge handicap that even the most experienced speaker will suffer. Not knowing if you should speed up or slow down to adapt to your audience is weird. Not knowing if your audience is laughing at your jokes or thinking you are a weirdo.

And probably your speakers are not familiar with your platform, which means they will be extra nervous of messing up. If the mic fails on a room full of people, you notice immediately and you can even just talk louder while it gets fixed. If the mic fails on an online event, you may not even notice until several minutes have already passed.

All these issues lead to talks not as good as those made face to face.

Disengagement

Let’s be fair: attendees will not be physically disconnected from their daily routines as they are when traveling to your event. Most of them will keep going to work, passing time with their families and even skipping some talks to do house work. These changes of context means losing attention.

Your event is just the side dish of their day to day, an extra appendix. They will never be as engaged as if they had to travel to a venue and be there physically.

Language Barrier on Online Events

I know, I know what you are thinking. No, María, this problem is the same, no matter if you are face to face or online. Let me guess: you are an English native speaker.

Even when having reliable broadband and enjoying high quality smooth 60fps video, understanding a foreign speaker over video is harder than in person. Maybe because the audience is shadowed and the speaker has no feedback on how well it is being received. Or maybe it is just because we speak differently face to face than online. Maybe we are missing corporal language. But it happens.

This could be fixed with subtitles on most cases. But, what kind of subtitles? The ones that run on the fly and so frequently confuse technical terms making the subtitle confusing? Or the ones that are attached to a pre-recorded video so the speaker is not really there?

No random meetings

As you don’t have to move from one room to the next, sit next to someone on a room or even mingle during the coffee breaks, you just lose all those opportunities to meet both friends and strangers.

Your event will be missing a space where attendees can meet more or less randomly. Some events are matching random attendees on private rooms or have general room halls for random chatting. And that’s a good idea, but still, it doesn’t replace introducing yourself to someone specific you feel is a good match when you see that person talking to someone you know or having a break and being open to engage in conversation.

Do you want to have an Ice Breaker or a Gala Dinner? Maybe you can try Mozilla hub for that. Too over complicated? High hardware requirements? But what other social platforms could be there that go beyond a chat and are not a simple video conference? How can we mimic this walking over the venue and saying hi to old friends or asking someone about their shirt drawing?

Financial Side

Contrary to popular belief, if you really want to have an online event that can compete with a face to face event, the cost will not be cheap. The proper software stack with the proper bandwidth can be as costly as the venue. And whatever you may be saving on catering you will spend in making the event more social and engaging.

It’s true that a basic online event can be done very cheaply, as I demonstrate on the next section. But still, it will be a cheap event and will not patch many of the social drawbacks exposed.

If you want to make the conference online, you have to work towards more lines of economic viability than in traditional face to face events. Because there are many things you can’t offer now (being the merchandise placement the most obvious one).

Sponsorship of Online Events

Obviously you will try to keep sponsors on board. But it would be insane to think sponsors will be willing to pay the same prices as in traditional events if you don’t offer anything above and beyond.

If your conference has some kind of sponsorship booth area, you need some kind of replacement. How do you plan to do that? And how are you going to make attendees go through it? Now you don’t have halls where the booths are showing fancy stuff.

If you host some kind of Business to Business slot, are you just going to run it as a normal online video session? How are you going to help them interact?

Tickets

Most face to face events reconverted to online events are “in panic” offering free tickets, which will make it difficult for any online event to sell tickets at a reasonable price in the near future.

To make sure people would want to pay for your specific event, you must prepare some extras besides the streaming. You have to offer some kind of exclusive content to paid attendees besides the basic chats on the streamings to make sure the ticket price is worth it.

Maybe promote more panels where only paid users can participate and ask questions? Adding prizes or badges to users that participate more or attend more sessions?

Some events are offering open price donation tickets. Which means, you can have the ticket for the price you deem it worth. I have seen (from the inside) a number of events doing this. I won’t share actual numbers but trust me: don’t count with this money.

Goodies

Have you thought of an online shop to sell goodies branded as the conference? Difficult to estimate if this will work if you haven’t sell goodies before, but tech conference attendees are well known for collecting branded shirts. This is worth a try.

Dare Donations

This was an idea that popped out watching the Games Done Quick events. They keep people engaged and donating money through a whole week. They propose “dares” and people donate to vote what to do. You have to find something that make attendees engage and control somehow the type of content they are going to see.

For example: If between 9:00 and 10:00 we get a minimum amount in donations, then some speaker will run a live demo on their presentation instead of a video. Or vote with donations what question to ask on a panel. Maybe even choose between two different talks.

This works awesomely well on the GDC but the GDC is a fun event with no consequence and that’s not the tone with most tech conferences. I have no way to estimate if this will work on our case. It may work if you choose the right “dares” and speakers are willing to collaborate.

Probably easier on FOSS conferences, where the camaraderie is a common denominator. Weirder in other types of conferences, specially academic ones.

Software Stack of Online Events

There are many valid software stacks to use for your online events. Here is a summary of what I have experimented with and what I am comfortable with.

Let’s start with a basic stack that covers minimum functionality for a cheap price and then proceed to more wholesome solutions.

The Basic Stack

This is the basic stack I have found that covers the minimum needed to offer a proper online conference: talk streaming and access to a VIP chat. The price would be ridiculously cheap if you use one of the popular free streamings like Twitch or YouTube. And remember that you can organize a full online conference using only free and open source software.

OBS Studio

OBS Studio is an old friend of streamers. And also FOSS. No matter what streaming platform you are going to use (something free or some custom server), it works. You can have combined screens and cams and different audio sources to get a beautifully orchestrated output.

If you follow gamer streamers, you will probably be familiar with OBS already.

If OBS Studio is too complex to use for you, your moderators or even your speakers, you can always try StreamYard. It offers a similar functionality on a web browser friendly interface. It will allow you to easily stream to popular free platforms too. Personally I am very satisfied with this tool, even if it is not FOSS. It is really easy to use and the hardware and knowledge requirements are low.

Mattermost

To cover the social side, Mattermost is my preferred option. It is a Slack-like messaging platform where you can offer special channels to your attendees according to their ticket (free or paid). There, you can have access to all social interactions; like asking the speakers, meeting other attendees or discussing on common rooms.

Website

Although this stack covers all the minimal requirements for an online conference, it is messy and with decoupled parts. Your attendees may feel lost.

Attendees will have two windows or tabs in the browser: Mattermost and the streaming. The links to each streaming talk may be on some schedule in some webpage. It will be worth adding some developing time to create a website that contains everything on an usable format and gives some sense of being a single component.

Offline Talks for Online Events

If you are afraid your software stack may fail at the worst moment, you can ask your speakers to send their recorded presentations before the online event. You can even ask them for optional subtitles to overcome any language barrier.

Asking for recordings in advance gives you time to test if the video is right and schedule the release at the proper time and date. This reduces a lot the possibility of real time failures. Also, this way you make sure all videos have the proper duration, making the schedule more stable.

This gives the speakers also the opportunity to edit their videos on a fancy friendly way, adding special effects or anything that may be too risky to run on real time.

The speakers can also be asked to be at that proper time and date on the chat to interact with the attendees so the Q&A can run as usual.

But the event will probably look less dynamic, with all these pre-recorded talks. No cross-referencing talks you just watched, no possible change mid talk because someone made a good point.

Promising Software Stacks

If you don’t want to use the free streaming options and/or you are looking for a more holistic solution, there are some platforms and software stacks you can check. Take into account that these options require you to have the proper bandwidth, which means, they are going to be more expensive than the previously described minimum viable plan.

So the first thing you have to ask yourself before jumping into any of these platforms is: is it worth the risk of using our own/rented streaming servers instead of something like YouTube or Twitch?

The most attractive (to me) alternatives right now are the following.

BigBlueButton

BBB is a FOSS Conferencing System for online learning. Not designed as a conference software, it provides the users with class-like rooms and a chat per room. As a tool for teachers I felt it was very complete. As a tool for an online event, I felt things missing. It felt like many disconnected talks, not like a full schedule.

Venueless

From the creators of Pretix and Pretalx, Venueless aspires to be the FOSS option for conferences. It is built on top of Big Blue Button, offering a wrapper that should make the experience more “conference-like”. I have to confess I haven’t tried it yet, but seeing how Pretix and Pretalx work, I’m confident this is going to be a big one. If you are considering using BBB, you should take a look at this one. If only because it provides features on top of it.

Both options have the huge advantage of being FOSS, which means, if you don’t like or need something in particular, you can dedicate your own developers into contributing for it. Even if you use a third party hosting, with proper care your contributions will make it to the main release so any third party hosting will offer them for you.

Yet Another Browser Mud

Based on traditional text-based online game spaces such as MUDs and MOOs, this FOSS platform for online events could be at least a nice secondary interface for your event.

Hopin

Hopin.to was used on ApacheCON. It is not FOSS, which means you are completely dependent on what the company behind it wants to implement. For me, that’s a big red flag, specially on online events, which is something the software stacks are not yet mature enough. You cannot be confident yet that a restricted licensed software will be mature enough to provide you with everything you need.

On the social side, Hopin offered fancy features like matching you with random attendees, mimicking what a coffee break would be, or offering specific booth rooms so you can talk with sponsors.

Truth is, I am usually very social on face to face events and talk with random people and visit the booths. But nothing was calling me to use these features on the online version of this conference. It’s not as if you see someone you want to talk to in the coffee break so you just approach them. It looks forced to me. The risk of matching a complete stranger with nothing in common and having an uncomfortable conversation was high.

Online Events Experiences

I’m sure all of you have some experiences with online events by now. Let me share some insights I have experienced and why I still think online events can’t replace face to face ones.

EsLibre experience

The EsLibre used Big Blue Button as the main conference platform, with RocketChat as the sidechat and MozillaHub for the Gala Dinner and social “after beers”. The experience was pretty good… except everything was disconnected, as we were using three different components completely decoupled.

And still, not the same as a face to face.

ApacheCON experience

ApacheCON used Hopin. It was in general a good experience, well organized, and we had more attendees than ever. But that may be misleading: most of the almost six thousand registered attendees logged in at some point in the event, but you rarely saw more than a thousand connected at the same time.

We used the “three timezone region” approach, which in the end led us to have like three different groups of attendees that didn’t mingle among each other.

Conclusion

Online Events can’t match Social Face to Face Events

At least, not for me, not with the technology and the social capabilities we have right now.

I really really hope this post ages badly. Because there’s a lot of people that can’t travel to on site events and they are missing all the awesomeness of a full event experience.

And, truth be told, having everyone traveling several times per year to different conferences is not ecological, not sustainable in the long term.

Maybe with experiments like the one from the Roguelike Celebration or the attempt at Google, we will be able to get closer to what we need.

en_GBEnglish (UK)