Kamelets

Introduction

Kamelets (Kamel route snippets) are a new concept introduced in Camel K that allow users to connect to external systems via a simplified interface, hiding all the low level details about how those connections are implemented.

A Kamelet can act as "source" of data or alternatively as "sink": a source allows to consume data from an external system, while a sink can let you send data to an external system or execute a particular action and get a result.

For example, a "twitter-stream" Kamelet may allow a user to consume all tweets containing a specified keyword. Another Kamelet "twitter-post" may allow the user to publish some data as a tweet. Those are simple examples: experienced Camel developers know how to map those Kamelets into Camel endpoint URIs directly. But, in general, a Kamelet is not expected to map 1:1 a Camel component. Kamelets are route templates, so they can be much more complex.

Being made of pure Camel DSL, Kamelets can embody all the logic that allows to consume or produce data from public SaaS or enterprise systems and only expose to the final users a clean interface that describes the expected parameters, input and output: just like an OpenAPI spec.

For example, a "store-orders" Kamelet may be used to consume all events related to orders created in a customer enterprise system. The Kamelet’s interface will define what parameters should be provided (e.g. filters on the type of order) and what’s the expected datashape of the produced events. Internally, the Kamelet defines how such events will be generated: it may involve connections to multiple systems using different protocols, transformations and so on. But everything will be hidden to the end user.

Kamelets are the fundamental unit of abstraction in the next-gen architecture of Apache Camel K. A system as a whole can be technically described as the set of operations that you can do with it: if you use the language of Kamelets to describe a specific system, then other users can have access to all those operations with ease, no matter how complicated is the internal logic underlying all those operations. Comparing it to the classical way of using Apache Camel, where developers used to write complex routes (containing both high-level and low-level details) to integrate systems, the Kamelet model tends to reduce the complexity by encapsulating low-level details into reusable components.

Kamelets are also expected to be rendered on visual tools that will provide additional value to the end users. They are generic connectors that can be used in multiples ways, depending on the context, so each UIs can use them for its own purpose.

Kamelet Example

Speaking technically, a Kamelet is a resource that can be installed on any Kubernetes cluster. The following is an example of Kamelet that we’ll use to discuss the various parts:

telegram-text-source.kamelet.yaml

apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
  name: telegram-text-source (1)
  annotations: (2)
    camel.apache.org/kamelet.icon: "data:image/svg+xml;base64,PD94bW..."
  labels: (3)
    camel.apache.org/kamelet.type: "source"
spec:
  definition: (4)
    title: "Telegram Text Source"
    description: |-
      Receive all text messages that people send to your telegram bot.

      # Instructions
      Description can include Markdown and guide the final user to configure the Kamelet parameters.
    required:
      - botToken
    properties:
      botToken:
        title: Token
        description: The token to access your bot on Telegram
        type: string
        x-descriptors:
        - urn:alm:descriptor:com.tectonic.ui:password

  types: (5)
    out:
      mediaType: text/plain
      # schema:
  flow: (6)
    from:
      uri: telegram:bots
      parameters:
        authorizationToken: "#property:botToken"
      steps:
        - convert-body-to:
            type: "java.lang.String"
            type-class: "java.lang.String"
            charset: "UTF8"
        - filter:
            simple: "${body} != null"
        - log: "${body}"
        - to: "kamelet:sink"

1	The Kamelet ID, to be used in integrations that want to leverage the Kamelet
2	Annotations such as icon provide additional display features to the Kamelet
3	Labels allow users to query Kamelets e.g. by kind ("source" vs. "sink")
4	Description of the Kamelets and parameters in JSON-schema specification format
5	The media type of the output (can include a schema)
6	The route template defining the behavior of the Kamelet

At a high level (more details are provided later), a Kamelet resource describes:

A metadata section containing the ID (metadata → name) of the Kamelet and other information, such as the type of Kamelet (source or sink)
A JSON-schema specification (definition) containing a set of parameters that you can use to configure the Kamelet
An optional section containing information about input and output expected by the Kamelet (types)
A Camel flow in YAML DSL containing the implementation of the Kamelet (flow)

Once installed on a Kubernetes namespace, the Kamelet can be used by any integration in that namespace.

Kamelets can be installed on a Kubernetes namespace with a simple command:

kubectl apply -f yourkamelet.kamelet.yaml

Kamelets are standard YAML files, but their common extension is .kamelet.yaml to help IDEs to recognize them and provide auto-completion (in the future).

Using Kamelets in Integrations

Kamelets can be used in integrations as if they were standard Camel components. For example, suppose that you’ve created the telegram-text-source Kamelet in the default namespace on Kubernetes, then you can write the following integration to use the Kamelet:

example.groovy

from('kamelet:telegram-text-source?botToken=XXXXYYYY')
  .to('log:INFO')

URI properties ("botToken") match the corresponding parameters in the Kamelet definition

Kamelets can also be used multiple times in the same route definition. This happens usually with sink Kamelets.

Suppose that you’ve defined a Kamelet named "my-company-log-sink" in your Kubernetes namespace, then you can write a route like this:

example.groovy

from('kamelet:telegram-text-source?botToken=XXXXYYYY')
  .to("kamelet:my-company-log-sink?bucket=general")
  .filter().simple('${body} contains "Camel"')
    .to("kamelet:my-company-log-sink?bucket=special")

The "my-company-log-sink" will obviously define what it means to write a log in the enterprise system and what is concretely a "bucket".

Configuration

When using a Kamelet, the instance parameters (e.g. "botToken", "bucket") can be passed explicitly in the URI or you can use properties. Properties can be also loaded implicitly by the operator from Kubernetes secrets (see below).

1. URI based configuration

You can configure the Kamelet by passing directly the configuration parameters in the URI, as in:

from("kamelet:telegram-text-source?botToken=the-token-value")
// ...

In this case, "the-token-value" is passed explicitly in the URI (you can also pass a custom property placeholder as value).

2. Property based configuration

An alternative way to configure the Kamelet is to provide configuration parameters as properties of the integration.

Taking for example a different version of the integration above:

from('kamelet:telegram-text-source')
  .to("kamelet:my-company-log-sink")
  .filter().simple('${body} contains "Camel"')
    .to("kamelet:my-company-log-sink/mynamedconfig")

The integration above does not contain URI query parameters and the last URI ("kamelet:my-company-log-sink/mynamedconfig") contains a path parameter with value "mynamedconfig"

The integration above needs some configuration in order to run properly. The configuration can be provided in a property file:

example.properties

# Configuration for the Telegram source Kamelet
camel.kamelet.telegram-text-source.botToken=the-token-value

# General configuration for the Company Log Kamelet
camel.kamelet.my-company-log-sink.bucket=general
# camel.kamelet.my-company-log-sink.xxx=yyy

# Specific configuration for the Company Log Kamelet corresponding to the named configuration "mynamedconfig"
camel.kamelet.my-company-log-sink.mynamedconfig.bucket=special
# When using "kamelet:my-company-log-sink/mynamedconfig", the bucket will be "special", not "general"

Then the integration can be run with the following command:

kamel run example.groovy --property-file example.properties

3. Implicit configuration using secrets

Property based configuration can also be used implicitly by creating secrets in the namespace that will be used to determine the Kamelets configuration.

To use implicit configuration via secret, we first need to create a configuration file holding only the properties of a named configuration.

mynamedconfig.properties

# Only configuration related to the "mynamedconfig" named config
camel.kamelet.my-company-log-sink.mynamedconfig.bucket=special
# camel.kamelet.my-company-log-sink.mynamedconfig.xxx=yyy

We can create a secret from the file and label it so that it will be picked up automatically by the operator:

# Create the secret from the property file
kubectl create secret generic my-company-log-sink.mynamedconfig --from-file=mynamedconfig.properties
# Bind it to the named configuration "mynamedconfig" of the "my-company-log-sink" Kamelet
kubectl label secret my-company-log-sink.mynamedconfig camel.apache.org/kamelet=my-company-log-sink camel.apache.org/kamelet.configuration=mynamedconfig

You can now write an integration that uses the Kamelet with the named configuration:

example.groovy

from('timer:tick')
  .setBody().constant('Hello')
  .to('kamelet:my-company-log-sink/mynamedconfig')

You can run this integration without specifying other parameters, the Kamelet endpoint will be implicitly configured by the Camel K operator that will automatically mount the secret into the integration Pod.

Binding Kamelets

In some contexts (for example "serverless") users often want to leverage the power of Apache Camel to be able to connect to various sources/sinks, without doing additional processing (such as tranformations or other enterprise integration patterns).

A common use case is that of Knative Sources, for which the Apache Camel developers maintain the Knative CamelSources. Kamelets represent an evolution of the model proposed in CamelSources, but they allow using the same declarative style of binding, via a resource named KameletBinding.

Binding to a Knative Destination

A KameletBinding allows to declaratively move data from a system described by a Kamelet towards a Knative destination (or other kind of destinations, in the future), or from a Knative channel/broker to another external system described by a Kamelet.

For example, here’s an example of binding:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: telegram-text-source-to-channel
spec:
  source: (1)
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: telegram-text-source
    properties:
      botToken: the-token-here
  sink: (2)
    ref:
      kind: InMemoryChannel
      apiVersion: messaging.knative.dev/v1
      name: messages

1	Reference to the source that provides data
2	Reference to the sink where data should be sent to

This binding takes the telegram-text-source Kamelet, configures it using specific properties ("botToken") and makes sure that messages produced by the Kamelet are forwarded to the Knative InMemoryChannel named "messages".

Note that source and sink are specified declaratively as standard Kubernetes object references.

The example shows how we can reference the "telegram-text-source" resource in a KameletBinding. It’s contained in the source section because it’s a Kamelet of type "source". A Kamelet of type "sink", by contrast, can only be used in the sink section of a KameletBinding.

Under the covers, a KameletBinding creates an Integration resource that implements the binding, but this is transparent to the end user.

Binding to a Kafka Topic

The example seen in the previous paragraph can be also configured to push data a Strimzi Kafka topic (Kamelets can be also configured to pull data from topics).

To do so, you need to:

Install Strimzi on your cluster
Create a Strimzi Kafka cluster using plain listener and no authentication
Create a Strimzi KafkaTopic named my-topic

Refer to the Strimzi documentation for instructions on how to do that.

The following binding can be created to push data into the my-topic topic:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: telegram-text-source-to-kafka
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: telegram-text-source
    properties:
      botToken: the-token-here
  sink:
    ref: (1)
      kind: KafkaTopic
      apiVersion: kafka.strimzi.io/v1beta1
      name: my-topic

1	Kubernetes reference to a Strimzi KafkaTopic

After creating it, messages will flow from Telegram to Kafka.

Binding to an explicit URI

An alternative way to use a KameletBinding is to configure the source/sink to be an explicit Camel URI. For example, the following binding is allowed:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: telegram-text-source-to-channel
spec:
  source:
    ref:
      kind: Kamelet
      apiVersion: camel.apache.org/v1alpha1
      name: telegram-text-source
    properties:
      botToken: the-token-here
  sink:
    uri: https://mycompany.com/the-service (1)

1	KameletBinding with explicitly URI

This KameletBinding explicitly defines an URI where data is going to be pushed.

the uri option is also conventionally used in Knative to specify a non-kubernetes destination. To comply with the Knative specifications, in case an "http" or "https" URI is used, Camel will send CloudEvents to the destination.

Error Handling

You can configure an error handler in order to specify what to do when some event ends up with failure. See Kamelet Bindings Error Handler User Guide for more detail.

Trait via annotations

You can easily tune your KameletBinding with traits configuration adding .metadata.annotations. Let’s have a look at the following example:

apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
  name: timer-2-log-annotation
  annotations: (1)
    trait.camel.apache.org/logging.level: DEBUG
    trait.camel.apache.org/logging.color: "false"
spec:
  source:
    uri: timer:foo
  sink:
    uri: log:bar

1	Include `.metadata.annotations` to specify the list of traits we want to configure

In this example, we’ve set the logging trait to specify certain configuration we want to apply. You can do the same with all the traits available, just by setting trait.camel.apache.org/trait-name.trait-property with the expected value.

if you need to specify an array of values, the syntax will be trait.camel.apache.org/trait.conf: "[\"opt1\", \"opt2\", …]"

Troubleshooting

A Kamelet is translated into a Route used from the Ìntegration. In order to troubleshoot any possible issue, you can have a look at the dedicated troubleshoot section.

Kamelet Specification

We’re now going to describe the various parts of the Kamelet in more details.

Metadata

The metadata section contains important information related to the Kamelet as Kubernetes resource.

Table 1. Metadata Fields
name	Description	Type	Example
`name`	ID of the Kamelet, used to refer to the Kamelet in external routes	`string`	E.g. `telegram-text-source`
`namespace`	The Kubernetes namespace where the resource is installed	`string`

The following annotations and labels are also defined on the resource:

Table 2. Annotations
name	Description	Type	Example
`camel.apache.org/kamelet.icon`	An optional icon for the Kamelet in URI data format	`string`	E.g. `data:image/svg+xml;base64,PD94bW…`
`trait.camel.apache.org/trait-name.trait-property`	An optional configuration setting for a trait	`string`	E.g. `trait.camel.apache.org/logging.level: DEBUG`

Table 3. Labels
name	Description	Type	Example
label: `camel.apache.org/kamelet.type`	Indicates if the Kamelet can be used as source or sink	enum: `source`, `sink`	E.g. `source`

Definition

The definition part of a Kamelet contains a valid JSON-schema document describing general information about the Kamelet and all defined parameters.

Table 4. Definition Fields
name	Description	Type	Example
`title`	Display name of the Kamelet	`string`	E.g. `Telegram Text Source`
`description`	A markdown description of the Kamelet	`string`	E.g. `Receive all text messages that people send to your telegram bot…`
`required`	List of required parameters (complies with JSON-schema spec)	array: `string`
`properties`	Map of properties that can be configured on the Kamelet	map: `string` → `schema`

Each property defined in the Kamelet has its own schema (normally a flat schema, containing only 1 level of properties). The following table lists some common fields allowed for each property.

Table 5. Definition Parameters
name	Description	Type	Example
`title`	Display name of the property	`string`	E.g. `Token`
`description`	Simple text description of the property	`string`	E.g. `The token to access your bot on Telegram`
`type`	JSON-schema type of the property	`string`	E.g. `string`
`x-descriptors`	Specific aids for the visual tools	array: `string`	E.g. `- urn:alm:descriptor:com.tectonic.ui:password` displays the property as a password field in a tectonic-type form

Data shapes

Kamelets are designed to be plugged as sources or sinks in more general routes, so they can accept data as input and/or produce their own data. To help visual tools and applications to understand how to interact with the Kamelet, the specification of a Kamelet includes also information about type of data that it manages.

# ...
spec:
  # ...
  types:
    out: (1)
      mediaType: application/json
      schema: (2)
        properties:
          # ...

1	Defines the type of the `output`
2	JSON-schema definition of the output

Data shape can be indicated for the following channels:

in: the input of the Kamelet, in case the Kamelet is of type sink
out: the output of the Kamelet, for both source and sink Kamelets
error: an optional error data shape, for both source and sink Kamelets

Data shapes contain the following information:

Table 6. Data Shape Options
name	Description	Type	Example
`mediaType`	The media type of the data	`string`	E.g. `application/json`
`schema`	An optional JSON-schema definition for the data	`object`

Flow

Each Kamelet contains a YAML-based Camel DSL that provides the actual implementation of the connector.

For example:

spec:
  # ...
  flow:
    from:
      uri: telegram:bots
      parameters:
        authorizationToken: "#property:botToken"
      steps:
        - convert-body-to:
            type: "java.lang.String"
            type-class: "java.lang.String"
            charset: "UTF8"
        - filter:
            simple: "${body} != null"
        - log: "${body}"
        - to: "kamelet:sink"

Source and sink flows will connect to the outside route via the kamelet:source or kamelet:sink special endpoints: - A source Kamelet must contain a call to kamelet:sink - A sink Kamelet must start from kamelet:source

The kamelet:source and kamelet:sink endpoints are special endpoints that are only available in Kamelet route templates and will be replaced with actual references at runtime.

Kamelets contain a single route template written in YAML DSL, as in the previous example.

Kamelets, however, can also contain additional sources in the spec → sources field. Those sources can be of any kind (not necessarily route templates) and will be added once to all the integrations where the Kamelet is used. They main role is to do advanced configuration of the integration context where the Kamelet is used, such as registering beans in the registry or adding customizers.