Simple JMS Batch
Since Camel 2.16
Only consumer is supported
SJMS Batch is a specialized component for highly performant, transactional batch consumption from a JMS queue. It can be thought of as a hybrid of a consumer-only component and an aggregator.
A common use case in Camel is to consume messages from a queue and
aggregate them before sending the aggregated state to another endpoint.
In order to ensure that data is not lost if the system performing the
processing fails, it is typically consumed within a transaction from the
queue, and once aggregated stored in a persistent
AggregationRepository
, such as the one found in the
JDBC Component.
The behavior of the aggregator pattern involves fetching data from the
AggregationRepository
before an incoming message is aggregated, and
writing back the result afterwards. By nature, the reads and writes take
progressively longer as the number of aggregated artifacts increases. A
rough example of this using arbitrary time units that demonstrates the
impact of this is as follows:
Item | Read Time | Write Time | Total Time |
---|---|---|---|
0 |
0 |
1 |
1 |
1 |
1 |
2 |
4 |
2 |
2 |
3 |
9 |
3 |
3 |
4 |
16 |
4 |
4 |
5 |
25 |
5 |
5 |
6 |
36 |
6 |
6 |
7 |
49 |
7 |
7 |
8 |
64 |
8 |
8 |
9 |
81 |
9 |
9 |
10 |
100 |
In contrast, consumption performance using the SJMS Batch component is
linear. Each message is consumed and aggregated using an
AggregationStrategy
before the next one is fetched. As all of the
consumption and aggregation is performed in a single JMS transaction no
external storage is required to persist the intermediate state - this
avoids the read and write costs described above. In practice, this
yields multiple orders of magnitude higher throughput.
Once a completion condition is met, either by size or period since first
message, the aggregated Exchange
is passed into the route. During the
processing of this Exchange
, if an exception is thrown or the system
shuts down, all of the original consumed messages end up back on the
queue (or are placed on the dead-letter queue depending on the broker
configuration).
Unlike using a regular aggregator, there is no facility for an aggregation condition, meaning that it is not possible to batch consume into multiple groups of messages. All consumed messages are aggregated together into a single batch.
Multiple JMS consumer support is available which allows you to consume in parallel using the one route, and at the same time use facilities like JMS message groups to group related messages.
Maven users will need to add the following dependency to their pom.xml
for this component:
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-sjms</artifactId>
<version>x.x.x</version>
<!-- use the same version as your Camel core version -->
</dependency>
URI format
sjms:[queue:]destinationName[?options]
Where destinationName
is a JMS queue. By default, the
destinationName
is interpreted as a queue name.
sjms:FOO.BAR
You can include the optional queue:
prefix, if you prefer:
sjms:queue:FOO.BAR
Topic consumption is not supported, as there is no advantage to using batch consumption within that context. Topic messages are usually non-persistent, and loss is acceptable. If consumed within a transaction that fails, a topic message will likely not be redelivered by the broker. A plain SJMS consumer endpoint can be used in conjunction with a regular non-persistence backed aggregator in this scenario.
Component Options and Configurations
The Simple JMS Batch component supports 6 options, which are listed below.
Name | Description | Default | Type |
---|---|---|---|
bridgeErrorHandler (consumer) |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
boolean |
asyncStartListener (advanced) |
Whether to startup the consumer message listener asynchronously, when starting a route. For example if a JmsConsumer cannot get a connection to a remote JMS broker, then it may block while retrying and/or failover. This will cause Camel to block while starting routes. By setting this option to true, you will let routes startup, while the JmsConsumer connects to the JMS broker using a dedicated thread in asynchronous mode. If this option is used, then beware that if the connection could not be established, then an exception is logged at WARN level, and the consumer will not be able to receive messages; You can then restart the route to retry. |
false |
boolean |
autowiredEnabled (advanced) |
Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. |
true |
boolean |
connectionFactory (advanced) |
A ConnectionFactory is required to enable the SjmsBatchComponent. |
ConnectionFactory |
|
recoveryInterval (advanced) |
Specifies the interval between recovery attempts, i.e. when a connection is being refreshed, in milliseconds. The default is 5000 ms, that is, 5 seconds. |
5000 |
int |
headerFilterStrategy (filter) |
To use a custom org.apache.camel.spi.HeaderFilterStrategy to filter header to and from Camel message. |
HeaderFilterStrategy |
The Simple JMS Batch endpoint is configured using URI syntax:
sjms-batch:destinationName
with the following path and query parameters:
Path Parameters (1 parameters):
Name | Description | Default | Type |
---|---|---|---|
destinationName |
Required The destination name. Only queues are supported, names may be prefixed by 'queue:'. |
String |
Query Parameters (23 parameters):
Name | Description | Default | Type |
---|---|---|---|
aggregationStrategy (consumer) |
Required The aggregation strategy to use, which merges all the batched messages into a single message |
AggregationStrategy |
|
allowNullBody (consumer) |
Whether to allow sending messages with no body. If this option is false and the message body is null, then an JMSException is thrown. |
true |
boolean |
bridgeErrorHandler (consumer) |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
boolean |
completionInterval (consumer) |
The completion interval in millis, which causes batches to be completed in a scheduled fixed rate every interval. The batch may be empty if the timeout triggered and there was no messages in the batch. Notice you cannot use both completion timeout and completion interval at the same time, only one can be configured. |
1000 |
int |
completionPredicate (consumer) |
The completion predicate, which causes batches to be completed when the predicate evaluates as true. The predicate can also be configured using the simple language using the string syntax. You may want to set the option eagerCheckCompletion to true to let the predicate match the incoming message, as otherwise it matches the aggregated message. |
String |
|
completionSize (consumer) |
The number of messages consumed at which the batch will be completed |
200 |
int |
completionTimeout (consumer) |
The timeout in millis from receipt of the first first message when the batch will be completed. The batch may be empty if the timeout triggered and there was no messages in the batch. Notice you cannot use both completion timeout and completion interval at the same time, only one can be configured. |
500 |
int |
consumerCount (consumer) |
The number of JMS sessions to consume from |
1 |
int |
eagerCheckCompletion (consumer) |
Use eager completion checking which means that the completionPredicate will use the incoming Exchange. As opposed to without eager completion checking the completionPredicate will use the aggregated Exchange. |
false |
boolean |
includeAllJMSXProperties (consumer) |
Whether to include all JMSXxxx properties when mapping from JMS to Camel Message. Setting this to true will include properties such as JMSXAppID, and JMSXUserID etc. Note: If you are using a custom headerFilterStrategy then this option does not apply. |
false |
boolean |
mapJmsMessage (consumer) |
Specifies whether Camel should auto map the received JMS message to a suited payload type, such as javax.jms.TextMessage to a String etc. See section about how mapping works below for more details. |
true |
boolean |
pollDuration (consumer) |
The duration in milliseconds of each poll for messages. completionTimeOut will be used if it is shorter and a batch has started. |
1000 |
int |
sendEmptyMessageWhenIdle (consumer) |
If using completion timeout or interval, then the batch may be empty if the timeout triggered and there was no messages in the batch. If this option is true and the batch is empty then an empty message is added to the batch so an empty message is routed. |
false |
boolean |
exceptionHandler (consumer) |
To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
ExceptionHandler |
|
exchangePattern (consumer) |
Sets the exchange pattern when the consumer creates an exchange. There are 3 enums and the value can be one of: InOnly, InOut, InOptionalOut |
ExchangePattern |
|
asyncStartListener (advanced) |
Whether to startup the consumer message listener asynchronously, when starting a route. For example if a JmsConsumer cannot get a connection to a remote JMS broker, then it may block while retrying and/or failover. This will cause Camel to block while starting routes. By setting this option to true, you will let routes startup, while the JmsConsumer connects to the JMS broker using a dedicated thread in asynchronous mode. If this option is used, then beware that if the connection could not be established, then an exception is logged at WARN level, and the consumer will not be able to receive messages; You can then restart the route to retry. |
false |
boolean |
headerFilterStrategy (advanced) |
To use a custom HeaderFilterStrategy to filter header to and from Camel message. |
HeaderFilterStrategy |
|
jmsKeyFormatStrategy (advanced) |
Pluggable strategy for encoding and decoding JMS keys so they can be compliant with the JMS specification. Camel provides two implementations out of the box: default and passthrough. The default strategy will safely marshal dots and hyphens (. and -). The passthrough strategy leaves the key as is. Can be used for JMS brokers which do not care whether JMS header keys contain illegal characters. You can provide your own implementation of the org.apache.camel.component.jms.JmsKeyFormatStrategy and refer to it using the # notation. |
JmsKeyFormatStrategy |
|
keepAliveDelay (advanced) |
The delay in millis between attempts to re-establish a valid session. If this is a positive value the SjmsBatchConsumer will attempt to create a new session if it sees an Exception during message consumption. This delay value allows you to pause between attempts to prevent spamming the logs. If this is a negative value then the SjmsBatchConsumer will bail out and the consumer will not stop consuming new messages. The default is 5000 ms, that is, 5 seconds. |
5000 |
int |
messageCreatedStrategy (advanced) |
To use the given MessageCreatedStrategy which are invoked when Camel creates new instances of javax.jms.Message objects when Camel is sending a JMS message. |
MessageCreatedStrategy |
|
recoveryInterval (advanced) |
Specifies the interval between recovery attempts, i.e. when a connection is being refreshed, in milliseconds. The default is 5000 ms, that is, 5 seconds. |
5000 |
int |
synchronous (advanced) |
Sets whether synchronous processing should be strictly used, or Camel is allowed to use asynchronous processing (if supported). |
false |
boolean |
timeoutCheckerExecutorService (advanced) |
If using the completionInterval option a background thread is created to trigger the completion interval. Set this option to provide a custom thread pool to be used rather than creating a new thread for every consumer. |
ScheduledExecutorService |
The completionSize
endpoint attribute is used in conjunction with
completionTimeout
, where the first condition to be met will cause the
aggregated Exchange
to be emitted down the route.
Spring Boot Auto-Configuration
When using sjms with Spring Boot make sure to use the following Maven dependency to have support for auto configuration:
<dependency>
<groupId>org.apache.camel.springboot</groupId>
<artifactId>camel-sjms-starter</artifactId>
<version>x.x.x</version>
<!-- use the same version as your Camel core version -->
</dependency>
The component supports 27 options, which are listed below.
Name | Description | Default | Type |
---|---|---|---|
camel.component.sjms-batch.async-start-listener |
Whether to startup the consumer message listener asynchronously, when starting a route. For example if a JmsConsumer cannot get a connection to a remote JMS broker, then it may block while retrying and/or failover. This will cause Camel to block while starting routes. By setting this option to true, you will let routes startup, while the JmsConsumer connects to the JMS broker using a dedicated thread in asynchronous mode. If this option is used, then beware that if the connection could not be established, then an exception is logged at WARN level, and the consumer will not be able to receive messages; You can then restart the route to retry. |
false |
Boolean |
camel.component.sjms-batch.autowired-enabled |
Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. |
true |
Boolean |
camel.component.sjms-batch.bridge-error-handler |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
Boolean |
camel.component.sjms-batch.connection-factory |
A ConnectionFactory is required to enable the SjmsBatchComponent. The option is a javax.jms.ConnectionFactory type. |
ConnectionFactory |
|
camel.component.sjms-batch.enabled |
Whether to enable auto configuration of the sjms-batch component. This is enabled by default. |
Boolean |
|
camel.component.sjms-batch.header-filter-strategy |
To use a custom org.apache.camel.spi.HeaderFilterStrategy to filter header to and from Camel message. The option is a org.apache.camel.spi.HeaderFilterStrategy type. |
HeaderFilterStrategy |
|
camel.component.sjms-batch.recovery-interval |
Specifies the interval between recovery attempts, i.e. when a connection is being refreshed, in milliseconds. The default is 5000 ms, that is, 5 seconds. |
5000 |
Integer |
camel.component.sjms.autowired-enabled |
Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. |
true |
Boolean |
camel.component.sjms.bridge-error-handler |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
Boolean |
camel.component.sjms.connection-client-id |
The client ID to use when creating javax.jms.Connection when using the default org.apache.camel.component.sjms.jms.ConnectionFactoryResource. |
String |
|
camel.component.sjms.connection-count |
The maximum number of connections available to endpoints started under this component |
1 |
Integer |
camel.component.sjms.connection-factory |
A ConnectionFactory is required to enable the SjmsComponent. It can be set directly or set set as part of a ConnectionResource. The option is a javax.jms.ConnectionFactory type. |
ConnectionFactory |
|
camel.component.sjms.connection-max-wait |
The max wait time in millis to block and wait on free connection when the pool is exhausted when using the default org.apache.camel.component.sjms.jms.ConnectionFactoryResource. |
5000 |
Long |
camel.component.sjms.connection-password |
The password to use when creating javax.jms.Connection when using the default org.apache.camel.component.sjms.jms.ConnectionFactoryResource. |
String |
|
camel.component.sjms.connection-resource |
A ConnectionResource is an interface that allows for customization and container control of the ConnectionFactory. See Plugable Connection Resource Management for further details. The option is a org.apache.camel.component.sjms.jms.ConnectionResource type. |
ConnectionResource |
|
camel.component.sjms.connection-test-on-borrow |
When using the default org.apache.camel.component.sjms.jms.ConnectionFactoryResource then should each javax.jms.Connection be tested (calling start) before returned from the pool. |
true |
Boolean |
camel.component.sjms.connection-username |
The username to use when creating javax.jms.Connection when using the default org.apache.camel.component.sjms.jms.ConnectionFactoryResource. |
String |
|
camel.component.sjms.destination-creation-strategy |
To use a custom DestinationCreationStrategy. The option is a org.apache.camel.component.sjms.jms.DestinationCreationStrategy type. |
DestinationCreationStrategy |
|
camel.component.sjms.enabled |
Whether to enable auto configuration of the sjms component. This is enabled by default. |
Boolean |
|
camel.component.sjms.header-filter-strategy |
To use a custom org.apache.camel.spi.HeaderFilterStrategy to filter header to and from Camel message. The option is a org.apache.camel.spi.HeaderFilterStrategy type. |
HeaderFilterStrategy |
|
camel.component.sjms.jms-key-format-strategy |
Pluggable strategy for encoding and decoding JMS keys so they can be compliant with the JMS specification. Camel provides one implementation out of the box: default. The default strategy will safely marshal dots and hyphens (. and -). Can be used for JMS brokers which do not care whether JMS header keys contain illegal characters. You can provide your own implementation of the org.apache.camel.component.jms.JmsKeyFormatStrategy and refer to it using the # notation. The option is a org.apache.camel.component.sjms.jms.JmsKeyFormatStrategy type. |
JmsKeyFormatStrategy |
|
camel.component.sjms.lazy-start-producer |
Whether the producer should be started lazy (on the first message). By starting lazy you can use this to allow CamelContext and routes to startup in situations where a producer may otherwise fail during starting and cause the route to fail being started. By deferring this startup to be lazy then the startup failure can be handled during routing messages via Camel’s routing error handlers. Beware that when the first message is processed then creating and starting the producer may take a little time and prolong the total processing time of the processing. |
false |
Boolean |
camel.component.sjms.message-created-strategy |
To use the given MessageCreatedStrategy which are invoked when Camel creates new instances of javax.jms.Message objects when Camel is sending a JMS message. The option is a org.apache.camel.component.sjms.jms.MessageCreatedStrategy type. |
MessageCreatedStrategy |
|
camel.component.sjms.reconnect-back-off |
Backoff in millis on consumer pool reconnection attempts |
5000 |
Long |
camel.component.sjms.reconnect-on-error |
Try to apply reconnection logic on consumer pool |
true |
Boolean |
camel.component.sjms.timed-task-manager |
To use a custom TimedTaskManager. The option is a org.apache.camel.component.sjms.taskmanager.TimedTaskManager type. |
TimedTaskManager |
|
camel.component.sjms.transaction-commit-strategy |
To configure which kind of commit strategy to use. Camel provides two implementations out of the box, default and batch. The option is a org.apache.camel.component.sjms.TransactionCommitStrategy type. |
TransactionCommitStrategy |