camel-azure-storage-datalake-kafka-connector source configuration
Connector description: Camel Azure Datalake Gen2 Component
When using camel-azure-storage-datalake-kafka-connector as source make sure to use the following Maven dependency to have support for the connector:
<dependency>
<groupId>org.apache.camel.kafkaconnector</groupId>
<artifactId>camel-azure-storage-datalake-kafka-connector</artifactId>
<version>x.x.x</version>
<!-- use the same version as your Camel Kafka connector version -->
</dependency>To use this Source connector in Kafka connect you’ll need to set the following connector.class
connector.class=org.apache.camel.kafkaconnector.azurestoragedatalake.CamelAzurestoragedatalakeSourceConnectorThe camel-azure-storage-datalake source connector supports 81 options, which are listed below.
| Name | Description | Default | Required | Priority |
|---|---|---|---|---|
camel.source.path.accountName |
name of the azure account |
null |
false |
MEDIUM |
camel.source.path.fileSystemName |
name of filesystem to be used |
null |
false |
MEDIUM |
camel.source.endpoint.accountKey |
account key for authentication |
null |
false |
MEDIUM |
camel.source.endpoint.clientId |
client id for azure account |
null |
false |
MEDIUM |
camel.source.endpoint.clientSecret |
client secret for azure account |
null |
false |
MEDIUM |
camel.source.endpoint.clientSecretCredential |
client secret credential for authentication |
null |
false |
MEDIUM |
camel.source.endpoint.close |
Whether or not a file changed event raised indicates completion (true) or modification (false) |
null |
false |
MEDIUM |
camel.source.endpoint.closeStreamAfterRead |
check for closing stream after read |
null |
false |
MEDIUM |
camel.source.endpoint.dataCount |
count number of bytes to download |
null |
false |
MEDIUM |
camel.source.endpoint.dataLakeServiceClient |
service client of datalake |
null |
false |
MEDIUM |
camel.source.endpoint.directoryName |
directory of the file to be handled in component |
null |
false |
MEDIUM |
camel.source.endpoint.downloadLinkExpiration |
download link expiration time |
null |
false |
MEDIUM |
camel.source.endpoint.expression |
expression for queryInputStream |
null |
false |
MEDIUM |
camel.source.endpoint.fileDir |
directory of file to do operations in the local system |
null |
false |
MEDIUM |
camel.source.endpoint.fileName |
name of file to be handled in component |
null |
false |
MEDIUM |
camel.source.endpoint.fileOffset |
offset position in file for different operations |
null |
false |
MEDIUM |
camel.source.endpoint.maxResults |
maximum number of results to show at a time |
null |
false |
MEDIUM |
camel.source.endpoint.maxRetryRequests |
no of retries to a given request |
null |
false |
MEDIUM |
camel.source.endpoint.openOptions |
set open options for creating file |
null |
false |
MEDIUM |
camel.source.endpoint.path |
path in azure datalake for operations |
null |
false |
MEDIUM |
camel.source.endpoint.permission |
permission string for the file |
null |
false |
MEDIUM |
camel.source.endpoint.position |
This parameter allows the caller to upload data in parallel and control the order in which it is appended to the file. |
null |
false |
MEDIUM |
camel.source.endpoint.recursive |
recursively include all paths |
null |
false |
MEDIUM |
camel.source.endpoint.regex |
regular expression for matching file names |
null |
false |
MEDIUM |
camel.source.endpoint.retainUncommitedData |
Whether or not uncommitted data is to be retained after the operation |
null |
false |
MEDIUM |
camel.source.endpoint.serviceClient |
datalake service client for azure storage datalake |
null |
false |
MEDIUM |
camel.source.endpoint.sharedKeyCredential |
shared key credential for azure datalake gen2 |
null |
false |
MEDIUM |
camel.source.endpoint.tenantId |
tenant id for azure account |
null |
false |
MEDIUM |
camel.source.endpoint.timeout |
Timeout for operation |
null |
false |
MEDIUM |
camel.source.endpoint.umask |
umask permission for file |
null |
false |
MEDIUM |
camel.source.endpoint.userPrincipalNameReturned |
whether or not to use upn |
null |
false |
MEDIUM |
camel.source.endpoint.bridgeErrorHandler |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
false |
MEDIUM |
camel.source.endpoint.sendEmptyMessageWhenIdle |
If the polling consumer did not poll any files, you can enable this option to send an empty message (no body) instead. |
false |
false |
MEDIUM |
camel.source.endpoint.exceptionHandler |
To let the consumer use a custom ExceptionHandler. Notice if the option bridgeErrorHandler is enabled then this option is not in use. By default the consumer will deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
null |
false |
MEDIUM |
camel.source.endpoint.exchangePattern |
Sets the exchange pattern when the consumer creates an exchange. One of: [InOnly] [InOut] [InOptionalOut] |
null |
false |
MEDIUM |
camel.source.endpoint.pollStrategy |
A pluggable org.apache.camel.PollingConsumerPollingStrategy allowing you to provide your custom implementation to control error handling usually occurred during the poll operation before an Exchange have been created and being routed in Camel. |
null |
false |
MEDIUM |
camel.source.endpoint.backoffErrorThreshold |
The number of subsequent error polls (failed due some error) that should happen before the backoffMultipler should kick-in. |
null |
false |
MEDIUM |
camel.source.endpoint.backoffIdleThreshold |
The number of subsequent idle polls that should happen before the backoffMultipler should kick-in. |
null |
false |
MEDIUM |
camel.source.endpoint.backoffMultiplier |
To let the scheduled polling consumer backoff if there has been a number of subsequent idles/errors in a row. The multiplier is then the number of polls that will be skipped before the next actual attempt is happening again. When this option is in use then backoffIdleThreshold and/or backoffErrorThreshold must also be configured. |
null |
false |
MEDIUM |
camel.source.endpoint.delay |
Milliseconds before the next poll. |
500L |
false |
MEDIUM |
camel.source.endpoint.greedy |
If greedy is enabled, then the ScheduledPollConsumer will run immediately again, if the previous run polled 1 or more messages. |
false |
false |
MEDIUM |
camel.source.endpoint.initialDelay |
Milliseconds before the first poll starts. |
1000L |
false |
MEDIUM |
camel.source.endpoint.repeatCount |
Specifies a maximum limit of number of fires. So if you set it to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means fire forever. |
0L |
false |
MEDIUM |
camel.source.endpoint.runLoggingLevel |
The consumer logs a start/complete log line when it polls. This option allows you to configure the logging level for that. One of: [TRACE] [DEBUG] [INFO] [WARN] [ERROR] [OFF] |
"TRACE" |
false |
MEDIUM |
camel.source.endpoint.scheduledExecutorService |
Allows for configuring a custom/shared thread pool to use for the consumer. By default each consumer has its own single threaded thread pool. |
null |
false |
MEDIUM |
camel.source.endpoint.scheduler |
To use a cron scheduler from either camel-spring or camel-quartz component. Use value spring or quartz for built in scheduler |
"none" |
false |
MEDIUM |
camel.source.endpoint.schedulerProperties |
To configure additional properties when using a custom scheduler or any of the Quartz, Spring based scheduler. |
null |
false |
MEDIUM |
camel.source.endpoint.startScheduler |
Whether the scheduler should be auto started. |
true |
false |
MEDIUM |
camel.source.endpoint.timeUnit |
Time unit for initialDelay and delay options. One of: [NANOSECONDS] [MICROSECONDS] [MILLISECONDS] [SECONDS] [MINUTES] [HOURS] [DAYS] |
"MILLISECONDS" |
false |
MEDIUM |
camel.source.endpoint.useFixedDelay |
Controls if fixed delay or fixed rate is used. See ScheduledExecutorService in JDK for details. |
true |
false |
MEDIUM |
camel.component.azure-storage-datalake.accountKey |
account key for authentication |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.clientId |
client id for azure account |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.client Secret |
client secret for azure account |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.client SecretCredential |
client secret credential for authentication |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.close |
Whether or not a file changed event raised indicates completion (true) or modification (false) |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.closeStream AfterRead |
check for closing stream after read |
null |
false |
MEDIUM |
* camel.component.azure-storage-datalake.configuration* |
configuration object for datalake |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.dataCount |
count number of bytes to download |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.directory Name |
directory of the file to be handled in component |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.download LinkExpiration |
download link expiration time |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.expression |
expression for queryInputStream |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.fileDir |
directory of file to do operations in the local system |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.fileName |
name of file to be handled in component |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.fileOffset |
offset position in file for different operations |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.maxResults |
maximum number of results to show at a time |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.maxRetry Requests |
no of retries to a given request |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.openOptions |
set open options for creating file |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.path |
path in azure datalake for operations |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.permission |
permission string for the file |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.position |
This parameter allows the caller to upload data in parallel and control the order in which it is appended to the file. |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.recursive |
recursively include all paths |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.regex |
regular expression for matching file names |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.retain UncommitedData |
Whether or not uncommitted data is to be retained after the operation |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.service Client |
datalake service client for azure storage datalake |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.sharedKey Credential |
shared key credential for azure datalake gen2 |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.tenantId |
tenant id for azure account |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.timeout |
Timeout for operation |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.umask |
umask permission for file |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.user PrincipalNameReturned |
whether or not to use upn |
null |
false |
MEDIUM |
camel.component.azure-storage-datalake.bridgeError Handler |
Allows for bridging the consumer to the Camel routing Error Handler, which mean any exceptions occurred while the consumer is trying to pickup incoming messages, or the likes, will now be processed as a message and handled by the routing Error Handler. By default the consumer will use the org.apache.camel.spi.ExceptionHandler to deal with exceptions, that will be logged at WARN or ERROR level and ignored. |
false |
false |
MEDIUM |
camel.component.azure-storage-datalake.autowired Enabled |
Whether autowiring is enabled. This is used for automatic autowiring options (the option must be marked as autowired) by looking up in the registry to find if there is a single instance of matching type, which then gets configured on the component. This can be used for automatic configuring JDBC data sources, JMS connection factories, AWS Clients, etc. |
true |
false |
MEDIUM |
The camel-azure-storage-datalake source connector has no converters out of the box.
The camel-azure-storage-datalake source connector has no transforms out of the box.
The camel-azure-storage-datalake source connector has no aggregation strategies out of the box.