What we wanted to achieve was to have the following setup for each queue:
- several consumers to spread the load and avoid that one message gets stuck if a route gets into trouble or takes a long time to complete
- each consumer should get only one message at a time so no message gets stuck in the local client buffer
- the camel route should be transacted so if something goes wrong the message is not lost but gets delivered to another consumer
So defining several consumers is rather easy.
We used prefetch=1 to make sure that only one message at a time is delivered to a route .
And we used a transaction manager for our JmsConfiguration bean.
So my config looked like something like that
<bean id="jmsConfig" class="org.apache.camel.component.jms.JmsConfiguration">
<property name="connectionFactory" ref="pooledConnectionFactory" />
<property name="transacted" value="true" />
<property name="transactionManager" ref="jmsTransactionManager" />
<property name="cacheLevelName" value="CACHE_CONSUMER" />
</bean>
So far nothing special.
What I then noticed was that sometimes a consumer could get 2 messages assigned, which puzzled me since I specified prefetch=1!
One being processed and waiting for an ACK to finish the transaction, and the other one just waiting to be taken and not being taken by another consumer even if one is available.
And this was bad because sometimes my route took several minutes to completed, effectively preventing the 2nd message to get through right away.
The culprit was CACHE_CONSUMER. This pre-assigns the next message to the consumer to speed things up.
So I changed this to CACHE_NONE and now everything behaves as expected!
It looks like you were running into the "prefetch extension logic" issue.
It's documented here: https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_A-MQ/6.0/html/Client_Connectivity_Guide/files/AMQClientConnectPrefetch.html
Disabling the prefetch extension logic
The default behavior of a broker is to use delivery acknowledgements to determine the state of a consumer's prefect buffer. For example, if a consumer's prefect limit is configured as 1 the broker will dispatch 1 message to the consumer and when the consumer acknowledges receiving the message, the broker will dispatch a second message. If the initial message takes a long time to process, the message sitting in the prefect buffer cannot be processed by a faster consumer.
This behavior can also cause issues when using the JCA resource adapter and transacted clients.
If the behavior is causing issues, it can be changed such that the broker will wait for the consumer to acknowledge that the message is processed before refilling the prefetch buffer. This is accomplished by setting a destination policy on the broker to disable the prefetch extension for specific destinations.
...
...
ML, yes you are correct :)
By using CACHE_NONE we made sure that we would never have 2 message assigned to a given consumer, but we ended up in another problem which is messages not always delivered in order.
Cf my post from today: http://minibiti.blogspot.no/2016/06/activemq-message-ordering.html