Clusters

Apache ActiveMQ Artemis clusters allow groups of Apache ActiveMQ Artemis servers to be grouped together in order to share message processing load. Each active node in the cluster is an active Apache ActiveMQ Artemis server which manages its own messages and handles its own connections.

The cluster is formed by each node declaring cluster connections to other nodes in the core configuration file broker.xml. When a node forms a cluster connection to another node, internally it creates a core bridge (as described in Core Bridges) connection between it and the other node, this is done transparently behind the scenes - you don’t have to declare an explicit bridge for each node. These cluster connections allow messages to flow between the nodes of the cluster to balance load.

Nodes can be connected together to form a cluster in many different topologies, we will discuss a couple of the more common topologies later in this chapter.

We’ll also discuss client side load balancing, where we can balance client connections across the nodes of the cluster, and we’ll consider message redistribution where Apache ActiveMQ Artemis will redistribute messages between nodes to avoid starvation.

Another important part of clustering is server discovery where servers can broadcast their connection details so clients or other servers can connect to them with the minimum of configuration.

It is important to note that while the goal of clustering is to increase overall message throughput via horizontal scaling it is not a "silver bullet." In certain situations clustering can, in fact, reduce message throughput so care must be taken when choosing a clustered configuration. Here’s a few general guidelines:

The main way a cluster can reduce overall message throughput is if there are are not enough producers & consumers on each node leading to message build-up on some nodes and consumer starvation others. The cluster has mechanisms to deal with this (i.e. message load-balancing & redistribution, which will be covered later), but you really don’t want the broker to intervene and move messages between nodes unless absolutely necessary because that adds latency.

Therefore, when thinking in performance terms the main question one must answer when choosing a clustered configuration is: Do I have enough clients so that each node in the cluster has sufficient consumers to receive all the messages produced on that node? If the answer to that question is "yes" then clustering may, in fact, improve overall message throughput for you. If the answer to that question is "no" then you’re likely to get better performance from either a smaller cluster or just a single broker.

Also keep in mind that a connection router may improve performance of your cluster by grouping related consumers and producers together on the same node.

Server discovery is a mechanism by which servers can propagate their connection details to:

This information, let’s call it the Cluster Topology, is actually sent around normal Apache ActiveMQ Artemis connections to clients and to other servers over cluster connections. This being the case we need a way of establishing the initial first connection. This can be done using dynamic discovery techniques like UDP and JGroups, or by providing a list of initial connectors.

If cluster connections are defined between nodes of a cluster, then Apache ActiveMQ Artemis will load balance messages arriving at a particular node from a client.

Let’s take a simple example of a cluster of four nodes A, B, C, and D arranged in a symmetric cluster (described in Symmetrical Clusters section). We have a queue called OrderQueue deployed on each node of the cluster.

We have client Ca connected to node A, sending orders to the server. We have also have order processor clients Pa, Pb, Pc, and Pd connected to each of the nodes A, B, C, D. If no cluster connection was defined on node A, then as order messages arrive on node A they will all end up in the OrderQueue on node A, so will only get consumed by the order processor client attached to node A, Pa.

If we define a cluster connection on node A, then as ordered messages arrive on node A instead of all of them going into the local OrderQueue instance, they are distributed in a round-robin fashion between all the nodes of the cluster. The messages are forwarded from the receiving node to other nodes of the cluster. This is all done on the server side, the client maintains a single connection to node A.

For example, messages arriving on node A might be distributed in the following order between the nodes: B, D, C, A, B, D, C, A, B, D. The exact order depends on the order the nodes started up, but the algorithm used is round robin.

Apache ActiveMQ Artemis cluster connections can be configured to always blindly load balance messages in a round robin fashion irrespective of whether there are any matching consumers on other nodes, but they can be a bit cleverer than that and also be configured to only distribute to other nodes if they have matching consumers. We’ll look at both these cases in turn with some examples, but first we’ll discuss configuring cluster connections in general.

With Apache ActiveMQ Artemis client-side load balancing, subsequent sessions created using a single session factory can be connected to different nodes of the cluster. This allows sessions to spread smoothly across the nodes of a cluster and not be "clumped" on any particular node.

The load balancing policy to be used by the client factory is configurable. Apache ActiveMQ Artemis provides four out-of-the-box load balancing policies, and you can also implement your own and use that.

The out-of-the-box policies are

You can also implement your own policy by implementing the interface org.apache.activemq.artemis.api.core.client.loadbalance.ConnectionLoadBalancingPolicy

Specifying which load balancing policy to use differs whether you are using JMS or the core API. If you don’t specify a policy then the default will be used which is org.apache.activemq.artemis.api.core.client.loadbalance.RoundRobinConnectionLoadBalancingPolicy.

The parameter connectionLoadBalancingPolicyClassName can be set on the URI to configure what load balancing policy to use:

The set of servers over which the factory load balances can be determined in one of two ways:

Sometimes you want to explicitly define a cluster more explicitly, that is control which server connect to each other in the cluster. This is typically used to form non symmetrical clusters such as chain cluster or ring clusters. This can only be done using a static list of connectors and is configured as follows:

In this example we have set the attribute allow-direct-connections-only which means that the only server that this server can create a cluster connection to is server1-connector. This means you can explicitly create any cluster topology you want.

Another important part of clustering is message redistribution. Earlier we learned how server side message load balancing round robins messages across the cluster. If message-load-balancing is OFF or ON_DEMAND then messages won’t be forwarded to nodes which don’t have matching consumers. This is great and ensures that messages aren’t moved to a queue which has no consumers to consume them. However, there is a situation it doesn’t solve: What happens if the consumers on a queue close after the messages have been sent to the node? If there are no consumers on the queue the message won’t get consumed and we have a starvation situation.

This is where message redistribution comes in. With message redistribution Apache ActiveMQ Artemis can be configured to automatically redistribute messages from queues which have no consumers or consumers with filters that don’t match messages. The messages are re-routed to other nodes in the cluster which do have matching consumers. To enable this functionality message-load-balancing must be ON_DEMAND or OFF_WITH_REDISTRIBUTION

Message redistribution can be configured to kick in immediately after the need to redistribute is detected, or to wait a configurable delay before redistributing. By default, message redistribution is disabled.

Message redistribution can be configured on a per address basis, by specifying the redistribution delay in the address settings. For more information on configuring address settings, please see Configuring Addresses and Queues via Address Settings.

Here’s an address settings snippet from broker.xml showing how message redistribution is enabled for a set of queues:

The above address-settings block would set a redistribution-delay of 0 for any queue which is bound to any address. So the above would enable instant (no delay) redistribution for all addresses.

The attribute match can be an exact match or it can be a string that conforms to the Apache ActiveMQ Artemis wildcard syntax (described in Wildcard Syntax).

The element redistribution-delay defines the delay in milliseconds between detecting the need for redistribution and actually attempting redistribution. A delay of zero means the messages will be immediately redistributed. A value of -1 signifies that messages will never be redistributed. The default value is -1.

It often makes sense to introduce a delay before redistributing as it’s a common case that a consumer closes but another one quickly is created on the same queue, in such a case you probably don’t want to redistribute immediately since the new consumer will arrive shortly.

Apache ActiveMQ Artemis clusters can be connected together in many different topologies, let’s consider the two most common ones here