How do I configure 10s of 1000s of Queues in a single broker
Scaling to tens of thousands of Queues in a single broker is relatively straightforward - but requires some configuration changes from the default.
With the default configuration, ActiveMQ is configured to use a dispatch thread per Queue - you can use set the optimizedDispatch property on the destination policy entry - see configuring Queues.
ActiveMQ can optionally use internally a thread pool to control dispatching of messages - but as a lot of deployment operating systems are good at handling a large number of threads, this is off by default. To enable this option, either set the ACTIVEMQ_OPTS to disable dedicated task runners in the start up script, INSTALL_DIR/bin/activemq -e.g.
or you can set
Note: From ActiveMQ 5.6 onwards the dedicated task runner is disabled by default (see jira-issue AMQ-3667).
To reduce the number of threads used for the transport - take a look at using the NIO transport - see Configuring Transports
Here is an example of this in use in one of the provided sample broker configuration files.
Reducing Memory Consumption
Reduce the memory used per thread - see reducing memory consumption
Reduce number of file descriptors
ActiveMQ uses the
amqPersistenceAdapter by default for persistent messages. Unfortunately, this persistence adapter (as well as the kahaPersistenceAdapter) opens a file descriptor for each queue. When creating large numbers of queues, you’ll quickly run into the limit for your OS.