ActiveMQ is specifically designed to allow custom transports to be plugged in. You can programatically create TransportConnector instances and add them to the BrokerService in Java code using the BrokerService.addConnector() method
or you can use the transport discovery mechanism, so that folks can just use your new transport using the neat URI syntax.
The way it works is that the URI scheme (e.g. "vm") is used to find a text file on the classpath - here are all the schemes provided.
then the contents of the file is the class name of a Java class on the classpath which extends the TransportFactory class.
Then once you've implemented a Transport and TransportFactory class, folks can use your new transport using a URI in the JMS client or broker XML config file using the URI using your new schema name. e.g. if you wrote a new foo transport, you could use a URI of foo://localhost?myParam=bar