There are 2 kinds of transaction support in ActiveMQ.
They are both implemented fairly similarly. When operations are carried out on a transacted (or XA transacted) session, a transaction command is sent to the broker, with a unique transaction ID which is then followed by all the usual commands (send message, acknowledge message etc).
Then when a commit() or rollback() is called on the Session, this command is sent to the broker for it to commit or rollback the transaction.
Now the operations carried out on a transacted session inside a transaction, like a send message or acknowledge message, do not really perform a real send or acknowledge until the commit occurs. So the Broker explicitly handles these cases separately - essentially buffering up the commands until the commit occurs when the messages are really sent or acknowledged.
For that we have TransactionStore (implemented for all persistence adapters) that handle transactions. TransactionStore will cache all messages and acks until commit or rollback occurs. Besides storing messages, broker will withhold dispatching any of the messages until the session commit.
If you wanna see the code, take a look at MemoryTransactionStore which proxies transactions for Memory and JDBC persistence adapters.
The only real difference with XA transactions is that at the PREPARE stage we MUST write every command we have received (the send message or acknowledge message commands) to a persistent store so that we can recover properly.