Cause a broker failover

Now go back to your Amazon MQ Broker Console. If you do not have a tab open, go to https://console.aws.amazon.com/amazon-mq and then select the “Broker” broker.

Scroll to the top of your brokers details page and click on Actions -> Reboot broker, and then select reboot

Failover 4

Now switch to the browser tab with your terminals. Shortly after, you should see an exception in both consoles because the primary broker isn’t reachable any more. This will not happen instantly.

[ActiveMQ Transport: ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com/52.28.200.138:61617] WARN org.apache.activemq.transport.failover.FailoverTransport - Transport (ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com:61617) failed , attempting to automatically reconnect: {}
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
    at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
    at java.lang.Thread.run(Thread.java:748)

After waiting a few more seconds, you should see a successful reconnect from your clients to the secondary broker, which is now the new primary one (compare the connection urls):

[ActiveMQ Task-3] INFO org.apache.activemq.transport.failover.FailoverTransport - Successfully reconnected to ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-1.mq.eu-central-1.amazonaws.com:61617

The sender terminal log output should look similar to this one:

12.04.2018 12:02:59.869 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4668'
12.04.2018 12:02:59.895 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4669'
12.04.2018 12:02:59.919 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4670'
[ActiveMQ Transport: ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com/52.28.200.138:61617] WARN org.apache.activemq.transport.failover.FailoverTransport - Transport (ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com:61617) failed , attempting to automatically reconnect: {}
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
    at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
    at java.lang.Thread.run(Thread.java:748)
[ActiveMQ Task-3] INFO org.apache.activemq.transport.failover.FailoverTransport - Successfully reconnected to ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-1.mq.eu-central-1.amazonaws.com:61617
12.04.2018 12:03:11.534 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4671'
12.04.2018 12:03:11.670 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4672'
12.04.2018 12:03:11.783 - Sender: sent '[queue://workshop.queueA] [Sender-1] Message number 4673'

The receiver terminal log output should look similar to this one:

12.04.2018 12:02:59.876 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4668'
12.04.2018 12:02:59.902 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4669'
12.04.2018 12:02:59.926 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4670'
[ActiveMQ Transport: ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com/52.28.200.138:61617] WARN org.apache.activemq.transport.failover.FailoverTransport - Transport (ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-2.mq.eu-central-1.amazonaws.com:61617) failed , attempting to automatically reconnect: {}
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
    at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
    at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
    at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
    at java.lang.Thread.run(Thread.java:748)
[ActiveMQ Task-3] INFO org.apache.activemq.transport.failover.FailoverTransport - Successfully reconnected to ssl://b-4e4bfd69-7b83-4a27-9faf-4684cfa80443-1.mq.eu-central-1.amazonaws.com:61617
12.04.2018 12:03:11.541 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4671'
12.04.2018 12:03:11.677 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4672'
12.04.2018 12:03:11.790 - Receiver: received '[queue://workshop.queueA] [Sender-1] Message number 4673'

Now try to return to the ActiveMQ broker console from the step above. This broker is no longer accessible as it has rebooted. You should still however see that the sender and receiver have reconnected and started to send the messages again.

Now go back to the Amazon MQ “Broker” console, scroll to the connections list as shown below.

Failover 5

Now click on the ActiveMQ Web Console link that failed before. Now you should be able to connect to that console(the one that was in Standby mode). Log into the console using the username workshopUser, and either workshopUser or the password you set before.

In the queues screen, you should see the “Messages Enqueued” and the “Messages Dequeued” increasing. The original Standy broker is now Active. Verify that the other brokers is no longer accessible by selecting the url in the Amazon MQ Connections screen (above).

You have successfully seen how the Amazon MQ Active-Standby model works, where rebooting the Active broker caused a very small glitch and then the clients immediately connected to the the Standby brokers.