How to connect to Java instances running on EC2 using JMX

Question

We are having a problem connecting to our Java applications running in Amazon's EC2 cluster. We definitely have allowed both the "JMX port" (which is usually the RMI registry port) and the server port (which does most of the work) to the security-group for the instances in question. Jconsole connects but seems to hang and never show any information.

We are running our java with something like the following:

java -server -jar foo.jar other parameters here > java.log 2>&1

We have tried:

• Telnets to the ports connect but no information is displayed.
• We can run jconsole on the instance itself using remote-X11 over ssh and it connects and shows information. So the JRE is exporting it locally.
• Opening all ports in the security group.
• Using tcpdump to make sure the traffic is not going to other ports.
• Simulating it locally. We can always connect to our local JREs or those running elsewhere on our network using the same application parameters.

java -version outputs:

OpenJDK Runtime Environment (IcedTea6 1.11.5) (amazon-53.1.11.5.47.amzn1-x86_64)

OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)

As an aside, we are using my Simple JMX package which allows us to set both the RMI registry and server ports which are typically semi-randomly chosen by the RMI registry. You can also force this with something like the following JMX URI:

service:jmx:rmi://localhost:" + serverPort + "/jndi/rmi://:" + registryPort + "/jmxrmi"

These days we use the same port for both the server and the registry. In the past, we have used X as the registry-port and X+1 for the server-port to make the security-group rules easy. You connect to the registry-port in jconsole or whatever JMX client you are using.

Answer 1

It seems that the problem was a combination of 2 missing settings. The first forces the JRE to prefer ipv4 and not v6. This was necessary (I guess) since we tend to try to connect to that via a v4 address:

-Djava.net.preferIPv4Stack=true

The real blocker was the very fact that JMX works by initially contacting the RMI port that responds with the hostname and port for the JMX client to attach. With no further settings, it'll use the local IP of the box that is a 10.X.X.X virtual address which a remote client cannot route to. We needed to add the following setting that is the external hostname or IP of the server -- during this case, it's the elastic hostname of the server.

-Djava.rmi.server.hostname=ec2-107-X-X-X.compute-1.amazonaws.com

The trick, if you're attempting to automate your EC2 instances (and why the hell would you not), is the way to find this address at runtime. To do that you just need to put something just like the following in our application boot script:

# get our _external_ hostname

RMI_HOST=`wget -q -O - http://169.254.169.254/latest/meta-data/public-hostname`

...

java -server \

    -Djava.net.preferIPv4Stack=true -Djava.rmi.server.hostname=$RMI_HOST \

    -jar foo.jar other parameters here > java.log 2>&1

The mysterious 169.254.169.254 IP within the wget command above provides information that the EC2 instance will request about itself. I'm disappointed that this doesn't include tags that are solely available in an authenticated call.

I initially was using the extern ipv4 address but it looks like the JDK tries to make a connection to the server-port when it starts up. If it uses the external IP then this was slowing our application boot time until that timed out. The public-hostname resolves locally to the 10-net address and to the public-ipv4 externally. So the application now's beginning fast and JMX clients still work.