Oracle Coherence on Docker

This post is about running Oracle Coherence on Docker. Docker is getting a lot of traction and is certainly flavour of the month in a lot of IT shops. Whether Docker is the right tool to use everywhere it is being used is probably a whole blog post on it own but assuming you have decided to go down that route and have worked out all the ins and out and caveats of using Docker and you have decided for whatever reason to run Coherence in Docker then hopefully this post will help you get a working Coherence cluster.

I am going to assume that you know something about Docker already, you have Docker installed and you can build images and run containers. I am also assuming that you have a good understanding of Coherence. If you don’t know Docker then you should go away and read up on it before you try to run Coherence in containers. If you don’t know much Coherence then go away and learn about it before you try muddying the waters by adding Docker into the mix.

In this post we will cover:

1. Building Docker Images of Java, Coherence and applications
2. Running Coherence Clusters including multi-host clustering
3. Using Coherence *Extend with containers, also Coherence REST and Coherence Memcached functionality
4. Using JMX Management with containers
5. Coherence Persistence on Docker
6. Coherence Federation on Docker

A Word on Docker Networking

Before we get into Coherence specifically, a quick word on Docker’s networking functionality. At the time of writing the current version of Docker is 1.9, which is the first release with multi-host network support. Prior to this it was not possible to run containerised Coherence clusters across multiple hosts as Docker’s default bridge network did not support this.

Many of the issues or caveats I am going to cover below when running Coherence in Docker relate to the way Docker’s networking functionality has been implemented. The different networking modes all introduce problems that need to be worked around. I know some people who have resorted to just running the containers using the --net=host  parameter as this removes all the network problems but it brings with it a set of other issues. In this post we are not going to use host networking as that would be a bit of a cop-out; we are going to look at what you need to do to get Coherence working in a standard Docker environment.

The same applies to some of the third-party network plugins available for Docker – for example Weave. In testing we have had some success with some of these and some of the do remove some of the restrictions imposed by standard Docker networking. Again though we are not going to cover these in this post.

Building Coherence Docker Images

If you are going to run Coherence applications in Docker containers then you need Docker images containing Java, Coherence and your application. Due to Coherence being licensed software from Oracle there are restrictions on uploading images containing Coherence (and most other Oracle software) to public image repositories like Docker Hub. For that reason you need to build the images yourself, you cannot obtain them from or upload them to the public Docker Hub repository.

Just as there are many ways to deploy Coherence to a real server there are many ways to build a Docker image containing a Coherence application. You could just take a suitable base Linux image and then add everything you need in one big single image, or alternatively you could split the image up into multiple layered images. Even if you decide on multiple images there is the choice of whether to create an image with Coherence installed using the Coherence installer or whether you just install the required Coherence jar files along with your applications jar files.

I am going to go down the route of multiple image files and using the Coherence installer as I think there are advantages to this. Having multiple images allows you to easily standardise your images, for example a standard Java image, a standard Coherence image etc. This becomes useful in environments where you want to make sure all applications have a common build structure, versions, configuration, security etc. If you are just knocking out an image for yourself then it isn’t that important but in a corporate environment where standardisation makes a difference it can be important. Consistency and standards makes your developers, DevOps and Support team’s lives so much easier.

For the multiple image route I am going to build the following images:

1. A base Java image – this is just a Linux image with Java installed; in this case it will be Java 8 as that is what the latest Coherence version requires.
2. A base Coherence image – this will use the Java image as its base and install Coherence on top. This will use the Coherence installer to add all of the Coherence artifacts to the image.
3. An application image – this will use the Coherence image as its base image and will install an application and its required artifacts.

Building the Java Image

For the Java image we are going to use a standard Oracle Linux image as our base. Oracle Linux images are available on Docker Hub as Oracle Linux is open source so it makes sense to use these as our base. You can find more information on Oracle Linux on Docker on the Oracle Blog site. The Dockerfile for the Java image is very simple and is shown below:

FROM oraclelinux:7

ENV JAVA_RPM jdk-linux-x64.rpm

RUN curl -s -j -k -L -H "Cookie: oraclelicense=accept-securebackup-cookie" \
  http://download.oracle.com/otn-pub/java/jdk/8u65-b17/jdk-8u65-linux-x64.rpm > $JAVA_RPM \
  && yum -y install $JAVA_RPM \
  && rm $JAVA_RPM

ENV JAVA_HOME /usr/java/default

When we run Docker to build an image from this file it will do the following:

1. Pull the oraclelinux:7 base image from the image repository if it has not already been downloaded to the Docker host.
2. Set and environment variable containing the name to use for the Java rpm – this is just to save typing and typos later.
3. Run curl to download the Java installer from Oracle’s web site – this command automatically accepts the required Oracle license agreement. In this case we are downloading Java 8 update 65 which is the latest at the time of writing. To download a different version just change the url in the Dockerfile.
4. Run yum to install Java and then delete the rpm file as we don’t need it taking space in the image
5. Finally set and environment variable for JAVA_HOME that will be available to all images or containers started from this image.
6. Update: Since I wrote this post I discovered that my original Docker file could be improved by putting all of the RUN commands on a single line. This reduces the number of layers in the resulting image file and also reduces the size of the image because the RMP is downloaded and removed in the same layer.

To build an image from this file you just run the normal Docker build command from the folder containing the Dockerfile, such as:

$ docker build -t oraclejava:8u65 -f Dockerfile .

Once this has run, which should not take long, depending on whether you need to download the Oracle Linux image, then you should have a new image called oraclejava:8u65 on the Docker host. We now have a standard Oracle Java image that can be used as a base image for anything that requires Java.

Building the Coherence Image

Building the Coherence image is almost as straight forward as the Java image was; we are basically just going to run the Coherence installer. As there are a number of different Coherence installers we will not download one when the image is built as we did with the Java image; in this case you need to download the Coherence installer you require and put it into the same folder as the Dockerfile when you run the build command. In our case we are going to use the Coherence standalone installer called fmw_12.2.1.0.0_coherence.jar  unzipped from the Coherence download that we can obtain following the links on Oracle’s Coherence download page.

When we run the Coherence installer as part of building the image it will need to run in silent mode and for this we need a suitable response file for the Oracle installer. A simple response file for Coherence looks like this:

[ENGINE]

Response File Version=1.0.0.0.0

[GENERIC]

ORACLE_HOME=/u01/app/oracle/coherence

INSTALL_TYPE=Typical With Examples

We will put this into a file called coh.rsp in the same folder as the Dockerfile. The final piece we need to make sure that the installer runs correctly is a location file. The location file looks like this:

inventory_loc=/u01/oracle/.inventory
inst_group=oracle

We place this into a file called oraInst.loc again in the same folder as the Dockerfile. The Dockerfile we are going to use is below:

FROM oraclejava:8u65

ENV COH_INSTALLER fmw_12.2.1.0.0_coherence.jar

RUN mkdir /u01 && chmod a+xr /u01 && \
    useradd -b /u01 -m -s /bin/bash oracle && \
    echo oracle:oracle | chpasswd && \
    chown oracle /u01 && \
    mkdir /home/oracle && \
    chown oracle /home/oracle

COPY $COH_INSTALLER /u01/
COPY coh.rsp /u01/
COPY oraInst.loc /u01/oraInst.loc

RUN chown oracle /u01/$COH_INSTALLER \
 && chown oracle /u01/coh.rsp \
 && chown oracle /u01/oraInst.loc

ENV COHERENCE_HOME=/u01/app/oracle/coherence/coherence

USER oracle

RUN java -jar /u01/$COH_INSTALLER -silent -force -responseFile /u01/coh.rsp -invPtrLoc /u01/oraInst.loc -jreLoc $JAVA_HOME \
  && rm /u01/$COH_INSTALLER \
  && rm /u01/coh.rsp \
  && rm /u01/oraInst.loc \
  && rm -rf /tmp/OraInstall*

The Dockerfile above performs a number of steps

• First we use the Java image we built earlier as the base image. If you created a base Java image with a different name then you should change this line to have the correct image name.
• We set an environment variable to the name of the Coherence installer we are using. In our case this is fmw_12.2.1.0.0_coherence.jar  but if you have a different installer name then you only need to change this line.
• We then set up various folders and create an oracle user. These commands are all concatenated into a single line, which is usual practice in a Dockerfile. The reason we need an oracle user is because the default user for Docker is root and the root user cannot run the Oracle installer.
• After creating the folders and user we copy the installer file, response file and location file into the /u01 folder of the image
• We next set the COHERENCE_HOME environment variable to the location we are going to install Coherence to in the image. This will match the Oracle Home location specified in the response file with /coherence  appended to it.
• We now switch to the oracle user we created earlier so that we can run the installer.
• Finally we actually run the installer in silent mode and then remove all of the install files and clean up as we no longer need them taking space in the image.

We can again build the image with the standard Docker build command, for example:

$ docker build -t oraclecoherence:12.2.1-0-0 -f Dockerfile .

In this case we would have a new image called oraclecoherence:12.2.1-0-0 as we are using Coherence version 12.2.1-0-0, but you can change the name in the build command to match whichever version you are using. Now that we have a Coherence image we can actually run this to start a cache server. This image obviously has no application code in it and is primarily used as a base image but it might still be useful on its own.

Building an Application Image

Now we have a suitable base image we can build an application and an application image. A normal Coherence server application is usually made up of a set of dependent jar files, configuration files and probably a start script.

For the purposes of simplicity I will skip building a real application and we’ll imagine we have one I made earlier.

A simple shell script to start a DefaultCacheServer inside a container might look something like this:

#!/usr/bin/env bash

set -eu

exec ${JAVA_HOME}/bin/java \
-cp ${COHERENCE_HOME}/lib/coherence.jar:${APP_LIB}/* \
-Dcoherence.cacheconfig=app-cache-config.xml \
-Dcoherence.pof.config=app-pof-config.xml \
com.tangosol.net.DefaultCacheServer

If you remember when we created the Java and Coherence images above we set some environment variables for JAVA_HOME  and COHERENCE_HOME  and we can use these in the script. This script simply sets the classpath to include the Coherence jar (from the install in the Coherence image) and the value of the ${APP_LIB}/*  which will put all the jar files on the classpath that are under the folder pointed to by the APP_LIB  environment variable (we will come to where APP_LIB  is set in a minute). The script also sets the cache and pof configuration files and starts DefaultCacheServer. So assuming we now have our application code, cache and pof configurations packaged into a jar (lets call it app.jar ) and the above shell script (lets call it run.sh ) in a folder along with a Dockerfile to build our image then we are good to go. The Dockerfile might look something like this:

FROM oraclecoherence:12.2.1-0-0

ENV APP_LIB="/home/oracle/lib"
RUN mkdir $APP_LIB

COPY app.jar /home/oracle/lib/app.jar
COPY run.sh /home/oracle/run.sh

ENTRYPOINT ["/bin/sh"]
CMD ["/home/oracle/run.sh"]

There are many ways to structure an application image so in real life yours will probably look somewhat different but this gets across the concepts. The Dockerfile does the following:

1. The base image is our Coherence image we built earlier oraclecoherence:12.2.1-0-0
2. We set the APP_LIB  environment variable to /home/oracle/lib . If you remember we used this environment variable in the shell script classpath above.
3. We then make sure that the folder is created; we created /home/oracle in the Coherence image when we created the oracle user.
4. We then copy the application’s jar file app.jar to the APP_LIB folder. If we had other dependent jar files, scripts, configuration, etc, we would also copy those at this point too.
5. We copy the run.sh shell script we wrote above into the /home/oracle folder
6. Finally we set the entry point of the image to the run.sh script. This means that if we run the container without specifying any command then run.sh will be run by default.

Assuming that we have a folder containing the above run.sh file, the above Dockerfile and the app.jar file containing our imaginary application then we can build an image with the following command from that folder.

$ docker build -t coherencedemo:1.0 -f Dockerfile .

We will then have an image called coherencedemo:1.0  that we can run inside a container with the following command.

$ docker run -itd coherencedemo:1.0

When we run the this command Docker will start the container and run the run.sh script which will start a DefaultCacheServer instance with the parameters we specified in the script. We will cover more about running Coherence containers in the next section.

Running Coherence Containers

Once we have an image containing Coherence we can run a container from that image. We can use the following command to interactively run a Coherence com.tangosol.net.DefaultCacheServer instance using the Coherence base image we created above:

$ docker run -i -t --rm oraclecoherence:12.2.1-0-0 \
/usr/java/default/bin/java \
-cp /u01/app/oracle/coherence/coherence/lib/coherence.jar \
com.tangosol.net.DefaultCacheServer

And we should see that the server starts successfully

If we start another interactive Docker container, this time running a storage disabled com.tangosol.net.CacheFactory console on the same Docker host using the following command in another command prompt:

$ docker run -i -t --rm oraclecoherence:12.2.1-0-0 \
/usr/java/default/bin/java \
-cp /u01/app/oracle/coherence/coherence/lib/coherence.jar \
-Dcoherence.distributed.localstorage=false \
com.tangosol.net.CacheFactory

NOTE: As of Coherence 12.2.1 system properties that used to be prefixed with tangosol.coherence. can now be shortened to just coherence. and that is what we will use in the examples that follow. If you are trying to use an earlier version of Coherence you will need to use the old property names with the old prefix. The old prefix is still compatible with 12.2.1 but we use the new one as it is less typing.

We can see that the two containers have formed a cluster.

Then just to prove it is working we can then type some commands into Coherence console and see that we can put and get from a cache.

Running Coherence Clusters in Containers on Multiple Docker Hosts

The above examples were very simplistic to show that Coherence will run inside Docker. In the real world though we would run a cluster of Coherence containers spread across mutiple Docker hosts. Outside of a simple test environment you would almost certainly want to run your cluster across multiple hosts so as to protect your cached data from machine failure. Running Coherence containers across multiple hosts is where we start to hit issues with what Docker allows us to do.

Originally when we first started looking at Docker there was no way that Coherence containers on two different hosts could see each other to form a cluster. As of Docker 1.9 we have the overlay network that allows networking between containers to span multiple Docker hosts. Unfortunately, at the current time, the overlay network does not allow multicast to span hosts; although it appears to allow multicast to work over the overlay network between containers on a single host and the network interface inside the container reports that it supports multicast.

The only soultion then to allow Coherence to form a cluster of Docker containers across multiple Docker hosts is to use well-known-addressing. This is easy to set up as we can give each container a unique host name; in fact we have to do this when using the overlay network. We can run some DefaultCacheServer containers from the command line using WKA and see that they do indeed form a cluster.

For the examples in this section we are going to use the example Docker hosts that are created if you follow the Get started with multi-host networking section of the Docker documentation.

If you have an environment the same as the one in the above Docker documentation link then you should have two Docker hosts called mhs-demo0  and mhs-demo1  set up with Swarm and an overlay network called my-net . Make sure that you have also built the Coherence container oraclecoherence:12.2.1-0-0 from above on both of the Docker hosts, so both hosts should have the oraclelinux:7 , the oraclejava:8u65  and oraclecoherence:12.2.1-0-0  containers on them.

First, targeting the mhs-demo0  host, we will start a DefaultCacheServer  instance. As we have not deployed any application code or configuration in our images we will be using the default configuration files from the Coherence jar for these examples. This means that to set WKA we need to set a Java system property called coherence.wka  to the host name of the first member we are going to start. When using the overlay network we will be able to use the same value that we specify for the --name  parameter when starting the container as the host name for that container. We could also use the combination of the --name  value and the network name. In the console window we need to target our Docker commands to the first Docker host mhs-demo0 with the following command:

$ eval $(docker-machine env mhs-demo0)

Now we can start an instance of DefaultCacheServer with the following command:

$ docker run -itd --name=dcs1 --hostname=dcs1 \
--net=my-net oraclecoherence:12.2.1-0-0 \
/usr/java/default/bin/java \
-cp /u01/app/oracle/coherence/coherence/lib/coherence.jar \
-Dcoherence.wka=dcs1 com.tangosol.net.DefaultCacheServer

You can see that we have called this container dcs1  using --name=dcs1  parameter and we set the hostname of this container to dcs1  too using the --hostname=dcs1  parameter. We have therfore set the WKA property to also be dcs1  using -Dcoherence.wka=dcs1 . The value of the name parameter is used by Docker as the host name to inject into other containers /etc/hosts  files so that they can see this container on the overlay network. Using the hostname parameter means that Docker will also inject the host name into this containers /etc/hosts  file so that it can see itself using the host name.

We can look at the output of the container we have just started using the following command…

$ docker logs dcs1

…and eventually we should see that the cache server has started.

If we scroll up the log a bit we should see that the WKA list has been set to a single member – which is this member’s IP address from the overlay network.

Now we can target the second host mhs-demo1 and start a second DefaultCacheServer instance:

$ eval $(docker-machine env mhs-demo1)

$ docker run -itd --name=dcs2 --hostname=dcs2 --net=my-net \
oraclecoherence:12.2.1-0-0 \
/usr/java/default/bin/java \
-cp /u01/app/oracle/coherence/coherence/lib/coherence.jar \
-Dcoherence.wka=dcs1 com.tangosol.net.DefaultCacheServer

This time we called the container dcs2 with the --name=dcs2  parameter and --hostname=dcs2  parameter and we have used the same -Dcoherence.wka=dcs1 parameter so that WKA points to the hostname of the previous container. If we look at the logs for the dcs2 container with the Docker logs command

$ docker logs dcs2

We will see that the DefaultCacheServer has started And if we scroll up we will see that the dcs2 container has the correct WKA list and has clustered with the dcs1 container on the mhs-demo0 host. So there you can see that it is possible to run Coherence inside Docker containers across multiple Docker hosts.

Clustering from Outside Docker

As the Coherence containers form a cluster using Docker’s overlay network it is impossible for an external non-container process to join the cluster; the overlay network is invisible to external non-Docker processes. Any process that you want to join the cluster must be running in a Docker container using the same overlay network.

Coherence *Extend

Unless your whole application runs inside Docker containers then you are going to have some sort of client application that accesses your data over Coherence Extend. This means that the external application just connects over TCP to a proxy listening on a TCP port inside the conatiner; this should be easy, but there is a catch.

If you started a Coherence container using the pre-Docker 1.9 method without any overlay network and exposed a port that the Extend proxy was listening on then everything worked fine. The proxy bound to the only network interface the container had and the proxy listened on the exposed port that Docker NAT’ed nicley for you. The problem is that with the introduction of the overlay network when you start the cache server the proxy can be bound to the overlay network, but this is not visible to external processes and Docker will not NAT ports to the overlay network.

In Coherence 12.2.1 you can configure a Proxy without needing to specify any listen address at all and Coherence will bind to an ephemeral port. In Docker this is not good as we need to know the port beforehand so that we can expose it when the container starts. When we built our demo application image earlier we used a custom cache configuration called app-cache-config.xml that might look something like this:

<?xml version="1.0"?>
<cache-config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xmlns="http://xmlns.oracle.com/coherence/coherence-cache-config"
              xsi:schemaLocation="http://xmlns.oracle.com/coherence/coherence-cache-config coherence-cache-config.xsd">

  <defaults>
    <serializer>pof</serializer>
  </defaults>

  <caching-scheme-mapping>
    <cache-mapping>
      <cache-name>*</cache-name>
      <scheme-name>distributed-scheme</scheme-name>
    </cache-mapping>
  </caching-scheme-mapping>

  <caching-schemes>
    <distributed-scheme>
      <scheme-name>distributed-scheme</scheme-name>
      <service-name>DockerDistributedService</service-name>
      <backing-map-scheme>
        <local-scheme/>
      </backing-map-scheme>
      <autostart>true</autostart>
    </distributed-scheme>

    <proxy-scheme>
      <service-name>Proxy</service-name>
      <acceptor-config>
        <tcp-acceptor>
          <local-address>
            <address system-property="test.extend.address.local">localhost</address>
            <port    system-property="test.extend.port">20000</port>
          </local-address>
        </tcp-acceptor>
      </acceptor-config>
      <load-balancer>client</load-balancer>
      <autostart system-property="test.extend.enabled">true</autostart>
    </proxy-scheme>
  </caching-schemes>
</cache-config>

You can see that in this configuration we have a Proxy service where we specify the address to bind to; in this case localhost unless we override this with the test.extend.address.local  system property. We also specify that we bind to port 20000. So what do we use for the bind address for the Proxy, we have two options, we can bind to any local address or bind to the externally visible address.

Bind to Any Local Address

We can set the <address> value to 0.0.0.0 to bind the Proxy to all local addresses; this is a simple solution and works well.

Bind to Specifically to the External Facing Address

If you don’t want to bind the Proxy to all local addresses and need to use a specific address then you need to do a bit more work to figure out what that address is. When using the overlay network each container now has two network interfaces (eth0 and eth1) and the solution is to get the proxy to bind to the externally facing network interface (eth1) of the container not the overlay interface (eth0). Whiles Docker will add in entries into the hosts file for the eth0 interface the problem here is that there is no host name for the external interface that we can use in our configuration file to tell the proxy to bind to. We only have an IP address and worse, we will not know what that address is until the container starts and Docker allocates one at random. This though is easy enough to work around by using a shell script inside the container that can detect the IP address and pass it thhrough to the cache server command line.

When we built the demo application image above we included a simple shell script to start DefaultCacheServer. We now need to modify this shell script to detect the IP address of the externally facing interface (eth1). When we have this value we can set it into the test.extend.address.local  system property.

We need to change the simple shell script we used above to look like this:

#!/usr/bin/env bash

set -eu

EXTERNAL_IP=$(ip addr show dev eth1 | grep "inet" | awk 'NR==1{print}' | cut -d'/' -f 1)

exec ${JAVA_HOME}/bin/java \
 -cp ${COHERENCE_HOME}/lib/coherence.jar:${APP_LIB}/* \
 -Dcoherence.cacheconfig=app-cache-config.xml \
 -Dcoherence.pof.config=app-pof-config.xml \
 -Dtest.extend.address.local=${EXTERNAL_IP} \
 com.tangosol.net.DefaultCacheServer

You can see we have added a couple of extra lines to the script.

• First we set the EXTERNAL_IP  variable to the IP address we require. We find this using a combination of the ip  command, grep  and awk . Basically we want the IP address that coressponds to the eth1  network interface.
• We then set this IP address into the test.extend.address.local  system property on the command line to start the server.

We can now remove and rebuild the coherencedemo:1.0 container

$ docker rmi coherence demo:1.0

$ docker build -t coherencedemo:1.0 -f Dockerfile .

So we now have an image that will correctly bind the Extend proxy to the external eth1  address that Docker will NAT exposed ports to. which we can verify easily by running a container from the coherencedemo:1.0 image.

$ docker run -idt --name=dcs1 --hostname=dcs1 --net=my-net -p 20000 coherencedemo:1.0

The command above will start our server with the hostname dcs1  on the my-net  overlay network. We have also exposed the Extend port 20000 using the -p 20000  parameter so that Docker will expose and NAT this port for us on the Docker host. If we checking the log output for the line specifiying the proxy address we can see where the proxy listening. The overlay network will normally have a 10.* IP address and the external network will usually have a 172.* IP address.

You can see above in this case our proxy is bound to 172.18.0.2:20000.

Coherence *Extend Load Balancing

One of the features of Coherence is that it can automatically load balance Extend client connections between Proxy services. Unfortunately this feature will not work if you are running Coherence inside Docker containers and your clients are external.

The bridge network used by the container and that Extend is bound to is not directly visible to the outside world – i.e. your Extend clients. Docker uses NAT’ing to forward a port on the Docker host through to the exposed ports on the container. This means in our example above our proxy is listening on 172.18.0.2:20000 but clients cannot see this address. Clients would need to connect to the IP address of the Docker host and whatever port Docker forwarded from the host to the container’s port 20000.

When we try to use Proxy load balancing in Coherence the load balance only knows the internal address of the proxy, i.e 172.18.0.2:20000 but that is not the address the client needs. If the load balancer tries to redirect the client to another proxy it will fail.

It may be possible to write a custom load balance that will work inside the container and knows the external addresses, but this has yet to be proven out.

In the meantime proxy load balancing needs to be disabled if you are running in containers by using the client setting…

<proxy-scheme>
  ...
  <load-balancer>client</load-balancer>
  ...
</proxy-scheme>

…in the proxy-scheme configuration.

Coherence *Extend Clients

Now that we have seen how to configure and run a containerised *Extend Proxy we can turn our attention to the client application.

Containerised Coherence Clients

If your application all runs inside Docker containers; that is, your clients are also containerised, then you do not need to go through the hoops above. Extend, REST, Memcache clients running in a container can see the overlay network and hence can connect to a Proxy listening on the overlay network. The Proxy, REST and Memcache services and load balancing will all work as normal.

Non-Containerised Coherence *Extend Clients

If your client application runs outside of Docker, which is a likely scenario from what I can see, then you will need to configure your client application so that it knows the addresses of the proxies to connect to. There are a number of ways to configure an *Extend client from the zero configuration options, using Coherence NameService lookups, and using fixed address lists or address providers.

Configuring a Fixed *Extend Proxy Endpoint

Starting with the simplest to make work in Docker we have the fixed address list option. We know the IP address of the Docker host and we know or can find the port that Docker has NAT’ed from the host to the port that our Proxy is listening on. We can then easily configure the client like this:

<remote-cache-scheme>
  <scheme-name>proxy-scheme</scheme-name>
  <service-name>RemoteCacheService</service-name>
  <initiator-config>
    <tcp-initiator>
      <remote-addresses>
        <socket-address>
          <address system-property="extend.address">127.0.0.1</address>
          <port system-property="extend.port">20000</port>
        </socket-address>
      </remote-addresses>
    </tcp-initiator>
  </initiator-config>
</remote-cache-scheme>

If we use the above configuration we just then need to set the extend.address  and extend.port  system properties to match our Docker host and NAT’ed port. Alternatively we could have used a custom AddressProvider implementation that provides the list of Docker host and NAT’ed port combinations for all of the Proxy servers in our cluster. This is pretty easy and is how Coherence *Extend was configured for a long time prior to the NameService being available.

Using a NameService Configuration

The NameService makes it easier to configure *Extend and there will be a NameService running inside each DefaultCacheServer in the cluster. By default the NameService runs on a sub-port of the member’s cluster port (Coherence uses 32bit port numbers to support sub-ports, so for example if the cluster port is 7574 you will see in the logs that the NameService currently binds to 7574.3).

It would appear that Docker’s NAT’ing of ports from the host to the container works fine with sub-ports and if make sure that our containerised DefaultCacheServer has a fixed local port (by setting the -Dcoherence.localport system property) and then expose that port when we start the container then the client application can connect to the NameService over the sub-port. You would configure the client like this:

<remote-cache-scheme>
  <scheme-name>proxy-scheme</scheme-name>
  <service-name>RemoteCacheService</service-name>
  <proxy-service-name>Proxy</proxy-service-name>
  <initiator-config>
    <tcp-initiator>
      <name-service-addresses>
        <socket-address>
          <address system-property=“extend.address">127.0.0.1</address>
          <port system-property=“extend.port”>7574</port>
        </socket-address>
      </name-service-addresses>
    </tcp-initiator>
  </initiator-config>
</remote-cache-scheme>

If we use the above configuration we just then need to set the extend.address  and extend.port  system properties to match our Docker host and NAT’ed port for the containers Coherence local port. So it is possible to configure a client to use the NameService and it is possible for the client to look up the Extend Proxy end points through the NameService. The issue though is that the NameService lookup will return the internal addresses of the sockets that the proxy is listening on. The NameService is running inside the containerised cluster and knows nothing of NAT’ed ports and host addresses. For this reason it is not possible to use the NameService in an *Extend client.

Using Zero Configuration *Extend Clients

New with Coherence 12.2.1 is the ability to configure an *Extend client with practically zero configuration like this:

<remote-cache-scheme>
  <scheme-name>proxy-scheme</scheme-name>
  <service-name>RemoteCacheService</service-name>
  <proxy-service-name>Proxy</proxy-service-name>
</remote-cache-scheme>

In this case the client is going to attempt to locate a cluster with the same name as the client’s cluster and then connect to the NameService on one of the members of that cluster. By default the cluster location would be done over multicast the same as any other Coherence cluster lookup but we know that multicast will not work outside the cluster. We could configure the client with a WKA list of Docker hosts and NAT’ed cluster ports but this too appears to fail at the moment. Finally even if we could get the cluster lookup to work the client would again attempt to find an *Extend proxy end point from the NameService and be given back an internal IP address and port. So we can see that zero configuration *Extend does not work with Docker.

So, in summary the only way currently to allow *Extend clients to connect to a containerised proxy is to configure the clients with a fixed set of Proxy end point socket addresses.

Coherence REST and Memcached

Coherence REST and Memcached servers work in the same way that Extend works and we would need to follow the same techniques we did above to find the external facing IP address and pass that as Java system properties through to the configuration files.

Containerised Coherence Clients

If your application all runs inside Docker containers; that is, your clients are also containerised, then you do not need to go through the hoops above. Extend, REST, Memcache clients running in a container can see the overlay network and hence can connect to a Proxy listening on the overlay network. The Proxy, REST and Memcache services and load balancing will all work as normal.

Coherence JMX Management

Coherence provide various management capabilites via MBeans exposed through JMX. There are some issues with using JMX inside containers; these are not specific to Coherence but due to how JMX itself works; or more accurately how the JVM’s default of JMX over RMI works.

As with Extend we need to be able to expose the JMX server over the externally facing network interface and not over the overlay network. Because of the way that JMX works we need to be able to tell JMX what the IP address of the server should be. Fortunately JMX provides a system property to allow us to specify this address:

-Djava.rmi.server.hostname 

This address must be the IP address of the Docker host, not the IP address of either of the interfaces inside the container.

The second problem with JMX is the issue with ports. JMX actually uses two ports, one for the JMX connection and one for the RMI server. When using Docker we need to make sure that we specify both of these ports and expose both ports when starting containers so that Docker will NAT them correctly. There are two system properties used to specify these ports and both can be set to the same value so that we only need to expose a single port in the container:

-Dcom.sun.management.jmxremote.port  -Dcom.sun.management.jmxremote.rmi.port 

In theory that should be enough to allow us to connect to a JMX server inside a container and almost every example of JMX in Docker on the web shows these settings with everything working fine.

For example, say we now change out start script run.sh we have been using to add in the appropriate JMX system properties like this:

#!/usr/bin/env bash

set -e

EXTERNAL_IP=$(ip addr show dev eth1 | grep "inet" | awk 'NR==1{print $2}' | cut -d'/' -f 1)

exec ${JAVA_HOME}/bin/java \
-cp ${COHERENCE_HOME}/lib/coherence.jar:${APP_LIB}/* \
-Dcoherence.cacheconfig=app-cache-config.xml \
-Dcoherence.pof.config=app-pof-config.xml \
-Dtest.extend.address.local=${EXTERNAL_IP} \
-Dcoherence.management=all \
-Dcoherence.management.remote=true \
-Djava.rmi.server.hostname=${DOCKER_HOST} \
-Dcom.sun.management.jmxremote.port=9000 \
-Dcom.sun.management.jmxremote.rmi.port=9000 \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.local.only=false \
com.tangosol.net.DefaultCacheServer

You can see that we have set the java.rmi.server.hostname  property to the value of an environment variable. We do not set this in the script but we will pass it in when we start the container. We cannot set it in our shell script as we have no idea what the address of the Docker host will be.

If we rebuild the coherencedemo:1.0 image with the above changes to the run.sh script we can run a container with the following command.

$ docker run -idt --name=dcs1 --net=my-net --hostname=dcs1 \
-p 20000 -p 9000:9000 -e "DOCKER_HOST=192.168.99.102" \
coherencedemo:1.0

In the command above we have started our container called dcs1 and in this case we have exposed port 20000 (the Extend port) and port 9000 (the JMX port) and we have set the DOCKER_HOST environment variable to 192.168.99.102 (which is the IP address of my Docker host. We have explictly told Docker to NAT port 9000 in the container to port 9000 on the Docker host and this means we should be able to connect JConsole to our server with the following URL:

service:jmx:rmi:///jndi/rmi://192.168.99.102:9000/jmxrmi

This all works fine, and as I have said, almost every Docker and JMX example works this way. The problem though is that we have explicitly told Docker the port to use for NAT’ing the JMX port, i.e. we mapped port 9000 in the container to port 9000 on the Docker host. If we had tried to map to a different port or left Docker to assign a random port then we would have issues. If we run the container like this:

$ docker run -idt --name=dcs1 --net=my-net --hostname=dcs1 \
-p 20000 -p 9001:9000 -e "DOCKER_HOST=192.168.99.102" \
coherencedemo:1.0

We are now NAT’ing the JMX port from 9001 on the Docker host to 9000 in the container. We then need to connect JConsole to the NAT’ed port like this:

$ jconsole -debug service:jmx:rmi:///jndi/rmi://192.168.99.102:9001/jmxrmi

If we run the above command (which also enables JConsole debugging) we will see this:

java.rmi.ConnectException: Connection refused to host: 192.168.99.102; nested exception is:
 java.net.ConnectException: Connection refused
 at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
 at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
 at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
 at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
 at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
 ...

So what is going on, everything works swimmingly before, the only difference is that we have changed the NAT mapping; and that is the issue. The way that a standard JMX client works is that it uses two connections and uses the first to look up the URL of the second.

So the client connects to the JMX server on 192.168.99.102:9001 and this is NAT’ed to port 9000 on the container and everything works fine. The client then asks the JMX server for the second connection URL and it gets back as a result 192.168.99.102:9000. Now the address is correct as this comes from the -Djava.rmi.server.hostname  property but the port is 9000 as that is the port the JMX server knows it is listening on, it knows nothing about the NAT’ed port. Hence when the client then tries to connect to 192.168.99.102:9000 it gets connection refused.

This is why the majority of examples work, as they show a fixed NAT apping of matchng ports. In the real world we don’t want to have to manage ports (that’s one of the reasons we are using Docker!) but using random ports will not work.

One possible solution may be to pass the JMX port through to the script as an environment variable the same way we have to pass the IP address. This way we can start each container with a different JMX port and still set up the one-to-one port mapping required.

Use JMXMP Instead of RMI

We have seen above that using JMX over RMI is troublesome inside Docker containers. An alternative is to use a different transport than RMI, one such alternative being JMXMP. The advantage of JMXMP is that it is TCP and Java serialization based and only requires a single port; which is ideal for Docker. There is a lot of documentation on JMXMP and a number of people who think it is superior to RMI, it is certainly less cumbersome to get working. The issue is that JMXMP does not come bundled with the JVM, which is a shame as it has been around for a while.

To use JMXMP you will need a jar containing an implementation; we are using the one from Glass Fish that can be found as a Maven dependency here:

<dependency>
  <groupId>org.glassfish.external</groupId>
  <artifactId>opendmk_jmxremote_optional_jar</artifactId>
  <version>1.0-b01-ea</version>
</dependency>

This jar will need to be on the classpath of your application. This is where building standard Docker base images for Java and Coherence is and advantage as you can install common libraries like this into the base images and they are available to all applications.

Now we have the JMXMP implementation we need to make sure that we have an MBean connector server running that we can connect to from an external client. The code for starting a connector server runs to about five lines of Java and Coherence makes it very simple to add in as there is already a nice hook in the configuration.

If you look at the Coherence Operational Configuration in the tangosol-coherence.xml  file in the Coherence jar you will see a secion tagged <management-config> . This contains an element like this:

<server-factory>
  <class-name system-property="coherence.management.serverfactory"</class-name>
</server-factory>

As you can see there is no server factory specified by default but all we need to do to provide one is set the coherence.management.serverfactory  system property to the name of our implementation class. The server factory is is documented in the Coherence documentation in a section called Using an Existing MBean Server. Basically we need to implement a class called com.tangosol.net.management.MBeanServerFinder the purpose of which is to provide the MBean server that Coherence will use to register its MBeans. Our implementation to provide a JMXMP connector is very simple.

package com.tangosol.util;

import com.tangosol.net.management.MBeanServerFinder;

import javax.management.MBeanServer;

import javax.management.remote.JMXConnectorServer;
import javax.management.remote.JMXConnectorServerFactory;
import javax.management.remote.JMXServiceURL;

import java.io.IOException;

import java.lang.management.ManagementFactory;


public class JmxmpServer
        implements MBeanServerFinder
    {
    // ----- MBeanServerFinder methods --------------------------------------

    @Override
    public JMXServiceURL findJMXServiceUrl(String s)
        {
        return s_jmxServiceURL;
        }

    @Override
    public MBeanServer findMBeanServer(String s)
        {
        return ensureServer().getMBeanServer();
        }

    // ----- helper methods -------------------------------------------------

    /**
     * Obtain the JMXMP protocol {@link JMXConnectorServer} instance, creating
     * the instance of the connector server if one does not already exist.
     *
     * @return  the JMXMP protocol {@link JMXConnectorServer} instance.
     */
    private static synchronized JMXConnectorServer ensureServer()
        {
        try
            {
            if (s_connectorServer == null)
                {
                MBeanServer server = ManagementFactory.getPlatformMBeanServer();
                int         nPort  = Integer.getInteger("coherence.jmxmp.port", 9000);

                s_jmxServiceURL   = new JMXServiceURL("jmxmp", "0.0.0.0", nPort);
                s_connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(s_jmxServiceURL, null, server);

                s_connectorServer.start();
                }

            return s_connectorServer;
            }
        catch (IOException e)
            {
            throw Base.ensureRuntimeException(e);
            }
        }

    // ----- data members ---------------------------------------------------

    /**
     * The JMXServiceURL for the MBeanConnector used by the Coherence JMX framework.
     */
    private static JMXServiceURL      s_jmxServiceURL;

    /**
     * The {@link JMXConnectorServer} using the JMXMP protocol.
     */
    private static JMXConnectorServer s_connectorServer;
    }

The class above creates a singleton instance of the JMXConnectorServer  in the static ensureServer()  method; this ensures that there is only one instance in the JVM. The JMXConnectorServer  uses a JMXServiceURL  made up of three parts, the protocol, in this case jmxmp, the address to bind to (0.0.0.0) and the port to listen on. We have defaulted the port to 9000 but allowed it to be configured by setting the coherence.jmxmp.port  system property.

To use JMXMP in our demo application all we need to do is put the JmxmpServer  class above and the opendmk_jmxremote_optional_jar-1.0-b01-ea.jar  file on our classpath and set the coherence.management.serverfactory  and coherence.jmxmp.port  system properties. Or alternatively the JmxmpServer  class and opendmk_jmxremote_optional_jar-1.0-b01-ea.jar  can be put into the base Java or Coherence image.

So assuming the classes are in one of our images all we need to do is change our run.sh shell script that starts our application to have the correct system properties.

#!/usr/bin/env bash

set -e

EXTERNAL_IP=$(ip addr show dev eth1 | grep "inet" | awk 'NR==1{print $2}' | cut -d'/' -f 1)

exec ${JAVA_HOME}/bin/java \
-cp ${COHERENCE_HOME}/lib/coherence.jar:${APP_LIB}/* \
-Dcoherence.cacheconfig=app-cache-config.xml \
-Dcoherence.pof.config=app-pof-config.xml \
-Dtest.extend.address.local=${EXTERNAL_IP} \
-Dcoherence.management=all \
-Dcoherence.management.remote=true \
-Dcoherence.management.serverfactory=com.tangosol.util.JmxmpServer \
-Dcoherence.jmxmp.port=9001 \
-Dcom.sun.management.jmxremote=true \
-Dcom.sun.management.jmxremote.authenticate=false \
-Dcom.sun.management.jmxremote.ssl=false \
com.tangosol.net.DefaultCacheServer

You can see that this is a bit simpler that the previous RMI version. All we have had to specify is the server factory as our JmxmpServer  class and set the port to 9001.

If we rebuild our demo application image we should be able to run a container that runs a DefaultCacheServer and exposes our MBeans over JMXMP. We can run the container like this:

$ docker run -idt --name=dcs1 --net=my-net --hostname=dcs1 -p 20000 -p 9001 coherencedemo:1.0

This time we have not had to worry about passing the hosts IP address and we are just exposing ports 20000 for extend and port 9001 for JMX; we don’t care where Docker NATs them to.

Once the container is running we can find out the port that has been NAT’ed from the host to our container’s JMXMP port using the Docker port command for our dcs1 container:

$ docker port dcs1

The output will be something like this:

In this case Docker has NAT’ed port 9001 in the container to port 32771 on the host. In my case my host’s IP address is 192.168.99.102 so the URL we need to use to connect a JMX client to my DefaultCacheServer is:

service:jmx:jmxmp://192.168.99.102:32771

Now we know the URL connect to it with our JMX client; for example JConsole or JVisualVM or something similar. First though we need to make sure that our client can use the JMXMP protocol; which JConsole and JVisualVM cannot by default. To support the JMXMP protocol the client will need to have the opendmk_jmxremote_optional_jar-1.0-b01-ea.jar on the classpath. For JConsole and JVisualVM this is pretty easy.

For JConsole:

$ jconsole -J-Djava.class.path="$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:opendmk_jmxremote_optional_jar-1.0-b01-ea.jar"

For JVisualVM:

$ jvisualvm -cp "$JAVA_HOME/lib/tools.jar:opendmk_jmxremote_optional_jar-1.0-b01-ea.jar"

That is all there is to it and now we can connect to a containerized Java JMX server. This technique should work with any Dockerized Java process, not just Coherence.

Coherence Persistence

A new feature of Coherence 12.2.1 was Persistence, which is basically saving cache state to disc. There is no reason that Persistence will not work inside Docker containers.

When running a container the disc storage is by default owned by the container and is internal to the container. The container’s storage cannot be accessed by external processes and when the container stops and is removed the storage is also removed. Obviously this may not be desirable for something like Persistence where you want the data to be available outside of the lifetime of a container or you want the data to be availabel to multiple containers or on multiple Docker hosts. Docker provides a lot of functionality to configure data volumes to store Persistence data outside of the containers.

Best practice when using Coherence Persistence functionality would be to configure the directories used by Persistence to point to locations that mapped to volumes that are external to the container.

The Docker documentation has a whole section on about how to Manage Data in Containers that goes into all the details of the different options available to manage data volumes externally to containers.

Coherence Federation

A new feature of Coherence 12.2.1 is Federated Caching. The federated caching feature replicates cache data asynchronously across multiple geographically dispersed clusters. Cached data is replicated across clusters to provide redundancy, off-site backup, and multiple points of access for application users in different geographical locations. Obviously Federation is going to rely on communication over the network between members of the different cluster and as we have seen Docker can introduce headaches with things network related.

The most likely case for federation on Docker would be to replicate data between two Coherence clusters both running in Docker containers and each on its own overlay network. Typically federation is used to replicate data across data centers so it is unlikely that both clusters will share the same Docker overlay network. By default federated caching is configured something like this:

<federation-config>
  <participants>
    <participant>
      <name>ClusterA</name>
      <remote-addresses>
        <socket-address>
          <address>192.168.1.100</address>
          <port>7574</port>
        </socket-address>
      </remote-addresses>
    </participant>
    <participant>
      <name>ClusterB</name>
      <remote-addresses>
        <socket-address>
          <address>192.168.99.100</address>
          <port>7574</port>
        </socket-address>
      </remote-addresses>
    </participant>
  </participants>
</federation-config>

Where the socket addresses used for each cluster contain the socket address and cluster port of at least one member of each cluster. Coherence then uses this address to connect from one cluster to the other and to look up the federation endpoints to connect to. As we have seen with *Extend above this is going to cause issues as any end point returned from a lookup is going to be the internal socket address of the container and not the NAT’ed address that is required. At the moment this makes it pretty much impossible to use federation inside Docker as there is no way to have a lookup return the correct address.

Summary

So to summarize, you have seen that it is possible to run Coherence clusters across multiple Docker hosts (with Docker version 1.9 and above). It is possible to write *Extend clients that can connect to these containerised cluster and it is possible to use JMX management to manage the clusters.

What Doesn’t Work

1. Multicast across different Docker hosts; cluster membership must use WKA.
2. Non-Containerised cluster membership; cluster members must be on the same Docker overlay network.
3. NameService look-ups; the NameService returns internal socket addresses when asked to lookup Extend proxy service or federated caching end points.
4. Extend load-balancing must be set to “client”.
5. JMX using RMI so use JMXMP instead.