Oracle Coherence Testing with Oracle Tools

This blog post is going to cover the new Oracle Tools project hosted on the Coherence Comunity GitHub site and how to use this to help test your applications, specifically Java and Oracle Coherence applications. For anyone who has been using the Oracle Coherence Incubator you will already be familiar with some of this post and I have blogged about the Incubator runtime package in the past here.

There have been a lot of changes around the Oracle Coherence Incubator over the last year thanks to a lot of work by the guys at Oracle. The biggest positive is that everything is now properly open sourced on GitHub. You can take the code and do whatever you like with it but more importantly it makes keeping up to date with changes much easier and it makes contributing changes and fixes back to the project a lot easier too. If you look at the Coherence Comunity on GitHub you will see it is made up of a number of different repositories. In this post I am going to talk about just one of those, the Oracle Tools project. If you do not want to get the source you can just pull the jar files from the central Maven repository as the releases of Oracle Tools are all published to Maven under com.oracle.tools.

After some discussions a while back it was clear that there was a lot of code in the Oracle Coherence Incubator Commons runtime package that would be useful for development other applications that may have nothing to do with Oracle Coherence. The runtime package allowed you to control any external process or virtualised Java process but being part of the Incubator dragged in the dependency on Coherence. A decision was made to refactor all of this code out of the Incubator and into its own project and so Oracle Tools was born.

If you look at the project you will see that it is pretty much all of the code from the old Incubator runtime package refactored into various modules. On the subject of modules, all of the projects on GitHub are now proper Maven projects so there should be no more headaches trying to figure out how to build them. I’m not getting into the pros and cons of build tools, Maven was chosen as the build tool and that is the way it is.

So, on with the meat of the blog post…

The Oracle Tools Modules

Oracle Tools is made up of the following modules:

• oracle-tools-core
• oracle-tools-runtime
• oracle-tools-testing-support
• oracle-tools-runtime-tests
• oracle-tools-coherence

The functionality provided by Oracle Tools right now is pretty much all geared around running “processes” and supporting easy testing of your applications and as you would expect specifically testing clustered applications made up of a number of processes. This means that typically to use Oracle Tools only your tests need a dependency on the Oracle Tools artefacts, your application code does not. I think this is quite an important point, I know a number of teams that are somehow scared of using the Incubator code in their application even though it is open source and after all, who does not already use open source code in their applications. But for those of a nervous disposition, you can add Oracle Tools as a test dependancy without impacting your production code. Another point regarding dependencies is that although the Oracle Tools Coherence module is built against the current version of Coherence and has a dependdency on it as far as I know it will work fine with older versions so you should be fine using Oracle Tools even if your project is on an older version of Coherence (or your project is like mine where you have clients who may take your client API and run it with old versions).

For Maven users these are the dependencies you want.

<dependency>
    <groupId>com.oracle.tools</groupId>
    <artifactId>oracle-tools-core</artifactId>
    <version>${oracle.tools.revision}</version>
</dependency>
<dependency>
    <groupId>com.oracle.tools</groupId>
    <artifactId>oracle-tools-runtime</artifactId>
    <version>${oracle.tools.revision}</version>
</dependency>
<dependency>
    <groupId>com.oracle.tools</groupId>
    <artifactId>oracle-tools-testing-support</artifactId>
    <version>${oracle.tools.revision}</version>
</dependency>
<dependency>
    <groupId>com.oracle.tools</groupId>
    <artifactId>oracle-tools-coherence</artifactId>
    <version>${oracle.tools.revision}</version>
</dependency>

Where ${oracle.tools.revision} is whatever the latest version is.

Deferred – Waiting for conditions to be met

The main functionality in the oracle-tools-core module is the deferred package. This code provides a way to reference objects that may or may not yet exist but probably will at some point in the future. Thinking about Coherence, this could typically be something like an Extend Proxy Service that may not yet exist but at some point will and at some other future point will be started and listening for connections. Used in combination with the oracle-tools-test-support module that provides deferred assertions it becomes very easy to write tests cases based on waiting for something to happen.

Still keeping with the Coherence example, as that is what I mostly work with, you can imagine that using the deferred functionality can be useful in tests to make sure that the pre-conditions of the tests have been met, for example, wait until the cluster has started and has the correct number of members then wait until the proxy is running, once those conditions are met, then the tests can start. Now you could write that code yourself using various combinations of waits or sleeps, catching exceptions when services are not present etc, but that is all now nicely wrapped away.

Typically, waiting for a cluster to be started and have a specific number of nodes is as simple as this

assertThat(eventually(invoking(cluster).getClusterSize()), is(3));

The code above will wait for a default time of 30 seconds for the condition to be met, otherwise an exception is thrown and the test would fail. If you want to wait for a different amount of time then you can pass other parameters, for example to wait for five minutes…

assertThat(eventually(invoking(cluster).getClusterSize(), 5, TimeUnit.MINUTES), is(3));

As you can see, this makes waiting for conditions to be met in your tests very easy and can be used in a number of other situations, for example waiting for events, messages etc…

The deferred package contains a number of different classes that allow you to work with various types of deferred values. From a Java and Coherence perspective, one of the most useful is the deferred JMX functionality that allows you to obtain values from JMX for a process but instead of you having to write code to wait for the process to start, wait for the JMX server to start and wait for the MBean to be available, that is all taken care of. For example, to wait for the proxy service to start on the extend proxy cluster member you can write this in your test…

Int nodeId = proxyMember.getLocalMemberId();
ObjectName objectName = new ObjectName("Coherence:type=Service,name=ExtendTcpProxyService,nodeId=" + nodeId));
assertThat(eventually(invoking(proxyMember).getMBeanAttribute(objectName, "Running", Boolean.class)), is(true));

You will see these assertions used in the examples below.

Runtime – Process Control

As its name suggests the oracle-tools-runtime module contains the code from the old Coherence incubator’s runtime package, in this case the non-Coherence specific code. As I have mentioned in a previous post the Runtime functionality is all about making it easy to configure and control proceses that you may need as part of your application testing. These processes can be Java or non-Java and can be externally forked processes, or if they are Java, they can be isolated, virtualized processes inside the same JVM as your application tests. The runtime package is similar to what I previously blogged about with a few package and class name changes and new functionality.

The runtime package is based on the concept of Schema that define a process and builders that take a schema and realize a running process. Oracle Tools comes with schema for various types of processes, Java, non-Java and Coherence and builders for external processes and virtualized java processes.

I will run through a few simple examples to show how easy this is to use.

Simple External Processes

To run a simple external process you will need to have a dependency on the jar files from the oracle-tool-core and oracle-tools-runtime modules; so no Coherence dependencies yet.

This is a very basic example of running an external executable. For this example I wanted to do something pretty generic that would work anywhere. I work at home on a Mac Book Pro, in the office on Windows and I know some of you work on various flavours of Linux desktop so I chose to run the Java executable. It is a bit pointless in terms of doing anything useful, but it shows the functionality.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.runtime.SimpleApplication;
import com.oracle.tools.runtime.SimpleApplicationBuilder;
import com.oracle.tools.runtime.SimpleApplicationSchema;
import com.oracle.tools.runtime.console.SystemApplicationConsole;

public class RunSomethingForked
{
    public static void main(String[] args) throws Exception
    {
        SimpleApplicationSchema schema =
                new SimpleApplicationSchema("java")
                    .setArgument("-version");

        SimpleApplicationBuilder builder
                = new SimpleApplicationBuilder();

        SimpleApplication application
                = builder.realize(schema, "Java", new SystemApplicationConsole());

        int exitCode = application.waitFor();
    }
}

The interesting bit is lines 12 – 22.

• On line 12 we create an instance of a SimpleApplicationSchema which defines the process we want to run. This is the most simple type of process that requires as a minimum just the name of the executable to run; in this case “java”. We then call setArgument to pass in the command line argument -version that will cause the Java executable to print out the version information then exit.
• On line 16 we create an instance of the SimpleApplicationBuilder that will realize a running process from a given schema. As the name suggests SimpleApplicationBuilder can only build processes based on a SimpleApplicationSchema.
• On line 19 we call realize on the SimpleApplicationBuilder to build a running instance of a process based on our SimpleApplicationSchema. There are various overloaded versions of the realize method. At a minimum you need to pass a SimpleApplicationSchema. You also have the option to pass a process name and finally the version we have used where we pass in a name and an instance of an ApplicationConsole, in this case a SystemApplicationConsole.
  An ApplicationConsole is used to capture the output of the process and send it somewhere useful. In this case the SystemApplicationConsole captures the process output and sends it to the stdout of our parent process. There are a few built in implementations for sending the output to stdout or stderr or just ignoring it but it is easy enough to add your own if you want to do anything special.
• Finally on line 22 we wait for the process to exit and capture it’s exit code.

If we run the code above we see something like this displayed on the console (this image is from running inside IntelliJ)

This is the captured output of our process that has been formatted by the ApplicationConsole. The format of a line starts with [application-name:output-type:pid] where application-name is the name of the application that we passed to the builder’s realize method. The output-type is either out for the process stdout stream or err for the process stderr stream. Finally pid is the O/S PID of the process. This is then followed by a line number and colon. So [Java:err:37401] 1: is line 1 of the stderr stream of our process. We gave the process the name “Java” and the O/S assigned it a PID of 37401.
As you can see the first two err lines are typically what you would expect from running java -version. The two lines containing (terminated) are from the Oracle Tools framework to signify the end of the process output for that stream, so there will be one for stderr and one for stdout.

So what we have ended up with is basically the same as running java -version from a command line in our current working directory.

That is the most basic example of running a process. Using SimplaApplicationSchema we can configure the process executable, its command line arguments, working directory and environment variables. This is pretty much all you would need to run any externally forked process, after all it is no less than you get with Java’s own ProcessBuilder. Oracle Tools though has some specialisations of the ApplicationSchema that make it easier to configure Java applications and Coherence applications.

Non-Coherence Java Processes

To run an external non-Coherence java process you will need to have a dependency on the jar files from the oracle-tool-core and oracle-tools-runtime modules; so still no Coherence dependencies yet.

When working with Java processes you now have two options on how to run them. You can run them externally, i.e. as a forked process as we did above; or you can run them in a virtualised sandbox inside the same process. Running internally has advantages, especially when testing, in that you are not left with various forked processes still running if your parent process dies.

To configure a Java process instead of SimpleApplicationSchema we can use SimpleJavaApplicationSchema which has a number of useful methods to configure settings specific to Java processes. At a minimum we need to create an instance of SimpleJavaApplicationSchema and specify a class name to run. We then have the option to specify a whole host of other options such as class path, system properties, JVM options and JMX configuration.

External Java Process

First we will run an example of a simple Java class. As with our first example above I wanted to run something that everyone will have on their system without me having to use a custom class. In this case given my site is pretty much all about Oracle Coherence I assume you have a Coherence jar file somewhere so I am going to run com.tangosol.net.DefaultCacheServer. I know this is a Coherence class and this section of the blog is supposed to be running a non-Coherence Java process, so we will have to pretend for a while.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.ExternalJavaApplicationBuilder;
import com.oracle.tools.runtime.java.JavaApplication;
import com.oracle.tools.runtime.java.SimpleJavaApplicationSchema;

import java.io.File;

public class RunJavaClassForked
{
    public static void main(String[] args) throws Exception
    {
        String classpath = "/m2/repository/com/oracle/coherence/coherence/3.7.1.7/coherence-3.7.1.7.jar";

        SimpleJavaApplicationSchema schema =
                  new SimpleJavaApplicationSchema("com.tangosol.net.DefaultCacheServer", classpath)
                    .setArgument("coherence-cache-config.xml")
                    .setOption("-Xmx256m")
                    .setSystemProperty("tangosol.coherence.clusterport", "12345")
                    .setWorkingDirectory(new File("/Users/jonathanknight"));

        ExternalJavaApplicationBuilder builder =
                new ExternalJavaApplicationBuilder();

        JavaApplication application =
                builder.realize(schema, "DCS", new SystemApplicationConsole());

        Thread.sleep(10000);
        application.destroy();
    }
}

As with the previous example the interesting bit is lines 14 – 30.

• The first thing we need to do on lines 14 is work out what the class path will be for running our class. In this case we need a coherence.jar on the class path and on my Mac Book Pro this lives in a Maven repository.
• On line 16 and 17 we create our SimpleJavaApplicationSchema that configures the java application, we specify the class to run, in this case com.tangosol.net.DefaultCacheServer and the class path
  At this point we have nothing else you need to configure to run DefaultCacheServer, as it will happily start up with no more configuration at all but that would not be much of an example so we will set some other options
• On line 18 we set an argument that will be passed to the class’s main method – in this case coherence-cache-config.xml as DefaultCacheServer allows you to pass the name of the cache configuration file as the first argument..
  On line 19 we set a JVM option to set the maximum heap size to 256 MB.
  On line 20 we set a the multi-cast cluster port using the relevant system property
  On line 21 we set the working directory to my home directory.
• On line 23 we create our application builder, in this case we want to run a forked external Java process so we create an instance of ExternalJavaApplicationBuilder
• On line 26 we realize a running instance of our JConsole application, which we have named “DCS” and we are using a SystemApplicationConsole to capture the output
• On line 29 we sleep for 10 seconds and on line 30 we destroy (kill) the process (there is no real need to sleep, but this example would run so fast otherwise the process would be killed before anything was output). This is different to the previous example where we waited for the application to finish by calling waitFor(). DefaultCacheServer is a server application that does not end until we somehow shut it down it. If we used waitFor() as in the previous example then this code would block until we shut down the DefaultCacheServer. The code we have here will force the DefaultCacheServer to close after 10 seconds.

So if you run the code above you will see in the console output DefaultCacheServer start then after 10 seconds die as the process is killed.
When running an external Java process like the example above the class being run must have a public static void main(String[] args) method as with any Java class being run from the command line. In effect what we have just run above would equate to a command line of…
java -Xmx256m -tangosol.coherence.clusterport=12345 -cp=/m2/repository/com/oracle/coherence/coherence/3.7.1.7/coherence-3.7.1.7.jar com.tangosol.net.DefaultCacheServer coherence-cache-config.xml

While DefaultCacheServer is running, if you look at the processes running on your machine you should see an new Java process that is DefaultCacheServer and the PID should match the PID in the console output.

You can see that when we relized the process we recieved an instance of JavaApplication instead of the SimpleApplication in the previous examples. The JavaApplication class has a number of Java specific methods on it.

Internal Java Process

A feature of Oracle Tools is that it allows you to run Java processes in a virtualized sandbox inside the parent process. This sandbox isolates the applications class path, system properties and JMX server from the parent so for most purposes it can be treated like a separate application.

There are a few differences to be aware of between external and virtualized processes.

• You are not restricted to using the usual Java main method to start the virtualized process, you can use any method with the caveat that the method must not block.
• It is not possible for the destroy method to kill the process in the same way that it does with an external process. To stop the virtualized process you can specify a stop method that will be called on the class to programatically shut it down.

As an example we will run the same DefaultCacheServer again but this time as an internal virtualized process.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.JavaApplication;
import com.oracle.tools.runtime.java.SimpleJavaApplicationSchema;
import com.oracle.tools.runtime.java.VirtualizedJavaApplicationBuilder;

import java.io.File;

public class RunJavaClassInProcess
{
    public static void main(String[] args) throws Exception
    {
        String classpath = "/m2/repository/com/oracle/coherence/coherence/3.7.1.7/coherence-3.7.1.7.jar";

        SimpleJavaApplicationSchema schema =
                  new SimpleJavaApplicationSchema("com.tangosol.net.DefaultCacheServer", classpath)
                      .setStartMethodName("startDaemon")
                      .setStopMethodName("shutdown")
                      .setSystemProperty("tangosol.coherence.cacheconfig", "coherence-cache-config.xml")
                      .setOption("-Xmx256m")
                      .setSystemProperty("tangosol.coherence.clusterport", "12345")
                      .setWorkingDirectory(new File("/Users/jonathanknight"));

        VirtualizedJavaApplicationBuilder builder =
                new VirtualizedJavaApplicationBuilder();

        JavaApplication application =
                builder.realize(schema, "DCS", new SystemApplicationConsole());

        Thread.sleep(10000);
        application.destroy();
    }
}

You can see that the code is almost identical

The rest of the code is identical to the previous example. If we run the code you can see exactly the same output in the console as previously with the exception that there is no PID shown at the start of each line. This is because there is no external process and if you look at the processes running on your machine while the code runs you will see not other processes are spawned.

Coherence Processes

As we saw above it is possible to start a Coherence cache server by treating it as just another Java class, which it obviously is, but there is an easier way and more functionaly rich way if we use the Coherence specific parts of Oracle Tools. To run a Coherence process you will need to have a dependency on the jar files from the oracle-tool-core, oracle-tools-runtime and oracle-tools-coherence modules as well as providing a coherence.jar file. As Coherence applications are Java processes you have the same options regarding whether you run externally or in a virtualized process. Oracle Tools also provides a cluster builder that allows you to control a whole cluster of processes at once.

To create Coherence cluster member processes instead of using SimpleJavaApplicationSchema we use ClusterMemberSchema. This class has all of the same functionality as the previous two schema classes we have looked at, so you can set all of the normal process stuff, and Java specific properties, but it also has a host of Coherence specific settings too. For example, instead of setting the cache configuration using the setSystemProperty method as we did above, and having to remember to type the correct property name, we can just use ClusterMemberSchema.setCacheConfigURI instead.

So, on with a few examples…

External Coherence Process

In this example we will run a single cache server process, the same as the previous example but using ClusterMemberSchema.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.runtime.coherence.ClusterMember;
import com.oracle.tools.runtime.coherence.ClusterMemberSchema;
import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.ExternalJavaApplicationBuilder;

import java.io.File;

import static com.oracle.tools.deferred.DeferredAssert.assertThat;
import static com.oracle.tools.deferred.DeferredHelper.eventually;
import static com.oracle.tools.deferred.DeferredHelper.invoking;
import static org.hamcrest.CoreMatchers.is;

public class RunCacheServerForked
{
    public static void main(String[] args) throws Exception
    {
        String classpath = "/m2/repository/com/oracle/coherence/coherence/3.7.1.7/coherence-3.7.1.7.jar";

        ClusterMemberSchema schema = new ClusterMemberSchema("com.tangosol.net.DefaultCacheServer", classpath)
                .setCacheConfigURI("coherence-cache-config.xml")
                .setPofEnabled(true)
                .setPofConfigURI("pof-config.xml")
                .setClusterPort(12345)
                .setStorageEnabled(true)
                .setJMXManagementMode(ClusterMemberSchema.JMXManagementMode.LOCAL_ONLY)
                .setOption("-Xmx256m")
                .setWorkingDirectory(new File("/Users/jonathanknight"));

        ExternalJavaApplicationBuilder<ClusterMember, ClusterMemberSchema> builder =
                    new ExternalJavaApplicationBuilder<ClusterMember, ClusterMemberSchema>();

        ClusterMember member = builder.realize(schema, "Data", new SystemApplicationConsole());

        assertThat(eventually(invoking(member).getClusterSize()), is(1));

        Thread.sleep(10000);
        member.destroy();
    }
}

So, very similar to our previous examples, we create a schema and use a builder to realize the running process. This time though things are a bit more Coherence specific.

When we realize the Coherence process the builder returns an instance of ClusterMember which is a Coherence specific version of the previous JavaApplication. We can still get all of the same functionality as the JavaProcess but we also get some extra Coherence methods.

Internal Coherence Process

Running our Coherence process as a virtualized process is nothing more than change the builder used.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.runtime.coherence.ClusterMember;
import com.oracle.tools.runtime.coherence.ClusterMemberSchema;
import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.VirtualizedJavaApplicationBuilder;

import java.io.File;

import static com.oracle.tools.deferred.DeferredAssert.assertThat;
import static com.oracle.tools.deferred.DeferredHelper.eventually;
import static com.oracle.tools.deferred.DeferredHelper.invoking;
import static org.hamcrest.CoreMatchers.is;

public class RunCacheServerInProcess
{
    public static void main(String[] args) throws Exception
    {
        String classpath = "/m2/repository/com/oracle/coherence/coherence/3.7.1.7/coherence-3.7.1.7.jar";

        ClusterMemberSchema schema = new ClusterMemberSchema("com.tangosol.net.DefaultCacheServer", classpath)
                .setCacheConfigURI("coherence-cache-config.xml")
                .setPofEnabled(true)
                .setPofConfigURI("pof-config.xml")
                .setClusterPort(12345)
                .setStorageEnabled(true)
                .setJMXManagementMode(ClusterMemberSchema.JMXManagementMode.LOCAL_ONLY)
                .setOption("-Xmx256m")
                .setWorkingDirectory(new File("/Users/jonathanknight"));

        VirtualizedJavaApplicationBuilder<ClusterMember, ClusterMemberSchema> builder =
                    new VirtualizedJavaApplicationBuilder<ClusterMember, ClusterMemberSchema>();

        ClusterMember member = builder.realize(schema, "Data", new SystemApplicationConsole());

        assertThat(eventually(invoking(member).getClusterSize()), is(1));

        Thread.sleep(10000);
        member.destroy();
    }
}

You can see the code above is identical to the previous example except that it uses a VirtualizedJavaApplicationBuilder and runs the Coherence node as a virtualized process. We still have all the same functioanlity as the previous external process.

External Coherence Cluster

Typically when working with and testing Coherence application we are not working with a single process but a whole cluster. It is uaual in an accepance or integration test to run a couple of storage nodes and maybe, if you require it, an extend proxy node. This is where Oracle Tools makes it easier by allowing you to build and control a whole cluster together.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.deferred.DeferredAssert;
import com.oracle.tools.runtime.coherence.Cluster;
import com.oracle.tools.runtime.coherence.ClusterBuilder;
import com.oracle.tools.runtime.coherence.ClusterMember;
import com.oracle.tools.runtime.coherence.ClusterMemberSchema;
import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.ExternalJavaApplicationBuilder;
import com.oracle.tools.runtime.network.AvailablePortIterator;

import java.io.File;

import static com.oracle.tools.deferred.DeferredHelper.eventually;
import static com.oracle.tools.deferred.DeferredHelper.invoking;
import static org.hamcrest.CoreMatchers.is;

public class RunClusterForked
{
    public static void main(String[] args) throws Exception
    {
        AvailablePortIterator ports = new AvailablePortIterator(40000);

        ClusterMemberSchema storage
            = new ClusterMemberSchema()
                    .setCacheConfigURI("coherence-cache-config.xml")
                    .setPofEnabled(true)
                    .setPofConfigURI("pof-config.xml")
                    .setClusterPort(12345)
                    .setJMXManagementMode(ClusterMemberSchema.JMXManagementMode.LOCAL_ONLY)
                    .setJMXPort(ports)
                    .setOption("-Xmx256m")
                    .setWorkingDirectory(new File("/Users/jonathanknight"))
                    .setStorageEnabled(true);

        ClusterMemberSchema extend
            = new ClusterMemberSchema()
                    .setCacheConfigURI("coherence-cache-config.xml")
                    .setPofEnabled(true)
                    .setPofConfigURI("pof-config.xml")
                    .setClusterPort(12345)
                    .setJMXManagementMode(ClusterMemberSchema.JMXManagementMode.LOCAL_ONLY)
                    .setJMXPort(ports)
                    .setOption("-Xmx256m")
                    .setWorkingDirectory(new File("/Users/jonathanknight"))
                    .setStorageEnabled(false)
                    .setSystemProperty("tangosol.coherence.extend.enabled", true)
                    .setSystemProperty("tangosol.coherence.extend.port", ports);

        ExternalJavaApplicationBuilder<ClusterMember,ClusterMemberSchema> builder
                = new ExternalJavaApplicationBuilder<ClusterMember,ClusterMemberSchema>();

        ClusterBuilder clusterBuilder = new ClusterBuilder();
        clusterBuilder.addBuilder(builder, storage, "Data", 2);
        clusterBuilder.addBuilder(builder, extend, "Proxy", 1);

        Cluster cluster = clusterBuilder.realize(new SystemApplicationConsole());

        DeferredAssert.assertThat(eventually(invoking(cluster).getClusterSize()), is(3));

        cluster.destroy();
    }
}

If you run the code above you will see in the console output that there are three Coherence nodes started and all three should form a cluster. As soon as the assertion has determined that all three nodes have joined the cluster, the cluster is destroyed. If you are quick enough you should see the three external processes fire up on your machine.

Based on what we have already covered it should be easy to follow what the code is doing.

• One new things is line 22, where we create an AvailablePortIterator. This allows us to obtain free port numbers without having to hard code them in our tests and risk clashing or in use port numbers. The ports will be dynamically assigned as required when applications are realized from schema that use the iterator. We have a few places in the code where we assign ports, lines 31 and 43 where we assign JMX ports and line 48 where we set the extend port.
• Lines 24 – 34 create the schema that configures a storage node
• Lines 36 – 48 create the schema that configures an extend proxy node
• Lines 50 and 51 create the ExternalJavaApplicationBuilder as we want to fork the cluster as external processes
• Line 53 is the new part where we create our ClusterBuilder that will realize and control the whole cluster of processes
• Line 54 we tell the ClusterBuilder we want to realize two instances of processes from the storage node schema, the processes should be realized using the ExternalJavaApplicationBuilder and we will name the processes with the prefix “Data”. When the processes are realized they will actually be named Data-n where n is the instance number from zero to the number of required instance.
• Line 55 we tell the ClusterBuilder we want to realize a single instance of a process using the proxy schema that we will name “Proxy” and realize with the ExternalJavaApplicationBuilder
• Line 57 is where we actually realize the running cluster using the ClusterBuilder and obtain an instance of a Cluster. We use a SystemApplicationConsole to capture the output of all of the processes to stdout
• Line 59 we wait for the result of cluster.getClusterSize() to be three, the expected cluster size
• Line 61 we destroy the cluster, which will destroy all of the processes

Being able to control and group all the various members of the cluster together makes the code a bit tider than having everything separate.

Internal Coherence Cluster

As with the previous internal example if we want to run all the members of the cluster as virtualized processes all we need to do is change a single line in the code. We would change line 50 to create a VirtualizedJavaApplicationBuilder and that would be it, the whole thing would run just the same. I will not bother repeating all the code again.

Testing With Oracle Tools

To finish off the examples here is an example test class that uses what I have covered above to start a Coherence cluster then run a test method that interacts with the cluster. This example uses JUnit but you should easily be able to convert it if you use another test framework.

package com.thegridman.oracle.tools.examples;

import com.oracle.tools.deferred.DeferredAssert;
import com.oracle.tools.runtime.coherence.Cluster;
import com.oracle.tools.runtime.coherence.ClusterBuilder;
import com.oracle.tools.runtime.coherence.ClusterMember;
import com.oracle.tools.runtime.coherence.ClusterMemberSchema;
import com.oracle.tools.runtime.console.SystemApplicationConsole;
import com.oracle.tools.runtime.java.VirtualizedJavaApplicationBuilder;
import com.oracle.tools.runtime.network.AvailablePortIterator;
import com.tangosol.net.CacheFactory;
import com.tangosol.net.CacheFactoryBuilder;
import com.tangosol.net.ConfigurableCacheFactory;
import com.tangosol.net.NamedCache;
import com.tangosol.run.xml.XmlDocument;
import com.tangosol.run.xml.XmlElement;
import com.tangosol.run.xml.XmlHelper;
import com.tangosol.run.xml.XmlValue;
import org.junit.After;
import org.junit.AfterClass;
import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

import javax.management.ObjectName;
import java.io.File;
import java.util.List;
import java.util.Properties;

import static com.oracle.tools.deferred.DeferredHelper.eventually;
import static com.oracle.tools.deferred.DeferredHelper.invoking;
import static org.hamcrest.CoreMatchers.is;
import static org.hamcrest.CoreMatchers.notNullValue;

public class ClusterTest
{
    private static Cluster cluster;

    private ConfigurableCacheFactory clientCacheFactory;

    @BeforeClass
    public static void startCluster() throws Exception
    {
        AvailablePortIterator ports = new AvailablePortIterator(40000);

        ClusterMemberSchema storage = createStorageNodeSchema(ports);
        ClusterMemberSchema extend =  createExtendProxySchema(ports);

        VirtualizedJavaApplicationBuilder<ClusterMember,ClusterMemberSchema> builder
                = new VirtualizedJavaApplicationBuilder<ClusterMember,ClusterMemberSchema>();

        ClusterBuilder clusterBuilder = new ClusterBuilder();
        clusterBuilder.addBuilder(builder, storage, "Data", 2);
        clusterBuilder.addBuilder(builder, extend, "Proxy", 1);

        cluster = clusterBuilder.realize(new SystemApplicationConsole());
    }

    @Before
    public void setupTest() throws Exception
    {
        // Assert the cluster is ready
        Assert.assertThat(cluster, is(notNullValue()));
        DeferredAssert.assertThat(eventually(invoking(cluster).getClusterSize()), is(3));

        // Assert the Proxy Service is running
        assertServiceIsRunning("Proxy-0", "TcpProxyService");

        // Get the extend port the proxy is using
        ClusterMember proxyNode = cluster.getApplication("Proxy-0");
        String extendPort = proxyNode.getSystemProperty("tangosol.coherence.extend.port");

        // Create the client properties
        Properties properties = new Properties(System.getProperties());
        properties.setProperty("tangosol.coherence.extend.port", extendPort);

        // Create the client Cache Factory
        clientCacheFactory = createClientCacheFactory("client-cache-config.xml", properties);
    }

    @Test
    public void shouldPutDataIntoCache() throws Exception
    {
        NamedCache cache = clientCacheFactory.ensureCache("dist-test", null);
        cache.put("Key-1", "Value-1");
        Assert.assertThat((String)cache.get("Key-1"), is("Value-1"));
    }

    @After
    public void shutdownClient()
    {
        if (clientCacheFactory != null)
        {
            CacheFactory.getCacheFactoryBuilder().release(clientCacheFactory);
            clientCacheFactory = null;
        }
    }

    @AfterClass
    public static void stopCluster()
    {
        if (cluster != null)
        {
            cluster.destroy();
            cluster = null;
        }
    }

    public static ClusterMemberSchema createStorageNodeSchema(AvailablePortIterator ports) {
        return createCommonSchema(ports)
                    .setStorageEnabled(true);
    }

    public static ClusterMemberSchema createExtendProxySchema(AvailablePortIterator ports) {
        return createCommonSchema(ports)
                    .setStorageEnabled(false)
                    .setSystemProperty("tangosol.coherence.extend.enabled", true)
                    .setSystemProperty("tangosol.coherence.extend.port", ports);
    }

    public static ClusterMemberSchema createCommonSchema(AvailablePortIterator ports) {
        return new ClusterMemberSchema()
                .setCacheConfigURI("coherence-cache-config.xml")
                .setPofEnabled(true)
                .setPofConfigURI("pof-config.xml")
                .setClusterPort(12345)
                .setJMXManagementMode(ClusterMemberSchema.JMXManagementMode.LOCAL_ONLY)
                .setJMXPort(ports)
                .setOption("-Xmx256m")
                .setWorkingDirectory(new File("/Users/jonathanknight"));
    }

    private void assertServiceIsRunning(String memberName, String serviceName) throws Exception
    {
        Assert.assertThat(cluster, is(notNullValue()));
        ClusterMember member = cluster.getApplication(memberName);
        int nodeId = member.getLocalMemberId();
        ObjectName objectName = new ObjectName(String.format("Coherence:type=Service,name=%s,nodeId=%d", serviceName, nodeId));
        DeferredAssert.assertThat(eventually(invoking(member).getMBeanAttribute(objectName, "Running", Boolean.class)), is(true));
    }

    private ConfigurableCacheFactory createClientCacheFactory(String cacheConfigurationURI, Properties properties)
    {
        XmlDocument clientConfigXml = XmlHelper.loadFileOrResource(cacheConfigurationURI, "Client");
        replacePropertiesInXml(clientConfigXml, "system-property", properties);

        CacheFactoryBuilder cacheFactoryBuilder = CacheFactory.getCacheFactoryBuilder();
        cacheFactoryBuilder.setCacheConfiguration(cacheConfigurationURI, null, clientConfigXml);
        return cacheFactoryBuilder.getConfigurableCacheFactory(cacheConfigurationURI, null);
    }

    @SuppressWarnings("unchecked")
    private void replacePropertiesInXml(XmlElement xml, String propertyAttributeName, Properties properties)
    {
        XmlValue attribute = xml.getAttribute(propertyAttributeName);
        if (attribute != null)
        {
            xml.setAttribute(propertyAttributeName, null);
            try
            {
                String propertyValue = properties.getProperty(attribute.getString());
                if (propertyValue != null)
                {
                    xml.setString(propertyValue);
                }
            }
            catch (Exception _ignored)
            {
                // ignored on purpose
            }
        }

        for (XmlElement child : (List<XmlElement>) xml.getElementList())
        {
            replacePropertiesInXml(child, propertyAttributeName, properties);
        }
    }
}

We will go through the class bit by bit

@BeforeClass Method – Starting the Cluster

The obvious place to start the cluster for a test class is before anything else runs, so in this case we use a method annotated with @BeforeClass I have called obviously enough startCluster().

@BeforeClass
public static void startCluster() throws Exception
{
    AvailablePortIterator ports = new AvailablePortIterator(40000);

    ClusterMemberSchema storage = createStorageNodeSchema(ports);
    ClusterMemberSchema extend =  createExtendProxySchema(ports);

    VirtualizedJavaApplicationBuilder<ClusterMember,ClusterMemberSchema> builder
            = new VirtualizedJavaApplicationBuilder<ClusterMember,ClusterMemberSchema>();

    ClusterBuilder clusterBuilder = new ClusterBuilder();
    clusterBuilder.addBuilder(builder, storage, "Data", 2);
    clusterBuilder.addBuilder(builder, extend, "Proxy", 1);

    cluster = clusterBuilder.realize(new SystemApplicationConsole());
}

The startCluster() method should be easy to understand as it is pretty identical to the last example. We have extracted out the schema creation into two other methods createStorageNodeSchema and createExtendProxySchema. If you look at these methods you will see we have a further optimisation where we extract creation of all the common schema settings into another method createCommonSchema. This meas there is less code and is obviously less error prone.

@Before – Verify The Cluster is Ready & Setup the Test

Before we run any test methods we want to make sure that the cluster is available, and also in our case we are going to connect over Extend, so we want to make sure the proxy is running. On top of that we will also set up the client Cache Factory that the tests will use to connect to the cluster. It makes sense to do this in one common method rather than every test method (even though in this example we only have one test method). All of the code I have put in the @Before method could just have easily been put in the @BeforeClass method if you only want to do it once for the whole test class (or even somewhere once for a suite).

@Before
public void setupTest() throws Exception
{
    // Assert the cluster is ready
    Assert.assertThat(cluster, is(notNullValue()));
    DeferredAssert.assertThat(eventually(invoking(cluster).getClusterSize()), is(3));

    // Assert the Proxy Service is running
    assertServiceIsRunning("Proxy-0", "TcpProxyService");

    // Get the extend port the proxy is using
    ClusterMember proxyNode = cluster.getApplication("Proxy-0");
    String extendPort = proxyNode.getSystemProperty("tangosol.coherence.extend.port");

    // Create the client properties
    Properties properties = new Properties(System.getProperties());
    properties.setProperty("tangosol.coherence.extend.port", extendPort);

    // Create the client Cache Factory
    clientCacheFactory = createClientCacheFactory("client-cache-config.xml", properties);
}

So, in our @Before method the first thing we do is assert we actually have a cluster.
Next we assert that the cluster size is three, using a deferred assertion the same as we have done in other examples.
Next we assert that the proxy service is running. For this we use a utility method that checks via JMX that the service is running.

private void assertServiceIsRunning(String memberName, String serviceName) throws Exception
{
    Assert.assertThat(cluster, is(notNullValue()));
    ClusterMember member = cluster.getApplication(memberName);
    int nodeId = member.getLocalMemberId();
    ObjectName objectName = new ObjectName(String.format("Coherence:type=Service,name=%s,nodeId=%d", serviceName, nodeId));
    DeferredAssert.assertThat(eventually(invoking(member).getMBeanAttribute(objectName, "Running", Boolean.class)), is(true));
}

This is quite a useful method and I may at some point feed it back into the Oracle Tool project.
The method takes two parameters, the name of the member the service is on and the name of the service. If you remember our proxy member will be called Proxy-0 as we told the ClusterBuilder to use the prefix “Proxy” for the extend proxy member and there is only a single instance which will be instance zero. The second parameter is the name of the service; this name comes from the cache configuration file. We are using the coherence-cache-config.xml file from inside the coherence jar file and the name of the proxy service in this file is TcpProxyService.

• The first thing we do is assert we have a cluster, this is better than just throwing a NullPointerException later.
• Next we get the proxy member from the cluster using the specified member name.
• Now we have the member we need to get its cluster node ID, which Oracle Tools does via JMX.
• Once we have the node ID we can create the JMX ObjectName for the service. All service names in Coherence follow the same naming pattern so this is easy to do. There is a method on the member called getMBeanAttribute that will retunr the value of a given MBean attribute. We use this to get the value of the “Running” attribute of the service. We wrap this in a deferred assertion to give the service a reasonable time to start up.

Now we know the proxy is started we can configure our Cache Factory that the tests will use. If you look at the code of the test method you will see that the NamedCache is not obtained using the static CacheFactory.getCache method but from a real instance of a ConfigurableCacheFactory. Personally I would avoid using the static methods in Coherence to do things as much as you can. Using statics makes tests hard to control and makes things hard to mock. I know there are mock frameworks that let you mock static calls but doing this is like worshiping the devil, so better not to use statics in the first place. So, how do we create our client cache factory and tell it the ports for the proxy…

ClusterMember proxyNode = cluster.getApplication("Proxy-0");
String extendPort = proxyNode.getSystemProperty("tangosol.coherence.extend.port");

// Create the client properties
Properties properties = new Properties(System.getProperties());
properties.setProperty("tangosol.coherence.extend.port", extendPort);

// Create the client Cache Factory
clientCacheFactory = createClientCacheFactory("client-cache-config.xml", properties);

private ConfigurableCacheFactory createClientCacheFactory(String cacheConfigurationURI, Properties properties)
{
    XmlDocument clientConfigXml = XmlHelper.loadFileOrResource(cacheConfigurationURI, "Client");
    replacePropertiesInXml(clientConfigXml, "system-property", properties);

    CacheFactoryBuilder cacheFactoryBuilder = CacheFactory.getCacheFactoryBuilder();
    cacheFactoryBuilder.setCacheConfiguration(cacheConfigurationURI, null, clientConfigXml);
    return cacheFactoryBuilder.getConfigurableCacheFactory(cacheConfigurationURI, null);
}

@Test
public void shouldPutDataIntoCache() throws Exception
{
    NamedCache cache = clientCacheFactory.ensureCache("dist-test", null);
    cache.put("Key-1", "Value-1");
    Assert.assertThat((String)cache.get("Key-1"), is("Value-1"));
}

@After
public void shutdownClient()
{
    if (clientCacheFactory != null)
    {
        CacheFactory.getCacheFactoryBuilder().release(clientCacheFactory);
        clientCacheFactory = null;
    }
}

@AfterClass
public static void stopCluster()
{
    if (cluster != null)
    {
        cluster.destroy();
        cluster = null;
    }
}