This blog post is a write up of the London Oracle Coherence SIG presentation that I did on 31st May 2012. The post is about how to integrate Oracle Coherence and the Groovy programming language. This post is not a tutorial about Groovy, if you want to lean Groovy then there are a lot of good things already on the web in particular the Groovy site http://groovy.codehaus.org/ .Some of the techniques discussed here could be used with other scripting type languages too – although Scala Oracle Coherence – Yeah Baby! does not quite work as well as a blog title.
What is Groovy and Why integrate it with Oracle Coherence
Groovy is a light-weight, OO, dynamic programming language that runs on top of the JVM and has been around since about 2003.
By dynamic, we mean it has the ability to change the behaviour and structure of objects at runtime. They allow you to do at runtime what static languages do at compile time. This can be very powerful but of course with great power comes great responsibility and overuse or the wrong use of dynamic features of a language can leave you in a mess.
Groovy has a number of things going for it that make it ideal to integrate into Coherence. If you are a Java developer (which is highly likely if you are using Coherence) then the learning curve is low as Groovy’s semantics are a lot like Java. As already mentioned it is dynamic which means you do not have to have thought of everything before you build and deploy your cluster. You can do things after the fact without having to have deployed code. For me when investigating problems in running clusters this is a massive advantage. Groovy also has a number of quite cool extensions to the JDK that make it easier to do things in less lines of code.
You can use Groovy in two ways with Coherence. You can treat it just as a scripting language and only execute scripts from normal Java code or you can go the whole hog and use Coherence and extend Coherence from within Groovy code. We will cover both of these in turn.
Groovy Script
Groovy Script is basically a snippet of Groovy code that is executed as a standalone piece of script from within an application. This can be very useful for adding dynamic functionality to applications as any ah-hoc piece of script can be compiled and executed at runtime.
Obviously there are a number of uses for dynamic script in Coherence. As I’ve already said, for me personally, the place I would be most likely to use it would be when investigating issues or bugs as it allows me to execute code to query the state of data or Coherence services without having to have thought up the scenarios I might need in advance. It is always the case that it is not until I need to investigate an issue with a running cluster that I wish I had thought of deploying a bit of code – an example would be when we had corruption of indexes a year or so back and we had no code on the server side that allowed us to look in detail at what indexes were present on a node and what data they contained. Using Groovy Script it would have been much easier.
Java Scripting API
Groovy is one of the languages supported by JSR 223 – the scripting API so this makes it very easy to execute Groovy Script from within a Java application.
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
String script = "(1..10).sum()"
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine engine = factory.getEngineByName("groovy");
Object result = engine.eval(script);
The code above shows how simple it is. We have a String that contains the Groovy code (in this case we just sum up all the integers from 1 to 10). We then use the Java Scripting API to run the code. In the code above we end up executing the Groovy script and the return value of 55 is assigned to the result object.
Groovy Script in Coherence
As you can see Groovy script is just a String and is simple to execute which makes it very easy to integrate into various parts of Coherence. It becomes very easy to build a Groovy Script Invocable, EntryProcessor, Aggregator, in fact many parts of Coherence that execute ad-hoc code can be integrated with Groovy script.
A Groovy Script Invocable
As an example we will write a Groovy Invocable as this is about as basic as we can get. All our Invocable needs to do is take a piece of Groovy script as a parameter, execute the script and return any result.
Here is the code
public class GroovyScriptInvocable extends AbstractInvocable implements PortableObject
{
private String script;
public GroovyScriptInvocable()
{
}
public GroovyScriptInvocable(String script)
{
this.script = script;
}
@Override
public void run()
{
try
{
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine engine = factory.getEngineByName("groovy");
engine.put("invocationService", getService());
Object scriptResult = engine.eval(script);
setResult(scriptResult);
}
catch (ScriptException e)
{
throw ensureRuntimeException(e, "Error running Groovy script\n" + script);
}
}
@Override
public void readExternal(PofReader pofReader) throws IOException
{
this.script = pofReader.readString(0);
}
@Override
public void writeExternal(PofWriter pofWriter) throws IOException
{
pofWriter.writeString(0, this.script);
}
}
As you can see it is pretty basic. The run() method does the following
Sets up the scripting engine to execute Groovy script
We then set a script variable into the engine; in this case we add the InvocationService that is executing the Invocable as a parameter with the name invocationService. This InvocationService will then be available to code in the script by using the specified name.
Finally we execute the script and any return value is set as the result of the Invocable.
Using the GroovyScriptInvocable is just as simple, like any Invocable
String "(1..10).sum()";
Invocable invocable = new GroovyScriptInvocable(script);
InvocationService service = (InvocationService)
CacheFactory.getService("invocation-service");
Object result = service.query(invocable, null);
The above code sets a String variable with some Groovy Script – in this case we are summing the integer values between 1 and 10 (which comes to 55).
After the call to service.query the result variable will be set to 55.
Looking at the above code you can see it would be equally as simple to then write a Groovy Script EntryProcessor as the code would be virtually identical. This time though instead of setting an InvocationService as the parameter we can pass the InvocableMap.Entry the EntryProcessor is being invoked against.
public class GroovyScriptEntryProcessor extends AbstractProcessor implements PortableObject
{
private String script;
public GroovyScriptEntryProcessor()
{
}
public GroovyScriptEntryProcessor(String script)
{
this.script = script;
}
@Override
public Object process(InvocableMap.Entry entry)
{
try
{
ScriptEngineManager factory = new ScriptEngineManager();
ScriptEngine engine = factory.getEngineByName("groovy");
engine.put("entry", entry);
return engine.eval(script);
}
catch (ScriptException e)
{
throw ensureRuntimeException(e, "Error running Groovy script\n" + script);
}
}
@Override
public void readExternal(PofReader pofReader) throws IOException
{
script = pofReader.readString(0);
}
@Override
public void writeExternal(PofWriter pofWriter) throws IOException
{
pofWriter.writeString(0, script);
}
}
Integrating Coherence Directly Into Groovy
Now we have seen how easy it is to use Groovy Script we can get onto the cooler stuff and integrate Coherence directly into Groovy so that you can use it from Groovy code. Obviously as Coherence is a Java application you could use it from Groovy anyway without any changes by calling the standard API methods. What we are going to look at though is how we can extend the Coherence API in Groovy to make use of some of Groovy’s cooler features especially Closures.
Closures
I’m not going to go into a great deal of detail about what a closure is, there are plenty of other resources on the web that do that better than I can. Basically though a Closure is self contained block of code that optionally takes parameters and may return a value. In Java it is little like creating an anonymous inner class for something like a Runnable. The difference with a Closure is that it can be declared and then used later in the code, but also as it is a program variable then like any other Object it can be passed around and executed later. One important feature of a Groovy Closure though is that they can be dynamic and created and used on the fly.
As a Groovy Closure is an Object then it should be possible to pass it into Coherence providing it is possible to serialize the Closure.
Serializing a Closure
When I started to look at integrating Groovy and Coherence I was vey pleased to see that the Closure class implements Serializable, so joy of joys, it should be easy to serialize over the wire to a Coherence cluser. Well, my joy was to be short lived.
Whilst a Closure is indeed serializable it contains some internal fields that are not going over the wire and typically may not be Serializable. After a bit of Googling I found that later versions of Groovy have added a dehydrate method to a Closure which returns the same closure with these fields set to null so it can be serialized. Just about every cluster I have ever worked on uses POF so the next step was to write a POF serializer that can serialize the Closure.
Here is attempt number one…
Before the eagle eyed of you spot that Coherence itself can already cope with POF serializing Java Serialable classes, this is only attempt one of the serializer; we will need to expand on this code.
import com.tangosol.io.DefaultSerializer;
import com.tangosol.io.pof.PofReader;
import com.tangosol.io.pof.PofSerializer;
import com.tangosol.io.pof.PofWriter;
import com.tangosol.util.Base;
import com.tangosol.util.Binary;
import com.tangosol.util.ExternalizableHelper;
import groovy.lang.Closure;
import java.io.IOException;
public class GroovyClosureSerializer extends Base implements PofSerializer
{
@Override
public void serialize(PofWriter pofWriter, Object o) throws IOException
{
Closure closure = (Closure)o;
Closure dehydratedClosure = closure.dehydrate();
DefaultSerializer serializer = new DefaultSerializer(getContextClassLoader());
Binary binaryClosure = ExternalizableHelper.toBinary(dehydratedClosure, serializer);
pofWriter.writeBinary(1, binaryClosure);
pofWriter.writeRemainder(null);
}
@SuppressWarnings({"unchecked"})
@Override
public Object deserialize(PofReader pofReader) throws IOException
{
Binary binary = pofReader.readBinary(1);
pofReader.readRemainder();
DefaultSerializer serializer = new DefaultSerializer(getContextClassLoader());
Closure closure = (Closure) ExternalizableHelper.fromBinary(binary, serializer);
return closure;
}
}
The code above is a little verbose and could be done in less actual lines but it makes it easier to explain.
Coherence contains a class com.tangosol.io.DefaultSerializer that can serialize a Java Serializable class to Binary so we make use of this in our POF serializer. In our serialize method we do the following…
Cast our Object parameter to a Closure
Dehydrate the Closure to remove the potentially non-serializable parts
Use ExternalizableHelper to serialize the Closure to Binary
Write the Binary to our POF stream
Write a null remainder to the POF stream; we have to do this as we are writing an external serializer.
The deserialize method is basically the opposite…
Read the Binary from the POF stream
Read any remainder from the POF stream; in this case we throw any remainder away, it will be null anyway.
Finally use ExternalizableHelper and our DefaultSerializer to turn the Binary back to a Closure and return it.
So… that was easy but hold on lets think about this for a minute.
A Groovy Closure is in a lot of cases a dynamic piece of code that can be compiled on the fly on a client and assigned to an instance of a Closure. If we serialize this and send it over the wire to the cluster to execute there is something missing – the byte code for the closure will not be on the cluster as the serialized representation of a Closure does not contain the byte code. If we tried to run the code above we would get a ClassNotFoundException when the cluster tried to deserialize the closure. So how do we add the byte code to our GroovyClosureSerializer?
This is where things start to get tricky. After a lot of searching on the Groovy site and in Google it is actually not very easy to get hold of the byte code of a Closure. You would think that Groovy would have an API that lets you do this but it does not, maybe it should or will in the future. In the end I only found vague references to how to do this and only one place with any working code; that was the GroovyMX project, which is a Groovy JMX implementation here: https://github.com/nickman/GroovyMX and in particular the blog post here:
In effect what the GroovyMX does is use the Java Agent API to attach to the current process and then use the instrumentation API to be able to intercept class loading and see byte code. Rather than copy the techniques or pull out bits of code it was easier at this point to just include the GroovyMX jar as a dependency of my code and use the couple of classes I needed directly.
To obtain the byte code of a class GroovyMX contains a class called org.helios.gmx.util.ByteCodeNet which has a method called getClassBytes. This method returns a Map keyed on class name with values as byte[] which are then easy to POF serialize.
We can then change our serialize method of the GroovyClosureSerializer to also serialize the byte code.
public void serialize(PofWriter pofWriter, Object o) throws IOException
{
Closure closure = (Closure)o;
Closure dehydratedClosure = closure.dehydrate();
DefaultSerializer serializer = new DefaultSerializer(getContextClassLoader());
Binary binaryClosure = ExternalizableHelper.toBinary(dehydratedClosure, serializer);
pofWriter.writeBinary(1, binaryClosure);
Map<String, byte[]> classBytes = ByteCodeNet.getClassBytes(o.getClass());
pofWriter.writeMap(2, classBytes);
pofWriter.writeRemainder(null);
}
As you can see it is pretty simple with the addition of two lines to the serialize method. We can also change the deserialize method to deserialize the byte code.
public Object deserialize(PofReader pofReader) throws IOException
{
Binary binary = pofReader.readBinary(1);
Map<String,byte[]> byteCode = pofReader.readMap(2, new HashMap<String,byte[]>());
pofReader.readRemainder();
DefaultSerializer serializer = new DefaultSerializer(getContextClassLoader());
Closure closure = (Closure) ExternalizableHelper.fromBinary(binary, serializer);
return closure;
}
But we still have a problem as what do we do with the byte code to make sure it gets used. The obvious answer is we need a special ClassLoader that we can pass this byte code to and that will be used when we deserialize the Closure.
Lets think what we need this ClassLoader to be able to do. Basicaly we need to pass it the Map of class names to byte code and then when the ClassLoader is asked to load any class with that name, it returns the byte code we gave it.
public class GroovyClosureClassLoader extends ClassLoader
{
private final Map<String, byte[]> byteCode;
private final Map<String, Class> classes;
public GroovyClosureClassLoader(ClassLoader parent, Map<String, byte[]> byteCode)
{
super(parent);
this.byteCode = byteCode;
this.classes = new HashMap<String, Class>();
}
protected synchronized Class<?> loadClass(String name, boolean resolve)
throws ClassNotFoundException
{
if (classes.containsKey(name)) {
return classes.get(name);
}
if (byteCode.keySet().contains(name)) {
return findClass(name);
}
return super.loadClass(name, resolve);
}
@Override
protected Class<?> findClass(String name) throws ClassNotFoundException
{
if (byteCode.containsKey(name)) {
byte[] b = byteCode.get(name);
Class cls = defineClass(name, b, 0, b.length);
classes.put(name, cls);
return cls;
}
return super.findClass(name);
}
}
That is all of the code, not too complicated; now we can add this to our derserialize method and make our DefaultSerializer use this ClassLoader instead of the context ClassLoader.
public Object deserialize(PofReader pofReader) throws IOException
{
Binary binary = pofReader.readBinary(1);
Map<String,byte[]> byteCode = pofReader.readMap(2, new HashMap<String,byte[]>());
pofReader.readRemainder();
GroovyClosureClassLoader classLoader = new GroovyClosureClassLoader(getClass().getClassLoader(), byteCode);
DefaultSerializer serializer = new DefaultSerializer(classLoader);
Closure closure = (Closure) ExternalizableHelper.fromBinary(binary, serializer);
return closure;
}
There we are, job done. We can now successfully serialize a Closure into a POF stream to send to Coherence so now the fun begins…
Closure EntryProcessor
We’ll jump straight in and start with a Closure EntryProcessor. Our EntryProcessor will need to take a Closure as a parameter and just like the Groovy Script EntryProcessor we will pass the InvocableMap.Entry as a parameter to the Closure when we execute it.
Here is out GroovyClosureEntryProcessor
public class GroovyClosureEntryProcessor extends AbstractProcessor implements PortableObject
{
private Closure closure;
public GroovyClosureEntryProcessor()
{
}
public GroovyClosureEntryProcessor(Closure closure)
{
this.closure = closure;
}
@Override
public Object process(InvocableMap.Entry entry)
{
return closure.call(entry);
}
@Override
public void readExternal(PofReader pofReader) throws IOException
{
closure = (Closure) pofReader.readObject(0);
}
@Override
public void writeExternal(PofWriter pofWriter) throws IOException
{
pofWriter.writeObject(0, closure);
}
}
So that is even less code than the Groovy Script EntryProcessor. You can see that all the process method does is call the Closure passing the entry as a parameter and returns any result.
We can use the GroovyClosureEntryProcessor from Groovy like this…
As a trivial example say we have a cache of Accounts and we want to update the account with ID “A12345” to reduce its balance by 1000 then return the new balance. Assume we have adjustBalance() and getBalance() methods on our Account class. With our GroovyClosureEntryProcessor The Closure might look like this
def balanceUpdater { entry ->
def account = entry.getValue();
account.adjustBalance(-1000);
account.getBalance();
}
The balanceUpdater Closure takes a single parameter called event – there is no type declared for this parameter, as is usual in Groovy, but we know it will be an InvocableMap.Entry. So, we get the Account from the value of the entry and adjust its balance. We then return the account balance, there is no need to use the “return” keyword in Groovy, it always returns whatever the last line of code in the Closure evaluates to.
We can run the above Closure like this…
def balanceUpdater { entry ->
def account = entry.getValue();
account.adjustBalance(-1000);
account.getBalance();
}
def processor = new GroovyClosureEntryProcessor(balanceUpdater);
def accountsCache = CacheFactory.getCache("Accounts");
def balance = accountsCache.invoke("A12345", processor);
Extend Coherence Classes
The code above is not really any different to how you would use any EntryProcessor in Java, it is just using Groovy syntax. It would be much better, and cooler, to make the Coherence API work the same way that Groovy has extended normal Java Collections classes. That is, we want to be able to write code like this…
def accountsCache = CacheFactory.getCache("Accounts");
def balance = accountsCache.invoke("A12345", { entry ->
def account = entry.getValue();
account.adjustBalance(-1000);
account.getBalance();
});
In the code above we do not create a GroovyClosureEntryProcessor but call the invoke method on the cache as though it takes a Closure as a parameter instead of an EntryProcessor. If you already know any Groovy you will know that it is easy to extend the API of classes to do just this.
In Groovy we can use the metaClass attribute, which all classes have to alter the behaviour of classes at runtime. This is a very powerful feature and allows us to modify the API of classes that we do not have the source code for, such as Coherence.
Basically we need to add a method to NamedCache, or actually to com.tangosol.util.InvocableMap as this is the actual interface with the invoke method on it. Using the metaClass in Groovy we do it like this
InvocableMap.metaClass.invoke = {Object key, Closure c ->
invoke(key, new GroovyClosureEntryProcessor(c))
}
What we are basically doing above is adding a method to InvocableMap called invoke that takes and Object and a Closure as parameters. The body of the new method calls the real invoke method by wrapping the closure inside a GroovyClosureEntryProcessor.
Conclusions
That is it, and as you can see with Groovy we can alter the Coherence API to integrate it very well into Groovy and provide some very flexible and dynamic functionality. You could easily use closure to replace many of the Coherence classes that execute code, Invocables, Aggregators, Filters, ValueExtractors and so on…
The code above is early days and I’m sure there may be optimisations, especially if there was a way to not have to send the byte code with every closure.
The biggest use I can see for this code in my daily life would be investigating cluster issues. There are so many times where I need to figure out what is happening on a cluster but have not deployed the code to the cluster that I need. Combining this code with something like the Groovy Console would be a very powerful replacement for the standard Coherence console application.
Indeed this is really handy .