blog.armstrongconsulting.com

I had a hard time figuring out what was required to cluster our Wicket apps. None of the Wicket books discuss it in any detail – presumably because its not considered a Wicket-specific topic. I bothered the guys in the ##Wicket IRC Channel for a while and Victor Igumnov helped me figure it out and provided me with all the Wicket and Jetty pieces I needed.

Before I start, I should explain that we’re running our Wicket apps with embedded Jetty (”don’t run your apps in Jetty, run Jetty in your apps”). We really like it because it essentially turns web apps into plain old java apps which you can develop, debug and deploy without all that tomcat clutter. So we wanted clustering solution which worked with embedded Jetty, preferably in an uncomplicated fashion.

My first idea was to use Hazelcast – we’re already using it for distributed locking and queuing (its a paragon of simplicity, exemplifying the “it just works” principle). However, I was unable to find exactly what I needed based on Hazelcast, so I followed Victor’s advice and used memcached instead.

You need three things to cluster a Wicket app:
(1) A load balancer – we use HAProxy (as does everybody else).
(2) A distributed map to store the application sessions – we’re using memcached here.
(3) A http session implementation which stores the session in the distributed map instead of locally.

Since the main point of clustering is fault-tolerance, you need to make sure that you avoid single points of failure in the system. Unfortunately, this means that you have to cluster not only your application, but also provide failover for HAProxy and for memcached. I won’t cover that here, but you just need to be aware that when you install those components, you’ll need to install at least two of each with failover. This is simple for memcached – for HAProxy its a bit harder, but at least you can reuse this infrastructure for multiple apps.

HAProxy is the load balancer which will take care of presenting one address for your app to clients, even though your app is running as a cluster of servers. I’m not going to explain how to configure HAProxy here except to say that you’ll need session affinity (aka sticky-sessions) turned on – this means that users stay on a server for a whole session unless the server goes down. Wicket prefers sticky-session load balancing, and its more efficient, so its best to use it.

Installing memcached is no problem. Just follow the instructions. The default configuration only allows connections from localhost, so you’ll need to allow access from the subnet containing the app servers, but otherwise there’s nothing to configure.

Then you need to have your http session persist itself to memcached instead of to the local machine. Frsi download Victor’s tcache distributed cache client library from http://github.com/victori/tcache. Then get Victor’s jetty-session-cache from http://github.com/victori/jetty-session-cache. Add the built jars to your project (or maven repository). Victor describes how to configure jetty via XML to use the MemcachedSessionManager. Since we confgure the embedded Jetty programatically, we needed the following line in our Jetty startup code:

 
context.setSessionHandler(
  new SessionHandler(
     new MemcachedSessionManager(memcachedServers, "my_app_session_store")));

where memcachedServers is a String[] containing a list of your memcached servers, like “server1:11211″, “server2:11211″.

You then need to tell Wicket to store its pages in the http session instead of the default disk-based page store. However, this apparently doesn’t work very efficiently, so the recommended approach is to use a custom page store which stores its pages directly to memcached instead of using the http session.

Victor has a solution here too, go to http://letsgetdugg.com/2010/02/07/clustering-wicket-for-fun-and-profit/ and follow the instructions to override your applications newSessionStore() and provide a memcached page store.

Now your Wicket app is clustered. You can try it out by accessing it via HAProxy, logging on to your app, killing the server with your app’s session and then continue using the app (for test purposes, I display the server running the app session, so I can see when it fails over).

I’d still like to replace memcached with Hazelcast (fully java-based embedded clustering in your app with no single point of failure)- maybe Victor will embrace Hazelcast and add it to the list of caches supported by tcache.

=================================
26.08.2010
I got a mail from Talip Ozturk of Hazelcast to say that Hazelcast supports the memcached protocol, so it should be possible to use a memcached session and page store implementation against a Hazelcast server. This would be fantastic, since it would eliminate the need to have multiple memcached servers – one less piece of infrastructure to worry about. I’ll try to get this working and let you know how I get on.

Hazelcast (hazelcast.com) provides an easy way to implement distributed locking to allow your applications to run multiple, fault-tolerant instances without worrying about issues related concurrent access to shared resources (like files, databases or whatever).

try {
	java.util.concurrent.locks.Lock lock = Hazelcast.getLock("mylock");
	while (true) {
		lock.lock();
		try {
			// do some work involving access to shared resources
		} finally {
			lock.unlock();
		}
	}
} finally {
	Hazelcast.shutdown();
}

We have an SMS server which retrieved mail messages from a POP3 mailbox, entered them to a database and then delivered via the Clickatell messaging gateway. Making it run multiple instances concurrently would have been a headache since it would involve various issues relating to transactions across the POP3 and the Clickatell interfaces. It was a whole lot easier to wrap the business logic with a hazelcast lock and let it run on two servers. The beauty of hazelcast is that it just works – since the default configuration uses multicast to detect other members of the cluster, there’s no additional infrastructure – just add the jar to your application and off you go. We might still implement Zookeeper or Terracotta in future, but both of them require more infrastructure (i.e. dedicated (virtual) servers) and configuration.
Postscript: I had occasional hangs with Hazelcast 1.8.4 which caused me to switch to Zookeeper. As expected, Zookeeper was a lot harder to use than Hazelcast – you need Zookeeper installed on 3 servers. There’s no official java client, just some recipes and I found an implementation of Zookeeper locks called Cages on google code. For a java developer, Hazelcast is obviously way easier to use. Anyway, after upgrading to Hazelcast version 1.8.5, the hang problem went away so I happily went back to using Hazelcast. We’re also now using the distributed queues – works great so far.

Many of our applications require scripting support (allowing users to create scripts to customize workflow within the application). Java provides very straightforward scripting via the javax.script script-engine library. A simple integration is shown below where a Javascript onSave method provided by the user is called, if available, passing a business-logic object “item”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
 
public class Scripting extends Base {
 private ScriptEngineManager mgr;
 private ScriptEngine engine;
 private Invocable invocable;
 
 private boolean initialize (String script) {
  mgr = new ScriptEngineManager();
  engine = mgr.getEngineByName("js");
  try {
   engine.eval(script);
  } catch (ScriptException e) {
   warn("The script could not be loaded: " + e.getMessage());
   return false;
  }
  invocable = (Invocable)engine;
  return true;
 }
 
 public void onSave(Item item) {
  try {
   invocable.invokeFunction("onSave", item);
  } catch (ScriptException e) {
   warn("The onSave method of the script caused an error: " + e.getMessage());
  } catch (NoSuchMethodException e) {
   // no onSave method provided by the user - that's OK, just ignore it
  }
 }
}

Note: the javax.script library supports multiple scripting engines including javascript, python, groovy and java. Javascript was the easiest to get working because the Rhino Javascript engine is included in JDK 6.