Before I start, I should explain that we’re running our Wicket apps with embedded Jetty (”don’t run your apps in Jetty, run Jetty in your apps”). We really like it because it essentially turns web apps into plain old java apps which you can develop, debug and deploy without all that tomcat clutter. So we wanted clustering solution which worked with embedded Jetty, preferably in an uncomplicated fashion.

My first idea was to use Hazelcast – we’re already using it for distributed locking and queuing (its a paragon of simplicity, exemplifying the “it just works” principle). However, I was unable to find exactly what I needed based on Hazelcast, so I followed Victor’s advice and used memcached instead.

You need three things to cluster a Wicket app:
(1) A load balancer – we use HAProxy (as does everybody else).
(2) A distributed map to store the application sessions – we’re using memcached here.
(3) A http session implementation which stores the session in the distributed map instead of locally.

Since the main point of clustering is fault-tolerance, you need to make sure that you avoid single points of failure in the system. Unfortunately, this means that you have to cluster not only your application, but also provide failover for HAProxy and for memcached. I won’t cover that here, but you just need to be aware that when you install those components, you’ll need to install at least two of each with failover. This is simple for memcached – for HAProxy its a bit harder, but at least you can reuse this infrastructure for multiple apps.

HAProxy is the load balancer which will take care of presenting one address for your app to clients, even though your app is running as a cluster of servers. I’m not going to explain how to configure HAProxy here except to say that you’ll need session affinity (aka sticky-sessions) turned on – this means that users stay on a server for a whole session unless the server goes down. Wicket prefers sticky-session load balancing, and its more efficient, so its best to use it.

Installing memcached is no problem. Just follow the instructions. The default configuration only allows connections from localhost, so you’ll need to allow access from the subnet containing the app servers, but otherwise there’s nothing to configure.

Then you need to have your http session persist itself to memcached instead of to the local machine. Frsi download Victor’s tcache distributed cache client library from http://github.com/victori/tcache. Then get Victor’s jetty-session-cache from http://github.com/victori/jetty-session-cache. Add the built jars to your project (or maven repository). Victor describes how to configure jetty via XML to use the MemcachedSessionManager. Since we confgure the embedded Jetty programatically, we needed the following line in our Jetty startup code:

 
context.setSessionHandler(
  new SessionHandler(
     new MemcachedSessionManager(memcachedServers, "my_app_session_store")));

where memcachedServers is a String[] containing a list of your memcached servers, like “server1:11211″, “server2:11211″.

You then need to tell Wicket to store its pages in the http session instead of the default disk-based page store. However, this apparently doesn’t work very efficiently, so the recommended approach is to use a custom page store which stores its pages directly to memcached instead of using the http session.

Victor has a solution here too, go to http://letsgetdugg.com/2010/02/07/clustering-wicket-for-fun-and-profit/ and follow the instructions to override your applications newSessionStore() and provide a memcached page store.

Now your Wicket app is clustered. You can try it out by accessing it via HAProxy, logging on to your app, killing the server with your app’s session and then continue using the app (for test purposes, I display the server running the app session, so I can see when it fails over).

I’d still like to replace memcached with Hazelcast (fully java-based embedded clustering in your app with no single point of failure)- maybe Victor will embrace Hazelcast and add it to the list of caches supported by tcache.

=================================
26.08.2010
I got a mail from Talip Ozturk of Hazelcast to say that Hazelcast supports the memcached protocol, so it should be possible to use a memcached session and page store implementation against a Hazelcast server. This would be fantastic, since it would eliminate the need to have multiple memcached servers – one less piece of infrastructure to worry about. I’ll try to get this working and let you know how I get on.

try {
	java.util.concurrent.locks.Lock lock = Hazelcast.getLock("mylock");
	while (true) {
		lock.lock();
		try {
			// do some work involving access to shared resources
		} finally {
			lock.unlock();
		}
	}
} finally {
	Hazelcast.shutdown();
}

We have an SMS server which retrieved mail messages from a POP3 mailbox, entered them to a database and then delivered via the Clickatell messaging gateway. Making it run multiple instances concurrently would have been a headache since it would involve various issues relating to transactions across the POP3 and the Clickatell interfaces. It was a whole lot easier to wrap the business logic with a hazelcast lock and let it run on two servers. The beauty of hazelcast is that it just works – since the default configuration uses multicast to detect other members of the cluster, there’s no additional infrastructure – just add the jar to your application and off you go. We might still implement Zookeeper or Terracotta in future, but both of them require more infrastructure (i.e. dedicated (virtual) servers) and configuration.
Postscript: I had occasional hangs with Hazelcast 1.8.4 which caused me to switch to Zookeeper. As expected, Zookeeper was a lot harder to use than Hazelcast – you need Zookeeper installed on 3 servers. There’s no official java client, just some recipes and I found an implementation of Zookeeper locks called Cages on google code. For a java developer, Hazelcast is obviously way easier to use. Anyway, after upgrading to Hazelcast version 1.8.5, the hang problem went away so I happily went back to using Hazelcast. We’re also now using the distributed queues – works great so far.

In an ideal world, your Machine class would have “Owner getOwner()” and “setOwner (Owner owner)” methods and your MachineService class would have a static “List getOwners()” method and wicket’s DropDownChoice would just work with:

  final Form<Machine> form = new Form<Machine>("form", new CompoundPropertyModel<Machine>(machine));
  add(form);
  form.add(new DropDownChoice("owner", MachineService.getOwners()));

However, in our Machine class, the owner property is an id of an Owner and not an Owner object. So how do we use the DropDownChoice without changing our Machine class? There are a few different approaches, but a simple one is to use a custom model with the DropDownChoice which converts between object and id representation, as follows:

  final Form<Machine> form = new Form<Machine>("form", new CompoundPropertyModel<Machine>(machine));
  add(form);
 
  Model<Owner> model = new Model<Owner>() {
  private static final long serialVersionUID = 1L;
    @Override
    public Owner getObject() {
      if (machine.getOwner() == null)
	return null;
      return new Owner(machine.getOwner());
    }
 
    @Override
    public void setObject(Owner owner) {
      if (owner == null) {
        machine.setOwner(null);
      } else {
        machine.setOwner(owner.getOwnerNr());
      }
    }
 };
 form.add(new DropDownChoice<Owner>("owner", model, ControlCenterService.getOwners()));

So now, when wicket needs to get the current selection, it calls getObject on your custom model and gets an object which it can compare. When it needs to set the selection, it calls setObject which writes the correct id into the machine object. Note: Your Machine class will need to implement toString() for the display in the dropdown. Additionally it will need a constructor which takes an id and an overridden equals() and hashCode() based on the id field.

getSecuritySettings().setAuthorizationStrategy(new SimplePageAuthorizationStrategy(ProtectedPage.class, LoginPage.class) {
  @Override
  protected boolean isAuthorized() {
    return ((ApplicationSession) Session.get()).getLoggedInUser() != null;
  }
});

This will cause any access to a protected page by a non-logged-in user to be redirected to the login page first.

In the real world, its likely however, that your application will also require administrator privileges for access to certain pages. To do this, you need to use a wicket CompoundAuthorizationStrategy as follows:

  CompoundAuthorizationStrategy pageAuthorizationStrategy = new CompoundAuthorizationStrategy();
  pageAuthorizationStrategy.add(new SimplePageAuthorizationStrategy(AdminPage.class, AccessDeniedPage.class) {
    @Override
     protected boolean isAuthorized() {
       return (((ApplicationSession) Session.get()).getLoggedInUser() != null) && (((ApplicationSession) Session.get()).getLoggedInUser().isAdministrator());
     }
   });
  pageAuthorizationStrategy.add(new SimplePageAuthorizationStrategy(ProtectedPage.class, LoginPage.class) {
    @Override
     protected boolean isAuthorized() {
       return ((ApplicationSession) Session.get()).getLoggedInUser() != null;
     }
   });
   getSecuritySettings().setAuthorizationStrategy(pageAuthorizationStrategy);

The above requires the user to be logged in to access any page inherited from ProtectedPage and to be both logged in AND be an administrator to access any page inherited from AdminPage.

Thanks to selckin for the tip!

We run all of our apps as multiple instances behind a load-balancer (HAProxy), so its convenient to run each app on a separate port and let the load-balancer handle the forwarding from port 80.

If you like JDBC and PreparedStatements, you’ll feel comfortable with IBATIS. It doesn’t force any major changes on you and reduces the lines of boilerplate code you’ll need for your database access. It has other benefits as well, such as automatically handling boolean and date mappings which are tedious and error-prone in JDBC.

We had successfully used Hibernate in a number of other projects, but in this particular project, Hibernate turned out to be an awkward fit. The problem was that we already had a database model which we couldn’t change and furthermore, we were porting an existing code base. This combination proved overly complex with Hibernate and we eventually redid the DAO in IBATIS, resulting in a much simpler port.

In summary, if you’re having difficulties integrating Hibernate into your application, consider using IBATIS instead – you get many of the benefits of ORM without the paradigm shifts which Hibernate implies.

public class SessionAttributeListener implements HttpSessionAttributeListener {
	private static List<String> activeUsers = Collections.synchronizedList(new ArrayList<String>()); // listener is being called from multiple threads and ArrayList is unsynchronized
 
	public static List<String> getActiveUsers() {
		return activeUsers;
	}
 
	@Override
	public void attributeAdded(HttpSessionBindingEvent event) {
		if (event.getName().equals("wicket:wicket.acpro:user.name"))
			activeUsers.add(event.getValue().toString());
	}
 
	@Override
	public void attributeRemoved(HttpSessionBindingEvent event) {
		if (event.getName().equals("wicket:wicket.acpro:user.name"))
			activeUsers.remove(event.getValue().toString());
	}
 
	@Override
	public void attributeReplaced(HttpSessionBindingEvent event) {
	}
}

Then from within your application login, you can set a session attribute when a user logs on

Session.get().setAttribute("user.name", getLoggedOnUser().getName());

You don’t need to remove the attribute on logout, as long as you are invalidating the session – this will automatically remove the attribute and invoke the attributeRemoved method of the listener.

The final step is to add an activity monitor display to you application which calls SessionAttributeListener.getActiveUsers() and displays the list.

<img src="images/search_icon.png"/>

This is fine as long as you don’t need any control over the image.

<img wicket:id="search_icon"/>

JAVA:

.add(new Image("search_icon", new ContextRelativeResource("/images/search_icon.png")))

The ContextRelativeResource method here tells wicket to load the image from the images folder in your application (the same path as used in the relative src attribute used in the simple HTML example 1).

<div class="search"/>

CSS:

.search {
	background-image: url(images/search_icon.png);
	background-position: center center;
	background-repeat: no-repeat;
	width: 20px;
	height: 20px;
}

This approach is preferred for images which are really only part of the styling of the application (like say a green tick beside a list item) because it reduces the image to an an implicit part of a CSS class, allowing it to be restyled or removed without affecting the code. To change the CSS class of a component at runtime using wicket, you can use a wicket AttributeModifier to change the CSS class of the component.


Note: in the example above, the image path is relative to the referring stylesheet (so if the stylesheet is in the /CSS folder, the image will have to be in /CSS/images folder). However, since the image is simply part of the styles, it makes sense to keep such images together with the stylesheet.

.add(new AttributeModifier("class", true, new Model<String>("search")));