Category Archives: Other

Clustering scheduled jobs with Quartz and Terracotta

We’re using Quartz to handle scheduled jobs in our java applications. Since we cluster our apps for failover using Terracotta, we need to address the issue of how to failover scheduled jobs (we only want each job to execute exactly once, but for fault tolerance it should be scheduled on multiple servers).

Quartz handles this situation with clustered job stores and Terracotta provides a job store for Quartz. Its easy to use – just add the quartz-terracotta jar (you’ll also need a quartz version > 1.7 – I’m using 1.8.4 below).



then set two systems properties. I’m doing it programmatically below because I use a commandline option to enable clustering in applications using embedded jetty. The “terracotta” commandline option is a terracotta url which tells the terracotta client where its servers are (something like “terracotta:9510,terracotta2:9510” for a fault-tolerant pair of terracotta servers).

if (cmd.hasOption("terracotta")) {
	System.setProperty("org.quartz.jobStore.class", "org.terracotta.quartz.TerracottaJobStore");
	System.setProperty("org.quartz.jobStore.tcConfigUrl", cmd.getOptionValue("terracotta"));

That’s it basically. The only additional thing you have to take care of is that while setting up the scheduled job, you need to be aware that another member of the cluster may have already set it up, so you should expect an ObjectAlreadyExistsException while setting up the quartz job.

Weird racing clock problem on VMWare Linux Guests

A while back we installed a new Dell server with a low-power Xeon 3GHz L3110 CPU to run some other essential network infrastructure. We chose the specific server configuration because it dissipates less than 30W while running 5 VMware VMs. It runs for hours on a UPS and doesn’t require cooling, so even if our server room air-conditioning were to die, this server should keep our network, firewalls, VPNs, DNS, DHCP and primary Terracotta server and a few other vital services up long enough for us to figure out what’s going on.

The server is running CentOS 5.5 and VMWare 1.0.10 – normally a very stable combination. However, we found that Linux guests running on this server were not keeping time properly. their clocks were running too fast – 50% too fast in fact. We finally figured out that the problem was caused by the fact that the CPU power-saving causes Linux to think that has a 2GHz processor instead of a 3GHz processor and this causes the 50% clock speedup in the Linux guests under VMWare. We disabled the demand-based power saving feature of the CPU in the BIOS and now it works correctly.

Hazelcast distributed locks for easy fault tolerance

Hazelcast ( provides an easy way to implement distributed locking to allow your applications to run multiple, fault-tolerant instances without worrying about issues related concurrent access to shared resources (like files, databases or whatever).

try {
	java.util.concurrent.locks.Lock lock = Hazelcast.getLock("mylock");
	while (true) {
		try {
			// do some work involving access to shared resources
		} finally {
} finally {

We have an SMS server which retrieved mail messages from a POP3 mailbox, entered them to a database and then delivered via the Clickatell messaging gateway. Making it run multiple instances concurrently would have been a headache since it would involve various issues relating to transactions across the POP3 and the Clickatell interfaces. It was a whole lot easier to wrap the business logic with a hazelcast lock and let it run on two servers. The beauty of hazelcast is that it just works – since the default configuration uses multicast to detect other members of the cluster, there’s no additional infrastructure – just add the jar to your application and off you go. We might still implement Zookeeper or Terracotta in future, but both of them require more infrastructure (i.e. dedicated (virtual) servers) and configuration.
Postscript: I had occasional hangs with Hazelcast 1.8.4 which caused me to switch to Zookeeper. As expected, Zookeeper was a lot harder to use than Hazelcast – you need Zookeeper installed on 3 servers. There’s no official java client, just some recipes and I found an implementation of Zookeeper locks called Cages on google code. For a java developer, Hazelcast is obviously way easier to use. Anyway, after upgrading to Hazelcast version 1.8.5, the hang problem went away so I happily went back to using Hazelcast. We’re also now using the distributed queues – works great so far.

Wicket dropdowns with mismatched models

Sometimes you have a class containing an id of another (lets say a Machine class containing an owner id rather than an Owner object). The owner id is actually a primary key from the Owner table in the database. Now you want to display the Machine class in a wicket form with a dropdown to select the Owner.

In an ideal world, your Machine class would have “Owner getOwner()” and “setOwner (Owner owner)” methods and your MachineService class would have a static “List getOwners()” method and wicket’s DropDownChoice would just work with:

  final Form<machine> form = new Form<machine>("form", new CompoundPropertyModel<machine>(machine));
  form.add(new DropDownChoice("owner", MachineService.getOwners()));

However, in our Machine class, the owner property is an id of an Owner and not an Owner object. So how do we use the DropDownChoice without changing our Machine class? There are a few different approaches, but a simple one is to use a custom model with the DropDownChoice which converts between object and id representation, as follows:

  final Form<machine> form = new Form<machine>("form", new CompoundPropertyModel<machine>(machine));

  Model<owner> model = new Model<owner>() {
  private static final long serialVersionUID = 1L;
    public Owner getObject() {
      if (machine.getOwner() == null)
	return null;
      return new Owner(machine.getOwner());

    public void setObject(Owner owner) {
      if (owner == null) {
      } else {
 form.add(new DropDownChoice<owner>("owner", model, ControlCenterService.getOwners()));

So now, when wicket needs to get the current selection, it calls getObject on your custom model and gets an object which it can compare. When it needs to set the selection, it calls setObject which writes the correct id into the machine object. Note: Your Machine class will need to implement toString() for the display in the dropdown. Additionally it will need a constructor which takes an id and an overridden equals() and hashCode() based on the id field.

Enforcing multiple levels of page access in Wicket

In Wicket when you want to require user login for some pages, you typically inherit the pages from a superclass called say ProtectedPage and then use a wicket SimplePageAuthorizationStrategy to protect these pages as follows:

getSecuritySettings().setAuthorizationStrategy(new SimplePageAuthorizationStrategy(ProtectedPage.class, LoginPage.class) {
  protected boolean isAuthorized() {
    return ((ApplicationSession) Session.get()).getLoggedInUser() != null;

This will cause any access to a protected page by a non-logged-in user to be redirected to the login page first.

In the real world, its likely however, that your application will also require administrator privileges for access to certain pages. To do this, you need to use a wicket CompoundAuthorizationStrategy as follows:

  CompoundAuthorizationStrategy pageAuthorizationStrategy = new CompoundAuthorizationStrategy();
  pageAuthorizationStrategy.add(new SimplePageAuthorizationStrategy(AdminPage.class, AccessDeniedPage.class) {
     protected boolean isAuthorized() {
       return (((ApplicationSession) Session.get()).getLoggedInUser() != null) && (((ApplicationSession) Session.get()).getLoggedInUser().isAdministrator());
  pageAuthorizationStrategy.add(new SimplePageAuthorizationStrategy(ProtectedPage.class, LoginPage.class) {
     protected boolean isAuthorized() {
       return ((ApplicationSession) Session.get()).getLoggedInUser() != null;

The above requires the user to be logged in to access any page inherited from ProtectedPage and to be both logged in AND be an administrator to access any page inherited from AdminPage.

Thanks to selckin for the tip!

Embedded Jetty instead of Tomcat

I recently used embedded Jetty for a small REST server project and it was a revelation. It took exactly no time to get it running and now I can deploy web-apps just by starting a runnable jar. That’s reason enough – no more Tomcat with all the complexity of sharing a single server amongst multiple apps, but, even more importantly, its wonderful for development. No more fooling around with Tomcat in Eclipse – just debug web-apps directly as a Java application.

We run all of our apps as multiple instances behind a load-balancer (HAProxy), so its convenient to run each app on a separate port and let the load-balancer handle the forwarding from port 80.

JDBC vs IBATIS vs Hibernate

Christoph has spent the last couple of weeks refactoring a web project to use IBATIS instead of Hibernate. I also refactored a smaller project to use IBATIS instead of JDBC. Here’s our take on the issues:

If you like JDBC and PreparedStatements, you’ll feel comfortable with IBATIS. It doesn’t force any major changes on you and reduces the lines of boilerplate code you’ll need for your database access. It has other benefits as well, such as automatically handling boolean and date mappings which are tedious and error-prone in JDBC.

We had successfully used Hibernate in a number of other projects, but in this particular project, Hibernate turned out to be an awkward fit. The problem was that we already had a database model which we couldn’t change and furthermore, we were porting an existing code base. This combination proved overly complex with Hibernate and we eventually redid the DAO in IBATIS, resulting in a much simpler port.

In summary, if you’re having difficulties integrating Hibernate into your application, consider using IBATIS instead – you get many of the benefits of ORM without the paradigm shifts which Hibernate implies.

LDAPAuthentication with Active Directory

Here’s something I had to piece together myself from various code fragments. Its a class to authenticate a user in Windows Active Directory environment using LDAP. It first locates the domain controllers (DNS lookup of SRV records for _ldap._tcp.domain), parses out the server part and then tries to authenticate the user against a domain controller.

import java.util.ArrayList;
import java.util.Hashtable;
import java.util.List;

import javax.naming.AuthenticationException;
import javax.naming.CommunicationException;
import javax.naming.Context;
import javax.naming.NamingException;

import com.sun.jndi.ldap.LdapCtxFactory;

* LDAPAuthentication class for authenticating Microsoft Active Directory users
* If the user or password is wrong, you’ll get an AuthenticationException If
* none of the domain controllers are reachable, you’ll get a
* CommunicationException. If a domain controller cannot be located (via DNS)
* you’ll get a NamingException.
* @author Roger Armstrong, Armstrong Consulting GmbH
public class LDAPAuthentication {
public static void authenticateUser(String user, String password, String domain) throws AuthenticationException, NamingException {
List ldapServers = findLDAPServersInWindowsDomain(domain);
if (ldapServers.isEmpty())
throw new NamingException(“Can’t locate an LDAP server (try nslookup type=SRV _ldap._tcp.” + domain + “)”);

Hashtable props = new Hashtable();
String principalName = user + “@” + domain;
props.put(Context.SECURITY_PRINCIPAL, principalName);
props.put(Context.SECURITY_CREDENTIALS, password);
Integer count = 0;
for (String ldapServer : ldapServers) {
try {
LdapCtxFactory.getLdapCtxInstance(“ldap://” + ldapServer, props);
} catch (CommunicationException e) { // this is what’ll happen if one of the domain controllers is unreachable
if (count.equals(ldapServers.size())) {
// we’ve got no more servers to try, so throw the CommunicationException to indicate that we failed to reach an LDAP server
throw e;

private static List findLDAPServersInWindowsDomain(String domain) throws NamingException {
List servers = new ArrayList();
Hashtable env = new Hashtable();
env.put(Context.INITIAL_CONTEXT_FACTORY, “com.sun.jndi.dns.DnsContextFactory”);
env.put(“java.naming.provider.url”, “dns:”);
DirContext ctx = new InitialDirContext(env);
Attributes attributes = ctx.getAttributes(“_ldap._tcp.” + domain, new String[] { “SRV” }); // that’s how Windows domain controllers are registered in DNS
Attribute a = attributes.get(“SRV”);
for (int i = 0; i < a.size(); i++) { String srvRecord = a.get(i).toString(); // each SRV record is in the format "0 100 389" // priority weight port server (space separated) servers.add(srvRecord.split(" ")[3]); } ctx.close(); return servers; } } [/sourcecode]

Adding an activity monitor to your wicket application

Sometimes its useful in an application to have a list of users who are currently logged onto an application on a server.

To do this, you first need a SessionAttributeListener which listens for attributes being added to and removed from the session:

Then from within your application login, you can set a session attribute when a user logs on

You don’t need to remove the attribute on logout, as long as you are invalidating the session – this will automatically remove the attribute and invoke the attributeRemoved method of the listener.
The final step is to add an activity monitor display to your application which calls SessionAttributeListener.getActiveUsers() and displays the list.

How to display images in Wicket

For someone starting out with Wicket, it can be tricky to figure out basic things like how to display images. Here are basic three scenarios:

1. Simple fixed image – no wicket code required:

This is fine as long as you don’t need any control over the image.

2. Image set from code:


The ContextRelativeResource method here tells wicket to load the image from the images folder in your application (the same path as used in the relative src attribute used in the simple HTML example 1).

3. Image set using CSS:


This approach is preferred for images which are really only part of the styling of the application (like say a green tick beside a list item) because it reduces the image to an an implicit part of a CSS class, allowing it to be restyled or removed without affecting the code. To change the CSS class of a component at runtime using wicket, you can use a wicket AttributeModifier to change the CSS class of the component.

Note: in the example above, the image path is relative to the referring stylesheet (so if the stylesheet is in the /CSS folder, the image will have to be in /CSS/images folder). However, since the image is simply part of the styles, it makes sense to keep such images together with the stylesheet.