Testing Complex Web Applications with Cucumber and Selenium in Java

At GMO Internet when we started using Cucumber and Selenium for testing our web apps. One thing that we found early on was: creating the preconditions for testing and cleaning up after testing often requires more than just Selenium.

This article explains a useful technique not covered by Cucumber or Selenium but complementary to both.

Before getting started, we recommend using Cucumber and Selenium as described in this excellent article at IBM developerworks: Automated testing with Selenium and Cucumber. The article covers binding Cucumber tests to Java test classes, and includes a powerful Java project structure for using Selenium to test potentially complex web apps.

However great the Cucumber/Selenium combo is, it is still lacking for testing real world complex web applications. For example we need to be able to execute arbitrary commands on remote servers to start servers, run batch programs and cleanup left-over test artifacts.

Executing commands on remote servers

Of course commands can be run on remote Unix type servers easily from the command line using SSH as shown below:

ssh user@host <<'ENDSSH'
cd /batch_dir
mvn exec:java -Dexec.mainClass="jp.gmo.internet.MyBatch"
ENDSSH

But we want to be able to run such commands from our Java tests. For this we will use JSch the excellent pure Java implementation of SSH2: http://www.jcraft.com/jsch/

Using code scavenged from the internet we were able to quickly knock up a remote command library in Java for easy execution from within our tests.

Here is an example of what it looks like:

import static jp.gmo.internet.bdd.util.RemoteCommand.command;
import static jp.gmo.internet.bdd.util.RemoteServerFactory.batchServer;

for(String out: command(batchServer()).run("cd /batch_dir && mvn exec:java -Dexec.mainClass="jp.gmo.internet.MyBatch"")) {
   logger.debug(out);
}

The code imports two static factory methods. Firstly, command() a factory method to get an instance of a RemoteCommand which we use to
invoke commands. Secondly, batchServer(); a method providing a server configuration for our batch server. Then we simply invoke the desired commands and loop through and log the command output.

The command will executed by the user specified in the server configuration returned by the batchServer method. So let’s take a closer look at that method.

private static RemoteServer batchServer;

public static RemoteServer batchServer() {
   if (batchServer == null) {
      batchServer = new RemoteServer(
        Config.getConfigString("batch.user.name"),
        Config.getConfigString("batch.user.private.key"),
        Config.getConfigString("batch.user.public.key"),
        Config.getConfigString("batch.server.ip"),
        Config.getConfigInt("batch.server.ssh.port", 22));
   }
   return batchServer;
}

If the batchServer object doesn’t already exist, the batchServer() method instantiates the batchServer object using application settings. You could
just as easily substitute your own custom configuration code here, or bypass this factory method completely and create the RemoteServer object as you need it. We prefer this approach because we use the batch RemoteServer object more than once within our test.

Usage Scenarios in Testing

Before the Test

Test setup can be handled with methods declared using the Cucumber @Before annotation, or the JUnit annotation if you are not using Cucumber. Such a method will be run before each feature test. For example it is common that you need to create files or run batch programs before a test.

@Before
public void setup() throws Throwable {
   try {
      command(fileServer()).run("mkdir /var/sftp/" + contract.getSftpUser());
   } catch (Exception e) {
      logger.warn("failed to create sftp directory: " + contract, e);
   }
}

Cleaning Up After

@After
public void cleanUp() throws Throwable {
   try {
      command(fileServer()).run("rm -rf /var/sftp/" + contract.getSftpUser());
   } catch (Exception e) {
      logger.warn("failed to delete sftp directory: " + contract, e);
   }
}

During the Test

Given the method below that checks and waits for a file to be copied to HDFS:

    private boolean fileExistsInHDFS() {
        boolean found = false;
        int i = 0;
        final RemoteCommand batchCommand = command(batchServer());

        // wait a maximum of 5 minutes for the file to be copied to HDFS
        while (!found && i < 5) {
            for (String output: batchCommand.run("hadoop fs -cat /user/flume/log/log.* | " +
                    "grep " + this.id)) {
                found = output.contains(this.id);
                if (found) {
                    break;
                }
            }
            i++;
            if (!found) {
                try {
                    Thread.sleep(60000);
                } catch (InterruptedException e) {
                    logger.warn("woken up while waiting for file to be copied to HDFS", e);
                }
            }
        }
        return found;
    }

It becomes trivial to assert that the file exists in HDFS in our test:

    @Then("^the file exists in HDFS'(\\d+)'

quot;)

    public void the_exists_in_HDFS(int id) throws Throwable {

        this.id = id;

        Assert.assertTrue(fileExistsInHDFS());

    }

SSH Requirements

• There needs to be an SSH server running at the target server
• The target server configuration in you test app must be correct
• Password-less SSH login needs to be configured. In most cases this simply means copying your local users’s id_rsa.pub key into the target user’s $HOME/.ssh/authorized_keys on the target server. If these files don’t already exist you may refer SSH login without password
• Given that you have an SSH command line client, you should be able to login in manually, without a password, using the same configuration values. For example
  

  $ ssh <TARGET_USER>@<TARGET_SERVER> -p <SERVER_SSH_PORT>
  

Sample Code

The files referenced are available here. Also included is a maven pom with all the necessary dependencies included as well as common maven profiles for handling multiple different environments.

For example to compile using conf files (application.properties, logback.xml etc) located under /conf/local use the following:

$ mvn compile -P local

It is up to you to configure each environment as needed but I recommend using the Apache Commons Configuration for which there is a convenience class Config.java, also included. This class allows prioritized configuration. JVM System variables take precedent over Environment variables. Environment variables take precedent over variables configured in application.properties. You may specify defaults when using most of the Config class methods, which means that if no setting is found the default will be used.

Ideas for Possible Improvements

Currently we use The JSch ChannelExec class which allows for single-shot remote commands. But it would be nice to be able to keep an SSH connection open, possibly verifying output and inputting new commands as our test progresses. The JSch ChannelShell apparently supports this type of connection. Something to look at in the future.
Alternatively a browser based ssh web client could be controlled by Selenium, so that with a heads up browser we could see all the SSH input and output in the browser. This would be cool.

Links:

• The referenced source can be found here
  
• Cucumber
• Selenium
• Automated testing with Selenium and Cucumber
• Selenium と Cucumber を使用した自動テスト

次世代システム研究室では、アプリケーション開発や設計を行うアーキテクトを募集しています。アプリケーション開発者の方、次世代システム研究室にご興味を持って頂ける方がいらっしゃいましたら、ぜひ 募集職種一覧からご応募をお願いします。