Skip to content

 
  • Expertise in Automated Acceptance Tests and ATDD
  • Automated Acceptance and Web Testing
  • Continuous Integration and Continuous Delivery with Jenkins
  • Test-Driven Development Training and Coaching
  • Test-Driven Development Training and Coaching
  • Automated Acceptance and Web Testing
  • Test-Driven Development Training and Coaching
  • Test-Driven Development Training and Coaching

Running parallel acceptance tests using JBehave, Thucydides and Bamboo

This article was writen by Simeon Ross, who works in a large government organization. In it, he describes how he sets up parallel testing in JBehave and Thucydides using Bamboo. Simeon has also put a sample project illustrating the approach on Github

The place that I work for has a data mart project pulling from many sources and it must be accurate. To that end we decided to write a number of JBehave style, data driven JBehave tests utilising Thucydides to ensure its integrity. This was a success until the data grew and the tests were taking too long to complete - some of our stories had massive example tables! While we could speed up the process by tuning the database with indexes we decided that it since it was still in development it could be time wasted if the data/approach was wrong and that wasn't getting away from the issue of the sheer number of tests so we came to the conclusion that the first attack would be running the tests in parallel.

Parallel tests with JBehave and Thucydides wasn't possible out of the box so I came up with a solution utilising Bamboo and it's ability to run jobs in parallel. Basically, I would have three stages in my parallel plan. The first would check out the source and do a Maven clean compile. The second stage has three jobs running in parallel doing the actual tests. The third stage then takes pulls together the results of the jobs from the previous stage and pushes them into the Thucydides report. It is a fairly simple but I had to do some work in the automated tests POM file and extend the ThucydidesJUnitStories class.

Maven profile

The first step was creating a profile for running of the tests in parallel within the POM. We used all the normal Thucydides dependencies plus lambdaj. The failsafe plugin that actually kicks off the integration tests was setup to run only on the integration-test goal, not verify as we'd normally do. It also receives two systemPropertyVariables, being parallel.agent.number and parallel.agent.total. These are used by the ParallelAcceptanceTestSuite class (which extends the extended ThucydidesJUnitStories class) to determine which stories it should run. parallel.agent.total indicates how many parallel agents there are and parallel.agent.number indicates which of the agents the class is being called by. These are used to determine which stories to pick up.
The maven thucydides plugin likewise is setup to only run on the aggregate goal. The idea being that we can run both stages separately.


<project>
  ...
  <dependencies>
    <dependency>
      <groupId>net.thucydides</groupId>
      <artifactId>thucydides-core</artifactId>
      <version>0.9.121</version>
    </dependency>
    ...
    <dependency>
      <groupId>com.googlecode.lambdaj</groupId>
      <artifactId>lambdaj</artifactId>
      <version>2.3.3</version>
    </dependency>
    
  ...
  <profiles>      
    ....
    <profile>
      <id>parallel</id>
      <properties>
        <project.database.url>jdbc:oracle:thin:@Nosddb01:1521:DEV1</project.database.url>
        <project.parallel.agent.total>2</project.parallel.agent.total> <!-- This should be set by Bamboo -->
        <project.parallel.agent.number>1</project.parallel.agent.number> <!-- This should be set by Bamboo -->
      </properties>
      <build>
        <plugins>
          <plugin>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.11</version>
            <configuration>
              <includes>
                <include>**/ParallelAcceptanceTestSuite.java</include>
              </includes>
              <systemPropertyVariables>
                <database.url>${project.database.url}</database.url>
                <parallel.agent.number>${project.parallel.agent.number}</parallel.agent.number>
                <parallel.agent.total>${project.parallel.agent.total}</parallel.agent.total>
              </systemPropertyVariables>
            </configuration>
            <executions>
              <execution>
                <goals>
                  <goal>integration-test</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
          <plugin>
            <groupId>net.thucydides.maven.plugins</groupId>
            <artifactId>maven-thucydides-plugin</artifactId>
            <version>0.9.121</version>
            <executions>
              <execution>
                <id>thucydides-reports</id>
                <goals>
                  <goal>aggregate</goal>
                </goals>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>      
  </profiles>

</project>

ParallelAcceptanceTestSuite class

The ParallelAcceptanceTestSuite class is setup to be run by the maven project to deal with the stories. We saw that it gets two parameters from POM profile. It uses these to determine which of the available story files are being run by this class instance. It does this by getting all the available stories and then taking a subset to run. If you can imagine that there are 12 story files and 3 agents then the first agent will take the first 4 story files, the second agent the next four and so on. If there is an odd number of story files and one left over then it is picked up by the last agent.


import ch.lambdaj.Lambda;
import net.thucydides.jbehave.ThucydidesJUnitStories;

import java.util.List;

public class ParallelAcceptanceTestSuite extends ThucydidesJUnitStories {

    public ParallelAcceptanceTestSuite() {
        Integer agentPosition 
            = Integer.parseInt(System.getProperty("parallel.agent.number"));
        Integer agentCount
            = Integer.parseInt(System.getProperty("parallel.agent.total"));
        List<String> storyPaths = storyPaths();

        failIfAgentIsNotConfiguredCorrectly(agentPosition, agentCount);
        failIfThereAreMoreAgentsThanStories(agentCount, storyPaths.size());

        // The reminder should work out to be either be zero or one.
        int reminder = storyPaths.size() % agentCount; 
        int storiesPerAgent = storyPaths.size() / agentCount;

        int startPos = storiesPerAgent * (agentPosition - 1);
        int endPos = startPos + storiesPerAgent;
        if (agentPosition == agentCount)
        {
            // In the case of an uneven number the last agent 
            // picks up the extra story file.
            endPos += reminder;
        }
        List<String> stories = storyPaths.subList(startPos, endPos);

        outputWhichStoriesAreBeingRun(stories);
        findStoriesCalled(Lambda.join(stories, ";"));
    }

    private void failIfAgentIsNotConfiguredCorrectly(Integer agentPosition, 
                                                     Integer agentCount)
    {
        if (agentPosition == null)
        {
            throw new RuntimeException("The agent number needs to be specified");
        } else if (agentCount == null)
        {
            throw new RuntimeException("The agent total needs to be specified");
        } else if (agentPosition < 1)
        {
            throw new RuntimeException("The agent number is invalid");
        } else if (agentCount < 1)
        {
            throw new RuntimeException("The agent total is invalid");
        } else if (agentPosition > agentCount )
        {
            throw new RuntimeException(String.format("There were %d agents in total specified and this agent is outside that range (it is specified as %d)", agentPosition, agentCount));
        }
    }

    private void failIfThereAreMoreAgentsThanStories(Integer agentCount, 
                                                     int storyCount)
    {
        if (storyCount > agentCount)
        {
            throw new RuntimeException(
            "There are more agents then there are stories, this agent isn't necessary");
        }
    }

    private void outputWhichStoriesAreBeingRun(List<String> stories)
    {
        System.out.println("Running stories: ");
        for (String story : stories)
        {
            System.out.println(" - " + story);
        }
    }
}

Bamboo Configuration

The final step is to set this all up in Bamboo. An important thing to keep in mind with the configuration that I'm about to describe is that the results from each of the parallel jobs is in a separate directory, this stops concurrent conflicts. It also makes our setup a little more difficult as we then need to then pass these results between stages. They say that a picture tells a thousand stories, so lets just look at some examples of this configuration.

We can see here that the first stage (Check out) has one job with two tasks. These do a source code check and then a Maven clean. All vanilla operations.

The first stage also exports its results for the other stages to use, which is named Project.

We can see here the configuration for the first parallel job (Test Stream 1). It calls the integration-test maven goal and its important to note the maven parameters. The -P is telling it to use the parallel profile and the other two parameters tell the ParallelAcceptanceTestSuite class to run the stories for agent 1 (of 3 agents). The only other thing to note is that we use the Maven return code.

We can see here the artifact tab for the first parallel job. It depends on the artifact Project produced by the previous stage. It also shares its target results as an artifact called Test 1 Target. This will be consumed by the next stage.

We can see here the configuration for the second parallel job (Test Stream 2). It is basically identical to the other except that it tells the ParallelAcceptanceTestSuite class to run the stories for agent 2 (of 3 agents). We won't look at it but you can guess that the configuration for the third parallel job (Test Stream 3) is identical other than indicating that it is agent 3.

We can see here the artifact tab for the second parallel job. It is identical in configuration to the previous one except in name. The third parallel job is the same except in name.

We can see here the configuration for the final stage and its only job. We can see that it called the Maven goal thucydides:aggregate and that it uses our Maven profile again. This calls the Thucydides code to compile the test reports.

We can see here the artifact tab for the final stage. We can see that it depends on all the previously produced artifacts and that it creates one of its own, the Thucydides tests report.

Put this all together and you have the tests running in parallel and hopefully saving you time!

Check out our upcoming BDD/TDD Master classes, coming soon to Sydney, Melbourne, Canberra, Brisbane and Perth