the four levels of software testing

Wed, Feb 12, 2014

I was recently having a blether with a colleague about testing and how we should thrash out a shared understanding of the different types. There are so many ways to test software, all with their own advocates, terminology and fluff that we just had to write down what we meant when we said ‘the software is tested’. The whiteboard is your friend in cases like this.


So I thought I’d write all this down, picking out the different test types from our whiteboard session, refine them and come up with a nice diagram. Everyone likes a nice diagram!

Most people more or less agree what unit testing means. It’s the basic level of ‘competence’ in your system. If, for example, you have a User class that has to be serialised onto the wire for a front end client to work with, then you need a way to do that and you also need to be confident the class can actually do it. Hence the unit test:

public class EasyTest() {
  public void test() {
    String expected = "{\"name\": \"harrymcd\"}";
    User user = new User("harrymcd");
    Assert.assertEquals(expected, user.toJSON());

Your User class needs to respond with JSON when poked with toJSON(). Can it do that? Is it a competent producer of JSON? The test above is one basic unit test for the User class. There are many more you can add for that class. What if you create a new User with no name? A new User with a name with quotes in it? Does it respond with JSON or a String that looks like JSON? Is it correct JSON?

This is where testing enters its grey area. If you trundle a JSON parser onto the stage to test that the User class is producing correct JSON, is that a unit test or an integration test? I’d argue it’s still a unit test. You’re still testing one particular functionality of a single class. You’re just doing it with more specialised tools. You’ve gone beyond isEquals() and moved into the domain. But it’s still a unit test.

Now let’s plonk a User into a Spring application. We need to get that JSON off the server and into the client after all. So we now have a collection of entities, all working together, making use of each others’ services, to produce JSON. We have a Controller, a Model (User) and a View. We’re now in integration testing mode.

public class SlightlyHarderClass extends BaseClassWithSetupStuff {
  public static void init() {
  public void test() {
      try {
      MockBackendSystem mockSystem = new MockBackendSystem();
      UserService userService = (UserService)ctx.getBean("userService");
      Assert.assertEquals("yuck", userService.getCurrentStatus());
      UserController userController = (UserController)ctx.getBean("userController");
      ModelAndView mav =  handlerAdapter.handle(request, response, userController);
      assertEquals("jsp/user", mav.getViewName());
        catch(Exception aargh) {

In integration testing, you’re making sure that everything can live in harmony with everything else in the system. You’re testing ‘life in the framework’. At this point you don’t care ‘what’ is in the views, you just care that the ‘correct’ views are returned. You’re still in ‘internal’ mode. You’re testing the ‘structural integrity’ of the system. You use the Spring testing framework to load Spring, access the beans in, for example, applicationContext.xml, replace them with ones that have predetermined behaviour (mocks) and then invoke the Controllers with those mocks. You’re looking for expected exceptions being thrown, correct view names being returned. If you need to bring in a real system at this point, for example, a database, you’re making sure things were added/updated/deleted from the database.

So that’s the first two levels of testing. Let’s have a recap of ‘build driven tests’:


By ‘build driven tests’, I mean tests that are run automatically when you build the system. e.g. mvn clean install. Unit and integration tests, as we’ve seen, test the basic competency and ‘technical social skills’ of the individual components. By this point you should be fairly confident your system isn’t composed of a noisy rabble of antisocial misfits.

It’s time to bring in the client. We now need functional tests.

Functional testing moves up a level to testing the actual content of the views. We know, from integration testing, that the Controllers return the correct views. Now we need to test the views contain the correct content and the flow between the views is correct. We need to hit the system with a Cucumber:

Feature: Deny access to cake eater
  As an unauthenticated cake eater I must login to get cakes

  Scenario: Create partner
    Given I am on the home page of the website
    When I click on the Give Me Cakes button
    Then I should see the login page

What you see above is a behavioural feature. Where did it come from? It came from a user story. We sat down with the client, worked through different use cases for the system and created a set of user stories. This is one:

As a user
In order to eat cakes
I must first login

The client is telling us that only authenticated users can eat cakes. We’ve written this down on a small piece of card. Perhaps along with an estimate of how long it might take. Or perhaps not, depending on how you view these things. On the back we’ve written an ‘acceptance test’. The feature above. The acceptance test tells us how to verify the user story has been completed.

So we now have a solid requirement from the client, in the form of a user story along with the steps to verify the requested behaviour. The client writes the front of the card, the developer reads the back and produces tests:

public class FunctionalTests {
    // Cucumber step definition implementation
    @Given("^I am on the home page of the website$")
  public void onHomePage() {
    // Do stuff with Selenium

We’re now testing the system as it will be used by the client. If the above test passes, then the feature is complete, the user story has been completed and verified and the acceptance test passed. Done!

We’ve come a long way from our napkin based scribbles down the local coffee shop. We’ve assembled a crude collection of unruly classes, ‘interviewed’ them over and over using unit testing to ascertain their competency and finally turned them into ‘first class citizens’. We’ve then introduced them to other, similarly competent entities in a communal meeting area. Perhaps a Spring framework. We’ve made sure they all behave and have ironed out any differences of opinion. We’ve then let them loose in the real world and watched them under the microscope of Behaviour Driven Development (BDD).

Our little organic garden has flowered into a tropical oasis of deep calm, efficiently handling requests and responses. Working away in its own little world. Our client is as pleased as punch.

Until that is, more and more people start using our little system. Suddenly our client’s eyebrows furrow. They’re having to wait several minutes for their login request to complete and the user interface is like treacle. What on earth is going on? Our little system isn’t scaling very well. It’s time to tackle non functional testing. Gulp!

We know the system ‘works’. By that I mean, it does what it’s supposed to do. It just does it ‘slowly’ and the client wants it to do it ‘quicker’. Subjective terms. We can prove it ‘works’ from the testing output. The fact the client has signed off on it, via the acceptance tests, proves it ‘works’. It’s just that it ‘works’ a bit too ‘slowly’ now.

If we examine our original napkins for hastily scribbled requirements we won’t see anything like ‘as a user I need to login pretty darn quickly’. So we need to make an educated assumption about what ‘quickly’ means. Login within one second? Display account information within 300ms? That’s what non functional testing is about.

Unit tests prove a Controller can return a View. Integration tests prove a Controller returns the correct structural View. Functional tests prove a Controller returns the correct View content. Non functional tests prove the Controller returns the correct View content ‘in a reasonable amount of time’. What’s reasonable? That’s up to you, the client and the smoky room with the bare lightbulb.

So that’s about it. The four levels of testing, as defined by, well, me I suppose.

The elephant in the room is of course, how do you create your tests? Are you a purist who writes the tests first? Do you prefer a screenfull of red to start your coding day? Personally, I don’t take this approach. I find it depressing, wasteful and inefficient. Modern IDEs have powerful autocomplete functionality. Why write code over and over for non existent classes, all underlined in red and being constantly prompted by the IDE ‘do you want to create this class’? It’s a waste of time.

I normally write software like I write a book. Get the raw functionality down early. See it working. Get a feel for how difficult the project will be. These are called ‘agile spikes’. All the time however, I have testing in mind, asking myself ‘how can I actually test this?’. I use the putlogging technique when I’m creating software. I’ll start with raw functionality. Then I’ll identify the ‘putlog holes’ where I can expose test hatches. Over a day or two I’ll have refactored each ‘chapter’ (class) into a modular, testable, happy little thing. In the end, I have a fully level 3 tested application. I’ll then address level 4 testing when options for the hosting solutions settle down.

So that basically covers testing from cupboard to client. But what about the diagram I promised way back at the start of this journey into the life of a simple little system? Here it is. njoy!


comments powered by Disqus