Spocklight: Optimize Run Order Test Methods

Spock is able to change the execution order of test methods in a specification. We can tell Spock to re-run failing methods before successful methods. And if we have multiple failing or successful tests, than first run the fastest methods, followed by the slower methods. This way when we re-run the specification we immediately see the failing methods and could stop the execution and fix the errors. We must set the property optimizeRunOrder in the runner configuration of the Spock configuration file. A Spock configuration file with the name SpockConfig.groovy can be placed in the classpath of our test execution or in our USER_HOME/.spock directory. We can also use the Java system property spock.configuration and assign the filename of our Spock configuration file.

In the following example we have a specification with different methods that can be successful or fail and have different durations when executed:

package com.mrhaki.spock

import spock.lang.Specification
import spock.lang.Subject

class SampleSpec extends Specification {

    @Subject
    private Sample sample = new Sample()

    def "spec1 - slowly should return name property value"() {
        given:
        sample.name = testValue

        expect:
        sample.slowly() == testValue

        where:
        testValue = 'Spock rules'
    }

    def "spec2 - check name property"() {
        given:
        sample.name = testValue

        expect:
        sample.name == testValue

        where:
        testValue = 'Spock is gr8'
    }

    def "spec3 - purposely fail test at random"() {
        given:
        sample.name = testValues[randomIndex]

        expect:
        sample.name == testValues[0]

        where:
        testValues = ['Spock rules', 'Spock is gr8']
        randomIndex = new Random().nextInt(testValues.size())
    }

    def "spec4 - purposely fail test slowly"() {
        given:
        sample.name = 'Spock is gr8'

        expect:
        sample.slowly() == 'Spock rules'
    }

    def "spec5 - purposely fail test"() {
        given:
        sample.name = 'Spock rules'

        expect:
        sample.name == 'Spock is gr8'
    }
}

class Sample {
    String name

    String slowly() {
        Thread.sleep(2000)
        name
    }
}

Let's run our test where there is no optimised run order. We see the methods are executed as defined in the specification:

Next we create a Spock configuration file with the following contents:

runner {
    println "Optimize run order"
    optimizeRunOrder true
}

If we re-run our specification and have this file in the classpath we already see the order of the methods has changed. The failing tests are at the top and the successful tests are at the bottom. The slowest test method is last:

Another re-run has optimised the order by running the slowest failing test after the other failing tests.

Spock keeps track of the failing and successful methods and their execution time in a file with the specification name in the USER_HOME/.spock/RunHistory directory. To reset the information we must delete the file from this directory.

Written with Spock 1.0-groovy-2.4.