Back

Conditional Swift testing

The problem

Some tests need to be skipped when executing on CI (continuous integration like travis.org).

What tests? In my case, it's performance tests.

Why? because I can't trust all CI results and sometimes, randomly, some tests will fail without good reason, like this one:

error: testPerformance : failed: The relative standard deviation of the measurements is 11.061% which is higher than the max allowed of 10.000%.

This is due to actual, temporary machine load that affects results of performance measurements. Happens.

Solution

Active Compilation Conditions
a.k.a. SWIFT_ACTIVE_COMPILATION_CONDITIONS
a.k.a. conditional compilation flag.

"Active Compilation Conditions" is a new build setting for passing conditional compilation flags to the Swift compiler.

note for Objective-C programmers: it's -DMYFLAG from OTHER_FLAGS

usage

#if MYFLAG
// do stuff
#endif

I named it CI and added to my Travis build script. Since I'm using Swift Package Manager it's called like this

$ swift test -Xswiftc -DCI

Now, wherever I check for CI I may distinguish if a test is run on continuous integration server.

#if !CI
extension FooTests {
    func testPerformance() {
         // ...
    }
}
#endif

finally, enough adjust list of tests for Swift Package Manager to not try run tests not intended for CI

// ...
#if !CI
    tests += [
        ("testPerformance", testPerformance),
    ]
#endif
// ...

tadaaam.

Sidenote about #if

Swift #if ... #endif directive behaves differently than the one known from C, and it has implications. See comments to this tweet

thoughts?

Back

Back to top