Swift Testing: an ergonomic combination helper

July 29, 2025

This helper is installable with SPM, see GitHub repo

I’ve been using the Swift Testing framework more. Despite my previous grumble I do like the arguments: feature of the @Test macro and how much it can clean up tests:

@Test(arguments: [
	(true, "bob", 38),
	(false, "sue", 45)
])
func testUserPermissions(isBasketEmpty: Bool, username: String, age: Int) {
    #expect( ... )
}

The ability to supply a list of parameter variants to test is very useful. I think it tends to result in more exhaustive tests because of ease of use.

But being able to easily whack in a list of argument variants soon reveals a slight annoyance. It’s not unusual to end up with a test like this:

@Test("feature flag combinations", arguments: [
    (false, false, 0),
    (true, false, 0),
    (false, true, 0),
    (true, true, 0),
    (false, false, 1),
    (true, false, 1),
    (true, true, 1),
    (false, false, 2),
    (true, false, 2),
    (false, true, 2),
    (true, true, 2)
])
func featureFlags(flagNewUI: Int,
                  flagOfflineAllowed: Int,
                  numItemsInBasket: Int) {
    #expect( // some test on the three parameters )
}

In this test we’re trying all possible combinations of the three parameters. Hence the truth table in the arg list!¹

It’s not ideal. Can you quickly see if there’s a mistake been made? Actually, one combo has missed out. Which one? And was that deliberate?

Could we use some nested for loops in our test to generate the combos? We could, and I think it might be the lesser of two evils, but it’s not very ergonomic and we’d lose the lovely feature of Testing where it reports the test result for every argument individually.

And so I had a little hackathon this weekend and created a helper for generating combinations for Testing. It looks like this:

@Test("feature flag combinations", arguments:
    FeatureFlagParams.variants(
        .wild(\.newUI),
        .wild(\.offlineAllowed), 
        .values(\.numItemsInBasket, 0...2)
    )
)
func featureFlags(ffParams: FeatureFlagParams) {
    #expect( // some test based on ffParams )
}

We’ve replaced the tedious arguments list with FeatureFlagParams.variants. It’s much clearer to read thanks to the keypaths and it’s more type-safe (we can avoid some of the vague Any errors you can get in regular Testing use).

It works by generating a list of FeatureFlagParam combinations using keypaths wrapped in either .wild or .values:

.wild means use all possible variants of a value (think wildcard) – it can only be used for some simpler types like Bool, certain enum and Optional, and a few others.

.values means use the explicitly given values for that keypath. In this example we use a range 0...2, but that expression can be any Sequence.

In order to use the above code we must define a simple mutable helper struct:

// mutable struct with a no-param init
struct FeatureFlagParams: WildcardPrototyping {
    var retryEnabled = false
    var retryCount = false
    var numItemsInBasket = 0
}

This struct specifies a prototype value for your test. Any properties your .variants invocation doesn’t override have the default value defined in the prototype.

To conform to WildcardPrototyping you must be Equatable and have a no-param init (hint: initialise all your properties as you declare them and you’re covered).

The flexibility of `Sequence`

.values takes a Sequence which is great for expressiveness. Examples:

// this bool will only be false
.values(\.newUI, false)

// explicit int values
.values(\.numItemsInBasket, [0, 3, 15])

 // a stride over ints
.values(\.numItemsInBasket, stride(from: 2, to: 20, by: 2))

// first 5 vals of some infinite sequence
.values(\.numItemsInBasket, someInfiniteSequence.prefix(5))

// a single value like this is not actually a sequence but it gets converted
.values(\.newUI, false)

What can I use `.wild` on?

Bool
enum: must be CaseIterable compatible; add the WildcardEnum marker protocol
Error: some of your Error enums will be compatible (see enum)
Optional: its wrapped type must be .wild compatible; its generated values for the test are the nil value plus all possible values of the wrapped type
Result: your success and failure types must be .wild compatible. Support is provided for Result<Never, Error>, and Result<SomeType, Never>.
OptionSet: add the InvariantOptionSet marker protocol; must be Equatable

Can I call `.variants` on an arbitrary instance of my helper type?

Yes, just call .variants on your instance:

@Test("feature flag combinations", arguments:
    FeatureFlagParams(newUI: false, offlineAllow: true)
    .variants(
        .values(\.numItemsInBasket, 0...2)
    )
)
func featureFlags(ffParams: FeatureFlagParams) {

Note that you can safely just omit any values that are mentioned in .variants( in your init call.

Can I omit certain combinations?

You can use a .filter to remove any specific combos you don’t want. Example:

FeatureFlagParams.variants(
    .wild(\.newUI),
    .wild(\.offlineAllowed), 
    .values(\.numItemsInBasket, 0...2)
)
// remove combo where neither feature flag is set
.filter { $0.newUI || $0.offlineAllowed }

(Syntactic sugar idea: we could define a .remove variant of .filter that just does the opposite thing.)

Can I get all the variants into a test func as a single list?

Yes, by calling .variantsList instead of .variants:

@Test("if retry is disabled then shouldRetry always returns false", arguments:
    RetryParam.variantsList(   // NOTE we're calling .variantsList here
        .values(\.retryEnabled, false),
        .values(\.retryCount, 0..2),
        .values(\.lastAttemptErrorCode, [401, 403]),
        .wild(\.connectionStatus)
    )
)
// NOTE we're taking in [RetryParams] below
func ifRetryDisabledThenShouldRetryAlwaysReturnFalse(retryParam: [RetryParam]) {
    // this func will only be called once with all the variants in a list
}

This isn’t recommended though; Testing gives better test feedback when you use .variants.

Use outside of `Testing`

Although this experimental helper is made with Testing in mind, you can use it in any context, for example:

struct RetryParam: WildcardPrototyping {
    var retryEnabled = true
    var retryCount = 0
    var lastAttemptErrorCode = 0
    var connectionStatus = ConnectionStatus.offline
}

let variants = RetryParam.variants(
    .values(\.retryEnabled, false),
    .values(\.retryCount, 0..2),
    .values(\.lastAttemptErrorCode, [401, 403]),
    .wild(\.connectionStatus)
)

print(variants)

Ideas

Could add a peer macro to make it even easier to call, e.g. @TestWildcards which would rewrite to a @Test macro.

Known issues

If you repeat one of the variant spec lines, for example:

let variants = RetryParam.variants(
    .values(\.retryEnabled, false),
    .values(\.retryEnabled, false), // oh no, same thing twice!

then behaviour is undocumented. Usually, your generated variants will contain duplicates or the tests can crash, which isn’t ideal.

And yes, this is one of the things I was grumbling about in that previous post! BUT – and take note, Apple – I’m documenting a limitation I know about. \sermon

Try this out via SPM

https://github.com/alexhunsley/swift-testing-wildcards

there’s an argument to be made (pun intended) that in a well designed codebase you’d minimise the amount of state that has to be tested in just one place and so would avoid any sprawling truth-table like stuff. I agree with this sentiment. But in the real world sometimes you just want to write a clear test for what’s there ↩︎

The flexibility of Sequence

What can I use .wild on?

Can I call .variants on an arbitrary instance of my helper type?