Swift Testing: an ergonomic combination helper

July 29, 2025

This helper is installable with SPM, see GitHub repo

I’ve been using the Swift Testing framework more. Despite my previous grumble I do like the arguments: feature of the @Test macro and how much it can clean up tests:

@Test(arguments: [
	(true, "bob", 38),
	(false, "sue", 45)
])
func testUserPermissions(isBasketEmpty: Bool, username: String, age: Int) {
    #expect( ... )
}

The ability to supply a list of parameter variants to test is very useful. I think it tends to result in more exhaustive tests because of ease of use.

But being able to do easily whack in a list of argument variants soon reveals a slight annoyance. It’s not unusual to end up with a test like this:

@Test("feature flag combinations", arguments: [
    (false, false, 0),
    (true, false, 0),
    (false, true, 0),
    (true, true, 0),
    (false, false, 1),
    (true, false, 1),
    (true, true, 1),
    (false, false, 2),
    (true, false, 2),
    (false, true, 2),
    (true, true, 2)
])
func featureFlags(flagNewUI: Int,
                  flagOfflineAllowed: Int,
                  numItemsInBasket: Int) {
    #expect( // some test on the three parameters )
}

In this test we’re trying all possible combinations of the three parameters. Hence the truth table in the arg list!¹

It’s not ideal. Can you quickly see if there’s a mistake been made? Actually, one combo has missed out. Which one? And was that deliberate?

Could we use some nested for loops in our test to generate the combos? We could, and I think it might be the lesser of two evils, but it’s not very ergonomic and we’d lose the lovely feature of Testing where it reports the test result for every argument individually.

And so I had a little hackathon this weekend and created a helper for generating combinations for Testing. It looks like this:

@Test("feature flag combinations", arguments:
    FeatureFlagParams.variants(
        .wild(\.newUI),
        .wild(\.offlineAllowed), 
        .values(\.numItemsInBasket, 0...2)
    )
)
func featureFlags(ffParams: FeatureFlagParams) {
    #expect( // some test based on ffParams )
}

We’ve replaced the tedious arguments list with FeatureFlagParams.variants. It’s much clearer to read thanks to the keypaths and it’s more type-safe (we can avoid some of the vague Any errors you can get in regular Testing use).

It works by generating a list of FeatureFlagParam combinations using keypaths wrapped in either .wild or .values:

.wild means use all possible variants of a value (think wildcard) – it can only be used for some simpler types like Bool, certain enum and Optional, and a few others.

.values means use the explicitly given values for that keypath. In this example we use a range 0...2, but that expression can be any Sequence.

In order to use the above code we must define a simple mutable helper struct:

// mutable struct with a no-param init
struct FeatureFlagParams: WildcardPrototyping {
    var retryEnabled = false
    var retryCount = false
    var numItemsInBasket = 0
}

Note the use of a WildcardPrototyping protocol here. To conform to this you must have a no-param init; you get that for free if you initialise all your properties as you declare them (recommended).

This struct specifies a prototype value for your test. Any properties your .variants invocation doesn’t override have the default value defined in the prototype.

We enjoy the flexibility of `Sequence`

.values takes a Sequence which is great for expressiveness. Examples:

// explicit int values
.values(\.numItemsInBasket, [0, 3, 15])

 // a stride over ints
.values(\.numItemsInBasket, stride(from: 2, to: 20, by: 2))

// first 5 vals of some infinite sequence
.values(\.numItemsInBasket, someInfiniteSequence.prefix(5))

// a single value like this is not actually a sequence but it gets converted
.values(\.newUI, false)

Can I omit certain combinations?

You can use the usual .filter to remove any specific combos you don’t want. Example:

FeatureFlagParams.variants(
    .wild(\.newUI),
    .wild(\.offlineAllowed), 
    .values(\.numItemsInBasket, 0...2)
)
// remove combo where neither feature flag is set
.filter { $0.newUI || $0.offlineAllowed }

(For clarify, it might be nice to define a .remove variant of .filter that just does the opposite thing.)

Can I get all the variants passed into my test method as a list?

Yes, by calling .variantsList instead of .variants, and receiving an array to your func. Example:

@Test("if retry is disabled then shouldRetry always returns false", arguments:
    RetryParam.variantsList(   // NOTE we're calling .variantsList here
        .values(\.retryEnabled, false),
        .values(\.retryCount, 0..2),
        .values(\.lastAttemptErrorCode, [401, 403]),
        .wild(\.connectionStatus)
    )
)
// NOTE we're taking in [RetryParams] below
func ifRetryDisabledThenShouldRetryAlwaysReturnFalse(retryParam: [RetryParam]) {
    // this func will only be called once with all the variants in a list
}

This isn’t recommended though; Testing gives better test feedback when you use .variants.

What can I use `.wild` on?

Bool
enum: must be CaseIterable compatible and be marked with the WildcardEnum protocol. This means some of your Error types can be used too
Optional: its wrapped type must be .wild compatible; its generated values for the test are nil plus all possible values of the wrapped type
MutableResult: a provided helper similar to Result. Your success and failure types must be .wild compatible. You must use the provided MutableResult in your prototype struct and then in your test func you access the real Result via the someMutableResult.result property

Use outside of `Testing`

Although this experimental helper is made with Testing in mind, you can use it in any context, for example:

struct RetryParam: WildcardPrototyping {
    var retryEnabled = true
    var retryCount = 0
    var lastAttemptErrorCode = 0
    var connectionStatus = ConnectionStatus.offline
}

let variants = RetryParam.variants(
    .values(\.retryEnabled, false),
    .values(\.retryCount, 0..2),
    .values(\.lastAttemptErrorCode, [401, 403]),
    .wild(\.connectionStatus)
)

print(variants)

Ideas

Could add a peer macro to make it even easier to call, e.g. @TestWildcards which just rewrites as @Test
Might it be fruitful to add a kind of .wild support for non-finite types like Int, where we give a range of values, and a finite amount of values are pseudo-randomly selected and used? In the interests of deterministic (repeatable) testing the pseudo-random choice should be the same each time (by setting a seed)

Design choices

This helper uses mutability of a prototype to configure your test parameter. If instead we constructed the test parameter with the correct values from the get-go that might be nicer. This means you could use real-world types in your testing; for example, there would be no need for the MutableResult helper: we would be directly creating the Result and could pass that into the SUT.

I’ve not thought too much at this point about recursiveness e.g. allowing enum associated values with compatible types. I don’t think it’s worth it for the extra complexity in would add to the code.

Known issues

If you repeat one of the variant spec lines, for example:

let variants = RetryParam.variants(
    .values(\.retryEnabled, false),
    .values(\.retryEnabled, false), // oh no, same thing twice!

then behaviour is undocumented. Usually, your generated variants will contain duplicates or the tests can crash, which isn’t ideal.

And yes, this is one of the things I was grumbling about in that previous post! BUT – and take note, Apple – I’m documenting a limitation I know about. \sermon

Try this out via SPM

https://github.com/alexhunsley/swift-testing-wildcards

there’s an argument to be made (pun intended) that in a well designed codebase you’d minimise the amount state that has to be tested in just one place and so would avoid any sprawling truth-table like stuff. I agree with this sentiment. But in the real world sometimes you just want to write a good, clear test for what’s there ↩︎

We enjoy the flexibility of Sequence