I've been doing quite a bit of experimentation with recent features in Xcode lately, especially as regards trying to efficiently run my GitLab-powered Mac Mini build farm.

Recently, as I've been doing some work on CartoMobile, I've been updating the testing code there and stole some ideas from the Cartographica test suites, which I intentionally build for both Apple Silicon and x86_64 and then run on 2 OS variants with each processor family. In this case, I collect coverage information from all 4 and then merge them because I have variant code that runs on different CPUs and versions of the OS (more the former than the later, because some libraries are specific to one architecture or the other).

In Cartographica, the CI code follows these steps:

1. Build the code for testing

2. Run a matrix job across the CPUs and Operating Systems that I need to test, collecting junit and coverage information

   • Current macOS and x86_64
   • Current macOS and arm64
   • Previous macOS and x86_64
   • Previous macOS and arm64

For CartoMobile, I did something similar, but ran into a problem with slather in doing so and also realized that I was likely wasting time and effort.

First, the problem that I ran into was specifically with running slather without pointing at the correct directory. In this case, I wasn't pointing slather at the migrated directory when checking coverage, thus failing to find the coverage files when running.

However, more importantly, this led me to the realization that the way that I was going about this was wrong for CartoMobile. Unlike Cartographica, where I was doing matrixed coverage because I have code that only operates on specific CPU architectures or versions of macOS, CartoMobile has a single set of code that runs on all SDKs, and since I can't run coverage tests on the iphoneos SDK, that meant that all that was interesting was running the coverage tests for the simulator on both iPadOS and iOS.

In addition, for CartoMobile, I also run TSAN and ASAN tests (with UBSAN set). Although the builds for those can take a while, the runtimes are short, and the builds are completely independent. Further, the coverage for these are not important, since coverage tests can (and should) be run without the sanitizers. Thus, unlike Cartographica, for CartoMobile, I decided to matrix the build and test functions together. The result is:

1. Build for simulator and test with Coverage, collecting junit and coverage

2. Build and run simulator using an Xcode test plan (.xctestplan) that runs with:

   • TSAN + UBSAN
   • ASAN + UBSAN

3. Build and run my iOS snapshots on a minimum iPad and iPhone simulator (UI Test)

4. Build and run my iOS snapshots on all sizes and languages (AppStore snapshots)