For a number of years, Cartographica has had a lot of tests-on the order of 1500+, but a few of them are quite a bit bigger than they should have been, owing mostly to their data-driven nature.

This post describes the method used to provide dynamic test creation for Cartographica.

Cartographica File Formats

Cartographica uses a couple of big libraries to provide a wide variety of file import and export capabilities, a situation which is not uncommon in the GIS world, as most of the players, including ESRI use (and make available) libraries for accessing file formats that they either create or need access to. The result is that many applications have access to a large number of file formats.

In order to deal with this large variety of formats in an intentional way, I created an internal catalog that is used by Cartographica to handle most operations with file formats, including:

• User-readable format names
• System-usable identifiers
• Format limitations
• Available test files and verification data
• Identification of implementation source
• Identification of licensing

The files that contain this information are shipped as an internal part of the shipping code, and are part of the source bundle used to create the software.

As such, the test information in the file format data file provides essential information for programmatically testing Cartographica's import and export capabilities (including round-trip testing where applicable).

Implementing Dynamic Testing

Historically, Cartographica's file format testing appeared in the test system as a small set of tests which ran for a long time and provided little debugging information when a specific file format test failed. In addition, due to the data-driven nature, it was difficult to test just a single file format.

Over time, I considered a number of different ways to create dynamic tests, most of which required code generation, which was creating too much complexity for me to approve of. I needed a solution that would work within the confines of our Continuous Integration build system (Jenkins) and would create obvious signals if the tests either failed or were not executed for some reason.

This weekend, I took another stab at handling the problem and was successful. The mechanism is straight-forward, almost elegant, and very Objective-C.

After choosing to spend some time in this area, I read up on the official mechanism for creating dynamic tests, using defaultTestSuite, which looked like it would be a good option, except that it has a side-effect of not playing well with the IDE for re-running tests, and I really wanted to be able to do that. Because the suite creation code is only called when running the whole suite, it doesn't make available the individual tests if they are re-executed.

The examples I'd found also resulted in a large number of identically-named methods being called, which was pretty useless in terms of providing visibility to the tests. The solution to this problem was to create separate implementations using the dynamic runtime properties of Objective-C. By doing so, the second problem (identical naming) was solved and I could create uniquely-named, highly-identifiable test names.

As an added bonus, this also paved the way to fixing my first complaint through the removal of code. By creating all of my tests with names starting with test, the system will auto-discover the tests (as long as they are created in the +initialize method). Therefore, I was able to remove the code which otherwise created the default TestSuite, since the Objective-C introspection for methods beginning with test would find all of my new dynamic tests.

I had help from a couple of sites in putting this approach together, and the result has been great. My 2 file format tests are now approximately 200, and they are individually named for what they do and to what data types.

In-depth Code Review

The particular code that I'm using works with hundreds of different file formats, and most of them have sample data files and information that allows me to verify that they're all loading correctly. To index all of this information, I use a strongly-typed XML Schema for these descriptions. The strong typing facilitates the strong validation of the files. Since all of this information (including driver names) is in this XML file, the best way to truly test this data is by running automated tests that interpret the files.

The actual code implementation is relatively simple:

• +initialize method to parse the file and set up the tests
• +CTUTaddInstanceMethodWithSelectorName used to register a block for the method

Our test registration code calls the add instance method with the block that executes the actual tests. The parameter passed to the block (ImportExportFileTests) is just a subclass of XCTestCase, as that's what XCTestCase expects to call test* methods with.

In this particular example code, we're passing in a NSString containing an XML fragment which contains information on the individual test files. The -runFileTestWithFileNode method referred to inside of the block is a basic test routine which I was able to reuse without modification. In this case, I chose to place run all the tests for a file format in the same test.

Looking at the code below, checkTestFileList verifies that at least one test file exists for this file format and driver (and the level of test that we're doing... some tests take a long time and are only executed when isExhaustive is set).

fileFormatName is the human-readable name of the format, fileFormatType is the UTI for the file type, some of which are vendor-specific, others are defined in Cartographica. For the UTIs we define, we create associations for com.ClueTrust.Cartographica.external.<type_name>, where <type_name> is a unique identifier for each type. Since that's a large, mostly useless, prefix and we're guaranteed that it's unique (since we check that elsewhere when validating the file format file), we shorten it to external. Finally, . is replaced with _ to comply with method name requirements.

The block passed to +CTUTaddInstanceMethodWithSelectorName:block: executes the tests themselves, using the standard XCTest assertions to flag problems.

NSString *testMethodBaseName = @"testVectorImport_";

NSArray *checkTestFileList = [fileFormat nodesForXPath: testFileSearchString error:nil];
if (checkTestFileList.count<1)
	 continue;

NSString *fileFormatName = [[fileFormat attributeForName: @"name"] stringValue];
NSString *fileFormatType =[[[[fileFormat attributeForName:@"typeID"] stringValue]
		stringByReplacingOccurrencesOfString:@"com.ClueTrust.Cartographica.external" withString: @"external"]
		stringByReplacingOccurrencesOfString: @"." withString:@"_"];
NSString *testName = [testMethodBaseName stringByAppendingString: fileFormatType];
NSXMLDocument *doc = [NSXMLDocument documentWithRootElement: [fileFormat copy]];
NSString *xmlString = [doc XMLString];

[self CTUTaddInstanceMethodWithSelectorName: testName block:^(ImportExportFileTests *test) {
	 NSXMLDocument *xml = [[NSXMLDocument alloc] initWithXMLString: xmlString options: 0 error:nil];
	 NSAssert( xml, @"Need XML");

	 NSArray *testFileList = [xml.rootElement nodesForXPath: testFileSearchString error:nil];
	 NSAssert( testFileList, @"no test files for %@", fileFormatName);
	 NSAssert( testFileList.count>0, @"Need >1 test");

	 test.readRasterPixels = isExhaustive;
	 for (NSXMLElement *testFile in testFileList) {
		 [test runFileTestWithFileNode: testFile formatName: fileFormatName asRaster: isRaster];
	 }
}];

The actual code we use to add the selector is thanks mostly to a Stack Overflow Posting which describes exactly how to do this.

+ (BOOL)CTUTaddInstanceMethodWithSelectorName:(NSString *)selectorName block:(void(^)(id))block
{
	// don't accept nil name
	NSParameterAssert(selectorName);

	// don't accept NULL block
	NSParameterAssert(block);

	// See http://stackoverflow.com/questions/6357663/casting-a-block-to-a-void-for-dynamic-class-method-resolution

	id impBlockForIMP = (__bridge id)(__bridge void *)(block);

	IMP myIMP = imp_implementationWithBlock(impBlockForIMP);

	SEL selector = NSSelectorFromString(selectorName);
	return class_addMethod(self, selector, myIMP, "v@:");
}