Dangers of executable examples over dry documentation

My team is responsible for several web service APIs that are used by external parties. The standard pattern for educating users about these interfaces was to provide an API document which listed the calls possibly with the occasional example. I have always found this approach to be less than useful and not particularly accessible.

I prefer an approach where the API specification is very much example based. This is makes more sense for RESTful web services than a programmatic API as there is no accepted mechanism for declaring such an interface. The same is not true if you are declaring, for example, a Java interface.

We supplement the document with JMeter projects pointed at a reference system which allow these interfaces to be exercised by the consumer and provides a step by step demonstration.

This has been an excellent mechanism which has resulted in very quick uptake of our interfaces but yesterday I was provided with an example of a potential pitfall:

We provided an API which upgraded widgets. Version was supplied with the manifest of the software package being supplied and was used again in the query string to the upgrade service. The reference data supplied by the system put the current version into a title metadata field so that testers could easily see that the upgrade had been formed. The version in the title matched exactly the title in the manifest.

Weeks after delivery and demonstration it stopped working. Why? The client was using the title rather than the metadata it was supposed to. Since we had used the same value for both the end to end use case appeared to work. As soon as they started using real data where the title was no longer equal to the version everything broke.

A valuable lesson here for me to relearn (I have hit it before in similar contexts). Always make sure reference and test data attributes are not interchangable with each other without an error occurring (and being detected). I should have made the title attribute ‘The version of this widget is <x>’. That would have steered the client developer away from error and would have caused a validation error then first time he tried to submit it to our application.

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>