This document introduces a declarative RDF data model to represent test cases, allowing users to represent some input, execution instructions and then some expected output. While the framework is designed to be of general use, it has built-in test case types for SHACL validation [[shacl]], SPARQL queries [[sparql11-query]], SPARQL functions and Active Data Shapes scripts.

This document uses the prefix dash which represents the DASH Data Shapes namespace http://datashapes.org/dash# which is accessible via its URL http://datashapes.org/dash.

Scope

Note that this document covers both the general design of the test case framework and also illustrates specific tool support as part of TopBraid platform. The TopBraid binding should only be understood as one possible implementation and is in no way limiting the general applicability of the framework. Parts of the framework have also been published as part of the open-source TopBraid SHACL API. The document has been updated for TopBraid release 7.0.

General Concepts

The DASH Test Cases vocabulary provides a declarative data model for representing and sharing information about test cases. Test Cases represented using this vocabulary can be executed by tools to verify that a given test scenario still applies as originally designed. The vocabulary is intentionally minimalistic and kept simple.

Test Case Graphs and Files

Test cases are represented in RDF graphs. When published, these RDF graphs SHOULD be stored in a commonly used RDF serialization format such as [[turtle]]. Files containing test cases should contain the partial name .test. so that tools can automatically recognize them when walking a folder structure. A typical file name for a test case is 

	ExampleTests.test.ttl

In order to produce a new test case file with TopBraid Composer, use File > New > RDF Test Cases File. In order to execute a single test case or all test cases, use the following window, accessible from Window > Show View > Test Cases:

In TopBraid EDG, activate test cases on the Manage tab as follows:

You can then execute test cases using the Problems and Suggestions panel:

The Base Class dash:TestCase

The DASH namespace (prefix: dash, namespace: http://datashapes.org/dash#) includes a class dash:TestCase that serves as the (abstract) base class of all types of test cases supported by the framework. Extensions may subclass this base class. The DASH namespace includes a collection of built-in subclasses of dash:TestCase for frequently needed kinds of tests, as described in the following sections.

The Query Graph

Test cases typically need to be evaluated against a default query graph. In the current design of the framework, queries simply run against the graph that also contains the definition of the test case itself, plus its (transitive) owl:imports. While this design may in principle lead to situations where the test case itself may interfere with the evaluation results, this case is considered too rare to be of practical relevance. So in the current design, we went for the simplest possible approach. If this becomes an issue, then future versions may introduce additional properties to link to specific query graphs.

Deactivating Test Cases

Setting the property sh:deactivated to true on a dash:TestCase excludes the test case when the tests are evaluated.

SPARQL Functions Test Cases

The class dash:FunctionTestCase can be used to represent test cases to repeatedly evaluate a given SPARQL function (or more general: SPARQL expression). Instances of this type of test cases need to specify a SPARQL expression and an RDF node representing the expected result. For the test case to succeed, the provided SPARQL expression is evaluated against the query graph and needs to produce the expected result.

The following example illustrates a test case that verifies that a given SPARQL function strlen returns 5 for the parameter "Hello". 

ex:strlenTest
    rdf:type dash:FunctionTestCase ;
    dash:expectedResult 5 ;
    dash:expression "strlen(\"Hello\")" .

TopBraid Composer includes a convenience feature to quickly produce such test cases from the SPARQL view. Select the expression that you want to turn into a test case, right-click on it and select Turn into dash:FunctionTestCase...:

SPARQL Query Test Cases

The class dash:QueryTestCase can be used to represent test cases to repeatedly evaluate a given SPARQL SELECT query. Instances of this type of test cases need to specify a SPARQL query and a string-serialization of a SPARQL Result in JSON [[sparql11-results-json]]. For the test case to succeed, the provided SPARQL query is evaluated against the query graph and needs to produce exactly the same result set. Note that in order to produce reliable results, the query MUST include an ORDER BY clause.

The following example illustrates a test case that verifies that a given SPARQL query still produces the given result set. 

ex:queryTest
  rdf:type dash:QueryTestCase ;
  dash:expectedResult """{
  \"head\": {
    \"vars\": [ \"label\" ]
  } ,
  \"results\": {
    \"bindings\": [
      {
        \"label\": { \"type\": \"literal\" , \"value\": \"Nothing\" }
      } ,
      {
        \"label\": { \"type\": \"literal\" , \"value\": \"Thing\" }
      }
    ]
  }
}
"""^^rdf:JSON ;
  sh:select """
      SELECT ?label
      WHERE {
          ?class a owl:Class .
          ?class rdfs:label ?label .
      }
      ORDER BY ?label
      """ .

TopBraid tools include convenience features to quickly produce such test cases from the SPARQL panel. Enter and execute the query until you are happy with the result, then in TopBraid Composer select Turn into Test Case... from the context menu of the SPARQL view:

In TopBraid EDG, you can run a SPARQL query and then use the "camera" button to turn the query results into a repeatable test case:

SHACL Validation Test Cases

The class dash:GraphValidationTestCase can be used to represent test cases to repeatedly validate the query graph against a SHACL [[shacl]] shapes graph. Instances of this type of test cases need to specify zero or more instances of the SHACL validation results classes such as sh:ValidationResult. If the set of provided validation results is empty, the validation is expected to return no violations. For the test case to succeed, the query graph is used as the data graph and the shapes graph of a SHACL validation, and the actual validation results are compared with those stored in the test case.

The following example illustrates a graph validation test case that is expected to return two violations. The example graph also includes the shapes and instances that are part of the validation. 

ex:GraphValidationTestCase
  rdf:type dash:GraphValidationTestCase ;
  dash:expectedResult [
    a sh:ValidationReport ;
    sh:conforms false ;
    sh:result [
      rdf:type sh:ValidationResult ;
      sh:focusNode ex:InvalidResource1 ;
      sh:path ex:dateProperty ;
      sh:severity sh:Violation ;
      sh:sourceConstraintComponent sh:DatatypeConstraintComponent ;
	  sh:sourceShape ex:TestShape-dateProperty ;
      sh:value "2011-01-01"^^xsd:dateTime ;
    ] ;
    sh:result [
      rdf:type sh:ValidationResult ;
      sh:focusNode ex:InvalidResource1 ;
      sh:path ex:integerProperty ;
      sh:severity sh:Violation ;
      sh:sourceConstraintComponent sh:DatatypeConstraintComponent ;
	  sh:sourceShape ex:TestShape-integerProperty ;
      sh:value 11.1 ;
    ] ;
  ] ;
.
ex:InvalidResource1
  rdf:type ex:TestShape ;
  ex:dateProperty "2011-01-01"^^xsd:dateTime ;
  ex:integerProperty 11.1 ;
.
ex:TestShape
  rdf:type rdfs:Class ;
  rdf:type sh:NodeShape ;
  rdfs:label "Test shape" ;
  sh:property ex:TestShape-dateProperty ;
  sh:property ex:TestShape-integerProperty ;
.
ex:TestShape-dateProperty
  rdf:type sh:PropertyShape ;
  sh:path ex:dateProperty ;
  rdfs:label "date property" ;
  sh:datatype xsd:date ;
.
ex:TestShape-integerProperty
  rdf:type sh:PropertyShape ;
  sh:path ex:integerProperty ;
  rdfs:label "integer property" ;
  sh:datatype xsd:integer ;
.
ex:ValidResource
  rdf:type ex:TestShape ;
  ex:dateProperty "2014-09-01"^^xsd:date ;
  ex:integerProperty 0 ;
  ex:integerProperty 1234 ;
  rdfs:label "Valid resource" ;
.

TopBraid tools include convenience features to quickly produce such test cases with a single click. In TopBraid Composer's SHACL Validation view simply run the validation (with the green button) and, when satisfied with the results, use the Create a test case... button from the tool bar:

In TopBraid EDG, run the Problems and Suggestions report and use the "camera" button to produce a repeatable test case.

ADS Script Test Cases

The class dash:ScriptTestCase can be used to represent test cases to repeatedly execute Active Data Shapes script and verify that the result is equal to the one stored in the test case.

The following example illustrates a script test case for a small script that uses g:Afghanistan as its focus node and then counts the number of narrower concepts. 

g:VerifyThatAfghanistanHas1NarrowerConcept
  a dash:ScriptTestCase ;
  dash:expectedResult """{
    \"data\": {
        \"lex\": \"true\",
        \"dt\": \"boolean\"
    },
    \"kind\": \"node\"
}"""^^rdf:JSON ;
  dash:focusNode g:Afghanistan ;
  dash:js "focusNode.narrower.length == 1" ;
  rdfs:label "Verify that Afghanistan has 1 narrower concept"@en ;
.

TopBraid EDG includes a convenience feature to quickly produce such test cases from the Script Editor panel. Simply execute a script and when satisfied with the result, use the "camera" button to product a repeatable test case: