ATS
(e.g. download-atom) / Test
(e.g. A.01.TGR1) / Type of comment1 / Severity
(minor, medium, critical) / Comments / Proposed change / Resolution
all / * / GE / Critical / Based on ISO 19105 and the OGC Specification Model, each Markdown document represents a test case. A test case is part of a conformance class, which in turn is part of an abstract test suite for a specification document.
The current material seems to use “ATS” where “Conformance Class” is probably meant. The information which conformance classes form an ATS is, however, not represented.
In this context it is important to emphasize that conformance class is the key concept. An ATS is far less important as it is just the aggregation of all the conformance classes in a specification.
In addition, the term “ATS” is sometimes also used for a test case. / Use “Conformance Class” (or “CC”/“cc”) where “Abstract Test Suite” (or “ATS”/“ats”) is used now - and where the Technical Guidance includes (implicitly or explicitly) a conformance class. Avoid using wrong a misleading term for test cases.
Group the conformance classes into ATSs based on the specifications that define the conformance classes.
I.e., the structure would be for the current “ATS”s:
- ATS: TG View Service 3.11
- CC: WMS 1.3.0 profile
- TC: A.02.IR04.extended...
- TC: A.03.IR05.schema...
- TC: ... (no more test cases shown below)
- CC: WMTS 1.0.0 profile
- ATS: TG Discovery Service 3.1
- CC: CSW 2.0.2 ISO AP profile
- ATS: TG Download Service 3.1
- CC: Pre-defined Atom
- CC: Pre-defined WFS 2.0.0
- CC: Direct WFS 2.0.0
- CC: Quality of Service
- ATS: TG Metadata 1.3
- CC: ISO 19115/19119 profile
- ATS: TG Spatial Data Services 3.1
- CC: Invocable Spatial Data Services
- CC: Interoperable Spatial Data Services
- CC: Harmonized Spatial Data Services
- TG View Services: The document does not explicitly specify conformance classes for the two OGC standards, but implicitly these are defined as “profiles”.
- TG View and Discovery Services: It is unclear how the requirements regarding QoS have to be treated as the requirements are not identified as requirements with an identifier.
all / * / GE / Critical / An ATS belongs to a specification. Therefore, the references in a test case may only reference one specification.
However, currently sometimes an IR and a TG are referenced. This is not consistent with the notion of conformance classes or abstract test suites. Since IRs do not specify conformance classes and this is typically done in a technical guidance document, the reference should go to the TG (and the requirement in the TG should reference the relevant parts of the IR). / For each test case, identify/reference the relevant requirements in the TG document that specifies the conformance class.
For TGs that identify requirements, list the requirements, otherwise list the most specific section.
all / * / GE / Critical / For cases where a dependency exists to an external conformance class (dependencies are always on the conformance class level, not on the level of test suites or test cases), clearly identify the conformance class. / See comment. For example, do not reference tests “OGC FES 2.0, A.1 Test cases for query”, but reference OGC FES 2.0 Conformance Class “Query”.
all / * / GE / Critical / It would be very useful to clarify for each test case which messages should be reported in case of identified errors in the test object.
While the description of a test case should be independent from both the implementation and the values, the test case “should be complete in the sense that it is sufficient to enable a test verdict to be assigned unambiguously to each potentially observable test outcome (i.e. sequence of test events)” (see ISO 19105). This level of completeness would imply that the potential verdicts / messages can be derived easily from the test case description.
In some cases this is trivial, but in general there is a risk that the messages implemented in an ETS may not contain the information that the ATS authors expect. / For each test case, specify the messages (including the variable information derived from the test object) that should be reported.
all / * / GE / Critical / The section “Prerequisites” should have a reliable structure. Prerequisites should only be other test cases in the same conformance class or other conformance classes.
In some cases, this section does not state other tests that need to be passed first, but actually state new tests. / Update “Prerequisites” to be a list of other test cases in the same conformance class or other conformance classes.
Any prerequisites that actually are new tests need to be converted into separate test cases or included in the test method description.
all / * / GE / Critical / The documentation of the test cases is inconsistent. Sometimes the title is the label (A.xxx.xxx) and sometimes a text. / Make the test case documentation consistent.
all / * / GE / Critical / The test type is mostly “automated”, but some test are “manual” or it is unclear. However, in some cases a requirement is considered “not testable” where a manual test might be possible.
It is clear that the ETS will only implement automated tests, but it may be helpful to understand if manual tests are within the scope of the ATSs or not. / Clarify.
download-atom / A.02.TGR2.conformtoAtomSpecification / CT / GE / Critical / Is this only about validation against RelaxNG or checking conformance against RFC 4287? / If the latter, conformance to RFC 4287 should be a separate ATS and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom / A.03.TGR3.conformtoGeoRSS-Simple / CT / GE / Critical / Is this only about validation of selected elements in an Atom feed against the GeoRSS XML Schema or checking conformance against the GeoRSS specification? / If the latter, conformance to GeoRSS should be a separate ATS and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom / A.04.TGR4.conformtoOpenSearch1.1 / CT / GE / Critical / Validation of OpenSearch description document is more than a test case. There is no schema document or similar that could be used.
(If a single test case would be ok here, there would be no need to break INSPIRE conformance classes down into test cases, just one test case “test conformance against INSPIRE Technical Guidance” would be sufficient.) / Define ATS for the OpenSearch description and identify the test cases to a level that unambiguously identifies the assertions to test.
download-atom / A.04.TGR4.conformtoOpenSearch1.1 / GE / Medium / Open Search 1.1 is not a finalised specification. redirects to Draft 5. / Clarify which draft version is meant (probably Draft 5).
download-atom / A.06.IR511.TGR6.linkToMetadataForTheService / CT / Medium / “the INSPIRE language request parameters in the Resource Locator to the Atom Download Service Feed URLs must be ignored in the comparison” is unclear. Which comparison? The does not seem to be any comparisons of feed URLs only of metadata record URLs? / Clarify.
download-atom / A.06.IR511.TGR6.linkToMetadataForTheService / CT / Minor / “valid gmd:MD_Metadata element” is most likely understood to mean “schema valid”. / If something else is meant, the test should be clarified.
download-atom / A.07.TGR7.selfreference / CT / Medium / Test method seems incomplete. / Clarify Xpath reference for “he default language code defined in the OpenSearch description”.
download-atom / A.08.IR222.TGR8.linktoOpenSearchDescription / GE / Medium / How is this different from A.04.TGR4.conformtoOpenSearch1.1? / Drop one test.
download-atom / A.09.TGR9.feedid, A.21.TGR22.datasetFeedId / CT / Minor / Strictly, the test differs from the requirement. The requirement is not that the id is the same the feed URI provided, but that the id resolves to the same document.
download-atom / A.10.IR221.TGR10.rightselement / ED / Minor / Regarding note 1, the requirement is clear that only /feed/rights is covered. / Remove note 1.
download-atom / A.11.IR221.TGR11.updatedelement, A.23.IR221.TGR24.datasetFeedUpdated / CT / Medium / “... or too far in the past” is vague for testing. / Change to “... or before 2012 (first release of the Technical Guidance)“?
download-atom / A.12.IR221.TGR12.contactinformation, A.24.IR221.TGR25.datasetFeedContactinformation / CT / Medium / Unresolved issue: “A regular expression could be used to validate the email address. Several regular expressions are available; the workgroup could choose one.” / Delete or resolve.
download-atom / A.13.IR221.TGR13.datasetidentifiers / CT / Medium / „the dataset identifier code and dataset identifier namespace must be present in the metadata document of the service“ is too vague. What does „be present“ mean? Included somewhere in the document or in specific elements? / Clarify.
download-atom / A.14.IR221.TGR14.linksToDatasetMetadata / CT / Medium / Same issue as with A.13.IR221.TGR13.datasetidentifiers (“present in the metadata document of the service and in the Download Service feed”). What does present mean? / Clarify.
download-atom / A.14.IR221.TGR14.linksToDatasetMetadata / CT / Medium / gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may refer to multiple nodes. / Clarify how to deal with multiple identifiers in the test.
download-atom / A.14.IR221.TGR14.linksToDatasetMetadata / CT / Medium / gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may be gmd:MD_Identifier, RS_Identifier or some other element. / Clarify how to compare identifiers that use different data types.
download-atom / A.18.TGR19.entryUpdated / CT / Medium / Why is the updated test different from A.11.IR221.TGR11.updatedelement? (Any year will work here.) / Consider aligning tests.
download-atom / A.26.IR313.TGR27.separateEntriesCRSFormat / CT / Medium / “Find and retrieve the all the Download Service Feed documents containing an entry pointing to this Dataset Feed”.
Why “all” the Download Service feed documents? As the test object is “a” Download Service this test should only check the feed of the Download Service under test.
Consistent with this, the Download Service feed document accessed earlier must be used not any other feed document. / “For each category element in the Download Service feed entity, which included the link to the Dataset feed document, check that at least one entry exists in the Dataset feed containing a category element with an identical term attribute.”
download-atom / A.25.IR31.TGR26.datasetFeedDownloadLink, A.28.IR31.TGR29.datasetFeedDownloadLinkDetails / CT / Medium / These seem to overlap significantly. / Consider to merge both test cases.
download-atom / A.29.IR311.TGR31.languageForDownloadLink / CT / Medium / Unresolved issue: “Is the hreflang attribute still mandatory if data in only 1 language is provided? If not, this ATS is not automatically testable and the ATS should be removed.” / Clarify.
download-atom / A.34.IR222.TGR39.provideOpenSearchDescription / GE / Minor / This test case is not referenced from the overview and not testable. / Remove test case.
download-atom / A.36.TGR41.openSearchGenericSearchQueries,
A.37.IR4.TGR42.openSearchUrlDescribeSpatialDataset / CT / Medium / „test if it provides a document with content-type xxx’“ is vague. Is this a test on the HTTP header of the response or in some way a test of the response. / Clarify. Probably a test on the HTTP header of the response is meant.
download-atom / A.39.IR3.IR4.TGR44.openSearchQueryExample / CT / Minor / valid HTTP codes: “200,206,301,303,303” / Change first 303 to 302.
download-predefined-wfs / * / ED / Minor / Numbering of test cases jumps from A.04 to A.06 / Update numbering of test cases
download-predefined-wfs / A.02.IR2.IR4.TGR49.TGR50.TGR51.predefinedStoredQuery / CT / Critical / “For each combination of supported CRS, supported language, SpatialDataSetIdentifier ID code and SpatialDataSetIdentifier Namespace”
The mechanism to determine the “supported CRS” is problematic as the CRS information may differ from feature type to feature type. It is unclear in how far that is an issue in practice, but a conformant service may specify different CRSs per feature type. / See comment. As the TG is unclear, the requirement should be clarified in the TG first.
download-predefined-wfs / A.03.IR221.TGR53.serviceMetadata / CT / Medium / The test method is not consistent with the purpose. The test method checks that both all metadata elements are in the extended capabilities and that there is a MetadataURL pointing to a valid Metadata document. / Change test method to express the ”EITHER ... OR”.
download-predefined-wfs / A.03.IR221.TGR53.serviceMetadata / CT / Medium / “valid Metadata document” is most likely understood to mean “schema valid”. If something else is meant, which is likely as schema validity does not make the metadata document conformant with the service metadata requirements, the test should be clarified. / Clarify meaning of “valid”.
download-predefined-wfs / A.03.IR221.TGR53.serviceMetadata / CT / Minor / Note also that the TG seems to be incorrect and the reference to table 4 should be to table 19. / Update the TG to reference table 19, not table 4.
download-predefined-wfs / A.04.TGR55.TGR56.language.affects.capabilities / CT / Medium / VERSION is not a parameter of the GetCapabilities request. / Change to ACCEPTVERSIONS.
download-predefined-wfs / A.04.TGR55.TGR56.language.affects.capabilities / CT / Critical / “If the returned resource can be parsed as a valid XML document and if the document passes all the tests listed as prerequisites for this test”
Why do the tests need to be executed again, if they have been tested before (“prerequisites”)?
The test cases and conformance classes in “prerequisites” have a WFS 2.0.0 as the test object, while this statement is worded as if these would be test cases and conformance classes on a Capabilities document. / Either drop the prerequisites and explicitly state the steps/assertions or omit the sentence.
download-predefined-wfs / AT / Minor / Maybe a test for requirement 52 / 61 could be added by testing that for each feature type there is exactly one //wfs:FeatureType/wfs:MetadataURL and all values must be identical.
However, the TG does not seem to require this, so this would require a TG update first. / -
ats-download-directaccess-wfs / * / AT / Minor / Maybe a test for requirement 52 / 61 could be added by testing that for each feature type there is exactly one //wfs:FeatureType/wfs:MetadataURL and all values must be identical.
However, the TG does not seem to require this, so this would require a TG update first. / -
ats-metadata / A.01.validate / CT / minor / The ATS requires validation against three different XSD schema versions (at least one should pass). It is not clear how to determine which schema to use to validate the document. / The ATS should perhaps clarify that this is on the basis of the declared namespaces in the XML document (see also issue 10 on GitHub), i.e.:
If the document declares use
But otherwise, it is unclear what to use. The official schema for is or but in the context of Discovery services, INSPIRE also uses It is unclear how to select the schema document. Which schema document does the MD TG require?
ats-metadata / A.02.title / CR / minor / There is still an open question whether to include this test case, as there is no explicit requirement in the technical guidelines. / It is proposed to keep this test case, and change the technical guidelines (TG MD) including this requirement.
ats-metadata / A.03.abstract / CR / minor / There is still an open question whether to include this test case, as there is no explicit requirement in the technical guidelines. / It is proposed to keep this test case, and change the technical guidelines (TG MD) including this requirement.
ats-metadata / A.05.IR14.ds.keyword / ED / Another prerequisite is test case A.04. / Add the test case as a prerequisite.
ats-metadata / A.05.IR14.ds.keyword / CT / minor / Coverage: In theory, the single keyword referring to the INSPIRE data theme could also be a text value, in each of the official languages. Is it OK to assume that in practice all metadata providers will use the language-neutral code?
ats-metadata / A.06.IR15.srv.keyword / ED / Another prerequisite is test case A.04, needed to determine whether the resource is a service. / Add the test case as a prerequisite.
ats-metadata / A.07.IR05.IR06.ds.identification / CR,
CT / Ambiguity: The discussion seems to be still ongoing: “In case of MD_identifier, discussion is ongoing on how to match this element against a namespace-identifier in a capabilities document/service metadata.” (see issue MIWP-8 (L) Unique Resource Identifier)
ats-metadata / A.07.IR05.IR06.ds.identification / ED / Ambiguity: gmd:identificationInfo[1]/*/gmd:citation/*/gmd:identifier may refer to multiple nodes. It should be clarified how to deal with multiple identifiers in the test.
Note: Another prerequisite is test case A.04. / Clarify ambiguity and add test case as a prerequisite.
ats-metadata / A.08.IR03.ds.linkage / CT / Ambiguity: It is not entirely clear which parts of the WSDL or GetCapabilities document need to be tested: “If the response indicates a linkage is a service capabilities or WSDL document, some basic params in the service response are analysed”. Or does this only refer to “Any service response should be checked if it provides proper linkage. The service wsdl or capabilities document should have a featuretype that shares the resource unique identification.” / Perhaps CI_OnLineFunctionCode (optional element) could be used to determine the nature of the online resource locator if present.
ats-metadata / A.08.IR03.ds.linkage / CT / Testability: A manual test is suggested, if the resource locator is a web page with further instructions or a client application.
ats-metadata / A.09.IR04.srv.linkage / CT / Ambiguity: “The URL is resolved.” It is not entirely clear what resolving the URI means. Which maximum timeout can be applied? Should we only look at the HTTP status code (200), or also at the returned information?
Ambiguity: It is not entirely clear which parts of the WSDL or GetCapabilities document need to be tested: “If the response indicates a linkage is a service capabilities or WSDL document, some basic params in the service response are analysed.” Or does it only refer to: “Any service response should be checked if it provides proper linkage. The service wsdl or capabilities document should have a featuretype that shares the resource unique identification.”
Testability: A manual test is suggested, if the resource locator is a web page with further instructions or a client application.
Note: Perhaps CI_OnLineFunctionCode (optional element) could be used to determine the nature of the online resource locator if present (see also TG SDS req 3).
Note: What in case of access restrictions?
Note: Another prerequisite is test case A.04. / Specify which HTTP status codes are acceptable.
ats-metadata / A.10.IR08.IR09.ds.language / Coverage: The test should not go as far as testing whether there are textual values in the datasets or data series.
Ambiguity: The test should specify what the “valid 3-letter language codes according to ISO/TS 19139” are.
Note: The test could perhaps explicitly test if the code list is identified by the URI: . The US Library of Congress is the officially mandated organisation to maintain ISO639.
Note: Another prerequisite is test case A.04.
ats-metadata / A.11.IR10.IR11.ds.topic / ED / Another prerequisite is test case A.04. / Add the test case as a prerequisite.
ats-metadata / A.12.IR12.srv.type / ED / Another prerequisite is test case A.04. / Add the test case as a prerequisite.
ats-metadata / A.14.IR16.IR17.IR18.vocab / CT / Testability: Validating if the keyword is actually available in the indicated vocabulary is a challenge, since the vocabulary is usually not referenced by a URL. If a vocabulary is indicated that is available to the validator, then this check can be performed.