Schematron Replacement Options Paper
Contents
Contents 2
1 Background Information 3
2 Options 4
2.1 Option 1 – Keep using Schematron 4
2.2 Option 2 – Provide an Xml Format of the Business Rules 5
2.3 Option 3 – Provide .NET based components 5
2.4 Option 4 – Provide Other Language specific components 6
2.5 Option 5 – Provide an API call for validation 6
2.6 Option 6 – App Store based solution 7
2.7 Related Option – Use Schema Validation as much as possible 8
1 Background Information
During tax time 2016 the ATO has reviewed and changed the technologies that are used for transformation of Xbrl, Xml and JSON. These changes are based on the .NET framework and have resulted in the following:
· Substantially reduced memory consumption.
· Substantially increased throughput (10 to 100 times faster).
· Far better scalability for larger messages.
· Higher quality through improved development, testing and debugging support.
The ATO is now looking to apply the same change to our internal validation components during tax time 2017. This solution has undergone performance verification and will result in the following:
· Substantially reduced memory consumption.
· Substantially increased throughput (2 to 10 times faster).
· Far better scalability for larger messages.
· Higher quality through improved development, testing and debugging support.
· Reduced complexity.
· Reduced likelihood of EVTE and production issues through higher quality.
· Substantial decrease in delivery time.
· Substantial decrease in time taken to diagnose EVTE and production issues.
· By using the same technologies for validation and transformation this will result in even greater overall throughput and memory efficiency overall.
The benefits for the ATO and for software developers are substantial, however in making this change the ATO will potentially no longer use and/or provide Schematron for third parties. This would result in a material impact for software developers that needs to be discussed, and co-designed with software developers.
2 Options
Dropping support for Schematron has a material impact on software developers that use Schematron for testing purposes. Note that using Schematron within a third party software product at run-time in production is not recommended due to performance and deployment issues.
Note that the use of Formulae link bases has not been considered as it is Xbrl specific. The ATO is pursuing options that are useable for more than just Xbrl.
2.1 Option 1 – Keep using Schematron
In this option the ATO keeps with the status quo and provides Schematron for third parties. This choice also means that the ATO must continue to use Schematron as it is not viable for the ATO to delivery Schematron and a .NET based component. The technologies are too different to support side by side.
Advantages
· No material impact on third party software vendors.
· Software developers can continue to test message payloads without integrated testing with the ATO.
Disadvantages
· Throughput and memory consumption issues will remain. Throughput is 2 to 10 times slower and memory usage is up to 10 times higher.
· Scalability issues will remain.
· Quality issues will remain due to lack of ATO tooling support for development, testing and debugging of Xslt 2.
· Will continue to have an extended time to market.
· Will continue to have difficulties in diagnosing EVTE and production issues.
· Deployment concerns will remain.
o There will always be a deployment time lag when fixes are applied
o The ATO has to spend time on schematron deployment where efforts could have been better spent improving quality in other areas.
o Support still has to be considered for differing platforms and technologies where Xslt 2 support may be lacking.
o Continued risk of providing some third parties a competitive advantage over others depending on the technologies they are using. Some third parties have access to Xslt 2 processors and others may not.
o Introduces component and versioning dependencies for third parties that may not be desirable (e.g. the Java dependency for Saxon). However this should not be an application based dependency.
o This reduces the ability for eCommerce to innovate and provide better solutions.
2.2 Option 2 – Provide an Xml Format of the Business Rules
In this option the ATO will provide no source code\runtime based replacement for the Schematron files that are currently provided to third parties.
Instead third parties would rely on:
· Spreadsheets with improved level of detail for the message structures and business rules.
· An Xml format of the CST/MST, Business Rules and Reference data will be provided.
· As business rules are already moving to a parse-able format rather than the current structured English they also become machine readable.
Advantages
· Performance and quality improvements will be achieved. This arguably has more benefit for third parties than the provision of schematron does.
· The deployment concerns shown in Option 1 will no longer exist.
· As a reference design Schematron is very difficult to understand.
· Achievable within tax time 2017 timeframes.
Disadvantages
· Third parties will not be able to test that their product passes the ATO’s business rules without using the EVTE environment – and thus the internet.
2.3 Option 3 – Provide .NET based components
The ATO’s technology improvements outlined in the section 1 were based on this option. It should be noted that for many years the ATO provided schematron files based on Xslt 1 which had a dependency on the .NET framework.
This would not include the source code however there would be no obfuscation used or licensing restrictions placed on third parties being able to reverse engineer any provided assemblies that include business rules.
Advantages
· Performance and quality improvements will be achieved. This arguably has more benefit for third parties than the provision of schematron does.
· A .NET framework based solution would have more reach than the current XSLT2 based Schematron files do. As the supplied component is not intended to be embedded in third party products the .NET dependency would only exist for testing components.
· Allows third parties to test their messages offline without requiring the SBDH/Ebms3 headers (including security headers).
· Removes a Java dependency for third parties that use Saxon for the XSLT2 based Schematron rules.
· Achievable within tax time 2017 timeframes.
· Will reflect what the ATO would like to use at runtime.
Disadvantages
· Trades an XSLT2 dependency for a .NET framework dependency.
· This option is dependent on the eCommerce branch getting ATO architectural approval to provide externals software developers with executables.
· The ATO may at a later point move to an internal COTS based solution that cannot physically be delivered to third parties. This approach still constrains the ATO from making further improvements to its internal run-time architecture.
· The ATO will need to rework some dependencies on the current .NET solution.
· Deployment concerns will remain.
o There will always be a deployment time lag when fixes are applied
o The ATO has to spend time on component deployment where efforts would be better spent on improving quality.
o Support still has to be considered for differing platforms and technologies.
o Introduces component and versioning dependencies for third parties that may not be desirable – i.e. the .NET framework. However this should not be an application based dependency.
o This reduces the ability for eCommerce to innovate and provide better solutions.
o Security concern of providing hackers with ATO run time details.
2.4 Option 4 – Provide Other Language specific components
In addition to providing a .NET based component the ATO could also provide components based on other languages such as Java, JavaScript, C++.
Advantages:
· Extended coverage for major development platforms.
Disadvantages:
· These components would not be used by the ATO at runtime. Testing these components would reduce the ATO’s time to market rather than increase time to market. Note: Due to this disadvantage ATO would need to limit the number of other language components, and would look to provide a language component only if there was a significant amount of third parties requesting it.
· Would still not reach all development platforms.
· Not achievable within tax time 2017 timeframes.
· Deployment concerns will be exacerbated.
2.5 Option 5 – Provide an API call for validation
Rather than provide a runnable component – instead we provide a simple API that allows third parties to test validation without including security, ebms3 or SBR 1 header requirements. The only security on the API would be white listing.
This is intended only for EVTE testing – not for production usage.
Advantages
· Performance and quality improvements will be achieved and this arguably has more benefit for third parties than the provision of schematron does.
· The deployment concerns shown in Option 1 and Option 3 will no longer exist.
· This solution may also be able to provide access to the artefacts required to complete development.
Disadvantages
· There is a risk that this could not be delivered within tax time 2017 timelines.
· Another service for the ATO to deploy to, monitor and maintain.
· Third parties would need to test over the internet rather than being able to do so locally.
2.6 Option 6 – App Store based solution
This is still the provision of a runnable component – but instead of our current deployment techniques we deploy to the apps stores for Windows 10, Apple and Android – or at least those that our third parties want.
The store based app would only allow just validation of payloads.
Advantages
· Performance and quality improvements will be achieved and this arguably has more benefit for third parties than the provision of schematron does.
· The deployment concerns shown in Option 1 and Option 3 will no longer exist.
· Could be later extended to provide artefacts for developers to support their development.
· Potentially quicker to apply fixes to.
Disadvantages
· There is a high risk that this could not be delivered within tax time 2017 timelines.
· There are some restrictions around what can be placed into an app store which may make this an unrealistic proposition.
· Change for how third parties would be able to test payloads.
2.7 Related Option – Use Schema Validation as much as possible
This option is related to Schematron Replacement and is intended to be applied in addition to the options considered above.
The ATO currently writes a large number of format based rules that could be implemented in Xbrl, Xml and JSON Schemas rather than being explicit business rules. These include:
· Cardinality checks
· Format rules such as Monetary(‘S’, 13, 2)’ are really just regular expression patterns expressed in a less technical manner.
Advantages
· Simpler level of maintenance than is required for business rules.
· Allows third parties to more easily include these format checks within their application – this can be done through the ATO provided artefacts without a dependency on a component being provided.
· Separates out rule business rules from the data format rules.
· Simplifies the business rule components.
Disadvantages
· Schema validators generally provide less meaningful errors. This is only of concern if third parties do not use the ATO provided artefacts to include these checks within their product.
END OF DOCUMENT
Schematron Replacement Options Paper - TWG.docx 2 of 9