Prepared for:
TheUCAIug OpenADE Task Force, UCAIug SG
Prepared by:
The UCAIug OpenADE Task Force and QualityLogic, Inc
Managed by:
UCAIug OpenADE Task Force
Version
0.01
Revision History
Rev / Date / Summary / Marked0.01 / 2011-21-12 / Initial draft for team review / N
Open Editorial Items and Issues Log
As open items and issues are addressed in new versions of this document, they are removed from this list.
Item No. / Date / Provided By / Summary of the Issue / Status / DispositionInteroperability Test and Certification Management Assistance Analysis / Version – 0.01 / 1
UCA International Users Group / December 12, 2011
Executive Summary
Smart Grid Technical Standards are critically important to development of interoperable products, increasing competition, reducing costs of Smart Grid components, and speeding implementation of new technologies. These key principles enable mainstream market adoption of Smart Grid products and services.This is important for utilities and other market participants to be able to make viable infrastructure investments, and reduce project costs and product development investments required to leverage the opportunities that Smart Grid technologies make available.
The UCA International OpenADE Task Group developed the requirements that led to the North American Energy Standards Board, (NAESB) REQ18/WEQ19, PAP10 Energy Usage Information and NAESB REQ21 Energy Services Provider Interface (ESPI) standards.These are specifically referenced in the November 8, 2011, US Department of Energy Funding Opportunity to demonstrate or pilot innovative applications based on standardized energy usage availability.OpenADE/ESPI is a key standard in the NIST V2.0 Smart Grid Standards Roadmap and is the most important standard addressing technology standardization for enabling consumer access to energy usage information.
Two of the important tasks that the OpenADE Task Group plans to undertake are:
-Develop a conformance, certification, and testing process and program for OpenADE, coordinated with entities such as standard development organizations (SDOs), user groups, and Smart Grid activities.
-Develop programs to allow vendors to develop, test, and demonstrate their ability to integrate with OpenADEcommunications protocols.
Creating a test and certification program is a challenging activity, especially in light of the rigorous requirements embodied in V1.0 and Draft Version 2.0 of the Smart Grid Interoperability Panel (SGIP) Test and Certification Committee’s (TCC) Interoperability Process Reference Manual (IPRM)[1].
This UCA International OpenADE Task Group is aimingto establish an accelerated test and certification program for OpenADE/ESPI (NAESB REQ 21) that can:
- Accelerate the development and implementation of a successful interoperability certification program for OpenADE that meets or exceeds the applicable SGIP TCC IPRM requirements
- Create a standard test specification, test cases, test scripts, test harness,and other tools that can be used by vendors and others to test interoperable OpenADE products prior to certification
- Establish a maintenance and update process and program to e ensure currency of the certification program and pre-certification tools
The balance of this requirements document will review the requirements of the SGIP TCC IPRM; provide commentary on the challenges in achieving the IPRM goals; outline the tasks required to achieve the goals of OpenADEfor the interoperability test and certification; and suggesta set of next steps for the Task Group.
Table of Contents
SGIP IPRM Requirements
Trade Alliance Functions
1.1.ITCA Test and Certification Tasks Based on the IPRM
1.1.1.Organize the ITCA (Implied Tasks)
1.1.2.Manage and Promote the Standard (Implied Tasks)
1.1.3.Organize the Certification Program
1.1.4.Define Certification Program (Explicit Tasks)
1.1.5.Establish Vendor Partnerships (Implied Tasks)
1.1.6.Implement Certification Program (Explicit Tasks)
1.1.7.Improvements in the Standard and the Certification Program (Explicit Tasks)
1.1.8.Cyber-Security (Explicit Tasks)
1.1.9.Governance (Explicit Tasks)
Next Steps
Schedule
Summary
Authors
James Mater, QualityLogic
Steve Van Ausdall, Xtensible / SCE
Edited by: Dave Jollota, QualityLogic
Interoperability Test and Certification Management Assistance Analysis / Version – 0.01 / 1UCA International Users Group / December 12, 2011
SGIP IPRM Requirements
The OpenADE Task Group is fortunate to have access to a ground-breaking guide to developing and managing a world-class test and certification program.Indeed, until the issuance of Version 1.0 of the SGIP IPRM, nothing like it existed for the Smart Grid (or any other industry) that we know of.Every trade alliance like UCA International OpenADETask Group has had to create its own program,which has resulted in a great deal of variation in how such programs have evolved.Having a roadmap such as the IPRM can greatly accelerate achieving the goals of product interoperability based on a specific standard.
It is also critical to understand that the IPRM defines the standard against which the SGIP will be assessing the quality and maturity of certification programs for Smart Grid standards.Although NIST[2] is not directly bound to accept the conclusions and recommendations of the SGIP (in terms of which standards to adopt), it is clear that the SGIP process is closely watched by NIST managers, and SGIP assessments of certification program maturity are expected to influence the decisions that NIST makes.
The IPRM Version 1 identifies some 86 formal requirements in fivedistinct areas that serve as a specification for what a good test and certification program for a Smart Grid technology standard should look like.In addition, a number of guidelines are discussed that further clarify how such a program can achieve its goals of interoperable and conformant products based on the specification.
Version 2 of the IPRM is in draft form but contains significant changes from Version 1.These are noted in this analysis, but it should be understood that until Version 2 is approved by the SGIP these changes are subject to further modification.
The following is a high-level overview of the key requirements that the SGIP has embodied in the IPRM V1.0 and Draft Version 2.Not all may apply to the OpenADE Task Group,and those that do not will be noted as such.
The IPRM defines the structure and functions of an organization that takes industry responsibility for a test and certification program for a standard. The organization is identified as an ITCA, or Interoperability Test and Certification Authority, and is characterized by its authority and competence to design and implement a program.
1.Section 5.1: General Test Policies provides a high-level overview of key policies an ITCA should consider and adopt.These include:
a.The level and types of information provided to vendors for certifications
b.The information that should be included in the final test report
c.Any conditions or expiration limits placed on certifications
d.Conformance versus interoperability certifications.Conformance does not necessarily imply interoperability between products.
e.Trade-off between certification testing and economic/business considerations with attention to safety and Cybersecurity related issues
f.Establishing policies to ensure adequacy of test tools used in certifications
2.Section 5.2 details the requirements for a Test Suite Specification (TSS) and includes details.Section 5.3 is related and deals with attributes of a Test Profile in lieu of a complete TSS.A TSS consists of a suite of tests, categorized into logical functional areas, such as use cases or well-defined features.Each test suite consists of many related test cases corresponding to a particular feature set or use case.A test profile evaluates a subset of a TSS and is used to target specific areas of product interoperability.
a.A common Test Suite Specification (TSS)[3] shall be established when multiple test labs are deployed to test the same standard and/or profile. If common unique test procedures are required to support this test suite, then they shall also be defined. The TSS should be test tool agnostic. Test Suite Specifications (TSS) used for interoperability or conformance testing shall be managed in a well-defined, open and formal manner with change control.
b.The TSS shall be subject to revision control, including revision history, revision numbering, and a defect and expansion management process. The TSS should clearly identify the test purpose, references, resource requirements, test setup, procedures, observable results and possible problems / lessons learned with the test approach. Observables should clearly identify pass / fail / indeterminate requirements and informational elements.
c.A Test profile, which MUST be a subset of a TSS, specifies all mandatory and optional elements and restrictions of the standard specification and is treated as a companion to the technical standard, including submission to an SSO for formal standardization.
3.Section 5.4 enumerates the Technical Design of Test and Certification Programs.This section covers areas of general technical, inheritance, version control, general testing, conformance, testing, interoperability, performance, tools and test lead. There are 35 specific technical requirements for an ITCA. These are summarized below:
a.The ITCA MUSTspecify in the test program requirements those features that are mandatory, and those features that are optional.The ITCA shall require and enforce that vendors declare the optional features implemented in a product. (Tech-1 and -2)
b.The ITCA MUSTrequire that implementations of optional features be tested and certified for conformance and interoperability. Furthermore, the ITCA shall define common test cases for that optional feature to be used by all test labs when testing for that optional feature. (Tech-3)
c.An ITCA MUST define the record handling and retention requirements to be followed by the TL and CB functions, consistent with requirements of ISO 17025 and ISO Guide 65. (Techn-4)
d.TheITCA SHALLspecify conditions under which the use of components that have been certified by other programs can be used in products to be certified by the ITCA program.Basically, OpenADE could decide that specific components or classes of components are “pre-certified” but the ITCA remains responsible for ensuring conformance and interoperability of the OpenADE Task Group certified products.The ITCA SHALL implement a Compliant Portion Description (CPD)[4] to be used as a guide for assembling a product based on compatible sub-components. (Tech-5 to Tech-8)
e.Another section deals with differing versions of certified products and how the certification program should handle them.ITCAs need to manage re-certifications and the identification of current versions that are certified. (Tech-9 to Tech-12)
f.There shall be a defined correlation between implementations and required testing, commonly called a Proforma Implementation Conformance Statement (PICS). (Tech-14)
g.The testing and certification program shall maintain a current and upcoming list of applicable test cases to be called a Test Case Reference List.These test cases should be defined in an open, consensus-driven fashion. These test cases will be used by all test labs approved by the ITCA. There shall be a Test Plan derived from the Test Case Reference List and used by all authorized test labs. Tests shall be identified using the test plan. (Tech-13, Tech-15-16
h.The testing and certification program SHALL require that a static conformance review[5] take place prior to testing a product. (Tech-17)
i.The testing and certification program shall first validate the tests, and implement them utilizing validated test tools. Golden reference test equipment may be utilized where appropriate. (Tech-18)
j.The testing and certification program shall assure that defined product test cases cover application profiles for specific feature sets and functions defined by the specific application profile, and implement interoperability evaluation within that application profile.Where practical, the testing and certification program shall assure that defined product test cases cover all feature sets and functions. (Tech-20 to Tech-22)
k.The ITCA SHALL classify common or major market products according to their application profiles and conduct certification tests based on those applications.The testing and certification program SHALL assure that defined product use cases are covered in application profiles.Interoperability testing and evaluation SHALL be implemented within those application profiles. (Tech-23)
l.A section deals with interoperability testing. The section deals with what are commonly called “plugfests” but also addresses the selection and use of “golden” reference units.Plugfests are optional for application interface standards. (Tech-24 and Tech-25)
m.ITCAs SHALL use reference test tools[6] where appropriate to the technology under test (hardware and/or software) to provide a consistent and replicable approach in generating test results across ITCA test labs. Successful testing programs assure that there is a known reference or constant, to which the system is evaluated against the desired metrics to determine conformance. ITCA program tests that are performed across multiple test facilities SHALL implement processes to assure they are each measuring against a common known reference and achieving repeatable results regardless of location. (Tech-26 and Tech-27)
n.When used, a minimum of two golden units are to be selected by a defined process (Tech-28 to Tech-30)
o.If an ITCA Certification Program involves multiple Smart Grid systems, then the Program Requirements SHALL support end-to-end testing of Smart Grid systems involving multiple product implementations to the fullest extent possible. An ITCA SHALL involve all relevant parties to define various business logic models for the end-to-end system testing, and make scenarios and test harness systems available for testing. (Tech-30-31)
p.The testing and certification program shall ensure that when functional performance requirements are defined in an application profile, the performance test profile(s) shall be designed to implement test cases for evaluating these requirements. (Tech-32)
q.The ITCA SHALL ensure that test tools have a complete mandatory feature-set coverage of a standard. In cases where two or more implementations of optional features are available, the ITCA shall incorporate those feature-sets in the test tool. The ITCA shall define procedures and processes to validate the use of test tools and reference implementations. (Tech-33 to Tech-34)
r.An ITCA shall develop criteria for surveillance to insure that certified products and vendors fulfill the certification agreement(s). The surveillance is to be carried out by its certification body(ies). (Tech-35)
4.Section 5.5 addresses Program and Field Experience Feedback and points to the importance of end-user and product vendor experience with the standard and the certification program.
5.Section 6 of Draft 4, Version 2 of the IPRM is new and addresses Cybersecurity Testing.ITCAs are responsible for coordinating and overseeing the Cybersecurity criteria as applicable to the testing and certification programs that they operate.Section 6.7 includes specific requirements of an ITCA for Cybersecurity testing but these are still in discussion in terms of the ITCA responsibility in this area. This section includes proposed requirements that:
a.The ITCA SHALL define the procedures and processes thatwill be used to validate interoperability Cybersecurity requirements.Such tests are usually initiated by each vendor, to verify that a vendor product meets an industry established level of security, and tested by independent third-party labs using the same testing procedures that are pass/fail in nature. (Sec-1)
b.The testing and certification program shall ensure that Cybersecurity functional performance requirements are defined, and test cases designed to evaluate the requirements. Further, ITCAs are responsible to qualify testing personnel for Cybersecurity training and experience.(Sec-2 and Sec-5)
c.If applicable, ITCAs are responsible for Digital Certificate programs. (Sec-3 and Sec 4)
d.ITCAs are responsible for requiring widely-accepted security stress testing, including static analysis and penetration testing; assuring security policy models drive testing; ensuring that vendors submit threat analyses as part of certification process; documenting programs and standards used for security testing; and incorporating component-based Cybersecurity concepts in the testing program.
One interpretation of this proposed section is that an ITCA needs to establish a distinct certification program just for Cybersecurity issues and include this testing in the certification program.The IPRM is not yet clear on the qualifications for a lab to do Cybersecurity testing.
6.Section 7.2, titled Governance, provides a structural prescription for anITCA. The OpenADE Task Groupintends toestablish the ITCA for the OpenADE/ESPI and related standards, although one purpose of this document is to assist in determining what organization will actually operate the ITCA for OpenADE/ESPI.Key governance requirements are:
a.The ITCA defines and documentsthe interoperability test program for the standard and oversees its implementation, including roles, responsibilities and resources.The ITCA SHALL provide oversight to provide confidence that implementations of Standard(s) in certified products are indeed interoperable.This will be the primary task of the OpenADE Task Group. (Gov-1 and Gov-5 in D3Rev4)
b.The ITCA determines whether first-party testing, third-party testing or both areallowed and defines the circumstances and process for submission to a certification body (CB) as well as the CB responsibilities. (Gov-2 and Gov-3 in D3Rev4)
c.The ITCA SHALL define a corrective process for resolving reported interoperability problems (e.g., in the field or as part of the test) for products for which they are responsible.[7]Further, it SHALL implement preventive processes to avoid recurrence of such problems.A problem may be associated with the specification, the test processes and procedures or the test data. (Gov-4 in D3Rev4)
d.A key function is to ensure that issues that arise through the certification test process are fed back to appropriate parties for clarification or inclusion in subsequent versions of the standard.(Gov-6 in D3Rev4)
e.The ITCA SHALL maintain a publicly available certified product and systems list. (Gov 7 in D3Rev4)