Texas Market Test Plan
Prepared by
Texas SET Working Group
RMS approved 10/28/14
Table of Contents
Table of Contents
Document History
Date/Version
Summary of Changes
1. Texas Market Test Plan
1.1 Purpose and Scope
1.2 Testing Assumptions
1.3 Texas Market Test Plan Revision
2. Testing Website
2.1 Testing Worksheet (TW)
2.1.1 Contacts
2.1.2 Exceptions to the Test Plan
2.1.3 Manually-Assisted Processes
2.1.4 Testing Responsibilities
3. Testing Guidelines......
3.1 Testing Requirements Matrix
3.2 In-Flight Texas Retail Market Testing
3.2.1 New TX SET/ANSI X12 EDI Version Releases
3.2.2 Contingency EDI Providers
3.2.3 Certified Market Participant Changes to a Non-Established Service Provider
3.2.4 New Trading Partnership
3.2.5 Market Participants who Fail to Maintain Certification
3.2.6 Marketplace Functional Changes
3.2.7 Banking Changes
3.3 Out-of-Flight Texas Retail Market Testing
3.3.1 Timing Guidelines
3.3.2 Emergency Changes
3.3.3 Changes Constituting a Specified Ad Hoc Testing
3.4 System Changes
3.4.1 System Change Categories
3.4.2 Translator System Changes and/or Updates
3.4.3 Translator Change Checklist
3.4.4 Back-end System Changes and/or Updates
3.4.5 Marketplace Production Failures
4. Testing Details
4.1 Round Robin Testing
4.1.1 MP Testing Flights
4.1.2 Scenarios
4.1.3 Scripts
4.1.4 Test Days
4.1.5 Check Points
4.1.6 Simulated System Dates
4.1.7 Meter Read and Switching Cycles
4.1.8 EDI Testing
4.1.9 Testing Status Checklist
4.1.10 Conference Calls
4.1.11 Risk Mitigation
4.1.12 General Principals Guiding Test Structure and Completion
4.1.13 Escalation Procedures
4.2 Data Loading
4.2.1 Loading ESI IDs into ERCOT Systems
4.2.2 Providing ESI IDs to CRs
4.3 Certification
4.3.1 Pre-Flight Activities
4.4 Business Process Scenarios
4.4.1 Business Process Certification
5. Testing Requirements of ERCOT and Market Participants
5.1 General Marketplace Requirements
5.2 CR Requirements
5.2.1 Prior to Testing
5.2.2 During Testing
5.2.3 After Testing
5.3 TDSP Requirements
5.3.1 Before Testing
5.3.2 During Testing
5.3.3 After Testing
5.4 ERCOT Requirements
5.4.1 Before Testing
5.4.2 During Testing
5.4.3 After Testing
5.5 PUCT Requirements
5.5.1 Before Testing
5.5.2 During Testing
5.5.3 After Testing
5.6 Flight Administrator Requirements
6. Details of Testing Phases
6.1 Technical Connectivity and Verification
6.1.1 NAESB EDM Testing
6.1.2 TX SET Verification
6.2 End-to-End Testing
6.3 Point-to-Point Testing
Appendices......
Appendix A - Testing Worksheet
Appendix B - Resources
Appendix C - Marketplace Issue Resolution Form
Appendix D - Texas Retail Market Test Bed Load Form
Appendix E - Testing Requirements Matrix
Appendix F – Glossary of Terms & Acronyms Used in this Document not defined in Section 2 of the ERCOT Protocols
Appendix G – Approved Test Flights Schedule
Appendix H – Random ANSI X12 and Business Validation Testing – Procedure Document
Issue......
Goals and Objectives......
Value of Random Testing......
Effects of the Problem......
Causes of the Problem......
Recommended Solution......
Communicating Results......
Reject Transactions......
Document History
Date/Version / Summary of Changes03-23-11 v 1.9
07-15-09 v 1.8 /
- Cleanup related to merger of Texas Test Plan Team into Texas SET Working Group and updated outdated procedures
- Updated Section 4.1.8 to remove language regarding testing through TML
- Added language regarding Round Robin Testing Approach
- Removed Non Established Service Provider from definitions.
- Corrected cosmetic issues
05-13-09 v 1.7 /
- Changes were made to Section3.3.3, Changes Constituting a Specified Ad Hoc Testing
05-29-08 v 1.6 /
- “Out of Flight Testing” added to Table of Contents
- Clarified responsibilities of Market Flight Administrator in regards to pass/fail situations
- Added language on Connectivity Issues
- Defined Escalation Procedures
- Removed Affiliated REP (AREP) and Disconnect for Non Pay (DNP) from conditional testing scenarios
11-16-06 v1.5 /
- Added DUNS+4 Reference for clarification
- Changed Flight Test to Texas Retail Market testing to parallel the new verbiage in the Flight Orientation presentation
- Added clarifying language in section 3.1 regarding Retail Testing Matrix
- Included reference to Random Testing in the Responsibilities section of the Flight Administrator
- Added Appendix H for Random ANSI X 12 and Business Validation Testing Procedure Document
01-16-06 v1.4 /
- Eliminated separate references to TSW and TCW combining both documents into one entitled “Testing Worksheet”; Updated section 3.3.1 defining the blackout period schedule; Added Section 2.1.5 Connectivity Testing Schedule; Included information regarding out of flight testing scripts for current MPs involving SIM Entities and ERCOT; Incorporated TTPT White Paper; Included PIVAR language; Updated Appendix
01-08-05 v1.3 /
- Update to reflect market changes associated with the Market Solution to Stacking; Reformatted; Defined Testing Guidelines; Updated Appendices
01-08-04 v1.2 / Re-formatted TMTP document; Defined Testing Guidelines section; added appendix; updated TX SET business process scenarios
11-03-03 v1.11 / Updated Technical Connectivity and Verification section to include NAESB EDM data transport method. Added references to NAESB EDM data transport throughout document.
01-29-03 v1.10 / Updated to reflect ERCOT role as testing facilitator, Issue Resolution Process, transaction additions and testing requirements applicable to various market participants; deleted the term ITPTA
09-19-02 v1.09 / Updated the Re-testing guidelines timelines to be consistent across sections.
04-30-02 v1.09 / Revised Re-testing guidelines to provide for ERCOT notification to the TTPT Chair
04-26-02 v1.09 / Revised purpose and scope section and re-testing section to clarify that the TMTP was for the Retail Market
04-11-02 v1.09 / Updated Re-testing guidelines to add clarification
03-28-02 v1.09 / Incorporated edit following TTPT review on 3-15-02
03-12-02 v1.09 / Revised re-testing section to reflect the use of automated testing
02-10-02 v1.08 / Incorporate information on automated testing Provided addition explanation for the use of Testing Signoff Worksheet and Technical Connectivity Worksheet Provided additional details on testing the Replacement FTP process Included details on transactions not support by the ERCOT portal Added section on Point-to-Point testing Updated Texas Retail Testing website address to etod.ercot.com
12-13-01 v1.07 Final / Incorporated re-testing guidelines and change in ERCOT connectivity process
10-29-01 v1.07 Draft / Update to reflect changes required for testing during 2002
08-29-01 v1.06 / Update to reflect changes required for Flight 1001
06-17-01 v1.05 Draft / Update to reflect changes required for Flight 3801
05-03-01 v1.04 Final / Moved from DRAFT to FINAL Changed all ‘Certification’ references to ‘Qualification’
04-19-01 v1.04 Draft / Added language to clarify testing for ERCOT Texas Market Link Added section on Test Plan Change Control Added language for provisional qualification guidelines Deleted TTPT membership information and published on website
03-14-01 v1.03 / Final Draft Refreshed script table Clarified success criteria General syntax/grammar/consistency cleanup
02-21-01 v1.03d / Second draft
01-23-01 v1.0d / First draft
1. Texas Market Test Plan
1.1 Purpose and Scope
The purpose of this document is to define the market plan for testing retail commercial operations systems and business processes to support the Texas Electric Choice Market. This document covers all retail testing requirements and procedures between ERCOT and the Market Participants (MPs) and Point-to-Point retail testing between MPs. In an effort to diminish the potential risks that could be introduced into the Texas Retail Electric Market from new unproven systems or MPs or from the effects of new TX SET/ANSI X12 EDI Version Releases, the Texas Market Test Plan provides the mechanism for ensuring that the central retail systems operated by ERCOT are functioning properly, and that the retail systems operated by MPs interface properly with both ERCOT’s systems and other MPs’ systems. In addition to Testing Procedures, the Texas Standard Electronic Transaction (Texas SET) Working Group monitors and reviews metrics on production environments looking for opportunities to improve existing testing procedures. The Texas Market Test Plan addresses the following:
- Testing Guidelines
- Testing Details
- Testing Requirements for Market Participants and ERCOT
- Testing Phases
- Success Criteria
- Overview of Testing Scenarios for Certification in the Texas Market
Texas SET is responsible for maintaining and updating the information in this document as defined in Section 19.8 of the ERCOT Protocols. All references to testing in the document are directed to the Retail Market.
1.2 Testing Assumptions
MPs who wish to participate in the market using NAESB EDM for conducting retail operations in the Texas Marketplace will refer to this document for guidelines on these processes.
MPs may elect to not participate in testing optional processes as identified in this document but will inform their trading partners (TPs) and the Market Flight Administrator in advance. However, an MP does not have the option to refuse to test the basic processes necessary to ensure that the central retail systems operated by ERCOT are functioning properly, and that the retail systems operated by the MPs interface properly with both ERCOT’s systems and other MPs’ systems.
Automated internal processes are required when testing. Any areas that require manual interaction or data manipulation shall be documented in advance in the Testing Worksheet and communicated to testing partners at the beginning of the testing cycle.
- All entities participating in the Texas Retail Market Testing, with the goal of gaining a certification, will use dedicated test environments that are representative of their production environments.
The Market Flight Administrator is the final authority on all levels of Business Process Certification among trading partners, including the verification that a party has successfully passed testing and is eligible to go into production. At any time during flight testing, a CR that is not meeting testing expectations may be advised by the Flight Administrator to withdraw from the flight. This may be related to such scenarios as not sending transactions to Trading Partners in a timely manner, sending transactions containing NAESB/TX SET errors, and/or failure to successfully pass random testing (see Appendix H). Some scenarios would include:
- New CR not currently certified in Texas market – CR would be advised to retest in a future flight test
- Existing CR changing Service Providers (includes testing to bring EDI operations In House) – CR would be advised they must remain with their current Service Provider until they successfully complete testing in a future flight test
- Existing CR changing functionality (Ex. bank change, adding CSA, entering new TDSP territory, etc.) - CR would be advised they must retain all current functionality and would need to retest any changes in a future flight test
Flight Administrator will follow escalation procedures set forth in the TMTP. If at the end of the flight the MP has not withdrawn and the Flight Administrator determines the MP has failed flight testing, the testing certificate shall not be granted and the MP must complete testing in a future flight.
The Market Flight Administrator will moderate testing and report on test status including progress and issues to ERCOT, Retail Market Subcommittee (RMS), Texas SET, other appropriate committees, and/or the PUCT.
Functional Acknowledgements provide a critical audit trail. All parties will send Functional Acknowledgements (FA/997) for all EDI transactions (except for the receipt of a 997, which would create an endless cycle) during testing. Parties shall monitor acknowledgements sent and received, but are not a checklist item for flight success.
1.3 Texas Market Test Plan Revision
Revisions to the Texas Market Test Plan shall be reviewed by the Texas SET Working Group and approved by the Retail Market Subcommittee.
2. Testing Website
The Market Flight Administrator maintains a Texas Retail Testing website (RTW) that details the current status of the testing process. The URL address for this website can be found in Appendix B.
This website includes:
The Texas Market Test Plan (TMTP)
Test Scripts
- Approved Texas Retail Market Test Flight Schedule Timelines
Daily agenda and minutes of each conference call
Texas SET meeting schedule
Testing contact lists
Frequently Asked Questions (FAQs) on the Testing Process
Testing Status - Each organization will be able to obtain a status of the testing process, including its own status. Information will be secured by organization.
Market Links
File Cabinet for significant testing materials
Testing Worksheet (TW)
2.1 Testing Worksheet (TW)
Each MP completes a Testing Worksheet (TW) online. This worksheet includes basic contact information, as well as specific testing communications information, required for effective testing. It also identifies processes that will be tested including optional functions that the MP will use in their business plan and which they plan to test.
The TW link can be found in Appendix A.
2.1.1 Contacts
Parties shall provide daily and emergency contact information for the test lead and the test lead alternate. Issue Resolution procedures require that an executive level contact also be provided.
At least one Business Contact shall be an employee of the Market Participant, not a vendor or service provider.
2.1.2 Exceptions to the Test Plan
Parties cannot arbitrarily require other parties to test certain features, scenarios or scripts, nor can they arbitrarily refuse to test certain features, scenarios or scripts. This Test Plan details full-testing requirements for MPs. There are legitimate scenarios where a party will not support a feature or scenario that is identified in a test script. In these cases, a party can claim an ‘exception to the Test Plan’. These exceptions shall be documented in the TW, and shall be approved by the Market Flight Administrator. The Market Flight Administrator will review exceptions on a case-by-case basis to determine the impact on the Marketplace. Parties that claim “approved” exceptions will not be required to test those features. Once approved, this information will be shared with trading partners.
2.1.3 Manually-Assisted Processes
Each party shall identify the different processes that directly support data exchanges that require manual intervention. Manual intervention increases the risk of errors or process failures and could serve to conceal systemic problems that might introduce transaction errors or hazards into the Market. ANSI X12-formatted files shall never be altered manually except in the case of a simulated error for a test script. This information will be documented in advance on the Testing Worksheet and shared with trading partners.
2.1.4 Testing Responsibilities
The ‘Testing Responsibilities’ section details the responsibilities each party has in the testing process. This Test Plan is focused on testing the most significant features of the marketplace. Also, each party has certain obligations prior to, during, and after testing which are outlined in the ‘Testing Responsibilities’ section.
2.1.5 Connectivity Schedules
Connectivity schedules are arranged by the dates stated in the approved Texas Retail Market Test Flight Schedule.
If a New MP chooses to use a Service Provider they must communicate their choice of Service Provider to the Flight Administrator by noon of the day the Flight Administrator is scheduled to send the testing matrix. (see approved Texas Retail Market Test Flight Schedule. ) No Service Provider changes will be made after the Flight Administrator has sent the testing matrix unless all testing participants have agreed to the change.
3. Testing Guidelines
Pursuant to PUCT rules, any entity intending to participate in the Texas Market must successfully certify their retail commercial applications through Texas Retail Market testing and maintain that certification in accordance with TX SET Version upgrades. Market Testing can be categorized as two types: In-Flight and Out-of-Flight Texas Retail Market testing. In-Flight Texas Retail Market testing consists of market approved scheduled Texas Retail Market testing. There are a defined number of test flights adopted by Texas SET and approved each year by the Retail Market Subcommittee, as directed by the PUCT. Out-of-Flight Texas Retail Market testing is considered only for those changes deemed an “emergency” or a “Specified Ad-Hoc Testing” for existing Market Participants in a specific service territory.
As mentioned above, “emergency” changes or those deemed a “Specified Ad Hoc Testing” for existing Market Participants in a specific service territory are the only changes that will be considered for Out-of-Flight Texas Retail Market testing. Out-of-Flight Texas Retail Market testing requires advance notice to ERCOT and the Market Participant testing contacts listed on the testing website. Upon confirmation of the “emergency” change or “Specified Ad Hoc Testing” by the Market Flight Administrator, based on the scenarios described further in this section, a mutually agreeable Out–of-Flight Texas Retail Market testing schedule will be developed between parties. If an MP is unsure of the lead-time required, it is best practice to contact the Market Flight Administrator for guidance or clarification.
This section provides baseline requirements to assist a Market Participant in determining whether their change qualifies for Out-of-Flight Texas Retail Market testing, or needs to be tested in an approved test flight. These guidelines are intended to minimize risk to the Marketplace. Market Participants (MPs) shall follow well-defined internal change management processes that document results and demonstrate due diligence when making changes.
3.1 Testing Requirements Matrix
A tool has been developed to assist in determining the testing requirements for any changes made to systems or contracts. This matrix is a dynamic guide, which may be changed by Texas SET, to assist with Retail Testing requirements; all testing requirements shall be verified with the Flight Administrator. (See Appendix E for the current version)
3.2 In-Flight Texas Retail Market Testing
There are a defined number of test flights adopted by Texas SET and approved each year by the Retail Market Subcommittee, as directed by the PUCT. Test flights approved by Texas SET, RMS and Technical Advisory Committee (TAC) and are posted on the Texas Retail Testing Website.
3.2.1 New TX SET/ANSI X12 EDI Version Releases
All market participants, including ERCOT, shall complete required certification Texas Retail Market testing as defined by Texas SET when a new TX SET/ANSI X12 EDI Version Release is approved by the Market. On occasion, a Version Release will not consist of any system or transaction changes to the Market or its participants. In that instance, Texas SET and RMS may determine that an additional test flight for a particular release is not necessary. In that event, all parties would not be required to test that specific release