Creating your Sizing Guide at the
iSeries Benchmark Center
....getting started
Helen Olson-Williams
Richard Smitherman
Howard Sykes
Cindy Mestad
Mike Meyers
Terry Ford
April 4, 2004
Process Overview 3
1.0 Documentation 3
2.0 Planning Session Calls 4
2.1 Introduction to Workload Solution Offering 4
2.2 Planning Calls – In depth planning and review 5
2.3 Final Call - Readiness Review 5
3.0 End-User questions 6
4.0 Preparation: Spreadsheets and Questionnaires 8
4.1 Workload Definition Template 9
4.2 Detailed User Group Scenarios - Paper Scripts 9
4.3 Run Matrix Template 10
4.4 Hardware/Software Questionnaire 13
4.5 System Customization, Application Install Procedure 15
4.6 Database Questionnaire 16
4.7 Run Procedures 17
4.8 Workload Simulation Questionnaire 18
4.9 CPW/CIW Requirements Template 19
4.10 Memory Requirements Template 19
4.11 Disk Requirements Template 19
4.12 Daily Activities Workplan 20
5.0 Create and Test the Workload 22
6.0 Save the workload as XML and HTML 22
7.0 Validation 23
Appendix A – Example of Paper Scripts 24
Appendix B – Legal Documents Required 30
Customer Copyright Release (customer owns the copyright) 30
Customer Third-Party Copyright Release (customer is a licensed user) 31
Inadvertent Disclosure Letter 32
Internet Usage Agreement 33
Remote Usage Agreement 36
Process Overview
· Submit a Benchmark Nomination, http://www.ibm.com/eserver/iseries/developer/cbc
· Read the documentation provided.
· Participate in planning conference calls with the iSeries Benchmark Center.
· Decide on the end-user questions.
· Fill out the planning spreadsheet templates and questionnaires provided by the iSeries Benchmark Center. Provide paper scripts of Business Transactions.
· Run tests, gather and analyze performance data at the iSeries Benchmark Center.
· Create the Workload Solution for your application at the iSeries Benchmark Center, using the Workload Developer.
· Have your solution approved in the IBM Global Solutions Directory (GSD) and enrolled in IBM eServer Solution Connection (eSC).
The Workload Solution that you create at the iSeries Benchmark Center will be the foundation for your sizing tool. In it you will specify the end-user questions and the calculations that will produce the recommended servers. Your customers and end users can link to your web site to execute the IBM Workload Estimator and to get the recommended servers for your solution.
1.0 Documentation
The following documents should be read and understood prior to filling out the planning templates and questionnaires.
· “Developing Workload Solutions for the IBM eServer Workload Estimator” provides a step-by-step how-to guide for creating a workload solution.
http://www.developer.ibm.com/graphics/csf/common/pdf/IBMeServerWL-HowTo.pdf
· “IBM eServer Workload Developer Users Guide” is the Users Guide for a tool called the Workload Developer. You will create the Workload Solution for your application using this tool and the data collected at the Benchmark Center.
http://www.developer.ibm.com/graphics/csf/common/pdf/IBMeServerWL-UsersGuide.pdf
2.0 Planning Session Calls
Planning Time Line
2.1 Introduction to Workload Solution Offering
The purpose of the Introductory Planning Call is primarily to describe the offering. The program requirements will be described along with the scope of the offering, hardware available, and the effort involved, including required planning and preparation. The schedule for preparation and a timeframe for the week-long test in Rochester will be discussed.
Following this call the customer will need to identify the application(s), the database and the business transactions to be tested. Templates and questionnaires will be made available to collect this information. These must be filled out and returned prior to the next planning call.
The requirements of the Workload Solution offering are as follows:
· A fully tested and stable application - no test code, please
· A database representative in size and number of records of a typical customer’s data
· An application or sub-set of an application to be sized
NOTE: Because of the short duration and the amount of specialized work involved, the workloads to be sized should be targeted toward those most often marketed or those workloads that are problematic to your marketing efforts
· The interface to identified workload must be HTML or 5250-based
· An understanding of your customers interaction with the application, i.e., how do they navigate though the application
· An understanding of your customer set, i.e., number of users, business transaction volumes, orders entered in a day, hits-per-second, number of customer accounts, etc.
2.2 Planning Calls – In depth planning and review
Prior to the first planning conference call, draft versions of the following documentation should be completed and sent to the Benchmark Center.
· End-User Questions,
· Workload Definition Template,
· Run Matrix Template,
· Hardware/Software Questionnaire,
· Database Questionnaire,
· Workload Simulation Questionnaire, and
· Daily Activities Workplan
These planning conference calls are used as working sessions to answer any questions on the process and the preparation templates. The number of calls needed will vary.
At the end of the planning phase, “final” versions of all the documentation mentioned above should be available, in addition to paper scripts documenting the end-user’s navigation through the application.
2.3 Final Call - Readiness Review
Prior to this call, the preparation templates and questionnaires, including the paper scripts should be complete and emailed to the Benchmark Center. The purpose of this call is to complete a final review of the planning templates, review the paper scripts, and assess the readiness of the benchmark effort. Typical questions include:
· Have the End-User Questions been defined?
· Has the Workload Definition Template been completed? We will walk through these items.
· Has the Run Matrix Template been completed including prioritization of the runs? We will walk through these items.
· Have Paper Scripts been created? We will review these items.
· Has the Hardware/Software Questionnaire been completed? We will walk through these items.
· Has the Database Questionnaire been completed? We will walk through these items.
· Has the Workload Simulation Questionnaire been completed? We will walk through these items.
· Daily Activities Workplan – does a detailed benchmark plan exists which encompasses all necessary steps to bring this Workload Sizing exercise to a successful completion? This includes a detailed, realistic plan of events in Rochester that is documented, from the install through system clean-up.
· Are you adequately staffed and prepared to assume all responsibilities and complete all tasks as required in your benchmark activities work plan?
· Is a stable version of the database and programs saved off to tape?
· Is the Restore/Install procedure documented and fully “debugged”?
· Are all application / system set up procedures (creation of user profiles, authority changes, pool sizes, etc. for the scripts) identified?
· Are copyright releases and any necessary confidential disclosure and internet usage agreements signed?
3.0 End-User questions
The Workload Estimator is primarily a sales tool for your solution. It should be designed to be used quickly by the end user. Pick a few significant features of your solution that the sales people will be asked about by potential customers. These will generally relate to the input questions for the tool. You could ask your sales representatives to develop a list of questions that they feel are the most important ones they will get asked when doing a sizing effort for a customer or prospect. Keep the list of questions under a dozen or so. Your questions will be presented to the end-user as in the following example. You can access this example WLE Workload Solution at the following URL. Look toward the bottom of the web page. Documentation on developing this workload is also available at:
http://www.developer.ibm.com/welcome/eserver/e3/CSFServlet?mvcid=main&packageid=3000
The screen above shows some questions from a order entry application: How many active users, peak number of orders entered per hour, etc. The questions will vary by application type. Some more examples are below.
· How many users will be logged on? How many will be actively using the system?
· How many concurrent active web sessions will you have?
· How many messages will you receive/send per hour? What is the average message size?
· How many transactions will be run per hour? (or minute) (by transaction type if you have multiple transaction types)
· How many of your users are light users, typical users, heavy users? (Don't forget to define the details of each of these user types.)
· What size database will you have (number of customers, items, orders,...)?
· Do you have seasonal peaks? If so how much higher is you peak load?
· What life span do you want for this system? How much growth should be allowed for over this life span?
Help text should be provided describing what is meant by each question.
Questions for your application are:
4.0 Preparation: Spreadsheets and Questionnaires
In order to come up with the performance calculations for the workload, it will be necessary to run the application on different iSeries models and gather performance data. The goal is to link the end-user questions to the processing resources required. (Refer to the following document:
http://www.developer.ibm.com/graphics/csf/common/pdf/IBMeServerWL-HowTo.pdf
for detailed information on this process.) Several tests will be required, varying the workload and the hardware until all of the necessary performance information is gathered. This data is analyzed, and calculations or formulas are determined to input into the workload generation tool.
These tests will be run in the iSeries Benchmark Center on Benchmark Center hardware. The LoadRunnerâ tool will most likely be used to generate scripts to drive the application. There are a series of templates available to guide you through the planning the tests. The above mentioned document must be read and the templates must be filled out and “paper scripts” must be developed before you will be ready for the one-week test at the iSeries Benchmark Center. Excel versions of the templates are available.
4.1 Workload Definition Template
The first template is the workload definition template. It is used to identify the different user groups and the key functions they perform. Scripts will be created for each User Group or Business Transaction, depending on the level of granularity desired in the sizing tool. Virtual users will execute the scripts on the iSeries. While the scripts are executing, system resource utilization will be measured. This data forms the basis of the sizing tool. Care should be taken to select the most representative functions. Due to practical and time constraints the number of functions must be kept to a minimum.
A key column in this template is “Peak Transactions per Hour”. The workload should represent a typical peak hourly rate of work for a given set of users.
The workload definition should be reviewed and approved prior to the creating paper scripts. No modifications can be made during the benchmark test.
Here is an example.
User Group / # Users / Business Transactions / Peak Trans/Hr / Trans/User/HrOrder Management / 50 / Order Entry
Order Inquiry
/ 150
125
/ 3
2.5
Inventory Management / 50 / Ship Confirm
Ship Tracking
/ 50
100
/ 1
2
Note: In the previous table, Peak Trans/Hr is the total transactions executed by all of the users,
i.e. Peak Trans/Hr = Trans/User/Hr * # Users.
4.2 Detailed User Group Scenarios - Paper Scripts
For each User Group (or Business Transaction) identified in the Workload Definition Matrix, there is a corresponding set of keystrokes that represent the typical workflow for the group of users. These keystrokes (and/or mouse movements) must be documented. Screen captures can be used to document the workflow. Variable and constant data should be identified.
See Appendix A for a partial example of a paper script.
More information about the LoadRunnerâ product can be found on the Mercury Interactive website:
http://www.mercuryinteractive.com/products/loadrunner/
4.3 Run Matrix Template
The Run Matrix template lays out the various runs or tests that are planned to be executed in the Benchmark Center. The User Group, hardware configuration and number of users are key components in this template. The number of users is a target for planning purposes. It is not known until after running some of the scripts if in fact the hardware configuration will support the targeted number of users. The three different numbers of users become three points on a curve in that illustrates how well the application scales and how much CPU is required per user or per transaction. Typically a first run is done with a minimal number of users on a small configuration. (In the example below, 50 users on the 810 running Order Management.) Then the number of users is increased until the CPU is about 60% busy. A final point is taken with the goal of 80-90% CPU utilization. Quite often the scripts can be designed such that the 3 measurement points are executed as a single run with users added and data gathered in stages. The following is an example of a completed Run Matrix.
Run # / User Group / Business Transactions / Hardware Configuration / Target Number of Users / Priority / Comments1 / Order Management / Order Entry
Order Inquiry
/ 810-2465
1-way / 50 / 100 / 150
/ 1
2 / Order Management / Order Entry
Order Inquiry
/ 810-2469
2-way / 100 / 150 / 500
/ 7
3 / Order Management / Order Entry
Order Inquiry
/ 825-2473
3-way / 150 / 500 / 1000
/ 3
4 / Inventory Management / Ship Confirm
Ship Tracking
/ 810-2465
1-way / 50 / 100 / 150
/ 2
5 / Inventory Management / Ship Confirm
Ship Tracking
/ 810-2469
2-way / 100 / 150 / 500
/ 6
6 / Inventory Management / Ship Confirm
Ship Tracking
/ 825-2473
3-way / 150 / 500 / 1000
/ 8
7 / Order Management / Order Entry
Order Inquiry
/ 810-2469
2-way / 50 / 100 / 150
/ 4 / Memory test
8 / Inventory Management / Ship Confirm
Ship Tracking
/ 810-2469
2-way / 50 / 100 / 150
/ 5 / Memory test
Notes:
- The example in the table above assumes that Order Entry and Order Inquiry are combined together as a single workload. For greater granularity in the sizing tool, each of the Business Transactions could be scripted and run separately. This would be advantageous if two different customer installations would typically have different ratios of order entry transactions to order inquiry transactions. For example, in a traditional phone order entry environment there may be twice as many new orders entered as there are inquiries on existing orders. In a web environment, where the inquiry function is available to the end-user customer, there may be many more inquiries than new orders. Capturing each transaction separately allows the sizing tool to size various ratios of the different transactions. If this variability in transaction mix is not necessary, then the transactions could be captured together in a single “order management” workload, as shown here.
- In the example above we show 2 different 810 models, a one-way and a two-way. These tests could be run on a single 810-2469 by logically removing a processor from the configuration to approximate an 810-2465.
- The first 6 runs listed are designed to size CPU and the last two are tests used to gather information for memory sizing. See the “Developing Workload Solutions” document for further information on sizing memory and disk. For the CPU tests listed here, memory and disk configurations are not specified. They need to be sized large enough so that neither memory nor disk are bottlenecks.
- After completing this matrix, the Daily Activities Workplan can be filled out. Each run can be mapped to a specific timeframe for planning purposes.
- The matrix here shows a fairly complete set of runs: 2 workloads on 3 different processor configurations. In many cases it won’t be necessary to complete every point in the matrix. Keep in mind that the goal of this exercise is to determine the relationship between business transactions and CPU required to process them. It may be sufficient to run one of the workloads on all 3 configurations and run the other workload(s) on just one or two of the configurations. In this example, the runs that are lower priority would be good candidates to leave out of the Workplan.
In the chart below we show how a single run can be performed that has 3 data points: 50 users, 100 users and 150 users.