Performance Testing
1Introduction
During performance testing projects, customers who do not require the full functionality of Compuware's ServerVantage monitoring product may still need to collect basic UNIX system activity data to enable bottlenecks to be found during analysis. These basic system statisticsare similar to those which may be collected for Windows systems by configuring Remote Monitoring in Compuware's QALoad.
This note discusses several possible ways forward:
Option 1. Use SNMP. Collect statistics and store in the SNMP database (known as the MIB). As QALoad can be configured to collect values of SNMP variables (again via Remote Monitoring), this offers a good level of integration with QALoad's Analyze module.
Option 2. Custom counters. Collect statistics which are added to QALoad's timing file by storing custom counters defined in a QALoad script. This would offer good integration but has the disadvantage of requiring scripting effort.
Option 3. Collect statistics which are loaded into QALoad after a scenario has run.
This note covers the following:
-what statistics to collect, and what Solaris provides
-configuring QALoad to collect values of these SNMP variables
-identify any customisation required to collect missing values
1.1Conclusions and recommendations
Regarding option 1, all the SNMP variables identified in this document can be monitored using Solaris SNMP agents. See section 2 for details.
Unfortunately QALoad does not directly support collection of all of these variables, and cannot be configured to collect custom variables. As a result, Option 2 has been explored via a proof of concept, which indicates the approach is feasible, and this is the recommended approach to collecting missing data.
2What statistics to collect,and what Solaris provides
2.1What to collect
As a baseline for the statistics that are required, the UNIX system resources implemented in Mercury’s LoadRunner v8.0 were examined, and these are listed in a table below in columns 1 and 2.
Articles on Mercury’s support site confirm that LoadRunner does not address clustering scenarios in any special way. So we will assume when servers run in a live-live configuration, the requirement is forstatistics on each to be collected separately and it should be possible to “drill-down” to each separately as well as viewing the overall picture.[1]
LoadRunner collects statistics by making RPC calls directly to the UNIX rstatd(1M) daemon running in the system under test. The comparison table shows the names of the statistics provided by rstatd in column 3, and their data types in column 4.
2.2What SNMP statistics Solaris provides
SNMP is based around an extensible, hierarchical structure within which organisations can define SNMP variables to collect values they require. This is done by defining a custom MIB that may be installed in a run-time environment to allow data collection.
SNMP traps are sent from an SNMP agent to a system management application when specified conditions are detected. This is designed for production system monitoring, and need not be enabled during performance testing.
There are two different MIB definitions which have arisen independently which are relevant to Solaris system monitoring. These are listed in Appendix B of ref[6], and they are UCD-SNMP-MIB and the sun.mib (aka SUN-SEA-EXTENSIONS-MIB).
2.2.1UCD-SNMP-MIB
The Solaris 10 SNMP agent (snmpd) is known simply as the System Management Agent. Compared with previous versions of Solaris, it is a re-write derived from the Open Source project Net-SNMP ( formerly known as ucd-snmp (as the package originated from UC Davis). The root of the sub-tree within the MIB is still named ucdavis.
In Solaris 9, it would be possible to install Net-SNMP as a sub-agent alongside mibiisa (see next section) under control of the SNMP Master Agent.
Variables collected by Net-SNMP are shown in columns 6 and 7 of the table below.
2.2.2SUN-SEA-EXTENSIONS-MIB
In Solaris 9 and earlier, a Sun specific MIB (sun.mib) was developed and supported by an SNMP agent known as mibiisa(1M). This was normally run as a sub-agent, for example running under control of the Sun Solstice Enterprise Master Agent (snmpdx(1M)).
In Solaris 10, the Sun Solstice Enterprise Master Agent is replaced with the System Management Agent derived from Net-SNMP (see previous section). It is still possible to run mibiisa as a sub-agent if required, for example for backwards compatibility, but the recommended approach (ref [7]) is for SNMP requests for values in sun.mib to be sent to the seaExtensions extension module of the System Management Agent. This is an optional module – it is important that it installed and configured to accept such requests.
The sun.mib has also been renamed SUN-SEA-EXTENSIONS-MIB.
Variables collected via the sun.mib are shown in column 5 of the table below.
2.3What rstatd statistics Solaris provides
The rstatd(1M) process was originally developed as part of SunOS (the precursor to Solaris) based around Sun’s RPC mechanism. So, the statistics which Mercury LoadRunner collects could be derived by making direct RPC calls.
An investigation was done by building and testing a Linux version of rstatd(1M) downloaded from This confirmed that the public domain version collects the statistics we require.
As yet, no command line monitor has been located to issue the RPC calls. However, an hour of assembling code fragments from the Internet was sufficient to write a Linux command line monitor using the RPC stub code from the rstatd package. Providing an RPC library is available under Windows, it should be possible to port.
Security concerns mean that RPC services (including portmapper) are often disabled on production systems. I would recommend SNMP is used in preference, unless other factors dictate that we reconsider the use of RPC.
UNIX server monitoring
Version 1.0
Author: David Cater, Acutest Ltd – 1 of 8
Performance Testing
2.4Table comparing SNMP values to collect against what is provided
LoadRunner description / LoadRunnerref [1] / Rstatd value
ref [2] / Rstatd type
ref [2] / sun.mibvariable
(all counter data types)
ref[3] / UCD-SNMP-MIB variable
ref [4] / UCD-SNMP-MIB data type
ref [4]
Average number of processes in ‘Ready’ state during the last minute / Average load / avenrun_1
avenrun_5
avenrun_15 / float / MISSING / ucdavis.laTable.laEntry.laLoad.1 (1m ave)
ucdavis.laTable.laEntry.laLoad.2 (5m ave)
ucdavis.laTable.laEntry.laLoad.3 (15m ave) / String
Collisions per second detected on the Ethernet / Collision rate / colls / counter / rsIfCollisions / MISSING / N/A
Number of switches between processes or threads, per second / Context switch rate / cswitch / counter / MISSING / ucdavis.systemStats.ssRawContexts.0 / Counter32
(number of switches)
Percent of time that the CPU is utilized / CPU utilization / cpubusy / counter / rsSystemProcessTime +
rsUserProcessTime / ucdavis.systemStats.ssCpuRawSystem.0 + ucdavis.systemStats.ssCpuRawUser.0 / Counter32
(100ths of seconds)
Rate of disk transfers / Disk rate / disk / counter / rsDiskXfer1 + rsDskXfer2 +
rsDiskXfer3 + rsDiskXfer4 / ucdavis.systemStats.ssIORawSent.0 +
ucdavis.systemStats.ssIORawReceived.0 / Counter32
(number of blocks)
Errors per second while receiving Ethernet packets / Incoming packets error rate / ipkts / counter / rsIfInErrors / if.ifInErrors / Counter32
Incoming Ethernet packets per second / Incoming packets rate / ierrs / counter / rsIfInPackets / if.ifInOctets. / Counter32
Number of device
interrupts per second / Interrupt rate / intr / counter / rsVIntr / ucdavis.systemStats.ssRawInterrupts.0 / Counter32
Errors per second while sending Ethernet packets / Outgoing packets error rate / oerrs / counter / rsIfOutErrors / if.ifOutErrors.1 / Counter32
Outgoing Ethernet packets per second / Outgoing packets rate / opkts / counter / rsIfOutPackets / if.ifOutOctets.1 / Counter32
Number of pages read to physical memory, per second / Page-in rate / pgin / counter / rsVPagesIn / ucdavis.systemStats.ssRawSwapIn.0 / Counter32
(number of blocks)
Number of pages written to pagefile(s) and removed from physical memory, per second / Page-out rate / pgout / counter / rsVPagesOut / ucdavis.systemStats.ssRawSwapOut.0 / Counter32
(number of blocks)
Number of pages read to physical memory or written to pagefile(s), per second / Paging rate / pgin + pgout / counter / rsVPagesIn +
rsVPagesOut / ucdavis.systemStats.ssRawSwapIn.0 +
ucdavis.systemStats.ssRawSwapOut.0 / Counter32
(number of blocks)
Number of processes being swapped(in)[2] / Swap-in rate / pswpin / counter / rsVSwapIn / MISSING / N/A
Number of processes being swapped(out) / Swap-out rate / pswpout / counter / rsVSwapOut / MISSING / N/A
Percent of time that the CPU is utilized in system mode / System mode CPU utilization / Not explicitly
stated / Not explicitly
stated / rsSystemProcessTime / ucdavis.systemStats.ssCpuRawSystem.0 / Counter32
(100ths of seconds)
Percent of time that the CPU is utilized in user mode / User mode CPU utilization / Not explicitly
stated / Not explicitly
stated / rsUserProcessTime / ucdavis.systemStats.ssCpuRawUser.0 / Counter32
(100ths of seconds)
Items shown in white are listed in the SNMP counters QALoad 5.1 collects for Solaris servers. How QALoad should be configured to achieve this is discussed in the next section.
Items shown in grey are not listed in the SNMP counters QALoad 5.1 collects for Solaris servers. Customisation to provide these statistics has not yet been addressed.
In summary, all the variables required can be collected by at least one of the SNMP agents reviewed. Also, the variables understood by QALoad 5.1 can all be collected via sun.mib (available by default on Solaris 9 and 10).
UNIX server monitoring
Version 1.0
Author: David Cater, Acutest Ltd – 1 of 8
Performance Testing
3Configuring QALoad
QALoad can only collect specific SNMP counters for each type of system under test.
The table in section 2 shows, in white, those which can be collected. There is no known way to configure QALoad to collect those shown in grey, or any other custom SNMP counters.
Providing the SNMP agent is running on the system under test, and firewall ports are open as need to allow UDP traffic to pass from the QALoad Conductor to the system under test, all the configuration required can be done via the QALoad Conductor GUI.
To include the SNMP counters, the Remote Monitoring button is selected on the Machine Configuration tab, then the New button used to launch a dialogue to discover and select the required SNMP counters on selected hosts.
4Customising QALoad
The table in section 2 shows, in grey, the SNMP values which cannot be collected directly via QALoad. As a result, some indirect method is needed.
QALoad allows the user to define custom counters using DEFINE_COUNTER, which may be either instance-type counters (for instantaneous values, or current rates) or cumulative-type counters (for counting errors and events). Values may be written to the QALoad Timing file using VALUE_COUNTER. A script can therefore be written which is run continuously by one virtual user, to collect and store the required values and then sleep for a period.
Remaining sections describe approaches whichhave been considered to producing a QALoad script to collect values for the counters.
4.1Calling an external program
QALoad allows a command line program (or a remote terminal session[3], or remote command execution) to be executed using standard Windows API calls. This is
the simpler method as the details of the programming interfaces and protocols are hidden from QALoad. Other monitoring commands could be invoked, instead of, or as well as, SNMP (eg. RPC clients as mentioned in section 2.3).
Following a proof of concept, a QALoad script was implemented to perform the following:
-Use _popen() and _pclose() to open a pipe to read SNMP counter values froma host using an SNMP command line program, snmpget, installed on Windows XP as part of NetSNMP.
-Host names arecurrently defined in a local datapool. Each virtual user configured on the same Player will monitor a unique host.
-For those SNMP value which are raw counters, derive a current rate for the SNMP counter value and store in an instance-type custom counter, using a microsecond clock timer to derive rate measurements accurately.
-For those SNMP values which are already averaged (eg. Load Average), just store in an instance-type custom counter.
-Sleep for 10 seconds (must be done BEFORE errors are reported, as errors cause DO_Sleep() to return immediately).
-As UDP is unreliable SNMP packets or responses may not be guaranteed. So record errors as failed transactions and continue. WWW_FATAL_ERROR is called to report the transaction as a fail. The script sets DO_AbortOnError(FALSE) to override any run-time settings.
The script was tested using Conductor and results reviewed using Analyze.
The script has been made thread-safe so may be run in process or thread mode.
The script has also been tested across a network connection to a Linux server hosting an SNMP agent which implements all the missing counters in section 2.
The following tasks remain outstanding:
-The script may need to be made portable to Linux, and re-tested, if Linux is to be used for Players.
There are a number of dependencies on configuring the test systems and networks:
-Net-SNMP would have to be installed on the Player systems and configured with the SNMP community names used by the QALoad script.
-SNMP agent would need to be installed on all hosts monitored and SNMP ports open for remote requests from Player systems.
- Local datapools are used forinformation which can be modified within QALoad:SNMP_HOSTNAMES.DAT
- Local configuration files which should not be changed during testing must be copied into the datapools directory: SNMP_CONFIG.csv and SNMP_COUNTERS.csv
4.2Using an SNMP API
WinSNMP is an SNMP programming interface implemented via shared libraries. QALoad also allows external libraries to be included (Options->C++ Compiler Options).
This approach has not been explored in detail as others seem more promising.
The following API implementations have been identified, but not reviewed in detail:
- WinSNMP Developers Kit.
$199 per runtime copy.
Includes C/C++ code samples.
– MG-SOFT SNMP Software Development Lab
$640 (SNMP v3 version)
$290 (SNMPv1 and SNMPv2c only)
Prices include one developer and 20 runtime licenses
Includes C/C++ code samples.
Also offers free time-limited evaluation downloads.
- Agent++ V3.5.
Source code only.
Free SNMP API for C++, unclear if it adheres to WinSNMP.
References
[1] LoadRunner Analysis User’s Guide, Mercury
[2] Sun Solstice Enterprise Manager 4.1 Answerbook, Sun – na.hostperf(8)
[3] Solaris 9 Reference Manual – man pages section 1(M) - mibiisa(1M)
[4] Net-SNMP MIB files (
[5] Monitoring CPU usage with OpenSTA via SNMP,
[6] Solaris 10 Sun System Management Agent Developer’s Guide
[7] Solaris 10 Sun System Management Agent Administration Guide
UNIX server monitoring
Version 1.0
Author: David Cater, Acutest Ltd – 1 of 8
[1] SNMP monitoring is described in the Sun Cluster 2.2 System Administration Guide, but this only covers state monitoring, and does not include any performance monitoring statistics.
[2] Though described as “number of processes” by Mercury, the underlying counters appear to represent number of pages swapped in/out in ref[2] and [3].
[3] Mercury LoadRunner KnowledgeBase article 18748 offers a sample script to run vmstat via a Mercury-specific terminal emulation session. Values for the missing SNMP counters (or equivalent) can all be derived from running vmstat(1M) on the system under test. This would require a telnet port, or similar, to be opened.