CARC - Hussein Al-Azzawi

Performance Benchmarking/Testing of CARC Network,and Connectivity (Psychology and IT-Core)

CARC – Hussein Al-Azzawi

April5, 2014

Introduction

The purpose of this document is to summarize testing and benchmarking results from the evaluation of CARC computer network connectivity to the Psychology Department over IT-Core network. All tests were performed during the first week of March 2014,by: H. Al-Azzawi (CARC/Psychology), B. Archuleta (CARC), S. Wilson (CARC), Ed May (IT), Ed Romero (IT), and J. Ragle (UL).

Summary and Results

Testing and benchmarking results confirmed 10G network connectivity all the way to CARC over the IT-Core network, and the end-to-end connectivity between CARC and Psychology also shows 10G network utilization.

During the testing process we noticed application limitations, for example transferring files using scp is limited to 60MBps (480Mbps) with basic options. When the test files were compressed, the observed transfer rates increased to 110 MBps per scpsession. In general the increase in transfer rate willdependon the nature of the data – i.e. whether it is compressible or not. We can transfer files simultaneously in multiple sessions; this will reduce each session’s transmission rate.For two sessions, the transfer rate will be greater than 110 MBps. But for more than two sessions transfer rate will be less than 110 MBps.

We installed an alternative point-to-point network file transfer tool, bbcp, on Psychology James server and CARC_10G_Client.We reached a stable file transfer rate after running many types of testing, Psychology-CARC maximum bbcp transfer rate was 540 MBps (4320 Mbps), this rate of bbcp is nine times scprate. bbcp is an advanced tool, that requires more tuning, the most important factor is the TCP window size, which should be set to half of the calculated window size, and its variable based on the network connection, in our Psychology-to-CARC case, the best window size to use is 278 KB.

Another important factor is time of day. Since the core network is a shared network, transmission rate is lower during peak hours, when there are more users on the network.When we increase or reduce the maximum transmission unitMTU (frame size) we get worse results.The reason is that the standard MTU is 1500 bytes; if we use larger (9000 bytes) MTU,the network switches will not be able to transmit the frames as they are.Switches will fragment large frames, adding more overhead and increasing packet loss. Reducing MTU size means poor network utilization.

Terms: RTT, NDT, NPAD, perfSONAR, CARC_perfSONAR_Server,SCP, BBCP, MMR, RSC, S2C, C2S,UNM_perfSONAR_Server, CARC_10G_Client, CARC_1G_Client.

Network Points of Testing

In each of the following tests, the perfSONARclient was connected to the switch being tested. The perfSONARserver is either internalto CARC, and connected to the RSC 10G Brocade switch, or externaland hosted at UNMIT. The following tests cover 4 major switches: 1. CARC RSC Brocade Switch (MMR_SW10) 2.Psychology Main Switch (Logan Hall) 3.CARC Main Switch (RR_SW01) 4.CARC to IT-Core Switch (RR_SW05).

  1. CARC RSC Brocade Switch (MMR_SW10)
  1. CARC Internal perfSONAR_Server – CARC_10G_Client
  1. NDT Test:
  1. C2S: 1521.68 Mbps
  2. S2C: 3836.10 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test:
  1. RTT: 5 msec
  2. Target rate: 10000 Mbps
  3. The maximum data rate was 4637.209463Mbps.
  4. Total loss rate 0.000028%.
  1. UNM External perfSonar_Server – CARC_10G_Client
  1. NDT Test:
  1. C2S: 1249.42 Mbps
  2. S2C: 316.26 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test:
  1. RTT: 5 msec
  2. Target rate: 10000 Mbps
  3. Maximum data rate: 1133.58568Mbps.
  4. Total loss rate 0.000013%.
  1. CARC Internal perfSONAR_Server – CARC_1G_Client
  1. NDT Test:
  1. C2S: 940.43 Mbps
  2. S2C: 941.51 Mbps
  3. The slowest link in end-to-end path is 1G.
  1. NPAD Test 1:
  1. RTT: 5 msec
  2. Target rate: 1000 Mbps
  3. The maximum data rate was 941.687700 Mbps.
  4. Pass: zero losses in 1833500 packets, loss rate less than 0.000055%.
  1. NPAD Test 2:
  1. RTT: 5 msec
  2. Target rate: 800 Mbps
  3. The maximum data rate was 942.263876 Mbps.
  4. Pass: zero losses in 1480013 packets, loss rate less than 0.000068%.
  1. UNM External perfSonar_Server – CARC_1G_Client
  1. NDT Test:
  1. C2S: 932.43 Mbps
  2. S2C: 696.27 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test 1:
  1. RTT: 5 msec
  2. Target rate: 1000 Mbps
  3. Maximum data rate: 939.798530 Mbps.
  4. Pass: zero losses in 2915120 packets, loss rate less than 0.000034%.
  1. NPAD Test 2:
  1. RTT: 5 msec
  2. Target rate: 800 Mbps
  3. Maximum data rate: 940.120138 Mbps.
  4. Pass: zero losses in 1694285 packets, loss rate less than 0.000059%.
  1. PsychologyMain Switch (Logan Hall)
  1. CARC Internal perfSONAR_Server – Psychology_10G_James_Server
  1. NDT Test:
  1. C2S: 1532.07 Mbps
  2. S2C: 3979.11 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test:
  1. RTT: 5 msec
  2. Target rate: 10000 Mbps
  3. The maximum data rate was 4809.354943Mbps.
  4. Total loss rate 0.000226%.
  1. UNM External perfSonar_Server – Psychology_10G_James_Server
  1. NDT Test:
  1. C2S: 277.50 Mbps
  2. S2C: 358.67 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test:
  1. RTT: 5 msec
  2. Target rate: 10000 Mbps
  3. Maximum data rate: 969.725658Mbps.
  4. Zero losses in 4587239 packets.
  1. CARC Main Switch (RR_SW01)
  1. CARC Internal perfSONAR_Server– CARC_1G_Client
  1. NDT Test:
  2. C2S: 940.41 Mbps
  3. S2C: 941.42 Mbps
  4. The slowest link in end-to-end path is 1G.
  1. NPAD Test 1:
  2. RTT: 5 msec
  3. Target rate: 1000 Mbps
  4. The maximum data rate was 941.669039Mbps.
  5. Zero losses in 2254727 packets, loss rate less than 0.000044%.
  6. NPAD Test 2:
  7. RTT: 5 msec
  8. Target rate: 800 Mbps
  9. The maximum data rate was 941.655697 Mbps.
  10. Zero losses in 2257376 packets, loss rate less than 0.000044%.
  1. UNM External perfSONAR_Server – CARC_1G_Client
  1. NDT Test:
  1. C2S: 919.38 Mbps
  2. S2C: 282.68 Mbps
  3. The slowest link in end-to-end path is 10G.
  1. NPAD Test 1:
  1. RTT: 5 msec
  2. Target rate: 1000 Mbps
  3. Maximum data rate: 941.716689 Mbps.
  4. Pass: zero losses in 2558529 packets, loss rate less than 0.000039%.
  1. NPAD Test 2:
  1. RTT: 5 msec
  2. Target rate: 800 Mbps
  3. Maximum data rate: 941.723600 Mbps.
  4. Zero losses in 1562044 packets, loss rate less than 0.000064%.
  1. CARC to IT-Core Switch(RR_SW05)
  1. CARC Internal perfSONAR_Server – CARC_1G_Client
  1. NDT Test:
  1. C2S: 937.03 Mbps
  2. S2C: 941.19 Mbps
  3. The slowest link in end-to-end path is 1G.
  1. NPAD Test 1:
  1. RTT: 5 msec
  2. Target rate: 1000 Mbps
  3. The maximum data rate was 941.603690 Mbps.
  4. Pass: zero losses in 2063274 packets, loss rate less than 0.000048%
  1. NPAD Test 2:
  1. RTT: 5 msec
  2. Target rate: 800 Mbps
  3. The maximum data rate was 941.642042Mbps.
  4. Pass: zero losses in 2096811 packets, loss rate less than 0.000048%.
  1. UNM External perfSONAR_Server – CARC_1G_Client

No external testing as the used port is part of internal private VLAN with no Internet access.

Questions

We can see a huge difference If we compare NDT test results between CARC internal perfSONAR server to CARC 10G Client or James server, and UNM external perfSONAR server to CARC 10G client or James server. For example: CARC 10G client on previous NDT S2C test (1,A,i,b) is 3836.10 Mbps, while on NDT S2C test (1,B,i,b) is 316.26 Mbps. Same thing if we compare Psychology James server NDT S2C test (2,A,i,b) is 3979.11 Mbps, while NDT S2C test (2,B,i,b) is 358.67 Mbps. Testing NDT S2C on CARC internal perfSONAR server is about 12 times faster than on UNM external perfSONAR server. Its known that the slowest link on end-to-end path for both of perfSONAR servers is 10G? Why is this difference?

File Transfer Testing

Test 1 - 03.07.14:Running one SCP session without compression

Average file transfer speed was 65MBps.

Test 2 - 03.07.14:Running one SCP session with compression

Average file transfer speed was 110MBps.

Test 3 - 03.12.14: Running two SCP sessions without compression

Average file transfer speed was 26MBps per session giving a total of 52MBps.

Test 4 - 03.12.14: Running two SCP sessions with compression

Average file transfer speed was 53MBps per session giving a total of 106MBps.

Test 5 - 03.12.14: Running three SCP sessions without compression

Average file transfer speed was17MBps per session giving a total of 51MBps.

Test 6 - 03.12.14:Running three SCP sessions with compression

Average file transfer speed was 33MBps per session giving a total of 100MBps.

Test 1 - 03.07.14: Running one BBCP session without compression, 278 KB window size.Average file transfer speed was 540MBps.

Glossary

  • CARC: Center for Advanced Research Computing [2].
  • CARC_Cuda01 and CARC_Cuda02: Two identical machines, each has: 8 Core i7-2600K CPU @ 3.40GHz, Myri-10G Dual-Protocol NIC, 16GB Ram, 2 x 1TB drives, 60GB SSD, 2 x MSI N560GTX-Ti [2].
  • CARC_10G_Client: Hosted on CARC_Cuda01, IP address: 129.24.240.15 [2].
  • CARC_1G_Client: 13-inch MacBook Pro with Retina display, 2.8GHz Dual-core Intel Core i7, Turbo Boost up to 3.3GHz, 16GB 1600MHz DDR3L SDRAM, 1TB PCIe-based Flash Storage [2].
  • CARC_perfSONAR_Server:

Internal perfSONAR server, 10G connection direct to the RSC 10G switch, used for internal network testing and troubleshooting, and secured external accessed through CARC’s firewall by request, hosting server: CARC_Cuda02, IP address: 129.24.240.14, access link: [2].

  • C2S: Client to server.
  • NDT:Allows end users to test the network path for a limited number of common problems, such as inadequate TCP buffer sizes and duplex mismatches. NDT attempts to determine what kind of performance the user should expect, and what the current limiting factor is [1].
  • MMR: CARC main machine room [2].
  • NPAD: Allows end-users to test limited portions of the network path and attempts to determine if there are issues that would adversely affect longer paths. The user provides a target data rate and round-trip-time (RTT) and NPAD attempts to determine if that should be possible, given the infrastructure on that limited portion of the path [1].
  • perfSONAR: Is a web services-based infrastructure for collecting and publishing network performance monitoring. A primary goal of perfSONAR is making it easier to solve end-to-end performance problems on paths crossing several networks. It contains a set of services delivering performance measurements in a federated environment [1].
  • RSC: Research Storage Consortium, storage system size is 0.56 Petabyte, lives in CARC MMR [2].
  • RTT: Stands for round trip time, the required time of a packet to travel from the source to destination, and the time of the acknowledge to travel back from the destination to the source.
  • S2C: Server to client.
  • UNM_perfSONAR_Server: UNM perfSONAR server, 10G connection speed, used for network testing and troubleshooting, IP address: 208.77.77.234, access link: 208.77.77.234/toolkit/ [2].

References

[1]Brian Tierney, Martin Swany, and Je Boote. perfSONAR: Instantiating a Global Network Measurement Framework. ACM SIGOPS Operating Systems Review, 43(4):50–53, January 2010.

[2]Systems Group, CARC Internal Documentation, CARC.

1