UCAR RFP R17-27363 Attachment 1A, Cheyenne Infiniband Fabric (V1)

UCAR RFP R17-27363 Attachment 1A, Cheyenne InfiniBand Fabric (v1)

UCAR RFP R17-27363 Attachment 1A

Cheyenne InfiniBand Fabric

Contents

1Cheyenne’s InfiniBand Fabric

2Cheyenne’s InfiniBand Hardware

3Cheyenne’s Fabric Topology

4Reference Information

1Cheyenne’s InfiniBand Fabric

NCAR’s SGI ICE XA[1] petaflop class supercomputer, Cheyenne, has a single plane 9D Enhanced Hypercube (EHC) high-speed communications fabric. The base layer of the fabric is composed of Mellanox EDR InfiniBand Switch-IBTM ASICs in pairs which form a single SGI Premium switch. The fabric is routed by an OpenSM Dimension-Order Routing (DOR) routing engine. DOR uses MinHop with optimizations for redundant cables in each dimension, which is determined by a cable’s portNum on a switch ASIC.

All Cheyenne nodes with InfiniBand currently run Mellanox OFED 3.2.2; but NCAR anticipates upgrading to Mellanox OFED 3.4 in the near future.

In Cheyenne’s fabric, one port of each ASIC is generally available for adding new nodes, such as the Data Analysis and Visualization (DAV) nodes being acquired in UCAR RFP R17-27363.

2Cheyenne’s InfiniBand Hardware

The table below contains the physical InfiniBand hardware present in Cheyenne to allow vendors to check for compatibility. Cable counts may change due to repairs and other hardware additions.

Hardware / Count / PSID or Product Number / Firmware Version
SGI Premium EDR Switch / Mellanox Switch-IBTM Pair / 224 / SGI_P0003833_X_1 and SGI_P0003833_X_0 / 11.1200.0102
SGI ConnectX-4 HCA / 4032 / SGI_P0001721_X / 12.14.2036
Mellanox ConnectX-4 HCA / 12 / MT_2180110032 / 12.14.2036
Mellanox ConnectX-4 HCA / 35 / MT_2190110032 / 12.18.2000
1M Copper EDR Cable / 1350 / MCP1600-E001
2M Copper EDR Cable / 1798 / MCP1600-E002
2M Copper EDR Cable / 898 / MCP1600-E01A
10M Fiber EDR Cable / 2448 / MFA1A00-E010
15M Fiber EDR Cable / 778 / MFA1A00-E015
30M Fiber EDR Cable / 38 / MFA1A00-E030
50M Fiber EDR Cable / 72 / MFS1200-E050

3Cheyenne’s Fabric Topology

Cheyenne’s InfiniBand Fabric topology is a partial 9D Enhanced Hypercube (EHC). There is a maximum of ten hops between any two nodes. The first two dimensions of the hypercube have three cables and the other dimensions have two cables to form an enhanced hypercube, instead of a conventional hypercube[2] - this is done for resiliency and added bandwidth.

Cheyenne’s EHC is based on the Mellanox Switch-IBTMASIC[3]. Each SGI Premium switch, composed of two interconnected Switch-IBTM ASICs, form the first (enhanced) dimension, as three ports are hard wired between each ASIC. Nine compute nodes are connected to a single ASIC. Each higher dimension is built by connecting lower dimensional pieces in a cubic pattern. This can generally best be shown by how the cables are physically connected to the switches.

SGI uses consistent port numbers on each switch ASIC for the cable connections to each EHC dimension, as shown in the following table. Duplicate cables do not change the shape of the hypercube.

Presently on Cheyenne, there are approximately 400 evenly distributed open EDR ports available for additional node integration.

Single Switch-IBTM ASIC SwitchPort Layout
Port Numbers / Nodes / Purpose
1-9 / 9 / Connection to node HCAs
10-12 / 18 / 1D (Connections between ASIC pairs in a Premium switch)
13-15 / 36 / 2D (1 IRU)
16-17 / 72 / 3D (2 IRUs or ½ Rack)
18-19 / 144 / 4D (1 Rack)
20-21 / 288 / 5D (2 Racks, 1 ICE XA E-Cell)
22-23 / 576 / 6D (4 Racks, 2 ICE XA E-Cells)
24-25 / 1152 / 7D (8 Racks, 4 ICE XA E-Cells)
26-27 / 2304 / 8D (16 Racks, 8 ICE XA E-Cells)
28-29 / 4032 / Partial 9D (28 Racks, 14 ICE XA E-Cells)
30 / 0 or 1 / Open for Service, Storage, and DAV nodes
31-36 / 0 / Ports physically unavailable

The interconnect topology of a single SGI ICE XA E-cell, or 5D hypercube, is depicted above. Each cube in the diagram is a 3D hypercube composed of eight Switch-IBTM ASICs, or four SGI Premium switches.

A 6D hypercube follows the same pattern, but in it each cube in the diagram is composed of 4D hypercubes. The nodes of an SGI ICE XA E-cell pair are interconnected as a 6D hypercube.

The following figure provides a representation of Cheyenne’s partial 9D Hypercube, which is comprised of seven 6D hypercubes (i.e. 14 SGI ICE-XA E-Cells).

4Reference Information

[1] SGI ICE XA and E-Cell:

[2] Network Hypercube Topology:

[3] Mellanox Switch-IBTM ASIC:

Page 1