Abstract
Issue the command no hw-module slot 1 oversubscription port-group 3 Similarly for your other link: - move the transceiver and link from port 5/14 to port 5/13. remove any links from ports 5/15 & 5/16.
- Country, as many bonuses are No Hw Module Slot 1 Oversubscriptiononly valid to players from certain countries. However, other ranging factors, such as the bonus value and the casino's rating, have been added No Hw Module Slot 1 Oversubscriptioninto the mix as well.
- Casino players Hw Module Slot 1 Oversubscriptionwill receive a free real cash bonus whenever they play at a casino for real money. Casinos usually give out bonuses in the form of deposit matches usually. That means a certain percentage of your deposit is given in free money, so the bigger your deposit, the bigger your bonus.
- Hw-module session 1/0 The Part 1/0 of above command depend from the Output of sh hw-module all attribute Command. Slot 1 Bay 0 Board 0 ModuleSM-X-ES3D-48-P. 1/0 is for Slot 1 Bay 0, than you connect to the EtherSwitch command and can configure this one.
- Rank 0=+n0 slot=1:0-2 rank 1=+n1 slot=0:0,1 rank 2=+n2 slot=1-2 $ mpirun -H aa,bb,cc,dd -rf myrankfile./a.out. Starting with Open MPI v1.7, all socket/core slot locations are be specified as logical indexes (the Open MPI v1.6 series used physical indexes). You can use tools such as HWLOC's 'lstopo' to find the logical indexes of socket.
The ThinkSystem SR650 is a mainstream 2U 2-socket server with industry-leading reliability, management, and security features, and is designed to handle a wide range of workloads.
New to the SR650 is support for up to 24 NVMe solid-state drives. With this support, the SR650 is an excellent choice for workloads that need large amounts of low-latency high-bandwidth storage, including virtualized clustered SAN solutions, software-defined storage, and applications leveraging NVMe over Fabrics (NVMeOF).
This article describes the three new configurations available for the SR650:
- 16 NVMe drives + 8 SAS/SATA drives
- 20 NVMe drives
- 24 NVMe drives
You can also learn about the offerings by watching the walk-through video below.
Change History
Changes in the April 16 update:
- Noted which second-generation Intel Xeon processors are not supported - Ordering information section
Walk-through video with David Watts and Patrick Caporale
Introduction
The Lenovo ThinkSystem SR650 is a mainstream 2U 2-socket server with industry-leading reliability, management, and security features, and is designed to handle a wide range of workloads.
New to the SR650 is support for up to 24 NVMe solid-state drives. With this support, the SR650 is an excellent choice for workloads that need large amounts of low-latency high-bandwidth storage, including virtualized clustered SAN solutions, software-defined storage, and applications leveraging NVMe over Fabrics (NVMeOF).
Figure 1. ThinkSystem SR650 with 24 NVMe drives
Three new configurations are now available:
- 16 NVMe drives + 8 SAS/SATA drives
- 20 NVMe drives
- 24 NVMe drives
NVMe (Non-Volatile Memory Express) is a technology that overcomes SAS/SATA SSD performance limitations by optimizing hardware and software to take full advantage of flash technology. Intel Xeon processors efficiently transfer data in fewer clock cycles with the NVMe optimized software stack compared to the legacy AHCI stack, thereby reducing latency and overhead. NVMe SSDs connect directly to the processor via the PCIe bus, further reducing latency. NVMe drives are characterized by very high bandwidth and very low latency.
Ordering information
These configurations are available configure-to-order (CTO) in the Lenovo Data Center Solution Configurator (DCSC), https://dcsc.lenovo.com. The following table lists the feature codes related to the NVMe drive subsystem. The configurator will derive any additional components that are needed.
Field upgrades: The 20x NVMe and 24x NVMe drive configurations are also available as field upgrades as described in the Field upgrades section.
| Feature code | Description |
|---|---|
| PCIe Switch Adapters | |
| B22D | ThinkSystem 810-4P NVMe Switch Adapter (PCIe x8 adapter with four x4 drive connectors) |
| AUV2 | ThinkSystem 1610-4P NVMe Switch Adapter (PCIe x16 adapter with four x4 drive connectors) |
| B4PA | ThinkSystem 1610-8P NVMe Switch Adapter (PCIe x16 adapter with four connectors to connect to eight drives) |
| NVMe Backplane | |
| B4PC | ThinkSystem SR650 2.5' NVMe 8-Bay Backplane |
| Riser Cards | |
| AUR3 | ThinkSystem SR550/SR590/SR650 x16/x8 PCIe FH Riser 1 Kit (x16+x8 PCIe Riser for Riser 1, for 16 and 20-drive configurations) |
| B4PB | ThinkSystem SR650 x16/x8/x16 PCIe Riser1 (x16+x8+x16 PCIe Riser for Riser 1, for 24-drive configurations) |
| AURC | ThinkSystem SR550/SR590/SR650 (x16/x8)/(x16/x16) PCIe FH Riser 2 Kit (x16+x16 PCIe Riser for Riser 2, for all three configurations) |
Note the following requirements for any of the three NVMe-rich configurations:
- Two processors
- No high-thermal processors:
- 200 W or 205 W TDP are not supported
- Gold 6126T, Gold 6144, Gold 6146, or Platinum 8160T processors are not supported
- Gold 6230N, Gold 6240Y, and Gold 6244 processors are not supported
- No GPU adapters installed
- No PCIe flash adapters installed
- No PCIe adapters with more than 25 W TDP installed
- 1100 W or 1600 W power supplies installed.
- Ambient temperature of up to 30 °C (86 °F)
- If a fan fails and the ambient temperature is above 27 °C, system performance may be reduced.
Although not required, it is expected that these configurations will be fully populated with NVMe drives. Maximum performance is achieved when all NVMe drive bays are filled with drives.
To verify support and ensure that the right power supply is chosen for optimal performance, validate your server configuration using the latest version of the Lenovo Capacity Planner:
http://datacentersupport.lenovo.com/us/en/solutions/lnvo-lcp
Supported NVMe drives
See the ThinkSystem SR650 product guide for the complete list of NVMe drives that are supported in the server: https://lenovopress.com/lp0644#drives-for-internal-storage
The NVMe drives listed in the following table are not supported in the three NVMe-rich configurations.
| Part number | Feature code | Description |
|---|---|---|
| Unsupported NVMe drives | ||
| 7SD7A05770 | B11L | ThinkSystem U.2 Intel P4600 6.4TB Mainstream NVMe PCIe3.0 x4 Hot Swap SSD |
| 7N47A00984 | AUV0 | ThinkSystem U.2 PM963 1.92TB Entry NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00985 | AUUU | ThinkSystem U.2 PM963 3.84TB Entry NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00095 | AUUY | ThinkSystem U.2 PX04PMB 960GB Mainstream NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00096 | AUMF | ThinkSystem U.2 PX04PMB 1.92TB Mainstream NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7XB7A05923 | AWG6 | ThinkSystem U.2 PX04PMB 800GB Performance NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7XB7A05922 | AWG7 | ThinkSystem U.2 PX04PMB 1.6TB Performance NVMe PCIe 3.0 x4 Hot Swap SSD |
Configuration 1: 16x NVMe drives + 8x SAS/SATA
The 16x NVMe drive configuration has the following features:
- 16 NVMe 2.5-inch drive bays plus eight SAS/SATA 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap).
- The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
- The eight SAS/SATA drive bays are connected to a supported 8-port RAID adapter or SAS HBA.
- One PCIe x16 slot is available for high-speed networking such as a 100 GbE adapter, InfiniBand or OPA adapter. If you elect not to configure the eight SAS/SATA drive bays, then you can free up an additional x8 slot for a second networking adapter.
- The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
- ThinkSystem 1Gb 2-port RJ45 LOM
- ThinkSystem 1Gb 4-port RJ45 LOM
- ThinkSystem 10Gb 2-port Base-T LOM
- ThinkSystem 10Gb 2-port SFP+ LOM
- ThinkSystem 10Gb 4-port Base-T LOM
- ThinkSystem 10Gb 4-port SFP+ LOM
- Additional support for one or two M.2 drives, if needed
The 16x NVMe drive configuration has the following performance characteristics:
- Balanced NVMe configuration. In this 16-NVMe drive configuration, each processor is connected to 8 drives. Such a balanced configuration ensures maximum performance by ensuring the processors are equally occupied handling I/O requests to and from the NVMe drives.
- No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.
In the 16x NVMe drive configuration, the drive bays are configured as follows:
- Bays 0-15: NVMe drives
- Bays 16-23: SAS or SATA drives
The PCIe slots in the server are configured as follows:
- Slot 1: 1610-4P NVMe Switch Adapter
- Slot 2: Not present
- Slot 3: Supported RAID adapter for SAS/SATA drives
- Slot 4: 810-4P NVMe Switch Adapter
- Slot 5: Available x16 slot
- Slot 6: 1610-4P NVMe Switch Adapter
- Slot 7 (internal slot): 810-4P NVMe Switch Adapter
The front and rear views of the SR650 with 16x NVMe drives and 8x SAS/SATA drives is shown in the following figure.
Figure 2. SR650 front and rear views of the 16-NVMe drive configuration
The following figure shows a block diagram of how the PCIe lanes are routed from the processors to the NVMe drives. Buying insurance in blackjack.
Figure 3. SR650 block diagram of the 16-NVMe drive configuration
The details of the connections are listed in the following table.
| Drive bay | Drive type | Drive lanes | Adapter | Slot | Host lanes | CPU |
|---|---|---|---|---|---|---|
| 0 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 1 | NVMe | PCIe x4 | 2 | |||
| 2 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 3 | NVMe | PCIe x4 | 2 | |||
| 4 | NVMe | PCIe x4 | 1610-4P | Slot 6 (Riser 2) | PCIe x16 | 2 |
| 5 | NVMe | PCIe x4 | 2 | |||
| 6 | NVMe | PCIe x4 | 2 | |||
| 7 | NVMe | PCIe x4 | 2 | |||
| 8 | NVMe | PCIe x4 | 810-4P | Slot 4 (vertical) | PCIe x8 | 1 |
| 9 | NVMe | PCIe x4 | 1 | |||
| 10 | NVMe | PCIe x4 | 810-4P | Slot 7 (internal) | PCIe x8 | 1 |
| 11 | NVMe | PCIe x4 | 1 | |||
| 12 | NVMe | PCIe x4 | 1610-4P | Slot 1 (Riser 1) | PCIe x16 | 1 |
| 13 | NVMe | PCIe x4 | 1 | |||
| 14 | NVMe | PCIe x4 | 1 | |||
| 15 | NVMe | PCIe x4 | 1 | |||
| 16 | SAS or SATA | RAID 8i | Slot 3 (Riser 1) | PCIe x8 | 1 | |
| 17 | SAS or SATA | 1 | ||||
| 18 | SAS or SATA | 1 | ||||
| 19 | SAS or SATA | 1 | ||||
| 20 | SAS or SATA | 1 | ||||
| 21 | SAS or SATA | 1 | ||||
| 22 | SAS or SATA | 1 | ||||
| 23 | SAS or SATA | 1 |
Configuration 2: 20x NVMe drives
The 20x NVMe drive configuration has the following features:
- 20 NVMe 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap). The other 4 bays are unavailable and are covered by a 4-bay blank.
- The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
- One PCIe x8 slot is available for networking or other needs. The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
- ThinkSystem 1Gb 2-port RJ45 LOM
- ThinkSystem 1Gb 4-port RJ45 LOM
- ThinkSystem 10Gb 2-port Base-T LOM
- ThinkSystem 10Gb 2-port SFP+ LOM
- ThinkSystem 10Gb 4-port Base-T LOM
- ThinkSystem 10Gb 4-port SFP+ LOM
- Additional support for one or two M.2 drives, if needed
The 20x NVMe drive configuration has the following performance characteristics:
- No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.
- Near-balanced NVMe configuration. Unlike the 16-drive and 24-drive configurations, that 20-drive configuration has eight NVMe drives connected to processor 1, and 12 NVMe drives connected to processor 2. As a result, we recommend you to only choose this configuration if you need the additional capacity that four drives provide above the 16-drive configuration, and your workload can fully operate without an equal number of drives connected to each processor.
The PCIe slots in the server are configured as follows:
- Slot 1: 1610-4P NVMe Switch Adapter
- Slot 2: Not present
- Slot 3: Available x8 slot
- Slot 4: 810-4P NVMe Switch Adapter
- Slot 5: 1610-4P NVMe Switch Adapter
- Slot 6: 1610-4P NVMe Switch Adapter
- Slot 7 (internal slot): 810-4P NVMe Switch Adapter
The front and rear views of the SR650 with 20x NVMe drives is shown in the following figure.
Figure 4. SR650 front and rear views of the 20-NVMe drive configuration
The following figure shows a block diagram of how the PCIe lanes are routed from the processors to the NVMe drives.
Figure 5. SR650 block diagram of the 20-NVMe drive configuration
The details of the connections are listed in the following table.
| Drive bay | Drive type | Drive lanes | Adapter | Slot | Host lanes | CPU |
|---|---|---|---|---|---|---|
| 0 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 1 | NVMe | PCIe x4 | 2 | |||
| 2 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 3 | NVMe | PCIe x4 | 2 | |||
| 4 | NVMe | PCIe x4 | 1610-4P | Slot 6 (Riser 2) | PCIe x16 | 2 |
| 5 | NVMe | PCIe x4 | 2 | |||
| 6 | NVMe | PCIe x4 | 2 | |||
| 7 | NVMe | PCIe x4 | 2 | |||
| 8 | NVMe | PCIe x4 | 1610-4P | Slot 5 (Riser 2) | PCIe x16 | 2 |
| 9 | NVMe | PCIe x4 | 2 | |||
| 10 | NVMe | PCIe x4 | 2 | |||
| 11 | NVMe | PCIe x4 | 2 | |||
| 12 | NVMe | PCIe x4 | 810-4P | Slot 4 (vertical) | PCIe x8 | 1 |
| 13 | NVMe | PCIe x4 | 1 | |||
| 14 | NVMe | PCIe x4 | 810-4P | Slot 7 (internal) | PCIe x8 | 1 |
| 15 | NVMe | PCIe x4 | 1 | |||
| 16 | NVMe | PCIe x4 | 1610-4P | Slot 1 (Riser 1) | PCIe x16 | 1 |
| 17 | NVMe | PCIe x4 | 1 | |||
| 18 | NVMe | PCIe x4 | 1 | |||
| 19 | NVMe | PCIe x4 | 1 | |||
| 20 | Blank bay - no connection | |||||
| 21 | Blank bay - no connection | |||||
| 22 | Blank bay - no connection | |||||
| 23 | Blank bay - no connection |
Configuration 3: 24x NVMe drives
These configurations are available configure-to-order (CTO) in the Lenovo Data Center Solution Configurator (DCSC), https://dcsc.lenovo.com. The following table lists the feature codes related to the NVMe drive subsystem. The configurator will derive any additional components that are needed.
Field upgrades: The 20x NVMe and 24x NVMe drive configurations are also available as field upgrades as described in the Field upgrades section.
| Feature code | Description |
|---|---|
| PCIe Switch Adapters | |
| B22D | ThinkSystem 810-4P NVMe Switch Adapter (PCIe x8 adapter with four x4 drive connectors) |
| AUV2 | ThinkSystem 1610-4P NVMe Switch Adapter (PCIe x16 adapter with four x4 drive connectors) |
| B4PA | ThinkSystem 1610-8P NVMe Switch Adapter (PCIe x16 adapter with four connectors to connect to eight drives) |
| NVMe Backplane | |
| B4PC | ThinkSystem SR650 2.5' NVMe 8-Bay Backplane |
| Riser Cards | |
| AUR3 | ThinkSystem SR550/SR590/SR650 x16/x8 PCIe FH Riser 1 Kit (x16+x8 PCIe Riser for Riser 1, for 16 and 20-drive configurations) |
| B4PB | ThinkSystem SR650 x16/x8/x16 PCIe Riser1 (x16+x8+x16 PCIe Riser for Riser 1, for 24-drive configurations) |
| AURC | ThinkSystem SR550/SR590/SR650 (x16/x8)/(x16/x16) PCIe FH Riser 2 Kit (x16+x16 PCIe Riser for Riser 2, for all three configurations) |
Note the following requirements for any of the three NVMe-rich configurations:
- Two processors
- No high-thermal processors:
- 200 W or 205 W TDP are not supported
- Gold 6126T, Gold 6144, Gold 6146, or Platinum 8160T processors are not supported
- Gold 6230N, Gold 6240Y, and Gold 6244 processors are not supported
- No GPU adapters installed
- No PCIe flash adapters installed
- No PCIe adapters with more than 25 W TDP installed
- 1100 W or 1600 W power supplies installed.
- Ambient temperature of up to 30 °C (86 °F)
- If a fan fails and the ambient temperature is above 27 °C, system performance may be reduced.
Although not required, it is expected that these configurations will be fully populated with NVMe drives. Maximum performance is achieved when all NVMe drive bays are filled with drives.
To verify support and ensure that the right power supply is chosen for optimal performance, validate your server configuration using the latest version of the Lenovo Capacity Planner:
http://datacentersupport.lenovo.com/us/en/solutions/lnvo-lcp
Supported NVMe drives
See the ThinkSystem SR650 product guide for the complete list of NVMe drives that are supported in the server: https://lenovopress.com/lp0644#drives-for-internal-storage
The NVMe drives listed in the following table are not supported in the three NVMe-rich configurations.
| Part number | Feature code | Description |
|---|---|---|
| Unsupported NVMe drives | ||
| 7SD7A05770 | B11L | ThinkSystem U.2 Intel P4600 6.4TB Mainstream NVMe PCIe3.0 x4 Hot Swap SSD |
| 7N47A00984 | AUV0 | ThinkSystem U.2 PM963 1.92TB Entry NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00985 | AUUU | ThinkSystem U.2 PM963 3.84TB Entry NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00095 | AUUY | ThinkSystem U.2 PX04PMB 960GB Mainstream NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7N47A00096 | AUMF | ThinkSystem U.2 PX04PMB 1.92TB Mainstream NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7XB7A05923 | AWG6 | ThinkSystem U.2 PX04PMB 800GB Performance NVMe PCIe 3.0 x4 Hot Swap SSD |
| 7XB7A05922 | AWG7 | ThinkSystem U.2 PX04PMB 1.6TB Performance NVMe PCIe 3.0 x4 Hot Swap SSD |
Configuration 1: 16x NVMe drives + 8x SAS/SATA
The 16x NVMe drive configuration has the following features:
- 16 NVMe 2.5-inch drive bays plus eight SAS/SATA 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap).
- The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
- The eight SAS/SATA drive bays are connected to a supported 8-port RAID adapter or SAS HBA.
- One PCIe x16 slot is available for high-speed networking such as a 100 GbE adapter, InfiniBand or OPA adapter. If you elect not to configure the eight SAS/SATA drive bays, then you can free up an additional x8 slot for a second networking adapter.
- The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
- ThinkSystem 1Gb 2-port RJ45 LOM
- ThinkSystem 1Gb 4-port RJ45 LOM
- ThinkSystem 10Gb 2-port Base-T LOM
- ThinkSystem 10Gb 2-port SFP+ LOM
- ThinkSystem 10Gb 4-port Base-T LOM
- ThinkSystem 10Gb 4-port SFP+ LOM
- Additional support for one or two M.2 drives, if needed
The 16x NVMe drive configuration has the following performance characteristics:
- Balanced NVMe configuration. In this 16-NVMe drive configuration, each processor is connected to 8 drives. Such a balanced configuration ensures maximum performance by ensuring the processors are equally occupied handling I/O requests to and from the NVMe drives.
- No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.
In the 16x NVMe drive configuration, the drive bays are configured as follows:
- Bays 0-15: NVMe drives
- Bays 16-23: SAS or SATA drives
The PCIe slots in the server are configured as follows:
- Slot 1: 1610-4P NVMe Switch Adapter
- Slot 2: Not present
- Slot 3: Supported RAID adapter for SAS/SATA drives
- Slot 4: 810-4P NVMe Switch Adapter
- Slot 5: Available x16 slot
- Slot 6: 1610-4P NVMe Switch Adapter
- Slot 7 (internal slot): 810-4P NVMe Switch Adapter
The front and rear views of the SR650 with 16x NVMe drives and 8x SAS/SATA drives is shown in the following figure.
Figure 2. SR650 front and rear views of the 16-NVMe drive configuration
The following figure shows a block diagram of how the PCIe lanes are routed from the processors to the NVMe drives. Buying insurance in blackjack.
Figure 3. SR650 block diagram of the 16-NVMe drive configuration
The details of the connections are listed in the following table.
| Drive bay | Drive type | Drive lanes | Adapter | Slot | Host lanes | CPU |
|---|---|---|---|---|---|---|
| 0 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 1 | NVMe | PCIe x4 | 2 | |||
| 2 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 3 | NVMe | PCIe x4 | 2 | |||
| 4 | NVMe | PCIe x4 | 1610-4P | Slot 6 (Riser 2) | PCIe x16 | 2 |
| 5 | NVMe | PCIe x4 | 2 | |||
| 6 | NVMe | PCIe x4 | 2 | |||
| 7 | NVMe | PCIe x4 | 2 | |||
| 8 | NVMe | PCIe x4 | 810-4P | Slot 4 (vertical) | PCIe x8 | 1 |
| 9 | NVMe | PCIe x4 | 1 | |||
| 10 | NVMe | PCIe x4 | 810-4P | Slot 7 (internal) | PCIe x8 | 1 |
| 11 | NVMe | PCIe x4 | 1 | |||
| 12 | NVMe | PCIe x4 | 1610-4P | Slot 1 (Riser 1) | PCIe x16 | 1 |
| 13 | NVMe | PCIe x4 | 1 | |||
| 14 | NVMe | PCIe x4 | 1 | |||
| 15 | NVMe | PCIe x4 | 1 | |||
| 16 | SAS or SATA | RAID 8i | Slot 3 (Riser 1) | PCIe x8 | 1 | |
| 17 | SAS or SATA | 1 | ||||
| 18 | SAS or SATA | 1 | ||||
| 19 | SAS or SATA | 1 | ||||
| 20 | SAS or SATA | 1 | ||||
| 21 | SAS or SATA | 1 | ||||
| 22 | SAS or SATA | 1 | ||||
| 23 | SAS or SATA | 1 |
Configuration 2: 20x NVMe drives
The 20x NVMe drive configuration has the following features:
- 20 NVMe 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap). The other 4 bays are unavailable and are covered by a 4-bay blank.
- The NVMe drives are connected to the processors either via NVMe Switch Adapters or via the onboard NVMe connectors on the system board of the server.
- One PCIe x8 slot is available for networking or other needs. The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
- ThinkSystem 1Gb 2-port RJ45 LOM
- ThinkSystem 1Gb 4-port RJ45 LOM
- ThinkSystem 10Gb 2-port Base-T LOM
- ThinkSystem 10Gb 2-port SFP+ LOM
- ThinkSystem 10Gb 4-port Base-T LOM
- ThinkSystem 10Gb 4-port SFP+ LOM
- Additional support for one or two M.2 drives, if needed
The 20x NVMe drive configuration has the following performance characteristics:
- No oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration, each drive is allocated 4 lanes from the processor. The 1:1 ratio means no oversubscription of the PCIe lanes from the processors and results in maximum NVMe drive bandwidth.
- Near-balanced NVMe configuration. Unlike the 16-drive and 24-drive configurations, that 20-drive configuration has eight NVMe drives connected to processor 1, and 12 NVMe drives connected to processor 2. As a result, we recommend you to only choose this configuration if you need the additional capacity that four drives provide above the 16-drive configuration, and your workload can fully operate without an equal number of drives connected to each processor.
The PCIe slots in the server are configured as follows:
- Slot 1: 1610-4P NVMe Switch Adapter
- Slot 2: Not present
- Slot 3: Available x8 slot
- Slot 4: 810-4P NVMe Switch Adapter
- Slot 5: 1610-4P NVMe Switch Adapter
- Slot 6: 1610-4P NVMe Switch Adapter
- Slot 7 (internal slot): 810-4P NVMe Switch Adapter
The front and rear views of the SR650 with 20x NVMe drives is shown in the following figure.
Figure 4. SR650 front and rear views of the 20-NVMe drive configuration
The following figure shows a block diagram of how the PCIe lanes are routed from the processors to the NVMe drives.
Figure 5. SR650 block diagram of the 20-NVMe drive configuration
The details of the connections are listed in the following table.
| Drive bay | Drive type | Drive lanes | Adapter | Slot | Host lanes | CPU |
|---|---|---|---|---|---|---|
| 0 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 1 | NVMe | PCIe x4 | 2 | |||
| 2 | NVMe | PCIe x4 | Onboard NVMe port | None | PCIe x8 | 2 |
| 3 | NVMe | PCIe x4 | 2 | |||
| 4 | NVMe | PCIe x4 | 1610-4P | Slot 6 (Riser 2) | PCIe x16 | 2 |
| 5 | NVMe | PCIe x4 | 2 | |||
| 6 | NVMe | PCIe x4 | 2 | |||
| 7 | NVMe | PCIe x4 | 2 | |||
| 8 | NVMe | PCIe x4 | 1610-4P | Slot 5 (Riser 2) | PCIe x16 | 2 |
| 9 | NVMe | PCIe x4 | 2 | |||
| 10 | NVMe | PCIe x4 | 2 | |||
| 11 | NVMe | PCIe x4 | 2 | |||
| 12 | NVMe | PCIe x4 | 810-4P | Slot 4 (vertical) | PCIe x8 | 1 |
| 13 | NVMe | PCIe x4 | 1 | |||
| 14 | NVMe | PCIe x4 | 810-4P | Slot 7 (internal) | PCIe x8 | 1 |
| 15 | NVMe | PCIe x4 | 1 | |||
| 16 | NVMe | PCIe x4 | 1610-4P | Slot 1 (Riser 1) | PCIe x16 | 1 |
| 17 | NVMe | PCIe x4 | 1 | |||
| 18 | NVMe | PCIe x4 | 1 | |||
| 19 | NVMe | PCIe x4 | 1 | |||
| 20 | Blank bay - no connection | |||||
| 21 | Blank bay - no connection | |||||
| 22 | Blank bay - no connection | |||||
| 23 | Blank bay - no connection |
Configuration 3: 24x NVMe drives
The 24x NVMe drive configuration has the following features:
- 24 NVMe 2.5-inch drive bays. All drives are hot-swap from the front of the server (provided the operating system supports hot-swap).
- The NVMe drives are connected to the processors via NVMe Switch Adapters. The onboard NVMe connectors are routed to a riser card installed in Riser slot 1.
- Two x16 slots (one connected to each processor) are available for high-speed networking such as a 100 GbE adapter, InfiniBand or OPA adapter.
- The LOM (LAN on Motherboard) slot is also available for 1Gb or 10Gb Ethernet connections. Supported LOM adapters are the following:
- ThinkSystem 1Gb 2-port RJ45 LOM
- ThinkSystem 1Gb 4-port RJ45 LOM
- ThinkSystem 10Gb 2-port Base-T LOM
- ThinkSystem 10Gb 2-port SFP+ LOM
- ThinkSystem 10Gb 4-port Base-T LOM
- ThinkSystem 10Gb 4-port SFP+ LOM
- Additional support for one or two M.2 drives, if needed
The 24x NVMe drive configuration has the following performance characteristics:
- Balanced NVMe configuration. In this 24-NVMe drive configuration, each processor is connected to 12 drives. Such a balanced configuration provides maximum performance by ensuring the processors are equally occupied handling I/O requests to and from the NVMe drives.
- 2:1 oversubscription. Lenovo NVMe drives connect using four PCIe lanes, and in this configuration each drive is allocated 2 lanes from the processor, resulting in a 2:1 oversubscription of the PCIe lanes. With 24 drives, there are simply not enough PCIe lanes in a two-socket server to support no oversubscription. As a result, the design objective is to minimize the oversubscription while still maintaining balance across all lanes.
- Balanced open slots. This configuration has two open PCIe x16 slots, one connected to each processor. These slots could be used for a pair of high-speed network cards and the result would be balanced configuration.
The PCIe slots in the server are configured as follows:
- Slot 1: 1610-8P NVMe Switch Adapter
- Slot 2: 810-4P NVMe Switch Adapter
- Slot 3: Available x16 slot
- Slot 4: 810-4P NVMe Switch Adapter
- Slot 5: Available x16 slot
- Slot 6: 810-4P NVMe Switch Adapter
- Slot 7 (internal slot): 810-4P NVMe Switch Adapter
The front and rear views of the SR650 with 24x NVMe drives is shown in the following figure.
Figure 6. SR650 front and rear views of the 24-NVMe drive configuration
The following figure shows a block diagram of how the PCIe lanes are routed from the processors to the NVMe drives.
Figure 7. SR650 block diagram of the 24-NVMe drive configuration
The details of the connections are listed in the following table.
| Drive bay | Drive type | Drive lanes | Adapter | Slot | Host lanes | CPU |
|---|---|---|---|---|---|---|
| 0 | NVMe | PCIe x4 | 810-4P | Slot 6 (Riser 2) | PCIe x8 | 2 |
| 1 | NVMe | PCIe x4 | ||||
| 2 | NVMe | PCIe x4 | 2 | |||
| 3 | NVMe | PCIe x4 | ||||
| 4 | NVMe | PCIe x4 | 1610-8P | Slot 1 (Riser 1) | PCIe x16 (from onboard NVMe ports) | 2 |
| 5 | NVMe | PCIe x4 | ||||
| 6 | NVMe | PCIe x4 | 2 | |||
| 7 | NVMe | PCIe x4 | ||||
| 8 | NVMe | PCIe x4 | 2 | |||
| 9 | NVMe | PCIe x4 | ||||
| 10 | NVMe | PCIe x4 | 2 | |||
| 11 | NVMe | PCIe x4 | ||||
| 12 | NVMe | PCIe x4 | 810-4P | Slot 4 (vertical) | PCIe x8 | 1 |
| 13 | NVMe | PCIe x4 | ||||
| 14 | NVMe | PCIe x4 | 1 | |||
| 15 | NVMe | PCIe x4 | ||||
| 16 | NVMe | PCIe x4 | 810-4P | Slot 7 (internal) | PCIe x8 | 1 |
| 17 | NVMe | PCIe x4 | ||||
| 18 | NVMe | PCIe x4 | 1 | |||
| 19 | NVMe | PCIe x4 | ||||
| 20 | NVMe | PCIe x4 | 810-4P | Slot 2 (Riser 1) | PCIe x8 | 1 |
| 21 | NVMe | PCIe x4 | ||||
| 22 | NVMe | PCIe x4 | 1 | |||
| 23 | NVMe | PCIe x4 |
Field upgrades
The following two field upgrade option kits are available to upgrade existing SAS/SATA or AnyBay drive configurations based on the 24x 2.5' chassis (feature code AUVV) to either the 20-drive or 24-drive NVMe configurations.
| Part number | Feature code | Description |
|---|---|---|
| 4XH7A09819 | B64L | ThinkSystem SR650 U.2 20-Bays Upgrade Kit |
| 4XH7A08810 | B64K | ThinkSystem SR650 U.2 24-Bays Upgrade Kit |
These kits include drive backplanes and required NVMe cables, power cables, drive bay fillers, and NVMe switch adapters.
No 16-drive upgrade kit: There is no upgrade kit for the 16x NVMe drive configuration.
The ThinkSystem SR650 U.2 20-Bays Upgrade Kit includes the following components:
- Two 810-4P NVMe Switch Adapters
- Three 1610-4P NVMe Switch Adapters
- One x16/x8 PCIe Riser for Riser 1
- One x16/x16 PCIe Riser for Riser 2
- Three 8-bay NVMe drive backplanes
- One 4-bay drive bay filler
- NVMe and power cables
- Brackets and screws
- Drive bay labels for the front bezel
The ThinkSystem SR650 U.2 24-Bays Upgrade Kit includes the following components:
- Four 810-4P NVMe Switch Adapters
- One 1610-8P NVMe Switch Adapter
- One x16/x8/x16 PCIe Riser for Riser 1
- One x16/x16 PCIe Riser for Riser 2
- Three 8-bay NVMe drive backplanes
- NVMe and power cables
- Brackets and screws
- Drive bay labels for the front bezel
Further information
For more information, see these resources:
- ThinkSystem SR650 product guide
https://lenovopress.com/lp0644-lenovo-thinksystem-sr650-server - Product Guides for ThinkSystem NVMe drives:
https://lenovopress.com/servers/options/drives#term=nvme&rt=product-guide - Paper, Implementing NVMe Drives on Lenovo Servers
https://lenovopress.com/lp0508-implementing-nvme-drives-on-lenovo-servers - Paper, Comparing the Effect of PCIe Host Connections on NVMe Drive Performance
https://lenovopress.com/lp0865-comparing-the-effect-of-pcie-host-connections-on-nvme-drive-performance - Data Center Solution Configurator (DCSC) configurator
https://dcsc.lenovo.com/
Related product families
Product families related to this document are the following:
Trademarks
Lenovo and the Lenovo logo are trademarks or registered trademarks of Lenovo in the United States, other countries, or both. A current list of Lenovo trademarks is available on the Web at https://www.lenovo.com/us/en/legal/copytrade/.
The following terms are trademarks of Lenovo in the United States, other countries, or both:
Lenovo®
AnyBay®
ThinkSystem
The following terms are trademarks of other companies:
Intel® and Xeon® are trademarks of Intel Corporation or its subsidiaries.
Other company, product, or service names may be trademarks or service marks of others.
The catalyst 6880-X has been here for a while now and has already been announced for End-of-Life in October 2019. Last day of support will be Oct 2025. Before that time customers are urged to replace the platform. Replacement for this platform will be the Catalyst 9500 series with all of its flavors.
I'm writing this, because I found it very difficult to find a suitable answer for my question based on oversubscription mode and performance mode. Does switching between these mode in production cause an outage, does the switch need to reboot, does the ASIC's need to be re-programmed?
I have the following hardware at my disposal:
To talk about oversubscription and performance mode we need to take a brief look at what they actually mean.
:: Oversubscription
Most of the catalyst platforms deliver line rate speed at 100Mb, 1Gb, 10Gb, 40 or 100Gb. Line rate is a Layer-1 terminology which essentially means the how fast hardware can put bits in the wire. But is doesn't mean that this speed will be reached over the box from Layer-1 to Layer-3. It depends strongly on how hardware interfaces are connected in a switch, how many of these interfaces are connected and how fast the bus can transfer cascaded bits, frames and packets.
With more complex switches vendors like to refer to a switch fabric or back plane. A back plane can be translated as the big hardware bus interface where all end ports connect to. The backplane can have various throughput speeds to transfer all the data in and out.
When the backplane delivers 1:1 throughput (48x 1Gb/s ports = backplane of 960Gb/s(full duplex)) we talk about non-blocking. We have to calculate full-duplex, otherwise it won't be non-blocking.
When the backplane delivers 2:1 throughput (48x 1Gb/s ports = backplane of 480Gb/s) we talk about oversubscription or oversubscribed.
In Utopia when all ports ask the maximum line rate at the same time, they would get 500Mb/s throughput each. With an oversubscripted interface throughput can shift over the backplane, still delivering line rate as long as not all interfaces require the same amount of throughput at the same time.
For heavy duty switches or DC switches I would not suggest using oversubscribed switches, linecards or fabric cards. Non-blocking is very recommended in DC environments.
Now back to the CAT6880-X…
:: Performance mode
Actually I've already given the answer for Performance mode. It's the same as non-blocking thus allowing line rate speeds and throughput full duplex to pass in and out of the switch.
:: Why switch between these modes?
The supervisors and linecards in my example have each 16 SFP+ Multi-rate ports onboard. You would have 64 10Gb port to use. Since we use VSS it's is required to have performance mode enabled on the switch were the VSL link are connected to. Cisco documents tell you that this is recommended. When they say that it actually means the following:
We recommend the feature to be turned off MEANS We order you to turn off this feature otherwise you won't receive support until you revert back to the recommended settings.
Please be aware of this.
Now with performance mode we can only use 8 ports of the 16 ports available. Why? We have to dive into the architectuur of the hardware.
Quote from the whitepapers:
'Cisco is extending the scale, performance, and capabilities of the venerable Cisco Catalyst 6000 Series with the introduction of the new Catalyst 6800 Series. The new Catalyst 6880-X chassis provides extremely high levels of scalability and performance, with the size and economics of an innovative extensible fixed architecture.
The new Catalyst 6880-X chassis is capable of delivering up to 220Gbps of per-slot bandwidth. This translates to a system capacity of up to approximately 2 Tbps.'
We have 16 10Gb ports per line card. The 220Gb/s slot can accept two linecards in it's bus. Each linecard support up to 80Gb/s.
We can only assume that this calculation is based on full duplex!
That would be 160Gb/s simplex or 320Gb/s full duplex. OMG! It's oversubscribed. To allow line rate non-blocking we must disable ports from usage. And that's where Performance mode comes into play.
When enabling Performance mode ports will be disabled. Both line cards have 2 ASIC group (or port-groups) per linecard. You can disable or enable the required mode per port-group per linecard.
Here are some examples:
Port-group 1: Performance mode
Port-group 2: Oversubscription mode
Port-group 1: 4 usable ports (4x 10Gbx2=80Gb/s)
Port-group 2: 8 usable ports (8x 10Gbx2=160Gb/s)
Total: 80Gb/s per linecard. Port-group one is 1:1 and port-group 2 is 2:1
Port-group 1: Oversubscription mode
Port-group 2: Oversubscription mode
Port-group 1: 8 usable ports (8x 10Gbx2=160Gb/s)
Port-group 2: 8 usable ports (8x 10Gbx2=160Gb/s)
Total: 80Gb/s per linecard. Both port-groups are oversubscribed 2:1
Port-group 1: Performance mode
Port-group 2: Performance mode
Port-group 1: 4 usable ports (4x 10Gbx2=80Gb/s)
Port-group 2: 4 usable ports (4x 10Gbx2=80Gb/s)
Total: 80Gb/s per linecard. 160Gb/s throughput.
Oversubscribed in a 2:1 relation
:: Does converting these mode cause impact?
Let's first distinct the two supervisor options for the 6880-X platform:
- Sup6T
- Sup2T
With the SUP6T architecture and converting from the default oversubscription mode to Performance mode: You'll need to reset the whole VSS system according to the Configuration Guide. Yaykes! Please read the SUP6T architecture and specifically sector VSS.
With the SUP2T there is actually nothing written about moving from one mode to the other. At least I couldn't find it. There is nothing about convert mode in the documentation when it comes to the SUP2T. Time for a TAC question!
Filed in a TAC case Prio 4 to ask the question. Luckily Cisco TAC is very fast in answering Prio 4 questions. The TAC engineer wasn't sure either, so he tested it for me in a testlab.
Result: No ports going down and up and no VSS system reload necessary. Cool!
BUT: as soon as I entered the commands per switch and per slot some things happened.
Ports went down, failover happened on the connected cluster devices.
What happened?
Well I was converting from performance mode to oversubscription mode of one of the port-groups on the linecards (No VSL links attached). The best move you can make. As TAC mentioned ports don't go down and VSS doesn't need a reload. He was right.
Ports went down because a failover was triggered on the WLC member. The ASIC (port-group) was re-programmed and caused a slight interuption on the dataplane. This happens quite fast, but not fast enough to avoid WLC Clusters failovers.
WLC's use keep-alives to their gateways to determine the overall health of a cluster. When the keep-alived fail the member of the cluster is deemed unhealthy and performs a failover. The former secondary member will reload to be put back into the cluster properly.
Then we see ports going down and up again. We also have a Firepower 4120 cluster connected to one of those port-groups and these didn't have any issues. Firepower 4120 cluster is active/active on data plane level, so the traffic will be directed temporary to the other FPR members.
:: Conclusion
So bottomline, Does converting modes in port-groups on the Cat 6880-X create an impact?
Yes and No, but it depends on which type of platform and which type of cluster you have connected to the port-groups. Also it's important how these are connected in the VSS system (straight or cross-connect). Cross-connecting those devices wouldn't have caused a failover in the first place as the gateway would always be available in the port-channels. We did not have these WLC's cross-connected.
Luckily SSO save the day and service was not impacted at all. Now we know that the port-groups are re-programmed and cause a little connectivity disruption when this happens.
No Hw-module Slot 1 Oversubscription Port-group 1
:: Converting in action
And there goes the WLC member…
I've performed this for 4 port-groups on the VSS cluster. Results were all the same.
Hope you've like the post and if you have any questions… 42 would be the answer.
No Hw-module Slot 1 Oversubscription
:: Resources
