Introduction
Virtual Fabrics (vFabrics)* allow multiple network applications running on the same fabric at the same time with limited interference. Using vFabrics, a physical fabric is divided into many overlapping virtual fabrics, which keep network applications separate even though they connect to the same physical fabric. Virtual fabric is a feature of the Intel® Omni-Path Architecture (Intel® OPA) Fabric Manager (FM). For a complete overview of the FM, please refer to the document Intel® Omni-Path Fabric (Intel® OP Fabric) Suite Fabric Manager User Guide.
Typical usage models for vFabrics include:
- Separating a cluster into multiple virtual fabrics so independent applications running in different virtual fabrics can run with minimal or no effect on each other.
- Separating classes of traffic. Each class of traffic runs in a different virtual fabric so that they don’t interfere with each other.
Each vFabric can be assigned quality of service (QoS) and security policies to control how common physical resources in the fabric are shared among the virtual fabrics. A virtual fabric is defined as a list of applications, a list of device groups, with a set of security and QoS along with other identifiers for the virtual fabric. Refer to Section 2.1 of the document Intel® Omni-Path Fabric (Intel® OP Fabric) Suite Fabric Manager User Guide for more information on virtual fabric.
This document shows how to configure vFabrics, how to use various Intel® OPA command lines to display virtual fabrics, to display port counters, and to clear them. The Intel® MPI Benchmarks, part of Intel® Parallel Studio XE 2018 Cluster Edition, is used to generate traffic in these virtual fabrics. Finally, you can verify the packets sent and received in the virtual fabrics with the Intel® OPA command lines.
Preparation
The following setup and tests are run on two systems equipped with the Intel® Xeon® processor E5-2698 v3 @2.30 GHz and the Intel® Xeon Phi™ processor 7250 @ 1.40 GHz. The first system has the IP address 10.23.3.28 with hostname lb0, the second system has the IP address 10.23.3.182 with hostname knl-sb2. Both systems are running Red Hat Enterprise Linux* 64-bit 7.2. Also, on each system, an Intel® Omni-Path Host Fabric Interface (Intel® OP HFI) Peripheral Component Interconnect Express (PCIe) x16 is installed and connected directly to the other HFI. The Intel® Omni-Path Fabric Software version 10.4.2.0.7 is installed on each system. Also, the Intel® Parallel Studio XE 2018 Cluster Edition is installed on both systems to run the Intel® MPI Benchmarks.
In this test, IP over Fabric (IPoFabric) is also configured. The corresponding IP addresses on the Intel® Xeon® processor host and the Intel® Xeon Phi™ processor are 192.168.100.101 and 192.168.100.102, respectively.
Creating Virtual Fabrics
The Intel® OPA fabric supports redundant FMs. This is implemented in order to ensure management coverage of the fabric continues in the case of failure on the master FM. When there are redundant FMs, only one is selected as a master FM arbitrary, and others are standby FMs. Only the master FM can create vFabrics.
Using the above systems, you will create the vFabrics and generate traffic in these vFabrics. First, you need to determine the master FM. Both commands opareport and opafabricinfo can show which FM is a master FM. For example, issue the command opafabricinfo in either machine lb0 or knl-sb2:
# opafabricinfo Fabric 0:0 Information: SM: knl-sb2 Guid: 0x00117501017444e0 State: Master SM: lb0 Guid: 0x0011750101790311 State: Inactive Number of HFIs: 2 Number of Switches: 0 Number of Links: 1 Number of HFI Links: 1 (Internal: 0 External: 1) Number of ISLs: 0 (Internal: 0 External: 0) Number of Degraded Links: 0 (HFI Links: 0 ISLs: 0) Number of Omitted Links: 0 (HFI Links: 0 ISLs: 0) -------------------------------------------------------------------------------
This indicates that the master FM resides in the machine knl-sb2.
Show the current virtual fabrics with the following command. You will see two virtual fabrics were configured by default: one is named Default; the other is Admin:
# opapaquery -o vfList Getting VF List... Number of VFs: 2 VF 1: Default VF 2: Admin opapaquery completed: OK
In the following exercise, you will add two virtual fabrics named “VirtualFabric1” and “VirtualFabric2”. To create virtual fabrics, you can modify the /etc/opa-fm/opafm.xml configuration file in the host where the master FM resides. Note that existing tools such as opaxmlextract, opaxmlindent, opaxmlfilter, and opaxmlgenerate can help to manipulate the opafm.xml file.
The /etc/opa-fm/opafm.xml configuration file stores information about how the fabric is managed via the master FM. This file uses standard XML syntax. In the node where the master FM resides, knl-sb2 in this case, edit the FM configure file /etc/opa-fm/opaf.xml and add two virtual fabrics by adding the following lines in the virtual fabrics configuration section:
# vi /etc/opa-fm/opafm.xml ........................ <VirtualFabric><Name>VirtualFabric1</Name><Application>AllOthers</Application><BaseSL>1</BaseSL><Enable>1</Enable><MaxMTU>Unlimited</MaxMTU><MaxRate>Unlimited</MaxRate><Member>All</Member><QOS>1</QOS></VirtualFabric><VirtualFabric><Name>VirtualFabric2</Name><Application>AllOthers</Application><BaseSL>2</BaseSL><Enable>1</Enable><MaxMTU>Unlimited</MaxMTU><MaxRate>Unlimited</MaxRate><Member>All</Member><QOS>1</QOS></VirtualFabric> ........................
A virtual fabric is created by adding an XML element <VirtualFabric>. Inside the XML element, you can define many parameters for this virtual fabric. The following parameters are defined for the virtual fabrics used in this example:
- Name: a unique name for this virtual fabric.
- Application: a catchall that can be used to identify all applications.
- BaseSL (base service level): allows a specified service level (0–15) to be used for the vFabric. In this example, VirtualFabric1 uses service level=1, and VirtualFabric2 uses service level=2.
- Enable: vFabric is enabled if this field is set to 1. If set to 0, the vFabric is simply ignored (allows user to easily disable the vFabric without deleting it).
- MaxMTU: the maximum of Maximum Transmission Unit (MTU) of the package. The actual value of MTU may be further reduced by hardware capability or by request. It can be set to unlimited.
- MaxRate: the maximum static rate. It can be set to unlimited.
- Member: the group name that specifies whether a device can talk to any other members.
- QoS: 1 means the QoS for this vFabric is enabled.
A complete list of parameters is shown in Section 6.5.12 of the document Intel® OPA Fabric Suite Fabric Manager User Guide.
Log in as root or a user with privileges, and restart the master FM in knl-sb2 to read the new /etc/opa-fm/opafm.xml configuration file:
# systemctl restart opafm
Verify the master FM is up and running:
# systemctl status opafm ● opafm.service - OPA Fabric Manager Loaded: loaded (/usr/lib/systemd/system/opafm.service; enabled; vendor preset: disabled) Active: active (running) since Tue 2017-11-14 08:39:58 EST; 18s ago Process: 36995 ExecStop=/usr/lib/opa-fm/bin/opafmd halt (code=exited, status=0/SUCCESS) …………
Now, display all current virtual fabrics:
# opapaquery -o vfList Getting VF List... Number of VFs: 4 VF 1: Default VF 2: Admin VF 3: VirtualFabric1 VF 4: VirtualFabric2 opapaquery completed: OK
The output shows that the two new virtual fabrics “VirtualFabric1” and “VirtualFabric2” are added. To show the configuration of the recently added virtual fabrics, issue the command “opareport -o vfinfo”:
# opareport -o vfinfo Getting All Node Records... Done Getting All Node Records Done Getting All Link Records Done Getting All Cable Info Records Done Getting All SM Info Records Done Getting vFabric Records vFabrics: Index:0 Name:Default PKey:0x8001 SL:0 Select:0x0 PktLifeTimeMult:2 MaxMtu:unlimited MaxRate:unlimited Options:0x00 QOS: Disabled PreemptionRank: 0 HoQLife: 8 ms Index:1 Name:Admin PKey:0x7fff SL:0 Select:0x1: PKEY PktLifeTimeMult:2 MaxMtu:unlimited MaxRate:unlimited Options:0x01: Security QOS: Disabled PreemptionRank: 0 HoQLife: 8 ms Index:2 Name:VirtualFabric1 PKey:0x2 SL:1 Select:0x3: PKEY SL PktLifeTimeMult:2 MaxMtu:unlimited MaxRate:unlimited Options:0x03: Security QoS QOS: Bandwidth: 33% PreemptionRank: 0 HoQLife: 8 ms Index:3 Name:VirtualFabric2 PKey:0x3 SL:2 Select:0x3: PKEY SL PktLifeTimeMult:2 MaxMtu:unlimited MaxRate:unlimited Options:0x03: Security QoS QOS: Bandwidth: 33% PreemptionRank: 0 HoQLife: 8 ms 4 VFs
This confirms that VirtualFabric1 and VirtualFabric2 have service level (SL) 1 and SL 2, respectively. Get membership information for all vFabrics in every node in the fabric:
# opareport -o vfmember
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Getting All Port VL Tables...
Done Getting All Port VL Tables
VF Membership Report
knl-sb2
LID 1 FI NodeGUID 0x00117501017444e0
Port 1
Neighbor Node: lb0
LID 2 FI NodeGUID 0x0011750101790311
Port 1
VF Membership:
VF Name VF Index Base SL Base SC VL
Default 0 0 0 0
Admin 1 0 0 0
VirtualFabric1 2 1 1 1
VirtualFabric2 3 2 2 2
lb0
LID 2 FI NodeGUID 0x0011750101790311
Port 1
Neighbor Node: knl-sb2
LID 1 FI NodeGUID 0x00117501017444e0
Port 1
VF Membership:
VF Name VF Index Base SL Base SC VL
Default 0 0 0 0
Admin 1 0 0 0
VirtualFabric1 2 1 1 1
VirtualFabric2 3 2 2 2
2 Reported Port(s)
-------------------------------------------------------------------------------
Virtual lanes (VLs) allow multiple logical flows over a single physical link. They are configurable from 0 to 8 VLs plus one management. The above output shows these two new virtual fabrics use VL 1 and VL 2, respectively. Note that local identifier (LID) is assigned to every port in the fabric.
Get the buffer control tables for all vFabrics on every node in the fabric:
# opareport -o bfrctrl
Getting All Node Records...
Done Getting All Node Records
Done Getting All Link Records
Done Getting All Cable Info Records
Done Getting All SM Info Records
Done Getting vFabric Records
Done Getting Buffer Control Tables
BufferControlTable Report
Port 0x00117501017444e0 1 FI knl-sb2 (LID 1)
Remote Port 0x0011750101790311 1 FI lb0 (LID 2)
BufferControlTable
OverallBufferSpace (AU/B): 2176/ 139264
Tx Buffer Depth (LTP/B): 128/ 16384
Wire Depth (LTP/B): 13/ 1664
TxOverallSharedLimit (AU/B): 1374/ 87936
VL | Dedicated ( Bytes) | Shared ( Bytes) | MTU
0 | 224 ( 14336) | 1374 ( 87936) | 10240
1 | 224 ( 14336) | 1374 ( 87936) | 10240
2 | 224 ( 14336) | 1374 ( 87936) | 10240
15 | 130 ( 8320) | 1374 ( 87936) | 2048
Port 0x0011750101790311 1 FI lb0 (LID 2)
Remote Port 0x00117501017444e0 1 FI knl-sb2 (LID 1)
BufferControlTable
OverallBufferSpace (AU/B): 2176/ 139264
Tx Buffer Depth (LTP/B): 128/ 16384
Wire Depth (LTP/B): 11/ 1408
TxOverallSharedLimit (AU/B): 1374/ 87936
VL | Dedicated ( Bytes) | Shared ( Bytes) | MTU
0 | 224 ( 14336) | 1374 ( 87936) | 10240
1 | 224 ( 14336) | 1374 ( 87936) | 10240
2 | 224 ( 14336) | 1374 ( 87936) | 10240
15 | 130 ( 8320) | 1374 ( 87936) | 2048
2 Reported Port(s)
-------------------------------------------------------------------------------For a detailed summary of all systems, users can issue the following command (a long report):
# opareport -V -o comps -d 10
Just before running traffic in the added virtual fabrics, show port status on VirtualFabric1 and VirtualFabric2 (–n mask 0x2 indicates port 1 and -w mask 0x6 indicates VL 1 and VL2):
# opapmaquery -o getportstatus -n 0x2 -w 0x6 | grep -v 0$
…………………………………
VL Number 1
Performance: Transmit
Xmit Data 0 MB (0 Flits)
Performance: Receive
Rcv Data 0 MB (0 Flits)
Performance: Congestion
Performance: Bubbles
Errors: Other
VL Number 2
Performance: Transmit
Xmit Data 0 MB (0 Flits)
Performance: Receive
Rcv Data 0 MB (0 Flits)
Performance: Congestion
Performance: Bubbles
Errors: OtherObserve that all port counters in VirtualFabric1 and VirtualFabric2 are null. Many job schedulers provide an integrated mechanism to launch jobs in a proper virtual fabric. When launching jobs manually, you can use the service level (SL) to identify the virtual fabric.
Find the LIDs assigned to the ports. Two LIDs are found, LID 0x0001 and 0x002:
# opareport -o lids Getting All Node Records... Done Getting All Node Records Done Getting All Link Records Done Getting All Cable Info Records Done Getting All SM Info Records Done Getting vFabric Records LID Summary 2 LID(s) in Fabric: LID(Range) NodeGUID Port Type Name 0x0001 0x00117501017444e0 1 FI knl-sb2 0x0002 0x0011750101790311 1 FI lb0 2 Reported LID(s) -------------------------------------------------------------------------------
Every HFI and switch has port counters to hold error counters and performance counters. Show the port counters for LID 0x0001, port 1:
# opapaquery -o portcounters -l 1 -P 1 | grep -v 0$
In the above command, since the output is a long list and most of the lines show 0 at the end, you may want to output only the lines that do not end in 0 just to see anything not normal.
Similarly, show the port counters for LID 0x0002, port 1:
# opapaquery -o portcounters -l 2 -P 1 | grep -v 0$
Getting Port Counters...
PM controlled Port Counters (total) for LID 0x0002, port number 1:
Performance: Transmit
Xmit Data 3113 MB (389186568 Flits)
Xmit Pkts 944353
MC Xmit Pkts 15
Performance: Receive
Rcv Data 3054 MB (381812586 Flits)
Rcv Pkts 1002110To clear counters on port 1 (-n 0x2) of VL 1 and VL 2 (-w 0x6):
# opapmaquery -o clearportstatus -n 0x2 -w 0x6 Port Select Mask 0x0000000000000002 Counter Sel Mask 0xffffffff
Testing and Verifying
In this section, after setting virtual fabrics, you will run MPI traffic on both VirtualFabric1 and VirtualFabric2 and verify that using the Intel® OPA command lines. You can use the Intel® MPI Benchmarks to generate traffic between two nodes. First, you need to set the secure shell password-less:
# ssh-keygen -t rsa # ssh-copy-id root@192.168.100.101
And disable the firewall for the MPI traffic test:
# systemctl status firewalld # systemctl stop firewalld # systemctl status firewalld
Set up proper environment variables before using Intel® MPI Benchmarks:
# source /opt/intel/parallel_studio_xe_2018.0.033/psxevars.sh intel64 Intel(R) Parallel Studio XE 2018 for Linux* Copyright (C) 2009-2017 Intel Corporation. All rights reserved.
Run the Intel® MPI Benchmarks IMB-MPI1 (benchmarks for MPI-1 functions) on VirtualFabric1 by setting the environment variable HFI_SL=1 (note that the flag –PSM2, Intel® Performance Scaled Messaging 2, was used for the Intel® OPA family of products):
# mpirun -genv HFI_SL=1 -PSM2 -host localhost -n 1 /opt/intel/impi/2018.0.128/bin64/IMB-MPI1 Sendrecv : -host 192.168.100.101 -n 1 /opt/intel/impi/2018.0.128/bin64/IMB-MPI1
#------------------------------------------------------------
# Intel (R) MPI Benchmarks 2018, MPI-1 part
#------------------------------------------------------------
# Date : Tue Nov 28 15:05:11 2017
# Machine : x86_64
# System : Linux
# Release : 3.10.0-327.22.2.el7.x86_64
# Version : #1 SMP Thu Jun 9 10:09:10 EDT 2016
# MPI Version : 3.1
# MPI Thread Environment:
# Calling sequence was:
# /opt/intel/impi/2018.0.128/bin64/IMB-MPI1 Sendrecv
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 2.77 2.77 2.77 0.00
1 1000 2.68 2.68 2.68 0.75
2 1000 2.62 2.62 2.62 1.53
4 1000 2.61 2.61 2.61 3.06
8 1000 2.64 2.64 2.64 6.06
16 1000 4.47 4.47 4.47 7.16
32 1000 4.36 4.36 4.36 14.67
64 1000 4.37 4.37 4.37 29.31
128 1000 4.41 4.41 4.41 58.07
256 1000 4.42 4.42 4.42 115.94
512 1000 4.59 4.59 4.59 223.34
1024 1000 4.76 4.76 4.76 430.62
2048 1000 5.02 5.02 5.02 815.80
4096 1000 5.79 5.79 5.79 1414.85
8192 1000 7.57 7.57 7.57 2164.05
16384 1000 12.25 12.25 12.25 2674.27
32768 1000 16.41 16.41 16.41 3993.64
65536 640 36.22 36.22 36.22 3618.56
131072 320 68.82 68.83 68.82 3808.70
262144 160 80.24 80.31 80.27 6528.68
524288 80 119.05 119.17 119.11 8798.73
1048576 40 202.05 202.25 202.15 10369.08
2097152 20 370.85 371.46 371.15 11291.52
4194304 10 673.70 674.99 674.34 12427.81
# All processes entering MPI_FinalizeNote that the performance numbers presented in this document are for reference purposes only.
On a separate shell, run the Intel® MPI Benchmarks on VirtualFabric2 by setting the environment variable HFI_SL=2:
# mpirun -genv HFI_SL=2 -PSM2 -host localhost -n 1 /opt/intel/impi/2018.0.128/bin64/IMB-MPI1 Sendrecv : -host 192.168.100.101 -n 1 /opt/intel/impi/2018.0.128/bin64/IMB-MPI1
#------------------------------------------------------------
# Intel (R) MPI Benchmarks 2018, MPI-1 part
#------------------------------------------------------------
# Date : Tue Nov 28 15:06:47 2017
# Machine : x86_64
# System : Linux
# Release : 3.10.0-327.22.2.el7.x86_64
# Version : #1 SMP Thu Jun 9 10:09:10 EDT 2016
# MPI Version : 3.1
# MPI Thread Environment:
# Calling sequence was:
# /opt/intel/impi/2018.0.128/bin64/IMB-MPI1 Sendrecv
# Minimum message length in bytes: 0
# Maximum message length in bytes: 4194304
#
# MPI_Datatype : MPI_BYTE
# MPI_Datatype for reductions : MPI_FLOAT
# MPI_Op : MPI_SUM
#
#
# List of Benchmarks to run:
# Sendrecv
#-----------------------------------------------------------------------------
# Benchmarking Sendrecv
# #processes = 2
#-----------------------------------------------------------------------------
#bytes #repetitions t_min[usec] t_max[usec] t_avg[usec] Mbytes/sec
0 1000 2.70 2.70 2.70 0.00
1 1000 2.66 2.66 2.66 0.75
2 1000 2.66 2.66 2.66 1.50
4 1000 3.55 3.55 3.55 2.26
8 1000 2.67 2.67 2.67 6.00
16 1000 4.40 4.40 4.40 7.27
32 1000 4.37 4.37 4.37 14.63
64 1000 4.27 4.27 4.27 29.95
128 1000 4.29 4.29 4.29 59.66
256 1000 4.39 4.39 4.39 116.63
512 1000 4.48 4.48 4.48 228.42
1024 1000 4.63 4.64 4.63 441.85
2048 1000 5.04 5.04 5.04 812.86
4096 1000 5.81 5.81 5.81 1409.28
8192 1000 7.54 7.54 7.54 2172.12
16384 1000 11.93 11.93 11.93 2747.13
32768 1000 16.24 16.25 16.24 4034.19
65536 640 35.99 35.99 35.99 3641.52
131072 320 67.15 67.16 67.15 3903.48
262144 160 77.56 77.62 77.59 6754.15
524288 80 118.23 118.37 118.30 8858.10
1048576 40 206.30 206.63 206.46 10149.53
2097152 20 367.50 368.05 367.77 11396.12
4194304 10 676.80 677.80 677.30 12376.23
# All processes entering MPI_FinalizeDisplay the port status for port 1, VL 1, and VL 2 (shows result only if it is different than 0):
# opapmaquery -o getportstatus -n 0x2 -w 0x6 | grep -v 0$
Port Number 1
VL Select Mask 0x00000006
Performance: Transmit
Xmit Data 2289 MB (286211521 Flits)
Xmit Pkts 746057
Performance: Receive
Rcv Data 2334 MB (291751815 Flits)
Errors: Signal Integrity
Link Qual Indicator 5 (Excellent)
Errors: Security
Errors: Routing and Other Errors
Performance: Congestion
Xmit Wait 10899
Performance: Bubbles
VL Number 1
Performance: Transmit
Xmit Data 736 MB (92044480 Flits)
Xmit Pkts 148719
Performance: Receive
Rcv Data 736 MB (92037226 Flits)
Rcv Pkts 148583
Performance: Congestion
Xmit Wait 6057
Performance: Bubbles
Errors: Other
VL Number 2
Performance: Transmit
Xmit Data 736 MB (92039568 Flits)
Xmit Pkts 147831
Performance: Receive
Rcv Data 736 MB (92038857 Flits)
Rcv Pkts 148803
Performance: Congestion
Xmit Wait 4833
Performance: Bubbles
Errors: OtherVerify that the port counters at VirtualFabric1 and VirtualFabric2 are incrementing by displaying port counters of fabric port for LID 1, port 1, and VirtualFabric1:
# opapaquery -o vfPortCounters -l 1 -P 1 -V VirtualFabric1 | grep -v 0$
Getting Port Counters...
PM Controlled VF Port Counters (total) for node LID 0x0001, port number 1:
VF name: VirtualFabric1
Performance: Transmit
Xmit Data 736 MB (92044480 Flits)
Xmit Pkts 148719
Performance: Receive
Rcv Data 736 MB (92037226 Flits)
Rcv Pkts 148583
Routing and Other Errors:
Congestion:
Xmit Wait 6057
Bubbles:
ImageTime: Tue Nov 21 21:45:53 2017
opapaquery completed: OKSimilarly, for VirtualFabric2:
# opapaquery -o vfPortCounters -l 1 -P 1 -V VirtualFabric2 | grep -v 0$
Getting Port Counters...
PM Controlled VF Port Counters (total) for node LID 0x0001, port number 1:
VF name: VirtualFabric2
Performance: Transmit
Xmit Data 736 MB (92039568 Flits)
Xmit Pkts 147831
Performance: Receive
Rcv Data 736 MB (92038857 Flits)
Rcv Pkts 148803
Routing and Other Errors:
Congestion:
Xmit Wait 4833
Bubbles:
ImageTime: Tue Nov 21 21:50:23 2017
opapaquery completed: OKView the data counters per virtual lane (shows only if different than 0):
# opapmaquery -o getdatacounters -n 0x2 -w 0x6 | grep -v 0$
Port Select Mask 0x0000000000000002
VL Select Mask 0x00000006
Port Number 1
Xmit Data 2226 MB (278332959 Flits)
Rcv Data 2236 MB (279538592 Flits)
Rcv Pkts 506372
Signal Integrity Errors:
Link Qual. Indicator 5 (Excellent)
Congestion:
Xmit Wait 10899
Bubbles:
VL Number 1
Xmit Data 736 MB (92044480 Flits)
Rcv Data 736 MB (92037226 Flits)
Xmit Pkts 148719
Rcv Pkts 148583
Xmit Wait 6057
VL Number 2
Xmit Data 736 MB (92039568 Flits)
Rcv Data 736 MB (92038857 Flits)
Xmit Pkts 147831
Rcv Pkts 148803
Xmit Wait 4833Finally, view the error counters for 16 virtual lanes; this shows errors if different than 0:
# opapmaquery -o geterrorcounters -n 0x2 -w 0xffff | grep -v 0$
Port Select Mask 0x0000000000000002
VL Select Mask 0x0000ffff
Port Number 1
Signal Integrity Errors:
Security Errors:
Routing and Other Errors:
VL Number 1
VL Number 2
VL Number 3
VL Number 4
VL Number 5
VL Number 6
VL Number 7
VL Number 8
VL Number 9
VL Number 11
VL Number 12
VL Number 13
VL Number 14
VL Number 15Conclusion
Configuring vFabrics in Intel® OPA requires users to manually edit the configuration file. This document shows how to set up two virtual fabrics by editing the opafm.xml file. The necessary command lines are shown to set up, to verify the newly created virtual fabrics, and to clear counters. The Intel® MPI Benchmarks are used to run traffic in each virtual fabric. Verification of traffic and errors in virtual fabric can be observed by looking at the counters.