From 7a1cf1acb5535c02c9532072a386e8b6517b9724 Mon Sep 17 00:00:00 2001
From: stansmith
The Windows OpenFabrics (WinOF) package is composed of software modules intended
@@ -30,10 +30,8 @@ OpenFabrics Infiniband core drivers and Upper Level Protocols (ULPs): HCA (Host Channel Adapter) driver
MTHCA -
- Mellanox
@@ -44,7 +42,7 @@ style='background-position: 0% 0%; mso-highlight:yellow; background-image:none;
Infiniband Core modules: IB verbs and IB access layer Upper Layer Protocols: IPoIB, WSD, VNIC, SRP Initiator and uDAPL Upper Layer Protocols: IPoIB, WSD, Network Direct, VNIC, SRP Initiator and uDAPL
OpenFabrics utilities: Windows OpenFabrics
User's Manual
-Release 1.1
+Release 2.0
-04/28/2008
+10/22/2008
Overview
@@ -66,17 +64,18 @@ style='background-position: 0% 0%; mso-highlight:yellow; background-image:none;
WinOF Features
IPoIB - Internet Protocols over InfiniBand
+
- IPoIB - Internet Protocols over InfiniBand
Winsock Direct Service Provider
+
- Winsock Direct Service Provider
+ Network Direct
+ Service Provider
DAT and uDAPL
@@ -97,6 +96,8 @@ style='background-position: 0% 0%; mso-highlight:yellow; background-image:none;
InfiniBand Software Development Kit
WinVerbs
opensm Open Subnet Management - configure and manage an InfiniBand subnet
osmtest - Subnet management tests
osmtest Subnet management tests
ib_trapgen Generate Infiniband Subnet +
ib_trapgen Generate Infiniband Subnet Management Traps for testing purposes
ib_send_lat Infiniband send +
ib_send_lat Infiniband send latency measurement
ib_send_bw Infiniband send bandwidth +
ib_send_bw Infiniband send bandwidth measurement
ib_write_lat Infiniband RDMA write @@ -150,11 +156,10 @@ TCP performance measurements
cmtest Connection Manager tests
PrintIP Display +
PrintIP Display an Internet Protocol address associated with an IB GUID.
vstat
-Display HCA attributes (lids), statistics and error counters.
vstat Display HCA attributes (lids), statistics and error counters.
+
The part_man.exe +application allows creating, deleting and viewing existing +host partitions.
+Usage : part_man.exe <show|add|rem> <port_guid> <pkey1 pkey2 +...>
+ + show - + + shows existing partitions+ Expected results after execution + part_man.exe show
++ 1. + Output has a format
++ port_guid1 pkey1 pkey2 + pkey3 pkey4 pkey5 pkey6 pkey7 pkey8
++ port_guid2 + pkey1 pkey2 pkey3 pkey4 pkey5 pkey6 pkey7 pkey8
++ where + port_guid is a port guid in + hexadecimal format, pkey + values of partition key (in hex format) for this port.
++ Default + partition key (0xFFFF) is not shown and can not be created by the + part_man.exe.
++
++ + add - create new partition(s) + on specified port
++ + port_guid + add <port_guid> + <pkey1> <pkey2> +
+creates new partition(s) on port specified by port_guid parameter (in + hexadecimal format) and pkey new partition key value in hexadecimal + format (e.g. 0xABCD or ABCD).
++ Port + guid is taken form vstat output and has a following format:
++ + XXXX:XXXX:XXXX:XXXX.
++ Vstat + prints node guid, so user has to add 1 to node guid value to obtain port + guid. For example, if node guid is + 0008:f104:0397:7ccc, port guid will be
++ + 0008:f104:0397:7ccd + for the first port,
++ + 0008:f104:0397:7cce + for the second port.
++ +
++ Expected + results of execution part_man.exe add + 0x0D99:9703:04f1:0800 0xABCD
++ 1. + part_man.exe output ends up with + Done message.
+
+ 2.
+ A new instance of a Network Adapter named OpenFabrics IPoIB
+ Adapter Partition will appear in Device manager
+ window.
+ If the new adapter appears with yellow label, manual device driver installation is
+ required.
In + the device manager view, right click OpenFabrics IPoIB Adapter Partition
++ select 'update driver' and follow the instructions.
don't + allow Windows Update to search the Internet
+ select install software automatically.
+ 3. + New adapter name ends with Partition, e.g. OpenFabrics IPoIB + Adapter Partition.
++
+ + + rem + + removes partition key on specified + port. ++ + part_man.exe rem <port_guid> + <pkey1> <pkey2>
++ + Port_guid in hexadecimal format (same as for + add command), identifies port for + operation.
++ Expected + results after execution part_man rem <port_guid> + <pkey>
++ 1. + Application prints Done + message.
++ 2. + In device manager window IPoIB network adapter will disappear.
++ 3. + Execution of part_man.exe + show will not show removed adapter.
++ +
@@ -903,6 +1045,12 @@ Area Connection X->Properties->(General Tab)Internet Protocol(TCP/IP)-> Infiniband HCA, the Local Area Connection will become enabled. If you discover the Local Area Connection to be disabled, then likely your subnet manager (opensm) is not running or functioning correctly. +IPoIB Partition Management
++
- +
+part_man + Manage (add/remove/show) IPoIB partitions.
<return-to-top>
@@ -1090,6 +1238,35 @@ Manual control is performed via the \Program Files\WinOF\installsp.exe utility.<
+
ND service
+is automatically installed and started as part of the 'default' installation for
+Windows server 2008, Vista or HPC systems.
+Manual control is performed via the %windir%\system32\ndinstall.exe utility.
++usage: ndinstall [-i | -r | -l]
+
+
+ -i Install (enable) the Network Direct (ND) service provider
+ -r Remove the ND service provider
+ -r <name> Remove the specified service provider
+ -l List service providers
The Microsoft Network Direct SDK can be downloaded from
+
+here. Once the ND SDK is installed, ND test programs can be located @
+%ProgramFiles%\Microsoft HPC Pack 2008 SDK\NetworkDirect\Bin\amd64\ as nd*.exe.
Known working ND test command invocations (loopback or remote +host)
+++svr: ndrpingpong s IPoIB_IPv4_addr 4096 p1
+
+ cli: ndrpingpong c IPoIB_IPv4_addr 4096 p1svr: ndpingpong s IPoIB_IPv4_addr 4096 b1
+
+ cli: ndpingpong c IPoIB_IPv4_addr 4096 b1
See ndping.exe /? for details.
+
-
+
+
WinVerbs is a userspace verbs and communication management interface optimized
for the Windows operating system. Its lower interface is designed to support
any RDMA based device, including Infiniband and iWarp. Its upper interface is
capable of providing a low latency verbs interface, plus supports Microsoft's
Network Direct Interface, DAPL, and OFED libibverbs interfaces. It consists of
a userspace library and a kernel filter driver.
+
The WinVerbs driver loads as an upper filter driver for Infiniband controllers.
(Open source iWarp drivers for Windows are not yet available.) A corresponding
userspace library installs as part of the Winverbs driver installation package.
Additionally, a Windows port of the OFED libibverbs library and several test
programs are also included.
+
A full list of available libibverbs test programs and their usage are given
below. Note that not all listed options apply to all applications
+
ibv_devinfo
-d <dev> use IB device <dev> (default first device found)
-I <port> use port <port> of IB device (default all ports)
-l print only the IB devices names
-v print all the attributes of the IB device(s)
+
ibv_rc_pingpong, ibv_uc_pingpong, ibv_ud_pingpong
no args start a server and wait for connection
-h <host> connect to server at <host>
-p <port> listen on/connect to port <port> (default 18515)
-d <dev> use IB device <dev> (default first device found)
-i <port> use port <port> of IB device (default 1)
-s <size> size of message to exchange (default 4096)
-m <size> path MTU (default 1024)
-r <dep> number of receives to post at a time (default 500)
-n <iters> number of exchanges (default 1000)
-l <sl> service level value
-e sleep on CQ events (default poll)
+
ibv_send_bw, ibv_send_lat
ibv_read_bw, ibv_read_lat
ibv_write_bw, ibv_write_lat
no args start a server and wait for connection
-h <host> connect to server at <host>
-p <port> listen on/connect to port <port> (default 18515)
-d <dev> use IB device <dev> (default first device found)
-i <port> use port <port> of IB device (default 1)
-c <RC/UC/UD> connection type RC/UC/UD (default RC)
-m <mtu> mtu size (256 - 4096. default for hermon is 2048)
-s <size> size of message to exchange (default 65536)
-a Run sizes from 2 till 2^23
-t <dep> size of tx queue (default 300)
-g send messages to multicast group (UD only)
-r <dep> make rx queue bigger than tx (default 600)
-n <iters> number of exchanges (at least 2, default 1000)
-I <size> max size of message to be sent in inline mode (default 400)
-b measure bidirectional bandwidth (default unidirectional)
-V display version number
-e sleep on CQ events (default poll)
-N cancel peak-bw calculation (default with peak-bw)
+
To verify correct WinVerbs and libibverbs installation, run ibv_devinfo. It
should report all RDMA devices in the system, along with limited port
attributes. Because of limitations in the WinOF stack, it is normal for it to
list several values as unknown.