From: stansmith Date: Tue, 7 Aug 2007 16:03:17 +0000 (+0000) Subject: [WinOF] keep only this version of the openib-windows/WinOF manual.htm file. WIX versi... X-Git-Url: https://openfabrics.org/gitweb/?a=commitdiff_plain;h=ba902ad3a834afc4b4568f359e1ac2bf88f8f31c;p=~shefty%2Frdma-win.git [WinOF] keep only this version of the openib-windows/WinOF manual.htm file. WIX version has been removed. git-svn-id: svn://openib.tc.cornell.edu/gen1@754 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86 --- diff --git a/trunk/docs/Manual.htm b/trunk/docs/Manual.htm index d56f13f2..6e3279f9 100644 --- a/trunk/docs/Manual.htm +++ b/trunk/docs/Manual.htm @@ -1,23 +1,43 @@ + + + + +

 

-

OpenFabrics Windows

+

Windows OpenFabrics

User's Manual

Release 1.0

-

3-20-07

+

+08/01/2007

Overview

-

This is the OpenFabrics Windows software package -supporting InfiniBand fabrics. It is composed of several software modules -intended for use on a computer cluster constructed as an InfiniBand fabric.

-

The OpenFabrics Windows (OFW) software package contains the +

+The Windows OpenFabrics (WinOF) package is composed of software modules intended +for use on Microsoft Windows based computer systems connected via an InfiniBand +fabric.

+

The Windows OpenFabrics software package contains the following:

-OpenFabrics core and ULPs:

+OpenFabrics Infiniband core drivers and Upper Level Protocols (ULPs):

@@ -43,54 +63,59 @@ OpenFabrics core and ULPs:

Tools

-

The OpenFabrics Alliance Windows release contains a set of +

+

The OpenFabrics Alliance Windows release contains a set of user mode tools which are designed to faciliate the smooth operation of an - OpenFabrics Windows installation.

-

Infiniband Subnet Management

-
    -
  • -

    opensm        Open Subnet - Management - configure a subnet

    • -
  • -

    osmtest         + Windows OpenFabrics installation. These tools are available from a command + window (cmd.exe) as the installation path '%SystemDrive%\Program + Files\WinOF' is appended to the system wide search path registry entry. + A start menu short-cut 'WinOF Cmd Window' is provided to faciliate + correction tool operation.

    +

    Infiniband Subnet Management

    +
      +
    • +

      opensm        Open Subnet + Management - configure and manage an InfiniBand subnet

      • +
    • +

      osmtest         Subnet management tests

      • -
    • -

      ib_trapgen    Generate Infiniband Subnet +

    • +

      ib_trapgen    Generate Infiniband Subnet Management Traps for testing purposes

      • -
    -

    Performance

    - +

    Performance

    +
      +
    • +

      ib_send_lat      Infiniband send latency measurement

      • -
    • -

      ib_send_bw     Infiniband send bandwidth +

    • +

      ib_send_bw     Infiniband send bandwidth measurement

      • -
    • -

      ib_write_lat     Infiniband RDMA write +

    • +

      ib_write_lat     Infiniband RDMA write latency measurement

      • -
    • -

      ib_write_bw    Infiniband RDMA write bandwidth +

    • +

      ib_write_bw    Infiniband RDMA write bandwidth measurement

      • -
    • -

      ttcp                 +

    • +

      ttcp                 TCP performance measurements

      • -
    -

    Diagnostics

    - +

    Diagnostics

    +
      +
    • +

      ib_limits         Infiniband verb tests

      • -
    • -

      cmtest           Connection Manager tests

      • -
    • -

      PrintIP           Display +

    • +

      cmtest           Connection Manager tests

      • +
    • +

      PrintIP           Display an Internet Protocol address associated with an IB GUID.

      • -
    • -

      vstat               -Display HCA attributes, statistics and error counters.
      - 

      • -
    +
  • +

    vstat               +Display HCA attributes, statistics and error counters.
     

    • +
+

 

 

-

 

-

User mode verbs micro-benchmarks
- 

-

These micro-benchmarks tests are intended as a useful benchmark for HW or SW +

User mode micro-benchmarks

+
+

The following user-mode test programs are intended as useful +micro-benchmarks for HW or SW tuning and/or functional testing.

-

- Tests use CPU cycle counter to get time stamps without - context switch.
Some CPU architectures do NOT have such capability. e.g. Intel 80486.
-
- measures round-trip time but reports half of that as one-way latency.
ie. May not be sufficiently accurate for - asymmetrical configurations.
-
- Min/Median/Max result is reported.
The median (vs. average) is less sensitive to extreme scores.
Typically the "Max" value is the first value measured.
-
- larger samples only marginally help. The default (1000) is pretty good.
Note that an array of cycles_t (typically unsigned long) is allocated
once to collect samples and again to store the difference between them.
Really big sample sizes (e.g. 1 million) might expose other problems
with the program.
-
- "-H" option will dump the histogram for additional statistical analysis.
See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other +

Tests use CPU cycle counters to get time stamps without + context switch.
+
Tests measure round-trip time but report half of that as one-way latency
+ (i.e.. May not be sufficiently accurate for + asymmetrical configurations).
+
Min/Median/Max result is reported.
The median (vs. average) is less sensitive to extreme scores.
Typically the "Max" value is the first value measured.
+
larger samples only marginally help. The default (1000) is pretty good.
Note that an array of cycles_t (typically unsigned long) is allocated
once to collect samples and again to store the difference between them.
Really big sample sizes (e.g. 1 million) might expose other problems
with the program.
+
"-H" option will dump the histogram for additional statistical analysis.
See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other
statistical math programs.

Architectures tested: x86, x86_64, ia64

ib_send_lat.exe      - latency test with @@ -234,7 +269,13 @@ transactions

<return-to-top>

+


+ 

ttcp - Test TCP performance

+

TTCP accesses the Windows socket layer, hence it does not access +IB verbs directly. IPoIB or WSD layers are invoked beneath the socket layer +depending on configuration. TTCP is included as a quick baseline performance +check.

Usage: ttcp -t [-options] host 
        ttcp -r [-options]
@@ -262,7 +303,9 @@ Options specific to -r:

<return-to-top>

 

-

Diagnostics

+

 

+

Diagnostics

+

 

ib_limits - Infiniband verbs tests

Usage: ib_limits [options]

@@ -328,26 +371,39 @@ Options specific to -r: counters

<return-to-top>

-

Install Infiniband Service Provider

-
-

usage: installsp [-i/-r [-p]]
-
- -i Install the IB service provider
- -r Remove the OpenIB service provider
- -r <name> Remove the specified service provider
- -l List service providers
- 

-
-

<return-to-top>

 

-

Subnet Management with OpenSM Rev: openib-1.2.0

+

Subnet Management with OpenSM Rev: openib-1.2.0

+

A single running process (opensm.exe) is required to configure -and thus make an Infiniband subnet useable.  For most cases, invoking -opensm.exe with no arguments is sufficient to correctly configure most -Infiniband fabrics.

+and thus make an Infiniband subnet useable.  For most cases, InfiniBand +Subnet Management as a Windows service is sufficient to correctly configure most +InfiniBand fabrics.

The Infiniband subnet management process (opensm) may exist on a -Windows node or a Linux (OFED) node but not both!

-

Usage: opensm [options]

+Windows (WinOF) node or a Linux (OFED) node.
+
+Limit the number of OpenSM processes per IB fabric; one SM is sufficient +although redundant SMs are supported. You do not need a Subnet Manager per +node/system.

+

OpenIB Subnet Management as a Windows Service

+

InfiniBand subnet management (OpenSM), as a Windows service, is installed by default, although it is NOT +started by default. There are two ways to enable the InfiniBand Subnet +Management service.

+
    +
  1. Reset the installed OpenSM service "InfiniBand Subnet Management" to + start automatically; See
    + My Computer->Manage->Services and Applications->Services->InfiniBand + subnet Management->Start.
     
    2. +
  2. Install OpenSM as a 'running' Windows service:
    + Request a 'Custom' install, selecting the OpenSM_service install feature. + Once the installation has completed, check the running InfiniBand Subnet + Management service status via the Windows service manager (see #1).
    + 
    3. +
  3. Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what + OpenSM thinks is happening.
    4. +
+

 

+

Manual InfiniBand Subnet Management from a command window

+

Usage: opensm.exe [options]

Options:

-c
@@ -545,7 +601,8 @@ Windows node or a Linux (OFED) node but not both!

<return-to-top>

 

Osmtest - Subnet Management Tests

-

Invoke open subnet management tests.

+

Invoke open subnet management tests. osmtest currently can not +run on the same HCA port which OpenSM is currently using.

 Usage: osmtest [options]

Options:

@@ -801,7 +858,8 @@ Options: one of the following optional flows:

<return-to-top>

 

 

-

IPoIB - Internet Protocols over InfiniBand

+

IPoIB - Internet Protocols over InfiniBand

+

IPoIB enables the use of Internet Protocol utilities (e.g., ftp, telnet) to function correctly over an Infiniband fabric. IPoIB is implemented as an NDIS Miniport driver with a WDM lower edge.

@@ -822,7 +880,8 @@ the Local Area Connection to be disabled, then likely your subnet manager

<return-to-top>

 

 

-

Winsock Direct Service Provider

+

Winsock Direct Service Provider

+

Winsock Direct (WSD) is Microsoft's proprietary protocol that predates SDP (Sockets Direct Protocol) for accelerating TCP/IP applications by using RDMA hardware. Microsoft had a significant role in defining the SDP @@ -835,14 +894,21 @@ Windows Server 2003, and is responsible for routing socket traffic over either the regular TCP/IP stack, or offload it to a WSD provider. The WSD provider is a hardware specific DLL that implements connection management and data transfers over particular RDMA hardware.

+

WinOF WSD is not supported in the Windows XP environment.

The WSD Protocol seamlessly transports TCP data using Infiniband data packets in 'buffered' mode or Infiniband RDMA in 'direct' mode. Either way the user mode socket application sees no - difference in the standard Inet socket which it created other than + behavioral difference in the standard Internet Protocol socket it created other than reduced data transfer times and increased bandwidth.

-The OpenFabrics Windows release includes a WSD provider library that has been -extensively tested with Microsoft Windows Server 2003.

+The Windows OpenFabrics release includes a WSD provider library that has been +extensively tested with Microsoft Windows Server 2003.
+During testing, bugs where found in the WSD switch that could lead to hangs, +crashes, data corruption, and other unwanted behavior. Microsoft released a +hotfix to address these issues which should be installed if using WSD; the +Microsoft Windows Server 2003 hotfix can be found +here.
+ 

@@ -907,7 +973,7 @@ extensively tested with Microsoft Windows Server 2003.

- + @@ -980,26 +1046,222 @@ extensively tested with Microsoft Windows Server 2003.

See https://wiki.openfabrics.org/tiki-index.php?page=Winsock+Direct for the latest WSD status.

-

 

-

installsp.exe - Installs the Winsock direct service provider - for Infiniband.

+

Winsock Direct Service Provider Installation

+

When the custom install option 'WSD' is selected the WSD service +is automatically installed and started as part of the installation.
+Manual control is performed via the \Program Files\WinOF\installsp.exe utility.

+
+

usage: installsp [-i | -r | -l]
+
+ -i    Install the Winsock Direct (WSD) service provider
+ -r    Remove the WSD service provider
+ -r <name>    Remove the specified service provider
+ -l    List service providers
+ 

+

<return-to-top>

 

 

-

DAT (Direct Access Transport) Library and uDAPL (usermode Direct Access Programing -Library)

-

DAT and uDAPL are based on the 1.1 DAT specification. The DAPL -(Direct Access Provider Library) provider now fully supports Infiniband RDMA and +

Direct Access Transport and usermode Direct Access Programming +Libraries

+
+

The DAT (Direct Access Transport) +API is a C programming interface developed by the +DAT Collaborative in +order provide a set of transport-independent, platform-independent Application +Programming Interfaces that exploit the RDMA (remote direct memory access) +capabilities of next-generation interconnect technologies such as InfiniBand, +and iWARP.

+

WinOF DAT and uDAPL are based on the 1.1 DAT specification. The DAPL +(Direct Access Provider Library) now fully supports Infiniband RDMA and IPoIB.

+
+
+
0x00000040CMCM (Connection Manager)
0x00000080
+ + + +
+
+
+
+ How  DAT objects map to equivalent + + InfiniBand objects:
+  + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Interface Adapter (IA) + HCA (Host Channel Adapter)
Protection Zone (PZ) PD (Protection Domain)
Local Memory Region (LMR) + MR (Memory Region)
Remote Memory Region (RMR) + MW (Memory Windows)
Event Dispatcher (EVD) + CQ (Completion Queue)
Endpoint (EP) QP (Queue Pair)
Public Service Point (PSP) + connection identifier
Reserved Service Point (RSP) + connection identifier
Connection Request (CR) + connection manager event +
+
+
+
+
+
+

 

- 
EXECUTION ENVIRONMENT:
+

DAT EXECUTION ENVIRONMENT:

-

In order for DAT/uDAPL programs to execute correctly, the - dat.dll file must be present in the current directory, windows\system32\ or - in the library search path.

-
-
NAME
+	
In order for DAT/uDAPL 
+	programs to execute correctly, the 'dat.dll' file must be present in the 
+	current directory,
+	%SystemRoot%\system32, %SystemRoot% or in the library search path.

+	
The default WinOF 
+	installation places the file dat.dll in the '%SystemRoot%' folder.

+	
The DAPL configuration 
+	file by default is defined as '%SystemDrive%\DAT\dat.conf'. This default 
+	specification can be overriden by use of the environment variable 
+	DAT_OVERRIDE; see following environment variable discussion.

+	
Within the dat.conf file, 
+	the DAPL library specification can be located as the 5th whitespace 
+	separated line argument. By default the DAPL library file is installed as

+	'%SystemRoot%\dapl.dll'.

+	
Should you choose to 
+	relocated the DAPL library file to a path where whitespace appears in the 
+	full library path specification, then the full library file specification 
+	must be contained within double-quotes. A side effect of the double-quotes 
+	is the library specification is treated as a Windows string which implies 
+	the '\' (backslash character) is treated as an 'escape' character.  Hence 
+	all backslashes in the library path must be duplicated when enclosed in 
+	double-quotes
+	(e.g., "C:\\Programs Files\\WinOF\\dapl.dll").

+	
A sample dat.conf file is 
+	installed as '\Program Files\WinOF\dat.conf '.

+	After the WinOF installation, move the \Program Files\WinOF\dat.conf file to 
+	the default DAT configuration file location
+	'%SystemDrive%\DAT\dat.conf'.

+	
In order to preserve existing installations, 
+	the dat.conf file is not automatically installed in it's default location.

+	
 

+	

+DAT library environment variables:

+ 

+	
+DAT_OVERRIDE
+------------
+Value used as the static registry configuration file, overriding the
+default location, 'C:\DAT\dat.conf'.
+
+Example: set DAT_OVERRIDE=%SystemDrive%\path\to\my\private.conf
+
+
+DAT_DBG_TYPE
+------------
+
+Value specifies which parts of the registry will print debugging
+information, valid values are 
+
+DAT_OS_DBG_TYPE_ERROR = 0x1
+DAT_OS_DBG_TYPE_GENERIC = 0x2
+DAT_OS_DBG_TYPE_SR = 0x4
+DAT_OS_DBG_TYPE_DR = 0x8
+DAT_OS_DBG_TYPE_PROVIDER_API = 0x10
+DAT_OS_DBG_TYPE_CONSUMER_API = 0x20
+DAT_OS_DBG_TYPE_ALL = 0xff
+
+or any combination of these. For example you can use 0xC to get both 
+static and dynamic registry output.
+
+Example set DAT_DBG_TYPE=0xC
+
+DAT_DBG_DEST
+------------ 
+
+Value sets the output destination, valid values are 
+
+DAT_OS_DBG_DEST_STDOUT = 0x1
+DAT_OS_DBG_DEST_SYSLOG = 0x2 
+DAT_OS_DBG_DEST_ALL = 0x3 
+
+For example, 0x3 will output to both stdout and the syslog. 
+

+	

+DAPL Provider library environment variables

+	

+

+DAPL_DBG_TYPE

+-------------

+

+Value specifies which parts of the registry will print 
+debugging information, valid values are 

+

+DAPL_DBG_TYPE_ERR = 0x0001

+DAPL_DBG_TYPE_WARN = 0x0002

+DAPL_DBG_TYPE_EVD = 0x0004

+DAPL_DBG_TYPE_CM = 0x0008

+DAPL_DBG_TYPE_EP = 0x0010

+DAPL_DBG_TYPE_UTIL = 0x0020

+DAPL_DBG_TYPE_CALLBACK = 0x0040

+DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080

+DAPL_DBG_TYPE_API = 0x0100

+DAPL_DBG_TYPE_RTN = 0x0200

+DAPL_DBG_TYPE_EXCEPTION = 0x0400

+

+or any combination of these. For example you can use 0xC to get both 

+EVD and CM output.

+

+Example set DAPL_DBG_TYPE=0xC

+

+

+DAPL_DBG_DEST

+-------------

+

+Value sets the output destination, valid values are 

+

+DAPL_DBG_DEST_STDOUT = 0x1

+DAPL_DBG_DEST_SYSLOG = 0x2 

+DAPL_DBG_DEST_ALL = 0x3 

+

+For example, 0x3 will output to both stdout and the syslog. 

+

+

+
+

+
+

DAPLTEST

+
 
     dapltest - test for the Direct Access Provider Library (DAPL)
 
@@ -1244,23 +1506,23 @@ USAGE - Limit test client
                         Default: run all tests
 
 
-EXAMPLES
+EXAMPLES
 
     dapltest -T S -d -D ibnic0
 
-                        Starts a server process with debug verbosity.
+                        Starts a local dapltest server process with debug verbosity.
+                        Server loops (listen for dapltest request, process request).
     
-    dapltest -T T -d -s winIB -D ibnic0 -i 100 \
-    client SR 4096 2 server SR 4096 2
+    dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2
 
                         Runs a transaction test, with both sides
                         sending one buffer with two 4KB segments,
                         one hundred times; dapltest server is on host winIB.
 
-    dapltest -T P -d -s winIB -D JniIbdd0 -i 100 SR 4096 2
+    dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2
 
                         Runs a performance test, with the client 
-                        sending one buffer with two 4KB segments,
+                        RDMA writing one buffer with two 4KB segments,
                         one hundred times.
 
     dapltest -T Q -s winIB -D ibnic0
@@ -1275,9 +1537,9 @@ EXAMPLES
                         when trying to exhaust resources.
 
     dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \
-    client RW  4096 1    server RW  2048 4    \
-    client SR  1024 4    server SR  4096 2    \
-    client SR  1024 3 -f server SR  2048 1 -f
+       client RW  4096 1    server RW  2048 4    \
+       client SR  1024 4    server SR  4096 2    \
+       client SR  1024 3 -f server SR  2048 1 -f
 
                         Runs a more complicated transaction test,
                         with two thread using four EPs each,
@@ -1292,3 +1554,55 @@ BUGS  (and  To Do List)
 
     Further limit tests could be added.

<return-to-top>

+

 

+

 

+

SRP - SCSI RDMA Protocol

+

+The +SCSI RDMA +Protocol  (SRP) is an emerging industry standard protocol for utilizing +block storage devices over an InfiniBand™ fabric. SRP is being defined in the +ANSI T-10 committee.

+

WinOF SRP is a storage +driver implementation that enables the SCSI RDMA protocol over an InfiniBand +fabric.
+The implementation conforms +to the T10 Working Group draft + +http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf.

+

Software Dependencies

+

The SRP driver depends on the installation of the WinOF stack +with a Subnet
+Manager running somewhere on the IB fabric.
+
+- Supported Operating Systems and Service Packs:
+   o Windows XP SP2 x64
+   o Windows Server 2003 SP1 (x86, x64)
+   o Windows Server 2003 CCS (x64)

+


+Testing Level

+

The SRP driver has undergone basic testing against Mellanox +Technologies'
+SRP Targets MTD1000 and MTD2000. Testing included format, read, and write
+operations.
+ 

+

Installation

+

The WinOF stack  does not install the SRP driver by default. +If SRP is +selected in the custom installation window, it will only be copied during WinOF +installation. To complete the SRP driver installation, an SRP target must be +detected; a Subnet Manager must be running somewhere in the IB subnet.
+
+Upon the detection of an SRP target, the "New Hardware Found" Wizard pops up.
+- Select Install Automatically and click Next. This installs the I/O unit +device.
+
+Once completed, the "New Hardware Found" Wizard pops up again.
+- Select Install Automatically and click Next. This installs the SRP driver.
+ 

+

New Features and Changes

+

- SRP supports WPP tracing tools by using the GUID: +'5AF07B3C-D119-4233-9C81-C07EF481CBE6'.
+  The flags and level of debug can be controlled at load-time or run-time.
+ 

+

<return-to-top>