From: stansmith Date: Tue, 7 Aug 2007 16:07:37 +0000 (+0000) Subject: [WinOF] house-cleaning, use the version in trunk\docs. X-Git-Url: https://openfabrics.org/gitweb/?a=commitdiff_plain;h=156d1f76bf0133e1c2dccae4c8219d620dc19ed9;p=~shefty%2Frdma-win.git [WinOF] house-cleaning, use the version in trunk\docs. git-svn-id: svn://openib.tc.cornell.edu/gen1@756 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86 --- diff --git a/branches/WinOF/WIX/Manual.htm b/branches/WinOF/WIX/Manual.htm deleted file mode 100644 index 6e3279f9..00000000 --- a/branches/WinOF/WIX/Manual.htm +++ /dev/null @@ -1,1608 +0,0 @@ - - - - - -

-  -

-

Windows OpenFabrics

-

User's Manual

-

Release 1.0

-

-08/01/2007

-

Overview

-

-The Windows OpenFabrics (WinOF) package is composed of software modules intended -for use on Microsoft Windows based computer systems connected via an InfiniBand -fabric.

-

The Windows OpenFabrics software package contains the -following:
-
-OpenFabrics Infiniband core drivers and Upper Level Protocols (ULPs):

- -

OpenFabrics utilities:

- -

Documentation

- -

Features

- -
-
-

The OpenFabrics Alliance Windows release contains a set of - user mode tools which are designed to faciliate the smooth operation of an - Windows OpenFabrics installation. These tools are available from a command - window (cmd.exe) as the installation path '%SystemDrive%\Program - Files\WinOF' is appended to the system wide search path registry entry. - A start menu short-cut 'WinOF Cmd Window' is provided to faciliate - correction tool operation.

-

Infiniband Subnet Management

-
    -
  • -

    opensm        Open Subnet - Management - configure and manage an InfiniBand subnet

    • -
  • -

    osmtest         - Subnet management tests

    • -
  • -

    ib_trapgen    Generate Infiniband Subnet - Management Traps for testing purposes

    • -
-

Performance

-
    -
  • -

    ib_send_lat      Infiniband send -latency measurement

    • -
  • -

    ib_send_bw     Infiniband send bandwidth - measurement

    • -
  • -

    ib_write_lat     Infiniband RDMA write -latency measurement

    • -
  • -

    ib_write_bw    Infiniband RDMA write bandwidth -measurement

    • -
  • -

    ttcp                 -TCP performance measurements

    • -
-

Diagnostics

-
    -
  • -

    ib_limits         - Infiniband verb tests

    • -
  • -

    cmtest           Connection Manager tests

    • -
  • -

    PrintIP           Display -an Internet Protocol address associated with an IB GUID.

    • -
  • -

    vstat               -Display HCA attributes, statistics and error counters.
     

    • -
-
-
- -

 

-

 

-

 

-

User mode micro-benchmarks

-
-

The following user-mode test programs are intended as useful -micro-benchmarks for HW or SW -tuning and/or functional testing.

-
-

Tests use CPU cycle counters to get time stamps without - context switch.
-
Tests measure round-trip time but report half of that as one-way latency
- (i.e.. May not be sufficiently accurate for - asymmetrical configurations).
-
Min/Median/Max result is reported.
The median (vs. average) is less sensitive to extreme scores.
Typically the "Max" value is the first value measured.
-
larger samples only marginally help. The default (1000) is pretty good.
Note that an array of cycles_t (typically unsigned long) is allocated
once to collect samples and again to store the difference between them.
Really big sample sizes (e.g. 1 million) might expose other problems
with the program.
-
"-H" option will dump the histogram for additional statistical analysis.
See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other -
statistical math programs.

Architectures tested: x86, x86_64, ia64

-
-

ib_send_lat.exe      - latency test with -send transactions

-
-

Usage:

-
-

ib_send_lat start a server and wait for connection
- ib_send_lat <host> connect to server at <host>

-
-

Options:

-
-

-p, --port=<port> listen on/connect to port <port> - (default 18515)
- -c, --connection=<RC/UC> connection type RC/UC (default RC)
- -m, --mtu=<mtu> mtu size (default 2048)
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)
- -i, --ib-port=<port> use port <port> of IB device (default 1)
- -s, --size=<size> size of message to exchange (default 1)
- -t, --tx-depth=<dep> size of tx queue (default 50)
- -l, --signal signal completion on each msg
- -a, --all Run sizes from 2 till 2^23
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)
- -C, --report-cycles report times in cpu cycle units (default - microseconds)
- -H, --report-histogram print out all results (default print summary - only)
- -U, --report-unsorted (implies -H) print out unsorted results (default - sorted)
- -V, --version display version number
- -e, --events sleep on CQ events (default poll)

-
-
-

ib_send_bw.exe     - BW (BandWidth) test with send transactions

-
-

Usage:

-
-

ib_send_bw start a server and wait for connection
- ib_send_bw <host> connect to server at <host>

-
-

Options:

-
-

-p, --port=<port> listen on/connect to port <port> - (default 18515)
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)
- -i, --ib-port=<port> use port <port> of IB device (default 1)
- -c, --connection=<RC/UC> connection type RC/UC/UD (default RC)
- -m, --mtu=<mtu> mtu size (default 1024)
- -s, --size=<size> size of message to exchange (default 65536)
- -a, --all Run sizes from 2 till 2^23
- -t, --tx-depth=<dep> size of tx queue (default 300)
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)
- -b, --bidirectional measure bidirectional bandwidth (default - unidirectional)
- -V, --version display version number
- -e, --events sleep on CQ events (default poll)

-
-
-

ib_write_lat.exe      - latency test with RDMA write -transactions

-
-

Usage:

-
-

ib_write_lat start a server and wait for connection
- ib_write_lat <host> connect to server at <host>

-
-

Options:

-
-

-p, --port=<port> listen on/connect to port <port> - (default 18515)
- -c, --connection=<RC/UC> connection type RC/UC (default RC)
- -m, --mtu=<mtu> mtu size (default 1024)
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)
- -i, --ib-port=<port> use port <port> of IB device (default 1)
- -s, --size=<size> size of message to exchange (default 1)
- -a, --all Run sizes from 2 till 2^23
- -t, --tx-depth=<dep> size of tx queue (default 50)
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)
- -C, --report-cycles report times in cpu cycle units (default - microseconds)
- -H, --report-histogram print out all results (default print summary - only)
- -U, --report-unsorted (implies -H) print out unsorted results (default - sorted)
- -V, --version display version number

-
-
-

ib_write_bw.exe     - BW test with RDMA write transactions

-
-

Usage:

-
-

ib_write_bw                - # start a server and wait for connection
- ib_write_bw <host>    # connect to server at <host>

-
-

Options:

-
-

-p, --port=<port> listen on/connect to port <port> - (default 18515)
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)
- -i, --ib-port=<port> use port <port> of IB device (default 1)
- -c, --connection=<RC/UC> connection type RC/UC (default RC)
- -m, --mtu=<mtu> mtu size (default 1024)
- -g, --post=<num of posts> number of posts for each qp in the chain - (default tx_depth)
- -q, --qp=<num of qp's> Num of qp's(default 1)
- -s, --size=<size> size of message to exchange (default 65536)
- -a, --all Run sizes from 2 till 2^23
- -t, --tx-depth=<dep> size of tx queue (default 100)
- -n, --iters=<iters> number of exchanges (at least 2, default 5000)
- -b, --bidirectional measure bidirectional bandwidth (default - unidirectional)
- -V, --version display version number

-
-
-

<return-to-top>

-


- 

-

ttcp - Test TCP performance

-

TTCP accesses the Windows socket layer, hence it does not access -IB verbs directly. IPoIB or WSD layers are invoked beneath the socket layer -depending on configuration. TTCP is included as a quick baseline performance -check.

-
- 
Usage: ttcp -t [-options] host 
-       ttcp -r [-options]
-Common options:
-	-l ##	length of bufs read from or written to network (default 8192)
-	-u	use UDP instead of TCP
-	-p ##	port number to send to or listen at (default 5001)
-	-A	align the start of buffers to this modulus (default 16384)
-	-O	start buffers at this offset from the modulus (default 0)
-	-d	set SO_DEBUG socket option
-	-b ##	set socket buffer size (if supported)
-	-f X	format for rate: k,K = kilo{bit,byte}; m,M = mega; g,G = giga
-Options specific to -t:
-	-n##	number of source bufs written to network (default 2048)
-	-D	don't buffer TCP writes (sets TCP_NODELAY socket option)
-Options specific to -r:
-	-B	for -s, only output full blocks as specified by -l (for TAR)
-	-T	"touch": access each byte as it's read
-

Requires a receiver (server) side and a transmitter (client) - side, host1 and host2 are IPoIB connected hosts.

-

at host1 (receiver)        - ttcp -r -f M -l 4096

-

at host2 (transmitter)    ttcp -t -f M -l - 4096 -n1000 host1

-
-

<return-to-top>

-

 

-

 

-

Diagnostics

-
-

 

-

ib_limits - Infiniband verbs tests

-

Usage: ib_limits [options]

-
-

Options:
-m or --memory
    Direct ib_limits to test memory registration
-c or --cq
    Direct ib_limits to test CQ creation
-r or --resize_cq
    direct ib_limits to test CQ resize
-q or --qp
    Directs ib_limits to test QP creation
-v or --verbose
    Enable verbosity level to debug console.
-h or --help
    Display this usage info then exit.

-
-

<return-to-top>

-

 

-

cmtest - Connection Manager Tests

-

Usage: cmtest [options]

-

    Options:

-
-

 -s --server This option directs cmtest to act as a Server
- -l - --local - This option specifies the local endpoint.
- -r - --remote - This option specifies the remote endpoint.
- -c - --connect - This option specifies the number of connections to open. Default of - 1.
- -m - --msize - This option specifies the byte size of each message. Default is 100 - bytes.
- -n - --nmsgs - This option specifies the number of messages to send at a time.
- -p --permsg This option indicates if a separate buffer should be used per - message. Default is one buffer for all messages.
- -i - --iterate - This option specifies the number of times to loop through 'nmsgs'. - Default of 1.
- -v --verbose This option enables verbosity level to debug console.
- -h --help Display this usage info then exit.

-
-

<return-to-top>

-

 

-

 

-

PrintIP - print ip adapters and their addresses

-
-

PrintIP is used to print IP adapters and their addresses, or - ARP (Address Resolution Protocol) and IP address.
-
- Usage:
-    printip <print_ips>
-    printip <remoteip> <ip>        - (example printip remoteip 10.10.2.20)

-
-

<return-to-top>

-

 

-

-
-vstat - HCA Stats and Counters

-
-

Display HCA (Host channel Adapter) attributes.

-

Usage: vstat [-v] [-c]
-          -v - verbose mode
-          -c - HCA error/statistic - counters

-
-

<return-to-top>

-

 

-

Subnet Management with OpenSM Rev: openib-1.2.0

-
-

A single running process (opensm.exe) is required to configure -and thus make an Infiniband subnet useable.  For most cases, InfiniBand -Subnet Management as a Windows service is sufficient to correctly configure most -InfiniBand fabrics.

-

The Infiniband subnet management process (opensm) may exist on a -Windows (WinOF) node or a Linux (OFED) node.
-
-Limit the number of OpenSM processes per IB fabric; one SM is sufficient -although redundant SMs are supported. You do not need a Subnet Manager per -node/system.

-

OpenIB Subnet Management as a Windows Service

-

InfiniBand subnet management (OpenSM), as a Windows service, is installed by default, although it is NOT -started by default. There are two ways to enable the InfiniBand Subnet -Management service.

-
    -
  1. Reset the installed OpenSM service "InfiniBand Subnet Management" to - start automatically; See
    - My Computer->Manage->Services and Applications->Services->InfiniBand - subnet Management->Start.
     
    2. -
  2. Install OpenSM as a 'running' Windows service:
    - Request a 'Custom' install, selecting the OpenSM_service install feature. - Once the installation has completed, check the running InfiniBand Subnet - Management service status via the Windows service manager (see #1).
    - 
    3. -
  3. Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what - OpenSM thinks is happening.
    4. -
-

 

-

Manual InfiniBand Subnet Management from a command window

-

Usage: opensm.exe [options]

-

Options:

-
-

-c
- --cache-options

-
-

Cache the given command line options into the file
- /var/cache/osm/opensm.opts for use next invocation
- The cache directory can be changed by the environment
- variable OSM_CACHE_DIR

-
-

-g[=]<GUID in hex>
- --guid[=]<GUID in hex>

-
-

This option specifies the local port GUID value with - which OpenSM should bind. OpenSM may be
- bound to 1 port at a time.  If GUID given is 0, OpenSM displays a - list of possible port GUIDs and waits for user input. Without -g, OpenSM - trys to use the default port.

-
-

-l <LMC>
- --lmc <LMC>

-
-

This option specifies the subnet's LMC value.
- The number of LIDs assigned to each port is 2^LMC.
- The LMC value must be in the range 0-7.
- LMC values > 0 allow multiple paths between ports.
- LMC values > 0 should only be used if the subnet
- topology actually provides multiple paths between
- ports, i.e. multiple interconnects between switches.
- Without -l, OpenSM defaults to LMC = 0, which allows
- one path between any two ports.

-
-

-p <PRIORITY>
- --priority <PRIORITY>

-
-

This option specifies the SM's PRIORITY.
- This will effect the handover cases, where master
- is chosen by priority and GUID.
- -smkey <SM_Key>
- This option specifies the SM's SM_Key (64 bits).
- This will effect SM authentication.

-
-

-r
- --reassign_lids

-
-


- This option causes OpenSM to reassign LIDs to all end nodes. Specifying - -r on a running subnet
- may disrupt subnet traffic.  Without -r, OpenSM attempts to - preserve existing LID assignments resolving multiple use of same LID.

-
-

-u
- --updn

-
-

This option activate UPDN algorithm instead of Min Hop - algorithm (default).

-
-

-a
- --add_guid_file <path to file>

-
-

Set the root nodes for the Up/Down routing algorithm to - the guids provided in the given file (one per line)

-
-

-o
- --once

-
-

This option causes OpenSM to configure the subnet once, - then exit. Ports remain in the ACTIVE state.

-
-

-s <interval>
- --sweep <interval>

-
-

This option specifies the number of seconds between - subnet sweeps. Specifying -s 0 disables sweeping.
- Without -s, OpenSM defaults to a sweep interval of 10 seconds.

-
-

-t <milliseconds>
- --timeout <milliseconds>

-
-

This option specifies the time in milliseconds
- used for transaction timeouts.
- Specifying -t 0 disables timeouts.
- Without -t, OpenSM defaults to a timeout value of
- 200 milliseconds.

-
-

-maxsmps <number>

-
-

This option specifies the number of VL15 SMP MADs - allowed on the wire at any one time.
- Specifying -maxsmps 0 allows unlimited outstanding SMPs.
- Without -maxsmps, OpenSM defaults to a maximum of one outstanding SMP.

-
-

-i <equalize-ignore-guids-file>
- -ignore-guids <equalize-ignore-guids-file>

-
-

This option provides the means to define a set of ports - (by guids) that will be ignored by the link load  - equalization algorithm.

-
-

-x
- --honor_guid2lid

-
-

This option forces OpenSM to honor the guid2lid file, - when it comes out of Standby state, if such file exists - under OSM_CACHE_DIR, and is valid. - By default this is FALSE.

-
-

-f
- --log_file

-
-

This option defines the log to be the given file. By - default the log goes to %SystemRoot%\temp\osm.log.
- For the log to go to standard output use -f stdout.

-
-

-e
- --erase_log_file

-
-

This option will cause deletion of the log file  - (if it previously exists). By default, the log file is accumulative.

-
-

-y
- --stay_on_fatal

-
-

This option will cause SM not to exit on fatal - initialization - issues: if SM discovers duplicated guids or 12x link with - lane reversal badly configured. - By default, the SM will exit on these errors.

-
-

-v
- --verbose

-
-

This option increases the log verbosity level. - The -v option may be specified multiple times - to further increase the verbosity level.  - See the -vf option for more information about. - log verbosity.

-
-

-V

-
-

This option sets the maximum verbosity level and - forces log flushing.
- The -V is equivalent to '-vf 0xFF -d 2'. - See the -vf option for more information about - log verbosity.

-
-

-D <flags>

-
-

This option sets the log verbosity level.  A flags - field must follow the -D option.
- A bit set/clear in the flags enables/disables a specific log level as - follows:
- BIT LOG LEVEL ENABLED
- ---- -----------------
- 0x01 - ERROR (error messages)
- 0x02 - INFO (basic messages, low volume)
- 0x04 - VERBOSE (interesting stuff, moderate volume)
- 0x08 - DEBUG (diagnostic, high volume)
- 0x10 - FUNCS (function entry/exit, very high volume)
- 0x20 - FRAMES (dumps all SMP and GMP frames)
- 0x40 - ROUTING (dump FDB routing information)
- 0x80 - currently unused.
- Without -D, OpenSM defaults to ERROR + INFO (0x3).
- Specifying -D 0 disables all messages.
- Specifying -D 0xFF enables all messages (see -V).
- High verbosity levels may require increasing the transaction timeout - with the -t option.

-
-

-d <number>
- --debug <number>

-
-

This option specifies a debug option. These options are - not normally needed. The number following -d selects the debug option to - enable as follows:
- OPT Description
- --- -----------------
- -d0 - Ignore other SM nodes
- -d1 - Force single threaded dispatching
- -d2 - Force log flushing after each log message
- -d3 - Disable multicast support
- -d4 - Put OpenSM in memory tracking mode
- -d10 - Put OpenSM in testability mode
- Without -d, no debug options are enabled

-
-

-h
- --help

-
-

Display this usage info then exit.

-
-

-?

-
-

Display this usage info then exit.

-
-
-

<return-to-top>

-

 

-

Osmtest - Subnet Management Tests

-

Invoke open subnet management tests. osmtest currently can not -run on the same HCA port which OpenSM is currently using.

-
-

 Usage: osmtest [options]

-

Options:

-
-

 -f <c|a|v|s|e|f|m|q|t>
- --flow <c|a|v|s|e|f|m|q|t>

- -

This option directs osmtest to run a specific flow:

-

FLOW DESCRIPTIONS
- c = create an inventory file with all nodes, ports & paths.
- a = run all validation tests (expecting an input inventory)
- v = only validate the given inventory file.
- s = run service registration, un-registration and lease.
- e = run event forwarding test.
- f = flood the SA with queries accoring to the stress mode.
- m = multicast flow.
- q = QoS info - VLArb and SLtoVL tables.
- t = run trap 64/65 flow; requires running an external tool.
- (default is all but QoS).

- -

-w <trap_wait_time>
- --wait <trap_wait_time>

-
-

This option specifies the wait time for trap 64/65 - in seconds.
- It is used only when running -f t - the trap 64/65 flow
- (default to 10 sec).

-
-

-d <number>
- --debug <number>

-
-

This option specifies a debug option. - These options are not normally needed.
- The number following -d selects the debug - option to enable as follows:
- OPT Description
- --- -----------------
- -d0 - Unused.
- -d1 - Do not scan/compare path records.
- -d2 - Force log flushing after each log message.
- -d3 - Use mem tracking.
- Without -d, no debug options are enabled.

-
-

-m <LID in hex>
- --max_lid <LID in hex>

-
-

This option specifies the maximal LID number to be - searched - for during inventory file build (default to 100).

-
-

-g <GUID in hex>
- --guid <GUID in hex>

-
-

This option specifies the local port GUID value - with which osmtest should bind. osmtest may be - bound to 1 port at a time. - Without -g, osmtest displays a menu of possible - port GUIDs and waits for user input.

-
-

-h
- --help

-
-

Display this usage info then exit.

-
-

-i <filename>
- --inventory <filename>

-
-

This option specifies the name of the inventory - file. - Normally, osmtest expects to find an inventory file, - which osmtest uses to validate real-time information - received from the SA during testing. - If -i is not specified, osmtest defaults to the file - 'osmtest.dat'.
- See the -c option for related information.

-
-

-s
- --stress

-
-

This option runs the specified stress test instead - of the normal test suite.
- Stress test options are as follows:
- OPT Description
- --- -----------------
- -s1 - Single-MAD response SA queries .
- -s2 - Multi-MAD (RMPP) response SA queries.
- -s3 - Multi-MAD (RMPP) Path Record SA queries.
- Without -s, stress testing is not performed.

-
-

-M
- --Multicast_Mode

-
-

This option specify length of Multicast test :
- OPT Description
- --- -----------------
- -M1 - Short Multicast Flow (default) - single mode.
- -M2 - Short Multicast Flow - multiple mode.
- -M3 - Long Multicast Flow - single mode.
- -M4 - Long Multicast Flow - multiple mode.
- Single mode - Osmtest is tested alone , with no other
- apps that interact vs. OpenSM MC.
- Multiple mode - Could be run with other apps using MC vs.
- OpenSM. Without -M, default flow testing is performed.

-
-

-t <milliseconds>

-
-

This option specifies the time in milliseconds used - for transaction timeouts.
- Specifying -t 0 disables timeouts.
- Without -t, osmtest defaults to a timeout value of 1 second.

-
-

-l
- --log_file

-
-

This option defines the log to be the given file.
- By default the log goes to stdout.

-
-

-v

-
-

This option increases the log verbosity level. The - -v option may be specified multiple times
- to further increase the verbosity level. See the -vf option for more - information about log verbosity.

-
-

-V

-
-

This option sets the maximum verbosity level and - forces log flushing.
- The -V is equivalent to '-vf 0xFF -d 2'.
- See the -vf option for more information about log verbosity.

-
-

-vf <flags>

-
-

This option sets the log verbosity level. A flags - field must follow the -vf option.
- A bit set/clear in the flags enables/disables a specific log level - as follows:
- BIT LOG LEVEL ENABLED
- ---- -----------------
- 0x01 - ERROR (error messages)
- 0x02 - INFO (basic messages, low volume)
- 0x04 - VERBOSE (interesting stuff, moderate volume)
- 0x08 - DEBUG (diagnostic, high volume)
- 0x10 - FUNCS (function entry/exit, very high volume)
- 0x20 - FRAMES (dumps all SMP and GMP frames)
- 0x40 - currently unused.
- 0x80 - currently unused.
- Without -vf, osmtest defaults to ERROR + INFO (0x3).
- Specifying -vf 0 disables all messages.
- Specifying -vf 0xFF enables all messages (see -V).
- High verbosity levels may require increasing
- the transaction timeout with the -t option.

-
-
-
-

<return-to-top>

-

 

-


-ibtrapgen - Generate Infiniband subnet management traps

-

Usage: ibtrapgen -t|--trap_num <TRAP_NUM> -n|--number <NUM_TRAP_CREATIONS>
-                          --r|--rate <TRAP_RATE> -l|--lid <LIDADDR>
-                          --s|--src_port <SOURCE_PORT> -p|--port_num <PORT_NUM>
-
-Options: one of the following optional flows:

-
-

-t <TRAP_NUM>
- --trap_num <TRAP_NUM>
-          This option specifies the - number of the trap to generate. Valid values are 128-131.
- -n <NUM_TRAP_CREATIONS>
- --number <NUM_TRAP_CREATIONS>
-          This option specifies the - number of times to generate this trap.
-          If not specified - - default to 1.
- -r <TRAP_RATE>
- --rate <TRAP_RATE>
-          This option specifies the - rate of the trap generation.
-          What is the time period - between one generation and another?
-          The value is given in - miliseconds.
-          If the number of trap - creations is 1 - this value is ignored.
- -l <LIDADDR>
- --lid <LIDADDR>
-          This option specifies the - lid address from where the trap should be generated.
- -s <SOURCE_PORT>
- --src_port <SOURCE_PORT>
-          This option specifies the - port number from which the trap should
-          be generated. If trap - number is 128 - this value is ignored (since
-          trap 128 is not sent with - a specific port number)
- -p <port num>
- --port_num <port num>
-          This is the port number - used for communicating with the SA.
- -h
- --help
-          Display this usage info - then exit.
- -o
- --out_log_file
-          This option defines the - log to be the given file.
-          By default the log goes - to stdout.
- -v
-          This option increases the - log verbosity level.
-          The -v option may be - specified multiple times to further increase the verbosity level.
-          See the -vf option for - more information about log verbosity.
- -V
-          This option sets the - maximum verbosity level and forces log flushing.
-          The -V is equivalent to - '-vf 0xFF -d 2'.
-          See the -vf option for - more information about. log verbosity.
- -x <flags>
-          This option sets the log - verbosity level.
-          A flags field must follow - the -vf option.
-          A bit set/clear in the - flags enables/disables a
-          specific log level as - follows:

-
-

BIT LOG LEVEL ENABLED
- ---- -----------------
- 0x01 - ERROR (error messages)
- 0x02 - INFO (basic messages, low volume)
- 0x04 - VERBOSE (interesting stuff, moderate volume)
- 0x08 - DEBUG (diagnostic, high volume)
- 0x10 - FUNCS (function entry/exit, very high volume)
- 0x20 - FRAMES (dumps all SMP and GMP frames)
- 0x40 - currently unused.
- 0x80 - currently unused.
- Without -x, ibtrapgen defaults to ERROR + INFO (0x3).
- Specifying -x 0 disables all messages.
- Specifying -x 0xFF enables all messages (see -V).

-
-
-

<return-to-top>

-

 

-

 

-

IPoIB - Internet Protocols over InfiniBand

-
-

IPoIB enables the use of Internet Protocol utilities (e.g., ftp, -telnet) to function correctly over an Infiniband fabric. IPoIB is implemented as -an NDIS Miniport driver with a WDM lower edge.

-

The IPoIB Network adapters are -located via 'My Computer->Manage->Device Manager->Network adapters->IPoIB'.
-'My -Network Places->Properties' will display IPoIB Local Area Connection instances and should be used to -configure IP addresses for the IPoIB interfaces; one Local Area Connection -instance per HCA port. The IP -(Internet Protocol) address bound to the IPoIB adapter instance can be assigned -by DHCP or as a static IP addresses via
-'My Network Places->Properties->Local -Area Connection X->Properties->(General Tab)Internet Protocol(TCP/IP)->Properties'.

-

When the subnet manager (opensm) configures/sweeps the local -Infiniband HCA, the Local Area Connection will become enabled. If you discover -the Local Area Connection to be disabled, then likely your subnet manager -(opensm) is not running or functioning correctly.

-

<return-to-top>

-

 

-

 

-

Winsock Direct Service Provider

-
-

Winsock Direct (WSD) is Microsoft's proprietary protocol that -predates SDP (Sockets Direct Protocol) for accelerating TCP/IP applications by -using RDMA hardware. Microsoft had a significant role in defining the SDP -protocol, hence SDP and WSD are remarkably similar, though unfortunately -incompatible.
-
-WSD is made up of two parts, the winsock direct switch and the winsock direct -provider. The WSD switch is in the winsock DLL that ships in all editions of -Windows Server 2003, and is responsible for routing socket traffic over either -the regular TCP/IP stack, or offload it to a WSD provider. The WSD provider is a -hardware specific DLL that implements connection management and data transfers -over particular RDMA hardware.

-

WinOF WSD is not supported in the Windows XP environment.

-

The WSD Protocol seamlessly transports TCP - data using Infiniband data packets in 'buffered' mode or Infiniband - RDMA in 'direct' mode. Either way the user mode socket application sees no - behavioral difference in the standard Internet Protocol socket it created other than -reduced data transfer times and increased bandwidth.
-
-The Windows OpenFabrics release includes a WSD provider library that has been -extensively tested with Microsoft Windows Server 2003.
-During testing, bugs where found in the WSD switch that could lead to hangs, -crashes, data corruption, and other unwanted behavior. Microsoft released a -hotfix to address these issues which should be installed if using WSD; the -Microsoft Windows Server 2003 hotfix can be found -here.
- 

-
-
- - - - -
-
-
- Environment variables can be used to change the behavior - of the WSD provider:
-
- IBWSD_NO_READ - Disables RDMA Read operations when set - to any value. Note that this variable must be used - consistently throughout the cluster or communication - will fail.
-
- IBWSD_POLL - Sets the number of times to poll the - completion queue after processing completions in - response to a CQ event. Reduces latency at the cost of - CPU utilization. Default is 500.
-
- IBWSD_SA_RETRY - Sets the number of times to retry SA - query requests. Default is 4, can be increased if - connection establishment fails.
-
- IBWSD_SA_TIMEOUT - Sets the number of milliseconds to - wait before retrying SA query requests. Default is 4, - can be increased if connection establishment fails.
-
- IBWSD_NO_IPOIB - SA query timeouts by default allow the - connection to be established over IPoIB. Setting this - environment variable to any value prevents fall back to - IPoIB if SA queries time out.
-
- IBWSD_DBG - Controls debug output when using a debug - version of the WSD provider. Takes a hex value, with - leading '0x', default value is '0x80000000'
-
-  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
0x00000001DLL
0x00000002socket info
0x00000004initialization code
0x00000008WQ related functions
0x00000010Enpoints related functions
0x00000020memory registration
0x00000040CM (Connection Manager)
0x00000080connections
0x00000200socket options
0x00000400network events
0x00000800Hardware
0x00001000Overlapped I/O request
0x00002000Socket Duplication
0x00004000Performance Monitoring
0x01000000More verbose than - IBSP_DBG_LEVEL3
0x02000000More verbose than - IBSP_DBG_LEVEL2
0x04000000More verbose than - IBSP_DBG_LEVEL1
0x08000000Verbose output
0x20000000Function enter/exit
0x40000000Warnings
0x80000000Errors
-
-
-
-
-
-


-See -https://wiki.openfabrics.org/tiki-index.php?page=Winsock+Direct for the -latest WSD status.

-

Winsock Direct Service Provider Installation

-

When the custom install option 'WSD' is selected the WSD service -is automatically installed and started as part of the installation.
-Manual control is performed via the \Program Files\WinOF\installsp.exe utility.

-
-

usage: installsp [-i | -r | -l]
-
- -i    Install the Winsock Direct (WSD) service provider
- -r    Remove the WSD service provider
- -r <name>    Remove the specified service provider
- -l    List service providers
- 

-
-

<return-to-top>

-

 

-

 

-

Direct Access Transport and usermode Direct Access Programming -Libraries

-
-

The DAT (Direct Access Transport) -API is a C programming interface developed by the -DAT Collaborative in -order provide a set of transport-independent, platform-independent Application -Programming Interfaces that exploit the RDMA (remote direct memory access) -capabilities of next-generation interconnect technologies such as InfiniBand, -and iWARP.

-

WinOF DAT and uDAPL are based on the 1.1 DAT specification. The DAPL -(Direct Access Provider Library) now fully supports Infiniband RDMA and -IPoIB.

-
-
- - - - -
-
-
-
- How  DAT objects map to equivalent - - InfiniBand objects:
-  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Interface Adapter (IA) - HCA (Host Channel Adapter)
Protection Zone (PZ) PD (Protection Domain)
Local Memory Region (LMR) - MR (Memory Region)
Remote Memory Region (RMR) - MW (Memory Windows)
Event Dispatcher (EVD) - CQ (Completion Queue)
Endpoint (EP) QP (Queue Pair)
Public Service Point (PSP) - connection identifier
Reserved Service Point (RSP) - connection identifier
Connection Request (CR) - connection manager event -
-
-
-
-
-
-

 

-
-

DAT EXECUTION ENVIRONMENT:

-
-
-

In order for DAT/uDAPL - programs to execute correctly, the 'dat.dll' file must be present in the - current directory, - %SystemRoot%\system32, %SystemRoot% or in the library search path.

-

The default WinOF - installation places the file dat.dll in the '%SystemRoot%' folder.

-

The DAPL configuration - file by default is defined as '%SystemDrive%\DAT\dat.conf'. This default - specification can be overriden by use of the environment variable - DAT_OVERRIDE; see following environment variable discussion.

-

Within the dat.conf file, - the DAPL library specification can be located as the 5th whitespace - separated line argument. By default the DAPL library file is installed as
- '%SystemRoot%\dapl.dll'.

-

Should you choose to - relocated the DAPL library file to a path where whitespace appears in the - full library path specification, then the full library file specification - must be contained within double-quotes. A side effect of the double-quotes - is the library specification is treated as a Windows string which implies - the '\' (backslash character) is treated as an 'escape' character.  Hence - all backslashes in the library path must be duplicated when enclosed in - double-quotes - (e.g., "C:\\Programs Files\\WinOF\\dapl.dll").

-

A sample dat.conf file is - installed as '\Program Files\WinOF\dat.conf '.
- After the WinOF installation, move the \Program Files\WinOF\dat.conf file to - the default DAT configuration file location - '%SystemDrive%\DAT\dat.conf'.

-

In order to preserve existing installations, - the dat.conf file is not automatically installed in it's default location.

-

 

-

-DAT library environment variables:
- 

- 
-DAT_OVERRIDE
-------------
-Value used as the static registry configuration file, overriding the
-default location, 'C:\DAT\dat.conf'.
-
-Example: set DAT_OVERRIDE=%SystemDrive%\path\to\my\private.conf
-
-
-DAT_DBG_TYPE
-------------
-
-Value specifies which parts of the registry will print debugging
-information, valid values are 
-
-DAT_OS_DBG_TYPE_ERROR = 0x1
-DAT_OS_DBG_TYPE_GENERIC = 0x2
-DAT_OS_DBG_TYPE_SR = 0x4
-DAT_OS_DBG_TYPE_DR = 0x8
-DAT_OS_DBG_TYPE_PROVIDER_API = 0x10
-DAT_OS_DBG_TYPE_CONSUMER_API = 0x20
-DAT_OS_DBG_TYPE_ALL = 0xff
-
-or any combination of these. For example you can use 0xC to get both 
-static and dynamic registry output.
-
-Example set DAT_DBG_TYPE=0xC
-
-DAT_DBG_DEST
------------- 
-
-Value sets the output destination, valid values are 
-
-DAT_OS_DBG_DEST_STDOUT = 0x1
-DAT_OS_DBG_DEST_SYSLOG = 0x2 
-DAT_OS_DBG_DEST_ALL = 0x3 
-
-For example, 0x3 will output to both stdout and the syslog. 
-
-

-DAPL Provider library environment variables

-

-
-DAPL_DBG_TYPE
--------------
-
-Value specifies which parts of the registry will print -debugging information, valid values are
-
-DAPL_DBG_TYPE_ERR = 0x0001
-DAPL_DBG_TYPE_WARN = 0x0002
-DAPL_DBG_TYPE_EVD = 0x0004
-DAPL_DBG_TYPE_CM = 0x0008
-DAPL_DBG_TYPE_EP = 0x0010
-DAPL_DBG_TYPE_UTIL = 0x0020
-DAPL_DBG_TYPE_CALLBACK = 0x0040
-DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080
-DAPL_DBG_TYPE_API = 0x0100
-DAPL_DBG_TYPE_RTN = 0x0200
-DAPL_DBG_TYPE_EXCEPTION = 0x0400
-
-or any combination of these. For example you can use 0xC to get both
-EVD and CM output.
-
-Example set DAPL_DBG_TYPE=0xC
-
-
-DAPL_DBG_DEST
--------------
-
-Value sets the output destination, valid values are
-
-DAPL_DBG_DEST_STDOUT = 0x1
-DAPL_DBG_DEST_SYSLOG = 0x2
-DAPL_DBG_DEST_ALL = 0x3
-
-For example, 0x3 will output to both stdout and the syslog.
-
-
- -

-
-

DAPLTEST

-
-
-    dapltest - test for the Direct Access Provider Library (DAPL)
-
-DESCRIPTION
-
-    Dapltest is a set of tests developed to exercise, characterize,
-    and verify the DAPL interfaces during development and porting.
-    At least two instantiations of the test must be run.  One acts
-    as the server, fielding requests and spawning server-side test
-    threads as needed.  Other client invocations connect to the
-    server and issue test requests.
-
-    The server side of the test, once invoked, listens continuously
-    for client connection requests, until quit or killed.  Upon
-    receipt of a connection request, the connection is established,
-    the server and client sides swap version numbers to verify that
-    they are able to communicate, and the client sends the test
-    request to the server.  If the version numbers match, and the
-    test request is well-formed, the server spawns the threads
-    needed to run the test before awaiting further connections.
-
-USAGE
-
-    dapltest [ -f script_file_name ]
-             [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]
-
-    With no arguments, dapltest runs as a server using default values,
-    and loops accepting requests from clients.  The -f option allows
-    all arguments to be placed in a file, to ease test automation.
-    The following arguments are common to all tests:
-
-    [ -T S|Q|T|P|L ]    Test function to be performed:
-                            S   - server loop
-                            Q   - quit, client requests that server
-                                  wait for any outstanding tests to
-                                  complete, then clean up and exit
-                            T   - transaction test, transfers data between 
-                                  client and server
-                            P   - performance test, times DTO operations
-                            L   - limit test, exhausts various resources,
-                                  runs in client w/o server interaction
-                        Default: S
-
-    [ -D device_name ]  Specifies the name of the device (interface adapter).
-                        Default: host-specific, look for DT_MdepDeviceName
-                                 in dapl_mdep.h
-
-    [ -d ]              Enables extra debug verbosity, primarily tracing
-			of the various DAPL operations as they progress.
-			Repeating this parameter increases debug spew.
-			Errors encountered result in the test spewing some
-			explanatory text and stopping; this flag provides
-			more detail about what lead up to the error.
-                        Default: zero
-
-    [ -R BE ]           Indicate the quality of service (QoS) desired.
-                        Choices are:
-                            HT  - high throughput
-                            LL  - low latency
-                            EC  - economy (neither HT nor LL)
-                            PM  - premium
-                            BE  - best effort
-                        Default: BE
-
-USAGE - Quit test client
-
-    dapltest [Common_Args] [ -s server_name ]
-
-    Quit testing (-T Q) connects to the server to ask it to clean up and
-    exit (after it waits for any outstanding test runs to complete).
-    In addition to being more polite than simply killing the server,
-    this test exercises the DAPL object teardown code paths.
-    There is only one argument other than those supported by all tests:
-
-    -s server_name      Specifies the name of the server interface.
-                        No default.
-
-
-USAGE - Transaction test client
-
-    dapltest [Common_Args] [ -s server_name ]
-             [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ] 
-             [ -V ] [ -P ] OPclient OPserver [ op3, 
-
-    Transaction testing (-T T) transfers a variable amount of data between 
-    client and server.  The data transfer can be described as a sequence of 
-    individual operations; that entire sequence is transferred 'iterations' 
-    times by each thread over all of its endpoint(s).
-
-    The following parameters determine the behavior of the transaction test:
-
-    -s server_name      Specifies the hostname of the dapltest server.
-                        No default.
-
-    [ -t threads ]      Specify the number of threads to be used.
-                        Default: 1
-
-    [ -w endpoints ]    Specify the number of connected endpoints per thread.
-                        Default: 1
-
-    [ -i iterations ]   Specify the number of times the entire sequence
-                        of data transfers will be made over each endpoint.
-                        Default: 1000
-
-    [ -Q ]              Funnel completion events into a CNO.
-			Default: use EVDs
-
-    [ -V ]              Validate the data being transferred.
-			Default: ignore the data
-
-    [ -P ]		Turn on DTO completion polling
-			Default: off
-
-    OP1 OP2 [ OP3, ... ]
-                        A single transaction (OPx) consists of:
-
-                        server|client   Indicates who initiates the
-                                        data transfer.
-
-                        SR|RR|RW        Indicates the type of transfer:
-                                        SR  send/recv
-                                        RR  RDMA read
-                                        RW  RDMA write
-                        Defaults: none
-
-                        [ seg_size [ num_segs ] ]
-                                        Indicates the amount and format
-                                        of the data to be transferred.
-                                        Default:  4096  1
-                                                  (i.e., 1 4KB buffer)
-
-                        [ -f ]          For SR transfers only, indicates
-                                        that a client's send transfer
-                                        completion should be reaped when
-                                        the next recv completion is reaped.
-					Sends and receives must be paired
-					(one client, one server, and in that
-					order) for this option to be used.
-
-    Restrictions:  
-    
-    Due to the flow control algorithm used by the transaction test, there 
-    must be at least one SR OP for both the client and the server.  
-
-    Requesting data validation (-V) causes the test to automatically append 
-    three OPs to those specified. These additional operations provide 
-    synchronization points during each iteration, at which all user-specified 
-    transaction buffers are checked. These three appended operations satisfy 
-    the "one SR in each direction" requirement.
-
-    The transaction OP list is printed out if -d is supplied.
-
-USAGE - Performance test client
-
-    dapltest [Common_Args] -s server_name [ -m p|b ]
-             [ -i iterations ] [ -p pipeline ] OP
-
-    Performance testing (-T P) times the transfer of an operation.
-    The operation is posted 'iterations' times.
-
-    The following parameters determine the behavior of the transaction test:
-
-    -s server_name      Specifies the hostname of the dapltest server.
-                        No default.
-
-    -m b|p		Used to choose either blocking (b) or polling (p)
-                        Default: blocking (b)
-
-    [ -i iterations ]   Specify the number of times the entire sequence
-                        of data transfers will be made over each endpoint.
-                        Default: 1000
-
-    [ -p pipeline ]     Specify the pipline length, valid arguments are in 
-                        the range [0,MAX_SEND_DTOS]. If a value greater than 
-                        MAX_SEND_DTOS is requested the value will be
-                        adjusted down to MAX_SEND_DTOS.
-                        Default: MAX_SEND_DTOS
-
-    OP
-                        An operation consists of:
-
-                        RR|RW           Indicates the type of transfer:
-                                        RR  RDMA read
-                                        RW  RDMA write
-                        Default: none
-
-                        [ seg_size [ num_segs ] ]
-                                        Indicates the amount and format
-                                        of the data to be transferred.
-                                        Default:  4096  1
-                                                  (i.e., 1 4KB buffer)
-
-USAGE - Limit test client
-
-    Limit testing (-T L) neither requires nor connects to any server
-    instance.  The client runs one or more tests which attempt to
-    exhaust various resources to determine DAPL limits and exercise
-    DAPL error paths.  If no arguments are given, all tests are run.
-
-    Limit testing creates the sequence of DAT objects needed to
-    move data back and forth, attempting to find the limits supported
-    for the DAPL object requested.  For example, if the LMR creation
-    limit is being examined, the test will create a set of
-    {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to
-    failure using that set of DAPL objects.  The 'width' parameter
-    can be used to control how many of these parallel DAPL object
-    sets are created before beating upon the requested constructor.
-    Use of -m limits the number of dat_*_create() calls that will
-    be attempted, which can be helpful if the DAPL in use supports
-    essentailly unlimited numbers of some objects.
-
-    The limit test arguments are:
-
-    [ -m maximum ]      Specify the maximum number of dapl_*_create()
-                        attempts.
-                        Default: run to object creation failure
-
-    [ -w width ]        Specify the number of DAPL object sets to
-                        create while initializing.
-                        Default: 1
-
-    [ limit_ia ]        Attempt to exhaust dat_ia_open()
-
-    [ limit_pz ]        Attempt to exhaust dat_pz_create()
-
-    [ limit_cno ]       Attempt to exhaust dat_cno_create()
-
-    [ limit_evd ]       Attempt to exhaust dat_evd_create()
-
-    [ limit_ep ]        Attempt to exhaust dat_ep_create()
-
-    [ limit_rsp ]       Attempt to exhaust dat_rsp_create()
-
-    [ limit_psp ]       Attempt to exhaust dat_psp_create()
-
-    [ limit_lmr ]       Attempt to exhaust dat_lmr_create(4KB)
-
-    [ limit_rpost ]     Attempt to exhaust dat_ep_post_recv(4KB)
-
-    [ limit_size_lmr ]  Probe maximum size dat_lmr_create()
-
-                        Default: run all tests
-
-
-EXAMPLES
-
-    dapltest -T S -d -D ibnic0
-
-                        Starts a local dapltest server process with debug verbosity.
-                        Server loops (listen for dapltest request, process request).
-    
-    dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2
-
-                        Runs a transaction test, with both sides
-                        sending one buffer with two 4KB segments,
-                        one hundred times; dapltest server is on host winIB.
-
-    dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2
-
-                        Runs a performance test, with the client 
-                        RDMA writing one buffer with two 4KB segments,
-                        one hundred times.
-
-    dapltest -T Q -s winIB -D ibnic0
-
-                        Asks the dapltest server at host 'winIB' to clean up and exit.
-
-    dapltest -T L -D ibnic0 -d -w 16 -m 1000
-
-                        Runs all of the limit tests, setting up
-                        16 complete sets of DAPL objects, and
-                        creating at most a thousand instances
-                        when trying to exhaust resources.
-
-    dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \
-       client RW  4096 1    server RW  2048 4    \
-       client SR  1024 4    server SR  4096 2    \
-       client SR  1024 3 -f server SR  2048 1 -f
-
-                        Runs a more complicated transaction test,
-                        with two thread using four EPs each,
-                        sending a more complicated buffer pattern
-                        for a larger number of iterations,
-                        validating the data received.
-
-
-BUGS  (and  To Do List)
-
-    Use of CNOs (-Q) is not yet supported.
-
-    Further limit tests could be added.
-

<return-to-top>

-

 

-

 

-

SRP - SCSI RDMA Protocol

-

-The -SCSI RDMA -Protocol  (SRP) is an emerging industry standard protocol for utilizing -block storage devices over an InfiniBand™ fabric. SRP is being defined in the -ANSI T-10 committee.

-

WinOF SRP is a storage -driver implementation that enables the SCSI RDMA protocol over an InfiniBand -fabric.
-The implementation conforms -to the T10 Working Group draft - -http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf.

-

Software Dependencies

-

The SRP driver depends on the installation of the WinOF stack -with a Subnet
-Manager running somewhere on the IB fabric.
-
-- Supported Operating Systems and Service Packs:
-   o Windows XP SP2 x64
-   o Windows Server 2003 SP1 (x86, x64)
-   o Windows Server 2003 CCS (x64)

-


-Testing Level

-

The SRP driver has undergone basic testing against Mellanox -Technologies'
-SRP Targets MTD1000 and MTD2000. Testing included format, read, and write
-operations.
- 

-

Installation

-

The WinOF stack  does not install the SRP driver by default. -If SRP is -selected in the custom installation window, it will only be copied during WinOF -installation. To complete the SRP driver installation, an SRP target must be -detected; a Subnet Manager must be running somewhere in the IB subnet.
-
-Upon the detection of an SRP target, the "New Hardware Found" Wizard pops up.
-- Select Install Automatically and click Next. This installs the I/O unit -device.
-
-Once completed, the "New Hardware Found" Wizard pops up again.
-- Select Install Automatically and click Next. This installs the SRP driver.
- 

-

New Features and Changes

-

- SRP supports WPP tracing tools by using the GUID: -'5AF07B3C-D119-4233-9C81-C07EF481CBE6'.
-  The flags and level of debug can be controlled at load-time or run-time.
- 

-

<return-to-top>