+++ /dev/null
-\r
-<head>\r
-<style>\r
-<!--\r
-div.Section1\r
- {page:Section1;}\r
--->\r
-</style>\r
-</head>\r
-\r
-<h1 align="left">\r
-<img border="0" src="openfabrics.gif" width="107" height="93"> \r
-<a name="TOP" href="#TOP"></a> </h1>\r
-<h1 align="center">Windows OpenFabrics </h1>\r
-<h1 align="center">User's Manual</h1>\r
-<h2 align="center">Release 1.0</h2>\r
-<h3 align="center">\r
-<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->08/01/2007<!--webbot bot="Timestamp" endspan i-checksum="12522" --></h3>\r
-<h2 align="left"><u>Overview</u></h2>\r
-<p align="left"><span style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'">\r
-The Windows OpenFabrics (WinOF) package is composed of software modules intended \r
-for use on Microsoft Windows based computer systems connected via an InfiniBand \r
-fabric.</span></p>\r
-<p align="left">The Windows OpenFabrics software package contains the \r
-following:<br>\r
-<br>\r
-OpenFabrics Infiniband core drivers and Upper Level Protocols (ULPs):</p>\r
-<ul>\r
- <li>\r
- <p align="left">HCA (Host Channel Adapter) driver </li>\r
- <li>\r
- <p style="text-indent: -.25in; margin-left: .5in"><span dir="LTR">MTHCA -<span style='background-position: 0% 0%; mso-highlight:yellow; background-image:none; background-repeat:repeat; background-attachment:scroll'> \r
- Mellanox </span></span><st1:stockticker>\r
- <span\r
- style='background-position: 0% 0%; mso-highlight:yellow; background-image:none; background-repeat:repeat; background-attachment:scroll'>HCA</span></st1:stockticker><span\r
-style='background-position: 0% 0%; mso-highlight:yellow; background-image:none; background-repeat:repeat; background-attachment:scroll'> low level driver. See \r
- Release_notes.htm for a list of supported devices. <o:p></o:p></span>\r
- </li>\r
-<li>\r
-<p align="left">Infiniband Core modules: IB verbs and IB access layer</li>\r
-<li>\r
-<p align="left">Upper Layer Protocols: IPoIB, WSD, VNIC, SRP Initiator and uDAPL</li>\r
-</ul>\r
-<p align="left">OpenFabrics utilities:</p>\r
-<ul>\r
- <li>\r
- <p align="left">OpenSM: InfiniBand Subnet Manager</li>\r
-<li>\r
-<p align="left">Performance tests</li>\r
-<li>\r
-<p align="left">Diagnostic tools</li>\r
-</ul>\r
-<p align="left">Documentation</p>\r
-<ul>\r
- <li>\r
- <p align="left">User's manual</li>\r
-<li>\r
-<p align="left">Release Notes</li>\r
-</ul>\r
-<h2 align="left"><u>Features</u></h2>\r
-<ul>\r
- <li>\r
- <h3 align="left">Tools</h3></li>\r
-</ul>\r
-<blockquote>\r
- <blockquote>\r
- <p align="left">The OpenFabrics Alliance Windows release contains a set of \r
- user mode tools which are designed to faciliate the smooth operation of an \r
- Windows OpenFabrics installation. These tools are available from a command \r
- window (cmd.exe) as the installation path '%SystemDrive%\Program \r
- Files\WinOF' is appended to the system wide search path registry entry. \r
- A start menu short-cut 'WinOF Cmd Window' is provided to faciliate \r
- correction tool operation.</p>\r
- <h4 align="left">Infiniband Subnet Management</h4>\r
- <ul>\r
- <li>\r
- <p align="left"><a href="#opensm">opensm</a> Open Subnet \r
- Management - configure and manage an InfiniBand subnet</li>\r
- <li>\r
- <p align="left"><a href="#osmtest">osmtest</a> \r
- Subnet management tests</li>\r
- <li>\r
- <p align="left"><a href="#ibtrapgen">ib_trapgen</a> Generate Infiniband Subnet \r
- Management Traps for testing purposes</li>\r
- </ul>\r
- <h4 align="left"><a href="#verbs_benchmarks">Performance</a></h4>\r
- <ul>\r
- <li>\r
- <p align="left"><a href="#ibsend_lat">ib_send_lat</a> Infiniband send \r
-latency measurement</li>\r
- <li>\r
- <p align="left"><a href="#ibsend_bw">ib_send_bw</a> Infiniband send bandwidth \r
- measurement</li>\r
- <li>\r
- <p align="left"><a href="#ibwrite_lat">ib_write_lat</a> Infiniband RDMA write \r
-latency measurement</li>\r
- <li>\r
- <p align="left">i<a href="#ibwrite_bw">b_write_bw</a> Infiniband RDMA write bandwidth \r
-measurement</li>\r
- <li>\r
- <p align="left"><a href="#ttcp">ttcp</a> \r
-TCP performance measurements</li>\r
- </ul>\r
- <h4 align="left"><a href="#diags">Diagnostics</a></h4>\r
- <ul>\r
- <li>\r
- <p align="left"><a href="#iblimits">ib_limits</a> \r
- Infiniband verb tests</li>\r
- <li>\r
- <p align="left"><a href="#cmtest">cmtest </a> Connection Manager tests</li>\r
- <li>\r
- <p align="left"><a href="#printip">PrintIP</a> Display \r
-an Internet Protocol address associated with an IB GUID.</li>\r
- <li>\r
- <p align="left"><a href="#vstat">vstat</a> \r
-Display HCA attributes, statistics and error counters.<br> </li>\r
- </ul>\r
- </blockquote>\r
-</blockquote>\r
-<ul>\r
- <li>\r
- <h3 align="left"><a href="#IPoIB">IPoIB - Internet Protocols over InfiniBand</a></h3>\r
- </li>\r
- <li>\r
- <h3 align="left"><a href="#winsockdirect">Winsock Direct Service Provider</a></h3>\r
- </li>\r
- <li>\r
- <h3 align="left"><a href="#DAT">DAT and uDAPL</a></h3>\r
-<ul>\r
- <li>\r
- <h4 align="left"><a href="#DAPLTEST">DAPLtest</a></h4></li>\r
- <li>\r
- <h4 align="left"><a href="#DAPLtest-examples">DAPLtest Examples</a></h4>\r
- </li>\r
-</ul></li>\r
- <li>\r
- <h3 align="left"><a href="#SRP">SRP - SCSI RDMA Protocol</a></h3></li>\r
-</ul>\r
-<h3 align="left"> </h3>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="verbs_benchmarks"></a>User mode micro-benchmarks</h2>\r
-<hr>\r
-<p align="left">The following user-mode test programs are intended as useful \r
-micro-benchmarks for HW or SW \r
-tuning and/or functional testing.</p>\r
-<blockquote>\r
- <p align="left">Tests use CPU cycle counters to get time stamps without \r
- context switch.<br>\r
- <br>Tests measure round-trip time but report half of that as one-way latency<br>\r
- (i.e.. May not be sufficiently accurate for \r
- asymmetrical configurations).<br>\r
- <br>Min/Median/Max result is reported.<br>The median (vs. average) is less sensitive to extreme scores.<br>Typically the "Max" value is the first value measured.<br>\r
- <br>larger samples only marginally help. The default (1000) is pretty good.<br>Note that an array of cycles_t (typically unsigned long) is allocated<br>once to collect samples and again to store the difference between them.<br>Really big sample sizes (e.g. 1 million) might expose other problems<br>with the program.<br>\r
- <br>"-H" option will dump the histogram for additional statistical analysis.<br>See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other\r
- <br>statistical math programs.<br><br>Architectures tested: x86, x86_64, ia64</p>\r
-</blockquote>\r
-<h4 align="left"><a name="ibsend_lat"></a>ib_send_lat.exe - latency test with \r
-send transactions</h4>\r
-<blockquote>\r
- <p align="left">Usage:</p>\r
- <blockquote>\r
- <p align="left">ib_send_lat start a server and wait for connection<br>\r
- ib_send_lat <host> connect to server at <host></p>\r
- </blockquote>\r
- <p align="left">Options:</p>\r
- <blockquote>\r
- <p align="left">-p, --port=<port> listen on/connect to port <port> \r
- (default 18515)<br>\r
- -c, --connection=<RC/UC> connection type RC/UC (default RC)<br>\r
- -m, --mtu=<mtu> mtu size (default 2048)<br>\r
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)<br>\r
- -i, --ib-port=<port> use port <port> of IB device (default 1)<br>\r
- -s, --size=<size> size of message to exchange (default 1)<br>\r
- -t, --tx-depth=<dep> size of tx queue (default 50)<br>\r
- -l, --signal signal completion on each msg<br>\r
- -a, --all Run sizes from 2 till 2^23<br>\r
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)<br>\r
- -C, --report-cycles report times in cpu cycle units (default \r
- microseconds)<br>\r
- -H, --report-histogram print out all results (default print summary \r
- only)<br>\r
- -U, --report-unsorted (implies -H) print out unsorted results (default \r
- sorted)<br>\r
- -V, --version display version number<br>\r
- -e, --events sleep on CQ events (default poll)</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a name="ibsend_bw"></a>ib_send_bw.exe - BW (BandWidth) test with send transactions</h4>\r
-<blockquote>\r
- <p align="left">Usage:</p>\r
- <blockquote>\r
- <p align="left">ib_send_bw start a server and wait for connection<br>\r
- ib_send_bw <host> connect to server at <host></p>\r
- </blockquote>\r
- <p align="left">Options:</p>\r
- <blockquote>\r
- <p align="left">-p, --port=<port> listen on/connect to port <port> \r
- (default 18515)<br>\r
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)<br>\r
- -i, --ib-port=<port> use port <port> of IB device (default 1)<br>\r
- -c, --connection=<RC/UC> connection type RC/UC/UD (default RC)<br>\r
- -m, --mtu=<mtu> mtu size (default 1024)<br>\r
- -s, --size=<size> size of message to exchange (default 65536)<br>\r
- -a, --all Run sizes from 2 till 2^23<br>\r
- -t, --tx-depth=<dep> size of tx queue (default 300)<br>\r
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)<br>\r
- -b, --bidirectional measure bidirectional bandwidth (default \r
- unidirectional)<br>\r
- -V, --version display version number<br>\r
- -e, --events sleep on CQ events (default poll)</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a name="ibwrite_lat"></a>ib_write_lat.exe - latency test with RDMA write \r
-transactions</h4>\r
-<blockquote>\r
- <p align="left">Usage:</p>\r
- <blockquote>\r
- <p align="left">ib_write_lat start a server and wait for connection<br>\r
- ib_write_lat <host> connect to server at <host></p>\r
- </blockquote>\r
- <p align="left">Options:</p>\r
- <blockquote>\r
- <p align="left">-p, --port=<port> listen on/connect to port <port> \r
- (default 18515)<br>\r
- -c, --connection=<RC/UC> connection type RC/UC (default RC)<br>\r
- -m, --mtu=<mtu> mtu size (default 1024)<br>\r
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)<br>\r
- -i, --ib-port=<port> use port <port> of IB device (default 1)<br>\r
- -s, --size=<size> size of message to exchange (default 1)<br>\r
- -a, --all Run sizes from 2 till 2^23<br>\r
- -t, --tx-depth=<dep> size of tx queue (default 50)<br>\r
- -n, --iters=<iters> number of exchanges (at least 2, default 1000)<br>\r
- -C, --report-cycles report times in cpu cycle units (default \r
- microseconds)<br>\r
- -H, --report-histogram print out all results (default print summary \r
- only)<br>\r
- -U, --report-unsorted (implies -H) print out unsorted results (default \r
- sorted)<br>\r
- -V, --version display version number</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a name="ibwrite_bw"></a>ib_write_bw.exe - BW test with RDMA write transactions</h4>\r
-<blockquote>\r
- <p align="left">Usage:</p>\r
- <blockquote>\r
- <p align="left">ib_write_bw \r
- # start a server and wait for connection<br>\r
- ib_write_bw <host> # connect to server at <host></p>\r
- </blockquote>\r
- <p align="left">Options:</p>\r
- <blockquote>\r
- <p align="left">-p, --port=<port> listen on/connect to port <port> \r
- (default 18515)<br>\r
- -d, --ib-dev=<dev> use IB device <dev> (default first device found)<br>\r
- -i, --ib-port=<port> use port <port> of IB device (default 1)<br>\r
- -c, --connection=<RC/UC> connection type RC/UC (default RC)<br>\r
- -m, --mtu=<mtu> mtu size (default 1024)<br>\r
- -g, --post=<num of posts> number of posts for each qp in the chain \r
- (default tx_depth)<br>\r
- -q, --qp=<num of qp's> Num of qp's(default 1)<br>\r
- -s, --size=<size> size of message to exchange (default 65536)<br>\r
- -a, --all Run sizes from 2 till 2^23<br>\r
- -t, --tx-depth=<dep> size of tx queue (default 100)<br>\r
- -n, --iters=<iters> number of exchanges (at least 2, default 5000)<br>\r
- -b, --bidirectional measure bidirectional bandwidth (default \r
- unidirectional)<br>\r
- -V, --version display version number</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"><br>\r
- </p>\r
-<h4 align="left"><a name="ttcp"></a>ttcp - Test TCP performance</h4>\r
-<p align="left">TTCP accesses the Windows socket layer, hence it does not access \r
-IB verbs directly. IPoIB or WSD layers are invoked beneath the socket layer \r
-depending on configuration. TTCP is included as a quick baseline performance \r
-check.</p>\r
-<blockquote>\r
- <pre>Usage: ttcp -t [-options] host \r
- ttcp -r [-options]\r
-Common options:\r
- -l ## length of bufs read from or written to network (default 8192)\r
- -u use UDP instead of TCP\r
- -p ## port number to send to or listen at (default 5001)\r
- -A align the start of buffers to this modulus (default 16384)\r
- -O start buffers at this offset from the modulus (default 0)\r
- -d set SO_DEBUG socket option\r
- -b ## set socket buffer size (if supported)\r
- -f X format for rate: k,K = kilo{bit,byte}; m,M = mega; g,G = giga\r
-Options specific to -t:\r
- -n## number of source bufs written to network (default 2048)\r
- -D don't buffer TCP writes (sets TCP_NODELAY socket option)\r
-Options specific to -r:\r
- -B for -s, only output full blocks as specified by -l (for TAR)\r
- -T "touch": access each byte as it's read</pre>\r
- <p align="left">Requires a receiver (server) side and a transmitter (client) \r
- side, host1 and host2 are IPoIB connected hosts.</p>\r
- <p align="left">at host1 (receiver) \r
- ttcp -r -f M -l 4096</p>\r
- <p align="left">at host2 (transmitter) ttcp -t -f M -l \r
- 4096 -n1000 host1</p>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="diags"></a>Diagnostics</h2>\r
-<hr>\r
-<p align="left"> </p>\r
-<h3 align="left"><a name="iblimits"></a>ib_limits - Infiniband verbs tests</h3>\r
-<p align="left">Usage: ib_limits [options]</p>\r
-<blockquote>\r
- <p align="left">Options:<br>-m or --memory<br> Direct ib_limits to test memory registration<br>-c or --cq<br> Direct ib_limits to test CQ creation<br>-r or --resize_cq<br> direct ib_limits to test CQ resize<br>-q or --qp<br> Directs ib_limits to test QP creation<br>-v or --verbose<br> Enable verbosity level to debug console.<br>-h or --help<br> Display this usage info then exit.</p>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<h3 align="left"><a name="cmtest"></a>cmtest - Connection Manager Tests</h3>\r
-<p>Usage: cmtest [options]</p>\r
-<p> Options:</p>\r
-<blockquote>\r
- <p> -s --server This option directs cmtest to act as a Server<br>\r
- -l\r
- <lid>--local\r
- <lid>This option specifies the local endpoint.<br>\r
- -r\r
- <lid>--remote\r
- <lid>This option specifies the remote endpoint.<br>\r
- -c\r
- <number>--connect\r
- <number>This option specifies the number of connections to open. Default of \r
- 1.<br>\r
- -m\r
- <bytes>--msize\r
- <bytes>This option specifies the byte size of each message. Default is 100 \r
- bytes.<br>\r
- -n\r
- <number>--nmsgs\r
- <number>This option specifies the number of messages to send at a time.<br>\r
- -p --permsg This option indicates if a separate buffer should be used per \r
- message. Default is one buffer for all messages.<br>\r
- -i\r
- <number>--iterate\r
- <number>This option specifies the number of times to loop through 'nmsgs'. \r
- Default of 1.<br>\r
- -v --verbose This option enables verbosity level to debug console.<br>\r
- -h --help Display this usage info then exit.</p>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h3 align="left"><a name="printip"></a>PrintIP - print ip adapters and their addresses</h3>\r
-<blockquote>\r
- <p align="left">PrintIP is used to print IP adapters and their addresses, or \r
- ARP (Address Resolution Protocol) and IP address.<br>\r
- <br>\r
- Usage:<br>\r
- printip <print_ips><br>\r
- printip <remoteip> <ip> \r
- (example printip remoteip 10.10.2.20)</p>\r
-</blockquote>\r
-<h3 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h3>\r
-<p align="left"> </p>\r
-<h3 align="left">\r
-<br>\r
-<a name="vstat"></a>vstat - HCA Stats and Counters</h3>\r
-<blockquote>\r
- <p align="left">Display HCA (Host channel Adapter) attributes.</p>\r
- <p align="left">Usage: vstat [-v] [-c]<br>\r
- -v - verbose mode<br>\r
- -c - HCA error/statistic \r
- counters</p>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="opensm"></a>Subnet Management with OpenSM Rev: openib-1.2.0</h2>\r
-<hr>\r
-<p align="left">A single running process (opensm.exe) is required to configure \r
-and thus make an Infiniband subnet useable. For most cases, InfiniBand \r
-Subnet Management as a Windows service is sufficient to correctly configure most \r
-InfiniBand fabrics.</p>\r
-<p align="left">The Infiniband subnet management process (opensm) may exist on a \r
-Windows (WinOF) node or a Linux (OFED) node.<br>\r
-<br>\r
-Limit the number of OpenSM processes per IB fabric; one SM is sufficient \r
-although redundant SMs are supported. You do not need a Subnet Manager per \r
-node/system.</p>\r
-<h4>OpenIB Subnet Management as a Windows Service</h4>\r
-<p>InfiniBand subnet management (OpenSM), as a Windows service, is installed by default, although it is NOT \r
-started by default. There are two ways to enable the InfiniBand Subnet \r
-Management service.</p>\r
-<ol style="margin-bottom: 0in">\r
- <li>Reset the installed OpenSM service "InfiniBand Subnet Management" to \r
- start automatically; See<br>\r
- My Computer->Manage->Services and Applications->Services->InfiniBand \r
- subnet Management->Start.<br> </li>\r
- <li>Install OpenSM as a 'running' Windows service:<br>\r
- Request a 'Custom' install, selecting the OpenSM_service install feature. \r
- Once the installation has completed, check the running InfiniBand Subnet \r
- Management service status via the Windows service manager (see #1).<br>\r
- </li>\r
- <li>Consult the OpenSM log file @ %SystemRoot%\temp\osm.log to see what \r
- OpenSM thinks is happening.</li>\r
-</ol>\r
-<p> </p>\r
-<h4 align="left">Manual InfiniBand Subnet Management from a command window</h4>\r
-<p align="left">Usage: opensm.exe [options]</p>\r
-<p align="left">Options:</p>\r
-<blockquote>\r
- <p align="left">-c<br>\r
- --cache-options</p>\r
- <blockquote>\r
- <p align="left">Cache the given command line options into the file<br>\r
- /var/cache/osm/opensm.opts for use next invocation<br>\r
- The cache directory can be changed by the environment<br>\r
- variable OSM_CACHE_DIR</p>\r
- </blockquote>\r
- <p align="left">-g[=]<GUID in hex><br>\r
- --guid[=]<GUID in hex></p>\r
- <blockquote>\r
- <p align="left">This option specifies the local port GUID value with \r
- which OpenSM should bind. OpenSM may be<br>\r
- bound to 1 port at a time. If GUID given is 0, OpenSM displays a \r
- list of possible port GUIDs and waits for user input. Without -g, OpenSM \r
- trys to use the default port.</p>\r
- </blockquote>\r
- <p align="left">-l <LMC><br>\r
- --lmc <LMC></p>\r
- <blockquote>\r
- <p align="left">This option specifies the subnet's LMC value.<br>\r
- The number of LIDs assigned to each port is 2^LMC.<br>\r
- The LMC value must be in the range 0-7.<br>\r
- LMC values > 0 allow multiple paths between ports.<br>\r
- LMC values > 0 should only be used if the subnet<br>\r
- topology actually provides multiple paths between<br>\r
- ports, i.e. multiple interconnects between switches.<br>\r
- Without -l, OpenSM defaults to LMC = 0, which allows<br>\r
- one path between any two ports.</p>\r
- </blockquote>\r
- <p align="left">-p <PRIORITY><br>\r
- --priority <PRIORITY></p>\r
- <blockquote>\r
- <p align="left">This option specifies the SM's PRIORITY.<br>\r
- This will effect the handover cases, where master<br>\r
- is chosen by priority and GUID.<br>\r
- -smkey <SM_Key><br>\r
- This option specifies the SM's SM_Key (64 bits).<br>\r
- This will effect SM authentication.</p>\r
- </blockquote>\r
- <p align="left">-r<br>\r
- --reassign_lids</p>\r
- <blockquote>\r
- <p align="left"><br>\r
- This option causes OpenSM to reassign LIDs to all end nodes. Specifying \r
- -r on a running subnet<br>\r
- may disrupt subnet traffic. Without -r, OpenSM attempts to \r
- preserve existing LID assignments resolving multiple use of same LID.</p>\r
- </blockquote>\r
- <p align="left">-u<br>\r
- --updn</p>\r
- <blockquote>\r
- <p align="left">This option activate UPDN algorithm instead of Min Hop \r
- algorithm (default).</p>\r
- </blockquote>\r
- <p align="left">-a<br>\r
- --add_guid_file <path to file></p>\r
- <blockquote>\r
- <p align="left">Set the root nodes for the Up/Down routing algorithm to \r
- the guids provided in the given file (one per line)</p>\r
- </blockquote>\r
- <p align="left">-o<br>\r
- --once</p>\r
- <blockquote>\r
- <p align="left">This option causes OpenSM to configure the subnet once, \r
- then exit. Ports remain in the ACTIVE state.</p>\r
- </blockquote>\r
- <p align="left">-s <interval><br>\r
- --sweep <interval></p>\r
- <blockquote>\r
- <p align="left">This option specifies the number of seconds between \r
- subnet sweeps. Specifying -s 0 disables sweeping.<br>\r
- Without -s, OpenSM defaults to a sweep interval of 10 seconds.</p>\r
- </blockquote>\r
- <p align="left">-t <milliseconds><br>\r
- --timeout <milliseconds></p>\r
- <blockquote>\r
- <p align="left">This option specifies the time in milliseconds<br>\r
- used for transaction timeouts.<br>\r
- Specifying -t 0 disables timeouts.<br>\r
- Without -t, OpenSM defaults to a timeout value of<br>\r
- 200 milliseconds.</p>\r
- </blockquote>\r
- <p align="left">-maxsmps <number></p>\r
- <blockquote>\r
- <p align="left">This option specifies the number of VL15 SMP MADs \r
- allowed on the wire at any one time.<br>\r
- Specifying -maxsmps 0 allows unlimited outstanding SMPs.<br>\r
- Without -maxsmps, OpenSM defaults to a maximum of one outstanding SMP.</p>\r
- </blockquote>\r
- <p align="left">-i <equalize-ignore-guids-file><br>\r
- -ignore-guids <equalize-ignore-guids-file></p>\r
- <blockquote>\r
- <p align="left">This option provides the means to define a set of ports\r
- (by guids) that will be ignored by the link load \r
- equalization algorithm.</p>\r
- </blockquote>\r
- <p align="left">-x<br>\r
- --honor_guid2lid</p>\r
- <blockquote>\r
- <p align="left">This option forces OpenSM to honor the guid2lid file,\r
- when it comes out of Standby state, if such file exists\r
- under OSM_CACHE_DIR, and is valid.\r
- By default this is FALSE.</p>\r
- </blockquote>\r
- <p align="left">-f<br>\r
- --log_file</p>\r
- <blockquote>\r
- <p align="left">This option defines the log to be the given file. By \r
- default the log goes to %SystemRoot%\temp\osm.log.<br>\r
- For the log to go to standard output use -f stdout.</p>\r
- </blockquote>\r
- <p align="left">-e<br>\r
- --erase_log_file</p>\r
- <blockquote>\r
- <p align="left">This option will cause deletion of the log file \r
- (if it previously exists). By default, the log file is accumulative.</p>\r
- </blockquote>\r
- <p align="left">-y<br>\r
- --stay_on_fatal</p>\r
- <blockquote>\r
- <p align="left">This option will cause SM not to exit on fatal \r
- initialization\r
- issues: if SM discovers duplicated guids or 12x link with\r
- lane reversal badly configured.\r
- By default, the SM will exit on these errors.</p>\r
- </blockquote>\r
- <p align="left">-v<br>\r
- --verbose</p>\r
- <blockquote>\r
- <p align="left">This option increases the log verbosity level.\r
- The -v option may be specified multiple times\r
- to further increase the verbosity level. \r
- See the -vf option for more information about.\r
- log verbosity.</p>\r
- </blockquote>\r
- <p align="left">-V</p>\r
- <blockquote>\r
- <p align="left">This option sets the maximum verbosity level and\r
- forces log flushing.<br>\r
- The -V is equivalent to '-vf 0xFF -d 2'.\r
- See the -vf option for more information about\r
- log verbosity.</p>\r
- </blockquote>\r
- <p align="left">-D <flags></p>\r
- <blockquote>\r
- <p align="left">This option sets the log verbosity level. A flags \r
- field must follow the -D option.<br>\r
- A bit set/clear in the flags enables/disables a specific log level as \r
- follows:<br>\r
- BIT LOG LEVEL ENABLED<br>\r
- ---- -----------------<br>\r
- 0x01 - ERROR (error messages)<br>\r
- 0x02 - INFO (basic messages, low volume)<br>\r
- 0x04 - VERBOSE (interesting stuff, moderate volume)<br>\r
- 0x08 - DEBUG (diagnostic, high volume)<br>\r
- 0x10 - FUNCS (function entry/exit, very high volume)<br>\r
- 0x20 - FRAMES (dumps all SMP and GMP frames)<br>\r
- 0x40 - ROUTING (dump FDB routing information)<br>\r
- 0x80 - currently unused.<br>\r
- Without -D, OpenSM defaults to ERROR + INFO (0x3).<br>\r
- Specifying -D 0 disables all messages.<br>\r
- Specifying -D 0xFF enables all messages (see -V).<br>\r
- High verbosity levels may require increasing the transaction timeout \r
- with the -t option.</p>\r
- </blockquote>\r
- <p align="left">-d <number><br>\r
- --debug <number></p>\r
- <blockquote>\r
- <p align="left">This option specifies a debug option. These options are \r
- not normally needed. The number following -d selects the debug option to \r
- enable as follows:<br>\r
- OPT Description<br>\r
- --- -----------------<br>\r
- -d0 - Ignore other SM nodes<br>\r
- -d1 - Force single threaded dispatching<br>\r
- -d2 - Force log flushing after each log message<br>\r
- -d3 - Disable multicast support<br>\r
- -d4 - Put OpenSM in memory tracking mode<br>\r
- -d10 - Put OpenSM in testability mode<br>\r
- Without -d, no debug options are enabled</p>\r
- </blockquote>\r
- <p align="left">-h<br>\r
- --help</p>\r
- <blockquote>\r
- <p align="left">Display this usage info then exit.</p>\r
- </blockquote>\r
- <p align="left">-?</p>\r
- <blockquote>\r
- <p align="left">Display this usage info then exit.</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<h3 align="left"> </h3>\r
-<h3 align="left"><a name="osmtest"></a>Osmtest - Subnet Management Tests</h3>\r
-<p align="left">Invoke open subnet management tests. osmtest currently can not \r
-run on the same HCA port which OpenSM is currently using.</p>\r
-<blockquote>\r
- <p align="left"> Usage: osmtest [options]</p>\r
- <p align="left">Options:</p>\r
- <blockquote>\r
- <p align="left"> -f <c|a|v|s|e|f|m|q|t><br>\r
- --flow <c|a|v|s|e|f|m|q|t></p>\r
-\r
- <p align="left">This option directs osmtest to run a specific flow:</p>\r
- <p align="left">FLOW DESCRIPTIONS<br>\r
- c = create an inventory file with all nodes, ports & paths.<br>\r
- a = run all validation tests (expecting an input inventory)<br>\r
- v = only validate the given inventory file.<br>\r
- s = run service registration, un-registration and lease.<br>\r
- e = run event forwarding test.<br>\r
- f = flood the SA with queries accoring to the stress mode.<br>\r
- m = multicast flow.<br>\r
- q = QoS info - VLArb and SLtoVL tables.<br>\r
- t = run trap 64/65 flow; requires running an external tool.<br>\r
- (default is all but QoS).</p>\r
- \r
- <p align="left">-w <trap_wait_time><br>\r
- --wait <trap_wait_time></p>\r
- <blockquote>\r
- <p align="left">This option specifies the wait time for trap 64/65 \r
- in seconds.<br>\r
- It is used only when running -f t - the trap 64/65 flow<br>\r
- (default to 10 sec).</p>\r
- </blockquote>\r
- <p align="left">-d <number><br>\r
- --debug <number></p>\r
- <blockquote>\r
- <p align="left">This option specifies a debug option.\r
- These options are not normally needed.<br>\r
- The number following -d selects the debug\r
- option to enable as follows:<br>\r
- OPT Description<br>\r
- --- -----------------<br>\r
- -d0 - Unused.<br>\r
- -d1 - Do not scan/compare path records.<br>\r
- -d2 - Force log flushing after each log message.<br>\r
- -d3 - Use mem tracking.<br>\r
- Without -d, no debug options are enabled.</p>\r
- </blockquote>\r
- <p align="left">-m <LID in hex><br>\r
- --max_lid <LID in hex></p>\r
- <blockquote>\r
- <p align="left">This option specifies the maximal LID number to be \r
- searched\r
- for during inventory file build (default to 100).</p>\r
- </blockquote>\r
- <p align="left">-g <GUID in hex><br>\r
- --guid <GUID in hex></p>\r
- <blockquote>\r
- <p align="left">This option specifies the local port GUID value\r
- with which osmtest should bind. osmtest may be\r
- bound to 1 port at a time.\r
- Without -g, osmtest displays a menu of possible\r
- port GUIDs and waits for user input.</p>\r
- </blockquote>\r
- <p align="left">-h<br>\r
- --help</p>\r
- <blockquote>\r
- <p align="left">Display this usage info then exit.</p>\r
- </blockquote>\r
- <p align="left">-i <filename><br>\r
- --inventory <filename></p>\r
- <blockquote>\r
- <p align="left">This option specifies the name of the inventory \r
- file.\r
- Normally, osmtest expects to find an inventory file,\r
- which osmtest uses to validate real-time information\r
- received from the SA during testing.\r
- If -i is not specified, osmtest defaults to the file\r
- 'osmtest.dat'.<br>\r
- See the -c option for related information.</p>\r
- </blockquote>\r
- <p align="left">-s<br>\r
- --stress</p>\r
- <blockquote>\r
- <p align="left">This option runs the specified stress test instead \r
- of the normal test suite.<br>\r
- Stress test options are as follows:<br>\r
- OPT Description<br>\r
- --- -----------------<br>\r
- -s1 - Single-MAD response SA queries .<br>\r
- -s2 - Multi-MAD (RMPP) response SA queries.<br>\r
- -s3 - Multi-MAD (RMPP) Path Record SA queries.<br>\r
- Without -s, stress testing is not performed.</p>\r
- </blockquote>\r
- <p align="left">-M<br>\r
- --Multicast_Mode</p>\r
- <blockquote>\r
- <p align="left">This option specify length of Multicast test :<br>\r
- OPT Description<br>\r
- --- -----------------<br>\r
- -M1 - Short Multicast Flow (default) - single mode.<br>\r
- -M2 - Short Multicast Flow - multiple mode.<br>\r
- -M3 - Long Multicast Flow - single mode.<br>\r
- -M4 - Long Multicast Flow - multiple mode.<br>\r
- Single mode - Osmtest is tested alone , with no other <br>\r
- apps that interact vs. OpenSM MC.<br>\r
- Multiple mode - Could be run with other apps using MC vs.<br>\r
- OpenSM. Without -M, default flow testing is performed.</p>\r
- </blockquote>\r
- <p align="left">-t <milliseconds></p>\r
- <blockquote>\r
- <p align="left">This option specifies the time in milliseconds used \r
- for transaction timeouts.<br>\r
- Specifying -t 0 disables timeouts.<br>\r
- Without -t, osmtest defaults to a timeout value of 1 second.</p>\r
- </blockquote>\r
- <p align="left">-l<br>\r
- --log_file</p>\r
- <blockquote>\r
- <p align="left">This option defines the log to be the given file.<br>\r
- By default the log goes to stdout.</p>\r
- </blockquote>\r
- <p align="left">-v</p>\r
- <blockquote>\r
- <p align="left">This option increases the log verbosity level. The \r
- -v option may be specified multiple times<br>\r
- to further increase the verbosity level. See the -vf option for more \r
- information about log verbosity.</p>\r
- </blockquote>\r
- <p align="left">-V</p>\r
- <blockquote>\r
- <p align="left">This option sets the maximum verbosity level and \r
- forces log flushing.<br>\r
- The -V is equivalent to '-vf 0xFF -d 2'.<br>\r
- See the -vf option for more information about log verbosity.</p>\r
- </blockquote>\r
- <p align="left">-vf <flags></p>\r
- <blockquote>\r
- <p align="left">This option sets the log verbosity level. A flags \r
- field must follow the -vf option.<br>\r
- A bit set/clear in the flags enables/disables a specific log level \r
- as follows:<br>\r
- BIT LOG LEVEL ENABLED<br>\r
- ---- -----------------<br>\r
- 0x01 - ERROR (error messages)<br>\r
- 0x02 - INFO (basic messages, low volume)<br>\r
- 0x04 - VERBOSE (interesting stuff, moderate volume)<br>\r
- 0x08 - DEBUG (diagnostic, high volume)<br>\r
- 0x10 - FUNCS (function entry/exit, very high volume)<br>\r
- 0x20 - FRAMES (dumps all SMP and GMP frames)<br>\r
- 0x40 - currently unused.<br>\r
- 0x80 - currently unused.<br>\r
- Without -vf, osmtest defaults to ERROR + INFO (0x3).<br>\r
- Specifying -vf 0 disables all messages.<br>\r
- Specifying -vf 0xFF enables all messages (see -V).<br>\r
- High verbosity levels may require increasing<br>\r
- the transaction timeout with the -t option.</p>\r
- </blockquote>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<h3 align="left"> </h3>\r
-<h3 align="left"><br>\r
-<a name="ibtrapgen"></a>ibtrapgen - Generate Infiniband subnet management traps</h3>\r
-<p align="left">Usage: ibtrapgen -t|--trap_num <TRAP_NUM> -n|--number <NUM_TRAP_CREATIONS><br>\r
- \r
--r|--rate <TRAP_RATE> -l|--lid <LIDADDR> <br>\r
- \r
--s|--src_port <SOURCE_PORT> -p|--port_num <PORT_NUM><br>\r
-<br>\r
-Options: one of the following optional flows:</p>\r
-<blockquote>\r
- <p align="left">-t <TRAP_NUM><br>\r
- --trap_num <TRAP_NUM><br>\r
- This option specifies the \r
- number of the trap to generate. Valid values are 128-131.<br>\r
- -n <NUM_TRAP_CREATIONS><br>\r
- --number <NUM_TRAP_CREATIONS><br>\r
- This option specifies the \r
- number of times to generate this trap.<br>\r
- If not specified - \r
- default to 1.<br>\r
- -r <TRAP_RATE><br>\r
- --rate <TRAP_RATE><br>\r
- This option specifies the \r
- rate of the trap generation.<br>\r
- What is the time period \r
- between one generation and another?<br>\r
- The value is given in \r
- miliseconds. <br>\r
- If the number of trap \r
- creations is 1 - this value is ignored.<br>\r
- -l <LIDADDR><br>\r
- --lid <LIDADDR><br>\r
- This option specifies the \r
- lid address from where the trap should be generated.<br>\r
- -s <SOURCE_PORT><br>\r
- --src_port <SOURCE_PORT><br>\r
- This option specifies the \r
- port number from which the trap should<br>\r
- be generated. If trap \r
- number is 128 - this value is ignored (since<br>\r
- trap 128 is not sent with \r
- a specific port number)<br>\r
- -p <port num><br>\r
- --port_num <port num><br>\r
- This is the port number \r
- used for communicating with the SA.<br>\r
- -h<br>\r
- --help<br>\r
- Display this usage info \r
- then exit.<br>\r
- -o<br>\r
- --out_log_file<br>\r
- This option defines the \r
- log to be the given file.<br>\r
- By default the log goes \r
- to stdout.<br>\r
- -v<br>\r
- This option increases the \r
- log verbosity level.<br>\r
- The -v option may be \r
- specified multiple times to further increase the verbosity level.<br>\r
- See the -vf option for \r
- more information about log verbosity.<br>\r
- -V<br>\r
- This option sets the \r
- maximum verbosity level and forces log flushing.<br>\r
- The -V is equivalent to \r
- '-vf 0xFF -d 2'.<br>\r
- See the -vf option for \r
- more information about. log verbosity.<br>\r
- -x <flags><br>\r
- This option sets the log \r
- verbosity level.<br>\r
- A flags field must follow \r
- the -vf option.<br>\r
- A bit set/clear in the \r
- flags enables/disables a<br>\r
- specific log level as \r
- follows:</p>\r
- <blockquote>\r
- <p align="left">BIT LOG LEVEL ENABLED<br>\r
- ---- -----------------<br>\r
- 0x01 - ERROR (error messages)<br>\r
- 0x02 - INFO (basic messages, low volume)<br>\r
- 0x04 - VERBOSE (interesting stuff, moderate volume)<br>\r
- 0x08 - DEBUG (diagnostic, high volume)<br>\r
- 0x10 - FUNCS (function entry/exit, very high volume)<br>\r
- 0x20 - FRAMES (dumps all SMP and GMP frames)<br>\r
- 0x40 - currently unused.<br>\r
- 0x80 - currently unused.<br>\r
- Without -x, ibtrapgen defaults to ERROR + INFO (0x3).<br>\r
- Specifying -x 0 disables all messages.<br>\r
- Specifying -x 0xFF enables all messages (see -V).</p>\r
- </blockquote>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="IPoIB"></a>IPoIB - Internet Protocols over InfiniBand</h2>\r
-<hr>\r
-<p align="left">IPoIB enables the use of Internet Protocol utilities (e.g., ftp, \r
-telnet) to function correctly over an Infiniband fabric. IPoIB is implemented as \r
-an NDIS Miniport driver with a WDM lower edge.</p>\r
-<p align="left">The IPoIB Network adapters are \r
-located via 'My Computer->Manage->Device Manager->Network adapters->IPoIB'.<br>\r
-'My \r
-Network Places->Properties' will display IPoIB Local Area Connection instances and should be used to \r
-configure IP addresses for the IPoIB interfaces; one Local Area Connection \r
-instance per HCA port. The IP \r
-(Internet Protocol) address bound to the IPoIB adapter instance can be assigned \r
-by DHCP or as a static IP addresses via<br>\r
-'My Network Places->Properties->Local \r
-Area Connection X->Properties->(General Tab)Internet Protocol(TCP/IP)->Properties'.</p>\r
-<p align="left">When the subnet manager (opensm) configures/sweeps the local \r
-Infiniband HCA, the Local Area Connection will become enabled. If you discover \r
-the Local Area Connection to be disabled, then likely your subnet manager \r
-(opensm) is not running or functioning correctly.</p>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="winsockdirect"></a>Winsock Direct Service Provider</h2>\r
-<hr>\r
-<p align="left">Winsock Direct (WSD) is Microsoft's proprietary protocol that \r
-predates SDP (Sockets Direct Protocol) for accelerating TCP/IP applications by \r
-using RDMA hardware. Microsoft had a significant role in defining the SDP \r
-protocol, hence SDP and WSD are remarkably similar, though unfortunately \r
-incompatible.<br>\r
-<br>\r
-WSD is made up of two parts, the winsock direct switch and the winsock direct \r
-provider. The WSD switch is in the winsock DLL that ships in all editions of \r
-Windows Server 2003, and is responsible for routing socket traffic over either \r
-the regular TCP/IP stack, or offload it to a WSD provider. The WSD provider is a \r
-hardware specific DLL that implements connection management and data transfers \r
-over particular RDMA hardware.</p>\r
-<p align="left">WinOF WSD is not supported in the Windows XP environment.</p>\r
-<p align="left">The WSD Protocol seamlessly transports TCP \r
- data using Infiniband data packets in 'buffered' mode or Infiniband \r
- RDMA in 'direct' mode. Either way the user mode socket application sees no \r
- behavioral difference in the standard Internet Protocol socket it created other than \r
-reduced data transfer times and increased bandwidth.<br>\r
-<br>\r
-The Windows OpenFabrics release includes a WSD provider library that has been \r
-extensively tested with Microsoft Windows Server 2003.<br>\r
-During testing, bugs where found in the WSD switch that could lead to hangs, \r
-crashes, data corruption, and other unwanted behavior. Microsoft released a \r
-hotfix to address these issues which should be installed if using WSD; the \r
-Microsoft Windows Server 2003 hotfix can be found\r
-<a class="wiki" href="http://support.microsoft.com/?kbid=910481">here</a>.<br>\r
- </p>\r
-<div id="tiki-main">\r
- <div id="tiki-mid">\r
- <table id="table1" cellSpacing="0" cellPadding="0" border="0">\r
- <tr>\r
- <td id="centercolumn">\r
- <div id="tiki-center">\r
- <div class="wikitext">\r
- Environment variables can be used to change the behavior \r
- of the WSD provider:<br>\r
- <br>\r
- IBWSD_NO_READ - Disables RDMA Read operations when set \r
- to any value. Note that this variable must be used \r
- consistently throughout the cluster or communication \r
- will fail.<br>\r
- <br>\r
- IBWSD_POLL - Sets the number of times to poll the \r
- completion queue after processing completions in \r
- response to a CQ event. Reduces latency at the cost of \r
- CPU utilization. Default is 500.<br>\r
- <br>\r
- IBWSD_SA_RETRY - Sets the number of times to retry SA \r
- query requests. Default is 4, can be increased if \r
- connection establishment fails.<br>\r
- <br>\r
- IBWSD_SA_TIMEOUT - Sets the number of milliseconds to \r
- wait before retrying SA query requests. Default is 4, \r
- can be increased if connection establishment fails.<br>\r
- <br>\r
- IBWSD_NO_IPOIB - SA query timeouts by default allow the \r
- connection to be established over IPoIB. Setting this \r
- environment variable to any value prevents fall back to \r
- IPoIB if SA queries time out.<br>\r
- <br>\r
- IBWSD_DBG - Controls debug output when using a debug \r
- version of the WSD provider. Takes a hex value, with \r
- leading '0x', default value is '0x80000000'<br>\r
- <br>\r
- <table class="wikitable" id="table2">\r
- <tr>\r
- <td class="wikicell">0x00000001</td>\r
- <td class="wikicell">DLL</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000002</td>\r
- <td class="wikicell">socket info</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000004</td>\r
- <td class="wikicell">initialization code</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000008</td>\r
- <td class="wikicell">WQ related functions</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000010</td>\r
- <td class="wikicell">Enpoints related functions</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000020</td>\r
- <td class="wikicell">memory registration</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000040</td>\r
- <td class="wikicell">CM (Connection Manager)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000080</td>\r
- <td class="wikicell">connections</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000200</td>\r
- <td class="wikicell">socket options</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000400</td>\r
- <td class="wikicell">network events</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00000800</td>\r
- <td class="wikicell">Hardware</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00001000</td>\r
- <td class="wikicell">Overlapped I/O request</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00002000</td>\r
- <td class="wikicell">Socket Duplication</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x00004000</td>\r
- <td class="wikicell">Performance Monitoring</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x01000000</td>\r
- <td class="wikicell">More verbose than \r
- IBSP_DBG_LEVEL3</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x02000000</td>\r
- <td class="wikicell">More verbose than \r
- IBSP_DBG_LEVEL2</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x04000000</td>\r
- <td class="wikicell">More verbose than \r
- IBSP_DBG_LEVEL1</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x08000000</td>\r
- <td class="wikicell">Verbose output</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x20000000</td>\r
- <td class="wikicell">Function enter/exit</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x40000000</td>\r
- <td class="wikicell">Warnings</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">0x80000000</td>\r
- <td class="wikicell">Errors</td>\r
- </tr>\r
- </table>\r
- </div>\r
- </div>\r
- </td>\r
- </tr>\r
- </table>\r
- </div>\r
-</div>\r
-<p align="left"><br>\r
-See <a href="https://wiki.openfabrics.org/tiki-index.php?page=Winsock+Direct">\r
-https://wiki.openfabrics.org/tiki-index.php?page=Winsock+Direct</a> for the \r
-latest WSD status.</p>\r
-<h3 align="left">Winsock Direct Service Provider Installation</h3>\r
-<p align="left">When the custom install option 'WSD' is selected the WSD service \r
-is automatically installed and started as part of the installation.<br>\r
-Manual control is performed via the \Program Files\WinOF\installsp.exe utility.</p>\r
-<blockquote>\r
- <p align="left">usage: installsp [-i | -r | -l]<br>\r
- <br>\r
- -i Install the Winsock Direct (WSD) service provider<br>\r
- -r Remove the WSD service provider<br>\r
- -r <name> Remove the specified service provider<br>\r
- -l List service providers<br>\r
- </p>\r
-</blockquote>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><a name="DAT"></a>Direct Access Transport and usermode Direct Access Programming \r
-Libraries</h2>\r
-<hr>\r
-<p align="left"><font face="Times New Roman">The DAT (Direct Access Transport) \r
-API is a C programming interface developed by the\r
-<a class="wiki" href="http://www.datcollaborative.org">DAT Collaborative</a> in \r
-order provide a set of transport-independent, platform-independent Application \r
-Programming Interfaces that exploit the RDMA (remote direct memory access) \r
-capabilities of next-generation interconnect technologies such as InfiniBand, \r
-and iWARP.</font></p>\r
-<p align="left">WinOF DAT and uDAPL are based on the 1.1 DAT specification. The DAPL \r
-(Direct Access Provider Library) now fully supports Infiniband RDMA and \r
-IPoIB.</p>\r
-<div id="tiki-main0">\r
- <div id="tiki-mid0">\r
- <table id="table3" cellSpacing="0" cellPadding="0" border="0">\r
- <tr>\r
- <td id="centercolumn0">\r
- <div id="tiki-center0">\r
- <div class="wikitext">\r
- <br>\r
- How DAT objects map to equivalent\r
- <a class="wiki" href="http://en.wikipedia.org/wiki/Infiniband">\r
- InfiniBand</a> objects:<br>\r
- <table class="wikitable" id="table4">\r
- <tr>\r
- <td class="wikicell">Interface Adapter (IA) \r
- </td>\r
- <td class="wikicell">HCA (Host Channel Adapter)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Protection Zone (PZ) </td>\r
- <td class="wikicell">PD (Protection Domain)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Local Memory Region (LMR) \r
- </td>\r
- <td class="wikicell">MR (Memory Region)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Remote Memory Region (RMR) \r
- </td>\r
- <td class="wikicell">MW (Memory Windows)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Event Dispatcher (EVD) \r
- </td>\r
- <td class="wikicell">CQ (Completion Queue)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Endpoint (EP) </td>\r
- <td class="wikicell">QP (Queue Pair)</td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Public Service Point (PSP) \r
- </td>\r
- <td class="wikicell">connection identifier </td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Reserved Service Point (RSP) \r
- </td>\r
- <td class="wikicell">connection identifier </td>\r
- </tr>\r
- <tr>\r
- <td class="wikicell">Connection Request (CR) \r
- </td>\r
- <td class="wikicell">connection manager event \r
- </td>\r
- </tr>\r
- </table>\r
- </div>\r
- </div>\r
- </td>\r
- </tr>\r
- </table>\r
- </div>\r
-</div>\r
-<p align="left"> </p>\r
-<div align="left">\r
- <h3 align="left"><font face="Courier New" size="3">DAT EXECUTION ENVIRONMENT</font><font face="Courier New" size="2">:</font></h3>\r
-</div>\r
-<blockquote>\r
- <p align="left"><font face="Courier New" size="2">In order for DAT/uDAPL \r
- programs to execute correctly, the 'dat.dll' file must be present in the \r
- current directory,\r
- %SystemRoot%\system32, %SystemRoot% or in the library search path.</font></p>\r
- <p align="left"><font face="Courier New" size="2">The default WinOF \r
- installation places the file dat.dll in the '%SystemRoot%' folder.</font></p>\r
- <p align="left"><font face="Courier New" size="2">The DAPL configuration \r
- file by default is defined as '%SystemDrive%\DAT\dat.conf'. This default \r
- specification can be overriden by use of the environment variable \r
- DAT_OVERRIDE; see following environment variable discussion.</font></p>\r
- <p align="left"><font face="Courier New" size="2">Within the dat.conf file, \r
- the DAPL library specification can be located as the 5th whitespace \r
- separated line argument. By default the DAPL library file is installed as<br>\r
- '%SystemRoot%\dapl.dll'.</font></p>\r
- <p align="left"><font face="Courier New" size="2">Should you choose to \r
- relocated the DAPL library file to a path where whitespace appears in the \r
- full library path specification, then the full library file specification \r
- must be contained within double-quotes. A side effect of the double-quotes \r
- is the library specification is treated as a Windows string which implies \r
- the '\' (backslash character) is treated as an 'escape' character. Hence \r
- all backslashes in the library path must be duplicated when enclosed in \r
- double-quotes\r
- (e.g., "C:\\Programs Files\\WinOF\\dapl.dll").</font></p>\r
- <p align="left"><font face="Courier New" size="2">A sample dat.conf file is \r
- installed as '\Program Files\WinOF\dat.conf '.<br>\r
- After the WinOF installation, move the \Program Files\WinOF\dat.conf file to \r
- the default DAT configuration file location\r
- '%SystemDrive%\DAT\dat.conf'.</font></p>\r
- <p align="left"><font face="Courier New" size="2">In order to preserve existing installations, \r
- the dat.conf file is not automatically installed in it's default location.</font></p>\r
- <p align="left"> </p>\r
- <h4>\r
-<u>DAT library environment variables</u>:<br>\r
- </h4>\r
- <pre>\r
-DAT_OVERRIDE\r
-------------\r
-Value used as the static registry configuration file, overriding the\r
-default location, 'C:\DAT\dat.conf'.\r
-\r
-Example: set DAT_OVERRIDE=%SystemDrive%\path\to\my\private.conf\r
-\r
-\r
-DAT_DBG_TYPE\r
-------------\r
-\r
-Value specifies which parts of the registry will print debugging\r
-information, valid values are \r
-\r
-DAT_OS_DBG_TYPE_ERROR = 0x1\r
-DAT_OS_DBG_TYPE_GENERIC = 0x2\r
-DAT_OS_DBG_TYPE_SR = 0x4\r
-DAT_OS_DBG_TYPE_DR = 0x8\r
-DAT_OS_DBG_TYPE_PROVIDER_API = 0x10\r
-DAT_OS_DBG_TYPE_CONSUMER_API = 0x20\r
-DAT_OS_DBG_TYPE_ALL = 0xff\r
-\r
-or any combination of these. For example you can use 0xC to get both \r
-static and dynamic registry output.\r
-\r
-Example set DAT_DBG_TYPE=0xC\r
-\r
-DAT_DBG_DEST\r
------------- \r
-\r
-Value sets the output destination, valid values are \r
-\r
-DAT_OS_DBG_DEST_STDOUT = 0x1\r
-DAT_OS_DBG_DEST_SYSLOG = 0x2 \r
-DAT_OS_DBG_DEST_ALL = 0x3 \r
-\r
-For example, 0x3 will output to both stdout and the syslog. \r
-</pre>\r
- <h4>\r
-<u>DAPL Provider library environment variables</u></h4>\r
- <p>\r
-<br>\r
-<font size="2">DAPL_DBG_TYPE<br>\r
--------------<br>\r
-<br>\r
-<font face="Courier New">Value specifies which parts of the registry will print \r
-debugging information, valid values are</font> <br>\r
-<br>\r
-</font><font face="Courier New"><font size="2">DAPL_DBG_TYPE_ERR = 0x0001<br>\r
-DAPL_DBG_TYPE_WARN = 0x0002<br>\r
-DAPL_DBG_TYPE_EVD = 0x0004<br>\r
-DAPL_DBG_TYPE_CM = 0x0008<br>\r
-DAPL_DBG_TYPE_EP = 0x0010<br>\r
-DAPL_DBG_TYPE_UTIL = 0x0020<br>\r
-DAPL_DBG_TYPE_CALLBACK = 0x0040<br>\r
-DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080<br>\r
-DAPL_DBG_TYPE_API = 0x0100<br>\r
-DAPL_DBG_TYPE_RTN = 0x0200<br>\r
-DAPL_DBG_TYPE_EXCEPTION = 0x0400<br>\r
-<br>\r
-or any combination of these. For example you can use 0xC to get both <br>\r
-EVD and CM output.<br>\r
-<br>\r
-Example set DAPL_DBG_TYPE=0xC<br>\r
-<br>\r
-<br>\r
-DAPL_DBG_DEST<br>\r
--------------<br>\r
-<br>\r
-Value sets the output destination, valid values are <br>\r
-<br>\r
-DAPL_DBG_DEST_STDOUT = 0x1<br>\r
-DAPL_DBG_DEST_SYSLOG = 0x2 <br>\r
-DAPL_DBG_DEST_ALL = 0x3 <br>\r
-<br>\r
-For example, 0x3 will output to both stdout and the syslog.</font> </font><br>\r
-<br>\r
-<br>\r
-\r
-</p>\r
-</blockquote><hr>\r
-<h3><a name="DAPLTEST"></a>DAPLTEST</h3>\r
-<pre>\r
-\r
- dapltest - test for the Direct Access Provider Library (DAPL)\r
-\r
-DESCRIPTION\r
-\r
- Dapltest is a set of tests developed to exercise, characterize,\r
- and verify the DAPL interfaces during development and porting.\r
- At least two instantiations of the test must be run. One acts\r
- as the server, fielding requests and spawning server-side test\r
- threads as needed. Other client invocations connect to the\r
- server and issue test requests.\r
-\r
- The server side of the test, once invoked, listens continuously\r
- for client connection requests, until quit or killed. Upon\r
- receipt of a connection request, the connection is established,\r
- the server and client sides swap version numbers to verify that\r
- they are able to communicate, and the client sends the test\r
- request to the server. If the version numbers match, and the\r
- test request is well-formed, the server spawns the threads\r
- needed to run the test before awaiting further connections.\r
-\r
-USAGE\r
-\r
- dapltest [ -f script_file_name ]\r
- [ -T S|Q|T|P|L ] [ -D device_name ] [ -d ] [ -R HT|LL|EC|PM|BE ]\r
-\r
- With no arguments, dapltest runs as a server using default values,\r
- and loops accepting requests from clients. The -f option allows\r
- all arguments to be placed in a file, to ease test automation.\r
- The following arguments are common to all tests:\r
-\r
- [ -T S|Q|T|P|L ] Test function to be performed:\r
- S - server loop\r
- Q - quit, client requests that server\r
- wait for any outstanding tests to\r
- complete, then clean up and exit\r
- T - transaction test, transfers data between \r
- client and server\r
- P - performance test, times DTO operations\r
- L - limit test, exhausts various resources,\r
- runs in client w/o server interaction\r
- Default: S\r
-\r
- [ -D device_name ] Specifies the name of the device (interface adapter).\r
- Default: host-specific, look for DT_MdepDeviceName\r
- in dapl_mdep.h\r
-\r
- [ -d ] Enables extra debug verbosity, primarily tracing\r
- of the various DAPL operations as they progress.\r
- Repeating this parameter increases debug spew.\r
- Errors encountered result in the test spewing some\r
- explanatory text and stopping; this flag provides\r
- more detail about what lead up to the error.\r
- Default: zero\r
-\r
- [ -R BE ] Indicate the quality of service (QoS) desired.\r
- Choices are:\r
- HT - high throughput\r
- LL - low latency\r
- EC - economy (neither HT nor LL)\r
- PM - premium\r
- BE - best effort\r
- Default: BE\r
-\r
-USAGE - Quit test client\r
-\r
- dapltest [Common_Args] [ -s server_name ]\r
-\r
- Quit testing (-T Q) connects to the server to ask it to clean up and\r
- exit (after it waits for any outstanding test runs to complete).\r
- In addition to being more polite than simply killing the server,\r
- this test exercises the DAPL object teardown code paths.\r
- There is only one argument other than those supported by all tests:\r
-\r
- -s server_name Specifies the name of the server interface.\r
- No default.\r
-\r
-\r
-USAGE - Transaction test client\r
-\r
- dapltest [Common_Args] [ -s server_name ]\r
- [ -t threads ] [ -w endpoints ] [ -i iterations ] [ -Q ] \r
- [ -V ] [ -P ] OPclient OPserver [ op3, \r
-\r
- Transaction testing (-T T) transfers a variable amount of data between \r
- client and server. The data transfer can be described as a sequence of \r
- individual operations; that entire sequence is transferred 'iterations' \r
- times by each thread over all of its endpoint(s).\r
-\r
- The following parameters determine the behavior of the transaction test:\r
-\r
- -s server_name Specifies the hostname of the dapltest server.\r
- No default.\r
-\r
- [ -t threads ] Specify the number of threads to be used.\r
- Default: 1\r
-\r
- [ -w endpoints ] Specify the number of connected endpoints per thread.\r
- Default: 1\r
-\r
- [ -i iterations ] Specify the number of times the entire sequence\r
- of data transfers will be made over each endpoint.\r
- Default: 1000\r
-\r
- [ -Q ] Funnel completion events into a CNO.\r
- Default: use EVDs\r
-\r
- [ -V ] Validate the data being transferred.\r
- Default: ignore the data\r
-\r
- [ -P ] Turn on DTO completion polling\r
- Default: off\r
-\r
- OP1 OP2 [ OP3, ... ]\r
- A single transaction (OPx) consists of:\r
-\r
- server|client Indicates who initiates the\r
- data transfer.\r
-\r
- SR|RR|RW Indicates the type of transfer:\r
- SR send/recv\r
- RR RDMA read\r
- RW RDMA write\r
- Defaults: none\r
-\r
- [ seg_size [ num_segs ] ]\r
- Indicates the amount and format\r
- of the data to be transferred.\r
- Default: 4096 1\r
- (i.e., 1 4KB buffer)\r
-\r
- [ -f ] For SR transfers only, indicates\r
- that a client's send transfer\r
- completion should be reaped when\r
- the next recv completion is reaped.\r
- Sends and receives must be paired\r
- (one client, one server, and in that\r
- order) for this option to be used.\r
-\r
- Restrictions: \r
- \r
- Due to the flow control algorithm used by the transaction test, there \r
- must be at least one SR OP for both the client and the server. \r
-\r
- Requesting data validation (-V) causes the test to automatically append \r
- three OPs to those specified. These additional operations provide \r
- synchronization points during each iteration, at which all user-specified \r
- transaction buffers are checked. These three appended operations satisfy \r
- the "one SR in each direction" requirement.\r
-\r
- The transaction OP list is printed out if -d is supplied.\r
-\r
-USAGE - Performance test client\r
-\r
- dapltest [Common_Args] -s server_name [ -m p|b ]\r
- [ -i iterations ] [ -p pipeline ] OP\r
-\r
- Performance testing (-T P) times the transfer of an operation.\r
- The operation is posted 'iterations' times.\r
-\r
- The following parameters determine the behavior of the transaction test:\r
-\r
- -s server_name Specifies the hostname of the dapltest server.\r
- No default.\r
-\r
- -m b|p Used to choose either blocking (b) or polling (p)\r
- Default: blocking (b)\r
-\r
- [ -i iterations ] Specify the number of times the entire sequence\r
- of data transfers will be made over each endpoint.\r
- Default: 1000\r
-\r
- [ -p pipeline ] Specify the pipline length, valid arguments are in \r
- the range [0,MAX_SEND_DTOS]. If a value greater than \r
- MAX_SEND_DTOS is requested the value will be\r
- adjusted down to MAX_SEND_DTOS.\r
- Default: MAX_SEND_DTOS\r
-\r
- OP\r
- An operation consists of:\r
-\r
- RR|RW Indicates the type of transfer:\r
- RR RDMA read\r
- RW RDMA write\r
- Default: none\r
-\r
- [ seg_size [ num_segs ] ]\r
- Indicates the amount and format\r
- of the data to be transferred.\r
- Default: 4096 1\r
- (i.e., 1 4KB buffer)\r
-\r
-USAGE - Limit test client\r
-\r
- Limit testing (-T L) neither requires nor connects to any server\r
- instance. The client runs one or more tests which attempt to\r
- exhaust various resources to determine DAPL limits and exercise\r
- DAPL error paths. If no arguments are given, all tests are run.\r
-\r
- Limit testing creates the sequence of DAT objects needed to\r
- move data back and forth, attempting to find the limits supported\r
- for the DAPL object requested. For example, if the LMR creation\r
- limit is being examined, the test will create a set of\r
- {IA, PZ, CNO, EVD, EP} before trying to run dat_lmr_create() to\r
- failure using that set of DAPL objects. The 'width' parameter\r
- can be used to control how many of these parallel DAPL object\r
- sets are created before beating upon the requested constructor.\r
- Use of -m limits the number of dat_*_create() calls that will\r
- be attempted, which can be helpful if the DAPL in use supports\r
- essentailly unlimited numbers of some objects.\r
-\r
- The limit test arguments are:\r
-\r
- [ -m maximum ] Specify the maximum number of dapl_*_create()\r
- attempts.\r
- Default: run to object creation failure\r
-\r
- [ -w width ] Specify the number of DAPL object sets to\r
- create while initializing.\r
- Default: 1\r
-\r
- [ limit_ia ] Attempt to exhaust dat_ia_open()\r
-\r
- [ limit_pz ] Attempt to exhaust dat_pz_create()\r
-\r
- [ limit_cno ] Attempt to exhaust dat_cno_create()\r
-\r
- [ limit_evd ] Attempt to exhaust dat_evd_create()\r
-\r
- [ limit_ep ] Attempt to exhaust dat_ep_create()\r
-\r
- [ limit_rsp ] Attempt to exhaust dat_rsp_create()\r
-\r
- [ limit_psp ] Attempt to exhaust dat_psp_create()\r
-\r
- [ limit_lmr ] Attempt to exhaust dat_lmr_create(4KB)\r
-\r
- [ limit_rpost ] Attempt to exhaust dat_ep_post_recv(4KB)\r
-\r
- [ limit_size_lmr ] Probe maximum size dat_lmr_create()\r
-\r
- Default: run all tests\r
-\r
-\r
-<a name="DAPLtest-examples"></a>EXAMPLES\r
-\r
- dapltest -T S -d -D ibnic0\r
-\r
- Starts a local dapltest server process with debug verbosity.\r
- Server loops (listen for dapltest request, process request).\r
- \r
- dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2\r
-\r
- Runs a transaction test, with both sides\r
- sending one buffer with two 4KB segments,\r
- one hundred times; dapltest server is on host winIB.\r
-\r
- dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2\r
-\r
- Runs a performance test, with the client \r
- RDMA writing one buffer with two 4KB segments,\r
- one hundred times.\r
-\r
- dapltest -T Q -s winIB -D ibnic0\r
-\r
- Asks the dapltest server at host 'winIB' to clean up and exit.\r
-\r
- dapltest -T L -D ibnic0 -d -w 16 -m 1000\r
-\r
- Runs all of the limit tests, setting up\r
- 16 complete sets of DAPL objects, and\r
- creating at most a thousand instances\r
- when trying to exhaust resources.\r
-\r
- dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \\r
- client RW 4096 1 server RW 2048 4 \\r
- client SR 1024 4 server SR 4096 2 \\r
- client SR 1024 3 -f server SR 2048 1 -f\r
-\r
- Runs a more complicated transaction test,\r
- with two thread using four EPs each,\r
- sending a more complicated buffer pattern\r
- for a larger number of iterations,\r
- validating the data received.\r
-\r
-\r
-BUGS (and To Do List)\r
-\r
- Use of CNOs (-Q) is not yet supported.\r
-\r
- Further limit tests could be added.</pre>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
-<p align="left"> </p>\r
-<p align="left"> </p>\r
-<h2 align="left"><u><a name="SRP"></a>SRP - SCSI RDMA Protocol</u></h2>\r
-<p align="left"><span style="color: black">\r
-<font face="Times New Roman" size="3">The\r
-<a href="http://infiniband.sourceforge.net/Storage/SRP/overview.htm">SCSI RDMA \r
-Protocol </a> (SRP) is an emerging industry standard protocol for utilizing \r
-block storage devices over an InfiniBand\99 fabric. SRP is being defined in the \r
-ANSI T-10 committee.</font></span></p>\r
-<p align="left"><font face="Times New Roman">WinOF </font>SRP is a storage \r
-driver implementation that enables the SCSI RDMA protocol over an InfiniBand \r
-fabric.<br>\r
-The implementation conforms\r
-to the T10 Working Group draft\r
-<a href="http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf">\r
-http://www.t10.org/ftp/t10/drafts/srp/srp-r16a.pdf</a>.</p>\r
-<h4 align="left"><u>Software Dependencies</u></h4>\r
-<p align="left">The SRP driver depends on the installation of the WinOF stack \r
-with a Subnet<br>\r
-Manager running somewhere on the IB fabric.<br>\r
-<br>\r
-- Supported Operating Systems and Service Packs:<br>\r
- o Windows XP SP2 x64<br>\r
- o Windows Server 2003 SP1 (x86, x64)<br>\r
- o Windows Server 2003 CCS (x64)</p>\r
-<h4 align="left"><br>\r
-<u>Testing Level</u></h4>\r
-<p align="left">The SRP driver has undergone basic testing against Mellanox \r
-Technologies'<br>\r
-SRP Targets MTD1000 and MTD2000. Testing included format, read, and write<br>\r
-operations.<br>\r
- </p>\r
-<h4 align="left"><u>Installation</u></h4>\r
-<p align="left">The WinOF stack does not install the SRP driver by default. \r
-If SRP is\r
-selected in the <i>custom</i> installation window, it will only be copied during WinOF\r
-installation. To complete the SRP driver installation, an SRP target must be\r
-detected; a Subnet Manager must be running somewhere in the IB subnet.<br>\r
-<br>\r
-Upon the detection of an SRP target, the "New Hardware Found" Wizard pops up.<br>\r
-- Select Install Automatically and click Next. This installs the I/O unit \r
-device.<br>\r
-<br>\r
-Once completed, the "New Hardware Found" Wizard pops up again.<br>\r
-- Select Install Automatically and click Next. This installs the SRP driver.<br>\r
- </p>\r
-<h4 align="left"><u>New Features and Changes</u></h4>\r
-<p align="left">- SRP supports WPP tracing tools by using the GUID: \r
-'5AF07B3C-D119-4233-9C81-C07EF481CBE6'.<br>\r
- The flags and level of debug can be controlled at load-time or run-time.<br>\r
- </p>\r
-<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
projects / ~shefty / rdma-win.git / commitdiff