<h1 align="center">User's Manual</h1>\r
<h2 align="center">Release 2.3</h2>\r
<h3 align="center">\r
-<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->05/12/2010<!--webbot bot="Timestamp" endspan i-checksum="12527" --></h3>\r
+<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->06/18/2010<!--webbot bot="Timestamp" endspan i-checksum="12625" --></h3>\r
<h2 align="left"><u>Overview</u></h2>\r
<p align="left"><span style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'">\r
The OpenFabrics Enterprise Distribution for Windows package is composed of software modules intended \r
<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
<p align="left"> </p>\r
<BLOCKQUOTE></BLOCKQUOTE>\r
-<h2 align="left"><a name="opensm"></a>Subnet Management with OpenSM version 3.3.3</h2>\r
+<h2 align="left"><a name="opensm"></a>Subnet Management with OpenSM version \r
+3.3.6</h2>\r
<hr>\r
<p align="left">A single running process (opensm.exe) is required to configure \r
and thus make an Infiniband subnet useable. For most cases, InfiniBand \r
When opensm.exe is run as a Windows Service, the 'normal' \r
case, %temp% is defined as %windir%\TEMP\.<br>\r
If opensm.exe is run from a command window, %TEMP% is <b>not</b> \r
- defined as %windir%\TEMP\; just use %TEMP%<br> </li>\r
+ defined as %windir%\TEMP\.<br> </li>\r
</ol>\r
<h3 align="left">InfiniBand Subnet Management from a command window</h3>\r
+\r
+opensm - InfiniBand subnet manager and administration (SM/SA)\r
<h3>SYNOPSIS</h3>\r
\r
<B>opensm</B>\r
[-console-port <port>]\r
[-i(gnore-guids) <equalize-ignore-guids-file>]\r
[-w | --hop_weights_file <path to file>]\r
-[-f <log file path> | --log_file <log file path> ]\r
+[-O | --dimn_ports_file <path to file>] [-f <log file path> | --log_file <log file path> ]\r
[-L | --log_limit <size in MB>] [-e(rase_log_file)]\r
[-P(config) <partition config file> ]\r
[-N | --no_part_enforce]\r
[--perfmgr_sweep_time_s <seconds>]\r
[--prefix_routes_file <path>]\r
[--consolidate_ipv6_snm_req]\r
-[-v(erbose)] [-V] [-D <flags>] [-d(ebug) <number>]\r
+[--log_prefix <prefix text>] [-v(erbose)] [-V] [-D <flags>] [-d(ebug) <number>]\r
[-h(elp)] [-?]\r
<h3>DESCRIPTION</h3>\r
\r
Administration. Such a software entity is required to run for in order\r
to initialize the InfiniBand hardware (at least one per each\r
InfiniBand subnet).\r
-\r
+<P>\r
opensm also now contains an experimental version of a performance\r
manager as well.\r
-\r
+<P>\r
opensm defaults were designed to meet the common case usage on clusters with up to a few hundred nodes. Thus, in this default mode, opensm will scan the IB\r
fabric, initialize it, and sweep occasionally for changes.\r
-\r
+<P>\r
opensm attaches to a specific IB port on the local machine and configures only\r
the fabric connected to it. (If the local machine has other IB ports,\r
opensm will ignore the fabrics connected to those other ports). If no port is\r
specified, it will select the first "best" available port.\r
-\r
-opensm can present the available ports and prompt for a port number to attach \r
-to. By default, the run is logged to two files:%windir%\temp\osm.syslog and \r
-%windir%\temp\osm.log.\r
+<P>\r
+opensm can present the available ports and prompt for a port number to\r
+attach to.\r
+<P>\r
+By default, the run is logged to two files: %TEMP%\osm.syslog (aka \r
+%windir%\temp\osm.syslog) and %windir%\temp\opensm.log.\r
The first file will register only general major events, whereas the second\r
will include details of reported errors. All errors reported in this second\r
file should be treated as indicators of IB fabric health issues.\r
(Note that when a fatal and non-recoverable error occurs, opensm will exit.)\r
Both log files should include the message "SUBNET UP" if opensm was able to\r
-setup the subnet correctly.<h3>OPTIONS</h3>\r
-\r
+setup the subnet correctly. Note when opensm.exe is run as a service, %TEMP% == \r
+%windir%\temp .<h3>OPTIONS</h3>\r
\r
+<P>\r
<P>\r
\r
<DL COMPACT>\r
Prints OpenSM version and exits.\r
<DT><B>-F</B>, <B>--config</B> <config file><DD>\r
The name of the OpenSM config file. When not specified\r
-<b>%ProgramFiles%\OFED\OpenSM\opensm.conf</b> will be used (if exists).\r
+<B> %ProgramFiles%\OFED\opensm\opensm.conf</B> will be used (if exists).\r
<DT><B>-c</B>, <B>--create-config</B> <file name><DD>\r
OpenSM will dump its configuration to the specified file and exit.\r
This is a way to generate OpenSM configuration file template.\r
recalculations: one when the host goes down, and the other when\r
the host comes back online.\r
<DT><B>-z</B>, <B>--connect_roots</B><DD>\r
-This option enforces a routing engine (currently up/down\r
-only) to make connectivity between root switches and in\r
+This option enforces routing engines (up/down and\r
+fat-tree) to make connectivity between root switches and in\r
this way to be fully IBA complaint. In many cases this can\r
violate "pure" deadlock free algorithm, so use it carefully.\r
<DT><B>-M</B>, <B>--lid_matrix_file</B> <file name><DD>\r
factor of 1. Lines starting with # are comments. Weights affect only the\r
output route from the port, so many useful configurations will require weights\r
to be specified in pairs.\r
+<DT><b>-O, --dimn_ports_file <path to file></b><dd>This option provides a \r
+mapping between hypercube dimensions and ports on a per switch basis for the DOR \r
+routing engine. The file consists of lines containing a switch node GUID \r
+(specified as a 64 bit hex number, with leading 0x) followed by a list of \r
+non-zero port numbers, separated by spaces, one switch per line. The order for \r
+the port numbers is in one to one correspondence to the dimensions. Ports not \r
+listed on a line are assigned to the remaining dimensions, in port order. \r
+Anything after a # is a comment.</dd>\r
<DT><B>-x</B>, <B>--honor_guid2lid</B><DD>\r
This option forces OpenSM to honor the guid2lid file,\r
when it comes out of Standby state, if such file exists\r
under OSM_CACHE_DIR, and is valid.\r
By default, this is FALSE.\r
<DT><B>-f</B>, <B>--log_file</B> <file name><DD>\r
-This option defines the log to be the given file. By default, the log goes to <b>\r
-%windir%\temp\</b>osm.log.\r
+This option defines the log to be the given file. By default, the log goes to \r
+%windir%\temp\opensm.log.\r
For the log to go to standard output use -f stdout.\r
<DT><B>-L</B>, <B>--log_limit</B> <size in MB><DD>\r
This option defines maximal log file size in MB. When\r
is accumulative.\r
<DT><B>-P</B>, <B>--Pconfig</B> <partition config file><DD>\r
This option defines the optional partition configuration file.\r
-The default name is <b>%ProgramFiles%\OFED\OpenSM\partitions.conf</b>.\r
+The default name is <B>%ProgramFiles%\OFED\opensm\partitions.conf</B>.\r
<DT><B>--prefix_routes_file</B> <file name><DD>\r
Prefix routes control how the SA responds to path record queries for\r
off-subnet DGIDs. By default, the SA fails such queries. The\r
<B>PREFIX ROUTES</B>\r
\r
section below describes the format of the configuration file.\r
-The default path is <b>%ProgramFiles%\OFED\OpenSM\prefix-routes.conf</b>.\r
+The default path is <B>%ProgramFiles%\OFED\opensm\prefix-routes.conf</B>.\r
<DT><B>-Q</B>, <B>--qos</B><DD>\r
This option enables QoS setup. It is disabled by default.\r
<DT><B>-Y</B>, <B>--qos_policy_file</B> <file name><DD>\r
This option defines the optional QoS policy file. The default\r
-name is <b>%ProgramFiles%\OFED\OpenSM\qos-policy.conf</b>. See\r
+name is <B>%ProgramFiles%\OFED\opensm\qos-policy.conf</B>. See\r
QoS_management_in_OpenSM.txt in opensm doc for more information on\r
configuring QoS policy via this file.\r
<DT><B>-N</B>, <B>--no_part_enforce</B><DD>\r
Specify the sweep time for the performance manager in seconds\r
(default is 180 seconds). Only takes\r
effect if --enable-perfmgr was specified at configure time.\r
-<DT><B>--consolidate_ipv6_snm_req</B>\r
-\r
-<DD>\r
-Consolidate IPv6 Solicited Node Multicast group join requests into one\r
-multicast group per MGID PKey.\r
+<DT><B>--consolidate_ipv6_snm_req</B><DD>\r
+Use shared MLID for IPv6 Solicited Node Multicast groups per MGID scope\r
+and P_Key.\r
+<DT><B>-log_prefix <prefix text></B><DD>\r
+This option specifies the prefix to the syslog messages from OpenSM. A suitable \r
+prefix can be used to identify the IB subnet in syslog messages when two or more \r
+instances of OpenSM run in a single node to manage multiple fabrics. For \r
+example, in a dual-fabric (or dual-rail) IB cluster, the prefix for the first \r
+fabric could be "mpi" and the other fabric could be "storage".\r
<DT><B>-v</B>, <B>--verbose</B><DD>\r
This option increases the log verbosity level.\r
The -v option may be specified multiple times\r
A flags field must follow the -D option.\r
A bit set/clear in the flags enables/disables a\r
specific log level as follows:\r
-\r
-<BR> BIT LOG LEVEL ENABLED\r
+<P>\r
+ BIT LOG LEVEL ENABLED\r
<BR> ---- -----------------\r
<BR> 0x01 - ERROR (error messages)\r
<BR> 0x02 - INFO (basic messages, low volume)\r
<BR> 0x20 - FRAMES (dumps all SMP and GMP frames)\r
<BR> 0x40 - ROUTING (dump FDB routing information)\r
<BR> 0x80 - currently unused.\r
-\r
+<P>\r
Without -D, OpenSM defaults to ERROR + INFO (0x3).\r
Specifying -D 0 disables all messages.\r
Specifying -D 0xFF enables all messages (see -V).\r
These options are not normally needed.\r
The number following -d selects the debug\r
option to enable as follows:\r
-\r
+<br>\r
<BR> OPT Description\r
<BR> --- -----------------\r
<BR> -d0 - Ignore other SM nodes\r
Display this usage info then exit.\r
<DT><B>-?</B><DD>\r
Display this usage info then exit.\r
-\r
-<br>\r
- </DL>\r
+<P>\r
+</DL>\r
<h3>ENVIRONMENT VARIABLES</h3>\r
\r
<P>\r
Examples:<P>\r
\r
sc.exe control OpenSM 128 \r
-# will clear the contents of %windir%\temp\osm.log<br>\r
+# will clear the contents of %windir%\temp\osm.log, logrotate.<br>\r
sc.exe control OpenSM 129 \r
# start a new heavy sweep<P>\r
\r
The default name of OpenSM partitions configuration file is\r
<B>%ProgramFiles\OFED\OpenSM\partitions.conf</B>. The default may be changed\r
by using the --Pconfig (-P) option with OpenSM.\r
-\r
+<P>\r
The default partition will be created by OpenSM unconditionally even\r
when partition configuration file does not exist or cannot be accessed.\r
-\r
+<P>\r
The default partition has P_Key value 0x7fff. OpenSM's port will always\r
have full membership in default partition. All other end ports will have\r
full membership if the partition configuration file is not found or cannot\r
be accessed, or limited membership if the file exists and can be accessed\r
but there is no rule for the Default partition.\r
-\r
+<P>\r
Effectively, this amounts to the same as if one of the following rules\r
below appear in the partition configuration file.\r
-\r
+<P>\r
In the case of no rule for the Default partition:\r
-\r
+<P>\r
Default=0x7fff : ALL=limited, SELF=full ;\r
-\r
+<P>\r
In the case of no partition configuration file or file cannot be accessed:\r
-\r
+<P>\r
Default=0x7fff : ALL=full ;\r
-\r
-\r
+<P>\r
+<P>\r
File Format\r
-\r
+<P>\r
Comments:\r
-\r
+<P>\r
Line content followed after '#' character is comment and ignored by\r
parser.\r
-\r
+<P>\r
General file format:\r
-\r
+<P>\r
<Partition Definition>:<PortGUIDs list> ;\r
-\r
+<P>\r
Partition Definition:\r
-\r
+<P>\r
[PartitionName][=PKey][,flag[=value]][,defmember=full|limited]\r
-\r
-<BR> PartitionName - string, will be used with logging. When omitted\r
+<P>\r
+ PartitionName - string, will be used with logging. When omitted\r
<BR> empty string will be used.\r
<BR> PKey - P_Key value for this partition. Only low 15 bits will\r
<BR> be used. When omitted will be autogenerated.\r
<BR> flag - used to indicate IPoIB capability of this partition.\r
<BR> defmember=full|limited - specifies default membership for port guid\r
<BR> list. Default is limited.\r
-\r
+<P>\r
Currently recognized flags are:\r
-\r
-<BR> ipoib - indicates that this partition may be used for IPoIB, as\r
+<P>\r
+ ipoib - indicates that this partition may be used for IPoIB, as\r
<BR> result IPoIB capable MC group will be created.\r
<BR> rate=<val> - specifies rate for this IPoIB MC group\r
<BR> (default is 3 (10GBps))\r
<BR> scope=<val> - specifies scope for this IPoIB MC group\r
<BR> (default is 2 (link local)). Multiple scope settings\r
<BR> are permitted for a partition.\r
-\r
+<P>\r
Note that values for rate, mtu, and scope should be specified as\r
defined in the IBTA specification (for example, mtu=4 for 2048).\r
-\r
+<P>\r
PortGUIDs list:\r
-\r
-<BR> PortGUID - GUID of partition member EndPort. Hexadecimal\r
+<P>\r
+ PortGUID - GUID of partition member EndPort. Hexadecimal\r
<BR> numbers should start from 0x, decimal numbers\r
<BR> are accepted too.\r
<BR> full or limited - indicates full or limited membership for this\r
<BR> port. When omitted (or unrecognized) limited\r
<BR> membership is assumed.\r
-\r
+<P>\r
There are two useful keywords for PortGUID definition:\r
-\r
-<BR> - 'ALL' means all end ports in this subnet.\r
+<P>\r
+ - 'ALL' means all end ports in this subnet.\r
<BR> - 'ALL_CAS' means all Channel Adapter end ports in this subnet.\r
<BR> - 'ALL_SWITCHES' means all Switch end ports in this subnet.\r
<BR> - 'ALL_ROUTERS' means all Router end ports in this subnet.\r
<BR> - 'SELF' means subnet manager's port.\r
-\r
+<P>\r
Empty list means no ports in this partition.\r
-\r
+<P>\r
Notes:\r
-\r
+<P>\r
White space is permitted between delimiters ('=', ',',':',';').\r
-\r
+<P>\r
The line can be wrapped after ':' followed after Partition Definition and\r
between.\r
-\r
+<P>\r
PartitionName does not need to be unique, PKey does need to be unique.\r
If PKey is repeated then those partition configurations will be merged\r
and first PartitionName will be used (see also next note).\r
-\r
+<P>\r
It is possible to split partition configuration in more than one\r
definition, but then PKey should be explicitly specified (otherwise\r
different PKey values will be generated for those definitions).\r
-\r
+<P>\r
Examples:\r
-\r
-<BR> Default=0x7fff : ALL, SELF=full ;\r
+<P>\r
+ Default=0x7fff : ALL, SELF=full ;\r
<BR> Default=0x7fff : ALL, ALL_SWITCHES=full, SELF=full ;\r
-\r
-<BR> NewPartition , ipoib : 0x123456=full, 0x3456789034=limi, 0x2134af2306 ;\r
-\r
-<BR> YetAnotherOne = 0x300 : SELF=full ;\r
+<P>\r
+ NewPartition , ipoib : 0x123456=full, 0x3456789034=limi, 0x2134af2306 ;\r
+<P>\r
+ YetAnotherOne = 0x300 : SELF=full ;\r
<BR> YetAnotherOne = 0x300 : ALL=limited ;\r
-\r
-<BR> ShareIO = 0x80 , defmember=full : 0x123451, 0x123452;\r
+<P>\r
+ ShareIO = 0x80 , defmember=full : 0x123451, 0x123452;\r
<BR> # 0x123453, 0x123454 will be limited\r
<BR> ShareIO = 0x80 : 0x123453, 0x123454, 0x123455=full;\r
<BR> # 0x123456, 0x123457 will be limited\r
<BR> ShareIO = 0x80 : defmember=limited : 0x123456, 0x123457, 0x123458=full;\r
<BR> ShareIO = 0x80 , defmember=full : 0x123459, 0x12345a;\r
<BR> ShareIO = 0x80 , defmember=full : 0x12345b, 0x12345c=limited, 0x12345d;\r
-\r
-\r
+<P>\r
+<P>\r
Note:\r
-\r
+<P>\r
The following rule is equivalent to how OpenSM used to run prior to the\r
partition manager:\r
-\r
-<BR> Default=0x7fff,ipoib:ALL=full;\r
+<P>\r
+ Default=0x7fff,ipoib:ALL=full;\r
\r
<A NAME="lbAI"> </A>\r
<h3>QOS CONFIGURATION</h3>\r
There are a set of QoS related low-level configuration parameters.\r
All these parameter names are prefixed by "qos_" string. Here is a full\r
list of these parameters:\r
-\r
-<BR> qos_max_vls - The maximum number of VLs that will be on the subnet\r
+<P>\r
+ qos_max_vls - The maximum number of VLs that will be on the subnet\r
<BR> qos_high_limit - The limit of High Priority component of VL\r
<BR> Arbitration table (IBA 7.6.9)\r
<BR> qos_vlarb_low - Low priority VL Arbitration table (IBA 7.6.9)\r
<BR> qos_sl2vl - SL2VL Mapping table (IBA 7.6.6) template. It is\r
<BR> a list of VLs corresponding to SLs 0-15 (Note\r
<BR> that VL15 used here means drop this SL)\r
-\r
+<P>\r
Typical default values (hard-coded in OpenSM initialization) are:\r
-\r
-<BR> qos_max_vls 15\r
+<P>\r
+ qos_max_vls 15\r
<BR> qos_high_limit 0\r
<BR> qos_vlarb_low 0:0,1:4,2:4,3:4,4:4,5:4,6:4,7:4,8:4,9:4,10:4,11:4,12:4,13:4,14:4\r
<BR> qos_vlarb_high 0:4,1:0,2:0,3:0,4:0,5:0,6:0,7:0,8:0,9:0,10:0,11:0,12:0,13:0,14:0\r
<BR> qos_sl2vl 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,7\r
-\r
+<P>\r
The syntax is compatible with rest of OpenSM configuration options and\r
values may be stored in OpenSM config file (cached options file).\r
-\r
+<P>\r
In addition to the above, we may define separate QoS configuration\r
parameters sets for various target types. As targets, we currently support\r
CAs, routers, switch external ports, and switch's enhanced port 0. The\r
names of such specialized parameters are prefixed by "qos_<type>_"\r
string. Here is a full list of the currently supported sets:\r
-\r
-<BR> qos_ca_ - QoS configuration parameters set for CAs.\r
+<P>\r
+ qos_ca_ - QoS configuration parameters set for CAs.\r
<BR> qos_rtr_ - parameters set for routers.\r
<BR> qos_sw0_ - parameters set for switches' port 0.\r
<BR> qos_swe_ - parameters set for switches' external ports.\r
-\r
+<P>\r
Examples:\r
<BR> qos_sw0_max_vls=2\r
<BR> qos_ca_sl2vl=0,1,2,3,5,5,5,12,12,0,\r
off-subnet DGID should return a path to the first available router.\r
This configuration yields the same behavior formerly achieved by\r
compiling opensm with -DROUTER_EXP which has been obsoleted.\r
-\r
-<A NAME="lbAK"> </A>\r
<h3>ROUTING</h3>\r
\r
<P>\r
\r
OpenSM now offers five routing engines:\r
-\r
+<P>\r
1. Min Hop Algorithm - based on the minimum hops to each node where the\r
path length is optimized.\r
-\r
+<P>\r
2. UPDN Unicast routing algorithm - also based on the minimum hops to each\r
node, but it is constrained to ranking rules. This algorithm should be chosen\r
if the subnet is not a pure Fat Tree, and deadlock may occur due to a\r
loop in the subnet.\r
-\r
+<P>\r
3. Fat Tree Unicast routing algorithm - this algorithm optimizes routing\r
for congestion-free "shift" communication pattern.\r
It should be chosen if a subnet is a symmetrical or almost symmetrical\r
fat-tree of various types, not just K-ary-N-Trees: non-constant K, not\r
fully staffed, any Constant Bisectional Bandwidth (CBB) ratio.\r
Similar to UPDN, Fat Tree routing is constrained to ranking rules.\r
-\r
+<P>\r
4. LASH unicast routing algorithm - uses Infiniband virtual layers\r
(SL) to provide deadlock-free shortest-path routing while also\r
distributing the paths between layers. LASH is an alternative\r
deadlock-free topology-agnostic routing algorithm to the non-minimal\r
UPDN algorithm avoiding the use of a potentially congested root node.\r
-\r
+<P>\r
5. DOR Unicast routing algorithm - based on the Min Hop algorithm, but\r
avoids port equalization except for redundant links between the same\r
two switches. This provides deadlock free routes for hypercubes when\r
the fabric is cabled as a hypercube and for meshes when cabled as a\r
mesh (see details below).\r
-\r
+<P>\r
OpenSM also supports a file method which\r
can load routes from a table. See 'Modular Routing Engine' for more\r
information on this.\r
-\r
+<P>\r
The basic routing algorithm is comprised of two stages:\r
-\r
+<P>\r
1. MinHop matrix calculation\r
<BR> How many hops are required to get from each port to each LID ?\r
<BR> The algorithm to fill these tables is different if you run standard\r
<BR> For Up/Down routing, a BFS from every target is used. The BFS tracks link\r
direction (up or down) and avoid steps that will perform up after a down\r
step was used.\r
-\r
+<P>\r
2. Once MinHop matrices exist, each switch is visited and for each target LID a\r
decision is made as to what port should be used to get to that LID.\r
<BR> This step is common to standard and Up/Down routing. Each port has a\r
the previous LID of the same LMC group)\r
<BR> c. if none - prefer those which go through another NodeGuid\r
<BR> d. fall back to the number of paths method (if all go to same node).\r
-\r
+<P>\r
Effect of Topology Changes\r
-\r
+<P>\r
OpenSM will preserve existing routing in any case where there is no change in\r
the fabric switches unless the -r (--reassign_lids) option is specified.\r
-\r
+<P>\r
-r\r
<BR>\r
\r
<BR> may disrupt subnet traffic.\r
<BR> Without -r, OpenSM attempts to preserve existing\r
<BR> LID assignments resolving multiple use of same LID.\r
-\r
+<P>\r
If a link is added or removed, OpenSM does not recalculate\r
the routes that do not have to change. A route has to change\r
if the port is no longer UP or no longer the MinHop. When routing changes\r
are performed, the same algorithm for balancing the routes is invoked.\r
-\r
+<P>\r
In the case of using the file based routing, any topology changes are\r
currently ignored The 'file' routing engine just loads the LFTs from the file\r
specified, with no reaction to real topology. Obviously, this will not be able\r
to recheck LIDs (by GUID) for disconnected nodes, and LFTs for non-existent\r
switches will be skipped. Multicast is not affected by 'file' routing engine\r
(this uses min hop tables).\r
-\r
-\r
+<P>\r
+<P>\r
Min Hop Algorithm\r
-\r
+<P>\r
The Min Hop algorithm is invoked by default if no routing algorithm is\r
specified. It can also be invoked by specifying '-R minhop'.\r
-\r
+<P>\r
The Min Hop algorithm is divided into two stages: computation of\r
min-hop tables on every switch and LFT output port assignment. Link\r
subscription is also equalized with the ability to override based on\r
port GUID. The latter is supplied by:\r
-\r
+<P>\r
-i <equalize-ignore-guids-file>\r
<BR>\r
\r
<BR> equalization algorithm. Note that only endports (CA,\r
<BR> switch port 0, and router ports) and not switch external\r
<BR> ports are supported.\r
-\r
+<P>\r
LMC awareness routes based on (remote) system or switch basis.\r
-\r
-\r
+<P>\r
+<P>\r
Purpose of UPDN Algorithm\r
-\r
+<P>\r
The UPDN algorithm is designed to prevent deadlocks from occurring in loops\r
of the subnet. A loop-deadlock is a situation in which it is no longer\r
possible to send data between any two hosts connected through the loop. As\r
such, the UPDN routing algorithm should be used if the subnet is not a pure\r
Fat Tree, and one of its loops may experience a deadlock (due, for example,\r
to high pressure).\r
-\r
+<P>\r
The UPDN algorithm is based on the following main stages:\r
-\r
+<P>\r
1. Auto-detect root nodes - based on the CA hop length from any switch in\r
the subnet, a statistical histogram is built for each switch (hop num vs\r
number of occurrences). If the histogram reflects a specific column (higher\r
the algorithm is statistical, it may not find any root nodes. The list of\r
the root nodes found by this auto-detect stage is used by the ranking\r
process stage.\r
-\r
+<P>\r
<BR> Note 1: The user can override the node list manually.\r
<BR> Note 2: If this stage cannot find any root nodes, and the user did\r
<BR> not specify a guid list file, OpenSM defaults back to the\r
<BR> Min Hop routing algorithm.\r
-\r
+<P>\r
2. Ranking process - All root switch nodes (found in stage 1) are assigned\r
a rank of 0. Using the BFS algorithm, the rest of the switch nodes in the\r
subnet are ranked incrementally. This ranking aids in the process of enforcing\r
rules that ensure loop-free paths.\r
-\r
+<P>\r
3. Min Hop Table setting - after ranking is done, a BFS algorithm is run from\r
each (CA or switch) node in the subnet. During the BFS process, the FDB table\r
of each switch node traversed by BFS is updated, in reference to the starting\r
node, based on the ranking rules and guid values.\r
-\r
+<P>\r
At the end of the process, the updated FDB tables ensure loop-free paths\r
through the subnet.\r
-\r
+<P>\r
Note: Up/Down routing does not allow LID routing communication between\r
switches that are located inside spine "switch systems".\r
The reason is that there is no way to allow a LID route between them\r
that does not break the Up/Down rule.\r
One ramification of this is that you cannot run SM on switches other\r
than the leaf switches of the fabric.\r
-\r
-\r
+<P>\r
+<P>\r
UPDN Algorithm Usage\r
-\r
+<P>\r
Activation through OpenSM\r
-\r
+<P>\r
Use '-R updn' option (instead of old '-u') to activate the UPDN algorithm.\r
Use '-a <root_guid_file>' for adding an UPDN guid file that contains the\r
root nodes for ranking.\r
If the `-a' option is not used, OpenSM uses its auto-detect root nodes\r
algorithm.\r
-\r
+<P>\r
Notes on the guid list file:\r
-\r
+<P>\r
1. A valid guid file specifies one guid in each line. Lines with an invalid\r
format will be discarded.\r
<BR>\r
2. The user should specify the root switch guids. However, it is also\r
possible to specify CA guids; OpenSM will use the guid of the switch (if\r
it exists) that connects the CA to the subnet as a root node.\r
-\r
-\r
+<P>\r
+<P>\r
Fat-tree Routing Algorithm\r
-\r
+<P>\r
The fat-tree algorithm optimizes routing for "shift" communication pattern.\r
It should be chosen if a subnet is a symmetrical or almost symmetrical\r
fat-tree of various types.\r
It supports not just K-ary-N-Trees, by handling for non-constant K,\r
cases where not all leafs (CAs) are present, any CBB ratio.\r
As in UPDN, fat-tree also prevents credit-loop-deadlocks.\r
-\r
+<P>\r
If the root guid file is not provided ('-a' or '--root_guid_file' options),\r
the topology has to be pure fat-tree that complies with the following rules:\r
<BR> - Tree rank should be between two and eight (inclusively)\r
<BR> - Switches of the same rank should have the same number\r
<BR> of ports in each DOWN-going port group.\r
<BR> - All the CAs have to be at the same tree level (rank).\r
-\r
+<P>\r
If the root guid file is provided, the topology doesn't have to be pure\r
fat-tree, and it should only comply with the following rules:\r
<BR> - Tree rank should be between two and eight (inclusively)\r
<BR> - All the Compute Nodes** have to be at the same tree level (rank).\r
<BR> Note that non-compute node CAs are allowed here to be at different\r
<BR> tree ranks.\r
-\r
+<P>\r
* ports that are connected to the same remote switch are referenced as\r
'port group'.\r
-\r
+<P>\r
** list of compute nodes (CNs) can be specified by '-u' or '--cn_guid_file'\r
OpenSM options.\r
-\r
+<P>\r
Topologies that do not comply cause a fallback to min hop routing.\r
Note that this can also occur on link failures which cause the topology\r
to no longer be "pure" fat-tree.\r
-\r
+<P>\r
Note that although fat-tree algorithm supports trees with non-integer CBB\r
ratio, the routing will not be as balanced as in case of integer CBB ratio.\r
In addition to this, although the algorithm allows leaf switches to have any\r
effective the "shift" communication pattern will be.\r
In general, even if the root list is provided, the closer the topology to a\r
pure and symmetrical fat-tree, the more optimal the routing will be.\r
-\r
+<P>\r
The algorithm also dumps compute node ordering file (opensm-ftree-ca-order.dump)\r
in the same directory where the OpenSM log resides. This ordering file provides\r
the CN order that may be used to create efficient communication pattern, that\r
will match the routing tables.\r
-\r
+<P>\r
Routing between non-CN nodes\r
-\r
+<P>\r
The use of the cn_guid_file option allows non-CN nodes to be located on different levels in the fat tree.\r
In such case, it is not guaranteed that the Fat Tree algorithm will route between two non-CN nodes.\r
To solve this problem, a list of non-CN nodes can be specified by '-G' or '--io_guid_file' option.\r
Theses nodes will be allowed to use switches the wrong way round a specific number of times (specified by '-H' or '--max_reverse_hops'.\r
With the proper max_reverse_hops and io_guid_file values, you can ensure full connectivity in the Fat Tree.\r
-\r
+<P>\r
Please note that using max_reverse_hops creates routes that use the switch in a counter-stream way.\r
This option should never be used to connect nodes with high bandwidth traffic between them ! It should only be used\r
to allow connectivity for HA purposes or similar.\r
Also having routes the other way around can in theory cause credit loops.\r
-\r
+<P>\r
Use these options with extreme care !\r
-\r
+<P>\r
Activation through OpenSM\r
-\r
+<P>\r
Use '-R ftree' option to activate the fat-tree algorithm.\r
Use '-a <root_guid_file>' to provide root nodes for ranking. If the `-a' option\r
is not used, routing algorithm will detect roots automatically.\r
Use '-u <root_cn_file>' to provide the list of compute nodes. If the `-u' option\r
is not used, all the CAs are considered as compute nodes.\r
-\r
+<P>\r
Note: LMC > 0 is not supported by fat-tree routing. If this is\r
specified, the default routing algorithm is invoked instead.\r
-\r
-\r
+<P>\r
+<P>\r
LASH Routing Algorithm\r
-\r
+<P>\r
LASH is an acronym for LAyered SHortest Path Routing. It is a\r
deterministic shortest path routing algorithm that enables topology\r
agnostic deadlock-free routing within communication networks.\r
-\r
+<P>\r
When computing the routing function, LASH analyzes the network\r
topology for the shortest-path routes between all pairs of sources /\r
destinations and groups these paths into virtual layers in such a way\r
as to avoid deadlock.\r
-\r
+<P>\r
Note LASH analyzes routes and ensures deadlock freedom between switch\r
pairs. The link from HCA between and switch does not need virtual\r
layers as deadlock will not arise between switch and HCA.\r
-\r
+<P>\r
In more detail, the algorithm works as follows:\r
-\r
+<P>\r
1) LASH determines the shortest-path between all pairs of source /\r
destination switches. Note, LASH ensures the same SL is used for all\r
SRC/DST - DST/SRC pairs and there is no guarantee that the return\r
path for a given DST/SRC will be the reverse of the route SRC/DST.\r
-\r
+<P>\r
2) LASH then begins an SL assignment process where a route is assigned\r
to a layer (SL) if the addition of that route does not cause deadlock\r
within that layer. This is achieved by maintaining and analysing a\r
channel dependency graph for each layer. Once the potential addition\r
of a path could lead to deadlock, LASH opens a new layer and continues\r
the process.\r
-\r
+<P>\r
3) Once this stage has been completed, it is highly likely that the\r
first layers processed will contain more paths than the latter ones.\r
To better balance the use of layers, LASH moves paths from one layer\r
to another so that the number of paths in each layer averages out.\r
-\r
+<P>\r
Note, the implementation of LASH in opensm attempts to use as few layers\r
as possible. This number can be less than the number of actual layers\r
available.\r
-\r
+<P>\r
In general LASH is a very flexible algorithm. It can, for example,\r
reduce to Dimension Order Routing in certain topologies, it is topology\r
agnostic and fares well in the face of faults.\r
-\r
+<P>\r
It has been shown that for both regular and irregular topologies, LASH\r
outperforms Up/Down. The reason for this is that LASH distributes the\r
traffic more evenly through a network, avoiding the bottleneck issues\r
related to a root node and always routes shortest-path.\r
-\r
+<P>\r
The algorithm was developed by Simula Research Laboratory.\r
-\r
-\r
+<P>\r
+<P>\r
Use '-R lash -Q ' option to activate the LASH algorithm.\r
-\r
+<P>\r
Note: QoS support has to be turned on in order that SL/VL mappings are\r
used.\r
-\r
+<P>\r
Note: LMC > 0 is not supported by the LASH routing. If this is\r
specified, the default routing algorithm is invoked instead.\r
-\r
+<P>\r
For open regular cartesian meshes the DOR algorithm is the ideal\r
routing algorithm. For toroidal meshes on the other hand there\r
are routing loops that can cause deadlocks. LASH can be used to\r
the dimension and size of a mesh. If it determines that the mesh\r
looks like an open or closed cartesian mesh it reorders the ports\r
in dimension order before the rest of the LASH algorithm runs.\r
-\r
+<P>\r
DOR Routing Algorithm\r
-\r
-The Dimension Order Routing algorithm is based on the Min Hop\r
-algorithm and so uses shortest paths. Instead of spreading traffic\r
-out across different paths with the same shortest distance, it chooses\r
-among the available shortest paths based on an ordering of dimensions.\r
-Each port must be consistently cabled to represent a hypercube\r
-dimension or a mesh dimension. Paths are grown from a destination\r
-back to a source using the lowest dimension (port) of available paths\r
-at each step. This provides the ordering necessary to avoid deadlock.\r
-When there are multiple links between any two switches, they still\r
-represent only one dimension and traffic is balanced across them\r
-unless port equalization is turned off. In the case of hypercubes,\r
-the same port must be used throughout the fabric to represent the\r
-hypercube dimension and match on both ends of the cable. In the case\r
-of meshes, the dimension should consistently use the same pair of\r
-ports, one port on one end of the cable, and the other port on the\r
-other end, continuing along the mesh dimension.\r
-\r
+<P>\r
+The Dimension Order Routing algorithm is based on the Min Hop algorithm and so \r
+uses shortest paths. Instead of spreading traffic<br>\r
+out across different paths with the same shortest distance, it chooses among the \r
+available shortest paths based on an ordering of dimensions. Each port \r
+must be consistently cabled to represent a hypercube dimension or a mesh \r
+dimension. Alternatively, the -O option can be<br>\r
+used to assign a custom mapping between the ports on a given switch, and the \r
+associated dimension. Paths are grown from a destination back<br>\r
+to a source using the lowest dimension (port) of available paths at each step. \r
+This provides the ordering necessary to avoid deadlock. When there are multiple \r
+links between any two switches, they still represent only one dimension and \r
+traffic is balanced across them<br>\r
+unless port equalization is turned off. In the case of hypercubes, the same port \r
+must be used throughout the fabric to represent the hypercube dimension and \r
+match on both ends of the cable, or the -O option used to accomplish the \r
+alignment. In the case of meshes, the dimension should consistently use the same \r
+pair of ports, one port on one end of the cable, and the other port on the other \r
+end, continuing along the mesh dimension, or the -O option used as an override.<P>\r
Use '-R dor' option to activate the DOR algorithm.\r
-\r
-\r
+<P>\r
+<P>\r
Routing References\r
-\r
+<P>\r
To learn more about deadlock-free routing, see the article\r
"Deadlock Free Message Routing in Multiprocessor Interconnection Networks"\r
by William J Dally and Charles L Seitz (1985).\r
-\r
+<P>\r
To learn more about the up/down algorithm, see the article\r
"Effective Strategy to Compute Forwarding Tables for InfiniBand Networks"\r
by Jose Carlos Sancho, Antonio Robles, and Jose Duato at the\r
Universidad Politecnica de Valencia.\r
-\r
+<P>\r
To learn more about LASH and the flexibility behind it, the requirement\r
for layers, performance comparisons to other algorithms, see the\r
following articles:\r
-\r
+<P>\r
"Layered Routing in Irregular Networks", Lysne et al, IEEE\r
Transactions on Parallel and Distributed Systems, VOL.16, No12,\r
December 2005.\r
-\r
+<P>\r
"Routing for the ASI Fabric Manager", Solheim et al. IEEE\r
Communications Magazine, Vol.44, No.7, July 2006.\r
-\r
+<P>\r
"Layered Shortest Path (LASH) Routing in Irregular System Area\r
Networks", Skeie et al. IEEE Computer Society Communication\r
Architecture for Clusters 2002.\r
-\r
-\r
+<P>\r
+<P>\r
Modular Routine Engine\r
-\r
+<P>\r
Modular routing engine structure allows for the ease of\r
"plugging" new routing modules.\r
-\r
+<P>\r
Currently, only unicast callbacks are supported. Multicast\r
can be added later.\r
-\r
+<P>\r
One existing routing module is up-down "updn", which may be\r
activated with '-R updn' option (instead of old '-u').\r
-\r
+<P>\r
General usage is:\r
$ opensm -R 'module-name'\r
-\r
+<P>\r
There is also a trivial routing module which is able\r
to load LFT tables from a file.\r
-\r
+<P>\r
Main features:\r
-\r
-<BR> - this will load switch LFTs and/or LID matrices (min hops tables)\r
+<P>\r
+ - this will load switch LFTs and/or LID matrices (min hops tables)\r
<BR> - this will load switch LFTs according to the path entries introduced\r
<BR> in the file\r
<BR> - no additional checks will be performed (such as "is port connected",\r
<BR> LFTs correctly if endport GUIDs are represented in the file\r
<BR> (in order to disable this, GUIDs may be removed from the file\r
<BR> or zeroed)\r
-\r
+<P>\r
The file format is compatible with output of 'ibroute' util and for\r
whole fabric can be generated with dump_lfts.sh script.\r
-\r
+<P>\r
To activate file based routing module, use:\r
-\r
-<BR> opensm -R file -U \path\to\lfts_file\r
-\r
+ opensm -R file -U /path/to/lfts_file\r
+<P>\r
If the lfts_file is not found or is in error, the default routing\r
algorithm is utilized.\r
-\r
+<P>\r
The ability to dump switch lid matrices (aka min hops tables) to file and\r
later to load these is also supported.\r
-\r
+<P>\r
The usage is similar to unicast forwarding tables loading from a lfts\r
file (introduced by 'file' routing engine), but new lid matrix file\r
-name should be specified by -M or --lid_matrix_file option. For example:\r
-\r
+name should be specified by -M or --lid_matrix_file option. For example: <br>\r
<BR> opensm -R file -M ./opensm-lid-matrix.dump\r
-\r
-The dump file is named 'opensm-lid-matrix.dump' and will be generated\r
-in standard opensm dump directory (/var/log by default) when\r
-OSM_LOG_ROUTING logging flag is set.\r
-\r
+<P>\r
+The dump file is named 'opensm-lid-matrix.dump' and will be generated in \r
+standard opensm dump directory (%TEMP% by default) when OSM_LOG_ROUTING logging flag is set.\r
+<P>\r
When routing engine 'file' is activated, but the lfts file is not specified\r
or not cannot be open default lid matrix algorithm will be used.\r
-\r
+<P>\r
There is also a switch forwarding tables dumper which generates\r
a file compatible with dump_lfts.sh output. This file can be used\r
as input for forwarding tables loading by 'file' routing engine.\r
Both or one of options -U and -M can be specified together with '-R file'.\r
-\r
-<A NAME="lbAL"> </A>\r
<h3>FILES</h3>\r
\r
<DL COMPACT>\r
\r
<DD>\r
default OpenSM config file.\r
+<P>\r
+<DT><B>%ProgramFiles%\OFED\OpenSM\ib-node-name-map</B>\r
\r
-<DT><B>%ProgramFiles\OFED\OpenSM\ib-node-name-map.conf</B><DD>\r
-default node name map file. See <a href="#NODE_NAME_MAP_FILE_FORMAT">ibnetdiscover</a> for more information on format.\r
-\r
-<DT><B>%ProgramFiles\OFED\OpenSM\partitions.conf</B>\r
+<DD>\r
+default node name map file. See ibnetdiscover for more information on format.\r
+<P>\r
+<DT><B>%ProgramFiles%\OFED\OpenSM\partitions.conf</B>\r
\r
<DD>\r
default partition config file\r
-\r
-<DT><B>%ProgramFiles\OFED\OpenSM\qos-policy.conf</B>\r
+<P>\r
+<DT><B>%ProgramFiles%\OFED\OpenSM\qos-policy.conf</B>\r
\r
<DD>\r
default QOS policy config file\r
-\r
-<DT><B>%ProgramFiles\OFED\OpenSM\prefix-routes.conf</B>\r
+<P>\r
+<DT><B>%ProgramFiles%\OFED\OpenSM\prefix-routes.conf</B>\r
\r
<DD>\r
default prefix routes file.\r
-\r
+<P>\r
</DL>\r
<h3>AUTHORS</h3>\r
\r
<<I><A HREF="mailto:weiny2@llnl.gov">weiny2@llnl.gov</A></I>>\r
\r
<DT>Stan Smith<DD>\r
-<<a href="mailto:weiny2@llnl.gov"><i>stan.smith</i></a><I><A HREF="mailto:weiny2@llnl.gov">@intel.com</A></I>></DL>\r
+<<a href="mailto:weiny2@llnl.gov"><i>stan.smith</i></a><I><A HREF="mailto:weiny2@llnl.gov">@intel.com</A></I>><dt>\r
+Dale Purdy<br>\r
+ < purdy@sgi.com ></dt>\r
+</DL>\r
\r
<h4 align="left"><a href="#TOP"><font color="#000000"><return-to-top></font></a></h4>\r
<h3 align="left"> </h3>\r
<br>\r
osmtest currently can not \r
run on the same HCA port which OpenSM is currently using.</p>\r
- <h3>SYNOPSIS</h3>\r
- <b>osmtest</b> [-f(low) <c|a|v|s|e|f|m|q|t>] [-w(ait) <trap_wait_time>] [-d(ebug) \r
- <number>] [-m(ax_lid) <LID in hex>] [-g(uid)[=]<GUID in hex>] [-p(ort)] [-i(nventory) \r
- <filename>] [-s(tress)] [-M(ulticast_Mode)] [-t(imeout) <milliseconds>] [-l \r
- | --log_file] [-v] [-vf <flags>] [-h(elp)] <a name="lbAD"> </a> \r
- <h3>DESCRIPTION</h3>\r
- <p>osmtest is a test program to validate InfiniBand subnet manager and \r
- administration (SM/SA). Default is to run all flows with the exception of \r
- the QoS flow. osmtest provides a test suite for opensm. osmtest has the \r
- following capabilities and testing flows: It creates an inventory file of \r
- all available Nodes, Ports, and PathRecords, including all their fields. It \r
- verifies the existing inventory, with all the object fields, and matches it \r
- to a pre-saved one. A Multicast Compliancy test. An Event Forwarding test. A \r
- Service Record registration test. An RMPP stress test. A Small SA Queries \r
- stress test. It is recommended that after installing opensm, the user should \r
- run "osmtest -f c" to generate the inventory file, and immediately \r
- afterwards run "osmtest -f a" to test OpenSM. Another recommendation for \r
- osmtest usage is to create the inventory when the IB fabric is stable, and \r
- occasionally run "osmtest -v" to verify that nothing has changed.\r
- <a name="lbAE"> </a> </p>\r
- <h3>OPTIONS</h3>\r
- <dl compact>\r
- <dt><b>-f</b>, <b>--flow</b> </dt>\r
- <dd>This option directs osmtest to run a specific flow: <br>\r
- FLOW DESCRIPTION <br>\r
- c = create an inventory file with all nodes, ports and paths <br>\r
- a = run all validation tests (expecting an input inventory) <br>\r
- v = only validate the given inventory file <br>\r
- s = run service registration, deregistration, and lease test <br>\r
- e = run event forwarding test <br>\r
- f = flood the SA with queries according to the stress mode <br>\r
- m = multicast flow <br>\r
- q = QoS info: dump VLArb and SLtoVL tables <br>\r
- t = run trap 64/65 flow (this flow requires running of external tool)\r
- <br>\r
- (default is all flows except QoS) \r
- </dd>\r
- <dt><b>-w</b>, <b>--wait</b> </dt>\r
- <dd>This option specifies the wait time for trap 64/65 in seconds It is \r
- used only when running -f t - the trap 64/65 flow (default to 10 sec) \r
- </dd>\r
- <dt><b>-d</b>, <b>--debug</b> </dt>\r
- <dd>This option specifies a debug option. These options are not normally \r
- needed. The number following -d selects the debug option to enable as \r
- follows: <br>\r
- OPT Description <br>\r
- --- ----------------- <br>\r
- -d0 - Ignore other SM nodes <br>\r
- -d1 - Force single threaded dispatching <br>\r
- -d2 - Force log flushing after each log message <br>\r
- -d3 - Disable multicast support \r
- </dd>\r
- <dt><b>-m</b>, <b>--max_lid</b> </dt>\r
- <dd>This option specifies the maximal LID number to be searched for \r
- during inventory file build (default to 100) \r
- </dd>\r
- <dt><b>-g</b>, <b>--guid</b> </dt>\r
- <dd>This option specifies the local port GUID value with which OpenSM \r
- should bind. OpenSM may be bound to 1 port at a time. If GUID given is \r
- 0, OpenSM displays a list of possible port GUIDs and waits for user \r
- input. Without -g, OpenSM trys to use the default port. \r
- </dd>\r
- <dt><b>-p</b>, <b>--port</b> </dt>\r
- <dd>This option displays a menu of possible local port GUID values with \r
- which osmtest could bind \r
- </dd>\r
- <dt><b>-i</b>, <b>--inventory</b> </dt>\r
- <dd>This option specifies the name of the inventory file Normally, \r
- osmtest expects to find an inventory file, which osmtest uses to \r
- validate real-time information received from the SA during testing If -i \r
- is not specified, osmtest defaults to the file 'osmtest.dat' See -c \r
- option for related information \r
- </dd>\r
- <dt><b>-s</b>, <b>--stress</b> </dt>\r
- <dd>This option runs the specified stress test instead of the normal \r
- test suite Stress test options are as follows: <br>\r
- OPT Description <br>\r
- --- ----------------- <br>\r
- -s1 - Single-MAD (RMPP) response SA queries <br>\r
- -s2 - Multi-MAD (RMPP) response SA queries <br>\r
- -s3 - Multi-MAD (RMPP) Path Record SA queries <br>\r
- -s4 - Single-MAD (non RMPP) get Path Record SA queries Without -s, \r
- stress testing is not performed \r
- </dd>\r
- <dt><b>-M</b>, <b>--Multicast_Mode</b> </dt>\r
- <dd>This option specify length of Multicast test: <br>\r
- OPT Description <br>\r
- --- ----------------- <br>\r
- -M1 - Short Multicast Flow (default) - single mode <br>\r
- -M2 - Short Multicast Flow - multiple mode <br>\r
- -M3 - Long Multicast Flow - single mode <br>\r
- -M4 - Long Multicast Flow - multiple mode Single mode - Osmtest is \r
- tested alone, with no other apps that interact with OpenSM MC Multiple \r
- mode - Could be run with other apps using MC with OpenSM. Without -M, \r
- default flow testing is performed \r
- </dd>\r
- <dt><b>-t</b>, <b>--timeout</b> </dt>\r
- <dd>This option specifies the time in milliseconds used for transaction \r
- timeouts. Specifying -t 0 disables timeouts. Without -t, OpenSM defaults \r
- to a timeout value of 200 milliseconds. \r
- </dd>\r
- <dt><b>-l</b>, <b>--log_file</b> </dt>\r
- <dd>This option defines the log to be the given file. By default the log \r
- goes to stdout. \r
- </dd>\r
- <dt><b>-v</b>, <b>--verbose</b> </dt>\r
- <dd>This option increases the log verbosity level. The -v option may be \r
- specified multiple times to further increase the verbosity level. See \r
- the -vf option for more information about. log verbosity. \r
- </dd>\r
- <dt><b>-V</b> </dt>\r
- <dd>This option sets the maximum verbosity level and forces log \r
- flushing. The -V is equivalent to '-vf 0xFF -d 2'. See the -vf option \r
- for more information about. log verbosity. \r
- </dd>\r
- <dt><b>-vf</b> </dt>\r
- <dd>This option sets the log verbosity level. A flags field must follow \r
- the -D option. A bit set/clear in the flags enables/disables a specific \r
- log level as follows: <br>\r
- BIT LOG LEVEL ENABLED <br>\r
- ---- ----------------- <br>\r
- 0x01 - ERROR (error messages) <br>\r
- 0x02 - INFO (basic messages, low volume) <br>\r
- 0x04 - VERBOSE (interesting stuff, moderate volume) <br>\r
- 0x08 - DEBUG (diagnostic, high volume) <br>\r
- 0x10 - FUNCS (function entry/exit, very high volume) <br>\r
- 0x20 - FRAMES (dumps all SMP and GMP frames) <br>\r
- 0x40 - ROUTING (dump FDB routing information) <br>\r
- 0x80 - currently unused. Without -vf, osmtest defaults to ERROR + INFO \r
- (0x3) Specifying -vf 0 disables all messages Specifying -vf 0xFF enables \r
- all messages (see -V) High verbosity levels may require increasing the \r
- transaction timeout with the -t option \r
- </dd>\r
- <dt><b>-h</b>, <b>--help</b> </dt>\r
- <dd>Display this usage info then exit. </dd>\r
- </dl>\r
- <h3>AUTHORS</h3>\r
- <dl compact>\r
- <dt>Hal Rosenstock </dt>\r
- <dd><<i><a href="mailto:hal.rosenstock@gmail.com">hal.rosenstock@gmail.com</a></i>> \r
- </dd>\r
- <dt>Eitan Zahavi </dt>\r
- <dd><<i><a href="mailto:eitan@mellanox.co.il">eitan@mellanox.co.il</a></i>> \r
- </dd>\r
- </dl>\r
- <dl compact>\r
- <h3>Examples</h3>\r
- <p>Note - osmtest will not run on the node where OpenSM is running.<br>\r
- See 'osmtest -h' for all options.</p>\r
- <p>Functionality:</p>\r
- <blockquote>\r
- <p>osmtest -f c \r
+<h3>SYNOPSIS</h3>\r
+\r
+<B>osmtest</B>\r
+\r
+[-f(low) <c|a|v|s|e|f|m|q|t>] [-w(ait) <trap_wait_time>] [-d(ebug) <number>]\r
+[-m(ax_lid) <LID in hex>] [-g(uid)[=]<GUID in hex>] [-p(ort)]\r
+[-i(nventory) <filename>] [-s(tress)] [-M(ulticast_Mode)]\r
+[-t(imeout) <milliseconds>] [-l | --log_file] [-v] [-vf <flags>]\r
+[-h(elp)]\r
+<h3>DESCRIPTION</h3>\r
+\r
+<P>\r
+\r
+osmtest is a test program used to validate the correct operation of the InfiniBand subnet manager and\r
+administration (SM/SA).\r
+<P>\r
+Default is to run all flows with the exception of the QoS flow.\r
+<P>\r
+osmtest provides a test suite for opensm.\r
+<P>\r
+osmtest has the following capabilities and testing flows:\r
+<P>\r
+It creates an inventory file of all available Nodes, Ports, and PathRecords,\r
+including all their fields.\r
+It verifies the existing inventory, with all the object fields, and matches it\r
+to a pre-saved one.\r
+A Multicast Compliancy test.\r
+An Event Forwarding test.\r
+A Service Record registration test.\r
+An RMPP stress test.\r
+A Small SA Queries stress test.\r
+<P>\r
+It is recommended that after installing opensm, the user should run\r
+"osmtest -f c" to generate the inventory file, and\r
+immediately afterwards run "osmtest -f a" to test OpenSM.\r
+<P>\r
+Another recommendation for osmtest usage is to create the inventory when the\r
+IB fabric is stable, and occasionally\r
+run "osmtest -v" to verify that nothing has changed.\r
+<h3>OPTIONS</h3>\r
+\r
+<P>\r
+<P>\r
+\r
+<DL COMPACT>\r
+<DT><B>-f</B>, <B>--flow</B><DD>\r
+This option directs osmtest to run a specific flow:\r
+<BR> FLOW DESCRIPTION\r
+<BR> c = create an inventory file with all nodes, ports and paths\r
+<BR> a = run all validation tests (expecting an input inventory)\r
+<BR> v = only validate the given inventory file\r
+<BR> s = run service registration, deregistration, and lease test\r
+<BR> e = run event forwarding test\r
+<BR> f = flood the SA with queries according to the stress mode\r
+<BR> m = multicast flow\r
+<BR> q = QoS info: dump VLArb and SLtoVL tables\r
+<BR> t = run trap 64/65 flow (this flow requires running of external tool)\r
+<BR> (default is all flows except QoS)\r
+<DT><B>-w</B>, <B>--wait</B><DD>\r
+This option specifies the wait time for trap 64/65 in seconds\r
+It is used only when running -f t - the trap 64/65 flow\r
+(default to 10 sec)\r
+<DT><B>-d</B>, <B>--debug</B><DD>\r
+This option specifies a debug option.\r
+These options are not normally needed.\r
+The number following -d selects the debug\r
+option to enable as follows:\r
+<P>\r
+ OPT Description\r
+<BR> --- -----------------\r
+<BR> -d0 - Ignore other SM nodes\r
+<BR> -d1 - Force single threaded dispatching\r
+<BR> -d2 - Force log flushing after each log message\r
+<BR> -d3 - Disable multicast support\r
+<DT><B>-m</B>, <B>--max_lid</B><DD>\r
+This option specifies the maximal LID number to be searched\r
+for during inventory file build (default to 100)\r
+<DT><B>-g</B>, <B>--guid</B><DD>\r
+This option specifies the local port GUID value\r
+with which OpenSM should bind. OpenSM may be\r
+bound to 1 port at a time.\r
+If GUID given is 0, OpenSM displays a list\r
+of possible port GUIDs and waits for user input.\r
+Without -g, OpenSM trys to use the default port.\r
+<DT><B>-p</B>, <B>--port</B><DD>\r
+This option displays a menu of possible local port GUID values\r
+with which osmtest could bind\r
+<DT><B>-i</B>, <B>--inventory</B><DD>\r
+This option specifies the name of the inventory file\r
+Normally, osmtest expects to find an inventory file,\r
+which osmtest uses to validate real-time information\r
+received from the SA during testing\r
+If -i is not specified, osmtest defaults to the file\r
+'osmtest.dat'\r
+See -c option for related information\r
+<DT><B>-s</B>, <B>--stress</B><DD>\r
+This option runs the specified stress test instead\r
+of the normal test suite\r
+Stress test options are as follows:\r
+<P>\r
+ OPT Description\r
+<BR> --- -----------------\r
+<BR> -s1 - Single-MAD (RMPP) response SA queries\r
+<BR> -s2 - Multi-MAD (RMPP) response SA queries\r
+<BR> -s3 - Multi-MAD (RMPP) Path Record SA queries\r
+<BR> -s4 - Single-MAD (non RMPP) get Path Record SA queries\r
+<P>\r
+Without -s, stress testing is not performed\r
+<DT><B>-M</B>, <B>--Multicast_Mode</B><DD>\r
+This option specify length of Multicast test:\r
+<P>\r
+ OPT Description\r
+<BR> --- -----------------\r
+<BR> -M1 - Short Multicast Flow (default) - single mode\r
+<BR> -M2 - Short Multicast Flow - multiple mode\r
+<BR> -M3 - Long Multicast Flow - single mode\r
+<BR> -M4 - Long Multicast Flow - multiple mode\r
+<P>\r
+Single mode - Osmtest is tested alone, with no other\r
+apps that interact with OpenSM MC\r
+<P>\r
+Multiple mode - Could be run with other apps using MC with\r
+OpenSM. Without -M, default flow testing is performed\r
+<DT><B>-t</B>, <B>--timeout</B><DD>\r
+This option specifies the time in milliseconds\r
+used for transaction timeouts.\r
+Specifying -t 0 disables timeouts.\r
+Without -t, OpenSM defaults to a timeout value of\r
+200 milliseconds.\r
+<DT><B>-l</B>, <B>--log_file</B><DD>\r
+This option defines the log to be the given file.\r
+By default the log goes to stdout.\r
+<DT><B>-v</B>, <B>--verbose</B><DD>\r
+This option increases the log verbosity level.\r
+The -v option may be specified multiple times\r
+to further increase the verbosity level.\r
+See the -vf option for more information about.\r
+log verbosity.\r
+<DT><B>-V</B><DD>\r
+This option sets the maximum verbosity level and\r
+forces log flushing.\r
+The -V is equivalent to '-vf 0xFF -d 2'.\r
+See the -vf option for more information about.\r
+log verbosity.\r
+<DT><B>-vf</B><DD>\r
+This option sets the log verbosity level.\r
+A flags field must follow the -D option.\r
+A bit set/clear in the flags enables/disables a\r
+specific log level as follows:\r
+<P>\r
+ BIT LOG LEVEL ENABLED\r
+<BR> ---- -----------------\r
+<BR> 0x01 - ERROR (error messages)\r
+<BR> 0x02 - INFO (basic messages, low volume)\r
+<BR> 0x04 - VERBOSE (interesting stuff, moderate volume)\r
+<BR> 0x08 - DEBUG (diagnostic, high volume)\r
+<BR> 0x10 - FUNCS (function entry/exit, very high volume)\r
+<BR> 0x20 - FRAMES (dumps all SMP and GMP frames)\r
+<BR> 0x40 - ROUTING (dump FDB routing information)\r
+<BR> 0x80 - currently unused.\r
+<P>\r
+Without -vf, osmtest defaults to ERROR + INFO (0x3)\r
+Specifying -vf 0 disables all messages\r
+Specifying -vf 0xFF enables all messages (see -V)\r
+High verbosity levels may require increasing\r
+the transaction timeout with the -t option\r
+<DT><B>-h</B>, <B>--help</B><DD>\r
+Display this usage info then exit.\r
+<P>\r
+</DL>\r
+<h3>AUTHORS</h3>\r
+\r
+<DL COMPACT>\r
+<DT>Hal Rosenstock<DD>\r
+<<I><A HREF="mailto:hal.rosenstock@gmail.com">hal.rosenstock@gmail.com</A></I>>\r
+\r
+<DT>Eitan Zahavi<DD>\r
+<<I><A HREF="mailto:eitan@mellanox.co.il">eitan@mellanox.co.il</A></I>>\r
+\r
+<dt> </dt>\r
+<h3>EXAMPLES</h3>\r
+<p>Note - osmtest will not run on the node where OpenSM is running.<br>See 'osmtest -h' for all options.</p>\r
+<p>Functionality:</p>\r
+<blockquote>\r
+ <p>osmtest -f c \r
# creates osmtest.dat inventory file in the current directory; \r
required by other osmtest runs.<br>\r
- osmtest -f a \r
+ osmtest -f v \r
+ # validate the default inventory file 'osmtest.dat'.<br>osmtest -f a \r
# run all validation tests (expecting an input inventory file 'osmtest.dat' \r
in the current folder).</p>\r
- </blockquote>\r
- </dl>\r
+</blockquote>\r
+</DL>\r
<p>Stress tests</p>\r
<blockquote>\r
<p>osmtest -f f -s1 # \r
DAT ENVIRONMENT</font><font face="Courier New" size="2">:</font></h3>\r
</div>\r
<blockquote>\r
- <p align="left"><font face="Courier New" size="2">DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as \r
+ <p align="left"><font face="Courier New" size="2">DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot%\System32 as \r
dat2.dll and dapl2.dll. Debug versions of the v2.0 runtime libraries \r
are located in '%SystemDrive%\%ProgramFiles%\OFED'.</font></p>\r
<p align="left"><font face="Courier New" size="2">IA32 (aka, 32-bit) \r
<p align="left"><font face="Courier New" size="2">In order for DAT/DAPL \r
programs to execute correctly, the runtime library files 'dat2.dll and \r
dapl2.dll' must be present in one of the following folders: current \r
- directory, %SystemRoot% or in the library search path.</font></p>\r
+ directory, %SystemRoot%, %SystemRoot%\System32 or in the library search path.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default OFED \r
- installation places the runtime library files dat2.dll and dapl2.dll in the '%SystemRoot%' folder; \r
+ installation places the runtime library files dat2.dll and dapl2.dll in the '%SystemRoot%\System32' folder; \r
symbol files (.pdb) are located in '%ProgramFiles%\OFED\'.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default DAPL configuration \r
file is defined as '%SystemDrive%\DAT\dat.conf'. This default \r
<p align="left"><font face="Courier New" size="2">Within the dat.conf file, \r
the DAPL library specification can be located as the 5th whitespace \r
separated line argument. By default the DAPL library file is installed as\r
- '%SystemRoot%\dapl2.dll'.</font></p>\r
+ '%SystemRoot%\System32\dapl2.dll'.</font></p>\r
<p align="left"><font face="Courier New" size="2">Should you choose to \r
relocated the DAPL library file to a path where whitespace appears in the \r
full library path specification, then the full library file specification \r
<ul>\r
<li>\r
<p align="left"><font face="Courier New" size="2">File: \r
- %windir%\dapl2.dll</font></p></li>\r
+ %windir%\System32\dapl2.dll</font></p></li>\r
<li>\r
<p align="left"><font face="Courier New" size="2">dat.conf Provider \r
name: ibnic0v2</font></p></li>\r
<p align="left"><u>Socket-CM Provider</u></p>\r
<ul>\r
<li>\r
- <p align="left"><font face="Courier New" size="2">File: %windir%\dapl2-ofa-scm.dll</font></p>\r
+ <p align="left"><font face="Courier New" size="2">File: %windir%\System32\dapl2-ofa-scm.dll</font></p>\r
</li>\r
<li>\r
<p align="left"><font face="Courier New" size="2"><u>dat.conf</u> \r
<p align="left"><font face="Times New Roman">RDMA-CM Provider</font></p>\r
<ul>\r
<li>\r
- <p align="left"><font face="Courier New" size="2">File: %windir%\dapl2-ofa-cma.dll</font></p>\r
+ <p align="left"><font face="Courier New" size="2">File: %windir%\System32\dapl2-ofa-cma.dll</font></p>\r
</li>\r
<li>\r
<p align="left"><font face="Courier New" size="2">dat.conf Provider \r
<h4>DESCRIPTION</h4>\r
<b>ibv_get_device_list()</b> returns a NULL-terminated array of RDMA devices \r
currently available. The argument <i>num_devices</i> is optional; if not NULL, \r
-it is set to the number of devices returned in the array.
+it is set to the number of devices returned in the array.\r
<p><b>ibv_free_device_list()</b> frees the array of devices <i>list</i> returned \r
by <b>ibv_get_device_list()</b>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_get_device_list()</b> returns the array of available RDMA devices, or \r
sets <i>errno</i> and returns NULL if the request fails. If no devices are found \r
-then <i>num_devices</i> is set to 0, and non-NULL is returned.
+then <i>num_devices</i> is set to 0, and non-NULL is returned.\r
<p><b>ibv_free_device_list()</b> returns no value.</p>\r
<h4>ERRORS</h4>\r
<dl COMPACT>\r
<dt><b>EPERM</b> </dt>\r
- <dd>Permission denied.
+ <dd>Permission denied.\r
</dd>\r
<dt><b>ENOSYS</b> </dt>\r
- <dd>No kernel support for RDMA.
+ <dd>No kernel support for RDMA.\r
</dd>\r
<dt><b>ENOMEM</b> </dt>\r
<dd>Insufficient memory to complete the operation.</dd>\r
ibv_open_device()</b> before calling <b>ibv_free_device_list()</b>. Once it \r
frees the array with <b>ibv_free_device_list()</b>, it will be able to use only \r
the open devices; pointers to unopened devices will no longer be valid.\r
-<a NAME="lbAH"> </a>
+<a NAME="lbAH"> </a>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_GET_DEVICE_NAME">ibv_get_device_name</a></b>, <b>\r
<a href="#IBV_GET_DEVICE_GUID">ibv_get_device_guid</a></b>, <b>\r
<h3><a name="IBV_GET_DEVICE_GUID">IBV_GET_DEVICE_GUID</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_get_device_guid - get an RDMA device's GUID
+ibv_get_device_guid - get an RDMA device's GUID\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<b>ibv_get_device_name()</b> returns the Global Unique IDentifier (GUID) of the \r
RDMA device <i>device</i>.<h4>RETURN VALUE</h4>\r
<b>ibv_get_device_guid()</b> returns the GUID of the device in network byte \r
-order. <a NAME="lbAF"> </a>
+order. <a NAME="lbAF"> </a>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_GET_DEVICE_LIST">ibv_get_device_list</a></b>, <b>\r
<a href="#IBV_GET_DEVICE_NAME">ibv_get_device_name</a></b>, <b>\r
<h3><a name="IBV_CLOSE_DEVICE">IBV_CLOSE_DEVICE</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_open_device, ibv_close_device - open and close an RDMA device context
+ibv_open_device, ibv_close_device - open and close an RDMA device context\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
</pre>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_open_device()</b> opens the device <i>device</i> and creates a context \r
-for further use.
+for further use.\r
<p><b>ibv_close_device()</b> closes the device context <i>context</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_open_device()</b> returns a pointer to the allocated device context, or \r
-NULL if the request fails.
+NULL if the request fails.\r
<p><b>ibv_close_device()</b> returns 0 on success, -1 on failure.</p>\r
<h4>NOTES</h4>\r
<b>ibv_close_device()</b> does not release all the resources allocated using \r
context <i>context</i>. To avoid resource leaks, the user should release all \r
-associated resources before closing a context.
+associated resources before closing a context.\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_GET_DEVICE_LIST">ibv_get_device_list</a></b>, <b>\r
<a href="#IBV_QUERY_DEVICE">ibv_query_device</a></b>, <b>\r
<hr>\r
<h4>NAME</h4>\r
ibv_get_async_event, ibv_ack_async_event - get or acknowledge asynchronous \r
-events <a NAME="lbAC"> </a>
+events <a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_get_async_event()</b> waits for the next async event of the RDMA device \r
context <i>context</i> and returns it through the pointer <i>event</i>, which is \r
-an ibv_async_event struct, as defined in <infiniband/verbs.h>.
+an ibv_async_event struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_async_event {\r
union {\r
</dl>\r
<p><b>ibv_ack_async_event()</b> acknowledge the async event <i>event</i>. </p>\r
<h4>RETURN VALUE</h4>\r
-<b>ibv_get_async_event()</b> returns 0 on success, and -1 on error.
+<b>ibv_get_async_event()</b> returns 0 on success, and -1 on error.\r
<p><b>ibv_ack_async_event()</b> returns no value. </p>\r
<h4>NOTES</h4>\r
All async events that <b>ibv_get_async_event()</b> returns must be acknowledged \r
using <b>ibv_ack_async_event()</b>. To avoid races, destroying an object (CQ, \r
SRQ or QP) will wait for all affiliated events for the object to be \r
acknowledged; this avoids an application retrieving an affiliated event after \r
-the corresponding object has already been destroyed.
+the corresponding object has already been destroyed.\r
<p><b>ibv_get_async_event()</b> is a blocking function. If multiple threads call \r
this function simultaneously, then when an async event occurs, only one thread \r
will receive it, and it is not possible to predict which thread will receive it.\r
</p>\r
<h4>EXAMPLES</h4>\r
The following code example demonstrates one possible way to work with async \r
-events in non-blocking mode. It performs the following steps:
+events in non-blocking mode. It performs the following steps:\r
<p>1. Set the async events queue work mode to be non-blocked <br>\r
2. Poll the queue until it has an async event <br>\r
3. Get the async event and ack it </p>\r
\r
</pre>\r
<h4>SEE ALSO</h4>\r
-<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>
+<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_QUERY_DEVICE">IBV_QUERY_DEVICE</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_query_device - query an RDMA device's attributes <a NAME="lbAC"> </a>
+ibv_query_device - query an RDMA device's attributes <a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_query_device()</b> returns the attributes of the device with context <i>\r
context</i>. The argument <i>device_attr</i> is a pointer to an ibv_device_attr \r
-struct, as defined in <infiniband/verbs.h>.
+struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_device_attr {\r
char fw_ver[64]; /* FW version */\r
};</pre>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_query_device()</b> returns 0 on success, or the value of errno on failure \r
-(which indicates the failure reason). <a NAME="lbAF"> </a>
+(which indicates the failure reason). <a NAME="lbAF"> </a>\r
<h4>NOTES</h4>\r
The maximum values returned by this function are the upper limits of supported \r
resources by the device. However, it may not be possible to use these maximum \r
users/processes.<h4>SEE ALSO</h4>\r
<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>, <b>\r
<a href="#IBV_QUERY_PORT">ibv_query_port</a></b>, <b><a href="#IBV_QUERY_PKEY">ibv_query_pkey</a></b>,\r
-<b><a href="#IBV_QUERY_GID">ibv_query_gid</a></b>
+<b><a href="#IBV_QUERY_GID">ibv_query_gid</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_QUERY_GID">IBV_QUERY_GID</a></h3>\r
<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>, <b>\r
<a href="#IBV_QUERY_DEVICE">ibv_query_device</a></b>, <b>\r
<a href="#IBV_QUERY_PORT">ibv_query_port</a></b>,\r
-<b><a href="#IBV_QUERY_PKEY">ibv_query_pkey</a></b>
+<b><a href="#IBV_QUERY_PKEY">ibv_query_pkey</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_QUERY_PKEY">IBV_QUERY_PKEY</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_query_pkey - query an InfiniBand port's P_Key table
+ibv_query_pkey - query an InfiniBand port's P_Key table\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<b>ibv_query_pkey()</b> returns the P_Key value (in network byte order) in entry\r
<i>index</i> of port <i>port_num</i> for device context <i>context</i> through \r
the pointer <i>pkey</i>.<h4>RETURN VALUE</h4>\r
-<b>ibv_query_pkey()</b> returns 0 on success, and -1 on error.
+<b>ibv_query_pkey()</b> returns 0 on success, and -1 on error.\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>, <b>\r
<a href="#IBV_QUERY_DEVICE">ibv_query_device</a></b>, <b>\r
<a name="IBV_QUERY_PORT">IBV_QUERY_PORT</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_query_port - query an RDMA port's attributes
+ibv_query_port - query an RDMA port's attributes\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_query_port()</b> returns the attributes of port <i>port_num</i> for \r
device context <i>context</i> through the pointer <i>port_attr</i>. The argument\r
-<i>port_attr</i> is an ibv_port_attr struct, as defined in <infiniband/verbs.h>.
+<i>port_attr</i> is an ibv_port_attr struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_port_attr {\r
enum ibv_port_state state; /* Logical port state */\r
};</pre>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_query_port()</b> returns 0 on success, or the value of errno on failure \r
-(which indicates the failure reason).
+(which indicates the failure reason).\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_CREATE_QP">ibv_create_qp</a></b>, <b><a href="#IBV_DESTROY_QP">ibv_destroy_qp</a></b>, <b>\r
<a href="#IBV_QUERY_QP">ibv_query_qp</a></b>, <b>\r
\r
<b>int ibv_dealloc_pd(struct ibv_pd </b><i>*pd</i><b>);</b></pre>\r
<h4>DESCRIPTION</h4>\r
-<b>ibv_alloc_pd()</b> allocates a PD for the RDMA device context <i>context</i>.
-
+<b>ibv_alloc_pd()</b> allocates a PD for the RDMA device context <i>context</i>.\r
+\r
<p><b>ibv_dealloc_pd()</b> deallocates the PD <i>pd</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_alloc_pd()</b> returns a pointer to the allocated PD, or NULL if the \r
-request fails.
+request fails.\r
<p><b>ibv_dealloc_pd()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason). <a NAME="lbAF"> </a> </p>\r
<h4>NOTES</h4>\r
<b>ibv_dealloc_pd()</b> may fail if any other resource is still associated with \r
-the PD being freed. <a NAME="lbAG"> </a>
+the PD being freed. <a NAME="lbAG"> </a>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_REG_MR">ibv_reg_mr</a></b>, <b><a href="#IBV_CREATE_SRQ">ibv_create_srq</a></b>, <b>\r
<a href="#IBV_CREATE_QP">ibv_create_qp</a></b>, <b>\r
<hr>\r
<h4>NAME</h4>\r
ibv_reg_mr, ibv_dereg_mr - register or deregister a memory region (MR)\r
-<a NAME="lbAC"> </a>
+<a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
protection domain <i>pd</i>. The MR's starting address is <i>addr</i> and its \r
size is <i>length</i>. The argument <i>access</i> describes the desired memory \r
protection attributes; it is either 0 or the bitwise OR of one or more of the \r
-following flags:
+following flags:\r
<p></p>\r
<dl COMPACT>\r
<dt><b>IBV_ACCESS_LOCAL_WRITE </b>Enable Local Write Access </dt>\r
lkey field of struct ibv_sge when posting buffers with ibv_post_* verbs, and the \r
the remote key (<b>R_Key</b>) field <b>rkey</b> is used by remote processes to \r
perform Atomic and RDMA operations. The remote process places this <b>rkey</b> \r
-as the rkey field of struct ibv_send_wr passed to the ibv_post_send function.
+as the rkey field of struct ibv_send_wr passed to the ibv_post_send function.\r
<p><b>ibv_dereg_mr()</b> returns 0 on success, or the value of errno on failure \r
(which indicates the failure reason).</p>\r
<h4>NOTES</h4>\r
SEE ALSO</h4>\r
<b><a href="#IBV_ALLOC_PD">ibv_alloc_pd</a></b>, <b><a href="#IBV_POST_SEND">ibv_post_send</a></b>, <b>\r
<a href="#IBV_POST_RECV">ibv_post_recv</a></b>, <b>\r
-<a href="#IBV_POST_SRQ_RECV">ibv_post_srq_recv</a></b>
+<a href="#IBV_POST_SRQ_RECV">ibv_post_srq_recv</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_CREATE_AH">IBV_CREATE_AH</a></h3>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_create_ah()</b> creates an address handle (AH) associated with the \r
protection domain <i>pd</i>. The argument <i>attr</i> is an ibv_ah_attr struct, \r
-as defined in <infiniband/verbs.h>.
+as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_ah_attr {\r
struct ibv_global_route grh; /* Global Routing Header (GRH) attributes */\r
<p><b>ibv_destroy_ah()</b> destroys the AH <i>ah</i>. </p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_create_ah()</b> returns a pointer to the created AH, or NULL if the \r
-request fails.
+request fails.\r
<p><b>ibv_destroy_ah()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason).</p>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_ALLOC_PD">ibv_alloc_pd</a></b>, <b>\r
<a href="#IBV_INIT_AH_FROM_WC">ibv_init_ah_from_wc</a></b>, <b>\r
-<a href="#IBV_CREATE_AH_FROM_WC">ibv_create_ah_from_wc</a></b>
+<a href="#IBV_CREATE_AH_FROM_WC">ibv_create_ah_from_wc</a></b>\r
<p align="left"> </p>\r
<h3><br>\r
<a name="IBV_CREATE_AH_FROM_WC">IBV_CREATE_AH_FROM_WC</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
ibv_init_ah_from_wc, ibv_create_ah_from_wc - initialize or create an address \r
-handle (AH) from a work completion <a NAME="lbAC"> </a>
+handle (AH) from a work completion <a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<b>ibv_init_ah_from_wc()</b> initializes the address handle (AH) attribute \r
structure <i>ah_attr</i> for the RDMA device context <i>context</i> using the \r
port number <i>port_num</i>, using attributes from the work completion <i>wc</i> \r
-and the Global Routing Header (GRH) structure <i>grh</i>.
-
+and the Global Routing Header (GRH) structure <i>grh</i>.\r
+\r
<p><b>ibv_create_ah_from_wc()</b> creates an AH associated with the protection \r
domain <i>pd</i> using the port number <i>port_num</i>, using attributes from \r
the work completion <i>wc</i> and the Global Routing Header (GRH) structure <i>\r
grh</i>. </p>\r
<h4>RETURN VALUE</h4>\r
-<b>ibv_init_ah_from_wc()</b> returns 0 on success, and -1 on error.
+<b>ibv_init_ah_from_wc()</b> returns 0 on success, and -1 on error.\r
<p><b>ibv_create_ah_from_wc()</b> returns a pointer to the created AH, or NULL \r
if the request fails. <a NAME="lbAF"> </a> </p>\r
<h4>NOTES</h4>\r
The filled structure <i>ah_attr</i> returned from <b>ibv_init_ah_from_wc()</b> \r
-can be used to create a new AH using <b>ibv_create_ah()</b>.
+can be used to create a new AH using <b>ibv_create_ah()</b>.\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_OPEN_DEVICE">ibv_open_device</a></b>, <b>\r
<a href="#IBV_ALLOC_PD">ibv_alloc_pd</a></b>, <b><a href="#IBV_CREATE_AH">ibv_create_ah</a></b>, <b>\r
<b>int ibv_destroy_comp_channel(struct ibv_comp_channel </b><i>*channel</i><b>);</b></pre>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_create_comp_channel()</b> creates a completion event channel for the RDMA \r
-device context <i>context</i>.
-
+device context <i>context</i>.\r
+\r
<p><b>ibv_destroy_comp_channel()</b> destroys the completion event channel <i>\r
channel</i>. </p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_create_comp_channel()</b> returns a pointer to the created completion \r
-event channel, or NULL if the request fails.
+event channel, or NULL if the request fails.\r
<p><b>ibv_destroy_comp_channel()</b> returns 0 on success, or the value of errno \r
on failure (which indicates the failure reason).</p>\r
<h4>NOTES</h4>\r
to deliver completion notifications to a userspace process. When a completion \r
event is generated for a completion queue (CQ), the event is delivered via the \r
completion channel attached to that CQ. This may be useful to steer completion \r
-events to different threads by using multiple completion channels.
+events to different threads by using multiple completion channels.\r
<p><b>ibv_destroy_comp_channel()</b> fails if any CQs are still associated with \r
the completion event channel being destroyed.</p>\r
<h4>SEE ALSO</h4>\r
<hr>\r
<h4>NAME</h4>\r
ibv_create_cq, ibv_destroy_cq - create or destroy a completion queue (CQ)\r
-<a NAME="lbAC"> </a>
+<a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
channel</i> is optional; if not NULL, the completion channel <i>channel</i> will \r
be used to return completion events. The CQ will use the completion vector <i>\r
comp_vector</i> for signaling completion events; it must be at least zero and \r
-less than <i>context</i>->num_comp_vectors.
-
+less than <i>context</i>->num_comp_vectors.\r
+\r
<p><b>ibv_destroy_cq()</b> destroys the CQ <i>cq</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_create_cq()</b> returns a pointer to the CQ, or NULL if the request \r
-fails.
+fails.\r
<p><b>ibv_destroy_cq()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason).</p>\r
<h4>NOTES</h4>\r
<b>ibv_create_cq()</b> may create a CQ with size greater than or equal to the \r
-requested size. Check the cqe attribute in the returned CQ for the actual size.
+requested size. Check the cqe attribute in the returned CQ for the actual size.\r
<p><b>ibv_destroy_cq()</b> fails if any queue pair is still associated with this \r
CQ. </p>\r
<h4>SEE ALSO</h4>\r
<h3><a name="IBV_POLL_CQ">IBV_POLL_CQ</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_poll_cq - poll a completion queue (CQ) <a NAME="lbAC"> </a>
+ibv_poll_cq - poll a completion queue (CQ) <a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<b>ibv_poll_cq()</b> polls the CQ <i>cq</i> for work completions and returns the \r
first <i>num_entries</i> (or all available completions if the CQ contains fewer \r
than this number) in the array <i>wc</i>. The argument <i>wc</i> is a pointer to \r
-an array of ibv_wc structs, as defined in <infiniband/verbs.h>.
+an array of ibv_wc structs, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_wc {\r
uint64_t wr_id; /* ID of the completed Work Request (WR) */\r
requested size. The cqe member of <i>cq</i> will be updated to the actual size.<h4>\r
SEE ALSO</h4>\r
<a href="#IBV_CREATE_CQ">\r
-<b>ibv_create_cq</b> </a> <b><a href="#IBV_DESTROY_CQ">ibv_destroy_cq</a></b>
+<b>ibv_create_cq</b> </a> <b><a href="#IBV_DESTROY_CQ">ibv_destroy_cq</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_GET_CQ_EVENT">IBV_GET_CQ_EVENT</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
ibv_get_cq_event, ibv_ack_cq_events - get and acknowledge completion queue (CQ) \r
-events
+events\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_get_cq_event()</b> waits for the next completion event in the completion \r
event channel <i>channel</i>. Fills the arguments <i>cq</i> with the CQ that got \r
-the event and <i>cq_context</i> with the CQ's context.
+the event and <i>cq_context</i> with the CQ's context.\r
<p><b>ibv_ack_cq_events()</b> acknowledges <i>nevents</i> events on the CQ <i>cq</i>.</p>\r
<h4>RETURN VALUE</h4>\r
-<b>ibv_get_cq_event()</b> returns 0 on success, and -1 on error.
+<b>ibv_get_cq_event()</b> returns 0 on success, and -1 on error.\r
<p><b>ibv_ack_cq_events()</b> returns no value. <a NAME="lbAF"> </a> </p>\r
<h4>NOTES</h4>\r
All completion events that <b>ibv_get_cq_event()</b> returns must be \r
acknowledged using <b>ibv_ack_cq_events()</b>. To avoid races, destroying a CQ \r
will wait for all completion events to be acknowledged; this guarantees a \r
-one-to-one correspondence between acks and successful gets.
+one-to-one correspondence between acks and successful gets.\r
<p>Calling <b>ibv_ack_cq_events()</b> may be relatively expensive in the \r
datapath, since it must take a mutex. Therefore it may be better to amortize \r
this cost by keeping a count of the number of events needing acknowledgement and \r
acking several completion events in one call to <b>ibv_ack_cq_events()</b>.</p>\r
<h4>EXAMPLES</h4>\r
The following code example demonstrates one possible way to work with completion \r
-events. It performs the following steps:
+events. It performs the following steps:\r
<p>Stage I: Preparation <br>\r
1. Creates a CQ <br>\r
2. Requests for notification upon a new (first) completion event </p>\r
<b>int ibv_req_notify_cq(struct ibv_cq </b><i>*cq</i><b>, int </b><i>solicited_only</i><b>);</b></pre>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_req_notify_cq()</b> requests a completion notification on the completion \r
-queue (CQ) <i>cq</i>.
-
+queue (CQ) <i>cq</i>.\r
+\r
<p>Upon the addition of a new CQ entry (CQE) to <i>cq</i>, a completion event \r
will be added to the completion channel associated with the CQ. If the argument\r
<i>solicited_only</i> is zero, a completion event is generated for any new CQE. \r
<b>ibv_req_notify_cq()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason).<h4>NOTES</h4>\r
The request for notification is "one shot." Only one completion event will be \r
-generated for each call to <b>ibv_req_notify_cq()</b>. <a NAME="lbAG"> </a>
+generated for each call to <b>ibv_req_notify_cq()</b>. <a NAME="lbAG"> </a>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_CREATE_COMP_CHANNEL">ibv_create_comp_channel</a></b>, <b>\r
<a href="#IBV_CREATE_CQ">ibv_create_cq</a></b>, <b><a href="#IBV_GET_CQ_EVENT">ibv_get_cq_event</a></b><p> </p>\r
<hr>\r
<h4>NAME</h4>\r
ibv_create_srq, ibv_destroy_srq - create or destroy a shared receive queue (SRQ)\r
-<a NAME="lbAC"> </a>
+<a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<b>int ibv_destroy_srq(struct ibv_srq </b><i>*srq</i><b>);</b></pre>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_create_srq()</b> creates a shared receive queue (SRQ) associated with the \r
-protection domain <i>pd</i>.
-
+protection domain <i>pd</i>.\r
+\r
<p><b>ibv_create_xrc_srq()</b> creates an XRC shared receive queue (SRQ) \r
associated with the protection domain <i>pd</i>, the XRC domain <i>xrc_domain</i> \r
and the CQ which will hold the XRC completion <i>xrc_cq</i>. </p>\r
<p><b>ibv_destroy_srq()</b> destroys the SRQ <i>srq</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_create_srq()</b> returns a pointer to the created SRQ, or NULL if the \r
-request fails.
+request fails.\r
<p><b>ibv_destroy_srq()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason). </p>\r
<h4>NOTES</h4>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_modify_srq()</b> modifies the attributes of SRQ <i>srq</i> with the \r
attributes in <i>srq_attr</i> according to the mask <i>srq_attr_mask</i>. The \r
-argument <i>srq_attr</i> is an ibv_srq_attr struct, as defined in <infiniband/verbs.h>.
+argument <i>srq_attr</i> is an ibv_srq_attr struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_srq_attr {\r
uint32_t max_wr; /* maximum number of outstanding work requests (WRs) in the SRQ */\r
<b>ibv_modify_srq()</b> returns 0 on success, or the value of errno on failure \r
(which indicates the failure reason).<h4>NOTES</h4>\r
If any of the modify attributes is invalid, none of the attributes will be \r
-modified.
+modified.\r
<p>Not all devices support resizing SRQs. To check if a device supports it, \r
check if the <b>IBV_DEVICE_SRQ_RESIZE</b> bit is set in the device capabilities \r
flags. </p>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_query_srq()</b> gets the attributes of the SRQ <i>srq</i> and returns \r
them through the pointer <i>srq_attr</i>. The argument <i>srq_attr</i> is an \r
-ibv_srq_attr struct, as defined in <infiniband/verbs.h>.
+ibv_srq_attr struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_srq_attr {\r
uint32_t max_wr; /* maximum number of outstanding work requests (WRs) in the SRQ */\r
(which indicates the failure reason).<h4>NOTES</h4>\r
If the value returned for srq_limit is 0, then the SRQ limit reached ("low \r
watermark") event is not (or no longer) armed, and no asynchronous events will \r
-be generated until the event is rearmed. <a NAME="lbAG"> </a>
+be generated until the event is rearmed. <a NAME="lbAG"> </a>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_CREATE_SRQ">ibv_create_srq</a></b>, <b>\r
<a href="#IBV_DESTROY_SRQ">ibv_destroy_srq</a></b>, <b>\r
will use <i>xrc_rcv_qpn</i> in <b>ibv_post_send()</b> when sending to an XRC SRQ \r
on this host in the same xrc domain as the XRC receive QP. This QP is created in \r
kernel space, and persists until the last process registered for the QP calls <b>\r
-ibv_unreg_xrc_rcv_qp()</b> (at which time the QP is destroyed).
+ibv_unreg_xrc_rcv_qp()</b> (at which time the QP is destroyed).\r
<p>The process which creates this QP is automatically registered for it, and \r
should also call <b>ibv_unreg_xrc_rcv_qp()</b> at some point, to unregister. </p>\r
<p>Processes which wish to receive on an XRC SRQ via this QP should call <b>\r
with the attributes in <i>attr</i> according to the mask <i>attr_mask</i> and \r
move the QP state through the following transitions: Reset -> Init -> RTR. <i>\r
attr_mask</i> should indicate all of the attributes which will be used in this \r
-QP transition and the following masks (at least) should be set:
+QP transition and the following masks (at least) should be set:\r
<p></p>\r
<pre>Next state Required attributes\r
---------- ----------------------------------------\r
</dl>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_modify_xrc_rcv_qp()</b> returns 0 on success, or the value of errno on \r
-failure (which indicates the failure reason).
+failure (which indicates the failure reason).\r
<h4>NOTES</h4>\r
If any of the modify attributes or the modify mask are invalid, none of the \r
-attributes will be modified (including the QP state).
+attributes will be modified (including the QP state).\r
<p>Not all devices support alternate paths. To check if a device supports it, \r
check if the <b>IBV_DEVICE_AUTO_PATH_MIG</b> bit is set in the device \r
capabilities flags.</p>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_OPEN_XRC_DOMAIN">ibv_open_xrc_domain</a></b>, <b>\r
<a href="#IBV_CREATE_XRC_RCV_QP">ibv_create_xrc_rcv_qp</a></b>, <b>\r
-<a href="#IBV_QUERY_XRC_RCV_QP">ibv_query_xrc_rcv_qp</a></b>
+<a href="#IBV_QUERY_XRC_RCV_QP">ibv_query_xrc_rcv_qp</a></b>\r
<h3> </h3>\r
<h3><br>\r
<a name="IBV_OPEN_XRC_DOMAIN">IBV_OPEN_XRC_DOMAIN</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
ibv_open_xrc_domain, ibv_close_xrc_domain - open or close an eXtended Reliable \r
-Connection (XRC) domain
+Connection (XRC) domain\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <fcntl.h></b>\r
<b>#include <infiniband/verbs.h></b>\r
context <i>context</i> or return a reference to an opened one. <i>fd</i> is the \r
file descriptor to be associated with the XRC domain. The argument <i>oflag</i> \r
describes the desired file creation attributes; it is either 0 or the bitwise OR \r
-of one or more of the following flags:
+of one or more of the following flags:\r
<p></p>\r
<dl COMPACT>\r
<dt><b>O_CREAT</b> </dt>\r
<dd>If a domain belonging to device named by context is already associated \r
with the inode, this flag has no effect, except as noted under <b>O_EXCL</b> \r
below. Otherwise, a new XRC domain is created and is associated with inode \r
- specified by <i>fd</i>.
-
+ specified by <i>fd</i>.\r
+\r
</dd>\r
<dt><b>O_EXCL</b> </dt>\r
<dd>If <b>O_EXCL</b> and <b>O_CREAT</b> are set, open will fail if a domain \r
associated with the inode exists. The check for the existence of the domain \r
and creation of the domain if it does not exist is atomic with respect to \r
- other processes executing open with <i>fd</i> naming the same inode.
+ other processes executing open with <i>fd</i> naming the same inode.\r
</dd>\r
</dl>\r
<p>If <i>fd</i> equals -1, no inode is is associated with the domain, and the \r
last reference, the XRC domain will be destroyed. </p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_open_xrc_domain()</b> returns a pointer to an opened XRC, or NULL if the \r
-request fails.
+request fails.\r
<p><b>ibv_close_xrc_domain()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason).</p>\r
<h4>NOTES</h4>\r
Not all devices support XRC. To check if a device supports it, check if the <b>\r
-IBV_DEVICE_XRC</b> bit is set in the device capabilities flags.
+IBV_DEVICE_XRC</b> bit is set in the device capabilities flags.\r
<p><b>ibv_close_xrc_domain()</b> may fail if any QP or SRQ are still associated \r
with the XRC domain being closed.</p>\r
<h4>SEE ALSO</h4>\r
<a name="IBV_QUERY_XRC_RCV_QP">IBV_QUERY_XRC_RCV_QP</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_query_xrc_rcv_qp - get the attributes of an XRC receive queue pair (QP)
+ibv_query_xrc_rcv_qp - get the attributes of an XRC receive queue pair (QP)\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
for the XRC receive QP with the number <i>xrc_qp_num</i> which is associated \r
with the XRC domain <i>xrc_domain</i> and returns them through the pointers <i>\r
attr</i> and <i>init_attr</i>. The argument <i>attr</i> is an ibv_qp_attr \r
-struct, as defined in <infiniband/verbs.h>.
+struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_qp_attr {\r
enum ibv_qp_state qp_state; /* Current QP state */\r
failure (which indicates the failure reason).<h4>NOTES</h4>\r
The argument <i>attr_mask</i> is a hint that specifies the minimum list of \r
attributes to retrieve. Some InfiniBand devices may return extra attributes not \r
-requested, for example if the value can be returned cheaply.
+requested, for example if the value can be returned cheaply.\r
<p>Attribute values are valid if they have been set using <b>\r
ibv_modify_xrc_rcv_qp()</b>. The exact list of valid attributes depends on the \r
QP state. </p>\r
<hr>\r
<h4>NAME</h4>\r
ibv_reg_xrc_rcv_qp, ibv_unreg_xrc_rcv_qp - register and unregister a user \r
-process with an XRC receive queue pair (QP) <a NAME="lbAC"> </a>
+process with an XRC receive queue pair (QP) <a NAME="lbAC"> </a>\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_reg_xrc_rcv_qp()</b> registers a user process with the XRC receive QP \r
(created via <b>ibv_create_xrc_rcv_qp()</b> ) whose number is <i>xrc_qp_num</i>, \r
-and which is associated with the XRC domain <i>xrc_domain</i>.
-
+and which is associated with the XRC domain <i>xrc_domain</i>.\r
+\r
<p><b>ibv_unreg_xrc_rcv_qp()</b> unregisters a user process from the XRC receive \r
QP number <i>xrc_qp_num</i>, which is associated with the XRC domain <i>\r
xrc_domain</i>. When the number of user processes registered with this XRC \r
<b>ibv_reg_xrc_rcv_qp()</b> and <b>ibv_unreg_xrc_rcv_qp()</b> may fail if the \r
number <i>xrc_qp_num</i> is not a number of a valid XRC receive QP (the QP is \r
not allocated or it is the number of a non-XRC QP), or the XRC receive QP was \r
-created with an XRC domain other than <i>xrc_domain</i>.
-
+created with an XRC domain other than <i>xrc_domain</i>.\r
+\r
<p>If a process is still registered with any XRC RCV QPs belonging to some \r
domain, <b>ibv_close_xrc_domain()</b> will return failure if called for that \r
domain in that process. </p>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_create_qp()</b> creates a queue pair (QP) associated with the protection \r
domain <i>pd</i>. The argument <i>qp_init_attr</i> is an ibv_qp_init_attr \r
-struct, as defined in <infiniband/verbs.h>.
+struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_qp_init_attr {\r
void *qp_context; /* Associated context of the QP */\r
<p><b>ibv_destroy_qp()</b> destroys the QP <i>qp</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_create_qp()</b> returns a pointer to the created QP, or NULL if the \r
-request fails. Check the QP number (<b>qp_num</b>) in the returned QP.
+request fails. Check the QP number (<b>qp_num</b>) in the returned QP.\r
<p><b>ibv_destroy_qp()</b> returns 0 on success, or the value of errno on \r
failure (which indicates the failure reason).</p>\r
<h4>NOTES</h4>\r
<b>ibv_create_qp()</b> will fail if a it is asked to create QP of a type other \r
-than <b>IBV_QPT_RC</b> or <b>IBV_QPT_UD</b> associated with an SRQ.
+than <b>IBV_QPT_RC</b> or <b>IBV_QPT_UD</b> associated with an SRQ.\r
<p>The attributes max_recv_wr and max_recv_sge are ignored by <b>ibv_create_qp()</b> \r
if the QP is to be associated with an SRQ. </p>\r
<p><b>ibv_destroy_qp()</b> fails if the QP is attached to a multicast group.</p>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_ALLOC_PD">ibv_alloc_pd</a></b>, <b><a href="#IBV_MODIFY_QP">ibv_modify_qp</a></b>, <b>\r
-<a href="#IBV_QUERY_QP">ibv_query_qp</a></b>
+<a href="#IBV_QUERY_QP">ibv_query_qp</a></b>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_MODIFY_QP">IBV_MODIFY_QP</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_modify_qp - modify the attributes of a queue pair (QP)
+ibv_modify_qp - modify the attributes of a queue pair (QP)\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
<h4>DESCRIPTION</h4>\r
<b>ibv_modify_qp()</b> modifies the attributes of QP <i>qp</i> with the \r
attributes in <i>attr</i> according to the mask <i>attr_mask</i>. The argument\r
-<i>attr</i> is an ibv_qp_attr struct, as defined in <infiniband/verbs.h>.
+<i>attr</i> is an ibv_qp_attr struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_qp_attr {\r
enum ibv_qp_state qp_state; /* Move the QP to this state */\r
<b>ibv_modify_qp()</b> returns 0 on success, or the value of errno on failure \r
(which indicates the failure reason).<h4>NOTES</h4>\r
If any of the modify attributes or the modify mask are invalid, none of the \r
-attributes will be modified (including the QP state).
+attributes will be modified (including the QP state).\r
<p>Not all devices support resizing QPs. To check if a device supports it, check \r
if the <b>IBV_DEVICE_RESIZE_MAX_WR</b> bit is set in the device capabilities \r
flags. </p>\r
with <i>wr</i> to the receive queue of the queue pair <i>qp</i>. It stops \r
processing WRs from this list at the first failure (that can be detected \r
immediately while requests are being posted), and returns this failing WR \r
-through <i>bad_wr</i>.
-
+through <i>bad_wr</i>.\r
+\r
<p>The argument <i>wr</i> is an ibv_recv_wr struct, as defined in <infiniband/verbs.h>.\r
</p>\r
<p></p>\r
};</pre>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_post_recv()</b> returns 0 on success, or the value of errno on failure \r
-(which indicates the failure reason).
+(which indicates the failure reason).\r
<h4>NOTES</h4>\r
The buffers used by a WR can only be safely reused after WR the request is fully \r
executed and a work completion has been retrieved from the corresponding \r
-completion queue (CQ).
+completion queue (CQ).\r
<p>If the QP <i>qp</i> is associated with a shared receive queue, you must use \r
the function <b>ibv_post_srq_recv()</b>, and not <b>ibv_post_recv()</b>, since \r
the QP's own receive queue will not be used. </p>\r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_CREATE_QP">ibv_create_qp</a></b>, <b><a href="#IBV_POST_SEND">ibv_post_send</a></b>, <b>\r
<a href="#IBV_POST_SRQ_RECV">ibv_post_srq_recv</a></b>,\r
-<b><a href="#IBV_POLL_CQ">ibv_poll_cq</a></b>
+<b><a href="#IBV_POLL_CQ">ibv_poll_cq</a></b>\r
<p> </p>\r
<p> </p>\r
<h3><br>\r
<a name="IBV_POST_SEND">IBV_POST_SEND</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-ibv_post_send - post a list of work requests (WRs) to a send queue
+ibv_post_send - post a list of work requests (WRs) to a send queue\r
<h4>SYNOPSIS</h4>\r
<pre><b>#include <infiniband/verbs.h></b>\r
\r
with <i>wr</i> to the send queue of the queue pair <i>qp</i>. It stops \r
processing WRs from this list at the first failure (that can be detected \r
immediately while requests are being posted), and returns this failing WR \r
-through <i>bad_wr</i>.
-
+through <i>bad_wr</i>.\r
+\r
<p>The argument <i>wr</i> is an ibv_send_wr struct, as defined in <infiniband/verbs.h>.\r
</p>\r
<p></p>\r
<dt><b>IBV_SEND_INLINE </b>Send data in given gather list as inline data\r
</dt>\r
<dd>in a send WQE. Valid only for Send and RDMA Write. The L_Key will not be \r
- checked.
+ checked.\r
</dd>\r
</dl>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_post_send()</b> returns 0 on success, or the value of errno on failure \r
-(which indicates the failure reason).
+(which indicates the failure reason).\r
<h4>NOTES</h4>\r
The user should not alter or destroy AHs associated with WRs until request is \r
fully executed and a work completion has been retrieved from the corresponding \r
-completion queue (CQ) to avoid unexpected behavior.
+completion queue (CQ) to avoid unexpected behavior.\r
<p>The buffers used by a WR can only be safely reused after WR the request is \r
fully executed and a work completion has been retrieved from the corresponding \r
completion queue (CQ). However, if the IBV_SEND_INLINE flag was set, the buffer \r
<b>ibv_post_srq_recv()</b> posts the linked list of work requests (WRs) starting \r
with <i>wr</i> to the shared receive queue (SRQ) <i>srq</i>. It stops processing \r
WRs from this list at the first failure (that can be detected immediately while \r
-requests are being posted), and returns this failing WR through <i>bad_wr</i>.
-
+requests are being posted), and returns this failing WR through <i>bad_wr</i>.\r
+\r
<p>The argument <i>wr</i> is an ibv_recv_wr struct, as defined in <infiniband/verbs.h>.\r
</p>\r
<p></p>\r
failure (which indicates the failure reason).<h4>NOTES</h4>\r
The buffers used by a WR can only be safely reused after WR the request is fully \r
executed and a work completion has been retrieved from the corresponding \r
-completion queue (CQ).
+completion queue (CQ).\r
<p>If a WR is being posted to a UD QP, the Global Routing Header (GRH) of the \r
incoming message will be placed in the first 40 bytes of the buffer(s) in the \r
scatter list. If no GRH is present in the incoming message, then the first bytes \r
<h4>SEE ALSO</h4>\r
<b><a href="#IBV_CREATE_QP">ibv_create_qp</a></b>, <b><a href="#IBV_POST_SEND">ibv_post_send</a></b>, <b>\r
<a href="#IBV_POST_RECV">ibv_post_recv</a></b>, <b>\r
-<a href="#IBV_POLL_CQ">ibv_poll_cq</a></b>
+<a href="#IBV_POLL_CQ">ibv_poll_cq</a></b>\r
<p> </p>\r
<p> </p>\r
<h3><br>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_query_qp()</b> gets the attributes specified in <i>attr_mask</i> for the \r
QP <i>qp</i> and returns them through the pointers <i>attr</i> and <i>init_attr</i>. \r
-The argument <i>attr</i> is an ibv_qp_attr struct, as defined in <infiniband/verbs.h>.
+The argument <i>attr</i> is an ibv_qp_attr struct, as defined in <infiniband/verbs.h>.\r
<p></p>\r
<pre>struct ibv_qp_attr {\r
enum ibv_qp_state qp_state; /* Current QP state */\r
The argument <i>attr_mask</i> is a hint that specifies the minimum list of \r
attributes to retrieve. Some RDMA devices may return extra attributes not \r
requested, for example if the value can be returned cheaply. This has the same \r
-form as in <b>ibv_modify_qp()</b>.
-
+form as in <b>ibv_modify_qp()</b>.\r
+\r
<p>Attribute values are valid if they have been set using <b>ibv_modify_qp()</b>. \r
The exact list of valid attributes depends on the QP state. </p>\r
<p>Multiple calls to <b>ibv_query_qp()</b> may yield some differences in the \r
<b>int ibv_detach_mcast(struct ibv_qp </b><i>*qp</i><b>, const union ibv_gid </b><i>*gid</i><b>,</b> <b>uint16_t </b><i>lid</i><b>);</b></pre>\r
<h4>DESCRIPTION</h4>\r
<b>ibv_attach_mcast()</b> attaches the QP <i>qp</i> to the multicast group \r
-having MGID <i>gid</i> and MLID <i>lid</i>.
-
+having MGID <i>gid</i> and MLID <i>lid</i>.\r
+\r
<p><b>ibv_detach_mcast()</b> detaches the QP <i>qp</i> to the multicast group \r
having MGID <i>gid</i> and MLID <i>lid</i>.</p>\r
<h4>RETURN VALUE</h4>\r
<b>ibv_attach_mcast()</b> and <b>ibv_detach_mcast()</b> returns 0 on success, or \r
the value of errno on failure (which indicates the failure reason).<h4>NOTES</h4>\r
Only QPs of Transport Service Type <b>IBV_QPT_UD</b> may be attached to \r
-multicast groups.
+multicast groups.\r
<p>If a QP is attached to the same multicast group multiple times, the QP will \r
still receive a single copy of a multicast message. </p>\r
<p>In order to receive multicast messages, a join request for the multicast \r
<h4>DESCRIPTION</h4>\r
<b>ibv_rate_to_mult()</b> converts the IB transmission rate enumeration <i>rate</i> \r
to a multiple of 2.5 Gbit/sec (the base rate). For example, if <i>rate</i> is <b>\r
-IBV_RATE_5_GBPS</b>, the value 2 will be returned (5 Gbit/sec = 2 * 2.5 Gbit/sec).
+IBV_RATE_5_GBPS</b>, the value 2 will be returned (5 Gbit/sec = 2 * 2.5 Gbit/sec).\r
<p><b>mult_to_ibv_rate()</b> converts the multiplier value (of 2.5 Gbit/sec) <i>\r
mult</i> to an IB transmission rate enumeration. For example, if <i>mult</i> is \r
2, the rate enumeration <b>IBV_RATE_5_GBPS</b> will be returned.</p>\r
<h4>RETURN VALUE</h4>\r
-<b>ibv_rate_to_mult()</b> returns the multiplier of the base rate 2.5 Gbit/sec.
+<b>ibv_rate_to_mult()</b> returns the multiplier of the base rate 2.5 Gbit/sec.\r
<p><b>mult_to_ibv_rate()</b> returns the enumeration representing the IB \r
transmission rate.</p>\r
<h4>SEE ALSO</h4>\r
-<b><a href="#IBV_QUERY_PORT">ibv_query_port</a></b>
+<b><a href="#IBV_QUERY_PORT">ibv_query_port</a></b>\r
</span>\r
<span style="font-size: 12pt; font-family: Times New Roman">\r
<p align="left"><a href="#TOP"><font color="#000000"><<b>return-to-top</b>></font></a></p>\r
<a name="RDMA_RESOLVE_ADDR">RDMA_RESOLVE_ADDR</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-rdma_resolve_addr - Resolve destination and optional source addresses.
+rdma_resolve_addr - Resolve destination and optional source addresses.\r
<h4>SYNOPSIS</h4>\r
<b>#include <rdma/rdma_cma.h></b><p> <b>int rdma_resolve_addr</b> <b>(struct \r
rdma_cm_id *</b><i>id</i><b>,</b> <b>struct sockaddr *</b><i>src_addr</i><b>,</b>\r
<h4>ARGUMENTS</h4>\r
<dl COMPACT>\r
<dt>id</dt>\r
- <dd>RDMA identifier.
+ <dd>RDMA identifier.\r
</dd>\r
<dt>src_addr</dt>\r
- <dd>Source address information. This parameter may be NULL.
+ <dd>Source address information. This parameter may be NULL.\r
</dd>\r
<dt>dst_addr</dt>\r
- <dd>Destination address information.
+ <dd>Destination address information.\r
</dd>\r
<dt>timeout_ms</dt>\r
- <dd>Time to wait for resolution to complete.
+ <dd>Time to wait for resolution to complete.\r
</dd>\r
</dl>\r
<h4>DESCRIPTION</h4>\r
Resolve destination and optional source addresses from IP addresses to an RDMA \r
address. If successful, the specified rdma_cm_id will be bound to a local \r
-device. <a NAME="lbAF"> </a>
+device. <a NAME="lbAF"> </a>\r
<h4>NOTES</h4>\r
This call is used to map a given destination IP address to a usable RDMA \r
address. The IP to RDMA address mapping is done using the local routing tables, \r
rdma_cm_id will be bound to a source address based on the local routing tables. \r
After this call, the rdma_cm_id will be bound to an RDMA device. This call is \r
typically made from the active side of a connection before calling \r
-rdma_resolve_route and rdma_connect. <a NAME="lbAG"> </a>
+rdma_resolve_route and rdma_connect. <a NAME="lbAG"> </a>\r
<h4>INFINIBAND SPECIFIC</h4>\r
This call maps the destination and, if given, source IP addresses to GIDs. In \r
order to perform the mapping, IPoIB must be running on both the local and remote \r
-nodes. <a NAME="lbAH"> </a>
+nodes. <a NAME="lbAH"> </a>\r
<h4>SEE ALSO</h4>\r
<a href="#RDMA_CREATE_ID">rdma_create_id</a>, <a href="#RDMA_RESOLVE_ROUTE">rdma_resolve_route</a>,\r
<a href="#RDMA_CONNECT">rdma_connect</a>, <a href="#RDMA_CREATE_QP">rdma_create_qp</a>,\r
<a href="#RDMA_GET_CM_EVENT">rdma_get_cm_event</a>, <a href="#RDMA_BIND_ADDR">rdma_bind_addr</a>,\r
<a href="#RDMA_GET_SRC_PORT">rdma_get_src_port</a>, <a href="#RDMA_GET_DST_PORT">rdma_get_dst_port</a>,\r
<a href="#RDMA_GET_LOCAL_ADDR">rdma_get_local_addr</a>,\r
-<a href="#RDMA_GET_PEER_ADDR">rdma_get_peer_addr</a>
+<a href="#RDMA_GET_PEER_ADDR">rdma_get_peer_addr</a>\r
<p> </p>\r
<h3><br>\r
<a name="RDMA_GET_CM_EVENT">RDMA_GET_CM_EVENT</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-rdma_get_cm_event - Retrieves the next pending communication event.
+rdma_get_cm_event - Retrieves the next pending communication event.\r
<h4>SYNOPSIS</h4>\r
<b>#include <rdma/rdma_cma.h></b><p> <b>int rdma_get_cm_event</b> <b>(struct \r
rdma_event_channel *</b><i>channel</i><b>,</b> <b>struct rdma_cm_event **</b><i>event</i><b>);</b>\r
<h4>ARGUMENTS</h4>\r
<dl COMPACT>\r
<dt>channel</dt>\r
- <dd>Event channel to check for events.
+ <dd>Event channel to check for events.\r
</dd>\r
<dt>event</dt>\r
- <dd>Allocated information about the next communication event.
+ <dd>Allocated information about the next communication event.\r
</dd>\r
</dl>\r
<h4>DESCRIPTION</h4>\r
Retrieves a communication event. If no events are pending, by default, the call \r
-will block until an event is received.
+will block until an event is received.\r
<h4>NOTES</h4>\r
The default synchronous behavior of this routine can be changed by modifying the \r
file descriptor associated with the given channel. All events that are reported \r
will block until related events have been acknowledged.<h4>EVENT DATA</h4>\r
Communication event details are returned in the rdma_cm_event structure. This \r
structure is allocated by the rdma_cm and released by the rdma_ack_cm_event \r
-routine. Details of the rdma_cm_event structure are given below.
+routine. Details of the rdma_cm_event structure are given below.\r
<dl COMPACT>\r
<dt>id</dt>\r
<dd>The rdma_cm identifier associated with the event. If the event type is \r
RDMA_CM_EVENT_CONNECT_REQUEST, then this references a new id for that \r
- communication.
+ communication.\r
</dd>\r
<dt>listen_id</dt>\r
<dd>For RDMA_CM_EVENT_CONNECT_REQUEST event types, this references the \r
- corresponding listening request identifier.
+ corresponding listening request identifier.\r
</dd>\r
<dt>event</dt>\r
<dd>Specifies the type of communication event which occurred. See EVENT \r
- TYPES below.
+ TYPES below.\r
</dd>\r
<dt>status</dt>\r
<dd>Returns any asynchronous error information associated with an event. The \r
- status is zero unless the corresponding operation failed.
+ status is zero unless the corresponding operation failed.\r
</dd>\r
<dt>param</dt>\r
<dd>Provides additional details based on the type of event. Users should \r
select the conn or ud subfields based on the rdma_port_space of the \r
rdma_cm_id associated with the event. See UD EVENT DATA and CONN EVENT DATA \r
- below.
+ below.\r
</dd>\r
</dl>\r
<h4>UD EVENT DATA</h4>\r
Event parameters related to unreliable datagram (UD) services: RDMA_PS_UDP and \r
RDMA_PS_IPOIB. The UD event data is valid for RDMA_CM_EVENT_ESTABLISHED and \r
-RDMA_CM_EVENT_MULTICAST_JOIN events, unless stated otherwise.
+RDMA_CM_EVENT_MULTICAST_JOIN events, unless stated otherwise.\r
<dl COMPACT>\r
<dt>private_data</dt>\r
<dd>References any user-specified data associated with \r
referenced by this field matches that specified by the remote side when \r
calling rdma_connect or rdma_accept. This field is NULL if the event does \r
not include private data. The buffer referenced by this pointer is \r
- deallocated when calling rdma_ack_cm_event.
+ deallocated when calling rdma_ack_cm_event.\r
</dd>\r
<dt>private_data_len</dt>\r
<dd>The size of the private data buffer. Users should note that the size of \r
the private data buffer may be larger than the amount of private data sent \r
- by the remote side. Any additional space in the buffer will be zeroed out.
+ by the remote side. Any additional space in the buffer will be zeroed out.\r
</dd>\r
<dt>ah_attr</dt>\r
<dd>Address information needed to send data to the remote endpoint(s). Users \r
- should use this structure when allocating their address handle.
+ should use this structure when allocating their address handle.\r
</dd>\r
<dt>qp_num</dt>\r
- <dd>QP number of the remote endpoint or multicast group.
+ <dd>QP number of the remote endpoint or multicast group.\r
</dd>\r
<dt>qkey</dt>\r
<dd>QKey needed to send data to the remote endpoint(s).<br>\r
<h4>CONN EVENT DATA</h4>\r
Event parameters related to connected QP services: RDMA_PS_TCP. The connection \r
related event data is valid for RDMA_CM_EVENT_CONNECT_REQUEST and \r
-RDMA_CM_EVENT_ESTABLISHED events, unless stated otherwise.
+RDMA_CM_EVENT_ESTABLISHED events, unless stated otherwise.\r
<dl COMPACT>\r
<dt>private_data</dt>\r
<dd>References any user-specified data associated with the event. The data \r
referenced by this field matches that specified by the remote side when \r
calling rdma_connect or rdma_accept. This field is NULL if the event does \r
not include private data. The buffer referenced by this pointer is \r
- deallocated when calling rdma_ack_cm_event.
+ deallocated when calling rdma_ack_cm_event.\r
</dd>\r
<dt>private_data_len</dt>\r
<dd>The size of the private data buffer. Users should note that the size of \r
the private data buffer may be larger than the amount of private data sent \r
- by the remote side. Any additional space in the buffer will be zeroed out.
+ by the remote side. Any additional space in the buffer will be zeroed out.\r
</dd>\r
<dt>responder_resources</dt>\r
<dd>The number of responder resources requested of the recipient. This field \r
matches the initiator depth specified by the remote node when calling \r
- rdma_connect and rdma_accept.
+ rdma_connect and rdma_accept.\r
</dd>\r
<dt>initiator_depth</dt>\r
<dd>The maximum number of outstanding RDMA read/atomic operations that the \r
recipient may have outstanding. This field matches the responder resources \r
- specified by the remote node when calling rdma_connect and rdma_accept.
+ specified by the remote node when calling rdma_connect and rdma_accept.\r
</dd>\r
<dt>flow_control</dt>\r
- <dd>Indicates if hardware level flow control is provided by the sender.
+ <dd>Indicates if hardware level flow control is provided by the sender.\r
</dd>\r
<dt>retry_count</dt>\r
<dd>For RDMA_CM_EVENT_CONNECT_REQUEST events only, indicates the number of \r
- times that the recipient should retry send operations.
+ times that the recipient should retry send operations.\r
</dd>\r
<dt>rnr_retry_count</dt>\r
<dd>The number of times that the recipient should retry receiver not ready \r
- (RNR) NACK errors.
+ (RNR) NACK errors.\r
</dd>\r
<dt>srq</dt>\r
- <dd>Specifies if the sender is using a shared-receive queue.
+ <dd>Specifies if the sender is using a shared-receive queue.\r
</dd>\r
<dt>qp_num</dt>\r
- <dd>Indicates the remote QP number for the connection.
+ <dd>Indicates the remote QP number for the connection.\r
</dd>\r
</dl>\r
<h4>EVENT TYPES</h4>\r
-The following types of communication events may be reported.
+The following types of communication events may be reported.\r
<dl COMPACT>\r
<dt>RDMA_CM_EVENT_ADDR_RESOLVED</dt>\r
- <dd>Address resolution (rdma_resolve_addr) completed successfully.
+ <dd>Address resolution (rdma_resolve_addr) completed successfully.\r
</dd>\r
<dt>RDMA_CM_EVENT_ADDR_ERROR</dt>\r
- <dd>Address resolution (rdma_resolve_addr) failed.
+ <dd>Address resolution (rdma_resolve_addr) failed.\r
</dd>\r
<dt>RDMA_CM_EVENT_ROUTE_RESOLVED</dt>\r
- <dd>Route resolution (rdma_resolve_route) completed successfully.
+ <dd>Route resolution (rdma_resolve_route) completed successfully.\r
</dd>\r
<dt>RDMA_CM_EVENT_ROUTE_ERROR</dt>\r
- <dd>Route resolution (rdma_resolve_route) failed.
+ <dd>Route resolution (rdma_resolve_route) failed.\r
</dd>\r
<dt>RDMA_CM_EVENT_CONNECT_REQUEST</dt>\r
<dd>Generated on the passive side to notify the user of a new connection \r
- request.
+ request.\r
</dd>\r
<dt>RDMA_CM_EVENT_CONNECT_RESPONSE</dt>\r
<dd>Generated on the active side to notify the user of a successful response \r
to a connection request. It is only generated on rdma_cm_id's that do not \r
- have a QP associated with them.
+ have a QP associated with them.\r
</dd>\r
<dt>RDMA_CM_EVENT_CONNECT_ERROR</dt>\r
<dd>Indicates that an error has occurred trying to establish or a \r
- connection. May be generated on the active or passive side of a connection.
+ connection. May be generated on the active or passive side of a connection.\r
</dd>\r
<dt>RDMA_CM_EVENT_UNREACHABLE</dt>\r
<dd>Generated on the active side to notify the user that the remote server \r
- is not reachable or unable to respond to a connection request.
+ is not reachable or unable to respond to a connection request.\r
</dd>\r
<dt>RDMA_CM_EVENT_REJECTED</dt>\r
<dd>Indicates that a connection request or response was rejected by the \r
- remote end point.
+ remote end point.\r
</dd>\r
<dt>RDMA_CM_EVENT_ESTABLISHED</dt>\r
<dd>Indicates that a connection has been established with the remote end \r
- point.
+ point.\r
</dd>\r
<dt>RDMA_CM_EVENT_DISCONNECTED</dt>\r
- <dd>The connection has been disconnected.
+ <dd>The connection has been disconnected.\r
</dd>\r
<dt>RDMA_CM_EVENT_DEVICE_REMOVAL</dt>\r
<dd>The local RDMA device associated with the rdma_cm_id has been removed. \r
- Upon receiving this event, the user must destroy the related rdma_cm_id.
+ Upon receiving this event, the user must destroy the related rdma_cm_id.\r
</dd>\r
<dt>RDMA_CM_EVENT_MULTICAST_JOIN</dt>\r
<dd>The multicast join operation (rdma_join_multicast) completed \r
- successfully.
+ successfully.\r
</dd>\r
<dt>RDMA_CM_EVENT_MULTICAST_ERROR</dt>\r
<dd>An error either occurred joining a multicast group, or, if the group had \r
already been joined, on an existing group. The specified multicast group is \r
- no longer accessible and should be rejoined, if desired.
+ no longer accessible and should be rejoined, if desired.\r
</dd>\r
<dt>RDMA_CM_EVENT_ADDR_CHANGE</dt>\r
<dd>The network device associated with this ID through address resolution \r
changed its HW address, eg following of bonding failover. This event can \r
serve as a hint for applications who want the links used for their RDMA \r
- sessions to align with the network stack.
+ sessions to align with the network stack.\r
</dd>\r
<dt>RDMA_CM_EVENT_TIMEWAIT_EXIT</dt>\r
<dd>The QP associated with a connection has exited its timewait state and is \r
now ready to be re-used. After a QP has been disconnected, it is maintained \r
in a timewait state to allow any in flight packets to exit the network. \r
- After the timewait state has completed, the rdma_cm will report this event.
+ After the timewait state has completed, the rdma_cm will report this event.\r
</dd>\r
</dl>\r
<h4>SEE ALSO</h4>\r
<span style="font-size: 12pt; ">\r
<dl COMPACT>\r
<dt>event</dt>\r
- <dd>Event to be released.
+ <dd>Event to be released.\r
</dd>\r
</dl>\r
</span>\r
call frees the event structure and any memory that it references.</span><span style="font-size: 12pt; font-family: Times New Roman"><h4>SEE ALSO</h4>\r
</span>\r
<span style="font-size: 12pt; ">\r
-<a href="#RDMA_GET_CM_EVENT">rdma_get_cm_event</a>, <a href="#RDMA_DESTROY_ID">rdma_destroy_id</a>
+<a href="#RDMA_GET_CM_EVENT">rdma_get_cm_event</a>, <a href="#RDMA_DESTROY_ID">rdma_destroy_id</a>\r
</span>\r
<span style="font-size: 12pt; font-family: Times New Roman">\r
<p> </p>\r
<a href="#RDMA_CONNECT">rdma_connect</a>, <a href="#RDMA_ACCEPT">rdma_accept</a>,\r
<a href="#RDMA_GET_CM_EVENT">rdma_get_cm_event</a>, <a href="#RDMA_GET_SRC_PORT">\r
rdma_get_src_port</a>, <a href="#RDMA_GET_LOCAL_ADDR">rdma_get_local_addr</a>,\r
-<a href="#RDMA_GET_PEER_ADDR">rdma_get_peer_addr</a>
+<a href="#RDMA_GET_PEER_ADDR">rdma_get_peer_addr</a>\r
<p> </p>\r
<h3><br>\r
<a name="RDMA_GET_LOCAL_ADDR">RDMA_GET_LOCAL_ADDR</a></h3>\r
<a name="RDMA_LEAVE_MULTICAST">RDMA_LEAVE_MULTICAST</a></h3>\r
<hr>\r
<h4>NAME</h4>\r
-rdma_leave_multicast - Leaves a multicast group.
+rdma_leave_multicast - Leaves a multicast group.\r
<h4>SYNOPSIS</h4>\r
<b>#include <rdma/rdma_cma.h></b><p> <b>int rdma_leave_multicast</b> <b>(struct \r
rdma_cm_id *</b><i>id</i><b>,</b> <b>struct sockaddr *</b><i>addr</i><b>);</b></p>\r
<h4>ARGUMENTS</h4>\r
<dl COMPACT>\r
<dt>id</dt>\r
- <dd>Communication identifier associated with the request.
+ <dd>Communication identifier associated with the request.\r
</dd>\r
<dt>addr</dt>\r
<dd>Multicast address identifying the group to leave.</dd>\r
after leaving a multicast group. Destroying an rdma_cm_id will automatically \r
leave all multicast groups.<h4>SEE ALSO</h4>\r
<a href="#RDMA_JOIN_MULTICAST">rdma_join_multicast</a>,\r
-<a href="#RDMA_DESTROY_QP">rdma_destroy_qp</a>
+<a href="#RDMA_DESTROY_QP">rdma_destroy_qp</a>\r
<p> </p>\r
<h3><br>\r
<a name="RDMA_SET_OPTION">RDMA_SET_OPTION</a></h3>\r
projects / ~shefty / rdma-win.git / commitdiff