Open Fabrics Enterprise Distribution (OFED)
- Diagnostic Tools in OFED 1.5 Release Notes
-
- December 2009
+ Diagnostic Tools in OFED 1.6.4 Release Notes
+ May 2014
-Repo: git://git.openfabrics.org/~sashak/management/management.git
+
+Repo: git://git.openfabrics.org/~iraweiny/infiniband-diags.git
URL: http://www.openfabrics.org/downloads/management
General
-------
-Model of operation: All diag utilities use direct MAD access to perform their
-operations. Operations that require QP0 mads only may use direct routed
-mads, and therefore can work even in unconfigured subnets. Almost all
-utilities can operate without accessing the SM, unless GUID to lid translation
-is required. The only exception to this is saquery which requires the SM.
+
+infiniband-diags is a set of utilities designed to help configure, debug, and
+maintain infiniband fabrics. Many tools and utilities are provided. Some with
+similar functionality.
+
+Here is a list of the utilties included in infiniband-diags
+
+ Basic fabric conectivity
+ See: ibnetdiscover, iblinkinfo
+
+ Node information
+ See: ibnodes, ibswitches, ibhosts, ibrouters
+
+ Port information
+ See: ibportstate, ibaddr
+
+ Switch Forwarding Table info
+ See: ibtracert, ibroute, dump_lfts, dump_mfts, check_lft_balance, ibfindnodesusing
+
+ Peformance counters
+ See: ibqueryerrors, perfquery
+
+ Local HCA info
+ See: ibstat, ibstatus
+
+ Connectivity check
+ See: ibping, ibsysstat
+
+ Low level query tools
+ See: smpquery, smpdump, saquery, sminfo
+
+In addition to the utilities provided a sub-library libibnetdisc is provided to
+scan an entire IB fabric and return data structures representing it. The
+interface to this library is _not_ guaranteed to be stable (though we try.)
Dependencies
------------
-Most diag utilities depend on libibmad and libibumad.
-All diag utilities depend on the ib_umad kernel module.
-
-
-Multiple port/Multiple CA support
----------------------------------
-When no IB device or port is specified (see the "local umad parameters" below),
-the libibumad library selects the port to use by the following criteria:
-1. the first port that is ACTIVE.
-2. if not found, the first port that is UP (physical link up).
-
-If a port and/or CA name is specified, the libibumad library attempts to
-satisfy the user request, and will fail if it cannot do so.
-
-For example:
- ibaddr # use the 'best port'
- ibaddr -C mthca1 # pick the best port from mthca1 only.
- ibaddr -P 2 # use the second (active/up) port from the
- first available IB device.
- ibaddr -C mthca0 -P 2 # use the specified port only.
-
-
-Common options & flags
-----------------------
-Most diagnostics take the following flags. The exact list of supported
-flags per utility can be found in the usage message and can be displayed
-using util_name -h syntax.
-
-# Debugging flags
- -d raise the IB debugging level. May be used
- several times (-ddd or -d -d -d).
- -e show umad send receive errors (timeouts and others)
- -h display the usage message
- -v increase the application verbosity level.
- May be used several times (-vv or -v -v -v)
- -V display the internal version info.
-
-# Addressing flags
- -D use directed path address arguments. The path
- is a comma separated list of out ports.
- Examples:
- "0" # self port
- "0,1,2,1,4" # out via port 1, then 2, ...
- -G use GUID address arguments. In most cases, it is the Port GUID.
- Examples:
- "0x08f1040023"
- -s <smlid> use 'smlid' as the target lid for SA queries.
-
-# Local umad parameters:
- -C <ca_name> use the specified ca_name.
- -P <ca_port> use the specified ca_port.
- -t <timeout_ms> override the default timeout for the solicited mads.
-
-
-CLI notation
-------------
-All utilities use the POSIX style notation, meaning that all options (flags)
-must precede all arguments (parameters).
+ 1) libibmad >= 1.3.10
+ 2) libibumad >= 1.3.7
+ 3) opensm-libs >= 3.3.10
+ 4) ib_umad kernel module
+ 5) glib2
+
+
+Documentation
+-------------
+
+See man pages: start with "infiniband-diags"
+
+
+Release notes v1.6.2 => 1.6.4
+-----------------------------
+
+ 1) bug fixes
+
+
+Authors since 1.6.2
+-------------------
+
+Al Chu <chu11@llnl.gov>
+ infiniband-diags: Check node-name-map strings when querying by name
+
+Dan Ben Yosef <danby@mellanox.com>
+ libibnetdisc/ibnetdisc.c: fix insert of invalid lid 0xFFFF into lid_port hash_table
+ ibtracert.c: Add 'force' flag to man page
+ infiniband-diags/vendstat: Fix GeneralInfo SW version
+ ibportstate: on/off option is missing in help
+ iblinkinfo: Remove "-R" option from man page ibaddr: Add missing -t option to manpage
+
+Hal Rosenstock <hal@dev.mellanox.co.il>
+ ibdiag_sa.c: Output attribute ID in hex rather than decimal
-Utilities descriptions
-----------------------
-See man pages
+Ira Weiny <ira.weiny@intel.com>
+ infiniband-diags: fail configure if glib2 is not found
+Bart Van Assche <bvanassche@acm.org>
+ Fix errors triggered by autoregen.sh
+ get_lid_from_name(): Remove an unused variable
-Bugs Fixed
-----------
Open Fabrics Enterprise Distribution (OFED)
- IB ACM in OFED 1.5 Release Notes
+ IB ACM in OFED 3.12 Release Notes
- July 2010
+ May 2014
===============================================================================
Table of Contents
===============================================================================
1. Overview
-2. Quick Start Guide
-3. Operation Details
-4. Known Issues
+2. Operation Details
+3. Known Issues
===============================================================================
1. Overview
===============================================================================
-The IB ACM package implements and provides a framework for experimental name,
+This describes ibacm release 1.0.8.
+
+The IB ACM package implements and provides a framework for name,
address, and route resolution services over InfiniBand. It is intended to
address connection setup scalability issues running MPI applications on
large clusters. The IB ACM provides information needed to establish a
connection, but does not implement the CM protocol.
-The librdmacm can invoke IB ACM services when built using the --with-ib_acm
-option. The IB ACM services tie in under the rdma_resolve_addr,
-rdma_resolve_route, and rdma_getaddrinfo routines. For maximum benefit,
-the rdma_getaddrinfo routine should be used, however existing applications
-should still see significant connection scaling benefits using the calls
-available in librdmacm 1.0.11 and previous releases.
+When available, the IB ACM service is automatically used by the librdmacm
+library.
The IB ACM is focused on being scalable and efficient. The current
implementation limits network traffic, SA interactions, and centralized
services. ACM supports multiple resolution protocols in order to handle
different fabric topologies.
-The IB ACM package is comprised of two components: the ib_acm service
-and a test/configuration utility - ib_acme. Both are userspace components
-and are available for Linux and Windows. Additional details are given below.
-
-===============================================================================
-2. Quick Start Guide
-===============================================================================
-
-1. Prerequisites: libibverbs and libibumad must be installed.
- The IB stack should be running with IPoIB configured.
- These steps assume that the user has administrative privileges.
-2. Install the IB ACM package
- This installs ib_acm, and ib_acme.
-3. Run ib_acme -A -O
- This will generate IB ACM address and options configuration files.
- (acm_addr.cfg and acm_opts.cfg)
-4. Run ib_acm and leave running.
- ib_acm will eventually be converted to a service/daemon, but for now
- is a userspace application. Because ib_acm uses the libibumad
- interfaces, it should be run with administrative privileges.
-5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v
- This will verify that the ib_acm service is running.
-5. Install librdmacm using the build option --with-ib_acm.
- The librdmacm will automatically use the ib_acm service.
- On failures, the librdmacm will fall back to normal resolution.
+The IB ACM package is comprised of two components: the ibacm service
+and a test/configuration utility - ib_acme.
===============================================================================
-3. Operation Details
+2. Operation Details
===============================================================================
ib_acme:
The ib_acme program serves a dual role. It acts as a utility to test
-ib_acm operation and help verify if the ib_acm service and selected
+ibacm operation and help verify if the ibacm service and selected
protocol is usable for a given cluster configuration. Additionally,
-it automatically generates ib_acm configuration files to assist with
+it automatically generates ibacm configuration files to assist with
or eliminate manual setup.
acm configuration files:
-The ib_acm service relies on two configuration files.
+The ibacm service relies on two configuration files.
-The acm_addr.cfg file contains name and address mappings for each IB
-<device, port, pkey> endpoint. Although the names in the acm_addr.cfg
+The ibacm_addr.cfg file contains name and address mappings for each IB
+<device, port, pkey> endpoint. Although the names in the ibacm_addr.cfg
file can be anything, ib_acme maps the host name and IP addresses to
the IB endpoints.
-The acm_opts.cfg file provides a set of configurable options for the
+The ibacm_opts.cfg file provides a set of configurable options for the
ib_acm service, such as timeout, number of retries, logging level, etc.
-ib_acme generates the acm_opts.cfg file using static information. A
-future enhancement would adjust options based on the current system
-and cluster size.
+ib_acme generates the ibacm_opts.cfg file using static information. If
+an ibacm_opts.cfg files is not available, internal default values are
+used by the ibacm service.
-ib_acm:
+ibacm:
The ib_acm service is responsible for resolving names and addresses to
-InfiniBand path information and caching such data. It is currently
-implemented as an executable application, but is a conceptual service
-or daemon that should execute with administrative privileges.
+InfiniBand path information and caching such data.
-The ib_acm implements a client interface over TCP sockets, which is
+The ibacm implements a client interface over TCP sockets, which is
abstracted by the librdmacm library. One or more back-end protocols are
-used by the ib_acm service to satisfy user requests. Although the
-ib_acm supports standard SA path record queries on the back-end, it
-provides an experimental multicast resolution protocol in hope of
-achieving greater scalability. The latter is not usable on all fabric
-topologies, specifically ones that may not have reversible paths.
-Users should use the ib_acme utility to verify that multicast protocol
-is usable before running other applications.
-
-Conceptually, the ib_acm service implements an ARP like protocol and either
+used by the ibacm service to satisfy user requests.
+
+Conceptually, the ibacm service implements an ARP like protocol and either
uses IB multicast records to construct path record data or queries the
SA directly, depending on the selected route protocol. By default, the
-ib_acm services uses and caches SA path record queries.
+ibacm services uses and caches SA path record queries.
Specifically, all IB endpoints join a number of multicast groups.
Multicast groups differ based on rates, mtu, sl, etc., and are prioritized.
All participating endpoints must be able to communicate on the lowest
-priority multicast group. The ib_acm assigns one or more names/addresses
-to each IB endpoint using the acm_addr.cfg file. Clients provide source
+priority multicast group. The ibacm assigns one or more names/addresses
+to each IB endpoint using the ibacm_addr.cfg file. Clients provide source
and destination names or addresses as input to the service, and receive
as output path record data.
data, which is returned to the client.
===============================================================================
-4. Known Issues
+3. Known Issues
===============================================================================
The current implementation of the IB ACM has several restrictions:
-- The ib_acm is limited in its handling of dynamic changes;
- the ib_acm must be stopped and restarted if a cluster is reconfigured.
-- Cached data does not timed out and is only updated if a new resolution
- request is received from a different QPN than a cached request.
-- Support for IPv6 has not been verified.
+- The ibacm is limited in its handling of dynamic changes;
+ the ibacm must be stopped and restarted if a cluster is reconfigured.
+- The ibacm service reads address data from an ibacm_addr.cfg file. If not
+ found, this file will automatically be generated. Once created, ibacm
+ will use the address stored in this file for its mapping purposes.
+ When dynamic IP addresses is used, or IP addresses change, the result
+ is that ibacm will use stale data. The current work-around is to
+ delete the ibacm_addr.cfg file before starting the ibacm daemon. This
+ problem will be addressed in the 1.0.9 release, and is available in
+ upstream source tree.
- The number of addresses that can be assigned to a single endpoint is
limited to 4.
- The number of multicast groups that an endpoint can support is limited to 2.
Open Fabrics Enterprise Distribution (OFED)
- RDMA CM in OFED 1.5 Release Notes
+ RDMA CM in OFED 3.12 Release Notes
- July 2010
+ May 2014
===============================================================================
===============================================================================
The RDMA CM is a communication manager used to setup reliable, connected
and unreliable datagram data transfers. It provides an RDMA transport
-neutral interface for establishing connections. The API is based on sockets,
-but adapted for queue pair (QP) based semantics: communication must be
-over a specific RDMA device, and data transfers are message based.
-
-
-The RDMA CM only provides the communication management (connection setup /
-teardown) portion of an RDMA API. It works in conjunction with the verbs
-API for data transfers.
+neutral interface for establishing connections. The API concepts are
+based on sockets, but adapted for queue pair (QP) based semantics:
+communication must be over a specific RDMA device, and data transfers
+are message based.
+
+The RDMA CM can control both the QP and communication management (connection setup /
+teardown) portions of an RDMA API, or only the communication management
+piece. It works in conjunction with the verbs
+API defined by the libibverbs library. The libibverbs library provides the
+underlying interfaces needed to send and receive data.
+
+The RDMA CM can operate asynchronously or synchronously. The mode of
+operation is controlled by the user.
+
+The RDMA CM also provides the rsocket implementation.
+Rsockets is a protocol over RDMA that supports a socket-level API
+for applications. Rsocket APIs are intended to match the behavior
+of corresponding socket calls. A preload library is included as part of
+the RDMA CM package, which allows many socket based applications to run
+unmodified over rsockets.
===============================================================================
2. New Features
===============================================================================
-for OFED 1.5.2:
-
-Several enhancements were added to librdmacm release 1.0.12 that
-are intended to simplify using RDMA devices and address scalability issues.
-These changes were in response to long standing requests to make
-connection establishment 'more like sockets'. For full details,
-users should refer to the appropriate man pages. Major changes include:
-
-* Support synchronous operation for library calls. Users can control
- whether an rdma_cm_id operates asynchronously or synchronously based on
- the rdma_event_channel parameter. Use of synchronous operations
- reduces the amount of application code required to use the librdmacm
- by eliminating the need for event processing code.
-
- An rdma_cm_id will be marked for synchronous operation if the
- rdma_event_channel parameter is NULL for rdma_create_id or
- rdma_migrate_id. Users can toggle between synchronous and
- asynchronous operation through the rdma_migrate_id call.
-
- Calls that operate synchronously include rdma_resolve_addr,
- rdma_resolve_route, rdma_connect, rdma_accept, and rdma_get_request.
- Synchronous event data is returned to the user through the
- rdma_cm_id.
-
-* The addition of a new API: rdma_getaddrinfo. This call is modeled
- after getaddrinfo, but for RDMA devices and connections. It has the
- following notable deviations from getaddrinfo:
-
- A source address is returned as part of the call to allow the
- user to allocate necessary local HW resources for connections.
-
- Optional routing information may be returned to support
- Infiniband fabrics. IB routing information includes necessary
- path record data. rdma_getaddrinfo will obtain this information
- if IB ACM support (see below) is enabled. The use of IB ACM
- is not required for rdma_getaddrinfo.
-
- rdma_getaddrinfo provides future extensions to support
- more complex address and route resolution mechanisms, such as
- multiple path support and failover.
-
-* Support for a new APIs: rdma_get_request, rdma_create_ep, and
- rdma_destroy_ep. rdma_get_request simplifies the passive side
- implementation by adding synchronous support for accepting new
- connections. rdma_create_ep combines the functionality of
- rdma_create_id, rdma_create_qp, rdma_resolve_addr, and rdma_resolve_route
- in a single API that uses the output of rdma_getaddrinfo as its input.
-
-* Support for optional parameters. To simplify support for casual RDMA
- developers and researchers, the librdmacm can allocate protection
- domains, completion queues, and queue pairs on a user's behalf.
- This simplifies the amount of information that a developer
- must learn in order to use RDMA, plus allows the user to take
- advantage of higher-level completion processing abstractions.
-
- In addition to optional parameters, a user can also specify that the
- librdmacm should automatically select usable values for RDMA read
- operations.
-
-* Add support for IB ACM. IB ACM (InfiniBand Assistant for Communication
- Management) defines a socket based protocol to an IB address and route
- resolution service. One implementation of that service is provided
- separately by the ibacm package, but anyone can implement the service
- provided that they adhere to the IB ACM socket protocol. IB ACM is an
- experimental service targeted at increasing the scalability of applications
- running on a large cluster.
-
- Use of IB ACM is not required and is controlled through the build option
- '--with-ib_acm'. If the librdmacm fails to contact the IB ACM service, it
- reverts to using kernel services to resolve address and routing data.
-
-* Add RDMA helper routines. The librdmacm provide a set of simpler verbs
- calls for posting work requests, registering memory, and checking for
- completions. These calls are wrappers around libibverbs routines.
+for OFED 3.12
+
+Enhancements to the librdmacm release 1.0.18, versus the 1.0.17 release,
+mostly centered around improvements to the rsocket protocol and implementation.
+Specific enhancements to rsockets included:
+
+* Now supports iWarp devices.
+
+* Support native IB addressing with SOCK_STREAM.
+
+* Provide the ability for an application to manually set Infiniband path
+ record data. This allows applications using rsockets to take advantage
+ of ibacm, a package which tries to address IB subnet scalability issues.
+
+* Add keepalive support for SOCK_STREAM.
+
+In addition to the rsocket enhancements, several bug fixes were incorporated,
+along with a new test program, cmtime, to measure connection establishment times.
===============================================================================
3. Known Issues
The RDMA CM relies on the operating system's network configuration tables to
map IP addresses to RDMA devices. Incorrectly configured network
configurations can result in the RDMA CM being unable to locate the correct
-RDMA device. Currently, the RDMA CM only supports IPv4 addressing.
-
-All RDMA interfaces must provide a way to map IP addresses to an RDMA device.
-For Infiniband, this is done using IPoIB, and requires correctly configured
-IPoIB device interfaces sharing the same multicast domain. For details on
-configuring IPoIB, refer to ipoib_release_notes.txt. For RDMA devices to
-communicate, they must support the same underlying network and data link
-layers.
+RDMA device.
If you experience problems using the RDMA CM, you may want to check the
following:
* For multicast issues, either bind directly to a specific RDMA device, or
configure the IP routing tables to route multicast traffic over an RDMA
device's IP address.
-
projects / ~ardavis / ofed_docs / .git / commitdiff