-Assistant for InfiniBand Communication Management (IB ACM)\r
-\r
-Note: The IB ACM should be considered experimental.\r
-\r
-\r
-Overview\r
---------\r
-The IB ACM package implements and provides a framework for experimental name,\r
-address, and route resolution services over InfiniBand. It is intended to\r
-address connection setup scalability issues running MPI applications on\r
-large clusters. The IB ACM provides information needed to establish a\r
-connection, but does not implement the CM protocol.\r
-\r
-The librdmacm can invoke IB ACM services when built using the --with-ib_acm\r
-option. The IB ACM services tie in under the rdma_resolve_addr,\r
-rdma_resolve_route, and rdma_getaddrinfo routines. For maximum benefit,\r
-the rdma_getaddrinfo routine should be used, however existing applications\r
-should still see significant connection scaling benefits using the calls\r
-available in librdmacm 1.0.11 and previous releases.\r
-\r
-The IB ACM is focused on being scalable and efficient. The current\r
-implementation limits network traffic, SA interactions, and centralized\r
-services. ACM supports multiple resolution protocols in order to handle\r
-different fabric topologies.\r
-\r
-This release is limited in its handling of dynamic changes.\r
-\r
-The IB ACM package is comprised of two components: the ib_acm service\r
-and a test/configuration utility - ib_acme. Both are userspace components\r
-and are available for Linux and Windows. Additional details are given below.\r
-\r
-\r
-Quick Start Guide\r
------------------\r
-1. Prerequisites: libibverbs and libibumad must be installed.\r
- The IB stack should be running with IPoIB configured.\r
- These steps assume that the user has administrative privileges.\r
-2. Install the IB ACM package\r
- This installs ib_acm, and ib_acme.\r
-3. Run ib_acme -A -O\r
- This will generate IB ACM address and options configuration files.\r
- (acm_addr.cfg and acm_opts.cfg)\r
-4. Run ib_acm and leave running.\r
- ib_acm will eventually be converted to a service/daemon, but for now\r
- is a userspace application. Because ib_acm uses the libibumad\r
- interfaces, it should be run with administrative privileges.\r
-5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v\r
- This will verify that the ib_acm service is running.\r
-5. Install librdmacm.\r
- The librdmacm will automatically use the ib_acm service.\r
- On failures, the librdmacm will fall back to normal resolution.\r
-\r
-\r
-Details\r
--------\r
-ib_acme:\r
-The ib_acme program serves a dual role. It acts as a utility to test\r
-ib_acm operation and help verify if the ib_acm service and selected\r
-protocol is usable for a given cluster configuration. Additionally,\r
-it automatically generates ib_acm configuration files to assist with\r
-or eliminate manual setup.\r
-\r
-\r
-acm configuration files:\r
-The ib_acm service relies on two configuration files.\r
-\r
-The acm_addr.cfg file contains name and address mappings for each IB\r
-<device, port, pkey> endpoint. Although the names in the acm_addr.cfg\r
-file can be anything, ib_acme maps the host name and IP addresses to\r
-the IB endpoints.\r
-\r
-The acm_opts.cfg file provides a set of configurable options for the\r
-ib_acm service, such as timeout, number of retries, logging level, etc.\r
-ib_acme generates the acm_opts.cfg file using static information. A\r
-future enhancement would adjust options based on the current system\r
-and cluster size. \r
-\r
-\r
-ib_acm:\r
-The ib_acm service is responsible for resolving names and addresses to\r
-InfiniBand path information and caching such data. It is currently\r
-implemented as an executable application, but is a conceptual service\r
-or daemon that should execute with administrative privileges.\r
-\r
-The ib_acm implements a client interface over TCP sockets, which is\r
-abstracted by the librdmacm library. One or more back-end protocols are\r
-used by the ib_acm service to satisfy user requests. Although the\r
-ib_acm supports standard SA path record queries on the back-end, it\r
-provides an experimental multicast resolution protocol in hope of\r
-achieving greater scalability. The latter is not usable on all fabric\r
-topologies, specifically ones that may not have reversible paths.\r
-Users should use the ib_acme utility to verify that multicast protocol\r
-is usable before running other applications.\r
-\r
-Conceptually, the ib_acm service implements an ARP like protocol and either\r
-uses IB multicast records to construct path record data or queries the\r
-SA directly, depending on the selected route protocol. By default, the\r
-ib_acm services uses and caches SA path record queries.\r
-\r
-Specifically, all IB endpoints join a number of multicast groups.\r
-Multicast groups differ based on rates, mtu, sl, etc., and are prioritized.\r
-All participating endpoints must be able to communicate on the lowest\r
-priority multicast group. The ib_acm assigns one or more names/addresses\r
-to each IB endpoint using the acm_addr.cfg file. Clients provide source\r
-and destination names or addresses as input to the service, and receive\r
-as output path record data.\r
-\r
-The service maps a client's source name/address to a local IB endpoint.\r
-If a client does not provide a source address, then the ib_acm service\r
-will select one based on the destination and local routing tables. If the\r
-destination name/address is not cached locally, it sends a multicast\r
-request out on the lowest priority multicast group on the local endpoint.\r
-The request carries a list of multicast groups that the sender can use.\r
-The recipient of the request selects the highest priority multicast group\r
-that it can use as well and returns that information directly to the sender.\r
-The request data is cached by all endpoints that receive the multicast\r
-request message. The source endpoint also caches the response and uses\r
-the multicast group that was selected to construct or obtain path record\r
-data, which is returned to the client.\r
-\r
-The current implementation of the IB ACM has several additional restrictions:\r
-- The ib_acm is limited in its handling of dynamic changes;\r
- the ib_acm must be stopped and restarted if a cluster is reconfigured.\r
-- Cached data does not timed out and is only updated if a new resolution\r
- request is received from a different QPN than a cached request.\r
-- Support for IPv6 has not been verified.\r
-- The number of addresses that can be assigned to a single endpoint is\r
- limited to 4.\r
-- The number of multicast groups that an endpoint can support is limited to 2.\r
-\r
+Assistant for InfiniBand Communication Management (IB ACM)
+
+Note: The IB ACM should be considered experimental.
+
+
+Overview
+--------
+The IB ACM package implements and provides a framework for experimental name,
+address, and route resolution services over InfiniBand. It is intended to
+address connection setup scalability issues running MPI applications on
+large clusters. The IB ACM provides information needed to establish a
+connection, but does not implement the CM protocol.
+
+The librdmacm can invoke IB ACM services when built using the --with-ib_acm
+option. The IB ACM services tie in under the rdma_resolve_addr,
+rdma_resolve_route, and rdma_getaddrinfo routines. For maximum benefit,
+the rdma_getaddrinfo routine should be used, however existing applications
+should still see significant connection scaling benefits using the calls
+available in librdmacm 1.0.11 and previous releases.
+
+The IB ACM is focused on being scalable and efficient. The current
+implementation limits network traffic, SA interactions, and centralized
+services. ACM supports multiple resolution protocols in order to handle
+different fabric topologies.
+
+This release is limited in its handling of dynamic changes.
+
+The IB ACM package is comprised of two components: the ib_acm service
+and a test/configuration utility - ib_acme. Both are userspace components
+and are available for Linux and Windows. Additional details are given below.
+
+
+Quick Start Guide
+-----------------
+1. Prerequisites: libibverbs and libibumad must be installed.
+ The IB stack should be running with IPoIB configured.
+ These steps assume that the user has administrative privileges.
+2. Install the IB ACM package
+ This installs ib_acm, and ib_acme.
+3. Run ib_acme -A -O
+ This will generate IB ACM address and options configuration files.
+ (acm_addr.cfg and acm_opts.cfg)
+4. Run ib_acm and leave running.
+ ib_acm will eventually be converted to a service/daemon, but for now
+ is a userspace application. Because ib_acm uses the libibumad
+ interfaces, it should be run with administrative privileges.
+5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v
+ This will verify that the ib_acm service is running.
+5. Install librdmacm.
+ The librdmacm will automatically use the ib_acm service.
+ On failures, the librdmacm will fall back to normal resolution.
+
+
+Details
+-------
+ib_acme:
+The ib_acme program serves a dual role. It acts as a utility to test
+ib_acm operation and help verify if the ib_acm service and selected
+protocol is usable for a given cluster configuration. Additionally,
+it automatically generates ib_acm configuration files to assist with
+or eliminate manual setup.
+
+
+acm configuration files:
+The ib_acm service relies on two configuration files.
+
+The acm_addr.cfg file contains name and address mappings for each IB
+<device, port, pkey> endpoint. Although the names in the acm_addr.cfg
+file can be anything, ib_acme maps the host name and IP addresses to
+the IB endpoints.
+
+The acm_opts.cfg file provides a set of configurable options for the
+ib_acm service, such as timeout, number of retries, logging level, etc.
+ib_acme generates the acm_opts.cfg file using static information. A
+future enhancement would adjust options based on the current system
+and cluster size.
+
+
+ib_acm:
+The ib_acm service is responsible for resolving names and addresses to
+InfiniBand path information and caching such data. It is currently
+implemented as an executable application, but is a conceptual service
+or daemon that should execute with administrative privileges.
+
+The ib_acm implements a client interface over TCP sockets, which is
+abstracted by the librdmacm library. One or more back-end protocols are
+used by the ib_acm service to satisfy user requests. Although the
+ib_acm supports standard SA path record queries on the back-end, it
+provides an experimental multicast resolution protocol in hope of
+achieving greater scalability. The latter is not usable on all fabric
+topologies, specifically ones that may not have reversible paths.
+Users should use the ib_acme utility to verify that multicast protocol
+is usable before running other applications.
+
+Conceptually, the ib_acm service implements an ARP like protocol and either
+uses IB multicast records to construct path record data or queries the
+SA directly, depending on the selected route protocol. By default, the
+ib_acm services uses and caches SA path record queries.
+
+Specifically, all IB endpoints join a number of multicast groups.
+Multicast groups differ based on rates, mtu, sl, etc., and are prioritized.
+All participating endpoints must be able to communicate on the lowest
+priority multicast group. The ib_acm assigns one or more names/addresses
+to each IB endpoint using the acm_addr.cfg file. Clients provide source
+and destination names or addresses as input to the service, and receive
+as output path record data.
+
+The service maps a client's source name/address to a local IB endpoint.
+If a client does not provide a source address, then the ib_acm service
+will select one based on the destination and local routing tables. If the
+destination name/address is not cached locally, it sends a multicast
+request out on the lowest priority multicast group on the local endpoint.
+The request carries a list of multicast groups that the sender can use.
+The recipient of the request selects the highest priority multicast group
+that it can use as well and returns that information directly to the sender.
+The request data is cached by all endpoints that receive the multicast
+request message. The source endpoint also caches the response and uses
+the multicast group that was selected to construct or obtain path record
+data, which is returned to the client.
+
+The current implementation of the IB ACM has several additional restrictions:
+- The ib_acm is limited in its handling of dynamic changes;
+ the ib_acm must be stopped and restarted if a cluster is reconfigured.
+- Cached data does not timed out and is only updated if a new resolution
+ request is received from a different QPN than a cached request.
+- Support for IPv6 has not been verified.
+- The number of addresses that can be assigned to a single endpoint is
+ limited to 4.
+- The number of multicast groups that an endpoint can support is limited to 2.
projects / ~shefty / ibacm.git / commitdiff