\r
Overview\r
--------\r
-The IB ACM package implements and provides a framework for experimental name, address, and route resolution services over InfiniBand. It is intended to address connection setup scalability issues running MPI applications on large clusters. The IB ACM provides information needed to establish a connection, but does not implement the CM protocol. Long term, the IB ACM may support multiple resolution mechanisms.\r
+The IB ACM package implements and provides a framework for experimental name,\r
+address, and route resolution services over InfiniBand. It is intended to\r
+address connection setup scalability issues running MPI applications on\r
+large clusters. The IB ACM provides information needed to establish a\r
+connection, but does not implement the CM protocol.\r
\r
-The IB ACM is focused on being scalable and efficient. The current implementation limits network traffic, SA interactions, and centralized services. As a trade-off, it is not expected to support all cluster routing configurations. However, it is anticipated that additional functionality, such as path record caching, can be incorporated into the IB ACM to support a wider range of configurations.\r
+The librdmacm can invoke IB ACM services when built using the --with-ib_acm\r
+option. The IB ACM services tie in under the rdma_resolve_addr,\r
+rdma_resolve_route, and rdma_getaddrinfo routines. For maximum benefit,\r
+the rdma_getaddrinfo routine should be used, however existing applications\r
+should still see significant connection scaling benefits using the calls\r
+available in librdmacm 1.0.11 and previous releases.\r
\r
-The IB ACM package is comprised of three components: the ib_acm service, a libibacm library, and a test/configuration utility - ib_acme. All are userspace components and are available for Linux and Windows. Additional details are given below.\r
+The IB ACM is focused on being scalable and efficient. The current\r
+implementation limits network traffic, SA interactions, and centralized\r
+services. ACM supports multiple resolution protocols in order to handle\r
+different fabric topologies.\r
+\r
+This release 1.0.0 is limited in its handling of dynamic changes.\r
+\r
+The IB ACM package is comprised of two components: the ib_acm service\r
+and a test/configuration utility - ib_acme. Both are userspace components\r
+and are available for Linux and Windows. Additional details are given below.\r
\r
\r
Quick Start Guide\r
-----------------\r
1. Prerequisites: libibverbs and libibumad must be installed.\r
- The IB stack should be running with IPoIB configured\r
+ The IB stack should be running with IPoIB configured.\r
+ These steps assume that the user has administrative privileges.\r
2. Install the IB ACM package\r
- This installs libibacm, ib_acm, and ib_acme.\r
-3. Run ib_acme \96A \96O\r
+ This installs ib_acm, and ib_acme.\r
+3. Run ib_acme -A -O\r
This will generate IB ACM address and options configuration files.\r
(acm_addr.cfg and acm_opts.cfg)\r
-4. Run ib_acm and leave running\r
-5. Optionally, run ib_acme \96s <source_ip> -d <dest_ip> \96v\r
+4. Run ib_acm and leave running.\r
+ ib_acm will eventually be converted to a service/daemon, but for now\r
+ is a userspace application. Because ib_acm uses the libibumad\r
+ interfaces, it should be run with administrative privileges.\r
+5. Optionally, run ib_acme -s <source_ip> -d <dest_ip> -v\r
This will verify that the ib_acm service is running.\r
- It also verifies the path is usable on the given cluster.\r
-5. Install librdmacm.\r
-6. Define the following environment variable: RDMA_CM_USE_IB_ACM=1\r
+5. Install librdmacm using the build option --with-ib_acm.\r
The librdmacm will automatically use the ib_acm service.\r
On failures, the librdmacm will fall back to normal resolution.\r
\r
\r
Details\r
-------\r
-libibacm:\r
-The libibacm is an end-user library with simple interfaces for communicating with the ib_acm service. The libibacm implements the ib_acm client protocol. Although the interfaces to the libibacm are considered experimental, it\92s expected that existing calls will be supported going forward.\r
-\r
-For simplicity, all calls operate synchronously and are serialized. Possible future changes to the libibacm would be to process calls in parallel and add asynchronous interfaces.\r
-\r
-\r
ib_acme:\r
-The ib_acme program serves a dual role. It acts as a utility to test ib_acm operation and help verify if the ib_acm is usable for a given cluster configuration. Additionally, it automatically generates ib_acm configuration files to assist with or eliminate manual setup.\r
+The ib_acme program serves a dual role. It acts as a utility to test\r
+ib_acm operation and help verify if the ib_acm service and selected\r
+protocol is usable for a given cluster configuration. Additionally,\r
+it automatically generates ib_acm configuration files to assist with\r
+or eliminate manual setup.\r
\r
\r
acm configuration files:\r
-The ib_acm service relies on two configuration files. The acm_addr.cfg file contains name and address mappings for each IB <device, port, pkey> endpoint. Although the names in the acm_addr.cfg file can be anything, ib_acme maps the host name and IP addresses to the IB endpoints.\r
+The ib_acm service relies on two configuration files.\r
\r
-The acm_opts.cfg file provides a set of configurable options for the ib_acm service, such as timeout, number of retries, logging level, etc. ib_acme generates the acm_opts.cfg file using static information. A future enhancement would adjust options based on the current system and cluster size. \r
+The acm_addr.cfg file contains name and address mappings for each IB\r
+<device, port, pkey> endpoint. Although the names in the acm_addr.cfg\r
+file can be anything, ib_acme maps the host name and IP addresses to\r
+the IB endpoints.\r
\r
+The acm_opts.cfg file provides a set of configurable options for the\r
+ib_acm service, such as timeout, number of retries, logging level, etc.\r
+ib_acme generates the acm_opts.cfg file using static information. A\r
+future enhancement would adjust options based on the current system\r
+and cluster size. \r
\r
-ib_acm:\r
-The ib_acm service is responsible for resolving names and addresses to InfiniBand path information and caching such data. It is currently implemented as an executable application, but is a conceptual service or daemon that should execute with administrative privileges.\r
-\r
-The ib_acm implements a client interface over TCP sockets, which is abstracted by the libibacm library. One or more back-end protocols are used by the ib_acm service to satisfy user requests. Although the ib_acm supports standard SA path record queries on the back-end, it provides an experimental resolution protocol in hope of achieving greater scalability. \r
-\r
-Conceptually, the ib_acm service implements an ARP like protocol and uses IB multicast records to construct path record data. It makes the assumption that a unicast path between two endpoints is realizable if those endpoints can communicate over a multicast group with similar properties (rate, mtu, etc.)\r
\r
-Specifically, all IB endpoints join a number of multicast groups. Multicast groups differ based on rates, mtu, sl, etc., and are prioritized. All participating endpoints must be able to communicate on the lowest priority multicast group. The ib_acm assigns one or more names/addresses to each IB endpoint using the acm_addr.cfg file. Clients provide source and destination names or addresses as input to the service, and receive as output path record data.\r
-\r
-The service maps a client's source name/address to a local IB endpoint. If the destination name/address is not cached locally, it sends a multicast request out on the lowest priority multicast group on the local endpoint. The request carries a list of multicast groups that the sender can use. The recipient of the request selects the highest priority multicast group that it can use as well and returns that information directly to the sender. The request data is cached by all endpoints that receive the multicast request message. The source endpoint also caches the response and uses the multicast group that was selected to construct path record data, which is returned to the client.\r
+ib_acm:\r
+The ib_acm service is responsible for resolving names and addresses to\r
+InfiniBand path information and caching such data. It is currently\r
+implemented as an executable application, but is a conceptual service\r
+or daemon that should execute with administrative privileges.\r
+\r
+The ib_acm implements a client interface over TCP sockets, which is\r
+abstracted by the librdmacm library. One or more back-end protocols are\r
+used by the ib_acm service to satisfy user requests. Although the\r
+ib_acm supports standard SA path record queries on the back-end, it\r
+provides an experimental multicast resolution protocol in hope of\r
+achieving greater scalability. The latter is not usable on all fabric\r
+topologies, specifically ones that may not have reversible paths.\r
+Users should use the ib_acme utility to verify that multicast protocol\r
+is usable before running other applications.\r
+\r
+Conceptually, the ib_acm service implements an ARP like protocol and either\r
+uses IB multicast records to construct path record data or queries the\r
+SA directly, depending on the selected route protocol. By default, the\r
+ib_acm services uses and caches SA path record queries.\r
+\r
+Specifically, all IB endpoints join a number of multicast groups.\r
+Multicast groups differ based on rates, mtu, sl, etc., and are prioritized.\r
+All participating endpoints must be able to communicate on the lowest\r
+priority multicast group. The ib_acm assigns one or more names/addresses\r
+to each IB endpoint using the acm_addr.cfg file. Clients provide source\r
+and destination names or addresses as input to the service, and receive\r
+as output path record data.\r
+\r
+The service maps a client's source name/address to a local IB endpoint.\r
+If a client does not provide a source address, then the ib_acm service\r
+will select one based on the destination and local routing tables. If the\r
+destination name/address is not cached locally, it sends a multicast\r
+request out on the lowest priority multicast group on the local endpoint.\r
+The request carries a list of multicast groups that the sender can use.\r
+The recipient of the request selects the highest priority multicast group\r
+that it can use as well and returns that information directly to the sender.\r
+The request data is cached by all endpoints that receive the multicast\r
+request message. The source endpoint also caches the response and uses\r
+the multicast group that was selected to construct or obtain path record\r
+data, which is returned to the client.\r
+\r
+The current implementation of the IB ACM has several additional restrictions:\r
+- The ib_acm is limited in its handling of dynamic changes;\r
+ the ib_acm must be stopped and restarted if a cluster is reconfigured.\r
+- Cached data does not timed out and is only updated if a new resolution\r
+ request is received from a different QPN than a cached request.\r
+- Support for IPv6 has not been verified.\r
+- The number of addresses that can be assigned to a single endpoint is\r
+ limited to 4.\r
+- The number of multicast groups that an endpoint can support is limited to 2.\r
\r
-The current implementation of the IB ACM has several additional restrictions. The ib_acm is limited in its handling of dynamic changes; the ib_acm must be stopped and restarted if a cluster is reconfigured. Cached data does not timed out and is only updated if a new resolution request is received from a different QPN than a cached request. Support for IPv6 has not been verified. The number of addresses that can be assigned to a single endpoint is limited to 4, and the number of multicast groups that an endpoint can support is limited to 2.\r
projects / ~shefty / ibacm.git / commitdiff