Hal Rosenstock [Thu, 1 Aug 2013 21:10:00 +0000 (14:10 -0700)]
acm: Save endpoint pkey in host order
acm_ep stores and uses a pkey value, assuming that it is in
host order. However, ibv_query_pkey returns the pkey in network
order. We need to byte swap the pkey when saving it.
Signed-off-by: Hal Rosenstock <hal@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Hal Rosenstock [Thu, 27 Jun 2013 18:48:11 +0000 (21:48 +0300)]
Add support for preloading IPv4 and IPv6 ACM caches
A hosts file (ibacm_hosts.cfg) is configured with tuples of
IP addresses (either IPv4 or IPv6) and IB GID. This information
in conjunction with the path record preloading allows
for the IPv4 and IPv6 ACM caches to be preloaded.
IPv4 was tested with both rping and ucmatose as well as
acme. IPv6 was tested with ucmatose and acme.
The remaining ACM cache to be preloaded is the name cache.
This will be added in subsequent patch.
Signed-off-by: Hal Rosenstock <hal@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Hal Rosenstock [Thu, 27 Jun 2013 13:48:24 +0000 (16:48 +0300)]
Add the ability to preload the destination GID and LID caches
Preloading of these caches is supported via a file which is
produced by OpenSM by the dump_pr plugin which contains
sufficient SA PathRecord information. Details on this
file format and configuring OpenSM for this are found in
dump_pr_notes.txt in dump_pr.
File format is specified in ibacm_opts.cfg as follows:
path_rec_fmt full_opensm_v1
File format defaults to none which means no preload of ACM cache.
Signed-off-by: Hal Rosenstock <hal@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Hal Rosenstock [Fri, 28 Jun 2013 01:08:27 +0000 (18:08 -0700)]
ib_acme: Enhance documention in generated ibacm_opt.cfg
Improve the description of the send_depth value in the generated
ibacm_opt.cfg file. Fixup a couple of other areas. Patch derived from
patches submitted by Hal.
Signed-off-by: Hal Rosenstock <hal@mellanox.com>. Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Fri, 28 Jun 2013 01:04:39 +0000 (18:04 -0700)]
Remove sample ibacm_*.cfg files
Configuration files can be automatically generated using ib_acme. Remove
the files from the source tree, versus maintaining them and ensuring that
they remain in sync with the files generated by ib_acme.
Sean Hefty [Tue, 9 Oct 2012 00:00:14 +0000 (17:00 -0700)]
ibacm: Timeout address and route data
Add an option that allows address and route data to
time out. Once data has timed out, it needs to be
resolved again. This helps handle cases where the
remote data has changed.
Sean Hefty [Mon, 4 Jun 2012 18:38:54 +0000 (11:38 -0700)]
ibacm: Automatically select local port if not specified by path record
If the user specifies a DLID or DGID as part of a path record lookup,
automatically select a local port. This allows a user to query an SA
without needing to specify the local SLID or SGID.
Sean Hefty [Fri, 30 Mar 2012 23:31:08 +0000 (16:31 -0700)]
ibacm: Fixes to ACM package to support distros
Set of changes to fixup the ibacm package for inclusion into RedHat 6.
Changes are based on feedback from Doug Ledford <dledford@redhat.com>.
These are primarily changes to the build files, along with name changes
to the man pages and sample configuration files.
Rename the ib_acm service to match the package name, ibacm.
Rename the ibacm configuration files to use the prefix 'ibacm' instead
of 'acm'. The new sample files are 'ibacm_addr.cfg' and 'ibacm_opts.cfg'.
Move location of ACM lock and configuration files and ibacm.pid
files. They are currently in non-standard locations.
Modify ibacm and ib_acme to use $sysconfdir, $bindir, and rdmadir
configure values. The ibacm_addr.cfg and ibacm_opt.cfg files will now be
read/written to $sysconfdir/$rdmadir by default, with rdmadir defaulting
to 'rdma' if not specified.. And ibacm will execute
$bindir/ib_acme if it needs to create the ibacm_addr.cfg file. Without
$bindir, the ibacm service can fail to launch ib_acme when started
from an init script.
Add init script as part of install. The init script is installed
into $sysconfdir/init.d. The init script is processed by configure,
so that it executes the correct ibacm service that was installed.
Fixup man pages based on changes.
Signed-off-by: Doug Ledford <dledford@redhat.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Sean Hefty [Fri, 6 Apr 2012 15:12:02 +0000 (08:12 -0700)]
ibacm: Set default route resolution to SA
SA resolution works in more environments than the ACM protocol.
SA is the default protocol for the ibacm_opts.cfg file, but if
that file is not available, the ibacm service defaults to ACM.
Hal Rosenstock [Thu, 5 Apr 2012 23:29:52 +0000 (16:29 -0700)]
ib_acme: Better error handling in resolve_gid
A return of 0 is an error for inet_pton but if 0 is returned
from resolve_gid, show_path is mistakenly called so if
0 is returned from inet_pton, return -1 instead (which is
what is done elsewhere in acme).
Signed-off-by: Hal Rosenstock <hal@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
Hal Rosenstock [Thu, 5 Apr 2012 23:24:09 +0000 (16:24 -0700)]
ib_acme: Use IPv4 rather than IPv6 when connecting to ACM server
When both IPv4 and IPv6 are running, getaddrinfo indicates IPv6
rather than IPv4 and that currently causes a failure to connect to
the ACM server. Fix is to set the address family in the hint to
getaddrinfo to IPv4.
Signed-off-by: Hal Rosenstock <hal@mellanox.com> Signed-off-by: Sean Hefty <sean.hefty@intel.com>
mc_dest has a default refcnt of 0. When a multicast destination is
released, an attempt is made to free it, which can result in the
ibacm service crashing.
Sean Hefty [Mon, 27 Jun 2011 17:29:36 +0000 (10:29 -0700)]
ibacm: Add performance counters
Add performance counters to track service usage. Counters are exposed
via a new perf query request. Update ib_acme to retrieve counters from
a specified endpoint.
Counters that are currently defined are:
- Address and route resolution errors
- Resolution requests
- Requests not satisfied as a result of no data available
- Requests requiring an address lookup
- Requests that found address information in the cache
- Requests requiring a route lookup (i.e. path record query)
- Requests that found route information in the cache
Sean Hefty [Fri, 1 Apr 2011 23:03:40 +0000 (16:03 -0700)]
ibacm: Allocate address handles dynamically when needed
ibacm allocates an address handle for every remote destination
that it tracks. However, under normal operation, the handle
is used infrequently - typically only once by the target
service to send a response and not at all on the initiator
service. Avoid the overhead of having 1 address handle per
destination by allocating them dynamically only when they are needed.
The exceptions to this are the address handles allocated to
communicate with the SA and the primary multicast group.
Sean Hefty [Fri, 1 Apr 2011 22:58:19 +0000 (15:58 -0700)]
ibacm: Join only the base multicast group if using SA path resolution
If the ACM service is configured to use the SA to resolve path
record data, then only join the base multicast group. We only
need to join other groups if we're creating path records based
on multicast data, using the acm route protocol.
Sean Hefty [Fri, 1 Apr 2011 21:46:34 +0000 (14:46 -0700)]
ibacm: Reduce overhead on multicast groups not used
ACM may join several multicast groups when using the multicast
group protocol. No data is ever sent on those groups. They exist
simply to see if a port *could* send data across those groups, for
example, to validate the mtu or rate between two endpoints. Since
we don't send data on them, there's no need to create an address
handle for them or attach the ACM UD QP to it.
Sean Hefty [Wed, 23 Feb 2011 22:41:48 +0000 (14:41 -0800)]
Handle port down events
Process port down or error events. While a port is down, the
ibacm service will fail all requests to endpoints on that port.
Cached data is not invalidated however, under the assumption
that the port state is temporary. (A future enhancement can
optimize the cleanup.)
Since a port can now transition between up and down, we need
to fixup a couple of places to avoid initializing structures
multiple times.
Sean Hefty [Tue, 22 Feb 2011 18:26:01 +0000 (10:26 -0800)]
Handle port up events
Handle port up events. The IB ACM service only checks once
during initialization for active ports to use. If a port later
becomes available, the service will not be able to use it.
Sean Hefty [Thu, 10 Feb 2011 20:31:44 +0000 (12:31 -0800)]
Add log message when incrementing SM destination reference count
The reference count for the SM destination structure is manually
incremented, but acm_put_dest is used to release it. This results
in an unmatched put message in the ibacm log file. Provide a matching
'get dest' log message when incrementing the SM destination.
Sean Hefty [Thu, 10 Feb 2011 20:06:46 +0000 (12:06 -0800)]
Set SM LID address in network byte order
acm_set_dest_addr expects LIDs to be in network order.
Convert the sm_lid retrieved from the port attributes into
network order. This formats the LID value correctly
when entered into the ibacm log.
Sean Hefty [Fri, 21 Jan 2011 01:06:28 +0000 (17:06 -0800)]
Move endpoint activation to another thread
To support dynamic changes, move endpoint activation from
the main thread into a dedicated thread that will also
process asynchronous events (to be added separately). The
main thread will still initialize the device and port
list, but activation code will be handled by the new thread,
so that endpoints can be deferred until their associated
ports are activated.
This also means that multicast joins are deferred, and
additional state checking is needed to prevent the main
thread (which processes client requests) from accessing
endpoints which are not yet fully active.
Sean Hefty [Thu, 20 Jan 2011 21:26:26 +0000 (13:26 -0800)]
Track inactive devices and ports
When obtaining a list of devices and ports, store all that are
available in the local device list, including those that are not
active. This will allow the ib_acm service to handle dynamic
changes, like port up/down events.
Sean Hefty [Thu, 20 Jan 2011 19:46:44 +0000 (11:46 -0800)]
Delay opening address file until assigning addresses
In order to handle ports that are down, we need to add
new endpoints only when they are brought online or assigned
new partitions (pkeys). This is easier to handle if we
delay the opening of the address configuration file until
it is needed to assign addresses.
Sean Hefty [Mon, 13 Dec 2010 18:35:40 +0000 (10:35 -0800)]
ibacm: add support for path lookup by LID/GID
Support path record lookup based on DLID or DGID. This allows
ibacm to be used purely for path record caching, with address
resolution being done by a user. This is needed to support
clients that exchange LID/GID information out of band before
connecting, or that make use of ipoib for address resolution.
Sean Hefty [Thu, 16 Dec 2010 01:23:26 +0000 (17:23 -0800)]
ibacm: fix lmc usage
Convert lmc to actual mask for use in slid comparisons. All SA
queries are initiated from the base lid, so set the src_path_bits
to 0. It is currently set incorrectly based on using the lmc as
a mask against the base lid.
Sean Hefty [Wed, 8 Dec 2010 19:34:36 +0000 (11:34 -0800)]
ib_acm: change default log file
Change the default logging file from stdout to
/var/log/ibacm.log, so we create a usable log file if
ib_acm is run without the user first having run ib_acme -O.
Sean Hefty [Tue, 7 Dec 2010 21:44:44 +0000 (13:44 -0800)]
ib_acme: add an 'unspecified' address format
Support an 'unspecified' address format. If no address
format is given, ib_acme will use getaddrinfo to try to
resolve the destination address.
This simplifies the user interface, plus allows the user
to specify a hostname for input, but allow getaddrinfo
to resolve the hostname into an IP address before contacting
the ACM service. The latter can provide a benefit of
being able to specify hostnames on a large cluster, but
allow ib_acme to populate ib_acm caches with IP address
data instead. This benefits MPI applications which may
exchange IP addresses out of band for communications, but
rely on hostnames for job distribution.
Sean Hefty [Tue, 7 Dec 2010 17:39:52 +0000 (09:39 -0800)]
ib_acme: Allow user to specify multiple destinations
In order to test the ACM service on large clusters and
force population of cached data, allow a user to indicate
that ib_acme should resolve information for multiple
destinations.
Multiple destinations can be specified by using bracket
syntax, such as [1-10,12,15-20], at the end of the
destination name.
Sean Hefty [Mon, 6 Dec 2010 21:07:41 +0000 (13:07 -0800)]
ibacm: support no delay options
Allow a user to specify that address and/or route resolution
protocols should be suppressed, and that only locally cached
data should be returned.
This helps support rdma_getaddrinfo options RAI_NUMERICHOST
and RAI_NOROUTE. If data for a request is available, it is
immediately returned. Otherwise, the client request is
failed, but the lookup is still initiated. This avoids
blocking a client for an extended period of time while
resolution completes, but allows future calls to find cached
data.
Sean Hefty [Mon, 6 Dec 2010 19:51:38 +0000 (11:51 -0800)]
ibacm: write port to /var/run/ibacm.port
Write used port data to /var/run/ibacm.port. This will allow
librdmacm and other libraries and applications to find the
ibacm service when it has been moved from its default port.
Sean Hefty [Mon, 6 Dec 2010 19:06:47 +0000 (11:06 -0800)]
ibacm: automatically generate addresses if missing acm_addr.cfg
If the acm_addr.cfg file is missing, automatically attempt to
generate the file. If no addresses can be assigned to any
endpoints because of a missing file, log an error and exit
the service. This avoids running a service that is basically
useless.
This change allows the ib_acm service to execute using defaults
without requireing that a user first run 'ib_acme -A -O' or
otherwise creating the acm configuration files.
Sean Hefty [Mon, 6 Dec 2010 16:42:43 +0000 (08:42 -0800)]
ibacm: modify logging output
Move some level 2 debug output to level 1, some level 0 to 1,
and mark output with 'notice' or 'ERROR' where needed. This
will help debug scaling problems on large clusters without needing to
parse through volumes of data. The original messages were guesses
anyway.
Sean Hefty [Fri, 3 Dec 2010 20:52:49 +0000 (12:52 -0800)]
ib_acme: Hide output for -A -O options unless verbose
Do not display the contents of generated config files by
default. Add a verbose option to ib_acme and only display
the contents if verbose is enabled. This reduces the amount
of noise using ib_acme to configuring address information
across a large cluster.
Sean Hefty [Thu, 2 Dec 2010 22:12:56 +0000 (14:12 -0800)]
ibacm: Add lock to prevent multiple daemons from running
Use a lock file to prevent multiple daemons from running
simultaneously.
Without this lock, a second instance of ib_acm eventually
fails to bind to the server's TCP port and exits, but not
before it overwrites a portion of the log file.
Sean Hefty [Wed, 1 Dec 2010 03:38:15 +0000 (19:38 -0800)]
ibacm: decrease default retries
With 15 retries, 2 seconds per timeout, the total timeout ends
up being too long when running on large clusters, where a node
can drop off at anytime. Decrease the default number of retries
to 2.
Sean Hefty [Tue, 16 Nov 2010 18:39:24 +0000 (10:39 -0800)]
ibacm: Introduce loopback resolution 'protocol'
OFED 1.5.2 introduced a kernel patch that disables looping
back multicast messages to the sending QP. This capability
was enabled by default. The current ACM address resolution
protocol depends on this feature of multicast. To handle
the case where multicast loopback has been disabled, add an
option to ACM to resolve loopback addresses using local data
only. This option is enabled by default.
When loopback resolution is set to 'local', ACM will automatically
insert a loopback destination into each endpoint. Requests
to resolve a loopback address will find this destination
and use it to respond to client requests.
Sean Hefty [Mon, 15 Nov 2010 20:08:46 +0000 (12:08 -0800)]
ibacm: fix issuing SA query after recording address
ACM has two ways that it can complete address resolution. The
first is to receive a response to an address resolution query.
The second is to receive a multicast message carrying an address
resolution request. In the second case, the address request
may be between two other nodes.
When this occurs, ACM will record the address information of
the source of the multicast message. However, it's possible
for ACM to be in the process of trying to resolve that address.
After it records the address, it must see if there's an
outstanding request against that address, so that it can kick
off route resolution.
This fixes an issue where ACM will hang resolving addresses
found during scale-up testing.
Sean Hefty [Wed, 17 Nov 2010 00:05:28 +0000 (16:05 -0800)]
ibacm: enhance debug messages
Prefix all log messages with time stamp information. This
provides useful debugging information regarding the timing of
various operations.
To help match up various operations with each other, improve
and simplify logging of endpoint and destination names. We
save printable names with all destinations. The
function to display address information is removed in favor
of using the normal logging function, a call to format address
data, and per thread logging data.
Finally, we add ability to handle LID/GID addresses in acm_log_addr.
This is required to display SA and multicast destination addresses
in the acm log file.
Sean Hefty [Wed, 25 Aug 2010 17:26:22 +0000 (10:26 -0700)]
ibacm: support distros with older versions of gcc
ibacm implements atomics using gcc intrinsics that were introduced
in gcc 4.1.2. If an older version of gcc is used to compile the
code, an error results. Check that the required atomic calls are
supported, and if not, provide our own implementation.
projects / ~shefty / ibacm.git / log