- OpenSM Release Notes 2.0.5
- ============================
-
-Version: OpenFabrics Enterprise Distribution (OFED) 1.2
-Repo: https://openib.org/svn/gen2/branches/1.2/src/userspace/management/osm
-Version: 9535 (openib-2.0.5)
-Date: April 2007
-
-1 Overview
-----------
-This document describes the contents of the OpenSM OFED 1.2 release.
-OpenSM is an InfiniBand compliant Subnet Manager and Administration,
-and runs on top of OpenIB. The OpenSM version for this release
-is openib-2.0.5
-
-This document includes the following sections:
-1 This Overview section (describing new features and software
- dependencies)
-2 Known Issues And Limitations
-3 Unsupported IB compliance statements
-4 Major Bug Fixes
-5 Main Verification Flows
-6 Qualified software stacks and devices
-
-1.1 Major New Features
-
-* Partition manager:
- The partition manager provides a means to setup multiple partitions
- by providing a partition policy file. For details please read the
- doc/partition-config.txt or the opensm man page.
-
-* Basic QoS Manager:
- Provides a uniform configuration of the entire fabric with values defined
- in the OpenSM options file. The options support different settings for
- CAs, Switches, and Routers. Note that this is disabled by default and
- using -Q enables QoS fabric setup.
-
-* Loading pre-routes from a file:
- A new routing module enables loading pre-routes from a file.
- To use this option you should use the command line options:
- "-R file --U <your routing file>" or
- "--routing_engine file --ucast_file <your routing file>"
- For more information refer to the file doc/modular-routing.txt
- or the opensm man page.
-
-* SA MultiPathRecord support:
- The SA can now handle requests for multiple PathRecords in one query.
- This includes methods SA GetMulti/GetMultiResp and dual sided RMPP.
-
-* PPC64 is now QAed and supported
-
-* Support LMC > 0 for Switch Enhanced Port 0:
- Allows enhanced switch port 0 (ESP0) to have a non zero
- LMC. Use the configured subnet wide LMC for this. Modifications were
- necessary to the LID assignment and routing to support this.
- Also, added an option to the configuration to use LMC configured for
- subnet for enhanced switch port 0 or set it to 0 even if a non zero
- LMC is configured for the subnet. The default is currently the
- latter option. The new configuration option is: lmc_esp0
-
-1.2 Minor New Features:
-
-* IPoIB broadcast group configuration:
- It is now possible to control the IPoIB broadcast group parameters
- (MTU, rate, SL) through the partitions configuration file.
-
-* Limiting OpenSM log file size:
- By providing the command line option: "-L <size in MB>" or
- "--log_limit <size in MB>" the user can limit the generated log
- file size. When specified, the log file will be truncated upon reaching
- this limit.
-
-* Favor 1K MTU for Tavor (MT23108) HCA
- In cases where a PathRecord or MultiPathRecord is queried and the
- requestor does not specify the MTU or does specify it in a way
- that allows for MTU to be 1K and one of the path ends in a Tavor,
- limit the MTU to 1K max.
-
-* Man pages:
- Added opensm.8 and osmtest.8
-
-* Leaf VL stall count control:
- A new parameter (leaf_vl_stall_count) for controlling the number of
- sequential packets dropped on a switch port driving a HCA/TCA/Router
- that cause the port to enter the VLStalled state was added to the
- options file.
-
-* SM Polling/Handover defaults changed
- The default SMInfo polling retries was decreased from 18 to 4
- which reduces the default handover time from 3 min to 40 seconds.
-
-1.3 Library API Changes
-
-* cl_mem* APIs deprecated in complib:
- These functions are now considered as deprecated and should be
- replaced by direct calls to malloc, free, memset, etc.
-
-* osm_log_init_v2 API added in libopensm:
- Supports providing the new option for log file truncation.
-
-1.4 Software Dependencies
-
-OpenSM depends on the installation of either OFED 1.2, OFED 1.0,
-OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD
-distribution), or Mellanox VAPI stacks. The qualified driver versions
-are provided in Table 2, "Qualified IB Stacks".
-
-1.5 Supported Devices Firmware
-
-The main task of OpenSM is to initialize InfiniBand devices. The
-qualified devices and their corresponding firmware versions
-are listed in Table 3.
-
-2 Known Issues And Limitations
-------------------------------
-
-* No Service / Key associations:
- There is no way to manage Service access by Keys.
-
-* No SM to SM SMDB synchronization:
- Puts the burden of re-registering services, multicast groups, and
- inform-info on the client application (or IB access layer core).
-
-* No "port down" event handling:
- Changing the switch port through which OpenSM connects to the IB
- fabric may cause incorrect operation. Please restart OpenSM whenever
- such a connectivity change is made.
-
-* Changing connections during SM operation:
- Under some conditions the SM can get confused by a change in
- cabling (moving a cable from one switch port to the other) and
- momentarily see this as having the same GUID appear connected
- to two different IB ports. Under some conditions, when the SM fails to
- get the corresponding change event it might mistakenly report this case
- as a "duplicated GUID" case and abort. It is advisable to double-check
- the syslog after each such change in connectivity and restart
- OpenSM if it has exited.
-
-3 Unsupported IB Compliance Statements
---------------------------------------
-The following section lists all the IB compliance statements which
-OpenSM does not support. Please refer to the IB specification for detailed
-information regarding each compliance statement.
-
-* C14-22 (Authentication):
- M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one
- SubnSet method. As a work-around, an OpenSM option is provided for
- defining the protect bits.
-
-* C14-67 (Authentication):
- On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then
- the SM shall generate a SubnGetResp if the M_Key matches, or
- silently drop the packet if M_Key does not match.
-
-* C15-0.1.23.4 (Authentication):
- InformInfoRecords shall always be provided with the QPN set to 0,
- except for the case of a trusted request, in which case the actual
- subscriber QPN shall be returned.
-
-* o13-17.1.2 (Event-FWD):
- If no permission to forward, the subscription should be removed and
- no further forwarding should occur.
-
-* C14-24.1.1.5 and C14-62.1.1.22 (Initialization):
- GUIDInfo - SM should enable assigning Port GUIDInfo.
-
-* C14-44 (Initialization):
- If the SM discovers that it is missing an M_Key to update CA/RT/SW,
- it should notify the higher level.
-
-* C14-62.1.1.12 (Initialization):
- PortInfo:M_Key - Set the M_Key to a node based random value.
-
-* C14-62.1.1.13 (Initialization):
- PortInfo:P_KeyProtectBits - set according to an optional policy.
-
-* C14-62.1.1.24 (Initialization):
- SwitchInfo:DefaultPort - should be configured for random FDB.
-
-* C14-62.1.1.32 (Initialization):
- RandomForwardingTable should be configured.
-
-* o15-0.1.12 (Multicast):
- If the JoinState is SendOnlyNonMember = 1 (only), then the endport
- should join as sender only.
-
-* o15-0.1.8 (Multicast):
- If a request for creating an MCG with fields that cannot be met,
- return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass).
-
-* C15-0.1.8.6 (SA-Query):
- Respond to SubnAdmGetTraceTable - this is an optional attribute.
-
-* C15-0.1.13 Services:
- Reject ServiceRecord create, modify or delete if the given
- ServiceP_Key does not match the one included in the ServiceGID port
- and the port that sent the request.
-
-* C15-0.1.14 (Services):
- Provide means to associate service name and ServiceKeys.
-
-4 Major Bug Fixes
------------------
-
-The following is a list of bugs that were fixed. Note that other less critical
-or visible bugs were also fixed.
-
-* "Broken" fabric (duplicated port GUIDs) handling improved
- Replace assert with a real check to handle invalid physical port
- in osm_node_info_rcv.c which could occur on a broken fabric
-
-* SA client synchronous request failed but status returned was IB_SUCCESS
- even if there was no response.
- There was a missing setting of the status in the synchronous case.
-
-* Memory leak fixes:
- 1. In libvendor/osm_vendor_ibumad.c:osm_vendor_get_all_port_attr
- 2. In libvendor/osm_vendor_ibumad_sa.c:__osmv_sa_mad_rcv_cb
- 3. On receiving SMInfo SA request from a node that does not share a
- partition, the response mad was allocated but never free'd
- as it was never sent.
-
-* Set(InformInfo) OpenSM Deadlock:
- When receiving a request with unknown LID
-
-* PathRecord to inconsistent multicast destination:
- Fix the return error when multicast destination is not consistently
- indicated.
-
-* Remove double calculation of reversible path
- In osm_sa_path_record.c:__osm_pr_rcv_get_lid_pair_path a PathRecord
- query used to double check if the path is reversible
-
-* Some PathRecord log messages use "net order":
- Fix GUID net to host conversion in some osm_log messages
-
-* DR/LID routed SMPs direction bit handling:
- osm_resp.c:osm_resp_make_resp_smp, set direction bit only if direct
- routed class. This bug caused two issues:
- 1. Get/Set responses always had direction bit set.
- 2. Trap represses never had direction bit set.
- The direction bit needs setting in direct routed responses and it
- doesn't exist in LID routed responses.
- osm_sm_mad_ctrl.c: did not detect the "direction bit" correctly.
-
-* OpenSM crash due to transaction lookup (interop with Cisco stack)
- When a wire TID that maps to internal TID of zero (after applying
- mask) was received the lookup of the transaction was successful.
- The stale transaction pointed to "free'd" memory.
-
-* Better handling for Path/MultiPath requests for raw traffic
-
-* Wrong ProducerType provided in Notice Reports:
- When formating an SM generated report, the ProducerType was using
- CL_NTOH32 which can not be used to format a 24bit network order number.
-
-* OpenSM break on PPC64
- complib: Fixed memory corruption in cl_pool.c:cl_qcpool_init. This
- affected big endian 64-bit architectures only.
-
-* Illegal Set(InformInfo) was wrongly successful in updating the SMDB
- osm_sa_informinfo.c: In osm_infr_rcv_process_set_method, if sending
- error, don't call osm_infr_rcv_process_set_method
-
-* RMPP queries of InformInfoRecord fail
- ib_types.h: Pad ib_inform_info_record_t to be modulo 8 in size so
- that attribute offset is calculated properly
-
-* Returning "invalid request" rather than "unsupported method/attribute"
- In these cases, a noncompliant response was being provided.
-
-* Noncompliant response for SubnAdmGet(PortInfoRecord) with no match
- osm_pir_rcv_process, now returns "SA no records error" for SubnAdmGet
- with 0 records found
-
-* Noncompliant non base LID returned by some queries:
- The following attributes used to return the request LID rather than
- its base LID in responses: PKeyTableRecord, GUIDInfoRecord,
- SLtoVLMappingTableRecord, VLArbitrationTableRecord, LinkRecord
-
-* Noncompliant SubnAdmGet and SubnAdmGetTable:
- Mixing of error codes in case of no records or multiple records
- fixed for the attributes:
- LinearForwardingTableRecord, GUIDInfoRecord,
- VLArbitrationTableRecord, LinkRecord, PathRecord
-
-* segfault in InformInfo flows
- Under stress concurrent Set/Delete/Get flows. Fixed by adding
- missing lock.
-
-* SA queries containing LID out if range did not return ERR_REQ_INVALID
-
-5 Main Verification Flows
--------------------------
-
-OpenSM verification is run using the following activities:
-* osmtest - a stand-alone program
-* ibmgtsim (IB management simulator) based - a set of flows that
- simulate clusters, inject errors and verify OpenSM capability to
- respond and bring up the network correctly.
-* small cluster regression testing - where the SM is used on back to
- back or single switch configurations. The regression includes
- multiple OpenSM dedicated tests.
-* cluster testing - when we run OpenSM to setup a large cluster, perform
- hand-off, reboots and reconnects, verify routing correctness and SA
- responsiveness at the ULP level (IPoIB and SDP).
-
-5.1 osmtest
-
-osmtest is an automated verification tool used for OpenSM
-testing. Its verification flows are described by list below.
-
-* Inventory File: Obtain and verify all port info, node info, link and path
- records parameters.
-
-* Service Record:
- - Register new service
- - Register another service (with a lease period)
- - Register another service (with service p_key set to zero)
- - Get all services by name
- - Delete the first service
- - Delete the third service
- - Added bad flows of get/delete non valid service
- - Add / Get same service with different data
- - Add / Get / Delete by different component mask values (services
- by Name & Key / Name & Data / Name & Id / Id only )
-
-* Multicast Member Record:
- - Query of existing Groups (IPoIB)
- - BAD Join with insufficient comp mask (o15.0.1.3)
- - Create given MGID=0 (o15.0.1.4)
- - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)
- - Create BAD MGID=0xFA. (o15.0.1.6)
- - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)
- - New MGID with invalid join state (o15.0.1.9)
- - Retry of existing MGID - See JoinState update (o15.0.1.11)
- - BAD RATE when connecting to existing MGID (o15.0.1.13)
- - Partial JoinState delete request - removing FullMember (o15.0.1.14)
- - Full Delete of a group (o15.0.1.14)
- - Verify Delete by trying to Join deleted group (o15.0.1.14)
- - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)
-
-* GUIDInfo Record:
- - All GUIDInfoRecords in subnet are obtained
-
-* MultiPathRecord:
- - Perform some compliant and noncompliant MultiPathRecord requests
- - Validation is via status in responses and IB analyzer
-
-* PKeyTableRecord:
- - Perform some compliant and noncompliant PKeyTableRecord queries
- - Validation is via status in responses and IB analyzer
-
-* LinearForwardingTableRecord:
- - Perform some compliant and noncompliant LinearForwardingTableRecord queries
- - Validation is via status in responses and IB analyzer
-
-* Event Forwarding: Register for trap forwarding using reports
- - Send a trap and wait for report
- - Unregister non-existing
-
-* Trap 64/65 Flow: Register to Trap 64-65, create traps (by
- disconnecting/connecting ports) and wait for report, then unregister.
-
-* Stress Test: send PortInfoRecord queries, both single and RMPP and
- check for the rate of responses as well as their validity.
-
-
-5.2 IB Management Simulator OpenSM Test Flows:
-
-The simulator provides ability to simulate the SM handling of virtual
-topologies that are not limited to actual lab equipment availability.
-OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily
-regressions use smaller (16 and 128 nodes clusters).
-
-The following test flows are run on the IB management simulator:
-
-* Stability:
- Up to 12 links from the fabric are randomly selected to drop packets
- at drop rates up to 90%. The SM is required to succeed in bringing the
- fabric up. The resulting routing is verified to be correct as well.
-
-* LID Manager:
- Using LMC = 2 the fabric is initialized with LIDs. Faults such as
- zero LID, Duplicated LID, non-aligned (to LMC) LIDs are
- randomly assigned to various nodes and other errors are randomly
- output to the guid2lid cache file. The SM sweep is run 5 times and
- after each iteration a complete verification is made to ensure that all
- LIDs that could possibly be maintained are kept, as well as that all nodes
- were assigned a legal LID range.
-
-* Multicast Routing:
- Nodes randomly join the 0xc000 group and eventually the
- resulting routing is verified for completeness and adherence to
- Up/Down routing rules.
-
-* osmtest:
- The complete osmtest flow as described in the previous table is run on
- the simulated fabrics.
-
-* Stress Test:
- This flow merges fabric, LID and stability issues with continuous
- PathRecord, ServiceRecord and Multicast Join/Leave activity to
- stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get
- were added to the test such both existing and non existing nodes
- perform them in random order.
-
-5.3 OpenSM Regression
-
-Using a back-to-back or single switch connection, the following set of
-tests is run nightly on the stacks described in table 2. The included
-tests are:
-
-* Stress Testing: Flood the SA with queries from multiple channel
- adapters to check the robustness of the entire stack up to the SA.
-
-* Dynamic Changes: Dynamic Topology changes, through randomly
- dropping SMP packets, used to test OpenSM adaptation to an unstable
- network & verify DB correctness.
-
-* Trap Injection: This flow injects traps to the SM and verifies that it
- handles them gracefully.
-
-* SA Query Test: This test exhaustively checks the SA responses to all
- possible single component mask. To do that the test examines the
- entire set of records the SA can provide, classifies them by their
- field values and then selects every field (using component mask and a
- value) and verifies that the response matches the expected set of records.
- A random selection using multiple component mask bits is also performed.
-
-5.4 Cluster testing:
-
-Cluster testing is usually run before a distribution release. It
-involves real hardware setups of 16 to 32 nodes (or more if a beta site
-is available). Each test is validated by running all-to-all ping through the IB
-interface. The test procedure includes:
-
-* Cluster bringup
-
-* Hand-off between 2 or 3 SM's while performing:
- - Node reboots
- - Switch power cycles (disconnecting the SM's)
-
-* Unresponsive port detection and recovery
-
-* osmtest from multiple nodes
-
-* Trap injection and recovery
-
-
-6 Qualification
-----------------
-
-Table 2 - Qualified IB Stacks
-=============================
-
-Stack | Version
------------------------------------------|--------------------------
-OFED | 1.1
-OFED | 1.0
-OpenIB Gen2 (IBG2 distribution) | 1.0
-OpenIB Gen1 (IBGD distribution) | 1.8.0
-VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later
-
-Table 3 - Qualified Devices and Corresponding Firmware
-======================================================
-
-Mellanox
-Device | FW versions
---------|-----------------------------------------------------------
-MT43132 | InfiniScale - fw-43132 5.2.0 (and later)
-MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)
-MT23108 | InfiniHost - fw-23108 3.3.2
-MT25204 | InfiniHost III Lx - fw-25204 1.0.1
-MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)
-MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)
-
-QLogic/PathScale
-Device | Note
---------|-----------------------------------------------------------
-iPath | QHT6040 (PathScale InfiniPath HT-460)
-iPath | QHT6140 (PathScale InfiniPath HT-465)
-iPath | QLE6140 (PathScale InfiniPath PE-880)
-
-Note: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose
-QP0 and QP1. However, it does support it as a device on the subnet.
+ OpenSM Release Notes 3.0.10\r
+ =============================\r
+\r
+Version: OpenFabrics Enterprise Distribution (OFED) 1.2\r
+Repo: git://git.openfabrics.org/~ofed_1_2/management.git (release)\r
+ git://git.openfabrics.org/~halr/management.git (development)\r
+Date: April 2007\r
+\r
+1 Overview\r
+----------\r
+This document describes the contents of the OpenSM OFED 1.2 release. \r
+OpenSM is an InfiniBand compliant Subnet Manager and Administration,\r
+and runs on top of OpenIB. The OpenSM version for this release \r
+is openib-3.0.9\r
+\r
+This document includes the following sections:\r
+1 This Overview section (describing new features and software\r
+ dependencies)\r
+2 Known Issues And Limitations\r
+3 Unsupported IB compliance statements\r
+4 Major Bug Fixes\r
+5 Main Verification Flows\r
+6 Qualified software stacks and devices\r
+\r
+1.1 Major New Features\r
+\r
+* Routing improvements\r
+ Two additional routing algorithms have been added in addition to\r
+ performance improvements to the existing routing algorithms. The\r
+ two new routing algorithms are FAT tree and LASH. LASH is\r
+ experimental. See the opensm man page for additional details.\r
+\r
+* SA Optional Record support now "virtually" complete\r
+ Includes SA InformInfo improvements and InformInfoRecord support in\r
+ addition to support for the remaining SA optional records \r
+ (MulticastForwardingTableRecord, SwitchInfoRecord). Also, SMInfoRecord \r
+ support was improved to include all SMs found.\r
+\r
+* SA database dump/restore\r
+ OpenSM now includes the ability to dump and restore the SA database.\r
+ This allows for all SA registrations (multicast, services, and events)\r
+ to be saved and restored later.\r
+\r
+ In verbose mode, OpenSM will dump SA DB (existing multicast groups,\r
+ services and InformInfo) into dump file which named "opensm-sa.dump"\r
+ and located under standard OpenSM dump directory (/var/log by default).\r
+\r
+ If option -S is specified and SA DB dump file name is provided, OpenSM\r
+ will try to restore SA database from this file. And if restore is\r
+ successful, OpenSM won't ask for client reregistration at subnet bring-up.\r
+\r
+* Modular routing for multicast\r
+ In conjunction was SA database dump/restore, there is the ability to\r
+ dump and load switch lid matrices (min hops tables) which are used\r
+ for multicast route calculation.\r
+\r
+* IB router enablement\r
+ OpenSM now supports router ports properly (in terms of PortInfo handling).\r
+ There is also some experiment support for IB routers which is enabled via\r
+ the ROUTER_EXP compile flag. This support includes SA PathRecord and \r
+ MCMemberRecord support for off subnet GIDs.\r
+\r
+* Socket support added to console\r
+ OpenSM console now supports remote in addition to local access.\r
+ Remote access is currently via telnet.\r
+\r
+1.2 Minor New Features:\r
+\r
+* Change output format of DR path from hex to decimal port numbers\r
+\r
+* Log rotation\r
+ The OpenSM log can now be rotated while OpenSM is running (without\r
+ stopping and restarting OpenSM). This is accomplished via SIGUSR1.\r
+\r
+* Support scope for IPoIB multicast groups in partition config\r
+\r
+* Dump filename changed from subnet.lst to osm-subnet.lst\r
+ Default temp directory for non Windows platforms was previously changed\r
+ from /tmp to /var/log.\r
+\r
+* Add option for force SDR link speed\r
+ Add option to opensm.opts to force link speed. Currently, only forcing\r
+ to SDR link speed is supported.\r
+\r
+1.3 Library API Changes\r
+\r
+ None\r
+\r
+1.4 Software Dependencies\r
+\r
+OpenSM depends on the installation of either OFED 1.2, OFED 1.1,\r
+OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD\r
+distribution), or Mellanox VAPI stacks. The qualified driver versions\r
+are provided in Table 2, "Qualified IB Stacks".\r
+\r
+1.5 Supported Devices Firmware\r
+\r
+The main task of OpenSM is to initialize InfiniBand devices. The\r
+qualified devices and their corresponding firmware versions \r
+are listed in Table 3.\r
+\r
+2 Known Issues And Limitations\r
+------------------------------\r
+\r
+* No Service / Key associations:\r
+ There is no way to manage Service access by Keys. \r
+\r
+* No SM to SM SMDB synchronization: \r
+ Puts the burden of re-registering services, multicast groups, and\r
+ inform-info on the client application (or IB access layer core).\r
+\r
+* No "port down" event handling:\r
+ Changing the switch port through which OpenSM connects to the IB\r
+ fabric may cause incorrect operation. Please restart OpenSM whenever\r
+ such a connectivity change is made.\r
+\r
+* Changing connections during SM operation:\r
+ Under some conditions the SM can get confused by a change in \r
+ cabling (moving a cable from one switch port to the other) and \r
+ momentarily see this as having the same GUID appear connected \r
+ to two different IB ports. Under some conditions, when the SM fails to \r
+ get the corresponding change event it might mistakenly report this case\r
+ as a "duplicated GUID" case and abort. It is advisable to double-check\r
+ the syslog after each such change in connectivity and restart\r
+ OpenSM if it has exited. The same error ("duplicated GUID") will \r
+ also appear with a loopback plug.\r
+\r
+3 Unsupported IB Compliance Statements\r
+--------------------------------------\r
+The following section lists all the IB compliance statements which\r
+OpenSM does not support. Please refer to the IB specification for detailed\r
+information regarding each compliance statement. \r
+\r
+* C14-22 (Authentication):\r
+ M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one\r
+ SubnSet method. As a work-around, an OpenSM option is provided for\r
+ defining the protect bits.\r
+\r
+* C14-67 (Authentication):\r
+ On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then\r
+ the SM shall generate a SubnGetResp if the M_Key matches, or\r
+ silently drop the packet if M_Key does not match.\r
+\r
+* C15-0.1.23.4 (Authentication):\r
+ InformInfoRecords shall always be provided with the QPN set to 0,\r
+ except for the case of a trusted request, in which case the actual\r
+ subscriber QPN shall be returned. \r
+\r
+* o13-17.1.2 (Event-FWD):\r
+ If no permission to forward, the subscription should be removed and\r
+ no further forwarding should occur.\r
+\r
+* C14-24.1.1.5 and C14-62.1.1.22 (Initialization):\r
+ GUIDInfo - SM should enable assigning Port GUIDInfo.\r
+\r
+* C14-44 (Initialization):\r
+ If the SM discovers that it is missing an M_Key to update CA/RT/SW,\r
+ it should notify the higher level.\r
+\r
+* C14-62.1.1.12 (Initialization):\r
+ PortInfo:M_Key - Set the M_Key to a node based random value.\r
+\r
+* C14-62.1.1.13 (Initialization):\r
+ PortInfo:P_KeyProtectBits - set according to an optional policy.\r
+\r
+* C14-62.1.1.24 (Initialization):\r
+ SwitchInfo:DefaultPort - should be configured for random FDB.\r
+\r
+* C14-62.1.1.32 (Initialization):\r
+ RandomForwardingTable should be configured.\r
+\r
+* o15-0.1.12 (Multicast):\r
+ If the JoinState is SendOnlyNonMember = 1 (only), then the endport\r
+ should join as sender only.\r
+\r
+* o15-0.1.8 (Multicast):\r
+ If a request for creating an MCG with fields that cannot be met,\r
+ return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass).\r
+\r
+* C15-0.1.8.6 (SA-Query):\r
+ Respond to SubnAdmGetTraceTable - this is an optional attribute.\r
+\r
+* C15-0.1.13 Services:\r
+ Reject ServiceRecord create, modify or delete if the given\r
+ ServiceP_Key does not match the one included in the ServiceGID port\r
+ and the port that sent the request.\r
+\r
+* C15-0.1.14 (Services):\r
+ Provide means to associate service name and ServiceKeys.\r
+\r
+4 Major Bug Fixes\r
+-----------------\r
+\r
+The following is a list of bugs that were fixed. Note that other less critical\r
+or visible bugs were also fixed.\r
+\r
+* osm_port_info_rcv.c: In __osm_pi_rcv_process_endport,\r
+ isSMdisabled also indicates that an SM is present so poll SMInfo \r
+\r
+* osm_sm_state_mgr.c: In __osm_sm_state_mgr_send_master_sm_info_req, \r
+ handle master GUID port not found properly\r
+\r
+* osm_sa_multipath_record.c: In __osm_mpr_rcv_get_path_parms, return \r
+ IB_NOT_FOUND rather than IB_ERROR when can't route to LID from switch\r
+\r
+* osm_sa_path_record.c: In __osm_pr_rcv_get_path_parms, return IB_NOT_FOUND\r
+ rather than IB_ERROR when can't route to LID from switch\r
+\r
+* osm_vendor_ibumad.c: In osm_vendor_set_sm, set issmfd to\r
+ -1 on open error\r
+\r
+* osm_vendor_ibumad: Termination crash fix\r
+ When OpenSM is terminated umad_receiver thread still running even after\r
+ the structures are destroyed and freed, this causes to random (but easily\r
+ reproducible) crashes. The reason is that osm_vendor_delete() does not\r
+ care about thread termination. This patch adds the receiver thread\r
+ cancellation (by using pthread_cancel() and pthread_join()) and cares to\r
+ keep have all mutexes unlocked upon termination. There is also minor\r
+ termination code consolidation - osm_vendor_port_close() function.\r
+\r
+* osm_port_profile.h: Fix reinsertion issue in osm_port_prof_set_ignored_port\r
+\r
+* osm_matrix.h: Fix segfault with up/down and root nodes file\r
+\r
+* osm_sa_path_record.c: In osm_pr_rcv_process, fix endian of hop_limit\r
+\r
+* osm_vendor_ibumad.c: Close umad port in osm_vendor_delete\r
+\r
+* osm_sa_(multipath path)_record.c: Fix MultiPathRecord/PathRecord issues\r
+ with using MTU/rate/PktLife explicitly ignoring selectors\r
+\r
+ OpenSM just uses the resulting path MTU/rate/pkt-life and fail the\r
+ query even though the selector might be allowing for selecting an\r
+ appropriate value.\r
+\r
+ After this fix, the following results are obtained for a case of\r
+ path allowing maximal 2K MTU.\r
+\r
+In standard mode:\r
+------------------------------------------------------------\r
+MTU greater than ... 256 (0x01) -> equal to ....... 2K\r
+MTU less than ...... 256 (0x41) -> NO PATHS\r
+MTU equal to ....... 256 (0x81) -> equal to ....... 256\r
+MTU largest possible 256 (0xc1) -> equal to ....... 2K\r
+MTU greater than ... 512 (0x02) -> equal to ....... 2K\r
+MTU less than ...... 512 (0x42) -> equal to ....... 256\r
+MTU equal to ....... 512 (0x82) -> equal to ....... 512\r
+MTU largest possible 512 (0xc2) -> equal to ....... 2K\r
+MTU greater than ... 1K (0x03) -> equal to ....... 2K\r
+MTU less than ...... 1K (0x43) -> equal to ....... 512\r
+MTU equal to ....... 1K (0x83) -> equal to ....... 1K\r
+MTU largest possible 1K (0xc3) -> equal to ....... 2K\r
+MTU greater than ... 2K (0x04) -> NO PATHS\r
+MTU less than ...... 2K (0x44) -> equal to ....... 1K\r
+MTU equal to ....... 2K (0x84) -> equal to ....... 2K\r
+MTU largest possible 2K (0xc4) -> equal to ....... 2K\r
+MTU greater than ... 4K (0x05) -> NO PATHS\r
+MTU less than ...... 4K (0x45) -> equal to ....... 2K\r
+MTU equal to ....... 4K (0x85) -> NO PATHS\r
+MTU largest possible 4K (0xc5) -> equal to ....... 2K\r
+============================================================\r
+\r
+With enable_quirks (when one of the ends is a Tavor device):\r
+------------------------------------------------------------\r
+MTU greater than ... 256 (0x01) -> equal to ....... 1K\r
+MTU less than ...... 256 (0x41) -> NO PATHS\r
+MTU equal to ....... 256 (0x81) -> equal to ....... 256\r
+MTU largest possible 256 (0xc1) -> equal to ....... 2K\r
+MTU greater than ... 512 (0x02) -> equal to ....... 1K\r
+MTU less than ...... 512 (0x42) -> equal to ....... 256\r
+MTU equal to ....... 512 (0x82) -> equal to ....... 512\r
+MTU largest possible 512 (0xc2) -> equal to ....... 2K\r
+MTU greater than ... 1K (0x03) -> NO PATHS\r
+MTU less than ...... 1K (0x43) -> equal to ....... 512\r
+MTU equal to ....... 1K (0x83) -> equal to ....... 1K\r
+MTU largest possible 1K (0xc3) -> equal to ....... 2K\r
+MTU greater than ... 2K (0x04) -> NO PATHS\r
+MTU less than ...... 2K (0x44) -> equal to ....... 1K\r
+MTU equal to ....... 2K (0x84) -> equal to ....... 2K\r
+MTU largest possible 2K (0xc4) -> equal to ....... 2K\r
+MTU greater than ... 4K (0x05) -> NO PATHS\r
+MTU less than ...... 4K (0x45) -> equal to ....... 1K\r
+MTU equal to ....... 4K (0x85) -> NO PATHS\r
+MTU largest possible 4K (0xc5) -> equal to ....... 2K\r
+============================================================\r
+\r
+* osm_pkey_rcv.c: rwlock double release fix\r
+ When the port is removed from subnet, but previously requested pkey\r
+ table block is received after this - the lock will be released twice.\r
+ This leads to deadlocks later when other MAD processor will try to\r
+ acquire the same lock.\r
+\r
+* osm_sa_informinfo.c: Fix InformInfoRecord searches\r
+\r
+* Better SA MCMemberRecord leave locking\r
+ Hold locked multicast group leave request (MCMember Record) processing.\r
+ This prevents kind of race with multicast group join request where\r
+ those requests can be reordered during processing.\r
+\r
+* osm_sa_informinfo.c: Conformance changes for subscribe component\r
+\r
+* osm_sa_path_record.c: Handle LID 0 as error\r
+\r
+* Fix comparing InformInfo records\r
+ 1. The received InformInfo struct was modified before dumping it.\r
+ 2. The function that compares InformInfo structures was just\r
+ comparing the whole memory allocated for it, including reserved\r
+ fields. Fixed to compare more selectively.\r
+\r
+ As for QPN, from the IB spec, table 119 InformInfo:\r
+ QPN : Ignored except when subscribe=0 (an unsubscribe\r
+ request). Queue pair to which Report()s were sent as\r
+ a result of a corresponding subscription. If no\r
+ subscription for this Report() with this QPN exists,\r
+ the request to unsubscribe performs no action and\r
+ produces GetResp() with status indicating an invalid\r
+ field value.\r
+\r
+* osm_trap_rcv.c: Reduce repeated trap messages so log doesn't fill\r
+ so quickly\r
+\r
+* osm_helper.c: Fix stack smashing detected problem in osm_dump_service_record\r
+\r
+* Fix permission on db files directory\r
+ When creating directory for db files (guid2lid) storing create it with\r
+ reasonable permissions (current 777 decimal = octal 01411) and don't do\r
+ it world writable.\r
+\r
+* Fix node_desc.description as string usages\r
+\r
+5 Main Verification Flows\r
+-------------------------\r
+\r
+OpenSM verification is run using the following activities:\r
+* osmtest - a stand-alone program\r
+* ibmgtsim (IB management simulator) based - a set of flows that\r
+ simulate clusters, inject errors and verify OpenSM capability to\r
+ respond and bring up the network correctly.\r
+* small cluster regression testing - where the SM is used on back to\r
+ back or single switch configurations. The regression includes\r
+ multiple OpenSM dedicated tests.\r
+* cluster testing - when we run OpenSM to setup a large cluster, perform\r
+ hand-off, reboots and reconnects, verify routing correctness and SA\r
+ responsiveness at the ULP level (IPoIB and SDP).\r
+\r
+5.1 osmtest\r
+\r
+osmtest is an automated verification tool used for OpenSM\r
+testing. Its verification flows are described by list below. \r
+\r
+* Inventory File: Obtain and verify all port info, node info, link and path\r
+ records parameters.\r
+\r
+* Service Record:\r
+ - Register new service\r
+ - Register another service (with a lease period)\r
+ - Register another service (with service p_key set to zero)\r
+ - Get all services by name\r
+ - Delete the first service\r
+ - Delete the third service\r
+ - Added bad flows of get/delete non valid service\r
+ - Add / Get same service with different data \r
+ - Add / Get / Delete by different component mask values (services\r
+ by Name & Key / Name & Data / Name & Id / Id only )\r
+\r
+* Multicast Member Record: \r
+ - Query of existing Groups (IPoIB)\r
+ - BAD Join with insufficient comp mask (o15.0.1.3)\r
+ - Create given MGID=0 (o15.0.1.4)\r
+ - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)\r
+ - Create BAD MGID=0xFA. (o15.0.1.6)\r
+ - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)\r
+ - New MGID with invalid join state (o15.0.1.9)\r
+ - Retry of existing MGID - See JoinState update (o15.0.1.11)\r
+ - BAD RATE when connecting to existing MGID (o15.0.1.13)\r
+ - Partial JoinState delete request - removing FullMember (o15.0.1.14)\r
+ - Full Delete of a group (o15.0.1.14)\r
+ - Verify Delete by trying to Join deleted group (o15.0.1.14)\r
+ - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)\r
+\r
+* GUIDInfo Record:\r
+ - All GUIDInfoRecords in subnet are obtained\r
+\r
+* MultiPathRecord:\r
+ - Perform some compliant and noncompliant MultiPathRecord requests\r
+ - Validation is via status in responses and IB analyzer\r
+\r
+* PKeyTableRecord:\r
+ - Perform some compliant and noncompliant PKeyTableRecord queries\r
+ - Validation is via status in responses and IB analyzer\r
+\r
+* LinearForwardingTableRecord:\r
+ - Perform some compliant and noncompliant LinearForwardingTableRecord queries\r
+ - Validation is via status in responses and IB analyzer\r
+\r
+* Event Forwarding: Register for trap forwarding using reports\r
+ - Send a trap and wait for report\r
+ - Unregister non-existing\r
+\r
+* Trap 64/65 Flow: Register to Trap 64-65, create traps (by\r
+ disconnecting/connecting ports) and wait for report, then unregister.\r
+\r
+* Stress Test: send PortInfoRecord queries, both single and RMPP and\r
+ check for the rate of responses as well as their validity.\r
+\r
+\r
+5.2 IB Management Simulator OpenSM Test Flows:\r
+\r
+The simulator provides ability to simulate the SM handling of virtual\r
+topologies that are not limited to actual lab equipment availability.\r
+OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily\r
+regressions use smaller (16 and 128 nodes clusters).\r
+\r
+The following test flows are run on the IB management simulator:\r
+\r
+* Stability:\r
+ Up to 12 links from the fabric are randomly selected to drop packets\r
+ at drop rates up to 90%. The SM is required to succeed in bringing the\r
+ fabric up. The resulting routing is verified to be correct as well.\r
+\r
+* LID Manager:\r
+ Using LMC = 2 the fabric is initialized with LIDs. Faults such as\r
+ zero LID, Duplicated LID, non-aligned (to LMC) LIDs are \r
+ randomly assigned to various nodes and other errors are randomly\r
+ output to the guid2lid cache file. The SM sweep is run 5 times and\r
+ after each iteration a complete verification is made to ensure that all\r
+ LIDs that could possibly be maintained are kept, as well as that all nodes\r
+ were assigned a legal LID range.\r
+\r
+* Multicast Routing:\r
+ Nodes randomly join the 0xc000 group and eventually the\r
+ resulting routing is verified for completeness and adherence to\r
+ Up/Down routing rules. \r
+\r
+* osmtest:\r
+ The complete osmtest flow as described in the previous table is run on\r
+ the simulated fabrics.\r
+\r
+* Stress Test:\r
+ This flow merges fabric, LID and stability issues with continuous \r
+ PathRecord, ServiceRecord and Multicast Join/Leave activity to\r
+ stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get\r
+ were added to the test such both existing and non existing nodes\r
+ perform them in random order.\r
+\r
+5.3 OpenSM Regression\r
+\r
+Using a back-to-back or single switch connection, the following set of\r
+tests is run nightly on the stacks described in table 2. The included\r
+tests are:\r
+\r
+* Stress Testing: Flood the SA with queries from multiple channel\r
+ adapters to check the robustness of the entire stack up to the SA.\r
+\r
+* Dynamic Changes: Dynamic Topology changes, through randomly\r
+ dropping SMP packets, used to test OpenSM adaptation to an unstable\r
+ network & verify DB correctness.\r
+\r
+* Trap Injection: This flow injects traps to the SM and verifies that it\r
+ handles them gracefully. \r
+\r
+* SA Query Test: This test exhaustively checks the SA responses to all\r
+ possible single component mask. To do that the test examines the\r
+ entire set of records the SA can provide, classifies them by their\r
+ field values and then selects every field (using component mask and a\r
+ value) and verifies that the response matches the expected set of records.\r
+ A random selection using multiple component mask bits is also performed.\r
+\r
+5.4 Cluster testing:\r
+\r
+Cluster testing is usually run before a distribution release. It\r
+involves real hardware setups of 16 to 32 nodes (or more if a beta site\r
+is available). Each test is validated by running all-to-all ping through the IB\r
+interface. The test procedure includes:\r
+\r
+* Cluster bringup \r
+\r
+* Hand-off between 2 or 3 SM's while performing:\r
+ - Node reboots\r
+ - Switch power cycles (disconnecting the SM's)\r
+\r
+* Unresponsive port detection and recovery\r
+\r
+* osmtest from multiple nodes\r
+\r
+* Trap injection and recovery\r
+\r
+\r
+6 Qualification \r
+----------------\r
+\r
+Table 2 - Qualified IB Stacks\r
+=============================\r
+\r
+Stack | Version\r
+-----------------------------------------|--------------------------\r
+OFED | 1.2\r
+OFED | 1.1\r
+OFED | 1.0\r
+OpenIB Gen2 (IBG2 distribution) | 1.0\r
+OpenIB Gen1 (IBGD distribution) | 1.8.0\r
+VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later\r
+\r
+Table 3 - Qualified Devices and Corresponding Firmware \r
+======================================================\r
+\r
+Mellanox\r
+Device | FW versions \r
+--------|-----------------------------------------------------------\r
+MT43132 | InfiniScale - fw-43132 5.2.0 (and later)\r
+MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)\r
+MT23108 | InfiniHost - fw-23108 3.3.2 (and later)\r
+MT25204 | InfiniHost III Lx - fw-25204 1.0.1i (and later)\r
+MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)\r
+MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)\r
+\r
+QLogic/PathScale\r
+Device | Note\r
+--------|-----------------------------------------------------------\r
+iPath | QHT6040 (PathScale InfiniPath HT-460)\r
+iPath | QHT6140 (PathScale InfiniPath HT-465)\r
+iPath | QLE6140 (PathScale InfiniPath PE-880)\r
+\r
+Note: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose \r
+QP0 and QP1. However, it does support it as a device on the subnet.\r
+\r
projects / compat-rdma / docs.git / commitdiff