- OpenSM Release Notes 3.0.11\r
- =============================\r
-\r
-Version: OpenFabrics Enterprise Distribution (OFED) 1.2\r
-Repo: git://git.openfabrics.org/~ofed_1_2/management.git (release)\r
- git://git.openfabrics.org/~halr/management.git (development)\r
-Date: May 2007\r
-\r
-1 Overview\r
-----------\r
-This document describes the contents of the OpenSM OFED 1.2 release. \r
-OpenSM is an InfiniBand compliant Subnet Manager and Administration,\r
-and runs on top of OpenIB. The OpenSM version for this release \r
-is openib-3.0.11\r
-\r
-This document includes the following sections:\r
-1 This Overview section (describing new features and software\r
- dependencies)\r
-2 Known Issues And Limitations\r
-3 Unsupported IB compliance statements\r
-4 Major Bug Fixes\r
-5 Main Verification Flows\r
-6 Qualified software stacks and devices\r
-\r
-1.1 Major New Features\r
-\r
-* Routing improvements\r
- Two additional routing algorithms have been added in addition to\r
- performance improvements to the existing routing algorithms. The\r
- two new routing algorithms are FAT tree and LASH. See the \r
- opensm man page for additional details.\r
-\r
-* SA Optional Record support now "virtually" complete\r
- Includes SA InformInfo improvements and InformInfoRecord support in\r
- addition to support for the remaining SA optional records \r
- (MulticastForwardingTableRecord, SwitchInfoRecord). Also, SMInfoRecord \r
- support was improved to include all SMs found.\r
-\r
-* SA database dump/restore\r
- OpenSM now includes the ability to dump and restore the SA database.\r
- This allows for all SA registrations (multicast, services, and events)\r
- to be saved and restored later.\r
-\r
- In verbose mode, OpenSM will dump SA DB (existing multicast groups,\r
- services and InformInfo) into dump file which named "opensm-sa.dump"\r
- and located under standard OpenSM dump directory (/var/log by default).\r
-\r
- If option -S is specified and SA DB dump file name is provided, OpenSM\r
- will try to restore SA database from this file. And if restore is\r
- successful, OpenSM won't ask for client reregistration at subnet bring-up.\r
-\r
-* Modular routing for multicast\r
- In conjunction was SA database dump/restore, there is the ability to\r
- dump and load switch lid matrices (min hops tables) which are used\r
- for multicast route calculation.\r
-\r
-* IB router enablement\r
- OpenSM now supports router ports properly (in terms of PortInfo handling).\r
- There is also some experimental support for IB routers which is enabled\r
- via the ROUTER_EXP compile flag. This support includes SA PathRecord and \r
- MCMemberRecord support for off subnet GIDs.\r
-\r
-* Socket support added to console\r
- OpenSM console now supports remote in addition to local access.\r
- Remote access is currently via telnet.\r
-\r
-1.2 Minor New Features:\r
-\r
-* Change output format of DR path from hex to decimal port numbers\r
-\r
-* Log rotation\r
- The OpenSM log can now be rotated while OpenSM is running (without\r
- stopping and restarting OpenSM). This is accomplished via SIGUSR1.\r
-\r
-* Support scope for IPoIB multicast groups in partition config\r
-\r
-* Dump filename changed from subnet.lst to osm-subnet.lst\r
- Default temp directory for non Windows platforms was previously changed\r
- from /tmp to /var/log.\r
-\r
-* Add option for force SDR link speed\r
- Add option to opensm.opts to force link speed. Currently, only forcing\r
- to SDR link speed is supported. This option is not supported as a\r
- command line option.\r
-\r
-1.3 Library API Changes\r
-\r
- None\r
-\r
-1.4 Software Dependencies\r
-\r
-OpenSM depends on the installation of either OFED 1.2, OFED 1.1,\r
-OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD\r
-distribution), or Mellanox VAPI stacks. The qualified driver versions\r
-are provided in Table 2, "Qualified IB Stacks".\r
-\r
-1.5 Supported Devices Firmware\r
-\r
-The main task of OpenSM is to initialize InfiniBand devices. The\r
-qualified devices and their corresponding firmware versions \r
-are listed in Table 3.\r
-\r
-2 Known Issues And Limitations\r
-------------------------------\r
-\r
-* No Service / Key associations:\r
- There is no way to manage Service access by Keys. \r
-\r
-* No SM to SM SMDB synchronization: \r
- Puts the burden of re-registering services, multicast groups, and\r
- inform-info on the client application (or IB access layer core).\r
-\r
-* No "port down" event handling:\r
- Changing the switch port through which OpenSM connects to the IB\r
- fabric may cause incorrect operation. Please restart OpenSM whenever\r
- such a connectivity change is made.\r
-\r
-* Changing connections during SM operation:\r
- Under some conditions the SM can get confused by a change in \r
- cabling (moving a cable from one switch port to the other) and \r
- momentarily see this as having the same GUID appear connected \r
- to two different IB ports. Under some conditions, when the SM fails to \r
- get the corresponding change event it might mistakenly report this case\r
- as a "duplicated GUID" case and abort. It is advisable to double-check\r
- the syslog after each such change in connectivity and restart\r
- OpenSM if it has exited. The same error ("duplicated GUID") will \r
- also appear with a loopback plug.\r
-\r
-3 Unsupported IB Compliance Statements\r
---------------------------------------\r
-The following section lists all the IB compliance statements which\r
-OpenSM does not support. Please refer to the IB specification for detailed\r
-information regarding each compliance statement. \r
-\r
-* C14-22 (Authentication):\r
- M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one\r
- SubnSet method. As a work-around, an OpenSM option is provided for\r
- defining the protect bits.\r
-\r
-* C14-67 (Authentication):\r
- On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then\r
- the SM shall generate a SubnGetResp if the M_Key matches, or\r
- silently drop the packet if M_Key does not match.\r
-\r
-* C15-0.1.23.4 (Authentication):\r
- InformInfoRecords shall always be provided with the QPN set to 0,\r
- except for the case of a trusted request, in which case the actual\r
- subscriber QPN shall be returned. \r
-\r
-* o13-17.1.2 (Event-FWD):\r
- If no permission to forward, the subscription should be removed and\r
- no further forwarding should occur.\r
-\r
-* C14-24.1.1.5 and C14-62.1.1.22 (Initialization):\r
- GUIDInfo - SM should enable assigning Port GUIDInfo.\r
-\r
-* C14-44 (Initialization):\r
- If the SM discovers that it is missing an M_Key to update CA/RT/SW,\r
- it should notify the higher level.\r
-\r
-* C14-62.1.1.12 (Initialization):\r
- PortInfo:M_Key - Set the M_Key to a node based random value.\r
-\r
-* C14-62.1.1.13 (Initialization):\r
- PortInfo:P_KeyProtectBits - set according to an optional policy.\r
-\r
-* C14-62.1.1.24 (Initialization):\r
- SwitchInfo:DefaultPort - should be configured for random FDB.\r
-\r
-* C14-62.1.1.32 (Initialization):\r
- RandomForwardingTable should be configured.\r
-\r
-* o15-0.1.12 (Multicast):\r
- If the JoinState is SendOnlyNonMember = 1 (only), then the endport\r
- should join as sender only.\r
-\r
-* o15-0.1.8 (Multicast):\r
- If a request for creating an MCG with fields that cannot be met,\r
- return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass).\r
-\r
-* C15-0.1.8.6 (SA-Query):\r
- Respond to SubnAdmGetTraceTable - this is an optional attribute.\r
-\r
-* C15-0.1.13 Services:\r
- Reject ServiceRecord create, modify or delete if the given\r
- ServiceP_Key does not match the one included in the ServiceGID port\r
- and the port that sent the request.\r
-\r
-* C15-0.1.14 (Services):\r
- Provide means to associate service name and ServiceKeys.\r
-\r
-4 Major Bug Fixes\r
------------------\r
-\r
-The following is a list of bugs that were fixed. Note that other less critical\r
-or visible bugs were also fixed.\r
-\r
-* osm_port_info_rcv.c: In __osm_pi_rcv_process_endport,\r
- isSMdisabled also indicates that an SM is present so poll SMInfo\r
-\r
-* osm_sm_state_mgr.c: In __osm_sm_state_mgr_send_master_sm_info_req, \r
- handle master GUID port not found properly\r
-\r
-* osm_sa_multipath_record.c: In __osm_mpr_rcv_get_path_parms, return \r
- IB_NOT_FOUND rather than IB_ERROR when can't route to LID from switch\r
-\r
-* osm_sa_path_record.c: In __osm_pr_rcv_get_path_parms, return IB_NOT_FOUND\r
- rather than IB_ERROR when can't route to LID from switch\r
-\r
-* osm_vendor_ibumad.c: In osm_vendor_set_sm, set issmfd to\r
- -1 on open error\r
-\r
-* osm_vendor_ibumad: Termination crash fix\r
- When OpenSM is terminated umad_receiver thread still running even after\r
- the structures are destroyed and freed, this causes to random (but easily\r
- reproducible) crashes. The reason is that osm_vendor_delete() does not\r
- care about thread termination. This patch adds the receiver thread\r
- cancellation (by using pthread_cancel() and pthread_join()) and cares to\r
- keep have all mutexes unlocked upon termination. There is also minor\r
- termination code consolidation - osm_vendor_port_close() function.\r
-\r
-* osm_port_profile.h: Fix reinsertion issue in osm_port_prof_set_ignored_port\r
-\r
-* osm_matrix.h: Fix segfault with up/down and root nodes file\r
-\r
-* osm_sa_path_record.c: In osm_pr_rcv_process, fix endian of hop_limit\r
-\r
-* osm_vendor_ibumad.c: Close umad port in osm_vendor_delete\r
-\r
-* osm_sa_(multipath path)_record.c: Fix MultiPathRecord/PathRecord issues\r
- with using MTU/rate/PktLife explicitly ignoring selectors\r
-\r
- OpenSM just uses the resulting path MTU/rate/pkt-life and fail the\r
- query even though the selector might be allowing for selecting an\r
- appropriate value.\r
-\r
- After this fix, the following results are obtained for a case of\r
- path allowing maximal 2K MTU.\r
-\r
-In standard mode:\r
-------------------------------------------------------------\r
-MTU greater than ... 256 (0x01) -> equal to ....... 2K\r
-MTU less than ...... 256 (0x41) -> NO PATHS\r
-MTU equal to ....... 256 (0x81) -> equal to ....... 256\r
-MTU largest possible 256 (0xc1) -> equal to ....... 2K\r
-MTU greater than ... 512 (0x02) -> equal to ....... 2K\r
-MTU less than ...... 512 (0x42) -> equal to ....... 256\r
-MTU equal to ....... 512 (0x82) -> equal to ....... 512\r
-MTU largest possible 512 (0xc2) -> equal to ....... 2K\r
-MTU greater than ... 1K (0x03) -> equal to ....... 2K\r
-MTU less than ...... 1K (0x43) -> equal to ....... 512\r
-MTU equal to ....... 1K (0x83) -> equal to ....... 1K\r
-MTU largest possible 1K (0xc3) -> equal to ....... 2K\r
-MTU greater than ... 2K (0x04) -> NO PATHS\r
-MTU less than ...... 2K (0x44) -> equal to ....... 1K\r
-MTU equal to ....... 2K (0x84) -> equal to ....... 2K\r
-MTU largest possible 2K (0xc4) -> equal to ....... 2K\r
-MTU greater than ... 4K (0x05) -> NO PATHS\r
-MTU less than ...... 4K (0x45) -> equal to ....... 2K\r
-MTU equal to ....... 4K (0x85) -> NO PATHS\r
-MTU largest possible 4K (0xc5) -> equal to ....... 2K\r
-============================================================\r
-\r
-With enable_quirks (when one of the ends is a Tavor device):\r
-------------------------------------------------------------\r
-MTU greater than ... 256 (0x01) -> equal to ....... 1K\r
-MTU less than ...... 256 (0x41) -> NO PATHS\r
-MTU equal to ....... 256 (0x81) -> equal to ....... 256\r
-MTU largest possible 256 (0xc1) -> equal to ....... 2K\r
-MTU greater than ... 512 (0x02) -> equal to ....... 1K\r
-MTU less than ...... 512 (0x42) -> equal to ....... 256\r
-MTU equal to ....... 512 (0x82) -> equal to ....... 512\r
-MTU largest possible 512 (0xc2) -> equal to ....... 2K\r
-MTU greater than ... 1K (0x03) -> NO PATHS\r
-MTU less than ...... 1K (0x43) -> equal to ....... 512\r
-MTU equal to ....... 1K (0x83) -> equal to ....... 1K\r
-MTU largest possible 1K (0xc3) -> equal to ....... 2K\r
-MTU greater than ... 2K (0x04) -> NO PATHS\r
-MTU less than ...... 2K (0x44) -> equal to ....... 1K\r
-MTU equal to ....... 2K (0x84) -> equal to ....... 2K\r
-MTU largest possible 2K (0xc4) -> equal to ....... 2K\r
-MTU greater than ... 4K (0x05) -> NO PATHS\r
-MTU less than ...... 4K (0x45) -> equal to ....... 1K\r
-MTU equal to ....... 4K (0x85) -> NO PATHS\r
-MTU largest possible 4K (0xc5) -> equal to ....... 2K\r
-============================================================\r
-\r
-* osm_pkey_rcv.c: rwlock double release fix\r
- When the port is removed from subnet, but previously requested pkey\r
- table block is received after this - the lock will be released twice.\r
- This leads to deadlocks later when other MAD processor will try to\r
- acquire the same lock.\r
-\r
-* osm_sa_informinfo.c: Fix InformInfoRecord searches\r
-\r
-* Better SA MCMemberRecord leave locking\r
- Hold locked multicast group leave request (MCMember Record) processing.\r
- This prevents kind of race with multicast group join request where\r
- those requests can be reordered during processing.\r
-\r
-* osm_sa_informinfo.c: Conformance changes for subscribe component\r
-\r
-* osm_sa_path_record.c: Handle LID 0 as error\r
-\r
-* Fix comparing InformInfo records\r
- 1. The received InformInfo struct was modified before dumping it.\r
- 2. The function that compares InformInfo structures was just\r
- comparing the whole memory allocated for it, including reserved\r
- fields. Fixed to compare more selectively.\r
-\r
- As for QPN, from the IB spec, table 119 InformInfo:\r
- QPN : Ignored except when subscribe=0 (an unsubscribe\r
- request). Queue pair to which Report()s were sent as\r
- a result of a corresponding subscription. If no\r
- subscription for this Report() with this QPN exists,\r
- the request to unsubscribe performs no action and\r
- produces GetResp() with status indicating an invalid\r
- field value.\r
-\r
-* osm_trap_rcv.c: Reduce repeated trap messages so log doesn't fill\r
- so quickly\r
-\r
-* osm_helper.c: Fix stack smashing detected problem in osm_dump_service_record\r
-\r
-* Fix permission on db files directory\r
- When creating directory for db files (guid2lid) storing create it with\r
- reasonable permissions (current 777 decimal = octal 01411) and don't do\r
- it world writable.\r
-\r
-* Fix node_desc.description as string usages\r
-\r
-5 Main Verification Flows\r
--------------------------\r
-\r
-OpenSM verification is run using the following activities:\r
-* osmtest - a stand-alone program\r
-* ibmgtsim (IB management simulator) based - a set of flows that\r
- simulate clusters, inject errors and verify OpenSM capability to\r
- respond and bring up the network correctly.\r
-* small cluster regression testing - where the SM is used on back to\r
- back or single switch configurations. The regression includes\r
- multiple OpenSM dedicated tests.\r
-* cluster testing - when we run OpenSM to setup a large cluster, perform\r
- hand-off, reboots and reconnects, verify routing correctness and SA\r
- responsiveness at the ULP level (IPoIB and SDP).\r
-\r
-5.1 osmtest\r
-\r
-osmtest is an automated verification tool used for OpenSM\r
-testing. Its verification flows are described by list below. \r
-\r
-* Inventory File: Obtain and verify all port info, node info, link and path\r
- records parameters.\r
-\r
-* Service Record:\r
- - Register new service\r
- - Register another service (with a lease period)\r
- - Register another service (with service p_key set to zero)\r
- - Get all services by name\r
- - Delete the first service\r
- - Delete the third service\r
- - Added bad flows of get/delete non valid service\r
- - Add / Get same service with different data \r
- - Add / Get / Delete by different component mask values (services\r
- by Name & Key / Name & Data / Name & Id / Id only )\r
-\r
-* Multicast Member Record: \r
- - Query of existing Groups (IPoIB)\r
- - BAD Join with insufficient comp mask (o15.0.1.3)\r
- - Create given MGID=0 (o15.0.1.4)\r
- - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)\r
- - Create BAD MGID=0xFA. (o15.0.1.6)\r
- - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)\r
- - New MGID with invalid join state (o15.0.1.9)\r
- - Retry of existing MGID - See JoinState update (o15.0.1.11)\r
- - BAD RATE when connecting to existing MGID (o15.0.1.13)\r
- - Partial JoinState delete request - removing FullMember (o15.0.1.14)\r
- - Full Delete of a group (o15.0.1.14)\r
- - Verify Delete by trying to Join deleted group (o15.0.1.14)\r
- - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)\r
-\r
-* GUIDInfo Record:\r
- - All GUIDInfoRecords in subnet are obtained\r
-\r
-* MultiPathRecord:\r
- - Perform some compliant and noncompliant MultiPathRecord requests\r
- - Validation is via status in responses and IB analyzer\r
-\r
-* PKeyTableRecord:\r
- - Perform some compliant and noncompliant PKeyTableRecord queries\r
- - Validation is via status in responses and IB analyzer\r
-\r
-* LinearForwardingTableRecord:\r
- - Perform some compliant and noncompliant LinearForwardingTableRecord queries\r
- - Validation is via status in responses and IB analyzer\r
-\r
-* Event Forwarding: Register for trap forwarding using reports\r
- - Send a trap and wait for report\r
- - Unregister non-existing\r
-\r
-* Trap 64/65 Flow: Register to Trap 64-65, create traps (by\r
- disconnecting/connecting ports) and wait for report, then unregister.\r
-\r
-* Stress Test: send PortInfoRecord queries, both single and RMPP and\r
- check for the rate of responses as well as their validity.\r
-\r
-\r
-5.2 IB Management Simulator OpenSM Test Flows:\r
-\r
-The simulator provides ability to simulate the SM handling of virtual\r
-topologies that are not limited to actual lab equipment availability.\r
-OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily\r
-regressions use smaller (16 and 128 nodes clusters).\r
-\r
-The following test flows are run on the IB management simulator:\r
-\r
-* Stability:\r
- Up to 12 links from the fabric are randomly selected to drop packets\r
- at drop rates up to 90%. The SM is required to succeed in bringing the\r
- fabric up. The resulting routing is verified to be correct as well.\r
-\r
-* LID Manager:\r
- Using LMC = 2 the fabric is initialized with LIDs. Faults such as\r
- zero LID, Duplicated LID, non-aligned (to LMC) LIDs are \r
- randomly assigned to various nodes and other errors are randomly\r
- output to the guid2lid cache file. The SM sweep is run 5 times and\r
- after each iteration a complete verification is made to ensure that all\r
- LIDs that could possibly be maintained are kept, as well as that all nodes\r
- were assigned a legal LID range.\r
-\r
-* Multicast Routing:\r
- Nodes randomly join the 0xc000 group and eventually the\r
- resulting routing is verified for completeness and adherence to\r
- Up/Down routing rules. \r
-\r
-* osmtest:\r
- The complete osmtest flow as described in the previous table is run on\r
- the simulated fabrics.\r
-\r
-* Stress Test:\r
- This flow merges fabric, LID and stability issues with continuous \r
- PathRecord, ServiceRecord and Multicast Join/Leave activity to\r
- stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get\r
- were added to the test such both existing and non existing nodes\r
- perform them in random order.\r
-\r
-5.3 OpenSM Regression\r
-\r
-Using a back-to-back or single switch connection, the following set of\r
-tests is run nightly on the stacks described in table 2. The included\r
-tests are:\r
-\r
-* Stress Testing: Flood the SA with queries from multiple channel\r
- adapters to check the robustness of the entire stack up to the SA.\r
-\r
-* Dynamic Changes: Dynamic Topology changes, through randomly\r
- dropping SMP packets, used to test OpenSM adaptation to an unstable\r
- network & verify DB correctness.\r
-\r
-* Trap Injection: This flow injects traps to the SM and verifies that it\r
- handles them gracefully. \r
-\r
-* SA Query Test: This test exhaustively checks the SA responses to all\r
- possible single component mask. To do that the test examines the\r
- entire set of records the SA can provide, classifies them by their\r
- field values and then selects every field (using component mask and a\r
- value) and verifies that the response matches the expected set of records.\r
- A random selection using multiple component mask bits is also performed.\r
-\r
-5.4 Cluster testing:\r
-\r
-Cluster testing is usually run before a distribution release. It\r
-involves real hardware setups of 16 to 32 nodes (or more if a beta site\r
-is available). Each test is validated by running all-to-all ping through the IB\r
-interface. The test procedure includes:\r
-\r
-* Cluster bringup \r
-\r
-* Hand-off between 2 or 3 SM's while performing:\r
- - Node reboots\r
- - Switch power cycles (disconnecting the SM's)\r
-\r
-* Unresponsive port detection and recovery\r
-\r
-* osmtest from multiple nodes\r
-\r
-* Trap injection and recovery\r
-\r
-\r
-6 Qualification \r
-----------------\r
-\r
-Table 2 - Qualified IB Stacks\r
-=============================\r
-\r
-Stack | Version\r
------------------------------------------|--------------------------\r
-OFED | 1.2\r
-OFED | 1.1\r
-OFED | 1.0\r
-OpenIB Gen2 (IBG2 distribution) | 1.0\r
-OpenIB Gen1 (IBGD distribution) | 1.8.0\r
-VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later\r
-\r
-Table 3 - Qualified Devices and Corresponding Firmware \r
-======================================================\r
-\r
-Mellanox\r
-Device | FW versions \r
---------|-----------------------------------------------------------\r
-MT43132 | InfiniScale - fw-43132 5.2.0 (and later)\r
-MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)\r
-MT23108 | InfiniHost - fw-23108 3.3.2 (and later)\r
-MT25204 | InfiniHost III Lx - fw-25204 1.0.1i (and later)\r
-MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)\r
-MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)\r
-\r
-QLogic/PathScale\r
-Device | Note\r
---------|-----------------------------------------------------------\r
-iPath | QHT6040 (PathScale InfiniPath HT-460)\r
-iPath | QHT6140 (PathScale InfiniPath HT-465)\r
-iPath | QLE6140 (PathScale InfiniPath PE-880)\r
-\r
-Note: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose \r
-QP0 and QP1. However, it does support it as a device on the subnet.\r
-\r
+ OpenSM Release Notes 3.0.11
+ =============================
+
+Version: OpenFabrics Enterprise Distribution (OFED) 1.2
+Repo: git://git.openfabrics.org/~ofed_1_2/management.git (release)
+ git://git.openfabrics.org/~halr/management.git (development)
+Date: May 2007
+
+1 Overview
+----------
+This document describes the contents of the OpenSM OFED 1.2 release.
+OpenSM is an InfiniBand compliant Subnet Manager and Administration,
+and runs on top of OpenIB. The OpenSM version for this release
+is openib-3.0.11
+
+This document includes the following sections:
+1 This Overview section (describing new features and software
+ dependencies)
+2 Known Issues And Limitations
+3 Unsupported IB compliance statements
+4 Major Bug Fixes
+5 Main Verification Flows
+6 Qualified software stacks and devices
+
+1.1 Major New Features
+
+* Routing improvements
+ Two additional routing algorithms have been added in addition to
+ performance improvements to the existing routing algorithms. The
+ two new routing algorithms are FAT tree and LASH. See the
+ opensm man page for additional details.
+
+* SA Optional Record support now "virtually" complete
+ Includes SA InformInfo improvements and InformInfoRecord support in
+ addition to support for the remaining SA optional records
+ (MulticastForwardingTableRecord, SwitchInfoRecord). Also, SMInfoRecord
+ support was improved to include all SMs found.
+
+* SA database dump/restore
+ OpenSM now includes the ability to dump and restore the SA database.
+ This allows for all SA registrations (multicast, services, and events)
+ to be saved and restored later.
+
+ In verbose mode, OpenSM will dump SA DB (existing multicast groups,
+ services and InformInfo) into dump file which named "opensm-sa.dump"
+ and located under standard OpenSM dump directory (/var/log by default).
+
+ If option -S is specified and SA DB dump file name is provided, OpenSM
+ will try to restore SA database from this file. And if restore is
+ successful, OpenSM won't ask for client reregistration at subnet bring-up.
+
+* Modular routing for multicast
+ In conjunction was SA database dump/restore, there is the ability to
+ dump and load switch lid matrices (min hops tables) which are used
+ for multicast route calculation.
+
+* IB router enablement
+ OpenSM now supports router ports properly (in terms of PortInfo handling).
+ There is also some experimental support for IB routers which is enabled
+ via the ROUTER_EXP compile flag. This support includes SA PathRecord and
+ MCMemberRecord support for off subnet GIDs.
+
+* Socket support added to console
+ OpenSM console now supports remote in addition to local access.
+ Remote access is currently via telnet.
+
+1.2 Minor New Features:
+
+* Change output format of DR path from hex to decimal port numbers
+
+* Log rotation
+ The OpenSM log can now be rotated while OpenSM is running (without
+ stopping and restarting OpenSM). This is accomplished via SIGUSR1.
+
+* Support scope for IPoIB multicast groups in partition config
+
+* Dump filename changed from subnet.lst to osm-subnet.lst
+ Default temp directory for non Windows platforms was previously changed
+ from /tmp to /var/log.
+
+* Add option for force SDR link speed
+ Add option to opensm.opts to force link speed. Currently, only forcing
+ to SDR link speed is supported. This option is not supported as a
+ command line option.
+
+1.3 Library API Changes
+
+ None
+
+1.4 Software Dependencies
+
+OpenSM depends on the installation of either OFED 1.2, OFED 1.1,
+OFED 1.0, OpenIB gen2 (e.g. IBG2 distribution), OpenIB gen1 (e.g. IBGD
+distribution), or Mellanox VAPI stacks. The qualified driver versions
+are provided in Table 2, "Qualified IB Stacks".
+
+1.5 Supported Devices Firmware
+
+The main task of OpenSM is to initialize InfiniBand devices. The
+qualified devices and their corresponding firmware versions
+are listed in Table 3.
+
+2 Known Issues And Limitations
+------------------------------
+
+* No Service / Key associations:
+ There is no way to manage Service access by Keys.
+
+* No SM to SM SMDB synchronization:
+ Puts the burden of re-registering services, multicast groups, and
+ inform-info on the client application (or IB access layer core).
+
+* No "port down" event handling:
+ Changing the switch port through which OpenSM connects to the IB
+ fabric may cause incorrect operation. Please restart OpenSM whenever
+ such a connectivity change is made.
+
+* Changing connections during SM operation:
+ Under some conditions the SM can get confused by a change in
+ cabling (moving a cable from one switch port to the other) and
+ momentarily see this as having the same GUID appear connected
+ to two different IB ports. Under some conditions, when the SM fails to
+ get the corresponding change event it might mistakenly report this case
+ as a "duplicated GUID" case and abort. It is advisable to double-check
+ the syslog after each such change in connectivity and restart
+ OpenSM if it has exited. The same error ("duplicated GUID") will
+ also appear with a loopback plug.
+
+3 Unsupported IB Compliance Statements
+--------------------------------------
+The following section lists all the IB compliance statements which
+OpenSM does not support. Please refer to the IB specification for detailed
+information regarding each compliance statement.
+
+* C14-22 (Authentication):
+ M_Key M_KeyProtectBits and M_KeyLeasePeriod shall be set in one
+ SubnSet method. As a work-around, an OpenSM option is provided for
+ defining the protect bits.
+
+* C14-67 (Authentication):
+ On SubnGet(SMInfo) and SubnSet(SMInfo) - if M_Key is not zero then
+ the SM shall generate a SubnGetResp if the M_Key matches, or
+ silently drop the packet if M_Key does not match.
+
+* C15-0.1.23.4 (Authentication):
+ InformInfoRecords shall always be provided with the QPN set to 0,
+ except for the case of a trusted request, in which case the actual
+ subscriber QPN shall be returned.
+
+* o13-17.1.2 (Event-FWD):
+ If no permission to forward, the subscription should be removed and
+ no further forwarding should occur.
+
+* C14-24.1.1.5 and C14-62.1.1.22 (Initialization):
+ GUIDInfo - SM should enable assigning Port GUIDInfo.
+
+* C14-44 (Initialization):
+ If the SM discovers that it is missing an M_Key to update CA/RT/SW,
+ it should notify the higher level.
+
+* C14-62.1.1.12 (Initialization):
+ PortInfo:M_Key - Set the M_Key to a node based random value.
+
+* C14-62.1.1.13 (Initialization):
+ PortInfo:P_KeyProtectBits - set according to an optional policy.
+
+* C14-62.1.1.24 (Initialization):
+ SwitchInfo:DefaultPort - should be configured for random FDB.
+
+* C14-62.1.1.32 (Initialization):
+ RandomForwardingTable should be configured.
+
+* o15-0.1.12 (Multicast):
+ If the JoinState is SendOnlyNonMember = 1 (only), then the endport
+ should join as sender only.
+
+* o15-0.1.8 (Multicast):
+ If a request for creating an MCG with fields that cannot be met,
+ return ERR_REQ_INVALID (currently ignores SL and FlowLabelTClass).
+
+* C15-0.1.8.6 (SA-Query):
+ Respond to SubnAdmGetTraceTable - this is an optional attribute.
+
+* C15-0.1.13 Services:
+ Reject ServiceRecord create, modify or delete if the given
+ ServiceP_Key does not match the one included in the ServiceGID port
+ and the port that sent the request.
+
+* C15-0.1.14 (Services):
+ Provide means to associate service name and ServiceKeys.
+
+4 Major Bug Fixes
+-----------------
+
+The following is a list of bugs that were fixed. Note that other less critical
+or visible bugs were also fixed.
+
+* osm_port_info_rcv.c: In __osm_pi_rcv_process_endport,
+ isSMdisabled also indicates that an SM is present so poll SMInfo
+
+* osm_sm_state_mgr.c: In __osm_sm_state_mgr_send_master_sm_info_req,
+ handle master GUID port not found properly
+
+* osm_sa_multipath_record.c: In __osm_mpr_rcv_get_path_parms, return
+ IB_NOT_FOUND rather than IB_ERROR when can't route to LID from switch
+
+* osm_sa_path_record.c: In __osm_pr_rcv_get_path_parms, return IB_NOT_FOUND
+ rather than IB_ERROR when can't route to LID from switch
+
+* osm_vendor_ibumad.c: In osm_vendor_set_sm, set issmfd to
+ -1 on open error
+
+* osm_vendor_ibumad: Termination crash fix
+ When OpenSM is terminated umad_receiver thread still running even after
+ the structures are destroyed and freed, this causes to random (but easily
+ reproducible) crashes. The reason is that osm_vendor_delete() does not
+ care about thread termination. This patch adds the receiver thread
+ cancellation (by using pthread_cancel() and pthread_join()) and cares to
+ keep have all mutexes unlocked upon termination. There is also minor
+ termination code consolidation - osm_vendor_port_close() function.
+
+* osm_port_profile.h: Fix reinsertion issue in osm_port_prof_set_ignored_port
+
+* osm_matrix.h: Fix segfault with up/down and root nodes file
+
+* osm_sa_path_record.c: In osm_pr_rcv_process, fix endian of hop_limit
+
+* osm_vendor_ibumad.c: Close umad port in osm_vendor_delete
+
+* osm_sa_(multipath path)_record.c: Fix MultiPathRecord/PathRecord issues
+ with using MTU/rate/PktLife explicitly ignoring selectors
+
+ OpenSM just uses the resulting path MTU/rate/pkt-life and fail the
+ query even though the selector might be allowing for selecting an
+ appropriate value.
+
+ After this fix, the following results are obtained for a case of
+ path allowing maximal 2K MTU.
+
+In standard mode:
+------------------------------------------------------------
+MTU greater than ... 256 (0x01) -> equal to ....... 2K
+MTU less than ...... 256 (0x41) -> NO PATHS
+MTU equal to ....... 256 (0x81) -> equal to ....... 256
+MTU largest possible 256 (0xc1) -> equal to ....... 2K
+MTU greater than ... 512 (0x02) -> equal to ....... 2K
+MTU less than ...... 512 (0x42) -> equal to ....... 256
+MTU equal to ....... 512 (0x82) -> equal to ....... 512
+MTU largest possible 512 (0xc2) -> equal to ....... 2K
+MTU greater than ... 1K (0x03) -> equal to ....... 2K
+MTU less than ...... 1K (0x43) -> equal to ....... 512
+MTU equal to ....... 1K (0x83) -> equal to ....... 1K
+MTU largest possible 1K (0xc3) -> equal to ....... 2K
+MTU greater than ... 2K (0x04) -> NO PATHS
+MTU less than ...... 2K (0x44) -> equal to ....... 1K
+MTU equal to ....... 2K (0x84) -> equal to ....... 2K
+MTU largest possible 2K (0xc4) -> equal to ....... 2K
+MTU greater than ... 4K (0x05) -> NO PATHS
+MTU less than ...... 4K (0x45) -> equal to ....... 2K
+MTU equal to ....... 4K (0x85) -> NO PATHS
+MTU largest possible 4K (0xc5) -> equal to ....... 2K
+============================================================
+
+With enable_quirks (when one of the ends is a Tavor device):
+------------------------------------------------------------
+MTU greater than ... 256 (0x01) -> equal to ....... 1K
+MTU less than ...... 256 (0x41) -> NO PATHS
+MTU equal to ....... 256 (0x81) -> equal to ....... 256
+MTU largest possible 256 (0xc1) -> equal to ....... 2K
+MTU greater than ... 512 (0x02) -> equal to ....... 1K
+MTU less than ...... 512 (0x42) -> equal to ....... 256
+MTU equal to ....... 512 (0x82) -> equal to ....... 512
+MTU largest possible 512 (0xc2) -> equal to ....... 2K
+MTU greater than ... 1K (0x03) -> NO PATHS
+MTU less than ...... 1K (0x43) -> equal to ....... 512
+MTU equal to ....... 1K (0x83) -> equal to ....... 1K
+MTU largest possible 1K (0xc3) -> equal to ....... 2K
+MTU greater than ... 2K (0x04) -> NO PATHS
+MTU less than ...... 2K (0x44) -> equal to ....... 1K
+MTU equal to ....... 2K (0x84) -> equal to ....... 2K
+MTU largest possible 2K (0xc4) -> equal to ....... 2K
+MTU greater than ... 4K (0x05) -> NO PATHS
+MTU less than ...... 4K (0x45) -> equal to ....... 1K
+MTU equal to ....... 4K (0x85) -> NO PATHS
+MTU largest possible 4K (0xc5) -> equal to ....... 2K
+============================================================
+
+* osm_pkey_rcv.c: rwlock double release fix
+ When the port is removed from subnet, but previously requested pkey
+ table block is received after this - the lock will be released twice.
+ This leads to deadlocks later when other MAD processor will try to
+ acquire the same lock.
+
+* osm_sa_informinfo.c: Fix InformInfoRecord searches
+
+* Better SA MCMemberRecord leave locking
+ Hold locked multicast group leave request (MCMember Record) processing.
+ This prevents kind of race with multicast group join request where
+ those requests can be reordered during processing.
+
+* osm_sa_informinfo.c: Conformance changes for subscribe component
+
+* osm_sa_path_record.c: Handle LID 0 as error
+
+* Fix comparing InformInfo records
+ 1. The received InformInfo struct was modified before dumping it.
+ 2. The function that compares InformInfo structures was just
+ comparing the whole memory allocated for it, including reserved
+ fields. Fixed to compare more selectively.
+
+ As for QPN, from the IB spec, table 119 InformInfo:
+ QPN : Ignored except when subscribe=0 (an unsubscribe
+ request). Queue pair to which Report()s were sent as
+ a result of a corresponding subscription. If no
+ subscription for this Report() with this QPN exists,
+ the request to unsubscribe performs no action and
+ produces GetResp() with status indicating an invalid
+ field value.
+
+* osm_trap_rcv.c: Reduce repeated trap messages so log doesn't fill
+ so quickly
+
+* osm_helper.c: Fix stack smashing detected problem in osm_dump_service_record
+
+* Fix permission on db files directory
+ When creating directory for db files (guid2lid) storing create it with
+ reasonable permissions (current 777 decimal = octal 01411) and don't do
+ it world writable.
+
+* Fix node_desc.description as string usages
+
+5 Main Verification Flows
+-------------------------
+
+OpenSM verification is run using the following activities:
+* osmtest - a stand-alone program
+* ibmgtsim (IB management simulator) based - a set of flows that
+ simulate clusters, inject errors and verify OpenSM capability to
+ respond and bring up the network correctly.
+* small cluster regression testing - where the SM is used on back to
+ back or single switch configurations. The regression includes
+ multiple OpenSM dedicated tests.
+* cluster testing - when we run OpenSM to setup a large cluster, perform
+ hand-off, reboots and reconnects, verify routing correctness and SA
+ responsiveness at the ULP level (IPoIB and SDP).
+
+5.1 osmtest
+
+osmtest is an automated verification tool used for OpenSM
+testing. Its verification flows are described by list below.
+
+* Inventory File: Obtain and verify all port info, node info, link and path
+ records parameters.
+
+* Service Record:
+ - Register new service
+ - Register another service (with a lease period)
+ - Register another service (with service p_key set to zero)
+ - Get all services by name
+ - Delete the first service
+ - Delete the third service
+ - Added bad flows of get/delete non valid service
+ - Add / Get same service with different data
+ - Add / Get / Delete by different component mask values (services
+ by Name & Key / Name & Data / Name & Id / Id only )
+
+* Multicast Member Record:
+ - Query of existing Groups (IPoIB)
+ - BAD Join with insufficient comp mask (o15.0.1.3)
+ - Create given MGID=0 (o15.0.1.4)
+ - Create given MGID=0xFF12A01C,FE800000,00000000,12345678 (o15.0.1.4)
+ - Create BAD MGID=0xFA. (o15.0.1.6)
+ - Create BAD MGID=0xFF12A01B w/ link-local not set (o15.0.1.6)
+ - New MGID with invalid join state (o15.0.1.9)
+ - Retry of existing MGID - See JoinState update (o15.0.1.11)
+ - BAD RATE when connecting to existing MGID (o15.0.1.13)
+ - Partial JoinState delete request - removing FullMember (o15.0.1.14)
+ - Full Delete of a group (o15.0.1.14)
+ - Verify Delete by trying to Join deleted group (o15.0.1.14)
+ - BAD Delete of IPoIB membership (no prev join) (o15.0.1.15)
+
+* GUIDInfo Record:
+ - All GUIDInfoRecords in subnet are obtained
+
+* MultiPathRecord:
+ - Perform some compliant and noncompliant MultiPathRecord requests
+ - Validation is via status in responses and IB analyzer
+
+* PKeyTableRecord:
+ - Perform some compliant and noncompliant PKeyTableRecord queries
+ - Validation is via status in responses and IB analyzer
+
+* LinearForwardingTableRecord:
+ - Perform some compliant and noncompliant LinearForwardingTableRecord queries
+ - Validation is via status in responses and IB analyzer
+
+* Event Forwarding: Register for trap forwarding using reports
+ - Send a trap and wait for report
+ - Unregister non-existing
+
+* Trap 64/65 Flow: Register to Trap 64-65, create traps (by
+ disconnecting/connecting ports) and wait for report, then unregister.
+
+* Stress Test: send PortInfoRecord queries, both single and RMPP and
+ check for the rate of responses as well as their validity.
+
+
+5.2 IB Management Simulator OpenSM Test Flows:
+
+The simulator provides ability to simulate the SM handling of virtual
+topologies that are not limited to actual lab equipment availability.
+OpenSM was simulated to bring up clusters of up to 10,000 nodes. Daily
+regressions use smaller (16 and 128 nodes clusters).
+
+The following test flows are run on the IB management simulator:
+
+* Stability:
+ Up to 12 links from the fabric are randomly selected to drop packets
+ at drop rates up to 90%. The SM is required to succeed in bringing the
+ fabric up. The resulting routing is verified to be correct as well.
+
+* LID Manager:
+ Using LMC = 2 the fabric is initialized with LIDs. Faults such as
+ zero LID, Duplicated LID, non-aligned (to LMC) LIDs are
+ randomly assigned to various nodes and other errors are randomly
+ output to the guid2lid cache file. The SM sweep is run 5 times and
+ after each iteration a complete verification is made to ensure that all
+ LIDs that could possibly be maintained are kept, as well as that all nodes
+ were assigned a legal LID range.
+
+* Multicast Routing:
+ Nodes randomly join the 0xc000 group and eventually the
+ resulting routing is verified for completeness and adherence to
+ Up/Down routing rules.
+
+* osmtest:
+ The complete osmtest flow as described in the previous table is run on
+ the simulated fabrics.
+
+* Stress Test:
+ This flow merges fabric, LID and stability issues with continuous
+ PathRecord, ServiceRecord and Multicast Join/Leave activity to
+ stress the SM/SA during continuous sweeps. InformInfo Set/Delete/Get
+ were added to the test such both existing and non existing nodes
+ perform them in random order.
+
+5.3 OpenSM Regression
+
+Using a back-to-back or single switch connection, the following set of
+tests is run nightly on the stacks described in table 2. The included
+tests are:
+
+* Stress Testing: Flood the SA with queries from multiple channel
+ adapters to check the robustness of the entire stack up to the SA.
+
+* Dynamic Changes: Dynamic Topology changes, through randomly
+ dropping SMP packets, used to test OpenSM adaptation to an unstable
+ network & verify DB correctness.
+
+* Trap Injection: This flow injects traps to the SM and verifies that it
+ handles them gracefully.
+
+* SA Query Test: This test exhaustively checks the SA responses to all
+ possible single component mask. To do that the test examines the
+ entire set of records the SA can provide, classifies them by their
+ field values and then selects every field (using component mask and a
+ value) and verifies that the response matches the expected set of records.
+ A random selection using multiple component mask bits is also performed.
+
+5.4 Cluster testing:
+
+Cluster testing is usually run before a distribution release. It
+involves real hardware setups of 16 to 32 nodes (or more if a beta site
+is available). Each test is validated by running all-to-all ping through the IB
+interface. The test procedure includes:
+
+* Cluster bringup
+
+* Hand-off between 2 or 3 SM's while performing:
+ - Node reboots
+ - Switch power cycles (disconnecting the SM's)
+
+* Unresponsive port detection and recovery
+
+* osmtest from multiple nodes
+
+* Trap injection and recovery
+
+
+6 Qualification
+----------------
+
+Table 2 - Qualified IB Stacks
+=============================
+
+Stack | Version
+-----------------------------------------|--------------------------
+OFED | 1.2
+OFED | 1.1
+OFED | 1.0
+OpenIB Gen2 (IBG2 distribution) | 1.0
+OpenIB Gen1 (IBGD distribution) | 1.8.0
+VAPI (Mellanox InfiniBand HCA Driver) | 3.2 and later
+
+Table 3 - Qualified Devices and Corresponding Firmware
+======================================================
+
+Mellanox
+Device | FW versions
+--------|-----------------------------------------------------------
+MT43132 | InfiniScale - fw-43132 5.2.0 (and later)
+MT47396 | InfiniScale III - fw-47396 0.5.0 (and later)
+MT23108 | InfiniHost - fw-23108 3.3.2 (and later)
+MT25204 | InfiniHost III Lx - fw-25204 1.0.1i (and later)
+MT25208 | InfiniHost III Ex (InfiniHost Mode) - fw-25208 4.6.2 (and later)
+MT25208 | InfiniHost III Ex (MemFree Mode) - fw-25218 5.0.1 (and later)
+
+QLogic/PathScale
+Device | Note
+--------|-----------------------------------------------------------
+iPath | QHT6040 (PathScale InfiniPath HT-460)
+iPath | QHT6140 (PathScale InfiniPath HT-465)
+iPath | QLE6140 (PathScale InfiniPath PE-880)
+
+Note: OpenSM does not run on an IBM Galaxy (eHCA) as it does not expose
+QP0 and QP1. However, it does support it as a device on the subnet.
+
projects / compat-rdma / docs.git / commitdiff