From: Tziporet Koren Date: Thu, 7 Feb 2008 09:00:35 +0000 (+0200) Subject: same name style for all README files X-Git-Url: https://openfabrics.org/gitweb/?a=commitdiff_plain;h=b2df6db165547206a36cd4645f7463a5e98cf7b7;p=~aditr%2Fcompat-rdma-docs.git same name style for all README files Signed-off-by: Tziporet Koren --- diff --git a/MSTFLINT_README.txt b/MSTFLINT_README.txt new file mode 100644 index 0000000..9e17adf --- /dev/null +++ b/MSTFLINT_README.txt @@ -0,0 +1,151 @@ +Mellanox Technologies, Inc. - www.mellanox.com +*********************************************** + +mstflint: A BURNING FIRMWARE APPLICATION +---------------------------------------- + +1) Overview + This package contains a burning tool, and diagnostic tools, for Mellanox + manufactured HCA cards. It also provides access to the relevant source + code. Please see the file LICENSE for licensing details. + + ---------------------------------------------------------------------------- + NOTE: + This burning tool should be used only with Mellanox-manufactured + HCA cards. Using it with cards manufactured by other vendors may be harmful + to the cards (due to different configurations). + Using diagnostic tools is usually safe for all HCAs. + ---------------------------------------------------------------------------- + +2) Package Contents + a) mstflint source code. + b) mflash lib. + This lib provides flash access through Mellanox HCAs. + c) mtcr linux utility. + This utility enables the mflash lib to access the HCA + hardware registers. + d) mstregdump utility + This utility dumps hardware registers from Mellanox hardware, + for later analysis by Mellanox + e) mstvpd + This utility dumps on-card VPD + +3) Installation + a) Build the the mstflint utility. + Example: + make + + b) Compilation should end with no error messages. + For example: + >make + g++ -O2 -g -Wall -I. -fno-exceptions -c flint.cpp -o mstflint.o + cc -O2 -g -Wall -I. -fno-exceptions mstflint.o -o mstflint "/usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../lib/libz.a" "/usr/lib/gcc/i486-linux-gnu/4.0.3/libstdc++.a" + cc -O2 -g -Wall -I. mread.c -o mstmread + cc -O2 -g -Wall -I. mwrite.c -o mstmwrite + cc -O2 -g -Wall -I. mstdump.c -o mstregdump + cc -O2 -g -Wall -I. vpd.c -o mstvpd + + c) Executables named mstflint, mstregdump, mstvpd will be generated in the current + directory + This is the mstflint utility that can be used to burn and examine + the HCA on-board flash, mstregdump and mstvpd diagnostic utilities. + You may copy these executables to an arbitrary location. + +4) Requirements: + a) Typically, you will need root privileges for hardware access + + b) For flash update, you will need the binary firmware-image file + supplied by your HCA card vendor. + + If you purchased your card from Mellanox, please use the Mellanox + secured site with your customer login account to download the firmware + binaries, or contact your local field application engineer. + + Or + + If you purchased your card from a vendor other than Mellanox, get a + specific board file from your HCA card vendor and generate the binary + image. + + c) If the mthca driver for the device is not loaded. + You must know the device location on the PCI bus. + For example, to find an InfiniHost HCA car manufactured by Mellanox use: + + /sbin/lspci -d 15b3:5a44 + 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1) + + In this example, 02:00.0 identifies the device in the form bus:dev.fn + +5) Usage (mstflint): + Read mstflint usage. Enter: "./mstflint -h" for short help, or + "./mstflint -hh" for detailed help message. + + Use mstflint to burn a device according to the burning instructions. + You will need to specify the device in the form mthcaX if the driver + has loaded successfully, or bus:dev.fn as explained in section 4c) + above, or in the alternative form /proc/bus/pci/bus/dev.fn . + + If the driver is loaded, unload and re-load it, or reset the device in + some other way, so that the new firmware is re-read from flash. + + Examples: + 1) mstflint -d mthca0 v + This is the recommended method, but it only works + if the mthca driver for the device is loaded. + + Note: it is always safe to try this access method first - + if the driver is not loaded, mstflint will fail with an error + message and you'll be able to retry with another method. + + To verify that the driver for the device mthca0 is loaded, you + can check whether /sys/class/infiniband/mthca0/device exists. + + 2) mstflint -d 02:00.0 v + This is functionally equivalent to access method 1, but less + convenient. This access method works with or without the driver + loaded. + + 3) mstflint -d /proc/bus/pci/02/00.0 v + This method is less safe: verify that driver is not running, + and that no other software accesses the device before using it. + +6) Usage (mstregdump): + Internal register dump is produced on standard output. + Store it in file for analysis in Mellanox. + + Example: + *) mstregdump mthca0 > dumpfile + +7) Usage (mstvpd): + VPD dump is produced on standard output. + A list of keywords to dump can be supplied after the -- flag + to apply an output filter: + + Examples: + 1) mstvpd mthca0 + ID: Lion cub DDR + PN: MHGA28-1T + EC: A3 + SN: MT0551X00740 + V0: PCIe x8 + V1: N/A + YA: R R + RW: + + 2) mstvpd mthca0 -- PN ID + PN: MHGA28-1T + ID: Lion cub DDR + +8) Problem Reporting: + Please collect the following information when reporting issues: + + uname -a + cat /etc/issue + cat /proc/bus/pci/devices + lspci + mstflint -d 02:00.0 v + mstflint -d 02:00.0 q + mstvpd 02:00.0 + + + diff --git a/SRPT_README.txt b/SRPT_README.txt new file mode 100644 index 0000000..709200d --- /dev/null +++ b/SRPT_README.txt @@ -0,0 +1,168 @@ +SCSI RDMA Protocol (SRP) Target driver for Linux +================================================= + +SRP Target driver is designed to work directly on top of OpenFabrics +OFED-1.x software stack (http://www.openfabrics.org) or Infiniband +drivers in Linux kernel tree (kernel.org). It also interfaces with +Generic SCSI target mid-level driver - SCST (http://scst.sourceforge.net) + +By interfacing with SCST driver we are able to work and support a lot IO +modes on real or virtual devices in the backend +1. scst_disk -- interfacing with scsi sub-system to claim and export real +scsi devices ie. disks, hardware raid volumes, tape library as SRP's luns + +2. scst_vdisk -- fileio and blockio modes. This allows you to turn software raid +volumes, LVM volumes, IDE disks, and normal files into SRP's luns (required to +work with latest scst's svn tree) + +3. NULLIO mode will allow you to measure the performance without sending IOs +to *real* devices + + +Prerequisites +------------- +1. First it is required to download and install SCST driver. Please +refer to SCST's Internet page http://scst.sourceforge.net for download +and installation instructions. You can also find SCST's design document, +svn repository development tree and utility tools for basic storage +management + +a. git clone git://git.openfabrics.org/~vu/srpt_inc ~/srpt_inc +b. Checking out scst's svn development tree revision 245 +svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk/scst -r 245 +c. cd scst +d. patch -p0 < ~/srpt_inc/scst_r245.patch +e. make and make install + +2. Download/install IB low-level driver from OFED package or linux kernel tree +a. SRP target is part of OFED-1.3 +or +b. Download and install OFED-1.x (x > 1) with kernel_ib development package rpm +or +c. Built and installed Infiniband driver in Linux "vanilla" kernel tree + from kernel.org + +3. Download SRP target driver from openfabrics.org (needed for 2b, 2c) +git clone git://git.openfabrics.org/~vu/srpt.git ~/srpt + + +Installation +------------ +A. For OFED-1.3 +---------------- +SRP target is part of ofed-1.3. Go through normal installation with srpt enable + + +B. Instruction to included in and built with OFED-1.2.5 development tree +----------------------------------------------------------------------- +1. cd /usr/src/ofa_kernel-1.2.5 or /usr/src/ofa_kernel-1.2.c or +~/ofa_1_2_c_kernel-200708xx-yyyy +2. patch -p1 < ~/srpt_inc/add_srpt_01.patch +3. patch -p1 < ~/srpt_inc/add_srpt_03.patch +4. patch -p1 < ~/srpt_inc/add_srpt_04.patch +5. cp -r ~/srpt drivers/infiniband/ulp/srpt +6. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod + --with-srp-target-mod +7. make and make install + +C. Instruction to be included in and built with OFED-1.2 development tree +-------------------------------------------------------------------------- +1. cd /usr/src/ofa_kernel-1.2 or /usr/src/ofa_kernel-1.2 or +~/ofa_1_2_kernel-200708xx-yyyy +2. patch -p1 < ~/srpt_inc/add_srpt_01.patch +3. patch -p1 < ~/srpt_inc/add_srpt_03.patch +4. patch -p1 < ~/srpt_inc/add_srpt_04.patch +5. cp -r ~/srpt drivers/infiniband/ulp/srpt +6. patch -p1 < ~/srpt_inc/add_srpt_ofed_1_2.patch +7. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod + --with-srp-target-mod +8. make and make install + +D. Instruction to be included in and built with "vanilla" kernel development tree +---------------------------------------------------------------------------------- +1. cd /usr/src/linux-2.6.1x +2. patch -p1 < ~/srpt_inc/add_srpt_03.patch +3. cp -r ~/srpt drivers/infiniband/ulp/srpt +4. configure and build SRP target module driver as normal + + +How-to run +----------- +A. On srp target machine +1. Please refer to SCST's README for loading scsi_tgt driver and its +dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...) + +Example 1: working with real back-end scsi disks +a. modprobe scsi_tgt +b. modprobe scst_disk +c. cat /proc/scsi_tgt/scsi_tgt + +ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt +Device (host:ch:id:lun or name) Device handler +0:0:0:0 dev_disk +4:0:0:0 dev_disk +5:0:0:0 dev_disk +6:0:0:0 dev_disk +7:0:0:0 dev_disk + +Now you want to exclude the first scsi disk and expose the last 4 scsi disks as +IB/SRP luns for I/O +echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices +echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices +echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices +echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices + +Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file) +a. modprobe scsi_tgt +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk +e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices + +Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0) +a. modprobe scsi_tgt +b. modprobe scst_vdisk +c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk +f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices +g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices +h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices + +2. modprobe ib_srpt + + +B. On initiator machines you can manualy do the following steps: +1. modprobe ib_srp +2. ipsrpdm -c (to discover new SRP target) +3. echo > /sys/class/infiniband_srp/srp-mthca0-1/add_target +4. fdisk -l (will show new discovered scsi disks) + +Example: +Assume that you use port 1 of first HCA in the system ie. mthca0 + +[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 +id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 +[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, +dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > +/sys/class/infiniband_srp/srp-mthca0-1/add_target + +OR + ++ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon +automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes ++ To set up and use high availability feature you need dm-multipath driver +and multipath tool ++ Please refer to OFED-1.x SRP's user manual for more in-details instructions +on how-to enable/use HA feature + + +TO DO +------ ++ Stress test and stabilize the code ++ Performance tuning ++ Integrate and test with SCST's admin/management tools for LUNs masking/sharing + among initiator hosts ++ Handle QP, shared-RQ, and asynchronous events diff --git a/VNIC.README b/VNIC.README deleted file mode 100644 index dc1f412..0000000 --- a/VNIC.README +++ /dev/null @@ -1,156 +0,0 @@ -This is a technology preview release of the QLogic VNIC driver on OFED 1.3. -This driver is currently supported on Intel x86 32 and 64 bit machines with -Mellanox HCAs. Supported OS are RHEL 4 Update 3, RHEL 4 Update 4, SLES 10 -and the vanilla 2.6.19 kernel. - -The VNIC driver in conjunction with the QLogic Virtual Ethernet I/O gateway -(VEx) provides Ethernet interfaces on a host with IB HCA(s) without the need -for any physical Ethernet NIC. - -This file describes the use of the VNIC ULP service on an OFED stack -and covers the following points: - -A) Creating VNIC interfaces -B) Discovering VEx IOCs present on the fabric using ibvexdm -C) Starting the VNIC driver and the VNIC interfaces -D) Assigning IP addresses etc for the VNIC interfaces -E) Information about the VNIC interfaces -F) Deleting a specific VNIC interface - -A) Creating VNIC interfaces - -The VNIC interfaces can be created with the help of -a configuration file which must be placed at /etc/infiniband/qlogic_vnic.cfg. - -Please take a look at qlogic_vnic.cfg.sample file (available as part of the -documentation) to see how VNIC configuration files are written. You can use -this configuration file as the basis for creating a VNIC configuration file by -copying it to /etc/infiniband/qlogic_vnic.cfg. Of course you will have to -replace the IOCGUID, IOCSTRING values etc in the sample configuration file -with those of the VEx IOCs present on your fabric. - -[As back ward compatibility for the QLogic Infiniserv stack, if the -/etc/infiniband/qlogic_vnic.cfg file is not present, the VNIC service will -look for the /etc/sysconfig/ics_inic.cfg configuration file] - -B) Discovering VEx IOCs present on the fabric using ibvexdm - -For writing the configuration file, you will need information -about the VEx IOCs present on the fabric like their IOCGUID, -IOCSTRING etc. The ibvexdm tool should be used to get this -information. - -When ibvexdm is executed without any options, it displays -detailed information about all the VEx IOCs present on the fabric: - -# ibvexdm -IO Unit Info: - port LID: 0003 - port GID: fe8000000000000000066a0258000001 - change ID: 0009 - max controllers: 0x03 - - - controller[ 1] - GUID: 00066a0130000001 - vendor ID: 00066a - device ID: 000030 - IO class : 2000 - ID: Chassis 0x00066A00010003F2, Slot 1, IOC 1 - service entries: 2 - service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01 - service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01 - - controller[ 2] - GUID: 00066a0230000001 - vendor ID: 00066a - device ID: 000030 - IO class : 2000 - ID: Chassis 0x00066A00010003F2, Slot 1, IOC 2 - service entries: 2 - service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02 - service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02 - - controller[ 3] - GUID: 00066a0330000001 - vendor ID: 00066a - device ID: 000030 - IO class : 2000 - ID: Chassis 0x00066A00010003F2, Slot 1, IOC 3 - service entries: 2 - service[ 0]: 1000066a00000003 / InfiniNIC.InfiniConSys.Control:03 - service[ 1]: 1000066a00000103 / InfiniNIC.InfiniConSys.Data:03 - - -When ibvexdm is run with -e option, it reports the IOCGUID information -and with -s option it reports the IOCSTRING information for the VEx IOCs -present on the fabric. - -# ibvexdm -e -ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff -ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff -ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff - -#ibvexdm -s -"Chassis 0x00066A00010003F2, Slot 1, IOC 1" -"Chassis 0x00066A00010003F2, Slot 1, IOC 2" -"Chassis 0x00066A00010003F2, Slot 1, IOC 3" - -#ibvexdm -es -ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 1" -ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 2" -ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 3" - -C) Starting the VNIC driver and the VNIC interfaces - -Once you have created a configuration file, you can start the VNIC driver -and create the VNIC interfaces specified in the configuration file with: -#/etc/init.d/openibd start - - -The VNIC driver is started by default as a part of openibd service. -(If openibd is configured to start on boot VNIC driver will also start on boot) -If you do not want to start the VNIC driver as a part of openibd service, set -VNIC_LOAD=no in /etc/infiniband/openib.conf - -You can stop the VNIC driver and openibd services with -#/etc/init.d/openibd stop - -You can restart the VNIC driver and openibd service using -#/etc/init.d/openibd restart - -After starting openibd service you can independently control the VNIC driver using ql_vnic service -To stop the VNIC driver you can use -#/etc/init.d/ql_vnic stop - -If you make changes to the VNIC configuration file and want the changes to be -reflected, you can restart the VNIC service independently, using -#/etc/init.d/ql_vnic restart - -Note that it is required to start openibd service before using ql_vnic service - -D) Assigning IP addresses etc for the VNIC interfaces - -This can be done with ifconfig or by setting up the ifcfg-XXX (ifcfg-veth0 for -an interface named veth0 etc) network files for the corresponding VNIC interfaces. - -E) Information about the VNIC interfaces - -Information about the created VNIC interfaces can be obtained from -/sys/class/infiniband_vnic/interfaces/. A directory is created -for each interface under this directory. - -The directory for each interface contains information about the interface -and the primary and secondary connections. - -F) Deleting a specific VNIC interface - -VNIC interfaces can be deleted by writing the name of the interface to -the /sys/class/infiniband_vnic/interfaces/delete_vnic file. - -For example to delete interface veth0 - -echo -n veth0 > /sys/class/infiniband_vnic/interfaces/delete_vnic - - - diff --git a/VNIC_README.txt b/VNIC_README.txt new file mode 100644 index 0000000..dc1f412 --- /dev/null +++ b/VNIC_README.txt @@ -0,0 +1,156 @@ +This is a technology preview release of the QLogic VNIC driver on OFED 1.3. +This driver is currently supported on Intel x86 32 and 64 bit machines with +Mellanox HCAs. Supported OS are RHEL 4 Update 3, RHEL 4 Update 4, SLES 10 +and the vanilla 2.6.19 kernel. + +The VNIC driver in conjunction with the QLogic Virtual Ethernet I/O gateway +(VEx) provides Ethernet interfaces on a host with IB HCA(s) without the need +for any physical Ethernet NIC. + +This file describes the use of the VNIC ULP service on an OFED stack +and covers the following points: + +A) Creating VNIC interfaces +B) Discovering VEx IOCs present on the fabric using ibvexdm +C) Starting the VNIC driver and the VNIC interfaces +D) Assigning IP addresses etc for the VNIC interfaces +E) Information about the VNIC interfaces +F) Deleting a specific VNIC interface + +A) Creating VNIC interfaces + +The VNIC interfaces can be created with the help of +a configuration file which must be placed at /etc/infiniband/qlogic_vnic.cfg. + +Please take a look at qlogic_vnic.cfg.sample file (available as part of the +documentation) to see how VNIC configuration files are written. You can use +this configuration file as the basis for creating a VNIC configuration file by +copying it to /etc/infiniband/qlogic_vnic.cfg. Of course you will have to +replace the IOCGUID, IOCSTRING values etc in the sample configuration file +with those of the VEx IOCs present on your fabric. + +[As back ward compatibility for the QLogic Infiniserv stack, if the +/etc/infiniband/qlogic_vnic.cfg file is not present, the VNIC service will +look for the /etc/sysconfig/ics_inic.cfg configuration file] + +B) Discovering VEx IOCs present on the fabric using ibvexdm + +For writing the configuration file, you will need information +about the VEx IOCs present on the fabric like their IOCGUID, +IOCSTRING etc. The ibvexdm tool should be used to get this +information. + +When ibvexdm is executed without any options, it displays +detailed information about all the VEx IOCs present on the fabric: + +# ibvexdm +IO Unit Info: + port LID: 0003 + port GID: fe8000000000000000066a0258000001 + change ID: 0009 + max controllers: 0x03 + + + controller[ 1] + GUID: 00066a0130000001 + vendor ID: 00066a + device ID: 000030 + IO class : 2000 + ID: Chassis 0x00066A00010003F2, Slot 1, IOC 1 + service entries: 2 + service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01 + service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01 + + controller[ 2] + GUID: 00066a0230000001 + vendor ID: 00066a + device ID: 000030 + IO class : 2000 + ID: Chassis 0x00066A00010003F2, Slot 1, IOC 2 + service entries: 2 + service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02 + service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02 + + controller[ 3] + GUID: 00066a0330000001 + vendor ID: 00066a + device ID: 000030 + IO class : 2000 + ID: Chassis 0x00066A00010003F2, Slot 1, IOC 3 + service entries: 2 + service[ 0]: 1000066a00000003 / InfiniNIC.InfiniConSys.Control:03 + service[ 1]: 1000066a00000103 / InfiniNIC.InfiniConSys.Data:03 + + +When ibvexdm is run with -e option, it reports the IOCGUID information +and with -s option it reports the IOCSTRING information for the VEx IOCs +present on the fabric. + +# ibvexdm -e +ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff +ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff +ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff + +#ibvexdm -s +"Chassis 0x00066A00010003F2, Slot 1, IOC 1" +"Chassis 0x00066A00010003F2, Slot 1, IOC 2" +"Chassis 0x00066A00010003F2, Slot 1, IOC 3" + +#ibvexdm -es +ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 1" +ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 2" +ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 3" + +C) Starting the VNIC driver and the VNIC interfaces + +Once you have created a configuration file, you can start the VNIC driver +and create the VNIC interfaces specified in the configuration file with: +#/etc/init.d/openibd start + + +The VNIC driver is started by default as a part of openibd service. +(If openibd is configured to start on boot VNIC driver will also start on boot) +If you do not want to start the VNIC driver as a part of openibd service, set +VNIC_LOAD=no in /etc/infiniband/openib.conf + +You can stop the VNIC driver and openibd services with +#/etc/init.d/openibd stop + +You can restart the VNIC driver and openibd service using +#/etc/init.d/openibd restart + +After starting openibd service you can independently control the VNIC driver using ql_vnic service +To stop the VNIC driver you can use +#/etc/init.d/ql_vnic stop + +If you make changes to the VNIC configuration file and want the changes to be +reflected, you can restart the VNIC service independently, using +#/etc/init.d/ql_vnic restart + +Note that it is required to start openibd service before using ql_vnic service + +D) Assigning IP addresses etc for the VNIC interfaces + +This can be done with ifconfig or by setting up the ifcfg-XXX (ifcfg-veth0 for +an interface named veth0 etc) network files for the corresponding VNIC interfaces. + +E) Information about the VNIC interfaces + +Information about the created VNIC interfaces can be obtained from +/sys/class/infiniband_vnic/interfaces/. A directory is created +for each interface under this directory. + +The directory for each interface contains information about the interface +and the primary and secondary connections. + +F) Deleting a specific VNIC interface + +VNIC interfaces can be deleted by writing the name of the interface to +the /sys/class/infiniband_vnic/interfaces/delete_vnic file. + +For example to delete interface veth0 + +echo -n veth0 > /sys/class/infiniband_vnic/interfaces/delete_vnic + + + diff --git a/mstflint.README b/mstflint.README deleted file mode 100644 index 9e17adf..0000000 --- a/mstflint.README +++ /dev/null @@ -1,151 +0,0 @@ -Mellanox Technologies, Inc. - www.mellanox.com -*********************************************** - -mstflint: A BURNING FIRMWARE APPLICATION ----------------------------------------- - -1) Overview - This package contains a burning tool, and diagnostic tools, for Mellanox - manufactured HCA cards. It also provides access to the relevant source - code. Please see the file LICENSE for licensing details. - - ---------------------------------------------------------------------------- - NOTE: - This burning tool should be used only with Mellanox-manufactured - HCA cards. Using it with cards manufactured by other vendors may be harmful - to the cards (due to different configurations). - Using diagnostic tools is usually safe for all HCAs. - ---------------------------------------------------------------------------- - -2) Package Contents - a) mstflint source code. - b) mflash lib. - This lib provides flash access through Mellanox HCAs. - c) mtcr linux utility. - This utility enables the mflash lib to access the HCA - hardware registers. - d) mstregdump utility - This utility dumps hardware registers from Mellanox hardware, - for later analysis by Mellanox - e) mstvpd - This utility dumps on-card VPD - -3) Installation - a) Build the the mstflint utility. - Example: - make - - b) Compilation should end with no error messages. - For example: - >make - g++ -O2 -g -Wall -I. -fno-exceptions -c flint.cpp -o mstflint.o - cc -O2 -g -Wall -I. -fno-exceptions mstflint.o -o mstflint "/usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../lib/libz.a" "/usr/lib/gcc/i486-linux-gnu/4.0.3/libstdc++.a" - cc -O2 -g -Wall -I. mread.c -o mstmread - cc -O2 -g -Wall -I. mwrite.c -o mstmwrite - cc -O2 -g -Wall -I. mstdump.c -o mstregdump - cc -O2 -g -Wall -I. vpd.c -o mstvpd - - c) Executables named mstflint, mstregdump, mstvpd will be generated in the current - directory - This is the mstflint utility that can be used to burn and examine - the HCA on-board flash, mstregdump and mstvpd diagnostic utilities. - You may copy these executables to an arbitrary location. - -4) Requirements: - a) Typically, you will need root privileges for hardware access - - b) For flash update, you will need the binary firmware-image file - supplied by your HCA card vendor. - - If you purchased your card from Mellanox, please use the Mellanox - secured site with your customer login account to download the firmware - binaries, or contact your local field application engineer. - - Or - - If you purchased your card from a vendor other than Mellanox, get a - specific board file from your HCA card vendor and generate the binary - image. - - c) If the mthca driver for the device is not loaded. - You must know the device location on the PCI bus. - For example, to find an InfiniHost HCA car manufactured by Mellanox use: - - /sbin/lspci -d 15b3:5a44 - 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1) - - In this example, 02:00.0 identifies the device in the form bus:dev.fn - -5) Usage (mstflint): - Read mstflint usage. Enter: "./mstflint -h" for short help, or - "./mstflint -hh" for detailed help message. - - Use mstflint to burn a device according to the burning instructions. - You will need to specify the device in the form mthcaX if the driver - has loaded successfully, or bus:dev.fn as explained in section 4c) - above, or in the alternative form /proc/bus/pci/bus/dev.fn . - - If the driver is loaded, unload and re-load it, or reset the device in - some other way, so that the new firmware is re-read from flash. - - Examples: - 1) mstflint -d mthca0 v - This is the recommended method, but it only works - if the mthca driver for the device is loaded. - - Note: it is always safe to try this access method first - - if the driver is not loaded, mstflint will fail with an error - message and you'll be able to retry with another method. - - To verify that the driver for the device mthca0 is loaded, you - can check whether /sys/class/infiniband/mthca0/device exists. - - 2) mstflint -d 02:00.0 v - This is functionally equivalent to access method 1, but less - convenient. This access method works with or without the driver - loaded. - - 3) mstflint -d /proc/bus/pci/02/00.0 v - This method is less safe: verify that driver is not running, - and that no other software accesses the device before using it. - -6) Usage (mstregdump): - Internal register dump is produced on standard output. - Store it in file for analysis in Mellanox. - - Example: - *) mstregdump mthca0 > dumpfile - -7) Usage (mstvpd): - VPD dump is produced on standard output. - A list of keywords to dump can be supplied after the -- flag - to apply an output filter: - - Examples: - 1) mstvpd mthca0 - ID: Lion cub DDR - PN: MHGA28-1T - EC: A3 - SN: MT0551X00740 - V0: PCIe x8 - V1: N/A - YA: R R - RW: - - 2) mstvpd mthca0 -- PN ID - PN: MHGA28-1T - ID: Lion cub DDR - -8) Problem Reporting: - Please collect the following information when reporting issues: - - uname -a - cat /etc/issue - cat /proc/bus/pci/devices - lspci - mstflint -d 02:00.0 v - mstflint -d 02:00.0 q - mstvpd 02:00.0 - - - diff --git a/srpt.README b/srpt.README deleted file mode 100644 index 709200d..0000000 --- a/srpt.README +++ /dev/null @@ -1,168 +0,0 @@ -SCSI RDMA Protocol (SRP) Target driver for Linux -================================================= - -SRP Target driver is designed to work directly on top of OpenFabrics -OFED-1.x software stack (http://www.openfabrics.org) or Infiniband -drivers in Linux kernel tree (kernel.org). It also interfaces with -Generic SCSI target mid-level driver - SCST (http://scst.sourceforge.net) - -By interfacing with SCST driver we are able to work and support a lot IO -modes on real or virtual devices in the backend -1. scst_disk -- interfacing with scsi sub-system to claim and export real -scsi devices ie. disks, hardware raid volumes, tape library as SRP's luns - -2. scst_vdisk -- fileio and blockio modes. This allows you to turn software raid -volumes, LVM volumes, IDE disks, and normal files into SRP's luns (required to -work with latest scst's svn tree) - -3. NULLIO mode will allow you to measure the performance without sending IOs -to *real* devices - - -Prerequisites -------------- -1. First it is required to download and install SCST driver. Please -refer to SCST's Internet page http://scst.sourceforge.net for download -and installation instructions. You can also find SCST's design document, -svn repository development tree and utility tools for basic storage -management - -a. git clone git://git.openfabrics.org/~vu/srpt_inc ~/srpt_inc -b. Checking out scst's svn development tree revision 245 -svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk/scst -r 245 -c. cd scst -d. patch -p0 < ~/srpt_inc/scst_r245.patch -e. make and make install - -2. Download/install IB low-level driver from OFED package or linux kernel tree -a. SRP target is part of OFED-1.3 -or -b. Download and install OFED-1.x (x > 1) with kernel_ib development package rpm -or -c. Built and installed Infiniband driver in Linux "vanilla" kernel tree - from kernel.org - -3. Download SRP target driver from openfabrics.org (needed for 2b, 2c) -git clone git://git.openfabrics.org/~vu/srpt.git ~/srpt - - -Installation ------------- -A. For OFED-1.3 ----------------- -SRP target is part of ofed-1.3. Go through normal installation with srpt enable - - -B. Instruction to included in and built with OFED-1.2.5 development tree ------------------------------------------------------------------------ -1. cd /usr/src/ofa_kernel-1.2.5 or /usr/src/ofa_kernel-1.2.c or -~/ofa_1_2_c_kernel-200708xx-yyyy -2. patch -p1 < ~/srpt_inc/add_srpt_01.patch -3. patch -p1 < ~/srpt_inc/add_srpt_03.patch -4. patch -p1 < ~/srpt_inc/add_srpt_04.patch -5. cp -r ~/srpt drivers/infiniband/ulp/srpt -6. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod - --with-srp-target-mod -7. make and make install - -C. Instruction to be included in and built with OFED-1.2 development tree --------------------------------------------------------------------------- -1. cd /usr/src/ofa_kernel-1.2 or /usr/src/ofa_kernel-1.2 or -~/ofa_1_2_kernel-200708xx-yyyy -2. patch -p1 < ~/srpt_inc/add_srpt_01.patch -3. patch -p1 < ~/srpt_inc/add_srpt_03.patch -4. patch -p1 < ~/srpt_inc/add_srpt_04.patch -5. cp -r ~/srpt drivers/infiniband/ulp/srpt -6. patch -p1 < ~/srpt_inc/add_srpt_ofed_1_2.patch -7. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod - --with-srp-target-mod -8. make and make install - -D. Instruction to be included in and built with "vanilla" kernel development tree ----------------------------------------------------------------------------------- -1. cd /usr/src/linux-2.6.1x -2. patch -p1 < ~/srpt_inc/add_srpt_03.patch -3. cp -r ~/srpt drivers/infiniband/ulp/srpt -4. configure and build SRP target module driver as normal - - -How-to run ------------ -A. On srp target machine -1. Please refer to SCST's README for loading scsi_tgt driver and its -dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...) - -Example 1: working with real back-end scsi disks -a. modprobe scsi_tgt -b. modprobe scst_disk -c. cat /proc/scsi_tgt/scsi_tgt - -ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt -Device (host:ch:id:lun or name) Device handler -0:0:0:0 dev_disk -4:0:0:0 dev_disk -5:0:0:0 dev_disk -6:0:0:0 dev_disk -7:0:0:0 dev_disk - -Now you want to exclude the first scsi disk and expose the last 4 scsi disks as -IB/SRP luns for I/O -echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices -echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices -echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices -echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices - -Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file) -a. modprobe scsi_tgt -b. modprobe scst_vdisk -c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk -d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk -e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices -f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices - -Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0) -a. modprobe scsi_tgt -b. modprobe scst_vdisk -c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk -f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices -g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices -h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices - -2. modprobe ib_srpt - - -B. On initiator machines you can manualy do the following steps: -1. modprobe ib_srp -2. ipsrpdm -c (to discover new SRP target) -3. echo > /sys/class/infiniband_srp/srp-mthca0-1/add_target -4. fdisk -l (will show new discovered scsi disks) - -Example: -Assume that you use port 1 of first HCA in the system ie. mthca0 - -[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0 -id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, -dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 -[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4, -dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 > -/sys/class/infiniband_srp/srp-mthca0-1/add_target - -OR - -+ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon -automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes -+ To set up and use high availability feature you need dm-multipath driver -and multipath tool -+ Please refer to OFED-1.x SRP's user manual for more in-details instructions -on how-to enable/use HA feature - - -TO DO ------- -+ Stress test and stabilize the code -+ Performance tuning -+ Integrate and test with SCST's admin/management tools for LUNs masking/sharing - among initiator hosts -+ Handle QP, shared-RQ, and asynchronous events