--- /dev/null
+Mellanox Technologies, Inc. - www.mellanox.com
+***********************************************
+
+mstflint: A BURNING FIRMWARE APPLICATION
+----------------------------------------
+
+1) Overview
+ This package contains a burning tool, and diagnostic tools, for Mellanox
+ manufactured HCA cards. It also provides access to the relevant source
+ code. Please see the file LICENSE for licensing details.
+
+ ----------------------------------------------------------------------------
+ NOTE:
+ This burning tool should be used only with Mellanox-manufactured
+ HCA cards. Using it with cards manufactured by other vendors may be harmful
+ to the cards (due to different configurations).
+ Using diagnostic tools is usually safe for all HCAs.
+ ----------------------------------------------------------------------------
+
+2) Package Contents
+ a) mstflint source code.
+ b) mflash lib.
+ This lib provides flash access through Mellanox HCAs.
+ c) mtcr linux utility.
+ This utility enables the mflash lib to access the HCA
+ hardware registers.
+ d) mstregdump utility
+ This utility dumps hardware registers from Mellanox hardware,
+ for later analysis by Mellanox
+ e) mstvpd
+ This utility dumps on-card VPD
+
+3) Installation
+ a) Build the the mstflint utility.
+ Example:
+ make
+
+ b) Compilation should end with no error messages.
+ For example:
+ >make
+ g++ -O2 -g -Wall -I. -fno-exceptions -c flint.cpp -o mstflint.o
+ cc -O2 -g -Wall -I. -fno-exceptions mstflint.o -o mstflint "/usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../lib/libz.a" "/usr/lib/gcc/i486-linux-gnu/4.0.3/libstdc++.a"
+ cc -O2 -g -Wall -I. mread.c -o mstmread
+ cc -O2 -g -Wall -I. mwrite.c -o mstmwrite
+ cc -O2 -g -Wall -I. mstdump.c -o mstregdump
+ cc -O2 -g -Wall -I. vpd.c -o mstvpd
+
+ c) Executables named mstflint, mstregdump, mstvpd will be generated in the current
+ directory
+ This is the mstflint utility that can be used to burn and examine
+ the HCA on-board flash, mstregdump and mstvpd diagnostic utilities.
+ You may copy these executables to an arbitrary location.
+
+4) Requirements:
+ a) Typically, you will need root privileges for hardware access
+
+ b) For flash update, you will need the binary firmware-image file
+ supplied by your HCA card vendor.
+
+ If you purchased your card from Mellanox, please use the Mellanox
+ secured site with your customer login account to download the firmware
+ binaries, or contact your local field application engineer.
+
+ Or
+
+ If you purchased your card from a vendor other than Mellanox, get a
+ specific board file from your HCA card vendor and generate the binary
+ image.
+
+ c) If the mthca driver for the device is not loaded.
+ You must know the device location on the PCI bus.
+ For example, to find an InfiniHost HCA car manufactured by Mellanox use:
+
+ /sbin/lspci -d 15b3:5a44
+ 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1)
+
+ In this example, 02:00.0 identifies the device in the form bus:dev.fn
+
+5) Usage (mstflint):
+ Read mstflint usage. Enter: "./mstflint -h" for short help, or
+ "./mstflint -hh" for detailed help message.
+
+ Use mstflint to burn a device according to the burning instructions.
+ You will need to specify the device in the form mthcaX if the driver
+ has loaded successfully, or bus:dev.fn as explained in section 4c)
+ above, or in the alternative form /proc/bus/pci/bus/dev.fn .
+
+ If the driver is loaded, unload and re-load it, or reset the device in
+ some other way, so that the new firmware is re-read from flash.
+
+ Examples:
+ 1) mstflint -d mthca0 v
+ This is the recommended method, but it only works
+ if the mthca driver for the device is loaded.
+
+ Note: it is always safe to try this access method first -
+ if the driver is not loaded, mstflint will fail with an error
+ message and you'll be able to retry with another method.
+
+ To verify that the driver for the device mthca0 is loaded, you
+ can check whether /sys/class/infiniband/mthca0/device exists.
+
+ 2) mstflint -d 02:00.0 v
+ This is functionally equivalent to access method 1, but less
+ convenient. This access method works with or without the driver
+ loaded.
+
+ 3) mstflint -d /proc/bus/pci/02/00.0 v
+ This method is less safe: verify that driver is not running,
+ and that no other software accesses the device before using it.
+
+6) Usage (mstregdump):
+ Internal register dump is produced on standard output.
+ Store it in file for analysis in Mellanox.
+
+ Example:
+ *) mstregdump mthca0 > dumpfile
+
+7) Usage (mstvpd):
+ VPD dump is produced on standard output.
+ A list of keywords to dump can be supplied after the -- flag
+ to apply an output filter:
+
+ Examples:
+ 1) mstvpd mthca0
+ ID: Lion cub DDR
+ PN: MHGA28-1T
+ EC: A3
+ SN: MT0551X00740
+ V0: PCIe x8
+ V1: N/A
+ YA: R R
+ RW:
+
+ 2) mstvpd mthca0 -- PN ID
+ PN: MHGA28-1T
+ ID: Lion cub DDR
+
+8) Problem Reporting:
+ Please collect the following information when reporting issues:
+
+ uname -a
+ cat /etc/issue
+ cat /proc/bus/pci/devices
+ lspci
+ mstflint -d 02:00.0 v
+ mstflint -d 02:00.0 q
+ mstvpd 02:00.0
+
+
+
--- /dev/null
+SCSI RDMA Protocol (SRP) Target driver for Linux
+=================================================
+
+SRP Target driver is designed to work directly on top of OpenFabrics
+OFED-1.x software stack (http://www.openfabrics.org) or Infiniband
+drivers in Linux kernel tree (kernel.org). It also interfaces with
+Generic SCSI target mid-level driver - SCST (http://scst.sourceforge.net)
+
+By interfacing with SCST driver we are able to work and support a lot IO
+modes on real or virtual devices in the backend
+1. scst_disk -- interfacing with scsi sub-system to claim and export real
+scsi devices ie. disks, hardware raid volumes, tape library as SRP's luns
+
+2. scst_vdisk -- fileio and blockio modes. This allows you to turn software raid
+volumes, LVM volumes, IDE disks, and normal files into SRP's luns (required to
+work with latest scst's svn tree)
+
+3. NULLIO mode will allow you to measure the performance without sending IOs
+to *real* devices
+
+
+Prerequisites
+-------------
+1. First it is required to download and install SCST driver. Please
+refer to SCST's Internet page http://scst.sourceforge.net for download
+and installation instructions. You can also find SCST's design document,
+svn repository development tree and utility tools for basic storage
+management
+
+a. git clone git://git.openfabrics.org/~vu/srpt_inc ~/srpt_inc
+b. Checking out scst's svn development tree revision 245
+svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk/scst -r 245
+c. cd scst
+d. patch -p0 < ~/srpt_inc/scst_r245.patch
+e. make and make install
+
+2. Download/install IB low-level driver from OFED package or linux kernel tree
+a. SRP target is part of OFED-1.3
+or
+b. Download and install OFED-1.x (x > 1) with kernel_ib development package rpm
+or
+c. Built and installed Infiniband driver in Linux "vanilla" kernel tree
+ from kernel.org
+
+3. Download SRP target driver from openfabrics.org (needed for 2b, 2c)
+git clone git://git.openfabrics.org/~vu/srpt.git ~/srpt
+
+
+Installation
+------------
+A. For OFED-1.3
+----------------
+SRP target is part of ofed-1.3. Go through normal installation with srpt enable
+
+
+B. Instruction to included in and built with OFED-1.2.5 development tree
+-----------------------------------------------------------------------
+1. cd /usr/src/ofa_kernel-1.2.5 or /usr/src/ofa_kernel-1.2.c or
+~/ofa_1_2_c_kernel-200708xx-yyyy
+2. patch -p1 < ~/srpt_inc/add_srpt_01.patch
+3. patch -p1 < ~/srpt_inc/add_srpt_03.patch
+4. patch -p1 < ~/srpt_inc/add_srpt_04.patch
+5. cp -r ~/srpt drivers/infiniband/ulp/srpt
+6. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod
+ --with-srp-target-mod <and any other modules/drivers as needed>
+7. make and make install
+
+C. Instruction to be included in and built with OFED-1.2 development tree
+--------------------------------------------------------------------------
+1. cd /usr/src/ofa_kernel-1.2 or /usr/src/ofa_kernel-1.2 or
+~/ofa_1_2_kernel-200708xx-yyyy
+2. patch -p1 < ~/srpt_inc/add_srpt_01.patch
+3. patch -p1 < ~/srpt_inc/add_srpt_03.patch
+4. patch -p1 < ~/srpt_inc/add_srpt_04.patch
+5. cp -r ~/srpt drivers/infiniband/ulp/srpt
+6. patch -p1 < ~/srpt_inc/add_srpt_ofed_1_2.patch
+7. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod
+ --with-srp-target-mod <and any other modules/drivers as needed>
+8. make and make install
+
+D. Instruction to be included in and built with "vanilla" kernel development tree
+----------------------------------------------------------------------------------
+1. cd /usr/src/linux-2.6.1x
+2. patch -p1 < ~/srpt_inc/add_srpt_03.patch
+3. cp -r ~/srpt drivers/infiniband/ulp/srpt
+4. configure and build SRP target module driver as normal
+
+
+How-to run
+-----------
+A. On srp target machine
+1. Please refer to SCST's README for loading scsi_tgt driver and its
+dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...)
+
+Example 1: working with real back-end scsi disks
+a. modprobe scsi_tgt
+b. modprobe scst_disk
+c. cat /proc/scsi_tgt/scsi_tgt
+
+ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt
+Device (host:ch:id:lun or name) Device handler
+0:0:0:0 dev_disk
+4:0:0:0 dev_disk
+5:0:0:0 dev_disk
+6:0:0:0 dev_disk
+7:0:0:0 dev_disk
+
+Now you want to exclude the first scsi disk and expose the last 4 scsi disks as
+IB/SRP luns for I/O
+echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices
+echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices
+echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices
+echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices
+
+Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file)
+a. modprobe scsi_tgt
+b. modprobe scst_vdisk
+c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk
+d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk
+e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
+f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
+
+Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0)
+a. modprobe scsi_tgt
+b. modprobe scst_vdisk
+c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
+d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
+e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
+f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
+g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
+h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
+
+2. modprobe ib_srpt
+
+
+B. On initiator machines you can manualy do the following steps:
+1. modprobe ib_srp
+2. ipsrpdm -c (to discover new SRP target)
+3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target
+4. fdisk -l (will show new discovered scsi disks)
+
+Example:
+Assume that you use port 1 of first HCA in the system ie. mthca0
+
+[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0
+id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
+dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4
+[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
+dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 >
+/sys/class/infiniband_srp/srp-mthca0-1/add_target
+
+OR
+
++ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon
+automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes
++ To set up and use high availability feature you need dm-multipath driver
+and multipath tool
++ Please refer to OFED-1.x SRP's user manual for more in-details instructions
+on how-to enable/use HA feature
+
+
+TO DO
+------
++ Stress test and stabilize the code
++ Performance tuning
++ Integrate and test with SCST's admin/management tools for LUNs masking/sharing
+ among initiator hosts
++ Handle QP, shared-RQ, and asynchronous events
+++ /dev/null
-This is a technology preview release of the QLogic VNIC driver on OFED 1.3.
-This driver is currently supported on Intel x86 32 and 64 bit machines with
-Mellanox HCAs. Supported OS are RHEL 4 Update 3, RHEL 4 Update 4, SLES 10
-and the vanilla 2.6.19 kernel.
-
-The VNIC driver in conjunction with the QLogic Virtual Ethernet I/O gateway
-(VEx) provides Ethernet interfaces on a host with IB HCA(s) without the need
-for any physical Ethernet NIC.
-
-This file describes the use of the VNIC ULP service on an OFED stack
-and covers the following points:
-
-A) Creating VNIC interfaces
-B) Discovering VEx IOCs present on the fabric using ibvexdm
-C) Starting the VNIC driver and the VNIC interfaces
-D) Assigning IP addresses etc for the VNIC interfaces
-E) Information about the VNIC interfaces
-F) Deleting a specific VNIC interface
-
-A) Creating VNIC interfaces
-
-The VNIC interfaces can be created with the help of
-a configuration file which must be placed at /etc/infiniband/qlogic_vnic.cfg.
-
-Please take a look at qlogic_vnic.cfg.sample file (available as part of the
-documentation) to see how VNIC configuration files are written. You can use
-this configuration file as the basis for creating a VNIC configuration file by
-copying it to /etc/infiniband/qlogic_vnic.cfg. Of course you will have to
-replace the IOCGUID, IOCSTRING values etc in the sample configuration file
-with those of the VEx IOCs present on your fabric.
-
-[As back ward compatibility for the QLogic Infiniserv stack, if the
-/etc/infiniband/qlogic_vnic.cfg file is not present, the VNIC service will
-look for the /etc/sysconfig/ics_inic.cfg configuration file]
-
-B) Discovering VEx IOCs present on the fabric using ibvexdm
-
-For writing the configuration file, you will need information
-about the VEx IOCs present on the fabric like their IOCGUID,
-IOCSTRING etc. The ibvexdm tool should be used to get this
-information.
-
-When ibvexdm is executed without any options, it displays
-detailed information about all the VEx IOCs present on the fabric:
-
-# ibvexdm
-IO Unit Info:
- port LID: 0003
- port GID: fe8000000000000000066a0258000001
- change ID: 0009
- max controllers: 0x03
-
-
- controller[ 1]
- GUID: 00066a0130000001
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: Chassis 0x00066A00010003F2, Slot 1, IOC 1
- service entries: 2
- service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
- service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
-
- controller[ 2]
- GUID: 00066a0230000001
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: Chassis 0x00066A00010003F2, Slot 1, IOC 2
- service entries: 2
- service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
- service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
-
- controller[ 3]
- GUID: 00066a0330000001
- vendor ID: 00066a
- device ID: 000030
- IO class : 2000
- ID: Chassis 0x00066A00010003F2, Slot 1, IOC 3
- service entries: 2
- service[ 0]: 1000066a00000003 / InfiniNIC.InfiniConSys.Control:03
- service[ 1]: 1000066a00000103 / InfiniNIC.InfiniConSys.Data:03
-
-
-When ibvexdm is run with -e option, it reports the IOCGUID information
-and with -s option it reports the IOCSTRING information for the VEx IOCs
-present on the fabric.
-
-# ibvexdm -e
-ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
-ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
-ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
-
-#ibvexdm -s
-"Chassis 0x00066A00010003F2, Slot 1, IOC 1"
-"Chassis 0x00066A00010003F2, Slot 1, IOC 2"
-"Chassis 0x00066A00010003F2, Slot 1, IOC 3"
-
-#ibvexdm -es
-ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 1"
-ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 2"
-ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 3"
-
-C) Starting the VNIC driver and the VNIC interfaces
-
-Once you have created a configuration file, you can start the VNIC driver
-and create the VNIC interfaces specified in the configuration file with:
-#/etc/init.d/openibd start
-
-
-The VNIC driver is started by default as a part of openibd service.
-(If openibd is configured to start on boot VNIC driver will also start on boot)
-If you do not want to start the VNIC driver as a part of openibd service, set
-VNIC_LOAD=no in /etc/infiniband/openib.conf
-
-You can stop the VNIC driver and openibd services with
-#/etc/init.d/openibd stop
-
-You can restart the VNIC driver and openibd service using
-#/etc/init.d/openibd restart
-
-After starting openibd service you can independently control the VNIC driver using ql_vnic service
-To stop the VNIC driver you can use
-#/etc/init.d/ql_vnic stop
-
-If you make changes to the VNIC configuration file and want the changes to be
-reflected, you can restart the VNIC service independently, using
-#/etc/init.d/ql_vnic restart
-
-Note that it is required to start openibd service before using ql_vnic service
-
-D) Assigning IP addresses etc for the VNIC interfaces
-
-This can be done with ifconfig or by setting up the ifcfg-XXX (ifcfg-veth0 for
-an interface named veth0 etc) network files for the corresponding VNIC interfaces.
-
-E) Information about the VNIC interfaces
-
-Information about the created VNIC interfaces can be obtained from
-/sys/class/infiniband_vnic/interfaces/. A directory is created
-for each interface under this directory.
-
-The directory for each interface contains information about the interface
-and the primary and secondary connections.
-
-F) Deleting a specific VNIC interface
-
-VNIC interfaces can be deleted by writing the name of the interface to
-the /sys/class/infiniband_vnic/interfaces/delete_vnic file.
-
-For example to delete interface veth0
-
-echo -n veth0 > /sys/class/infiniband_vnic/interfaces/delete_vnic
-
-
-
--- /dev/null
+This is a technology preview release of the QLogic VNIC driver on OFED 1.3.
+This driver is currently supported on Intel x86 32 and 64 bit machines with
+Mellanox HCAs. Supported OS are RHEL 4 Update 3, RHEL 4 Update 4, SLES 10
+and the vanilla 2.6.19 kernel.
+
+The VNIC driver in conjunction with the QLogic Virtual Ethernet I/O gateway
+(VEx) provides Ethernet interfaces on a host with IB HCA(s) without the need
+for any physical Ethernet NIC.
+
+This file describes the use of the VNIC ULP service on an OFED stack
+and covers the following points:
+
+A) Creating VNIC interfaces
+B) Discovering VEx IOCs present on the fabric using ibvexdm
+C) Starting the VNIC driver and the VNIC interfaces
+D) Assigning IP addresses etc for the VNIC interfaces
+E) Information about the VNIC interfaces
+F) Deleting a specific VNIC interface
+
+A) Creating VNIC interfaces
+
+The VNIC interfaces can be created with the help of
+a configuration file which must be placed at /etc/infiniband/qlogic_vnic.cfg.
+
+Please take a look at qlogic_vnic.cfg.sample file (available as part of the
+documentation) to see how VNIC configuration files are written. You can use
+this configuration file as the basis for creating a VNIC configuration file by
+copying it to /etc/infiniband/qlogic_vnic.cfg. Of course you will have to
+replace the IOCGUID, IOCSTRING values etc in the sample configuration file
+with those of the VEx IOCs present on your fabric.
+
+[As back ward compatibility for the QLogic Infiniserv stack, if the
+/etc/infiniband/qlogic_vnic.cfg file is not present, the VNIC service will
+look for the /etc/sysconfig/ics_inic.cfg configuration file]
+
+B) Discovering VEx IOCs present on the fabric using ibvexdm
+
+For writing the configuration file, you will need information
+about the VEx IOCs present on the fabric like their IOCGUID,
+IOCSTRING etc. The ibvexdm tool should be used to get this
+information.
+
+When ibvexdm is executed without any options, it displays
+detailed information about all the VEx IOCs present on the fabric:
+
+# ibvexdm
+IO Unit Info:
+ port LID: 0003
+ port GID: fe8000000000000000066a0258000001
+ change ID: 0009
+ max controllers: 0x03
+
+
+ controller[ 1]
+ GUID: 00066a0130000001
+ vendor ID: 00066a
+ device ID: 000030
+ IO class : 2000
+ ID: Chassis 0x00066A00010003F2, Slot 1, IOC 1
+ service entries: 2
+ service[ 0]: 1000066a00000001 / InfiniNIC.InfiniConSys.Control:01
+ service[ 1]: 1000066a00000101 / InfiniNIC.InfiniConSys.Data:01
+
+ controller[ 2]
+ GUID: 00066a0230000001
+ vendor ID: 00066a
+ device ID: 000030
+ IO class : 2000
+ ID: Chassis 0x00066A00010003F2, Slot 1, IOC 2
+ service entries: 2
+ service[ 0]: 1000066a00000002 / InfiniNIC.InfiniConSys.Control:02
+ service[ 1]: 1000066a00000102 / InfiniNIC.InfiniConSys.Data:02
+
+ controller[ 3]
+ GUID: 00066a0330000001
+ vendor ID: 00066a
+ device ID: 000030
+ IO class : 2000
+ ID: Chassis 0x00066A00010003F2, Slot 1, IOC 3
+ service entries: 2
+ service[ 0]: 1000066a00000003 / InfiniNIC.InfiniConSys.Control:03
+ service[ 1]: 1000066a00000103 / InfiniNIC.InfiniConSys.Data:03
+
+
+When ibvexdm is run with -e option, it reports the IOCGUID information
+and with -s option it reports the IOCSTRING information for the VEx IOCs
+present on the fabric.
+
+# ibvexdm -e
+ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
+ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
+ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff
+
+#ibvexdm -s
+"Chassis 0x00066A00010003F2, Slot 1, IOC 1"
+"Chassis 0x00066A00010003F2, Slot 1, IOC 2"
+"Chassis 0x00066A00010003F2, Slot 1, IOC 3"
+
+#ibvexdm -es
+ioc_guid=00066a0130000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 1"
+ioc_guid=00066a0230000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 2"
+ioc_guid=00066a0330000001,dgid=fe8000000000000000066a0258000001,pkey=ffff,"Chassis 0x00066A00010003F2, Slot 1, IOC 3"
+
+C) Starting the VNIC driver and the VNIC interfaces
+
+Once you have created a configuration file, you can start the VNIC driver
+and create the VNIC interfaces specified in the configuration file with:
+#/etc/init.d/openibd start
+
+
+The VNIC driver is started by default as a part of openibd service.
+(If openibd is configured to start on boot VNIC driver will also start on boot)
+If you do not want to start the VNIC driver as a part of openibd service, set
+VNIC_LOAD=no in /etc/infiniband/openib.conf
+
+You can stop the VNIC driver and openibd services with
+#/etc/init.d/openibd stop
+
+You can restart the VNIC driver and openibd service using
+#/etc/init.d/openibd restart
+
+After starting openibd service you can independently control the VNIC driver using ql_vnic service
+To stop the VNIC driver you can use
+#/etc/init.d/ql_vnic stop
+
+If you make changes to the VNIC configuration file and want the changes to be
+reflected, you can restart the VNIC service independently, using
+#/etc/init.d/ql_vnic restart
+
+Note that it is required to start openibd service before using ql_vnic service
+
+D) Assigning IP addresses etc for the VNIC interfaces
+
+This can be done with ifconfig or by setting up the ifcfg-XXX (ifcfg-veth0 for
+an interface named veth0 etc) network files for the corresponding VNIC interfaces.
+
+E) Information about the VNIC interfaces
+
+Information about the created VNIC interfaces can be obtained from
+/sys/class/infiniband_vnic/interfaces/. A directory is created
+for each interface under this directory.
+
+The directory for each interface contains information about the interface
+and the primary and secondary connections.
+
+F) Deleting a specific VNIC interface
+
+VNIC interfaces can be deleted by writing the name of the interface to
+the /sys/class/infiniband_vnic/interfaces/delete_vnic file.
+
+For example to delete interface veth0
+
+echo -n veth0 > /sys/class/infiniband_vnic/interfaces/delete_vnic
+
+
+
+++ /dev/null
-Mellanox Technologies, Inc. - www.mellanox.com
-***********************************************
-
-mstflint: A BURNING FIRMWARE APPLICATION
-----------------------------------------
-
-1) Overview
- This package contains a burning tool, and diagnostic tools, for Mellanox
- manufactured HCA cards. It also provides access to the relevant source
- code. Please see the file LICENSE for licensing details.
-
- ----------------------------------------------------------------------------
- NOTE:
- This burning tool should be used only with Mellanox-manufactured
- HCA cards. Using it with cards manufactured by other vendors may be harmful
- to the cards (due to different configurations).
- Using diagnostic tools is usually safe for all HCAs.
- ----------------------------------------------------------------------------
-
-2) Package Contents
- a) mstflint source code.
- b) mflash lib.
- This lib provides flash access through Mellanox HCAs.
- c) mtcr linux utility.
- This utility enables the mflash lib to access the HCA
- hardware registers.
- d) mstregdump utility
- This utility dumps hardware registers from Mellanox hardware,
- for later analysis by Mellanox
- e) mstvpd
- This utility dumps on-card VPD
-
-3) Installation
- a) Build the the mstflint utility.
- Example:
- make
-
- b) Compilation should end with no error messages.
- For example:
- >make
- g++ -O2 -g -Wall -I. -fno-exceptions -c flint.cpp -o mstflint.o
- cc -O2 -g -Wall -I. -fno-exceptions mstflint.o -o mstflint "/usr/lib/gcc/i486-linux-gnu/4.0.3/../../../../lib/libz.a" "/usr/lib/gcc/i486-linux-gnu/4.0.3/libstdc++.a"
- cc -O2 -g -Wall -I. mread.c -o mstmread
- cc -O2 -g -Wall -I. mwrite.c -o mstmwrite
- cc -O2 -g -Wall -I. mstdump.c -o mstregdump
- cc -O2 -g -Wall -I. vpd.c -o mstvpd
-
- c) Executables named mstflint, mstregdump, mstvpd will be generated in the current
- directory
- This is the mstflint utility that can be used to burn and examine
- the HCA on-board flash, mstregdump and mstvpd diagnostic utilities.
- You may copy these executables to an arbitrary location.
-
-4) Requirements:
- a) Typically, you will need root privileges for hardware access
-
- b) For flash update, you will need the binary firmware-image file
- supplied by your HCA card vendor.
-
- If you purchased your card from Mellanox, please use the Mellanox
- secured site with your customer login account to download the firmware
- binaries, or contact your local field application engineer.
-
- Or
-
- If you purchased your card from a vendor other than Mellanox, get a
- specific board file from your HCA card vendor and generate the binary
- image.
-
- c) If the mthca driver for the device is not loaded.
- You must know the device location on the PCI bus.
- For example, to find an InfiniHost HCA car manufactured by Mellanox use:
-
- /sbin/lspci -d 15b3:5a44
- 02:00.0 InfiniBand: Mellanox Technology: Unknown device 5a44 (rev a1)
-
- In this example, 02:00.0 identifies the device in the form bus:dev.fn
-
-5) Usage (mstflint):
- Read mstflint usage. Enter: "./mstflint -h" for short help, or
- "./mstflint -hh" for detailed help message.
-
- Use mstflint to burn a device according to the burning instructions.
- You will need to specify the device in the form mthcaX if the driver
- has loaded successfully, or bus:dev.fn as explained in section 4c)
- above, or in the alternative form /proc/bus/pci/bus/dev.fn .
-
- If the driver is loaded, unload and re-load it, or reset the device in
- some other way, so that the new firmware is re-read from flash.
-
- Examples:
- 1) mstflint -d mthca0 v
- This is the recommended method, but it only works
- if the mthca driver for the device is loaded.
-
- Note: it is always safe to try this access method first -
- if the driver is not loaded, mstflint will fail with an error
- message and you'll be able to retry with another method.
-
- To verify that the driver for the device mthca0 is loaded, you
- can check whether /sys/class/infiniband/mthca0/device exists.
-
- 2) mstflint -d 02:00.0 v
- This is functionally equivalent to access method 1, but less
- convenient. This access method works with or without the driver
- loaded.
-
- 3) mstflint -d /proc/bus/pci/02/00.0 v
- This method is less safe: verify that driver is not running,
- and that no other software accesses the device before using it.
-
-6) Usage (mstregdump):
- Internal register dump is produced on standard output.
- Store it in file for analysis in Mellanox.
-
- Example:
- *) mstregdump mthca0 > dumpfile
-
-7) Usage (mstvpd):
- VPD dump is produced on standard output.
- A list of keywords to dump can be supplied after the -- flag
- to apply an output filter:
-
- Examples:
- 1) mstvpd mthca0
- ID: Lion cub DDR
- PN: MHGA28-1T
- EC: A3
- SN: MT0551X00740
- V0: PCIe x8
- V1: N/A
- YA: R R
- RW:
-
- 2) mstvpd mthca0 -- PN ID
- PN: MHGA28-1T
- ID: Lion cub DDR
-
-8) Problem Reporting:
- Please collect the following information when reporting issues:
-
- uname -a
- cat /etc/issue
- cat /proc/bus/pci/devices
- lspci
- mstflint -d 02:00.0 v
- mstflint -d 02:00.0 q
- mstvpd 02:00.0
-
-
-
+++ /dev/null
-SCSI RDMA Protocol (SRP) Target driver for Linux
-=================================================
-
-SRP Target driver is designed to work directly on top of OpenFabrics
-OFED-1.x software stack (http://www.openfabrics.org) or Infiniband
-drivers in Linux kernel tree (kernel.org). It also interfaces with
-Generic SCSI target mid-level driver - SCST (http://scst.sourceforge.net)
-
-By interfacing with SCST driver we are able to work and support a lot IO
-modes on real or virtual devices in the backend
-1. scst_disk -- interfacing with scsi sub-system to claim and export real
-scsi devices ie. disks, hardware raid volumes, tape library as SRP's luns
-
-2. scst_vdisk -- fileio and blockio modes. This allows you to turn software raid
-volumes, LVM volumes, IDE disks, and normal files into SRP's luns (required to
-work with latest scst's svn tree)
-
-3. NULLIO mode will allow you to measure the performance without sending IOs
-to *real* devices
-
-
-Prerequisites
--------------
-1. First it is required to download and install SCST driver. Please
-refer to SCST's Internet page http://scst.sourceforge.net for download
-and installation instructions. You can also find SCST's design document,
-svn repository development tree and utility tools for basic storage
-management
-
-a. git clone git://git.openfabrics.org/~vu/srpt_inc ~/srpt_inc
-b. Checking out scst's svn development tree revision 245
-svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk/scst -r 245
-c. cd scst
-d. patch -p0 < ~/srpt_inc/scst_r245.patch
-e. make and make install
-
-2. Download/install IB low-level driver from OFED package or linux kernel tree
-a. SRP target is part of OFED-1.3
-or
-b. Download and install OFED-1.x (x > 1) with kernel_ib development package rpm
-or
-c. Built and installed Infiniband driver in Linux "vanilla" kernel tree
- from kernel.org
-
-3. Download SRP target driver from openfabrics.org (needed for 2b, 2c)
-git clone git://git.openfabrics.org/~vu/srpt.git ~/srpt
-
-
-Installation
-------------
-A. For OFED-1.3
-----------------
-SRP target is part of ofed-1.3. Go through normal installation with srpt enable
-
-
-B. Instruction to included in and built with OFED-1.2.5 development tree
------------------------------------------------------------------------
-1. cd /usr/src/ofa_kernel-1.2.5 or /usr/src/ofa_kernel-1.2.c or
-~/ofa_1_2_c_kernel-200708xx-yyyy
-2. patch -p1 < ~/srpt_inc/add_srpt_01.patch
-3. patch -p1 < ~/srpt_inc/add_srpt_03.patch
-4. patch -p1 < ~/srpt_inc/add_srpt_04.patch
-5. cp -r ~/srpt drivers/infiniband/ulp/srpt
-6. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod
- --with-srp-target-mod <and any other modules/drivers as needed>
-7. make and make install
-
-C. Instruction to be included in and built with OFED-1.2 development tree
---------------------------------------------------------------------------
-1. cd /usr/src/ofa_kernel-1.2 or /usr/src/ofa_kernel-1.2 or
-~/ofa_1_2_kernel-200708xx-yyyy
-2. patch -p1 < ~/srpt_inc/add_srpt_01.patch
-3. patch -p1 < ~/srpt_inc/add_srpt_03.patch
-4. patch -p1 < ~/srpt_inc/add_srpt_04.patch
-5. cp -r ~/srpt drivers/infiniband/ulp/srpt
-6. patch -p1 < ~/srpt_inc/add_srpt_ofed_1_2.patch
-7. ./configure --with-core-mod --with-mthca-mod --without-mthca_debug-mod
- --with-srp-target-mod <and any other modules/drivers as needed>
-8. make and make install
-
-D. Instruction to be included in and built with "vanilla" kernel development tree
-----------------------------------------------------------------------------------
-1. cd /usr/src/linux-2.6.1x
-2. patch -p1 < ~/srpt_inc/add_srpt_03.patch
-3. cp -r ~/srpt drivers/infiniband/ulp/srpt
-4. configure and build SRP target module driver as normal
-
-
-How-to run
------------
-A. On srp target machine
-1. Please refer to SCST's README for loading scsi_tgt driver and its
-dev_handlers drivers (scst_disk, scst_vdisk block or file IO mode, nullio, ...)
-
-Example 1: working with real back-end scsi disks
-a. modprobe scsi_tgt
-b. modprobe scst_disk
-c. cat /proc/scsi_tgt/scsi_tgt
-
-ibstor00:~ # cat /proc/scsi_tgt/scsi_tgt
-Device (host:ch:id:lun or name) Device handler
-0:0:0:0 dev_disk
-4:0:0:0 dev_disk
-5:0:0:0 dev_disk
-6:0:0:0 dev_disk
-7:0:0:0 dev_disk
-
-Now you want to exclude the first scsi disk and expose the last 4 scsi disks as
-IB/SRP luns for I/O
-echo "add 4:0:0:0 0" >/proc/scsi_tgt/groups/Default/devices
-echo "add 5:0:0:0 1" >/proc/scsi_tgt/groups/Default/devices
-echo "add 6:0:0:0 2" >/proc/scsi_tgt/groups/Default/devices
-echo "add 7:0:0:0 3" >/proc/scsi_tgt/groups/Default/devices
-
-Example 2: working with VDISK FILEIO mode (using md0 device and file 10G-file)
-a. modprobe scsi_tgt
-b. modprobe scst_vdisk
-c. echo "open vdisk0 /dev/md0" > /proc/scsi_tgt/vdisk/vdisk
-d. echo "open vdisk1 /10G-file" > /proc/scsi_tgt/vdisk/vdisk
-e. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
-f. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
-
-Example 3: working with VDISK BLOCKIO mode (using md0 device, sda, and cciss/c1d0)
-a. modprobe scsi_tgt
-b. modprobe scst_vdisk
-c. echo "open vdisk0 /dev/md0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-d. echo "open vdisk1 /dev/sda BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-e. echo "open vdisk2 /dev/cciss/c1d0 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk
-f. echo "add vdisk0 0" >/proc/scsi_tgt/groups/Default/devices
-g. echo "add vdisk1 1" >/proc/scsi_tgt/groups/Default/devices
-h. echo "add vdisk2 2" >/proc/scsi_tgt/groups/Default/devices
-
-2. modprobe ib_srpt
-
-
-B. On initiator machines you can manualy do the following steps:
-1. modprobe ib_srp
-2. ipsrpdm -c (to discover new SRP target)
-3. echo <new target info> > /sys/class/infiniband_srp/srp-mthca0-1/add_target
-4. fdisk -l (will show new discovered scsi disks)
-
-Example:
-Assume that you use port 1 of first HCA in the system ie. mthca0
-
-[root@lab104 ~]# ibsrpdm -c -d /dev/infiniband/umad0
-id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
-dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4
-[root@lab104 ~]# echo id_ext=0002c90200226cf4,ioc_guid=0002c90200226cf4,
-dgid=fe800000000000000002c90200226cf5,pkey=ffff,service_id=0002c90200226cf4 >
-/sys/class/infiniband_srp/srp-mthca0-1/add_target
-
-OR
-
-+ You can edit /etc/infiniband/openib.conf to load srp driver and srp HA daemon
-automatically ie. set SRP_LOAD=yes, and SRPHA_ENABLE=yes
-+ To set up and use high availability feature you need dm-multipath driver
-and multipath tool
-+ Please refer to OFED-1.x SRP's user manual for more in-details instructions
-on how-to enable/use HA feature
-
-
-TO DO
-------
-+ Stress test and stabilize the code
-+ Performance tuning
-+ Integrate and test with SCST's admin/management tools for LUNs masking/sharing
- among initiator hosts
-+ Handle QP, shared-RQ, and asynchronous events
projects / ~aditr / compat-rdma-docs.git / commitdiff