<a href="#TOP"></a> </h1>\r
<h1 align="center">Windows OpenFabrics </h1>\r
<h1 align="center">User's Manual</h1>\r
-<h2 align="center">Release 2.1</h2>\r
+<h2 align="center">Release 2.2</h2>\r
<h3 align="center">\r
-<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->12/10/2009<!--webbot bot="Timestamp" endspan i-checksum="12507" --></h3>\r
+<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->12/29/2009<!--webbot bot="Timestamp" endspan i-checksum="12659" --></h3>\r
<h2 align="left"><u>Overview</u></h2>\r
<p align="left"><span style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'">\r
The Windows OpenFabrics (WinOF) package is composed of software modules intended \r
<br> </li>\r
</ul></li>\r
<li>\r
- <h3 align="left"><a href="#SRP">SRP (SCSI RDMA) Protocol Driver</a></h3></li>\r
+ <h3 align="left"><a href="#SRP">SRP (SCSI over RDMA) Protocol Driver</a></h3></li>\r
<li>\r
<h3 align="left"><u><font color="#0000FF"><a href="#QLOGICVNIC">QLogic VNIC_Driver</a></font></u></h3></li>\r
<li>\r
Programming Interfaces that exploit the RDMA (remote direct memory access) \r
capabilities of next-generation interconnect technologies such as InfiniBand, \r
and iWARP.</font></p>\r
-<p align="left"><font face="Times New Roman">WinOF DAT and DAPL are based on the 1.1 DAT specification. The DAPL \r
+<p align="left"><font face="Times New Roman">WinOF uDAT and uDAPL are based on the \r
+2.0 DAT specification. The DAPL \r
(Direct Access Provider Library) which now fully supports Infiniband RDMA and \r
IPoIB.</font></p>\r
-<p align="left"><font face="Times New Roman">WinOF 1.0.1, and future WinOF \r
-releases, will include DAT/DAPL version 2.0 runtime libraries along with an optional \r
-v2.0 application build environment.<br>\r
-DAT 2.0 is configured with InfiniBand extensions enabled. The IB extensions \r
+<p align="left"><font face="Times New Roman">Previous WinOF \r
+releases supported the uDAT/uDAPL 1.1 provider which has now been deprecated.<br>\r
+uDAT/uDAPL version 2.0 runtime libraries along with an optional \r
+v2.0 application build environment are the only options.<br>\r
+uDAT 2.0 is configured with InfiniBand extensions enabled. The IB extensions \r
include</font></p>\r
<ul>\r
<li>\r
DAT ENVIRONMENT</font><font face="Courier New" size="2">:</font></h3>\r
</div>\r
<blockquote>\r
- <p align="left"><font face="Courier New" size="2">DAT/DAPL v1.1 (free-build) \r
- runtime libraries are installed into %SystemRoot%, with the v1.1 Debug \r
- versions located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'. Debug \r
- libraries are identified as datd.dll and dapld.dll.</font></p>\r
- <p align="left"><font face="Courier New" size="2">IA32 (aka, 32-bit) \r
- versions of DAT/DAPL 1.1 runtime libraries, found only on 64-bit systems, \r
- are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat32.dll and \r
- dapl32.dll.<br>\r
- <br>\r
- DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as \r
+ <p align="left"><font face="Courier New" size="2">DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as \r
dat2.dll and dapl2.dll. Debug versions of the v2.0 runtime libraries \r
- are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.</font></p>\r
+ are located in '%SystemDrive%\%ProgramFiles%\WinOF'.</font></p>\r
<p align="left"><font face="Courier New" size="2">IA32 (aka, 32-bit) \r
versions of DAT/DAPL 2.0 runtime libraries, found only on 64-bit systems, \r
- are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat232.dll \r
- and dapl232.dll.</font></p>\r
- <p align="left"><font face="Courier New" size="2">In order for DAT/uDAPL \r
- programs to execute correctly, the runtime library files 'dat.dll and \r
- dapl.dll' must be present in one of the following folders: current \r
+ are identified in '%ProgramFiles%\WinOF' as dat32.dll \r
+ and dapl32.dll.</font></p>\r
+ <p align="left"><font face="Courier New" size="2">In order for DAT/DAPL \r
+ programs to execute correctly, the runtime library files 'dat2.dll and \r
+ dapl2.dll' must be present in one of the following folders: current \r
directory, %SystemRoot% or in the library search path.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default WinOF \r
- installation places the runtime library files dat.dll and dapl.dll in the '%SystemRoot%' folder; \r
- symbol files (.pdb) are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.</font></p>\r
+ installation places the runtime library files dat2.dll and dapl2.dll in the '%SystemRoot%' folder; \r
+ symbol files (.pdb) are located in '%ProgramFiles%\WinOF\'.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default DAPL configuration \r
file is defined as '%SystemDrive%\DAT\dat.conf'. This default \r
specification can be overriden by use of the environment variable \r
<p align="left"><font face="Courier New" size="2">Within the dat.conf file, \r
the DAPL library specification can be located as the 5th whitespace \r
separated line argument. By default the DAPL library file is installed as\r
- '%SystemRoot%\dapl.dll'.</font></p>\r
+ '%SystemRoot%\dapl2.dll'.</font></p>\r
<p align="left"><font face="Courier New" size="2">Should you choose to \r
relocated the DAPL library file to a path where whitespace appears in the \r
full library path specification, then the full library file specification \r
CM is IBAL).<br> ibnic1v2-scm - InfiniBand HCA #one, DAPL version 2.0, CM is \r
'socket-CM'<br>\r
ibnic0v2-cma - InfiniBand HCA #zero, DAPL version 2.0, CM is \r
- 'rdma-CM'<br>\r
- ibnic0-scm - InfiniBand HCA #zero, DAPL version 1.1, CM is \r
- 'IBAL'</font></p>\r
+ 'rdma-CM'</font></p>\r
<p align="left"><font face="Courier New" size="2">Each non-comment line in \r
the dat.conf file describes a DAPL provider interface.<br>\r
The 2nd to the last field on the right (7th from the left) describes the \r
<p align="left"><b><u><a name="DAT_App_Build"></a>DAT application build environment</u>:</b></p>\r
<blockquote>\r
<p align="left"><font face="Courier New" size="2">DAT library header files \r
- are selectively installed in the DAT default configuration folder as<br>'%SystemDrive%\DAT\v1-1' \r
- or\r
- '%SystemDrive%\DAT\v2-0'. Your C language based DAT 1.1 application compilation command line \r
- should include'/I%SystemDrive%\DAT\v1-1' with C code referencing '#include <DAT\udat.h>'.<br>\r
+ are selectively installed in the DAT default configuration folder as<br>'%SystemDrive%\DAT\v2-0'. \r
+ Your C language based DAT application compilation command line \r
+ should include'/I%SystemDrive%\DAT\v2-0' with C code referencing '#include <DAT\udat.h>'.<br>\r
<br>\r
The 'default' DAT/DAPL C language calling convention is '__stdcall', not the \r
'normal' Visual Studio C compiler default. __stdcall was chosen as MS \r
<br>\r
Visual Studio 2005 command window - (nmake) Makefile Fragments:</font></p>\r
<blockquote>\r
- <p align="left"><font face="Courier New" size="2">DAT_PATH=%SystemDrive%\DAT\v1-1<br>\r
+ <p align="left"><font face="Courier New" size="2">DAT_PATH=%SystemDrive%\DAT\v2-0<br>\r
CC = cl<br>\r
INC_FLAGS = /I $(DAT_PATH)<br>\r
<br>\r
<br>\r
LINK_FLAGS = /nologo /subsystem:console /machine:X64 \r
/libpath:$(DAT_PATH) $(LIBS)</font><b><br><br>\r
- </b>When linking a DEBUG/Checked version make sure to use datd.lib or \r
- dat2d.lib for DAT v2.0.</p>\r
+ </b>When linking a DEBUG/Checked version make sure to use dat2d.lib .</p>\r
</blockquote>\r
</blockquote>\r
<h4><u>DAT library environment variables</u>:</h4>\r
<h3><a name="DAPLTEST"></a>DAPLTEST</h3>\r
<pre>\r
\r
- dapltest - test for the Direct Access Provider Library (DAPL)\r
+ dapltest - test for the Direct Access Provider Library (DAPL) v2.0\r
\r
DESCRIPTION\r
\r
and verify the DAPL interfaces during development and porting.\r
At least two instantiations of the test must be run. One acts\r
as the server, fielding requests and spawning server-side test\r
- threads as needed. Other client invocations connect to the\r
- server and issue test requests.\r
+ threads as needed. Other client invocation connects to the\r
+ Dapltest server and issue test requests.\r
\r
The server side of the test, once invoked, listens continuously\r
for client connection requests, until quit or killed. Upon\r
\r
<a name="DAPLtest-examples"></a>EXAMPLES\r
\r
- dapltest -T S -d -D ibnic0\r
+ dapltest -T S -d -D ibnic0v2\r
\r
Starts a local dapltest server process with debug verbosity.\r
Server loops (listen for dapltest request, process request).\r
\r
- dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2\r
+ dapltest -T T -d -s winIB -D ibnic0v2 -i 100 client SR 4096 2 server SR 4096 2\r
\r
Runs a transaction test, with both sides\r
sending one buffer with two 4KB segments,\r
one hundred times; dapltest server is on host winIB.\r
\r
- dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2\r
+ dapltest -T P -d -s winIB -D ibnic0v2 -i 100 RW 4096 2\r
\r
Runs a performance test, with the client \r
RDMA writing one buffer with two 4KB segments,\r
one hundred times.\r
\r
- dapltest -T Q -s winIB -D ibnic0\r
+ dapltest -T Q -s winIB -D ibnic0v2\r
\r
Asks the dapltest server at host 'winIB' to clean up and exit.\r
\r
- dapltest -T L -D ibnic0 -d -w 16 -m 1000\r
+ dapltest -T L -D ibnic0v2 -d -w 16 -m 1000\r
\r
Runs all of the limit tests, setting up\r
16 complete sets of DAPL objects, and\r
creating at most a thousand instances\r
when trying to exhaust resources.\r
\r
- dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \\r
+ dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0v2 \\r
client RW 4096 1 server RW 2048 4 \\r
client SR 1024 4 server SR 4096 2 \\r
client SR 1024 3 -f server SR 2048 1 -f\r
for a larger number of iterations,\r
validating the data received.</pre>\r
<pre>\r
-dt-svr.bat - DAPLtest server script; starts a DAPL2test.exe server on the local node.\r
+dt-svr.bat - DAPLtest server script; starts a DAPL2test.exe server on the local system.\r
dt-svr DAPL-provider [-D [hex-debug-bitmask] ]</pre>\r
<blockquote>\r
<blockquote>\r
dt-cli DAPL-provider host-IPv4-address testname [-D [hex-debug-bitmask] ]\r
example: dt-cli ibnic0v2 10.10.2.20 trans\r
dt-cli -h # outputs help text.\r
- dt-svr ibnic0v2 # IBAL on HCA0</pre>\r
+ dt-svr ibnic0v2 # IBAL dapltest server listening on port HCA0</pre>\r
<pre>\r
-Verify dt-*.bat script is running same dapltest.exe(v1.1) or dapl2test.exe(v2.0)\r
+Verify dt-*.bat script is running same dapl2test.exe(DAPL v2.0)\r
\r
\r
BUGS (and To Do List)\r
Manager running somewhere on the IB fabric.<br>\r
<br>\r
- Supported Operating Systems and Service Packs:<br>\r
- o Windows XP SP3 x86 & x64<br>\r
+ o Windows 7 (x86 & x64)<br>\r
+ o Windows Server 2008 R2 (x86, x64)<br>\r
o Windows Server 2008/Vista (x86, x64)<br>\r
- o Windows Server 2008 HPC (x64)<br>\r
+ o Windows Server 2008 HPC (x64,x86)<br>\r
o Windows Server 2003 SP2/R2 (x86, x64, IA64)</p>\r
<h4 align="left"><u>Testing Levels</u></h4>\r
<p align="left">The SRP driver has undergone basic testing against Mellanox \r
Technologies'\r
SRP Targets MTD1000 and MTD2000.<br>\r
-Additionally the Linux OFED 1.4 SRP target has been tested.<br>\r
+Additionally the Linux OFED 1.4.1 SRP target with scst 1.0.0.0 (vdisk with \r
+blockio) has been tested. Note: ONLY the scst-1.0.0.0 release will work as \r
+the OFED 1.4.1 ib_srpt driver has a symbol dependency on scst. later releases of \r
+scst.ko no longer export the required symbol hence ib_srpt fails to load.<br>\r
+When ib_srpt is updated later versions of scst can be used.<br>\r
+<br>\r
Testing included SRP target drive format, read, write and dismount/offline\r
operations.<br>\r
</p>\r
<p align="left">SRP supports WPP tracing tools by using the GUID: \r
'5AF07B3C-D119-4233-9C81-C07EF481CBE6'. The flags and level of debug can be controlled at load-time or run-time; \r
see ib_srp.inf file for details.</p>\r
+<h4 align="left"><u>Constructing a RHEL 5.1 OFED 1.4.1 SRP vdisk BLOCKIO target</u></h4>\r
+<p align="left">Example assumptions:</p>\r
+<ul>\r
+ <li>\r
+ <p align="left">Red Hat Enterprise Linux Server release 5 (Tikanga) EL 5.1 \r
+ (2.6.18-8.el5)<br>\r
+ Later RHEL 5.4 releases will work with some minor scst compile time fixes \r
+ and later OFED releases.</p></li>\r
+ <li>\r
+ <p align="left">Linux SRP Target has /dev/sdb[123]<br>\r
+ sizeof(sdb1) < sdb2 < sdb3; test convention only.</p></li>\r
+ <li>\r
+ <p align="left">scst-1.0.0.tgz is required as OFED 1.4.1 ib_srpt has symbol \r
+ dependency on<br>\r
+ scst.ko....sigh. Later scst release do not export the required ib_srpt<br>\r
+ symbol; stick with scst-1.0.0 until OFED ib_srpt is updated.</p></li>\r
+ <li>\r
+ <p align="left"><a href="http://sourceforge.net/projects/scst/files/">\r
+ download scst-1.0.0</a></p></li>\r
+</ul>\r
+<p align="left">Use out of the box scst defines which include (#undef \r
+STRICT_SERIALIZING),<br>\r
+'no' kernel mods are required for BLOCKIO access to /dev/sdb[123].<br>\r
+<br>\r
+cd scst-1.0.0.0<br>\r
+make all<br>\r
+make install<br>\r
+<br>\r
+cd OFED-1.4.1<br>\r
+build OFED select #3 for 'all' OFED components<br>\r
+ - no SRP loads in /etc/infiniband/openib.conf, edit prior to reboot.</p>\r
+<p align="left">REBOOT.<br>\r
+<br>\r
+./LOAD & ./UNLOAD scripts are manual versions of what scstAdmin (separate scst \r
+package) will do minus<br>\r
+loading the OFED driver ib_srpt.<br>\r
+<br>\r
+SRP targets formatted from Windows using default NTFS allocation size.<br>\r
+Partition size & numbering is derrived from local test conventions; your setup \r
+will be different.<br>\r
+<br>\r
+/dev/sdb1 NTFS < 1GB<br>\r
+/dev/sdb2 NTFS > 1GB<br>\r
+/dev/sdb3 NTFS > sdb2</p>\r
+<p align="left"> </p>\r
+<h4 align="left"><u>Manual SRP Target LOAD script</u></h4>\r
+<div align="left">\r
+ <pre>#!/bin/sh \r
+\r
+GRP=Default\r
+\r
+if [ ! -e /proc/scsi_tgt ] ; then\r
+ echo -n Loading scst driver\r
+ modprobe scst\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+if [ ! -e /proc/scsi_tgt/vdisk ] ; then\r
+ echo -n Loading scst_vdisk driver\r
+ modprobe scst_vdisk\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst_vdisk\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+if [ ! -e /proc/scsi_tgt/vdisk ] ; then\r
+ echo -n Loading scst_vdisk driver\r
+ modprobe scst_vdisk\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst_vdisk\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+fgrep -q ib_srpt /proc/modules\r
+if [ $? -ne 0 ] ; then\r
+ modprobe ib_srpt\r
+ echo ib_srpt...OK\r
+fi \r
+\r
+echo -n Open SRP devices srp[123]\r
+echo "open srp1 /dev/sdb1 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp1 /dev/sdb1\r
+ exit $?\r
+fi\r
+echo "open srp2 /dev/sdb2 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp2 /dev/sdb2\r
+ exit $?\r
+fi\r
+echo "open srp3 /dev/sdb3 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp3 /dev/sdb3\r
+ exit $?\r
+fi\r
+echo ...OK\r
+\r
+echo -n Set allowed hosts access...\r
+echo "add *" > /proc/scsi_tgt/groups/Default/names\r
+echo ...OK\r
+\r
+echo -n Adding targets srp[123] as LUNs [012] in group $GRP\r
+echo "add srp1 0" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp1 0\r
+ exit $?\r
+fi\r
+echo "add srp2 1" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp2 1\r
+ exit $?\r
+fi\r
+echo "add srp3 2" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp3 2\r
+ exit $?\r
+fi\r
+echo ...OK\r
+</pre>\r
+</div>\r
+<div align="left">\r
+ <h4 align="left"><u>Manual SRP Target UNLOAD script</u></h4>\r
+ <pre>#!/bin/sh\r
+\r
+if [ -w /proc/scsi_tgt/vdisk/vdisk ] ; then\r
+ echo -n Closing SRP Targets srp[321]...\r
+ echo "close srp3" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo "close srp2" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo "close srp1" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo Done.\r
+fi\r
+\r
+fgrep -q scst_vdisk /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r scst_vdisk\r
+fi\r
+\r
+fgrep -q ib_srpt /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r ib_srpt\r
+fi\r
+\r
+fgrep -q scst /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r scst\r
+fi\r
+</pre>\r
+</div>\r
<h4 align="left"><a href="#TOP"><font color="#000000" size="4"><return-to-top></font></a></h4>\r
<h3> </h3>\r
<BLOCKQUOTE></BLOCKQUOTE>\r
<a href="#TOP"></a> </h1>\r
<h1 align="center">Windows OpenFabrics </h1>\r
<h1 align="center">User's Manual</h1>\r
-<h2 align="center">Release 2.1</h2>\r
+<h2 align="center">Release 2.2</h2>\r
<h3 align="center">\r
-<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->12/10/2009<!--webbot bot="Timestamp" endspan i-checksum="12507" --></h3>\r
+<!--webbot bot="Timestamp" S-Type="EDITED" S-Format="%m/%d/%Y" startspan -->12/29/2009<!--webbot bot="Timestamp" endspan i-checksum="12659" --></h3>\r
<h2 align="left"><u>Overview</u></h2>\r
<p align="left"><span style="FONT-SIZE: 12pt; FONT-FAMILY: 'Times New Roman'">\r
The Windows OpenFabrics (WinOF) package is composed of software modules intended \r
<br> </li>\r
</ul></li>\r
<li>\r
- <h3 align="left"><a href="#SRP">SRP (SCSI RDMA) Protocol Driver</a></h3></li>\r
+ <h3 align="left"><a href="#SRP">SRP (SCSI over RDMA) Protocol Driver</a></h3></li>\r
<li>\r
<h3 align="left"><u><font color="#0000FF"><a href="#QLOGICVNIC">QLogic VNIC_Driver</a></font></u></h3></li>\r
<li>\r
Programming Interfaces that exploit the RDMA (remote direct memory access) \r
capabilities of next-generation interconnect technologies such as InfiniBand, \r
and iWARP.</font></p>\r
-<p align="left"><font face="Times New Roman">WinOF DAT and DAPL are based on the 1.1 DAT specification. The DAPL \r
+<p align="left"><font face="Times New Roman">WinOF uDAT and uDAPL are based on the \r
+2.0 DAT specification. The DAPL \r
(Direct Access Provider Library) which now fully supports Infiniband RDMA and \r
IPoIB.</font></p>\r
-<p align="left"><font face="Times New Roman">WinOF 1.0.1, and future WinOF \r
-releases, will include DAT/DAPL version 2.0 runtime libraries along with an optional \r
-v2.0 application build environment.<br>\r
-DAT 2.0 is configured with InfiniBand extensions enabled. The IB extensions \r
+<p align="left"><font face="Times New Roman">Previous WinOF \r
+releases supported the uDAT/uDAPL 1.1 provider which has now been deprecated.<br>\r
+uDAT/uDAPL version 2.0 runtime libraries along with an optional \r
+v2.0 application build environment are the only options.<br>\r
+uDAT 2.0 is configured with InfiniBand extensions enabled. The IB extensions \r
include</font></p>\r
<ul>\r
<li>\r
DAT ENVIRONMENT</font><font face="Courier New" size="2">:</font></h3>\r
</div>\r
<blockquote>\r
- <p align="left"><font face="Courier New" size="2">DAT/DAPL v1.1 (free-build) \r
- runtime libraries are installed into %SystemRoot%, with the v1.1 Debug \r
- versions located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'. Debug \r
- libraries are identified as datd.dll and dapld.dll.</font></p>\r
- <p align="left"><font face="Courier New" size="2">IA32 (aka, 32-bit) \r
- versions of DAT/DAPL 1.1 runtime libraries, found only on 64-bit systems, \r
- are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat32.dll and \r
- dapl32.dll.<br>\r
- <br>\r
- DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as \r
+ <p align="left"><font face="Courier New" size="2">DAT/DAPL 2.0 (free-build) libraries are identified in %SystemRoot% as \r
dat2.dll and dapl2.dll. Debug versions of the v2.0 runtime libraries \r
- are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.</font></p>\r
+ are located in '%SystemDrive%\%ProgramFiles%\WinOF'.</font></p>\r
<p align="left"><font face="Courier New" size="2">IA32 (aka, 32-bit) \r
versions of DAT/DAPL 2.0 runtime libraries, found only on 64-bit systems, \r
- are identified in '%SystemDrive%\%ProgramFiles(x86)%\WinOF' as dat232.dll \r
- and dapl232.dll.</font></p>\r
- <p align="left"><font face="Courier New" size="2">In order for DAT/uDAPL \r
- programs to execute correctly, the runtime library files 'dat.dll and \r
- dapl.dll' must be present in one of the following folders: current \r
+ are identified in '%ProgramFiles%\WinOF' as dat32.dll \r
+ and dapl32.dll.</font></p>\r
+ <p align="left"><font face="Courier New" size="2">In order for DAT/DAPL \r
+ programs to execute correctly, the runtime library files 'dat2.dll and \r
+ dapl2.dll' must be present in one of the following folders: current \r
directory, %SystemRoot% or in the library search path.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default WinOF \r
- installation places the runtime library files dat.dll and dapl.dll in the '%SystemRoot%' folder; \r
- symbol files (.pdb) are located in '%SystemDrive%\%ProgramFiles(x86)%\WinOF'.</font></p>\r
+ installation places the runtime library files dat2.dll and dapl2.dll in the '%SystemRoot%' folder; \r
+ symbol files (.pdb) are located in '%ProgramFiles%\WinOF\'.</font></p>\r
<p align="left"><font face="Courier New" size="2">The default DAPL configuration \r
file is defined as '%SystemDrive%\DAT\dat.conf'. This default \r
specification can be overriden by use of the environment variable \r
<p align="left"><font face="Courier New" size="2">Within the dat.conf file, \r
the DAPL library specification can be located as the 5th whitespace \r
separated line argument. By default the DAPL library file is installed as\r
- '%SystemRoot%\dapl.dll'.</font></p>\r
+ '%SystemRoot%\dapl2.dll'.</font></p>\r
<p align="left"><font face="Courier New" size="2">Should you choose to \r
relocated the DAPL library file to a path where whitespace appears in the \r
full library path specification, then the full library file specification \r
CM is IBAL).<br> ibnic1v2-scm - InfiniBand HCA #one, DAPL version 2.0, CM is \r
'socket-CM'<br>\r
ibnic0v2-cma - InfiniBand HCA #zero, DAPL version 2.0, CM is \r
- 'rdma-CM'<br>\r
- ibnic0-scm - InfiniBand HCA #zero, DAPL version 1.1, CM is \r
- 'IBAL'</font></p>\r
+ 'rdma-CM'</font></p>\r
<p align="left"><font face="Courier New" size="2">Each non-comment line in \r
the dat.conf file describes a DAPL provider interface.<br>\r
The 2nd to the last field on the right (7th from the left) describes the \r
<p align="left"><b><u><a name="DAT_App_Build"></a>DAT application build environment</u>:</b></p>\r
<blockquote>\r
<p align="left"><font face="Courier New" size="2">DAT library header files \r
- are selectively installed in the DAT default configuration folder as<br>'%SystemDrive%\DAT\v1-1' \r
- or\r
- '%SystemDrive%\DAT\v2-0'. Your C language based DAT 1.1 application compilation command line \r
- should include'/I%SystemDrive%\DAT\v1-1' with C code referencing '#include <DAT\udat.h>'.<br>\r
+ are selectively installed in the DAT default configuration folder as<br>'%SystemDrive%\DAT\v2-0'. \r
+ Your C language based DAT application compilation command line \r
+ should include'/I%SystemDrive%\DAT\v2-0' with C code referencing '#include <DAT\udat.h>'.<br>\r
<br>\r
The 'default' DAT/DAPL C language calling convention is '__stdcall', not the \r
'normal' Visual Studio C compiler default. __stdcall was chosen as MS \r
<br>\r
Visual Studio 2005 command window - (nmake) Makefile Fragments:</font></p>\r
<blockquote>\r
- <p align="left"><font face="Courier New" size="2">DAT_PATH=%SystemDrive%\DAT\v1-1<br>\r
+ <p align="left"><font face="Courier New" size="2">DAT_PATH=%SystemDrive%\DAT\v2-0<br>\r
CC = cl<br>\r
INC_FLAGS = /I $(DAT_PATH)<br>\r
<br>\r
<br>\r
LINK_FLAGS = /nologo /subsystem:console /machine:X64 \r
/libpath:$(DAT_PATH) $(LIBS)</font><b><br><br>\r
- </b>When linking a DEBUG/Checked version make sure to use datd.lib or \r
- dat2d.lib for DAT v2.0.</p>\r
+ </b>When linking a DEBUG/Checked version make sure to use dat2d.lib .</p>\r
</blockquote>\r
</blockquote>\r
<h4><u>DAT library environment variables</u>:</h4>\r
<h3><a name="DAPLTEST"></a>DAPLTEST</h3>\r
<pre>\r
\r
- dapltest - test for the Direct Access Provider Library (DAPL)\r
+ dapltest - test for the Direct Access Provider Library (DAPL) v2.0\r
\r
DESCRIPTION\r
\r
and verify the DAPL interfaces during development and porting.\r
At least two instantiations of the test must be run. One acts\r
as the server, fielding requests and spawning server-side test\r
- threads as needed. Other client invocations connect to the\r
- server and issue test requests.\r
+ threads as needed. Other client invocation connects to the\r
+ Dapltest server and issue test requests.\r
\r
The server side of the test, once invoked, listens continuously\r
for client connection requests, until quit or killed. Upon\r
\r
<a name="DAPLtest-examples"></a>EXAMPLES\r
\r
- dapltest -T S -d -D ibnic0\r
+ dapltest -T S -d -D ibnic0v2\r
\r
Starts a local dapltest server process with debug verbosity.\r
Server loops (listen for dapltest request, process request).\r
\r
- dapltest -T T -d -s winIB -D ibnic0 -i 100 client SR 4096 2 server SR 4096 2\r
+ dapltest -T T -d -s winIB -D ibnic0v2 -i 100 client SR 4096 2 server SR 4096 2\r
\r
Runs a transaction test, with both sides\r
sending one buffer with two 4KB segments,\r
one hundred times; dapltest server is on host winIB.\r
\r
- dapltest -T P -d -s winIB -D ibnic0 -i 100 RW 4096 2\r
+ dapltest -T P -d -s winIB -D ibnic0v2 -i 100 RW 4096 2\r
\r
Runs a performance test, with the client \r
RDMA writing one buffer with two 4KB segments,\r
one hundred times.\r
\r
- dapltest -T Q -s winIB -D ibnic0\r
+ dapltest -T Q -s winIB -D ibnic0v2\r
\r
Asks the dapltest server at host 'winIB' to clean up and exit.\r
\r
- dapltest -T L -D ibnic0 -d -w 16 -m 1000\r
+ dapltest -T L -D ibnic0v2 -d -w 16 -m 1000\r
\r
Runs all of the limit tests, setting up\r
16 complete sets of DAPL objects, and\r
creating at most a thousand instances\r
when trying to exhaust resources.\r
\r
- dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0 \\r
+ dapltest -T T -V -d -t 2 -w 4 -i 55555 -s winIB -D ibnic0v2 \\r
client RW 4096 1 server RW 2048 4 \\r
client SR 1024 4 server SR 4096 2 \\r
client SR 1024 3 -f server SR 2048 1 -f\r
for a larger number of iterations,\r
validating the data received.</pre>\r
<pre>\r
-dt-svr.bat - DAPLtest server script; starts a DAPL2test.exe server on the local node.\r
+dt-svr.bat - DAPLtest server script; starts a DAPL2test.exe server on the local system.\r
dt-svr DAPL-provider [-D [hex-debug-bitmask] ]</pre>\r
<blockquote>\r
<blockquote>\r
dt-cli DAPL-provider host-IPv4-address testname [-D [hex-debug-bitmask] ]\r
example: dt-cli ibnic0v2 10.10.2.20 trans\r
dt-cli -h # outputs help text.\r
- dt-svr ibnic0v2 # IBAL on HCA0</pre>\r
+ dt-svr ibnic0v2 # IBAL dapltest server listening on port HCA0</pre>\r
<pre>\r
-Verify dt-*.bat script is running same dapltest.exe(v1.1) or dapl2test.exe(v2.0)\r
+Verify dt-*.bat script is running same dapl2test.exe(DAPL v2.0)\r
\r
\r
BUGS (and To Do List)\r
Manager running somewhere on the IB fabric.<br>\r
<br>\r
- Supported Operating Systems and Service Packs:<br>\r
- o Windows XP SP3 x86 & x64<br>\r
+ o Windows 7 (x86 & x64)<br>\r
+ o Windows Server 2008 R2 (x86, x64)<br>\r
o Windows Server 2008/Vista (x86, x64)<br>\r
- o Windows Server 2008 HPC (x64)<br>\r
+ o Windows Server 2008 HPC (x64,x86)<br>\r
o Windows Server 2003 SP2/R2 (x86, x64, IA64)</p>\r
<h4 align="left"><u>Testing Levels</u></h4>\r
<p align="left">The SRP driver has undergone basic testing against Mellanox \r
Technologies'\r
SRP Targets MTD1000 and MTD2000.<br>\r
-Additionally the Linux OFED 1.4 SRP target has been tested.<br>\r
+Additionally the Linux OFED 1.4.1 SRP target with scst 1.0.0.0 (vdisk with \r
+blockio) has been tested. Note: ONLY the scst-1.0.0.0 release will work as \r
+the OFED 1.4.1 ib_srpt driver has a symbol dependency on scst. later releases of \r
+scst.ko no longer export the required symbol hence ib_srpt fails to load.<br>\r
+When ib_srpt is updated later versions of scst can be used.<br>\r
+<br>\r
Testing included SRP target drive format, read, write and dismount/offline\r
operations.<br>\r
</p>\r
<p align="left">SRP supports WPP tracing tools by using the GUID: \r
'5AF07B3C-D119-4233-9C81-C07EF481CBE6'. The flags and level of debug can be controlled at load-time or run-time; \r
see ib_srp.inf file for details.</p>\r
+<h4 align="left"><u>Constructing a RHEL 5.1 OFED 1.4.1 SRP vdisk BLOCKIO target</u></h4>\r
+<p align="left">Example assumptions:</p>\r
+<ul>\r
+ <li>\r
+ <p align="left">Red Hat Enterprise Linux Server release 5 (Tikanga) EL 5.1 \r
+ (2.6.18-8.el5)<br>\r
+ Later RHEL 5.4 releases will work with some minor scst compile time fixes \r
+ and later OFED releases.</p></li>\r
+ <li>\r
+ <p align="left">Linux SRP Target has /dev/sdb[123]<br>\r
+ sizeof(sdb1) < sdb2 < sdb3; test convention only.</p></li>\r
+ <li>\r
+ <p align="left">scst-1.0.0.tgz is required as OFED 1.4.1 ib_srpt has symbol \r
+ dependency on<br>\r
+ scst.ko....sigh. Later scst release do not export the required ib_srpt<br>\r
+ symbol; stick with scst-1.0.0 until OFED ib_srpt is updated.</p></li>\r
+ <li>\r
+ <p align="left"><a href="http://sourceforge.net/projects/scst/files/">\r
+ download scst-1.0.0</a></p></li>\r
+</ul>\r
+<p align="left">Use out of the box scst defines which include (#undef \r
+STRICT_SERIALIZING),<br>\r
+'no' kernel mods are required for BLOCKIO access to /dev/sdb[123].<br>\r
+<br>\r
+cd scst-1.0.0.0<br>\r
+make all<br>\r
+make install<br>\r
+<br>\r
+cd OFED-1.4.1<br>\r
+build OFED select #3 for 'all' OFED components<br>\r
+ - no SRP loads in /etc/infiniband/openib.conf, edit prior to reboot.</p>\r
+<p align="left">REBOOT.<br>\r
+<br>\r
+./LOAD & ./UNLOAD scripts are manual versions of what scstAdmin (separate scst \r
+package) will do minus<br>\r
+loading the OFED driver ib_srpt.<br>\r
+<br>\r
+SRP targets formatted from Windows using default NTFS allocation size.<br>\r
+Partition size & numbering is derrived from local test conventions; your setup \r
+will be different.<br>\r
+<br>\r
+/dev/sdb1 NTFS < 1GB<br>\r
+/dev/sdb2 NTFS > 1GB<br>\r
+/dev/sdb3 NTFS > sdb2</p>\r
+<p align="left"> </p>\r
+<h4 align="left"><u>Manual SRP Target LOAD script</u></h4>\r
+<div align="left">\r
+ <pre>#!/bin/sh \r
+\r
+GRP=Default\r
+\r
+if [ ! -e /proc/scsi_tgt ] ; then\r
+ echo -n Loading scst driver\r
+ modprobe scst\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+if [ ! -e /proc/scsi_tgt/vdisk ] ; then\r
+ echo -n Loading scst_vdisk driver\r
+ modprobe scst_vdisk\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst_vdisk\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+if [ ! -e /proc/scsi_tgt/vdisk ] ; then\r
+ echo -n Loading scst_vdisk driver\r
+ modprobe scst_vdisk\r
+ if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? modprobe scst_vdisk\r
+ exit $?\r
+ fi\r
+ echo ...OK\r
+fi \r
+\r
+fgrep -q ib_srpt /proc/modules\r
+if [ $? -ne 0 ] ; then\r
+ modprobe ib_srpt\r
+ echo ib_srpt...OK\r
+fi \r
+\r
+echo -n Open SRP devices srp[123]\r
+echo "open srp1 /dev/sdb1 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp1 /dev/sdb1\r
+ exit $?\r
+fi\r
+echo "open srp2 /dev/sdb2 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp2 /dev/sdb2\r
+ exit $?\r
+fi\r
+echo "open srp3 /dev/sdb3 512 BLOCKIO" > /proc/scsi_tgt/vdisk/vdisk\r
+if [ $? -ne 0 ] ; then\r
+ echo err $? open srp3 /dev/sdb3\r
+ exit $?\r
+fi\r
+echo ...OK\r
+\r
+echo -n Set allowed hosts access...\r
+echo "add *" > /proc/scsi_tgt/groups/Default/names\r
+echo ...OK\r
+\r
+echo -n Adding targets srp[123] as LUNs [012] in group $GRP\r
+echo "add srp1 0" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp1 0\r
+ exit $?\r
+fi\r
+echo "add srp2 1" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp2 1\r
+ exit $?\r
+fi\r
+echo "add srp3 2" > /proc/scsi_tgt/groups/Default/devices\r
+if [ $? -ne 0 ] ; then\r
+ echo\r
+ echo err $? add srp3 2\r
+ exit $?\r
+fi\r
+echo ...OK\r
+</pre>\r
+</div>\r
+<div align="left">\r
+ <h4 align="left"><u>Manual SRP Target UNLOAD script</u></h4>\r
+ <pre>#!/bin/sh\r
+\r
+if [ -w /proc/scsi_tgt/vdisk/vdisk ] ; then\r
+ echo -n Closing SRP Targets srp[321]...\r
+ echo "close srp3" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo "close srp2" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo "close srp1" > /proc/scsi_tgt/vdisk/vdisk\r
+ echo Done.\r
+fi\r
+\r
+fgrep -q scst_vdisk /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r scst_vdisk\r
+fi\r
+\r
+fgrep -q ib_srpt /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r ib_srpt\r
+fi\r
+\r
+fgrep -q scst /proc/modules\r
+if [ $? -eq 0 ] ; then\r
+ modprobe -r scst\r
+fi\r
+</pre>\r
+</div>\r
<h4 align="left"><a href="#TOP"><font color="#000000" size="4"><return-to-top></font></a></h4>\r
<h3> </h3>\r
<BLOCKQUOTE></BLOCKQUOTE>\r
projects / ~shefty / rdma-win.git / commitdiff