[MLX4] Fix ioremap to use consistent caching attribute

[MLX4] Fix ioremap to use consistent caching attribute

When mapping Blue Flame registers to user-mode, the UAR page is first mapped from I/O space to kernel-mode, then from kernel-mode to user-mode.  The mapping from I/O space uses the function ioremap, which is hard coded to use MmNonCached as the caching attribute.  The subsequent mapping to user-mode specifies MmWriteCombined.  On Windows Server 2008 R2, the memory manager catches the inconsistency and preserves the original MmNonCached attribute.  This leads to a 200-400% regression in small message latencies due to not write combining when writing the request to the blue flame register.

The following patch adds a caching type as a parameter to ioremap, changes all previous callers to specify MmNonCached except in __map_memory_for_user, which already took a caching type, in which case it forwards the caching type to ioremap.

With this patch, small message latency is identical on Windows Server 2008 and Windows Server 2008 R2.

Signed-off-by: Fab Tillier <ftillier@microsoft.com>
git-svn-id: svn://openib.tc.cornell.edu/gen1@2921 ad392aa1-c5ef-ae45-8dd8-e69d62a5ef86