次の方法で共有

Demo details for my TechEd 2014 presentation on File Server Networking

  • [アーティクル]

During my TechEd 2014 presentation on File Server Networking for a Private Cloud Storage Infrastructure in Windows Server 2012 R2, I did a demo showing a Scale-Out File Server Cluster. The demo consisted of showing the networking details of this cluster, looking both from the cluster perspective and the client side. I was asked after the presentation to share the details of the PowerShell script that I used. So, here it goes.

 

First, here is the output of the script. It’s broken into a few parts:

  • It starts by showing the cluster configuration (cluster nodes, cluster networks, cluster resources and cluster shared volumes)
  • Note that the name of the cluster itself is JOSE-S and the name of the Scale-Out File Server is JOSE-SO.
  • Then it shows the IP addresses on each node, plus what’s registered on DNS for the scale-out file server name.
  • Next, it outputs the SMB server network information, with focus on the scopes defined on the server.
  • Then we switch to the client view of connections, after just showing the directory from the 3 shares created. This shows that multichannel won’t engage until you have a read or a write.
  • Finally, after running a quick workload on the shares, it shows the full set of 24 connections (3 shares * 2 NICs * 4 connections per NIC).
  • The 10.1.x.100 IP addresses are associated with the cluster name used for cluster management. They don't relate directly to the Scale-Out File Server.
  • The scope name starting with "FE80" is the internal cluster scope used for inter-node communication.

 

Cluster Information

PS C:> Get-ClusterNode -Cluster JOSE-S

Name ID State
---- -- -----
JOSE-S1 2 Up
JOSE-S2 1 Up
JOSE-S3 3 Up

PS C:> Get-ClusterNetwork -Cluster JOSE-S

Cluster Name State Address Role
------- ---- ----- ------- ----
JOSE-S Cluster Network 1 Up 10.1.2.0 3
JOSE-S Cluster Network 2 Up 10.1.1.0 3

PS C:> Get-ClusterResource -Cluster JOSE-S

Name State OwnerGroup ResourceType
---- ----- ---------- ------------
Cluster Disk 1 Online Cluster Group Physical Disk
Cluster IP Address Online Cluster Group IP Address
Cluster IP Address 10.1.1.100 Online Cluster Group IP Address
Cluster Name Online Cluster Group Network Name
JOSE-SO Online JOSE-SO Distributed Network Name
Scale-Out File Server (\JOSE-SO) Online JOSE-SO Scale Out File Server

PS C:> Get-ClusterSharedVolume -Cluster JOSE-S

Name State Node
---- ----- ----
Cluster Disk 2 Online JOSE-S2
Cluster Disk 3 Online JOSE-S1
Cluster Disk 4 Online JOSE-S3

IP Addresses for each node, DNS for Scale-Out Name

PS C:> Get-NetIPAddress -CimSession <node>

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S1 12 10.1.1.100 24
JOSE-S1 12 10.1.1.2 24
JOSE-S1 14 10.1.2.100 24
JOSE-S1 14 10.1.2.2 24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S2 12 10.1.1.3 24
JOSE-S2 14 10.1.2.3 24

PSComputerName ifIndex IPAddress PrefixLength
-------------- ------- --------- ------------
JOSE-S3 12 10.1.1.4 24
JOSE-S3 14 10.1.2.4 24

PS C:> Resolve-DNSName JOSE-SO

Name Type TTL Section IPAddress
---- ---- --- ------- ---------
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.3
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.2
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.1.4
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.3
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.4
JOSE-SO.JOSE.TEST A 1200 Answer 10.1.2.2

IP Addresses on each node per Scope name

PS C:> Get-SmbServerNetworkInterface -CimSession JOSE-SO

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S1 FE80::9CBF:78F5:DFA8:803B 10.1.2.2
JOSE-S1 FE80::9CBF:78F5:DFA8:803B 10.1.1.2
JOSE-S1 JOSE-S 10.1.2.100
JOSE-S1 JOSE-S 10.1.1.100
JOSE-S1 JOSE-SO 10.1.2.2
JOSE-S1 JOSE-SO 10.1.1.2

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S2 FE80::69AF:5813:D729:CEFB 10.1.1.3
JOSE-S2 FE80::69AF:5813:D729:CEFB 10.1.2.3
JOSE-S2 JOSE-SO 10.1.2.3
JOSE-S2 JOSE-SO 10.1.1.3

PSComputerName ScopeName IPAddress
-------------- --------- ---------
JOSE-S3 FE80::F4BE:F77B:7EAC:6CED 10.1.1.4
JOSE-S3 FE80::F4BE:F77B:7EAC:6CED 10.1.2.4
JOSE-S3 JOSE-SO 10.1.2.4
JOSE-S3 JOSE-SO 10.1.1.4

Simple connection just to check directory

PS C:> Dir \JOSE-SO<share>

    Directory: \JOSE-SOShare1

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

    Directory: \JOSE-SOShare2

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

    Directory: \JOSE-SOShare3

Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 5/4/2014 5:40 PM 1717986918 testfile.dat
4

SMB Connection

PS C:> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO Share1 3.02 True JOSE.TESTAdministrator
JOSE-SO Share2 3.02 True JOSE.TESTAdministrator
JOSE-SO Share3 3.02 True JOSE.TESTAdministrator

TCP connections used by the client

PS C:> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5 10.1.1.2 49453 445
10.1.1.5 10.1.1.3 49470 445
10.1.1.5 10.1.1.4 49461 445

Putting some load on the shares

PS C:> C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>
sqlio v1.5.SG
using system counter for latency timings, 10000000 counts per second
8 threads reading for 10 secs from files \JOSE-SOShare1testfile.dat, \JOSE-SOShare2testfile.dat and \JOSE-SOShare3testfile.dat
using 8KB random IOs
enabling multiple I/Os per thread with 16 outstanding
buffering set to not use file nor disk caches (as is SQL Server)
using current size: 16384 MB for file: \JOSE-SOShare1testfile.dat
using current size: 16384 MB for file: \JOSE-SOShare2testfile.dat
using current size: 16384 MB for file: \JOSE-SOShare3testfile.dat
initialization done
CUMULATIVE DATA:
throughput metrics:
IOs/sec: 26822.20
MBs/sec: 209.54
latency metrics:
Min_Latency(ms): 0
Avg_Latency(ms): 13
Max_Latency(ms): 171
histogram:
ms: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24+
%: 0 0 0 0 1 2 4 6 8 8 9 8 8 7 6 5 5 4 3 3 2 2 1 1 7

SMB Connection

PS C:> Get-SmbConnection

ServerName ShareName Dialect Redirected Credential
---------- --------- ------- ---------- ----------
JOSE-SO Share1 3.02 True JOSE.TESTAdministrator
JOSE-SO Share2 3.02 True JOSE.TESTAdministrator
JOSE-SO Share3 3.02 True JOSE.TESTAdministrator

SMB Multichannel Connection

PS C:> Get-SmbMultichannelConnection

ClientIPAddress ServerIPAddress CurrentChannels
--------------- --------------- ---------------
10.1.1.5 10.1.1.2 4
10.1.1.5 10.1.1.3 4
10.1.1.5 10.1.1.4 4
10.1.2.5 10.1.2.2 4
10.1.2.5 10.1.2.3 4
10.1.2.5 10.1.2.4 4

TCP connections used by the client

PS C:> Get-NetTCPConnection -RemotePort 445

LocalAddress RemoteAddress LocalPort RemotePort
------------ ------------- --------- ----------
10.1.1.5 10.1.1.2 49511 445
10.1.1.5 10.1.1.2 49513 445
10.1.1.5 10.1.1.2 49516 445
10.1.1.5 10.1.1.2 49480 445
10.1.1.5 10.1.1.3 49518 445
10.1.1.5 10.1.1.3 49497 445
10.1.1.5 10.1.1.3 49512 445
10.1.1.5 10.1.1.3 49515 445
10.1.1.5 10.1.1.4 49514 445
10.1.1.5 10.1.1.4 49517 445
10.1.1.5 10.1.1.4 49492 445
10.1.1.5 10.1.1.4 49509 445
10.1.2.5 10.1.2.2 49498 445
10.1.2.5 10.1.2.2 49501 445
10.1.2.5 10.1.2.2 49508 445
10.1.2.5 10.1.2.2 49505 445
10.1.2.5 10.1.2.3 49500 445
10.1.2.5 10.1.2.3 49510 445
10.1.2.5 10.1.2.3 49506 445
10.1.2.5 10.1.2.3 49503 445
10.1.2.5 10.1.2.4 49507 445
10.1.2.5 10.1.2.4 49502 445
10.1.2.5 10.1.2.4 49499 445
10.1.2.5 10.1.2.4 49504 445

  

Next, here’s the PowerShell script behind what was shown above. Here are a few notes on the script:

  • It shows some text with a simplified view of the cmdlet about to be run, but the actual cmdlet is usually a bit more complex since it usually includes additional parameters and some formatting.
  • I tried to use some color and extra blank lines so things would be easier to read.
  • The “read-host” cmdlets are used to pause the script until I press enter to continue to the next phase.
  • I pipe the results to “Out-Host” to make sure everything shows in proper order on the screen (without it, some of the longer running cmdlets might show out of order).
  • The script is run from a client, so cmdlets that target the cluster use the –Cluster parameter and cmdlets that target a specific server use the –CimSession parameter.
  • SQLIO is used as a workload generator. For more details, see this other blog post on using SQLIO.
  • I make a point of using the PowerShell cmdlets for network information (Resolve-DnsName instead of NSLOOKUP.EXE, Get-NetTcpConnection instead of NETSTAT.EXE).

 

Cls

Write-Host -ForegroundColor Green "Cluster Information"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterNode -Cluster JOSE-S"
Get-ClusterNode -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterNetwork -Cluster JOSE-S"
Get-ClusterNetwork -Cluster JOSE-S | FT Cluster, Name, State, Address, Role -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterResource -Cluster JOSE-S"
Get-ClusterResource -Cluster JOSE-S | FT -AutoSize | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Get-ClusterSharedVolume -Cluster JOSE-S"
Get-ClusterSharedVolume -Cluster JOSE-S | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses for each node, DNS for Scale-Out Name"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetIPAddress -CimSession <node>"
"JOSE-S1", "JOSE-S2", "JOSE-S3" | % { Get-NetIPAddress -CimSession $_ -AddressFamily IPv4 -IPAddress 10.1.* | Sort IPAddress | FT PSComputerName, ifIndex, IPAddress, PrefixLength -AutoSize } | Out-Host

Write-Host -ForegroundColor Yellow "PS C:> Resolve-DNSName JOSE-SO"
Resolve-DnsName JOSE-SO | FT -AutoSize | Out-Host

Read-Host

Write-Host -ForegroundColor Green "IP Addresses on each node per Scope name", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbServerNetworkInterface -CimSession JOSE-SO"
1..3 | % { Get-SmbServerNetworkInterface -CimSession JOSE-S$_ | ? ScopeName -ne "*" | Sort ScopeName | FT PSComputerName, ScopeName, IPAddress -AutoSize } | Out-Host

Read-Host

Write-Host -ForegroundColor Green "Simple connection just to check directory", ""
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Dir \JOSE-SO<share>"
Dir \JOSE-SOShare1testfile.dat, \JOSE-SOShare2testfile.dat, \JOSE-SOShare3testfile.dat | Out-Host

Write-Host -ForegroundColor Green "SMB Connection", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort -AutoSize | Out-Host
Read-Host

Write-Host -ForegroundColor Green "Putting some load on the shares", ""
Write-Host " "
Write-Host -ForegroundColor Yellow "PS C:> C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN <file>"
C:SQLIOSQLIO.EXE -s10 -kR -frandom -b8 -t8 -o16 -LS -BN \JOSE-SOShare1testfile.dat \JOSE-SOShare2testfile.dat \JOSE-SOShare3testfile.dat

Write-Host -ForegroundColor Green "SMB Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbConnection"
Get-SmbConnection | FT ServerName, ShareName, Dialect, Redirected, Credential -AutoSize | Out-Host

Write-Host -ForegroundColor Green "SMB Multichannel Connection"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-SmbMultichannelConnection"
Get-SmbMultichannelConnection | Sort ClientIPAddress, ServerIPAddress | FT ClientIPAddress, ServerIPAddress, CurrentChannels | Out-Host

Write-Host -ForegroundColor Green "TCP connections used by the client"
Write-Host " "

Write-Host -ForegroundColor Yellow "PS C:> Get-NetTCPConnection -RemotePort 445"
Get-NetTCPConnection -RemotePort 445 | Sort LocalAddress, RemoteAddress | FT LocalAddress, RemoteAddress, LocalPort, RemotePort –AutoSize | Out-Host

 

You can find the slides and the recording for the presentation at https://channel9.msdn.com/Events/TechEd/NorthAmerica/2014/DCIM-B337

Comments

  • Anonymous
    January 01, 2003
    Great! Really easy to reproduce and make testing!
  • Anonymous
    January 01, 2003
    @ Jose

    Does JBOD / Storage Spaces / Scale Out File Server Support ODX?

    I have posted a question here: http://social.technet.microsoft.com/Forums/windowsserver/en-US/f55e2985-e297-48c7-8d97-e166c02eafc9/does-odx-work-with-hyper-v-vhdxs-on-smb3-shares-presented-from-sofs-storage-spaces?forum=winserver8gen

    It would seem not but I haven't had anyone from Microsoft give a yes or No.

    Thanks

    John.
  • Anonymous
    May 16, 2014
    (y)
  • Anonymous
    May 16, 2014
    Jose, thank you for the awesome content on your blog. We wouldn't--and couldn't--have a SOFS-based storage solution without it.

    I've been working with Microsoft support since January on a few issues, but my experience has been like this so far: 1) I get assigned to an engineer; 2) The engineer connects and observes the behavior of the system, and over the course of several calls, takes multiple traces; 3) The engineer says they will go to the lab to confirm the behavior and will get back to me; 4) Multiple weeks pass; 5) I get assigned to a new engineer; 6) Repeat. At this point, I've given up on any answers from support, and have resorted to figuring out almost every issue (those for which I couldn't find online answers) on my own. For the most part, I've found that the issues that prevented rolling this out into production (e.g., fixed VHDX creation via the SoFS CAP doesn't exceed about 55 MB/s) are apparently normal behavior.

    I'm now going back and re-running performance testing, and I'm finding behavior I just can't explain. I'm hoping you can help. I'm running a series of SQLIO tests against various scenarios--different write-back cache sizes (including 0), different CSV cache sizes (including 0), fixed vs. dynamic VHDX files, SSD tiering results, different column configurations, etc. Right now, I'm running fixed vs. dynamic VHDX tests. In every scenario, the dynamic VHDX greatly outperforms fixed, which is not only a huge surprise, but the numbers are seemingly impossible--sometimes nearly 40x performance difference for both read and write, random and sequential, 8KB and 512KB, queue depth 2-32, threads 2-16.

    For example, using a 3 column mirrored space (I thought it was made up of 18 HDDs, but 6 SSDs are also in the pool; I think it may not be relevant to the fixed vs. dynamic question, though) and a CSV cache size of 10 GB, I get the following results (all tests -s60 -LS -BN, testfile=1 GB):

    Seq Read, 512 KB, 2 threads, QD=2:
    Fixed IOPS: 1095
    Fixed MB/s: 547.5
    Dynamic IOPS: 6156
    Dynamic MB/s: 3078

    Random Read, 8 KB, 8 threads, QD=2:
    Fixed IOPS: 15218.59
    Fixed MB/s: 118.89
    Dynamic IOPS: 55021.66
    Dynamic MB/s: 429.85

    Seq Write, 8 KB, 8 threads, QD=2:
    Fixed IOPS: 934.08
    Fixed MB/s: 7.29
    Dynamic IOPS: 19683.15
    Dynamic MB/s: 153.77

    Random Write, 8 KB, 8 threads, QD=2:
    Fixed IOPS: 667.44
    Fixed MB/s: 5.21
    Dynamic IOPS: 26282.69
    Dynamic MB/s: 205.33

    For many of the tests, Perfmon shows no disk activity for the volume on the SOFS server at all, during both writes and reads. There's a bit of RDMA traffic, but the values don't seem to correspond to anything--they're usually under 20,000 Bytes/s. Yet on the Hyper-V host, Perfmon will show values exceeding 9 GB/s for the dynamic disk (Hyper-V Virtual Storage Device counter).

    Is this expected behavior? Are the results misleading, or am I really getting much better performance with dynamics disks over fixed? Why is there no SOFS disk activity for both reads and writes?

    Thank you again for all the help you provide on your blog--I look forward to every post!
  • Anonymous
    May 19, 2014
    I've done a bit more testing, and as far as I can tell, there must be some Hyper-V (Server 2012 R2) host-level read and write caching going done in memory with dynamic disks that doesn't happen with fixed disks, at least during SQLIO tests--using the -BN switch in both cases--from within a VM.

    What's most disconcerting is that, while the Hyper-V host shows the huge read and write numbers for the Hyper-V Virtual Storage Device, the SOFS cluster doesn't show any activity for the physical disk during the dynamic disk tests. In fact, the SMB Client Shares counter for the SOFS share on which the dynamic disk is located also shows no activity. It seems that the entire SQLIO -BN test on a dynamic disk stored on an SMB3 share is performed in memory--both reads and writes. This suggests that fixed disks are safe, whereas dynamic disks are not, even when the application plays by the rules.

    Is this really what's happening, or am I missing something? Is there a configuration step that can be made to prevent at least the write caching for dynamic disks (even for applications that require it)? I have not found anything particularly relevant to this issue so far. I found this article: http://support.microsoft.com/kb/2853952, but it doesn't apply to 2012 R2, and the issue I'm seeing is specific to dynamic disks.
  • Anonymous
    June 02, 2014
    Thanks for posting this up, Jose! You've made life much easier when testing out SMB 3.0 File shares now. It was great meeting you at the conference!