[Forum FAQ] Troubleshooting Network File Copy Slowness

Question

1. Introduction
The Server Message Block (SMB) Protocol is a network file sharing protocol, and as implemented in Microsoft Windows is known as Microsoft SMB Protocol. The set of message packets that defines a particular version of the protocol is called a dialect. The Common
Internet File System (CIFS) Protocol is a dialect of SMB. Both SMB and CIFS are also available on VMS, several versions of Unix, and other operating systems.
Microsoft SMB Protocol and CIFS Protocol Overview
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365233(v=vs.85).aspx
Server Message Block overview
http://technet.microsoft.com/en-us/library/hh831795.aspx
1.1
SMB Versions and Negotiated Versions
- Thanks for the
Jose Barreto's Blog
There are several different versions of SMB used by Windows operating systems:
CIFS – The ancient version of SMB that was part of Microsoft Windows NT 4.0 in 1996. SMB1 supersedes this version.
SMB 1.0 (or SMB1) – The version used in Windows 2000, Windows XP, Windows Server 2003 and Windows Server 2003 R2
SMB 2.0 (technically SMB2 version 2.002) – The version used in Windows Vista (SP1 or later) and Windows Server 2008 (or any SP)
SMB 2.1 ((technically SMB2 version 2.1) – The version used in Windows 7 (or any SP) and Windows Server 2008 R2 (or any SP)
SMB 3.0 (or SMB3) – The version used in Windows 8 and Windows Server 2012
SMB 3.02 (or SMB3) – The version used in Windows 8.1 and Windows Server 2012 R2
Windows NT is no longer supported, so CIFS is definitely out. Windows Server 2003 R2 with a current service pack is under Extended Support, so SMB1 is still around for a little while. SMB 2.x in Windows Server 2008 and Windows Server 2008
R2 are under Mainstream Support until 2015. You can find the most current information on the
support lifecycle page for Windows Server. The information is subject to the
Microsoft Policy Disclaimer and Change Notice.  You can use the support pages to also find support policy information for Windows
XP, Windows Vista, Windows 7 and Windows 8.
In Windows 8.1 and Windows Server 2012 R2, we introduced the option to completely disable CIFS/SMB1 support, including the actual removal of the related binaries. While this is not the default configuration, we recommend disabling this older
version of the protocol in scenarios where it’s not useful, like Hyper-V over SMB. You can find details about this new option in item 7 of this blog post:
What’s new in SMB PowerShell in Windows Server 2012 R2.
Negotiated Versions
Here’s a table to help you understand what version you will end up using, depending on what Windows version is running as the SMB client and what version of Windows is running as the SMB server:
OS
Windows 8.1  WS 2012 R2
Windows 8  WS 2012
Windows 7  WS 2008 R2
Windows Vista  WS 2008
Previous versions
Windows 8.1 WS 2012 R2
SMB 3.02
SMB 3.0
SMB 2.1
SMB 2.0
SMB 1.0
Windows 8 WS 2012
SMB 3.0
SMB 3.0
SMB 2.1
SMB 2.0
SMB 1.0
Windows 7 WS 2008 R2
SMB 2.1
SMB 2.1
SMB 2.1
SMB 2.0
SMB 1.0
Windows Vista WS 2008
SMB 2.0
SMB 2.0
SMB 2.0
SMB 2.0
SMB 1.0
Previous versions
SMB 1.0
SMB 1.0
SMB 1.0
SMB 1.0
SMB 1.0
* WS = Windows Server
1.2 Check, Enable and Disable SMB Versions in Windows operating systems
In Windows 8 or Windows Server 2012 and later, there is a new PowerShell cmdlet that can easily tell you what version of SMB the client has negotiated with the File Server. You simply access a remote file server (or create a new mapping to it) and use Get-SmbConnection.
To enable and disable SMBv1, SMBv2, and SMBv3 in Windows Vista, Windows Server 2008, Windows 7, Windows Server 2008 R2, Windows 8, and Windows Server 2012, please follow the steps in the article below.
Warning: We do not recommend that you disable SMBv2 or SMBv3. Disable SMBv2 or SMBv3 only as a temporary troubleshooting measure. Do not leave SMBv2 or SMBv3 disabled.
http://support.microsoft.com/kb/2696547
1.3 Features and Capabilities
- Thanks for the
Jose Barreto's Blog
Here’s a very short summary of what changed with each version of SMB:
From SMB 1.0 to SMB 2.0 - The first major redesign of SMB
Increased file sharing scalability
Improved performance
Request compounding
Asynchronous operations
Larger reads/writes
More secure and robust
Small command set
Signing now uses HMAC SHA-256 instead of MD5
SMB2 durability
From SMB 2.0 to SMB 2.1
File leasing improvements
Large MTU support
BranchCache
From SMB 2.1 to SMB 3.0
Availability
SMB Transparent Failover
SMB Witness
SMB Multichannel
Performance
SMB Scale-Out
SMB Direct (SMB 3.0 over RDMA)
SMB Multichannel
Directory Leasing
BranchCache V2
Backup
VSS for Remote File Shares
Security
SMB Encryption using AES-CCM (Optional)
Signing now uses AES-CMAC
Management
SMB PowerShell
Improved Performance Counters
Improved Eventing
From SMB 3.0 to SMB 3.02
Automatic rebalancing of Scale-Out File Server clients
Improved performance of SMB Direct (SMB over RDMA)
Support for multiple SMB instances on a Scale-Out File Server
You can get additional details on the SMB 2.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2008/12/09/smb2-a-complete-redesign-of-the-main-remote-file-protocol-for-windows.aspx
You can get additional details on the SMB 3.0 improvements listed above at
http://blogs.technet.com/b/josebda/archive/2012/05/03/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx
You can get additional details on the SMB 3.02 improvements in Windows Server 2012 R2 at
http://technet.microsoft.com/en-us/library/hh831474.aspx
1.4 Related Registry Keys
HKLM\SYSTEM\CurrentControlSet\Services\MrxSmb\Parameters\
DeferredOpensEnabled – Indicates whether the Redirector can defer opens for certain cases where the file does not really need to be opened, such as for certain delete requests and adjusting file attributes.
This defaults to true and is stored in the Redirector variable MRxSmbDeferredOpensEnabled.
OplocksDisabled – Whether the Redirector should not request oplocks, this defaults to false (the Redirector will request oplocks) and is stored in the variable MrxSmbOplocksDisabled.
CscEnabled – Whether Client Side Caching is enabled. This value defaults to true and stored in MRxSmbIsCscEnabled. It is used to determine whether to execute CSC operations when called. If CSC is enabled,
several other parameters controlling CSC behavior are checked, such as CscEnabledDCON, CscEnableTransitionByDefault, and CscEnableAutoDial. CSC will be discussed in depth in its own module, so will be only mentioned in this module when it is necessary to understanding
the operation of the Redirector.
DisableShadowLoopback – Whether to disable the behavior of the Redirector getting a handle to loopback opens (opens on the same machine) so that it can shortcut the network path to the resource and
just access local files locally. Shadow opens are enabled by default, and this registry value can be used to turn them off. It is stored in the global Redirector variable RxSmbDisableShadowLoopback.
IgnoreBindingOrder – Controls whether the Redirector should use the binding order specified in the registry and controlled by the Network Connections UI, or ignore this order when choosing a transport
provider to provide a connection to the server. By default the Redirector will ignore the binding order and can use any transport. The results of this setting are stored in the variable MRxSmbObeyBindingOrder.
HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\
Security Signature settings – The RequireSecuritySignature setting is stored in MRxSmbSecuritySignaturesRequired, EnableSecuritySignature in MRxSmbSecuritySignaturesEnabled, RequireExtendedSignature
in MRxSmbExtendedSignaturesRequired, and EnableExtendedSignature in MRxSmbExtendedSignaturesEnabled. Note that the Extended Security Signatures assume the regular security signatures are enabled, so those settings are adjusted if necessary based on the extended
settings. If extended signatures are required, regular signatures have to be required.
EnablePlainTextPassword – Support for using plain text passwords can be turned on using this key. They are disabled by default.
OffLineFileTimeoutIntervalInSeconds – Used to set the expiration time for timing out an Exchange (discussed later) when the exchange is accessing an offline file. This value defaults to 1000 seconds,
but can be changed in the registry and is stored in the global Redirector variable OffLineFileTimeoutInterval
SessTimeout – This is the amount of time the client waits for the server to respond to an outstanding request. The default value is 60 seconds (Windows Vista). When the client does not receive the
response to a request before the Request Expiration Timer expires, it will reset the connection because the operation is considered blocked. In Windows 8, the request expiration timer for the SMB 2 Negotiate is set to a smaller value, typically under 20 seconds,
so that if a node of a continuously available (CA) cluster server is not responding, the SMB 3.0 client can expedite failover to the other node.
ExtendedSessTimeout – Stored in the ExtendedSessTimeoutInterval variable, this value is used to extend the timeout on exchanges for servers that require an extended session timeout as listed in the
ServersWithExtendedSessTimeout key. These are third party servers that handle SMB sessions with different processes and vary dramatically on the time required to process SMB requests. The default value is 1000 seconds. If the client is running at least Windows
7 and ExtendedSessTimeout is not configured (By Default), the timeout is extended to four times the value of SessTimeout (4 * SessTimeout).
MaxNumOfExchangesForPipelineReadWrite – This value is used to determine the maximum number of write exchanges that can be pipelined to a server. The default is 8 and the value is stored in the variable
MaxNumOfExchangesForPipelineReadWrite.
Win9xSessionRestriction – This value defaults to false, but is used to impose a restriction on Windows 9x clients that they can only have one active non-NULL session with the server at a time. Also,
existing session based connections (VNETROOTS) are scavenged immediately, without a timeout to allow them to be reused.
EnableCachingOnWriteOnlyOpens – This value can cause the Redirector to attempt to open a file that is being opened for write only access in a manner that will enable the Redirector to cache the file
data. If the open fails, the request will revert back to the original requested access. The value of this parameter defaults to false and is stored in the MRxSmbEnableCachingOnWriteOnlyOpens variable.
DisableByteRangeLockingOnReadOnlyFiles – This parameter defaults to false, but if set to true will cause level II oplocks to automatically be upgraded to batch oplocks on read-only files opened for
read only access. It is stored in the variable DisableByteRangeLockingOnReadOnlyFiles.
EnableDownLevelLogOff – False by default, this value controls whether a Logoff SMB will be sent to down-level servers when a session is being closed. If this is false, and the server has not negotiated
to the NT SMB dialect or does not support NT Status codes, the logoff will not be sent because we aren’t sure that server will understand the request. The value is stored in MrxSmbEnableDownLevelLogOff.
HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\
ResilientTimeout – This timer is started when the transport connection associated with a resilient handle is lost. It controls the amount of time the server keeps a resilient handle active after the
transport connection to the client is lost. The default value is 300 seconds (Windows 7, Server 2008 R2, 8, Server 2012).
DurableHandleV2TimeoutInSecond – This timer is started when the transport connection associated with a durable handle is lost. It controls the amount of time the server keeps a durable handle active
after the transport connection to the client is lost. The default value is 60 seconds (Windows 8, Windows Server 2012). The maximum value is 300 seconds.
HKLM\SYSTEM\CurrentControlSet\Services\SMBWitness\Parameters\
KeepAliveInterval – This functionality was introduced for SMB 3.0 in Windows 8 and Windows Server 2012. The witness protocol is used to explicitly notify a client of resource changes that have occurred
on a highly available cluster server. This enables faster recovery from unplanned failures, so that the client does not need to wait for TCP timeouts. The default value is 20 minutes (Windows 8, Windows Server 2012).
HKLM\System\CurrentControlSet\Services\SmbDirect\Parameters\
ConnectTimeoutInMs – Establish a connection and complete negotiation. ConnectTimeoutInMs is the deadline for the remote peer to accept the connection request and complete SMB Direct negotiation. Default
is 120 seconds (Windows 8).
AcceptTimeoutInMs – Accept negotiation: The SMB Direct Negotiate request should be received before AcceptTimeoutInMs expires. The servers starts this timer as soon as it accepted the connection. Default
is 5 seconds (Windows 8).
IdleConnectionTimeoutInMs – This timer is per-connection. It is the amount of time the connection can be idle without receiving a message from the remote peer. Before the local peer terminates the
connection, it sends a keep alive request to the remote peer and applies a keep alive timer. Default is Default: 120 seconds (Windows 8).
KeepaliveResponseTimeoutInMs – This attribute is per-connection. It defines the timeout to wait for the peer response for a keep-alive message on an idle RDMA connection. Default is 5 seconds (Windows
8).
CreditGrantTimeoutInMs – This timer is per-connection.  It regulates the amount of time that the local peer waits for the remote peer to grant Send credits before disconnecting the connection.
This timer is started when the local peer runs out of Send credits. Default is 5 seconds (Windows 8).
References:
[MS-SMB]: Server Message Block (SMB) Protocol
http://msdn.microsoft.com/en-us/library/cc246231.aspx
[MS-SMB2]: Server Message Block (SMB) Protocol Versions 2 and 3
http://msdn.microsoft.com/en-us/library/cc246482.aspx
SMB 2.x and SMB 3.0 Timeouts in Windows
http://blogs.msdn.com/b/openspecification/archive/2013/03/27/smb-2-x-and-smb-3-0-timeouts-in-windows.aspx

Answer 1

3. How to Troubleshoot
3.1 Troubleshooting Decision Tree
1
Is the slowness occurring in browsing a network shared folder or   copying a file, or both?
Browsing, go to 1.1.
Copying, go to 1.2.
Both, go to 1.3.
1.1
Is the target a DFS path or not?
Yes, go to 1.1.1.
No, go to 1.1.2.
1.1.1
Is the client visiting the nearest DFS root server and file   server?
Yes, go to 1.1.1.1.
No, go to 1.1.1.2.
1.1.1.1
Browse the corresponding (Non-DFS) UNC path directly. Do you   still experience the slowness?
Yes, go to 1.1.1.1.1.
 No,
go to 1.1.1.1.2.
1.1.1.1.1
Issue is the particular file server responds to the share folder   enumeration requests slowly. Most probably it’s
unrelated to DFS. Follow   1.1.2.
1.1.1.1.2
Issue is that client experiences delay when browsing the DFS   path, but no delay is visiting the target file server
directly. Capture   Network Monitor trace from the client and study if the DFS path is cracked   down.
1.1.1.2
Use dfsutil.exe to clear local domain and referral cache. Then   visit the DFS path again and capture Network Monitor
trace from the client to   study why the client goes to a wrong file server or DFS root server.
1.1.2
Not a DFS issue. Issue is the particular file server responds to   the share folder enumeration requests slowly. “Dir”
the same share folder   from Command Prompt. Is it slow?
Yes, go to 1.1.2.1
No, go to 1.1.2.2
1.1.2.1
Check the number of subfolders and files in that share folder.   Is the number large?
Yes, go to 1.1.2.1.1
No, go to 1.1.2.1.2
1.1.2.1.1
Try to “dir” a different share folder on the same file server,   but with less items. Is it still slow or not?
Yes, go to 1.1.2.1.1.1
No, go to 1.1.2.1.1.2
1.1.2.1.1.1
Probably to be performance issue of the file server. Capture   Network Monitor trace from both sides, plus Performance
Monitor on the file   server.
1.1.2.1.1.2
Probably to be performance issue of the file server,   particularly, of the disk. Capture Network Monitor trace from
both sides,   plus Performance Monitor on the file server.
1.1.2.1.2
Same as 1.1.2.1.1.1. Probably to be performance issue of the   file server. Capture Network Monitor trace from both
sides, plus Performance   Monitor on the file server.
1.1.2.2
Explorer.exe browses the share folder slowly while “dir” does   fast. The issue should lie in the particular SMB traffic
incurred by   explorer.exe. It's a Shell issue.
1.2
Is the target a DFS path or not?
Yes, go to 1.2.1
No, go to 1.2.2
1.2.1
Is the client downloading/uploading against the nearest file   server?
Yes, go to 1.2.1.1
No, go to 1.2.1.2
1.2.1.1
Try to download/upload against that file server using the   Non-DFS share path. Still slow?
Yes, go to 1.2.1.1.1
No, go to 1.2.1.1.2
1.2.1.1.1
Not a DFS issue. Capture Network Monitor trace from both sides   to identify the pattern of the slowness.
1.2.1.1.2
This is unlikely to occur because the conclusion is   contradictory to itself. Start from the beginning to double
check.
1.2.1.2
Same situation as 1.1.1.2. Use dfsutil.exe to clear local domain   and referral cache. Then visit the DFS path again
and capture Network Monitor   trace from the client to study why the client goes to a wrong file server or   DFS root server.
1.2.2
Same as 1.2.1.1.1. It's not a DFS issue. Capture Network Monitor   trace from both sides to identify the pattern of
the slowness.
1.3
Follow 1.1 and then 1.2.
3.2 Troubleshooting Tools
Network Monitor or Message Analyzer
Download
http://www.microsoft.com/en-us/download/details.aspx?id=40308
Blog
http://blogs.technet.com/b/messageanalyzer/
Microsoft Message Analyzer Operating Guide
http://technet.microsoft.com/en-us/library/jj649776.aspx
Performance Monitor
http://technet.microsoft.com/en-us/library/cc749249.aspx
DiskMon
http://technet.microsoft.com/en-us/sysinternals/bb896646.aspx
Process Monitor
http://technet.microsoft.com/en-us/sysinternals/bb896645