App-V 5 Full Infrastructure Apps take long time to stream to the client

Question

Hi was wondering if anyone has the same issue as i am or knows a fix for this, below is my problem and the troubleshooting i have done.
Overview of problem
App-V 5 apps delivered via App-V 5 full infrastructure take a long time to stream to the client and this means the user has to wait if they try and run an application before it has streamed to the client. Users sometimes have to
wait 2 or 3 minutes for an application to stream and this is about 40 times slower than basic SMB and HTTP transfer tests show the system is capable of (see performance results below).
App-V 4.6 apps delivered via App-V 4.6 full infrastructure and HTTP streaming are fine.
Overview of environment
App-V 5.0 SP1 Full Infrastructure.
App-V servers are running Server 2012 on Hyper-V 3 or ESX 5.1 with 2 x vCPU and 4GB RAM.
SQL servers are a SQL 2012 cluster.
Separate servers for SQL, management, publishing, content and reporting.
Management, Publishing and Content servers have two servers per role and NLB to provide load balancing. So 7 servers (2 x Man, 2 x Pub, 2 x Content, 1 x Reporting)
Two further sites with 2 x Pub and 2 x Content each. All publishing servers pointed at the load balance address for management.
Content delivered via HTTP
Clients are physical desktops and laptops running Windows 7 SP1 x86 and Windows 8 x86
App-V client is 5.0SP1
Clients are pointed at their nearest publishing server NLB via a script which looks up the client IP address and uses PowerShell to configure the publishing server
Content is streamed from the nearest content server NLB by setting the PackageSourceRoot to the nearest content NLB (via the same PowerShell script above).
App-V apps delivered per-user via AD group. One AD group per application. Approximately 200 App-V apps published so far - will eventually reach 400 as we sequence more. About 9000 users.
Analysis performed so far
Servers not heavily loaded. CPU averages 5%. Lots of RAM free. Very low disk IO. Problem also occurs out-of-hours so we are 99.9% certain that server resources are not a cause.
Streaming performance is the same from all 6 content servers and all 3 NLB addresses (tested by changing the value of PackageSourceRoot). Wireshark used to confirm packages are really streaming from the correct location, enforcing
our belief that the problem isn't at the server end (unless all 6 servers are affected).
Streaming via both HTTP and SMB2.1 is approximately the same (tested by changing the value of PackageSourceRoot between http://xxxx and \\server\AppVContent).
Wireshark used to confirm we really are using the protocol we think we are using.
All clients exhibit the same behaviour. Issue reported by many users. 5 test PCs chosen at random at all 3 sites confirmed to have the slow streaming problem.
Slow streaming from both Hyper-V and VMware ESX servers.
Client not heavily loaded.
Affects all App-V apps although it obviously affects the larger ones more.
All App-V apps have a Feature Block 1 setup.
If we copy the ".appv" file from the server to the client via either HTTP or SMB then it's reasonably quick (up to 480Mb/s). So we don't believe the network or servers are at fault. For example:
We can copy a 149MB .appv file via SMB from the content server to the client in 5 seconds.
We copy HTTP download the .appv file from the content server using IE on the client in 5 seconds.
But if you ask the App-V 5 client to fully download the sequence then it takes 2 - 3 minutes.
The App-V 4.6 client takes about 8 - 10 seconds to fully download a similar sized application.
App-V 5 publishing works fine - when a new user logs on they get their list of application straight away, it's just the streaming which is slow.
Once the App-V app has streamed locally it runs fine and with a decent performance.
Looking at a Wireshark trace of the streaming you can see that the slow performance is due to the transfer stopping and starting a lot. You only notice this when you zoom into the performance graph a fair bit.
Each time the HTTP server stops sending traffic, it doesn't start again until the client sends a "TCP Window update". Each "stop" is of a different length, but just taking a few from the middle I get 0.06s, 0.11s,
0.13s wasted etc.
I can see that it's the client stopping the transfer by reducing its advertised TCP Window Size. I'll provide an example:
Server sends 9 x 1514 bytes. Client responds with an ACK and a Window size of 54016 bytes (256x211)
Server sends 11 x 1514 bytes. Client responds with an ACK and a Window size of 37888 bytes (256x148)
Server sends 10 x 1514 bytes. Client responds with an ACK and a Window size of 23296 bytes (256x91)
Server sends 15 x 1514 bytes. Client responds with an ACK and a Window size of 1280 bytes (5 x 256)
Server stops sending (I'm guessing because the client advertised Window size was less than a single packet's worth of bytes)
<0.1 seconds passes>
Client sends a "TCP Window Update" re-advertising a TCP window size of 65536 (256x256).
Server starts transmitting again
So the way I see this is that the App-V 5 client is controlling the transfer speed by utilising TCP Window flow control. The trace was taken at the client end so there's no room for anything on the network to be fiddling with flow
control (and we've confirmed there are no traffic shapers in the loop).
We've also tried streaming directly from the local client by copying some App-V 5 apps down to the client, creating a SMB share on the client and changing PackageSourceRoot to \\localhost\AppVContent (i.e. so we are streaming directly
from the client to the client - to remove the network from the equation) and there is only an improvement of 5 to 10 seconds. So we know it's nothing to do with the network or the servers.
We've tried turning off TCP auto-tuning on the client with:
netsh interface tcp set global autotuninglevel=disabled
and turning off TCP chimney offloading (which is off anyway because the NIC doesn't support it and Netstat -t output shows "InHost" for offload state for all connections) with:
netsh int tcp set global chimney=disabled
and nothing has improved.
So we've now focussed on the extraction of the .appv (ZIP) file on the client.
Using Windows Explorer it takes 75 seconds to extract the ZIP file
Using 7ZIP it takes 9 seconds to extract the ZIP file
Yeah we've always known that the Explorer ZIP engine is terrible. That's why we use 7ZIP or WinRAR on our clients.
So we've started to wonder if the problem with the slow App-V 5 streaming is because the client is downloading the .appv file and extracting it as it goes along in a single thread. If the App-V 5 client is using the same terrible
ZIP engine that Explorer does then that would explain the slow performance. The "download" appears to take a long time because the client is using TCP flow control to slow the transfer since it's extracting the .appv file using a very slow ZIP engine
and it's all in a single thread.

Answer 1

Guys,
Just wanted to give you a brief update basically close this thread as Answered.
We had submitted 4 App-V 5 Bugs to Microsoft and these were reproducible and an explanation was given on work around to them. Microsoft
sent down a App-v developer to have a look at our problems. They said they will try and include the Bug fixes in SP2 which should be out in a few weeks or they will definitely be included in SP3.
In regards to the slow streaming it all came down to the Disk IO.
We found that you could simply enable "Turn off
Windows write-cache buffer flushing on the device", then start streaming the app and then disable "Turn
off Windows write-cache buffer flushing on the device" immediately after
(we don't want to leave it on) and that basically fixed the issue.
But a normal user would not have permissions to do
this, so a code was written to enable and disable this option.
Apology for not going in detail, like my opening thread, its very late. 
but if you would like a detailed analysis please message me.
I would like to Thank the Talented Consultant who designed and implemented are App-V infrastructure who found the bugs and created all
the work around and who also emailed the detailed analysis of the problems to Microsoft that got them interested.
Simon Bond from Ultima Business Solutions.
Thank you