SQL Azure: More Intermittent Timeouts

Question

Hi guys,
We have a set of 5 online auction systems running on Windows Azure & SQL Azure. Each system consists of a single web worker and one or more web roles. Each system is using ASP.NET MVC 3 and Entity Framework, Repository Pattern and StructureMap.
The worker role is responsible for housekeeping and runs two groups of processes. One group is run every ten seconds, the other every second. Each process will likely run a database query or stored procedure. These are scheduled with Quartz.net
The web role serves the public interface and back office. Among other basic crud functionality, both of these provide screens which, when open, will repeatedly call controller methods which will result in execution of stored procedure read-only queries.
The frequency of repetition is about 2-3 seconds per client. A typical use case would be 5 back office windows open, and 25 end user windows open – all hitting the system repeatedly.
For a long time we have been experiencing intermittent SQL timeout errors. Three of the most common ones are:
System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - The semaphore timeout period has expired.)
System.Data.SqlClient.SqlException: Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding.
The only predictable scenario is during an auction where a specific controller -> sproc starts to timeout during the event (presumably due to load). All other times the errors appear to be completely random and come in singles, two’s, and three’s etc.
even during periods of user inactivity. For example the system will go 18 hours without an error and then could be 5 – 10 errors from different housekeeping methods, or perhaps a user logged on and viewed their account.
Other info:
I have tried to run the affected queries/sprocs on SQL Azure using both local SSMS and Azure web-based query tool – all seem to execute quickly, 1 second max. Query plans not showing anything too suspicious although I am by no means a SQL query performance
expert, or any other kind of expert for that matter
J
We have wrapped all affected areas in Azure SQL Transient Fault Handling Blocks – but as is discussed here
http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted/thread/7a50985d-92c2-472f-9464-a6591efec4b3, they do not catch timeouts, and according to “Valery M” this is for good reason.
We are not storing any session information in the database, although asp.net membership information is stored in the database.
We use 1 “SQL Azure server instance” which hosts all 5 databases, two for staging and three for production. All 5 systems are generally active at the same time although it is unlikely that more than one will be in a state of live load use at any given time.
All web roles, worker roles and the SQL Azure server reside in the same Azure Geographical Region.
Any thoughts on where we should be looking? Would it help giving each system it's own SQL Azure server? ... Failing a solution by ourselves - is it possible to get Microsoft to open a support ticket and take a look under the hood at what’s going on in with
our application – how does one go about this?
Thanks in advance.
Ilan

Answer 1

Ditto.
New website/database (has only been in production for a week or so). The model we have is pretty simple Azure website MVC EF 6.1.1 using a single database in same region as website. I have monitored for and have never found any throttled connections or deadlocks.
What I have tried so far
I bumped the timeout in the connection string to 60 seconds
I changed the web site to Basic and configured the auto scale to max instance size and count
That seemed to help - but then just recently during points of high volume usage (client requests, data in and out, CPU time) I started seeing this error. As I narrow the focus to look at metrics during the exact time of the error it doesn't reflect
the exact peak. For example peak usage may be at 3:45, but the errors occur at 4:30 when the usage is less than 3:45 by a fair amount.
To try to force recreate and see if it was caused by transaction data table locks I ran a query in a transaction block with a WAITFOR delay and then went to the web site and tried to access data that would be locked by that query. That test did generate
an error - but not this specific "semaphore timeout period has expired" - so I don't think it is caused by table locks.
I did notice that when the error happens - it generally happens to one user and then happens multiple times to that one user over the course of what can be a minute or so and in different queries (views). The last analysis showed that pattern for
one user with 2 other users getting the error but just once (and in different views/queries).
USER A - got about 9 of these errors in about 8 different views over the course of a little more than a minute (20:26 to 21:35)
USER B - got 1 (in the same timespan)
USER C - got 1 (in the same time span)
Analysis of the error logs showed it isn't unique to any single view/query or scenario wrt queries. The only clue that makes it less of an intermittent situation is does correspond with usage. However, it doesn't have to be specifically peak usage at the
exact moment - but in a window (within an hour), It makes me wonder if it is the load or just the fact that more usage just increases the odds (I will monitor this relation some more and note the exact metrics). 
I am now bumping the timeout in the connection string to 90 seconds and I have changed the database DTU setting to 50