SQL connection errors when stress testing mobile service: SQL failure after 3 attempts

Question

I have created a Mobile Service (node.js), which has been running for a few months with 50-100k api calls per day with no issues.
The usage of the service will be scaled up soon, and it will receive much higher traffic, so I am running stress tests (on a staging deployment of the same service).
The api in question queries data from the sql database that is attached to the mobile service, but caches the resulting data in a redis database for 5 minutes, so each call with identical parameters will not access the SQL database, just the redis cache.
My first test never has any problems:
Run a single api calls as fast as possible using the program wrk (https://github.com/wg/wrk). This test never has a problem, and I have run it an hour continuously with a sustained rate of 200 req/s and very few timeout errors.
The second test always causes problems:
I created a list of 5 different api calls to the same service (same endpoint, but different parameters), and I run those randomly interweaved as fast as possible. I use the program locust (http://locust.io) for this, spawing 10.000 clients at a rate of 500
per second. For the first few seconds everything goes ok, but then connections start failing massively.
I get mostly:
Error: SQL failure after 3 attempts. Error: [Microsoft][SQL Server Native Client 10.0]TCP Provider: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected
host has failed to respond.
 (SqlState: 08001, Code: 10060)
And I see this in the http response:
ConnectionError(ProtocolError('Connection aborted.', gaierror(8, 'nodename nor servname provided, or not known')),)
The strange thing is that since I am running the same five calls repeatedly, only the very first of each call will actually access the sql database, every subsequent call will just return data directly from redis, and not try to access the sql database at
all.
In the azure console for the sql database in question, I see 1860 successful connections and 11 failed connections, which does not explain the massive failure rate I am seeing.

Answer 1

Thanks for sharing your tips on this forum, as it will be useful for other users, that run into this issue.
Feedback like this will also help us improve error reporting in the next release, as missing class files in the packaged application is a common issue when developers are moving from the embeded oc4j instance packaged with JDewveloper to standalone instances.
-Eric