OLT Ramp down errors

Question

I have a script that ramps up to 750 users run on 3 agent machines, it includes a sync point so all agents start actions together - performs some navigation and then completes.
Looking at the session graphs all is fine the 750 users do their stuff and then start to end - and then approx 1/3 through the ramp down the errors begin- not on all the agents but on a max of 14 of the 750
Most of the errors are cannot connect to server on port 80 WinInet error code 12029 with additional timeouts after 120 seconds reported
Nothing obvious in the Agent or controller logs
Running the same scenario for 1500 users makes the errors happen earlier (before max ramp up) but again the majority on ramp down - so I can't be certain if the ramp down timing is a bit of a Red Herring
Has anyone seen similar on their sessions?

Answer 1

Are the users only performing a certain number of iterations of the script each, or are they running for a certain time before the test stops them? The way you describe this it could be interpreted as each user only completes a single transaction and then stops (while other VUs are still being added)? Do you actually get all 750 users running at the same time (or have most finished by the time the 750th is launched)?
How are you adding the users? I'd go for a slow stepped approach e.g. 50 users every 10 minutes (or something along those lines appropriate to the length of time for a single iteration). It really depends on how many samples you want to take at each step. i.e. only 5 completed iterations at every step isn't particularly good, but 100 might be OK?
That way it should be clear that the system is fine at lower levels (say 100 samples taken with 450 users) but at 500 users things start to struggle.
Another approach might be to run 1 hour tests at 200 users, then another at 400, then a 3rd at 600 (to get decent samples at each level). It does depend a lot on the average length of a single iteration though.
I'm always interested to hear how other people approach this sort of thing.