Office 365 Streaming Notifications, "One or more subscriptions in the request reside on another Client Access server."

Question

Hello all,
I am maintaining a part of our product that requires monitoring mailboxes for events.  This is currently being done by using streaming connections for getting the notifications.  Our solution has been successful for situations with smaller numbers
of mailboxes, ~200 or less.  However we are seeing some issues when scaling up to say, 5000 mailboxes.
The error and the sequence leading up to it are as follows:
Make an Exchange Service Account.
exchSvc.ConnectionGroupName = someGroupName;
add to the httpheaders ("X-AnchorMailbox", userSmtp) and ("X-PreferServerAffinity", "true");
create a new impersonated UserId for the userSmtp address that is our anchor mailbox.
set the Exchange Service account ImpersonatedUserID to the one we just made.
ExchangeServiceAccount.SubscribeToStreamingNotifications(new FolderId[] { WellKnownFolderName.Inbox }, _mailEvents);
to this point everything was successful, saw no error messages.
we create a second impersonated UserID for a different mailbox, and repeat the process above from that step forward.  Upon the final step, subscribing to the streaming notifications we get the error:
Exception: Microsoft.Exchange.WebServices.Data.ServiceResponseException: One or more subscriptions in the request reside on another Client Access server. GetStreamingEvents won't proxy in the event of a batch request.
This is only the second subscription that we are trying to add to this connection, and it is to a different mailbox than the first.
Can anyone please help point me to where this is going wrong?

Answer 1

>> Is there a good way to verify the number of subscriptions in a group?
Not that I know of you should be tracking this in your code there are no server side operations in EWS to even tell you if there are active subscriptions on a mailbox.
>>The error I am getting is on the second subscription in a new group, just after doing the anchor mailbox so I don't think we are hitting the 200 limit. 
It's hard to say without seeing your code but it sounds like there is problem with your grouping code. One way to validate this is that with every request you make with the EWS managed API there is a
RequestId header http://blogs.msdn.com/b/exchangedev/archive/2012/06/18/exchange-web-services-managed-api-1-2-1-now-released.aspx
you should be able to give that RequestId to the Office365 support people and they should be able to check the EWS Log on the server and tell you more about what's happening (it maybe server side bug). Something doesn't quite add up in that the X-BackEndOverrideCookie
is what ultimately determines what server the request ends up at and the error is essentially telling you its ending up at the wrong server (have you looked at the headers on the error message?). Is it always one group of users that fails have
you tried different groups and different combinations etc.
Cheers
Glen