Why am I getting ExchangeWebServices Inbox Error: Error, ErrorServerBusy. The server cannot service this request right now. Try again later

Question

I recently switched my application that uses EWS from an on-premise Exchange Server to Exchage Online through Office356.
The process worked just fine for several days, then I started getting the following errors;
Error accessing [USERNAME] email account.; ExchangeWebServices Inbox Error: Error, ErrorServerBusy, The server cannot service this request right now. Try again later. --> 
This has been happening for the past 14 hours now. 
I contacted my Office365 support team and they acted like they had never heard of the Exchange Web Services API, so no help there.
I can access the mailbox using the O365 web portal and I can access the mailbox account using the Outlook 2013 desktop client. The issue seems specific to EWS
My program is a Windows service, written in VB.Net. It connects to EWS, goes to the user account inbox, iterates through the inbox extracting attachments from messages, then moves the messages to a saved folder below the inbox.
I created the wrapper for EWS that I can reference in my project code using the following, run from an elevated VS2012 command prompt;
wsdl.exe /namespace:ExchangeWebServices /out:EWS.cs https://outlook.office365.com/ews/services.wsdl /username:[email protected] /password:p@ssw0rd
csc /out:EWS_E2K13_release /target:library EWS.cs
I bind to EWS in my class, using the following code;
Imports System.Net
Imports ExchangeWebServices
Public Class Exchange2013WebServiceClass
    Private ExchangeBinding As New ExchangeServiceBinding
    Public Sub New(ByVal userEmail As String, ByVal userPassword As String, ByVal URL As String)
        ExchangeBinding.Credentials = New NetworkCredential(userEmail, userPassword)
        ExchangeBinding.Url = URL
    End Sub
The error that is logged gets triggered when my code makes a call to the following method;
    Public Function GetInboxMessageIDs() As ArrayOfRealItemsType
        Dim returnInboxMessageIds As ArrayOfRealItemsType = Nothing
        Dim errMsg As String = String.Empty
        'Create the request and specify the travesal type.
        Dim FindItemRequest As FindItemType
        FindItemRequest = New FindItemType
        FindItemRequest.Traversal = ItemQueryTraversalType.Shallow
        'Define which item properties are returned in the response.
        Dim ItemProperties As ItemResponseShapeType
        ItemProperties = New ItemResponseShapeType
        ItemProperties.BaseShape = DefaultShapeNamesType.IdOnly
        'Add properties shape to the request.
        FindItemRequest.ItemShape = ItemProperties
        'Identify which folders to search to find items.
        Dim FolderIDArray(0) As DistinguishedFolderIdType
        FolderIDArray(0) = New DistinguishedFolderIdType
        FolderIDArray(0).Id = DistinguishedFolderIdNameType.inbox
        'Add folders to the request.
        FindItemRequest.ParentFolderIds = FolderIDArray
        Try
            'Send the request and get the response.
            Dim FindItemResponse As FindItemResponseType
            FindItemResponse = ExchangeBinding.FindItem(FindItemRequest)
            'Get the response messages.
            Dim ResponseMessage As ResponseMessageType()
            ResponseMessage = FindItemResponse.ResponseMessages.Items
            Dim FindItemResponseMessage As FindItemResponseMessageType
            If ResponseMessage(0).ResponseClass = ResponseClassType.Success Then
                FindItemResponseMessage = ResponseMessage(0)
                returnInboxMessageIds = FindItemResponseMessage.RootFolder.Item
            Else
                '' Server error
                Dim responseClassStr As String = [Enum].GetName(GetType(ExchangeWebServices.ResponseClassType), ResponseMessage(0).ResponseClass).ToString
                Dim responseCodeStr As String = [Enum].GetName(GetType(ExchangeWebServices.ResponseCodeType), ResponseMessage(0).ResponseCode).ToString
                Dim messageTextStr As String = ResponseMessage(0).MessageText.ToString
                Dim thisErrMsg As String = String.Format("ExchangeWebServices Inbox Error: {0}, {1}, {2}", responseClassStr, responseCodeStr, messageTextStr)
                errMsg = If(errMsg.Equals(String.Empty), String.Empty, errMsg & "; ") & thisErrMsg
            End If
        Catch ex As Exception
            'errMsg = String.Join("; ", errMsg, ex.Message)
            errMsg = If(errMsg.Equals(String.Empty), String.Empty, errMsg & "; ") & ex.Message
        End Try
        If Not errMsg.Equals(String.Empty) Then
            returnInboxMessageIds = Nothing
            Throw New System.Exception(errMsg)
        End If
        Return returnInboxMessageIds
    End Function  
Since the code worked just fine for several days and then suddenly stopped working with a server busy error, I have to think that this is some type of limit or throttling by EWS on the account. I process several thousand emails per day, in chunks of 300
at a time. 
But I have no idea how to check for any limits exceeded. I am nowhere close to my O365 mailbox size limit. Right now, there are over 4,000 messages in my inbox, and growing. 
Thanks in advance for any ideas you can offer.
Dave

Answer 1

All the API's EWS, MAPI, ActiveSync,Remote powershell are throttled on Office365 (based around what 1 particular user could resonably do). If you have had a read of this already i would recommend
http://msdn.microsoft.com/en-us/library/office/jj945066(v=exchg.150).aspx
 You can't adjust or even find your current throttle usage so you have to try to design your code around living inside the default limits. If your using One Service Account to access multiple Mailboxes (or if that account is because used across multiple
applications) that can cause problems. In this case using EWS Impersonation is good solution as described in
http://blogs.msdn.com/b/exchangedev/archive/2012/04/19/more-throttling-changes-for-exchange-online.aspx (this basically means the Target Mailbox is charged instead of the Service Account).
 Looking at the code one thing I notice missing is your don't appear to be paging the results of FindItems, also have versioned your requests to Exchagne2013. eg ". When the value of the
RequestServerVersion element indicates Exchange 2010 or an earlier version of Exchange, the server sends a failure response with error code
ErrorServerBusy. If the value of the RequestServerVersion
element indicates a version of Exchange starting with Exchange 2010 SP1
or Exchange Online, and the client is using paging, EWS may return a
partial result set instead of an error"
To Page FindItems Correctly you should use the IndexedPageViewType class and page the Items at no more the 1000 at a time eg something like
IndexedPageViewType indexedPageView = new IndexedPageViewType();
indexedPageView.BasePoint = IndexBasePointType.Beginning;
indexedPageView.Offset = 0;
indexedPageView.MaxEntriesReturned = 1000;
indexedPageView.MaxEntriesReturnedSpecified = true;
FindItemType findItemrequest = new FindItemType();
findItemrequest.Item = indexedPageView;
findItemrequest.ItemShape = new ItemResponseShapeType();
findItemrequest.ItemShape.BaseShape = DefaultShapeNamesType.IdOnly;
BasePathToElementType[] beAdditionproperties = new BasePathToElementType[3];
PathToUnindexedFieldType SubjectField = new PathToUnindexedFieldType();
SubjectField.FieldURI = UnindexedFieldURIType.itemSubject;
beAdditionproperties[0] = SubjectField;
PathToUnindexedFieldType RcvdTime = new PathToUnindexedFieldType();
RcvdTime.FieldURI = UnindexedFieldURIType.itemDateTimeReceived;
beAdditionproperties[1] = RcvdTime;
PathToUnindexedFieldType ReadStatus = new PathToUnindexedFieldType();
ReadStatus.FieldURI = UnindexedFieldURIType.messageIsRead;
beAdditionproperties[2] = ReadStatus;
findItemrequest.ItemShape.AdditionalProperties = beAdditionproperties;
DistinguishedFolderIdType[] faFolderIDArray = new DistinguishedFolderIdType[1];
faFolderIDArray[0] = new DistinguishedFolderIdType();
faFolderIDArray[0].Mailbox = new EmailAddressType();
faFolderIDArray[0].Mailbox.EmailAddress = "[email protected]";
faFolderIDArray[0].Id = DistinguishedFolderIdNameType.inbox;
bool moreAvailible = false;
findItemrequest.ParentFolderIds = faFolderIDArray;
int loopCount = 0;
do
FindItemResponseType frFindItemResponse = esb.FindItem(findItemrequest);
if (frFindItemResponse.ResponseMessages.Items[0].ResponseClass == ResponseClassType.Success)
foreach (FindItemResponseMessageType firmtMessage in frFindItemResponse.ResponseMessages.Items)
Console.WriteLine("Number of Items retreived : " + ((ArrayOfRealItemsType)firmtMessage.RootFolder.Item).Items.Length);
if (firmtMessage.RootFolder.IncludesLastItemInRange == false)
moreAvailible = true;
else
moreAvailible = false;
((IndexedPageViewType)findItemrequest.Item).Offset += ((ArrayOfRealItemsType)firmtMessage.RootFolder.Item).Items.Length;
Console.WriteLine("Offset : " + ((IndexedPageViewType)findItemrequest.Item).Offset);
if (firmtMessage.RootFolder.TotalItemsInView > 0)
foreach (ItemType miMailboxItem in ((ArrayOfRealItemsType)firmtMessage.RootFolder.Item).Items)
Console.WriteLine(miMailboxItem.Subject);
else
throw new Exception("error " + frFindItemResponse.ResponseMessages.Items[0].MessageText);
} while (moreAvailible);
The support people should be able to help you as long as you can get past the first level. The EWS Managed API has a RequestId header that gets submitted with requests
http://blogs.msdn.com/b/exchangedev/archive/2012/06/18/exchange-web-services-managed-api-1-2-1-now-released.aspx . In theory they should be able to take this and then from the Logs tell more information about why your request failed etc.
Cheers
Glen