Question regarding DocumentDB RU consumption when inserting documents & write performance

Question

Hi guys,
I do have some questions regarding the DocumentDB Public Preview capacity and performance quotas:
My use case is the following:
I need to store about 200.000.000 documents per day with a maximum of about 5000 inserts per second. Each document has a size of about 200 Byte.
According to to the documentation (http://azure.microsoft.com/en-us/documentation/articles/documentdb-manage/) i understand that i should be able to store about 500 documents per second with single inserts and about 1000 per second with a batch insert using
a stored procedure. This would result in the need of at least 5 CUs just to handle the inserts.
Since one CU consists of 2000 RUs i would expect the RU usage to be about 4 RUs per single document insert or 100 RUs for a single SP execution with 50 documents.
When i look at the actual RU consumption i get values i don’t really understand:
Batch insert of 50 documents: about 770 RUs
Single insert: about 17 RUs
Example document:
{"id":"5ac00fa102634297ac7ae897207980ce","Type":0,"h":"13F40E809EF7E64A8B7A164E67657C1940464723","aid":4655,"pid":203506,"sf":202641580,"sfx":5662192,"t":"2014-10-22T02:10:34+02:00","qg":3}
The consistency level is set to “Session”.
I am using the SP from the example c# project for batch inserts and the following code snippet for single inserts:
await client.CreateDocumentAsync(documentCollection.DocumentsLink, record);
Is there any flaw in my assumption (ok…obviously) regarding the throughput calculation or could you give me some advice how to achieve the throughput stated in the documentation?
With the current performance i would need to buy at least 40 CUs which wouldn’t be an option at all.
I have another question regarding document retention:
Since i would need to store a lot of data per day i also would need to delete as much data per day as i insert:
The data is valid for at least 7 days (it actually should be 30 days, depending on my options with documentdb). 
I guess there is nothing like a retention policy for documents (this document is valid for X day and will automatically be deleted after that period)?
Since i guess deleting data on a single document basis is no option at all i would like to create a document collection per day and delete the collection after a specified retention period.
Those historic collections would never change but would only receive queries. The only problem i see with creating collections per day is the missing throughput:
As i understand the throughput is split equally according to the number of available collections which would result in “missing” throughput on the actual hot collection (hot meaning, the only collection i would actually insert documents).
Is there any (better) way to handle this use case than buy enough CUs so that the actual hot collection would get the needed throughput?
Example: 
1 CU -> 2000 RUs
7 collections -> 2000 / 7 = 286 RUs per collection (per CU)
Needed throughput for hot collection (values from documentation): 20.000
=> 70 CUs (20.000 / 286)
vs. 10 CUs when using one collection and batch inserts or 20 CUs when using one collection and single inserts.
I know that DocumentDB is currently in preview and that it is not possible to handle this use case as is because of the limit of 10 GB per collection at the moment. I am just trying to do a POC to switch to DocumentDB when it is publicly available. 
Could you give me any advice if this kind of use case can be handled or should be handled with documentdb? I currently use Table Storage for this case (currently with a maximum of about 2500 inserts per second) but would like to switch to documentdb since i
had to optimize for writes per second with table storage and do have horrible query execution times with table storage because of full table scans.
Once again my desired setup:
200.000.000 inserts per day / Maximum of 5000 writes per second
Collection 1.2 -> Hot Collection: All writes (max 5000 p/s) will go to this collection. Will also be queried.
Collection 2.2 -> Historic data, will only be queried; no inserts
Collection 3.2 -> Historic data, will only be queried; no inserts
Collection 4.2 -> Historic data, will only be queried; no inserts
Collection 5.2 -> Historic data, will only be queried; no inserts
Collection 6.2 -> Historic data, will only be queried; no inserts
Collection 7.2 -> Historic data, will only be queried; no inserts
Collection 1.1 -> Old, so delete whole collection
As a matter of fact the perfect setup would be to have only one (huge) collection with an automatic document retention…but i guess this won’t be an option at all?
I hope you understand my problem and give me some advice if this is at all possible or will be possible in the future with documentdb.
Best regards and thanks for your help

Answer 1

Hi Aravind,
first of all thanks for your reply regarding my questions.
I sent you a mail a few days ago but since i did not receive a response i am not sure it got through.
My main question regarding the actual usage of RUs when inserting documents is still my main concern since i can not insert nearly
as many documents as expected per second and CU.
According to to the documentation (http://azure.microsoft.com/en-us/documentation/articles/documentdb-manage/)
i understand that i should be able to store about 500 documents per second with single inserts and about 1000 per second with a batch insert using a stored procedure (20 batches per second containing 50 documents each). 
As described in my post the actual usage is multiple (actually 6-7) times higher than expected…even when running the C# examples
provided at:
https://code.msdn.microsoft.com/windowsazure/Azure-DocumentDB-NET-Code-6b3da8af/view/SourceCode
I tried all ideas Steve posted (manual indexing & lazy indexing mode) but was not able to enhance RU consumption to a point
that 500 inserts per second where nearly possible.
Here again my findings regarding RU consumption for batch inserts:
Automatic indexing on: 777
RUs for 50 documents
Automatic indexing off &
mandatory path only: 655
RUs for 50 documents
Automatic indexing off & IndexingMode Lazy & mandatory path only:  645 RUs for
50 documents
Expected result: approximately 100
RUs (2000 RUs => 20x Batch insert of 50 => 100 RUs per batch)
Since DocumentDB is still Preview i understand that it is not yet capable to handle my use case regarding throughput, collection
size, amount of collections and possible CUs and i am fine with that. 
If i am able to (at least nearly) reach the stated performance of 500 inserts per second per CU i am totally fine for now. If not
i have to move on and look for other options…which would also be “fine”. ;-)
Is there actually any working example code that actually manages to do 500 single inserts per second with one CUs 2000 RUs or is
this a totally theoretical value? Or is it just because of being Preview and the stated values are planned to work.
Regarding your feedback:
...another thing to consider
is if you can amortize the request rate over the average of 200 M requests/day = 2000 requests/second, then you'll need to provision 16 capacity units instead of 40 capacity units. You can do this by catching "RequestRateTooLargeExceptions" and retrying
after the server specified retry interval…
Sadly this is not possible for me because i have to query the data in near real time for my use case…so queuing is not
an option.
We don't support a way to distribute throughput differently across hot and cold
collections. We are evaluating a few solutions to enable this scenario, so please do propose as a feature at http://feedback.azure.com/forums/263030-documentdb as this helps us prioritize
feature work. Currently, the best way to achieve this is to create multiple collections for hot data, and shard across them, so that you get more proportionate throughput allocated to it. 
I guess i could circumvent this by not clustering in “hot" and “cold" collections but “hot" and “cold"
databases with one or multiple collections (if 10GB will remain the limit per collection) each if there was a way to (automatically?) scale the CUs via an API. Otherwise i would have to manually scale down the DBs holding historic data. I
also added a feature requests as proposed by you.
Sorry for the long post but i am planning the future architecture for one of our core systems and want to be sure if i am on
the right track. 
So if you would be able to answer just one question this would be:
How to achieve the stated throughput of 500 single inserts per second with one CUs 2000 RUs in reality? ;-)
Best regards and thanks again