Fetch data from Azure table is quite slow

Question

We have a table on Azure as ID as partition key. We need to fetch 100 thousands IDs in one go. So we are running multiple threads with 10-20 IDs in one thread to access data, but it is taking approx 2 minutes to fetch it.
We are using dotnet application to communicate to Azure.
Following is the C# code-
private void FillDataFromAzure()
            DataTable transactionsList;
            DTTransaction transaction;
            try
ID, TID is fetched from database in dtTransactionsForAzure table before folowing code.
                List<Thread> threads = new List<Thread>();
                int tidCount = 0;
                string tidList = null;
                foreach (DataRow row in dtTransactionsForAzure.Rows)
                    if (tidCount < 30)
                        string tid = row["TransactionID"].ToString();
                        string id = row["ID"].ToString();
                        tidList = (tidList == null) ? tid + "&" + id : tidList + "," + tid + "&" + id;
                        tidCount++;
                    else
                        Thread thread = new Thread(new ParameterizedThreadStart(this.ParallelInvokeForTIDs));
                        thread.Priority = ThreadPriority.Lowest;
                        thread.Start(tidList);
                        threads.Add(thread);
                        tidList = null;
                        tidCount = 0;
                if (tidList != null)
                    Thread thread = new Thread(new ParameterizedThreadStart(this.ParallelInvokeForTIDs));
                    thread.Start(tidList);
                    threads.Add(thread);
                // waiting for all threads to complete
                foreach (var thread in threads)
                    thread.Join();
            catch (Exception e)
                Response.Write("Error occured in FillDataFromAzure " + e.ToString());
private void ParallelInvokeForTIDs(object transactionIDs)
            try
                string tids = (string)transactionIDs;
                #region Create array of Action delegate for paralell invoke
                List<Action> actionDelegates = new List<Action>();
                Action actionDelegate = null;
                foreach (string tid in tids.Split(",").ToArray())
                    string[] strParameters = tid.Split("&").ToArray();
                    string strTid = strParameters[0];
                    string strId = strParameters[1];
                    actionDelegate = delegate()
                        GetRecordsFromAzureByTID(strTid, strId);
                    actionDelegates.Add(actionDelegate);
                Action[] arrDelegates = actionDelegates.ToArray();
                #endregion
                // Parrallel Invoke for multiple TIDs to get data from azure using TPL
                Parallel.Invoke(arrDelegates);
            catch (Exception ex)
                AppLog.Log("Error occured in ParallelInvokeForTIDs " + ex.ToString());
private void GetRecordsFromAzureByTID(string strTid, string id)
            try
                string partitionKey = strTid;
                string rowKey = id;
TransactionEmailSent objTransactionEmailSentBounceBack;
                if (_azureTableHelper == null)
                    //_azureTableHelper = new AzureTableHelper(DTIslandSettings.ConnectionStringAzureStorage);
                if (_azureTableHelper.GetEntity<TransactionEmailSent>("TransactionEmailSent", partitionKey, rowKey, out objTransactionEmailSentBounceBack))
                    if (objTransactionEmailSentBounceBack != null)
                        if (objTransactionEmailSentBounceBack.IsHardBounce == true)
                            AddtoDatatable(objTransactionEmailSentBounceBack.TransactionID, objTransactionEmailSentBounceBack.BounceMessageId, true);
                        else if (objTransactionEmailSentBounceBack.IsSoftBounce == true)
                            AddtoDatatable(objTransactionEmailSentBounceBack.TransactionID, objTransactionEmailSentBounceBack.BounceMessageId, false);
            catch (Exception ex)
                AppLog.Log("Error occured in GetRecordsFromAzureByTID " + ex.ToString());
private void AddtoDatatable(string partitionKey, int? bounceMessageId, bool isHardBounce)
Add data into a datatable
Atul

Answer 1

Hi,
Do you mean 100 or 100 thousands? If there're 100,000 items, with your current code, you're trying to create 100,000 / 30 = 3333 threads. IMHO, that's too much for the OS to handle efficiently, please try to include more items in a single request. It is
true that concurrent requests can improve performance sometimes, but too many small concurrent requests will drop performance, especially so many concurrent requests can be made anyway due to network limitation. From my experience, there's no specific
recommendations for the number of concurrent requests, you can start with something small (such as 2 threads, each requesting 50,000 items), gradually add concurrent requests, till the performance no longer improves.
Best Regards,
Ming Xu
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.