Migrating Azure table storage content to On prem SQl database

hi
is it possible to import azure table storage data to On prem Sql database ?

Hi
You cannot do it directly from SQL or Azure Storage Explorer.
But you can have a little application that extracts your table storage data into some CSV file like this:
http://blogs.msdn.com/b/jmstall/archive/2012/08/03/converting-between-azure-tables-and-csv.aspx
And then import the CSV file that you generate into your on-prem sql database
Regards
Aram
Aram Koukia

Similar Messages

Azure table storage design for simple social networking

What is the best table design if I want to use Azure Table Service for a simple social networking website?
The website could have millions of users.
Users need to be able to view a list of all other users in the system sorted by the number of mutual connections.
Users must be able to view a list of their connections
User must be able to view content posted by themselves and their connections.
One major design constraint is that Azure table service queries are generally limited to the partition key and row key when there are a large number of records or else they get really slow. Another constraint is that query results are only sorted by the
partition key and then the row key.

For your scenario, I think making use of the SQL Azure makes more senses than usage of azure table storage service offering, nature of the data looks more relational in this particular context which is not recommended for table storage model design.
You can get started with SQL Azure at -
http://azure.microsoft.com/en-us/services/sql-database/
Bhushan | Blog |
LinkedIn | Twitter

How to improve performance for Azure Table Storage bulk loads

Hello all,
Would appreciate your help as we are facing a challenge.
We are tried to bulk load Azure table storage. We have a file that contains nearly 2 million rows.
We would need to reach a point where we could bulk load 100000-150000 entries per minute. Currently, it takes more than 10 hours to process the file..
We have tried Parallel.Foreach but it doesn't help. Today I discovered Partitioning in PLINQ. Would that be the way to go??
Any ideas? I have spent nearly two days in trying to optimize it using PLINQ, but still I am not sure what is the best thing to do.
Kindly, note that we shouldn't be using SQL/Azure SQL for this.
I would really appreciate your help.
Thanks

I'd think you're just pooling the parallel connections to Azure, if you do it on one system. You'd also have a bottleneck of round trip time from you, through the internet to Azure and back again.
You could speed it up by moving the data file to the cloud and process it with a Cloud worker role. That way you'd be in the datacenter (which is a much faster, more optimized network.)
Or, if that's not fast enough - if you can split the data so multiple WorkerRoles could each process part of the file, you can use the VM's scale to put enough machines to it that it gets done quickly.
Darin R.

How to select data from AZure table storage without row key and partition key

Hi
I need to select a data from azure table storage without rowkey and partition key. how in azure storage emulator click query it display all data from that table.
thanks
rajesh

Hi rajesh,
It seems that you didn't click query data using storage emulator. But I recommend you could use the azure server explore in your VS to view your data and query data. Please see this document (http://msdn.microsoft.com/en-us/library/azure/ff683677.aspx).
And base on my experience, you may need input the command on Azure storage emulator, such as this page(http://msdn.microsoft.com/en-us/library/azure/gg433005.aspx).
Regards,
Will
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

How can I transfer a XML file content to a MS SQL database by stored procedure using LabWindows/CVI SQL Toolkit?

Hi,
I have a problem to transfer a XML file content to a MS SQL database by a given/fixed stored procedure. I'm able to transfer the content of the file by using following method ...
hstmt = DBPrepareSQL (hdbc, EXEC usp_InsertReport '<Report> ..... </Report>');
resCode = DBExecutePreparedSQL (hstmt);
resCode = DBClosePreparedSQL (hstmt);
... but in this case I'm not able to fetch the return value of the stored procedure!
I have tried to follow the example of the stored procedure in the help documentation (DBPrepareSQL) but I miss a datatype for xml?!?
Any idea how to solve my problem?
KR Cake
Solved!
Go to Solution.

After some additional trials I found a solution by calling the stored procedure in this way
DBSetAttributeDefault (hdbc, ATTR_DB_COMMAND_TYPE, DB_COMMAND_STORED_PROC);
DBPrepareSQL (hdbc, "usp_InsertReport");
DBCreateParamInt (hstmt, "", DB_PARAM_RETURN_VALUE, -1);
DBCreateParamChar (hstmt, "XMLCONTENT", DB_PARAM_INPUT, sz_Buffer, (int) strlen(sz_Buffer) + 1 );
DBExecutePreparedSQL (hstmt);
DBClosePreparedSQL (hstmt);
DBGetParamInt (hstmt, 1, &s32_TestId);
where sz_Buffer is my xml file content and s32_TestID the return value of the stored procdure (usp_InsertReport(@XMLCONTENT XML))
Now I face the problem, that DBCreateParamChar limits the buffer size to 8000 Bytes.
Any idea to by-pass this shortage??

Azure table storage rest API including

How do I access my table storage using REST API.
Any example would be appreciated including enabling REST API.

Hi,
Please have a look at this article:
http://blogs.msdn.com/b/tconte/archive/2011/08/10/accessing-windows-azure-blob-storage-using-jquery.aspx, hope it helps. We could also consider use
Jquery to call codebehind to do some operation about Azure table storeage, if so we could choose azure SDK or azure storage rest API to do this.
Best Regards,
Jambor
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Azure Table Storage Unable to connect - Error on increasing load

I am using Windows Aure Website ( Reserved Instance) and doing the Inserts to an Azure table via http. ( Yes http). When i was doing load testing, foud the below error. It was like 2 hits / seconds... Not sure what is the reason?
Any thoughts? SDK issue?
HTTP_Request2_ConnectionException: Unable to connect to tcp://innovativetxtstorage.table.core.windows.net:80. Error: php_network_getaddresses: getaddrinfo failed: No such host is known.
C:\DWASFiles\Sites\innovativetxt\VirtualDirectory0\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 324
>HTTP_Request2_SocketWrapper->__construct('tcp://innovative…', 10, Array)</td><td>C:\DWASFiles\Sites\innovative in C:\DWASFiles\Sites\innovativetxt\VirtualDirectory0\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\SocketWrapper.php
on line 116

Here is further information ...
In normal circumstance with 30 Hits per Minute to our our API.... It works fine. After that latency is too high that it gives time out. See the Attached Report..
All I am doing is reading the Azure table to verify user Loginand Password... ( Reading Same Entity Again and Again)
See teh Performance Test Report here... http://www.innovativetxt.com/BlitzPerformanceTest.pdf
My customer cannot send us more than 30 request per seconds.. we feel ourself in trouble by complains every day.
We are on reserved Instance Website URL is http://www.innovativetxt.com and we feel like latency is killing us.
1. Is it because of Aure Storage? Cannot handel load of that much?
Is it bec
( I have already turend of Tanggling, TCP 100 etc all tips implemented )
See the below error message from Log file.
[03-Aug-2013 09:37:01 UTC] PHP Fatal error: Uncaught <table style="border: 1px" cellspacing="0">
<tr><td colspan="3" style="background: #ff9999"> HTTP_Request2_ConnectionException: Unable to connect to tcp://innovativetxtstorage.table.core.windows.net:80. Error: php_network_getaddresses: getaddrinfo failed:
No such host is known. in C:\DWASFiles\Sites\innovativetxt\VirtualDirectory0\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\Adapter\Socket.php on line 324</td></tr>
<tr><td colspan="3" style="background-color: #aaaaaa; text-align: center; font-weight: bold;">Exception trace</td></tr>
<tr><td style="text-align: center; background: #cccccc; width:20px; font-weight: bold;">#</td><td style="text-align: center; background: #cccccc; font-weight: bold;">Function</td><td style="text-align:
center; background: #cccccc; font-weight: bold;">Location</td></tr>
<tr><td style="text-align: center;">0</td><td>HTTP_Request2_SocketWrapper->__construct('tcp://innovative…', 10, Array)</td><td>C:\DWASFiles\Sites\innovative in C:\DWASFiles\Sites\innovativetxt\VirtualDirectory0\site\wwwroot\vendor\pear-pear.php.net\HTTP_Request2\HTTP\Request2\SocketWrapper.php
on line 116

EKM using the Azure Key Vault is now available for SQL Database and SQL Server running in Azure VM's

In preview today, you can create keys in the Azure Key Vault, and use them with Azure SQL Database, or SQL Server running in a Azure VM. Use the Extensible Key Management (EKM) for TDE, backup encryption, or cell level encryption. For more information, see
Extensible Key Management Using Azure Key Vault (SQL Server)
http://msdn.microsoft.com/en-us/library/dn198405.aspx.
The announcement:
Azure Key Vault in public preview
Key Vault offers an easy, cost-effective way to safeguard keys and other sensitive data used by cloud applications and services. Included are the following features:
Enhance data protection and compliance:
Protect cryptographic keys and sensitive data like passwords with keys stored in Hardware Security Modules (HSMs). For added assurance, import or generate your keys in HSMs certified to FIPS 140-2 level 2 and Common Criteria EAL4 standards,
so that keys stay within the HSM boundary. Key Vault is designed so that Microsoft doesn’t see or extract your keys.
All the control, none of the work:
Provision new vaults and keys in minutes and centrally manage keys, sensitive data, and policies. You maintain control over your encrypted data—simply grant permission for your own and third-party applications to use keys as needed. Enable
developers to easily manage keys used for dev/test and migrate seamlessly to production keys managed by security operations.
Boost performance and achieve global scale: Improve
performance and reduce latency of cloud applications by storing cryptographic keys in the cloud (versus on-premises). Key Vault rapidly scales to meet the cryptographic needs of your cloud applications and match peak demand.
Get started with Azure Key Vault by creating keys for applications you develop,
SQL Server encryption (TDE, CLE, and Backup), and partner solutions like
CloudLink SecureVM.
Key Vault is available now at no charge with discounted preview pricing starting on January 15, 2015.
For more information, please visit the
Key Vault webpage. For a comprehensive look at pricing, please visit the
Key Vault Pricing webpage.
Rick Byham, Microsoft, SQL Server Books Online, Implies no warranty

Thank you for sharing this Rick.
Regards,
Alberto Morillo
SQLCoffee.com

Using one azure table storage account for many customers with their own data

I'm developing app that will allow the customers to store their data in azure. However, currently I have no idea how to split accounts of the customers in azure. Yes, I'm just started to read the documentation, but maybe someone can point me to the
right topic?

It seems like it might be worth starting from the general guidance to developing multitenant cloud applications - this resource might help: http://msdn.microsoft.com/en-us/library/ff966499.aspx
The patterns covered in this guidance might apply to data storage mechanism chosen for the application - whether it's Azure Storage , Azure SQL DB or else.

Table only pulls 1st row from SQL database?

Hi,
I have created a table in Livecycle Designer ES that pulls in data from a SQL Server table with 2 rows, see table.gif attached..
It only ever pulls in the first row and doesn't repeat for all rows in the SQL table.
How can this be done?
Thank you for your time,
Ollie.

Rich -
The property loader step type does not look at data stored in the database as ordered. It assumes that anything in the recordset that meets the criteria specified on the filtering tab is importable. Whatever locals are defined within that filter are automatically applied and if two records apply to the same variable, then the last record applied wins.
Keep in mind that the locals variable scope applies to all steps in the sequence so to define a set of local values for one step and another for a different step is not quite what we had in mind for the step type. The idea is to import properties before executing any steps in the sequence and then run the sequence. Importing properties between steps goes beyond the intended scope of the step, but
it can be done if you limit the selected records using the filtering tab.
Hope this helps...
Scott Richardson - NI
Scott Richardson
National Instruments

Is this the best design for asynchronous notifications (such as email)? Current design uses Web Site, Azure Service Bus Queue, Table Storage and Cloud Service Worker Role.

I am asking for feedback on this design. Here is an example user story:
As a group admin on the website I want to be notified when a user in my group uploads a file to the group.
Easiest solution would be that in the code handling the upload, we just directly create an email message in there and send it. However, this seems like it isn't really the appropriate level of separation of concerns, so instead we are thinking to have a separate
worker process which does nothing but send notifications. So, the website in the upload code handles receiving the file, extracting some metadata from it (like filename) and writing this to the database. As soon as it is done handling the file upload it then
does two things: Writes the details of the notification to be sent (such as subject, filename, etc...) to a dedicated "notification" table and also creates a message in a queue which the notification sending worker process monitors. The entire sequence
is shown in the diagram below.
My questions are: Do you see any drawbacks in this design? Is there a better design? The team wants to use Azure Worker Roles, Queues and Table storage. Is it the right call to use these components or is this design unnecessarily complex? Quality attribute
requirements are that it is easy to code, easy to maintain, easy to debug at runtime, auditable (history is available of when notifications were sent, etc...), monitor-able. Any other quality attributes you think we should be designing for?
More info:
We are creating a cloud application (in Azure) in which there are at least 2 components. The first is the "source" component (for example a UI / website) in which some action happens or some condition is met that triggers a second component or "worker"
to perform some job. These jobs have details or metadata associated with them which we plan to store in Azure Table Storage. Here is the pattern we are considering:
Steps:
Condition for job met.
Source writes job details to table.
Source puts job in queue.
Asynchronously:
Worker accepts job from queue.
Worker Records DateTimeStarted in table.
Queue marks job marked as "in progress".
Worker performs job.
Worker updates table with details (including DateTimeCompleted).
Worker reports completion to queue.
Job deleted from queue.
Please comment and let me know if I have this right, or if there is some better pattern. For example sake, consider the work to be "sending a notification" such as an email whose template fields are filled from the "details" mentioned in
the pattern.

Hi,
Thanks for your posting.
This development mode can exclude some errors, such as the file upload complete at the same time... from my experience, this is a good choice to achieve the goal.
Best Regards,
Jambor
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.

Make the relationship in between multiple table storage's tables will affect the performance

Hi,
i'm going to develop business application,the product ID needs to be generic one and it should automatically generate the unique id(like identity in sql ) but,it has to be generate in formatted way
for example the ID would be "cityCode+areaCode+uniqueNumber" . here, cityCode and areaCode are going to maintain in separate table. while generate the product id, going to find the cityCode table and AreaCode table the generate unique
number by merge all the respective information.
1) while doing all this will affect the performance Azure table storage performance and web application ?
2) making multiple relationship among multi-Table Storage will decrease the performance ?.

Hello,
When you say tables, are referring to Azure Storage Tables or Relational Databases?
Please note Windows Azure tables do not function in the same manner as tables in a relational database since they do not make use of relationships or have schemas.
And if you are referring to relational databases, the latency in performance would depend on the logic used to generate the unique ID.
You should be able to use the logic in an On-Prem SQL database and check for the latency.
Regards,
Malar.

Max size of Azure table used in single ISS instance

Hi,
what is max size of an Azure Table used in a single
ISS instance?
Regards

Hi,
Please have a check on the below links.
http://blogs.msdn.com/b/avkashchauhan/archive/2011/11/30/how-the-size-of-an-entity-is-caclulated-in-windows-azure-table-storage.aspx
http://blogs.msdn.com/b/windowsazurestorage/archive/2010/07/09/understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx
http://msdn.microsoft.com/en-us/library/azure/jj553018.aspx
Hope this helps.
Regards,
Mekh.

Importing Azure table Data to Hive

Hi All,
Have created a table in Azure. Can see the same in Azure Storage explorer under tables section.
I want to import this table's data to Hive.
For the blob/csv/text files we will use the syntax
CREATE EXTERNAL TABLE <tablename>(col1 string , col2 string)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
LOCATION '/xxxx/xxxx/filename';
what's the syntax for importing the Azure table to Hive.
Please help.
Sriman N Vangala

Hi Sriman,
Unfortunately, Microsoft does not have a Azure table storage handler for HDInsight. It is in the roadmap for future but there is no specific dates on when it will be publically available. One of our product team members has a blog on how to achieve
this using a connector available from github. Please refer to that blog here:
http://blogs.msdn.com/b/mostlytrue/archive/2014/04/04/analyzing-azure-table-storage-data-with-hdinsight.aspx
The set up process in the blog is not very straight forward and it requires some extra config steps as there is no out of the box solution at present. Also, the ATS Hive connector is a community driven project, not a part of HDInsight right now which means
it is not officially supported if you run into any problems with it later on.
One easier alternative would be to get the data out of Azure Tables into CSV in your storage account and process it normally. A lot of folks who have built service telemetry pipelines are doing so.
Hope this helps.
DebarchanS - MSFT

Azure Tables or SQL Azure?

I am at the planning stage of a web application that will be hosted in Azure with ASP.NET for the web site and Silverlight within the site for a rich user experience. Should I use Azure Tables or SQL Azure for storing my application data?

Hi Carol,
Before we choose the Azure Table Or SQL Azure for storing
data, we need to know their differences and
Scenarios:
First ,Azure SQL Database is a relational database service that extends core SQL Server capabilities to the cloud.
While
Azure Table Storage is a fault-tolerant, ISO 27001 certified
NoSQL key-value store. It stores structured data
without schemas, it does not provide any way to represent relationships between the data.
Second, If your application stores and retrieves large data sets that do not require rich relational capabilities, Azure Table Storage might be a better choice. If your application requires data processing
over schematized data sets and is relational in nature, Azure SQL Database might better suit your needs.
In fact There are
several other of factors you should consider when choose which one to store data ,please refer to the links below for more information:
http://msdn.microsoft.com/en-us/library/azure/jj553018.aspx
http://msdn.microsoft.com/en-us/magazine/gg309178.aspx
Best Regards,
Kevin Shen.

Migrating Azure table storage content to On prem SQl database

Similar Messages

Maybe you are looking for