Large Files from FTP to Azure Storage Account.
We required to transfer files from FTP to Azure Storage Account.
Our FTP contains Almost 4 TB data and in future it will be growing.
We require some automate process which will transfer data from FTP to Windows Azure Storage.
Currently Implemented Solution: We have created
a windows service which will download files from FTP and converting to file stream and with Blob Storage object it will be uploaded to
Storage account.
With the mentioned approach bandwidth is utilized at 2 places
For downloading file from FTP
For Uploading to Azure Storage
Can you please provide any solution/suggestion which will reduce the consumption of bandwidth and large data will transfer in less time
Hi,
Please have a look at below article, it is talking about Blob Transfer Utility tool. Blob Transfer Utility is a GUI tool to upload and download thousands of small/large files to/from Windows Azure Blob Storage.
#https://blobtransferutility.codeplex.com/
Disclaimer: This is not a Microsoft official tool and it is not supported by Microsoft. It's just a (very useful) sample.
Best Regards,
Jambor
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey.
Similar Messages
-
How to Delete Directory from Azure Storage Account?
Hi All,
Currently i am working on Microsoft Windows Azure (SaaS - Application).
I am facing a problem with the AzureStorage.
I want to delete a whole directory from the Azure Storage Account.
For Example : I want to delete following Directory(14).
https://azurestorage.blob.core.windows.net/blobname/Images/14
Any help will be appriciated.
Thanks in Advance.
Rakesh T. Gupta, Web Engineer, Ahmedabad, IndiaHi Rakesh,
We have not heard you in days. Have you managed to write the code to delete a blob directory?
If not, I'd like to share my code for your reference:
public void RemoveBlobDirectory()
string containerName = "blobname";
string directoryName = "blobname/Images/14/";
var storageAccount = CloudStorageAccount.Parse("UseDevelopmentStorage=true");
var blobStorage = storageAccount.CreateCloudBlobClient();
// Ensure the container is exist.
var blobContainer = blobStorage.GetContainerReference(containerName);
blobContainer.CreateIfNotExist();
foreach (IListBlobItem item in blobStorage.ListBlobsWithPrefix(directoryName))
if (item.GetType() == typeof(CloudBlob) || item.GetType().BaseType == typeof(CloudBlob))
((CloudBlob)item).DeleteIfExists();
Please note that the ListBlobsWithPrefix method will scan through all blob entities in order to search the result. So it will potentially have low performance when you have large number of blobs in you storage account.
Thanks,
Wengchao Zeng
Please mark the replies as answers if they help or unmark if not.
If you have any feedback about my replies, please contact
[email protected]
Microsoft One Code Framework -
How do I move large files from a Windows computer to my MacBook Pro? There are a lot of photos (tiffs, jpgs, some photoshopped files), iTunes library, word docs. Is there a particular type of external hard drive that can be used or am I limited to copying stuff onto z-drive and moving that way?
If both computers are on the same network, you can share the files from the Windows host using SMB and mount the share on the Mac. Otherwise, the Mac should be able to read (not necessarily write) any storage device formatted in Windows.
-
Windows Azure Storage Account Location change
Hi Experts,
I have deployed my Web site on Windows azure. I have created Windows Azure Storage Account. By mistakely i have set the location wrong. How can we change the location of Windows storage account?
Thanks!Hello CRM Inn,
I am sorry to know that you have chosen an incorrect location and would like to change the location for your storage account. Unfortunately, there is no option available to change the storage location for an existing storage account.
Your only option would be to create a new storage account, migrate data to the new storage account and delete the existing one. You can refer to this article that will give you information on migrating data from one account to another:
http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx
I also suggest that you create affinity groups. By creating affinity groups you can be sure that all your related services are under the same data center. You can refer to this article to learn about Affinity groups:
http://msdn.microsoft.com/en-us/library/azure/jj156209.aspx
Syed irfan Hussain -
Is there any way to automate purging Application Logging in azure storage account tables
is there any way to automate purging Application Logging in azure storage account tables
Rohit Pasrijahi Rohit,
If you want to delete old data automatically, I think you need develop this feature by yourself. You could code the logic methods on your project, and set a timer to execute the methods to delete old data. Please refer to this thread (http://stackoverflow.com/questions/13602629/is-it-possible-to-acces-a-table-storage-in-azure-from-an-azure-web-site
), you could operate table storage data using azure sdk or REST API.
Please try it.
Regards,
Will
We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click
HERE to participate the survey. -
Error while downloading PDF file from FTP Server
Hi Friends,
I have sent a PDF file with data to FTP , Then i want to check that uploaded pdf file , whether that is correct or not?
for that , i have downloaded that file from FTP and i am trying to open the file . but it is giving this problem .
"There was an error opening the document . The file is damaged and could not be repaired."
will you suggest me regarding this.
thanks in advance.
balaji.T.CALL FUNCTION 'CONVERT_OTFSPOOLJOB_2_PDF'
EXPORTING
SRC_SPOOLID = spoolno
src_spoolid = wa_file-rqident
no_dialog = ' '
DST_DEVICE =
PDF_DESTINATION =
IMPORTING
pdf_bytecount = numbytes
pdf_spoolid = pdfspoolid
OTF_PAGECOUNT =
btc_jobname = jobname
btc_jobcount = jobcount
TABLES
pdf = pdf
EXCEPTIONS
err_no_otf_spooljob = 1
err_no_spooljob = 2
err_no_permission = 3
err_conv_not_possible = 4
err_bad_dstdevice = 5
user_cancelled = 6
err_spoolerror = 7
err_temseerror = 8
err_btcjob_open_failed = 9
err_btcjob_submit_failed = 10
err_btcjob_close_failed = 11.
because of this one PDF internal table is obtained.
OPEN DATASET L_FILENAME FOR OUTPUT in text mode MESSAGE MSG.
LOOP AT pdf.
CONCATENATE pdf-tdformat "Material group
pdf-TDLINE "Basic Material
INTO ITEXT-TLINE ..
APPEND ITEXT.
TRANSFER ITEXT TO L_FILENAME.
ENDLOOP. -
Problem while reading the file from FTP server
Hi Friends,
I have a problem while fetching files from FTP server.
I used FTP_Connect, FTP_COMMAND function modules. I can able to put the files into FTP server.
but I cant able to pick the files from FTP server.
anyone have faced similar issues kindly let me know.
Thanks
GowrishankarHi,
try this way..
for reading the file using FTP you need to use different unix command ..
Prabhuda -
How to read .xls file from FTP server t oInternal table
Hi
am using the FTP_SERVER_TO_R3 to read xls file from FTP server to internal table
but the data i get in LT_TEXT is special characters.
CALL FUNCTION 'FTP_SERVER_TO_R3'
EXPORTING
handle = hdl
fname = f_name "ProdDataFromCRM.xls.
* CHARACTER_MODE = 'X'
* IMPORTING
* BLOB_LENGTH =
TABLES
BLOB = lt_text
* TEXT = lt_text
EXCEPTIONS
TCPIP_ERROR = 1
COMMAND_ERROR = 2
DATA_ERROR = 3
OTHERS = 4
can any one help me out to get the exact data..
Really appreciate your quick response..
Thank YouHi, if you really retrieve an excel file, you can not see the data in ABAP. You may see them in Excel. For this you may use
CALL METHOD document->open_document_from_table
of the interface i_oi_document_proxy for OLE objects. You can access the data with reference to the interface i_oi_spreadsheet.
Please check [Desktop Office Integration (BC-CI)|http://help.sap.com/printdocu/core/Print46c/en/data/pdf/BCCIOFFI/BCCIOFFI.pdf] for details.
But, who knows, perhaps you want "And Now For Something Completely Different".
Regards
Clemens -
Need to copy .txt file from FTP server and downloaded on local server directory.
I need to figure out a way to copy .txt file from ftp server in local server directory using sql jobs.
Below links will help achieving it:
https://www.virtualobjectives.com.au/sqlserver/ftp_scripts.htm
http://www.mssqltips.com/sqlservertip/2884/sql-server-integration-services-ssis-ftp-task-for-data-exchange/ -
How to read XL file from FTP server
Hi all,
I have a requirement like to read file from FTP server using path
ftp:
10.212......\DTR\DTR_ Accounted_Out
and again save other file in same location ,
to doing this RFC connection is required?
give a procedure or program to do this requirement ..
To create rfc for FTP which connection type can i use ?
Give complete settings to create rfc destinationHi Rakhi,
Use the below code to check if you have received the proper data.
call function 'FTP_SERVER_TO_R3' "Get data as character instead of BLOB
exporting
handle = hdl
fname = docid
character_mode = 'X'
tables
text = chardata.
If the data is incorrect, it is possible that you are in the wrong directory.
Use
call function 'FTP_COMMAND'
exporting
handle = hdl
command = 'cd mydir\mysubdir' "cd <space> your path
tables
data = result
exceptions
tcpip_error = 1
command_error = 2
data_error = 3.
Regards,
Jovito. -
Problem in download file from FTP server
Hi
I want to download a file from FTP server for that i am using the apache FTP module but i am not getting how to download a file with that api i can dispaly the list of files and folders but not able to download the file or folder can any one help me in this
Thanks
NinadHi
I think you miss understood something I am writing a program to download the file in Java where i have used the jakarta.apache api for that and getting problem in that bellow is the code where I have written to print the directory & file names but I don't know how to download the file
FTPClient ftpConnection = new FTPClient();
ftpConnection.connect(host);
ftpConnection.login(FTPConnection.userName,FTPConnection.password);
FTPFile fileList[] = ftpConnection.listFiles();
for(int i=0;i<fileList.length;i++)
System.out.println(fileList.getName());
thanks
Ninad -
How to get file from FTP Server using File Control
Hi,
Any one did getting file from FTP Server?
Please let me know any one help me.
I would need to get file from FTP Server.
Thanks,
MadhuYes I have done that. But In FTP Server I cannt read file, because no previliges. Only I need to copy file from FTP Server to local server then only I can read that file.
I tried all options using FileConrol(getFiles(),read()).
getFiles() - It wont copy the file, it give information about file.
read() - I dont have previliges to read the file.
Please tell me any other procedure would be there for getting file from FTPServer.
Thanks,
Madhu -
Downloading excel file from FTP Server to Application Server
Hi,
I have to get data from an excel file available on FTP server into an Internal table.Can I use FTP_SERVER_TO_R3 to do so.
Please let me know if there are any function modules available to do this.
Thanks,
Prasuna.Dear Gayatri,
You can get the file from FTP to internal table...
I am sending you the code with inline comments ....Hope this will be helpful to you.
Data: lv_key TYPE i VALUE 26101957.
Data: lv_password(30) TYPE c.
i_rfc_destination = 'SAPFTP'.
lv_length = STRLEN( i_password ).
CALL FUNCTION 'HTTP_SCRAMBLE'
EXPORTING
SOURCE = i_password "give ftp server pwd
sourcelen = lv_length
key = lv_key
IMPORTING
destination = lv_password.
CALL FUNCTION 'FTP_CONNECT'
EXPORTING
user = i_user "give ftp user name
password = lv_password
host = i_host
rfc_destination = i_rfc_destination
IMPORTING
handle = lv_ftp_handle
EXCEPTIONS
not_connected = 1
OTHERS = 2.
CONCATENATE 'cd' i_folder_path INTO lv_cmd SEPARATED BY space.
*i_folder path is the path in ftp server where file is stored
CALL FUNCTION 'FTP_COMMAND'
EXPORTING
handle = lv_ftp_handle
command = lv_cmd
TABLES
data = result
EXCEPTIONS
command_error = 1
tcpip_error = 2.
lv_blob_length = 392.
TRANSLATE i_filename TO LOWER CASE.
CALL FUNCTION 'FTP_SERVER_TO_R3'
EXPORTING
handle = lv_ftp_handle
fname = i_filename "give required file name
IMPORTING
blob_length = lv_blob_length
TABLES
blob = lt_dummy.
Regards
Sajid -
[HELP] Download file from FTP server
hi,
I want to write a java program that can download and upload files from ftp server. Currently I only manage to upload a file to ftp server but i cant download file from ftp server. Here is the source code that only allow user to upload file. Anyone can give me some guidelines so that my program can download and also upload file? thx.
import java.io.*;
import java.net.*;
public class FTPUpload {
private static final int CTRLPORT = 21;
private static Socket ctrlSocket;
private static PrintWriter ctrlOutput;
private static BufferedReader ctrlInput;
private static byte[] localHostAddress;
public final static String DIR = "C:\\zip\\";
public static void main(String[] args) {
try {
String host = "192.168.1.1";
String loginName = "testuser";
String password = "password";
String dirName = "/home/testuser";
String fileName = "hello.zip";
ctrlSocket = new Socket(host, CTRLPORT);
localHostAddress = ctrlSocket.getLocalAddress().getAddress();
ctrlOutput = new PrintWriter(ctrlSocket.getOutputStream());
ctrlInput = new BufferedReader(new InputStreamReader(ctrlSocket.getInputStream()));
ctrlOutput.println("USER " + loginName);
ctrlOutput.flush();
ctrlOutput.println("PASS " + password);
ctrlOutput.flush();
ctrlOutput.println("CWD " + dirName);
ctrlOutput.flush();
ctrlOutput.println("TYPE I");
ctrlOutput.flush();
FileInputStream fis = new FileInputStream(DIR + fileName);
Socket dataSocket = dataConnection("STOR " + fileName);
OutputStream outstr = dataSocket.getOutputStream();
int n;
byte[] buff = new byte[1024];
while ((n = fis.read(buff)) > 0) {
outstr.write(buff,0,n);
dataSocket.close();
fis.close();
ctrlOutput.close();
ctrlInput.close();
ctrlSocket.close();
}catch (Exception e) {
e.printStackTrace();
private static Socket dataConnection(String ctrlcmd)
throws IOException,UnknownHostException {
String cmd = "PORT ";
ServerSocket serverDataSocket = new ServerSocket(0,1);
for (int i=0;i<4;i++) {
cmd = cmd + (localHostAddress[i] & 0xff) + ",";
cmd = cmd + (((serverDataSocket.getLocalPort())/256) & 0xff)
+ ","
+ (serverDataSocket.getLocalPort() & 0xff);
ctrlOutput.println(cmd);
ctrlOutput.flush();
ctrlOutput.println(ctrlcmd);
ctrlOutput.flush();
Socket dataSocket = serverDataSocket.accept();
serverDataSocket.close();
return dataSocket;
}Or just use a java.net.URL("ftp://...) ..., get its input stream, and away you go ...
-
Upload XL file from FTP server
Hi All,
Can anybady help me, how to upload Excel file from FTP server.
Thanks
Sri
Edited by: srikanthn on Apr 14, 2010 6:31 PMHello
How about using SAPFTP?
I hope SAP note 130106 will guide you on this.
Thanks
koju
Maybe you are looking for
-
ABAPER for FICO S_ALR_87012082
Hello Friends, This is a T.Code in FICO Module S_ALR_87012082 for Vendor Balance Report.This will give the entair Vendor Balance Report for the comapny.But i need out put according to the USER NAME in the Selection Criteria. For this what i have to
-
How can I convert an ART file to pdf?
How can I convert an ART file to pdf?
-
Infoset based Query creating locks on DSO Activations
Hello Experts, We are current on SAP NetWeaver 7.3 on ORACLE data base version 11.2.0.3.0. We have a query thats built on top of an infoset ( Based on left outer join on 0material and 0mat_plant) when user executes the query its causing locks on DSO
-
Primary database switchover problem
I am getting the following error when i execute 'alter database commit to switchover to standby with session shutdown' in primary database ora-16416 no viable physical standby switchover targets available What may be the reason??? I am using Oracle S
-
I had to rest my iphone 4s. Now I cant use my imessages. How do i fix this. It keeps saying waiting on activation.