File Chunking on upload

I have to solve a problem of uploading large 4gb files.
Anyone have any experience or point to an article on the following,
With a file reference we can get to the fileReferance.data and load it into memory.
I wonder if you could part load a chunk of data say 80mb, upload that 80mb, then discard that 80mb and load the next 80mb
You would loop until end of data, I can reassemble the data at the server not an issue.
The trick here is that I do not want to load the entire file into memory of the flash player just a chunk at a time and discard the uploaded chunk to free up the memory.
The idea would be to build on this is to be chunking the next bit of data while the previous chunk is uploading, also resumable uploading.
If we start the upload again, start so many chunks (bytes) into the file.
When we start the file process take the  file size from the file reference and divide by the chunk size, when we send the first chunk we tell the server how many chunks of data to expect.
Thoughts, ideas appreciated.
TIA

not sure if you can do this outside of AIR, but i've done it as an AIR app.
use FileStream to openAsync() a file for FileMode.READ
use FileStream's .readAhead property to define the amount of bytes you want to read in at a time
define your own chunkSize value for the size of the chunked files
use the ProgressEvent fired by FileStream to let you know when enough bytes are available to fill a buffer of chunkSize
create a new File and write those chunkSize bytes into it
when that file has finished writing to the local disk, use File.upload() to upload it to your server
repeat until all chunks are there.  using openAsync with a smaller chunk size means you don't have to load the entire file into memory.  using File.upload() gives you upload progress.  (Sockets are still broken in as3 in that they don't report the progress of the file)
it's quite simple to set up your own mini protocol for resuming broken transfers.  just keep track of the last chunk index that was successfully uploaded, and start from that index to resume a broken upload.  reassembling the chunked files back into the original file on the server side is trivial.
i don't think this will work without a lot of user intervention in a flash app running inside a browser, due to flash's security restrictions.  with flash 10 you can write to the local disk, yes, but i believe manual user intervention/permission is required each time.  furthermore, i don't think you could upload() the chunk files that were dynamically created without manual user intervention/permission for security reasons.

Similar Messages

  • File chunking and uploading/reasembling

    i'm working on a chunked file uploader that talks to a nginx modul that handles resumable/chunked uploads.
    right now i'm trying to verify that the file chunking process is working correctly. however i've run into some strange things with my local tests. for starters it works perfectly on all of the smaller file's i've tried. they've all passed the md5 checksum comparisons.
    however for all of the larger files i've tried they fail the md5 checksum comparison, and a lot of the time the videos won't play. i've noticed that upping the chunk size from 5mb to 100mb has fixed the playback issue for at least one of the videos i've tested with.
    as far as i can tell it's loading in the original file in chunks and storing those chunks into files. i can watch it's progress through the file and the numbers all match up as expected.
    here's my source code for the chunking and rebuilding:
            public function makeChunks():void{
                status = "Preparing";
                trace("VideoData::makeChunks");
                fileStream = new FileStream();
                fileStream.readAhead = chunkSize;
                fileStream.addEventListener(ProgressEvent.PROGRESS,onOpenProgress);
                fileStream.addEventListener(Event.COMPLETE,onOpenComplete);
                fileStream.openAsync(file,FileMode.READ);
            private function onOpenProgress(e:ProgressEvent):void{
                if(fileStream.bytesAvailable >= fileStream.readAhead){
                    trace("onOpenProgress |",fileStream.position,"|",e.bytesLoaded,"/",e.bytesTotal,"(",file.size,")|----|",fileStr eam.bytesAvailable,"/",fileStream.readAhead);
                    var cChunk:ByteArray = new ByteArray();
                    fileStream.readBytes(cChunk);
                    trace("--",chunkIndex,"*",cChunk.length,chunkIndex*cChunk.length);
                    trace("--",fileStream.bytesAvailable,"/",fileStream.readAhead);
                    var tFile:File = File.applicationStorageDirectory.resolvePath(file.name+"_chunks/"+chunkIndex+".part");
                    var wStream:FileStream = new FileStream();
                    wStream.open(tFile,FileMode.WRITE);
                    wStream.writeBytes(cChunk);
                    wStream.close();
                    chunkIndex++;
                    //fileStream.position += currentChunk.length;
                    trace("---------------",chunkIndex,"",cChunk.bytesAvailable);
                    //dispatchEvent(new MediohEvent("LocalVideosChanged",true,true));
                    current_progress = e.bytesLoaded / e.bytesTotal;
            private function onOpenComplete(e:Event):void{
                trace("onOpenComplete |",fileStream.position,"/",file.size,"|",file.size-fileStream.position,"|----|",fileStrea m.bytesAvailable,"/",fileStream.readAhead);
                if(fileStream.bytesAvailable > 0){
                    var cChunk:ByteArray = new ByteArray();
                    fileStream.readBytes(cChunk);
                    var tFile:File = File.applicationStorageDirectory.resolvePath(file.name+"_chunks/"+chunkIndex+".part");
                    var wStream:FileStream = new FileStream();
                    wStream.open(tFile,FileMode.WRITE);
                    wStream.writeBytes(cChunk);
                    wStream.close();
                    trace("chunking complete---------------",chunkIndex,"bytes length",cChunk.length,"bytes length",cChunk.length);
                trace("--",chunkIndex,"*",cChunk.length,chunkIndex*cChunk.length);
                trace("--",fileStream.bytesAvailable,"/",fileStream.readAhead);
                fileStream = null;
                chunk_path = file.name+"_chunks/";
                needs_chunks = false;
                status = "Uploading";
                current_progress = 0;
                dispatchEvent(new MediohEvent("LocalVideosChanged",true,true));
                dispatchEvent(new Event("saveVideos",true,true));
                rebuild();
            private function rebuild():void{
                var target_file:String = "C:\\Users\\sophia\\Videos\\backtogether\\"+file.name;
                var folder:File =  File.applicationStorageDirectory.resolvePath(chunk_path);
                var filesFound:Array = folder.getDirectoryListing();
                trace("blah",filesFound);
                var bigFile:File = new File(target_file);
                var wStream:FileStream = new FileStream();
                wStream.open(bigFile,FileMode.WRITE);
                for(var i:int = 0; i < filesFound.length; i++){
                    var fStream:FileStream = new FileStream();
                    fStream.open(filesFound[i],FileMode.READ);
                    var bytes:ByteArray = new ByteArray();
                    fStream.readBytes(bytes);
                    fStream.close();
                    wStream.writeBytes(bytes);
                wStream.close();
                status = "Complete";
                current_progress = 1;
                date_uploaded = new Date();
                clearChunks();
                dispatchEvent(new Event("uploadComplete",true,true));
                dispatchEvent(new Event("saveVideos",true,true));

    i've found a work around. instead of splitting the file up into a bunch of smaller files that i can then read the bytes from to upload i just upload them as i read them from the file. and have it wait to processes the next block of bytes that get loaded until after the previous bytes finish uploading.
    i also determined that the number of chunks effects my ability to put them back together again locally.
    i tested with a file i had successfully uploaded before with a much smaller chunk size (1mb instead of 5mb)  and it uploaded successfully but my local attempt to rebuild failed.
    i'm still kinda currious as to why i can put together a file broken into a couple chunks but not one broken into many.

  • Debatching and File Chunking

    I want to know
    1) the Difference between Debatching and File Chunking?
    2) how both work?
    anyone kindly explain me.

    i've found a work around. instead of splitting the file up into a bunch of smaller files that i can then read the bytes from to upload i just upload them as i read them from the file. and have it wait to processes the next block of bytes that get loaded until after the previous bytes finish uploading.
    i also determined that the number of chunks effects my ability to put them back together again locally.
    i tested with a file i had successfully uploaded before with a much smaller chunk size (1mb instead of 5mb)  and it uploaded successfully but my local attempt to rebuild failed.
    i'm still kinda currious as to why i can put together a file broken into a couple chunks but not one broken into many.

  • I have purchased a monthly subscription to Adobe PDF pack so that i can edit a PDF file, i have uploaded the file and its now in "my assets". how do i now edit this file??

    I have purchased a monthly subscription to Adobe PDF pack so that i can edit a PDF file, i have uploaded the file and its now in "my assets". how do i now edit this file??

    You can't edit a file with PDF Pack, but you should be able to convert it to a Word document, and if you are really lucky edit THAT and make a new PDF. Don't expect too much though!

  • How to get full file path while uploading a file in flex Applications

    How to get full file path while uploading a file in flex applications.
    FileReference Object is giving file name and other details but not the actual path.
    Is there any workaround to to get the file path?.
    Thanks

    Why not ask in the Flex forum; it is more likely that someone over there knows.

  • Some chinese character file name not uploading in SharePoint Document Library?

    Hi
    Some Chinese character pdf file name not uploading in SharePoint Document Library. I am getting following error.
    1. File not found error if i would upload using file browse option in Doc Lib.
    2. Can't read from the source file or disk. if i would tried to copy and paste using "Open with explorer" options.
    File name is "LA(未签署).pdf"
    Note: This file has been converted from "TIFF" to "PDF".
    Thanks & Regards
    Poomani Sankaran

    Hi Poomani,
    For troubleshooting your issue, please let's verify the followings:
    Whether other pdf file which are not converted from "TIFF" and contain Chinese char could be uploaded into the SharePoint Document library.
    Whether other pdf file which are converted from "TIFF" and don't contain Chinese char could be uploaded into the SharePoint Document library.
    Please upload these pdf files into other libraries, compare the result.
    Please check the log file to find more information about this issue. The path of the log file is:
    C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS.
    Best Regards,
    Wendy
    Wendy Li
    TechNet Community Support

  • Urgent : How to upload a tif file without using upload element

    could someone please tell me how to upload a tif file(any file) without using upload element. Function Module GUI_UPLOAD does not work. Please suggest. Appreciate your suggestions.

    Hello Suri,
    there's currently no way to achieve this.
    Best regards,
    Thomas

  • Image files email and upload problem

    Emails I send through mail are arriving very compressed. However when I try to upload them onto my webspace they are expanding.
    For example a jpeg 600 x 600px, at a resolution of 72ppi and a file size of 308kb:
    When emailed through Mail it arrives as 240 x 240px, at a resolution of 72pp and a file size of 92kb
    When emailed through my webmail (Virgin) it arrives at the original dimensions and file size
    When uploaded to my webspace using Cyberduck it expands from 308kb to 4GB and an estimated upload time of 18 hours!
    The larger the original file size, the worse the email problem, but the upload remains at around 4GB.
    I suspect that somehow the file is being compressed on arrival because it has expanded as it is being emailed or uploaded.
    My webspace provider says it isn't a problem their end. As things are ok through my Virgin webmail I don't think it can be them - though they are the usual suspects.

    When you email an image in Mail, set the Image Size pop-up in the lower right-hand corner of the window to "Actual Size".

  • CSV file template for Uploading PO confirmaiton

    Dear all,
    We have activated Badi for Upload PO confirmations for a supplier in SNC 7.0.
    Request if any of you can please let me know where can I find the CSV upload file template.
    Thanks,
    mahesh.

    Hi Mahesh,
    You can download the CSV file from Tools-->Download center(select purchase order confirmation.
    To create a purchase order confirmation, you enter X in the To Be Confirmed column of the schedule line. If the schedule line has an X, you can change the following data:
    ■Quantity
    ■Delivery date
    Note
    Depending on the system set-up, for example if you have set up Customizing for POs to allow shipping dates to be used instead of delivery dates, you can change the shipping date.
    End of the note.
    ■Confirmed price
    Note
    If you leave the Confirmed Price field empty in all schedule lines of an item, the requested price is used. If you enter a value in the Confirmed Price field of one of the schedule lines, or in more than one schedule line, but those values are the same, the system uses that value as the confirmed price for the item. However, if you enter two or more different confirmed price values in the schedule lines of an item, the system regards this as an error, and the item is not processed.
    End of the note.
    ■Confirmed MPN
    ■Confirmed Mfr
    ■To reject the PO item, you enter an X in the To Be Rejected column of the schedule line that has an X in the Requested column.
    Now upload this file in Tools->upload center.
    See the below link for more information.
    http://help.sap.com/saphelp_snc70/helpdata/EN/b4/79223dc5b54b36899ea4f731a712f6/frameset.htm
    Regards,
    Nikhil

  • How to increase max file size for uploading

    hi i have a form that uploads a file and save it in a database..my problem is when it exceeds 2MB it doesnt save in my database. the page doesnt show any error also.
    i use multipartrequest..here is multipartrequest...
    MultipartRequest multi= new MultipartRequest(request,"/var/www/myuploads/",1000*1024, new com.oreilly.servlet.multipart.DefaultFileRenamePolicy());my application server in tomcat, and my database in mysql..
    do i need to set something in tomcat to increase the file size that i can upload?
    i need help asap..thanks in advance.

    tyr to split ur file into separated files
    and then upload them,
    it is preferable not to upload a big sized file

  • One file maximum for uploads with Safari, Opera, FireFox and Chrome to OneDive

    Some Windows 7 users have complained to they can only mark one file maximum for uploading with Safari, Opera, FireFox and Chrome to OneDive

    Hi,
    You mean that you access OneDrive via web browser?
    From my tests, everything is normal and it seems there is no setting in browser to configure this number.
    if I misunderstand this issue, please give us more information on it.
    Alex Zhao
    TechNet Community Support

  • Ipad only uses one file name when uploading images via email

    We have an image tool that uploads images to a template website. The problem is the file names for these images must be unique any duplicate file names will be rejected. Some of our users want to be able to upload from the field but Ipad only assigns the file name "photo.JPG" to jpegs. It doesn't seem to include the sequential extension that actually does exist for the image on the file name when uploading. Is there a way to get ipad to do so, or is this something that Apple maybe want to consider looking into. I can easily see the average Joe emailing themselves images and hitting save only to replace and lose an older image because let's face it, not all users are on MACs and PC's do not allow multiple files to have the same file name in the same folder.

    Hi,
    Based on the description, you could send email to external addresses without the Twin Oaks software. However, with the Twin Oaks software, you couldn't send successfully.
    For this issue, I recommend you enable message tracking and check whether you could retrieve message tracking log entires when you send emails to external addresses through the Twin Oaks software.
    If you couldn't retrieve message these tracking log entires when you send emails to external addresses through the Twin Oaks software, it means that the Exchange server is OK and the crux of the problem is the Twin Oaks software.
    Here is an article about message tracking log for your reference.
    Get-MessageTrackingLog
    http://technet.microsoft.com/en-us/library/aa997573(v=exchg.141).aspx
    Best regards,
    Belinda
    Belinda Ma
    TechNet Community Support

  • File Visibility in uploading the file using web DynPro java

    hello,
    i have gone through the tutorial " uploading & downloading the file using web DynPro java".
    the upload functionalty is working fine. but i need to know thw path where these files are getting uploaded ??
    Waiting for the reply..
    Regards,
    Viren Gupta

    Hi Viren,
    Are you are refering to the below tutorial for upload and download:
    http://www.sdn.sap.com/irj/scn/go/portal/prtroot/docs/library/uuid/b0e10426-77ca-2910-7eb5-d7d8982cb83f?QuickLink=index&overridelayout=true
    If yes, then answer is that the file is stored in the context of the application only as "com.sap.ide.webdynpro.uielement-definitions.Resource" object. Please read through the tutorial to understand the full process.
    WD doesn't really upload to server automatically unless there is any explicit code written. All it does is to store the file in a temp context which you can access and write your own code to store it in a particular location.
    Regards,
    Mahesh

  • Allowing only zip files to be uploaded through af:inputFile ( in jdev 11g )

    I am able to upload all kinds of files through af:inputfile tag. I want to allow only zip files to be uploaded though the af:inputFile tag. Does inputFile provide any feature of this sort???

    Let me address two issues here:
    1. Chris has posted a correct answer about how to write a ValueChangeListener to allow only zip files. However, by the time this code runs, the file has actually already been uploaded to temporary storage. All this code does is specify what to do with the file once it is on the application server. So if you want to prevent the file from being uploaded unless it is zipped, this catches it too late. To catch it earlier, you have to write your own UploadedFileProcessor, and configure ADF to use it. Read more in Appendix A, section A.6.2.11 of the Fusion Web User Interface Developers Guide for Oracle ADF 11g.
    2. While you have to write some code to handle uploaded files yourself (hey - it isn't a lot of code, and I don't care how declarative ADF is, sooner or later you're going to write a little code) if you are uploading to an ORDSYS.ORDDOC column in your Oracle database, ADF CAN do that declaratively.

  • My "Pages" files won't upload

    My pages files won't upload to my university site ever since I upgraded it about a week ago. Everytime I try to upload it the files that I created with pages are shown below. Has anyone else had this problem?

    Safari > Preferences >Privacy
    Set "Block cookies and other website data" to "Never".

Maybe you are looking for

  • ITunes Crashed, am trying to import songs purchased from iTunes from burned

    My girlfriend just became an iPod Covert and while she was busy making custom playlists for an upcoming party, her system crashed. She purchased several songs from the Store, and burned them to a CD. Now when she is trying to rebuild her iTunes Libra

  • How can I specify table owners when I export project to Java source?

    Using TopLink Mapping Workbench, when I export project to Java source, the descriptors is automatically generated like: descriptor.addTableName("SOME_TABLE"); Is there a way to configure the TopLink Mapping Workbench so it generates the descriptors s

  • EDI Invoice printing

    Hey all you Oracle experts, Hopefully simple question here surrounding Invoice printing and EDI. In our current setup we print most of our invoices on a daily basis, and the image generated through BI publisher gets archived successfully. We have rec

  • Able to play live video by rtmp but not by http

    Hi Experts ,                    I need some help in proceeding further on my HDS live streaming setup . i have adobe flash media server 4.5 and adobe flash media live encoder 3.2 . followed all the steps provided in the following link http://help.ado

  • Error when trying to start Apache Chemistry Workbench (./workbench.sh)

    Hi colleagues, when trying to start the Apache Chemistry Workbench with ./workbench.sh (as described in Unit 4 of Week 3 of the SAP HANA course), i'am getting the following error: Can someone please help me? Thanks and regards Thorsten