MappedByteBuffer with huge files.

Greetings
I have an application for viewing and editing seismic files. Files sizes can range from 3600 bytes up to > 100 gigabytes. I/O performance on a RandomAccessFile is good until a file gets over 2.1 gigabytes then performance drops dramatically.
I have re-written my IO routine to use MappedByteBuffers. I map only a portion of the file, ( I have tried 1 megabyte and 16 megabyte chunks, no difference I can see),.
when I need to read or write a portion of the file that is not mapped I create a new MappedByteBuffer with the same name as previously.
Everything works OK except the direct memory allocated for the MappedByteBuffer is not released untill the program finishes. In the case of a 2.8 gig file the program crashed because it runs out of memory and swap space.
Is there a way to release the memory when a Buffer is no longer needed? Or should I be using a Non-direct byte buffer and reading big chunks of the file in to it.
Any insights would be appreciated.
Regards
Doug Bath

Is there a way to release the memory when a Buffer is no longer needed?No. There's been a lot of discussion about this over the years. GC is your only real answer at the moment, but see also the Bug Parade, 6417205, 4938372, 4954921, and 4724038.

Similar Messages

Working with Huge Files

My boss handed over a hard drive full of HD clips.
The files are too big to transfer to my internal ard drives.
It is impossible to play back without frame skipping.
I have to work in an FCP project already cut with these files.
I doubt if there is an acceptable solution, but thought I'd ask.

None of us have the answer because THERE IS NO ANSWER when the question isn't complete. For example, it would be extremely helpful to know what flavor of HD you're talking about and what type of hard drive was handed to you and how is it formatted. How "huge" are these files anyway? You know ... the little things that could possibly help someone in offering a valid answer.
By the way, how much are you willing to wager and are you actually good for it?
I recently carefully answered over 4000 questions and it really doesn't matter if the OP responded or not. What was your point? Or more precisely; do you have a clue?
-DH

Working with huge files (4GB-15GB) in iMovie 11

should i keep my smaller (4GB) mp4 files in storage or use the 15GB mov files already imported into iMovie '11?
I may want to work with these files again in the future, but is it worth the space to keep it in iMovie so I don't have to reload and optimize such large files?

Welcome to the Apple Community.
That's really a question only you can answer depending on whether you need the space in the meantime. If not keep the optimised files. If you want to work not only with the event files again but the project again, you will need to keep the optimised files.

Save Captivate 5.5 (.cptx) files in Captivate 7 (.cptx) with huge file size results.

I have been opening and saving Captivate 5.5 (.cptx) files in Captivate 7 (.cptx). The problem is that after saving a file in Captivate 7, the file size increases substantially from the original Captivate 5.5 file size. Can this be avoided or corrected? Thanks!

You're a star. I'd already done the switching off the page thumbnails and reducing page performance. I hadn't thought about cross-references! That's certainly a possibility.
I've just done the "Save As" twice - and that first file is down to 1,100 KB - which is a huge reduction from it's previous size! It's still growing already though - just updating the numbers in the book has taken it from 800 KB up to the new 1,100 without me doing anything else (not even opening it myself). Track Changes is (I believe) switched off ... in Edit > Preferences, unless there's somewhere else...
Shoudl have thought of the Save As as well - used to have to do that with Word docs a few versions ago (not so much now).
Thanks again.
Alison

VIDEO PODCAST WITH iWEB-how to deal with huge file size

Hello folks,
I have created simple website for my video podcast. It all works fine, except one problem. I had used just small video clips for now, just for testing... and it takes long time for website to download. I can't even imagine, how slow it will be, when I have full 15 video episodes within my website. So I am wondering, is there any way of how to maintain that cool iWeb podcast structure (with podcast page, entry page and archive page) but not to embed those videos onto the page, but instead, play it externally? I have tried to just post the link to the video clip, but than "publish to the folder alternative" does not create correct rss.xml feed, because it does not know where those clips are. So it doesn't work in iTunes.
Any suggestions? Thanks for any input or advise.

Hello,
unfortunately, no answer yet. The only solution I can think of, is to create a Blog instead of Podcast, then post hyperlink to your video file on entry page. That way, the actual video is not embeded on that webpage, but it only downloads when someone clicks on it. That way you probably won't be able to use subscribe button and RSS subscribe button, but you can create your own Feed with Feed For All software (http://www.feedforall.com), put it on your domain, then post it's URL on your website. So if someone wants to subscribe to your podcast, he will copy that address and paste it into the iTunes or some other podcast readers.
I am a beginner, so this most likely is not the best solution, however, it works so far.
Good Luck

When I watch tv shows in Front Row that I've bought off iTunes the audio gradually falls out of sync with the video. It does not happen with other files that I have converted into mp4s. Any ideas for fixing this? It's a huge bummer...

When I watch tv shows in Front Row that I've bought off iTunes the audio gradually falls out of sync with the video. It does not happen with other files that I have converted into mp4s. Any ideas for fixing this? It's a huge bummer...

I too have the same problem. The audio drop out of sync in about 5 sec in not uncommon for the audio to be 30+ sec behind video. Apple needs to fix this!!!!!!!!!!!!!!!! its a joke, my iphone is all messed up after the last update as well. at one time not to long ago apple products and software were top rated, not so much any more, more and more problems are showing up and apple is unwilling or unalbe to fix the many issues. Apple needs to fix the audio sync problem, it not the computer hardware or other software it ITUNES!!!!, there is no workaround other then hitting pause and hitting pause and hitting pause to temp fix the problem.

What seems to be automatic archiving lops out whole sections of data, but I can't figure out how to restore it. I get huge files with a funny green icon that won't do anything. Small ones can be moved to floppies.

Since mid-2009, I started losing whole blocks of data that was lopped off in blocks covering time periods. In Mozilla and Thunderbird. At the same time, I got huge files with little green icons. These icons can be moved to floppies, but cannot be opened.
At first I thought it was a virus, but my techie says No.
Restore Wizard will not work because it does not like the presence of two hard drives, C and D.

What are the names of those files?
Are they the result of disk errors and created by the chkdsk.exe program?
You can do a disk check with the chkdsk.exe program.<br />
If you run the chkdsk.exe program from a cmd.exe Command window then you can read the response from the chkdsk.exe program.
Open a cmd.exe window:<br />
Start > Run: cmd.exe <press Enter><br />
At the command prompt (>) type or Copy&Paste: chkdsk.exe /f /r <press Enter> (put a space before /f and /r)
If you get something like: Would you like to schedule this volume to be checked the next time the system restarts? y/n then answer the question with "Y" and close all programs and reboot the computer.
http://support.microsoft.com/kb/315265/ - How to perform disk error checking in Windows XP

Photoshop cs4 makes huge files - but with Aperture Whats the biggest file ?

When editing w/Photoshop cs4 by adding layers and masks etc, This makes HUGE files - but with using Aperture alone, what's the biggest file size when editing a RAW file?? (??? _Kb ??? __ mgB ????)
I think I heard -12kb. True ?_________ From what little I understand. Aperture changes the pixels once you have finished the editing. Or to put it another away, Aperture replaces the old or original pixels with the edited final pixels you prefer. IE -lighter darker. and with that said. is this still a RAW IMAGE?

Hello, and welcome to the Discussions.
It would seem that none of what you report hearing is accurate. A RAW file downloaded into Aperture, remains as the Master for this image, and is never changed -- that is why Aperture is said to utilize non-destructive editing. When editing in Aperture, only a small file of edit detail is saved, and no actual new version needs to be created until you Export for some purpose.
When you send an image file from Aperture to Photoshop to be edited, it typically will be a version created as a TIFF file, although there are 4 choices in the Preferences -- TIFF, TIFF 16-bit, PSD, and PSD 16-bit. When finished editing, the image is Saved in PS, that file is available to Aperture, and the version created for sending to PS will change to reflect those changes made in PS.
Aperture cannot send an edited RAW file to PS in this fashion, but you can Export the RAW Master, to later be opened in Photoshop.
You should watch all the available online tutorials about Aperture, and possibly buy a book as well.
Aperture may have trouble with super-sized images, but I have not had trouble with those in the range of 100 MB or less. Larger ones than that would likely be from scans and not from editing an image shot with your camera in Photoshop.
Ernie

I am having an issue with video file size increasing after editing in iMovie 11

So I start off my iMovie 11 video project with a .mp4 video file of roughly 900MB. After chopping off the beginning and ending (which is all I want to do) I have attempted finalizing and exporting using numerous options. The problem is, each time I do this my file turns into a 3.9GB .mov file! I don't want a huge file, I don't want a changed format file. I simply want to cut off the beginning and ending of my movie and leave everything else the same. Theoretically, this should make the file SMALLER since I am removing footage. Correct?
As you can probably tell, I am new to all this. But can anyone tell me how to keep my .mp4 file extension and simply delete the start and end?
Thanks!

The simple answer is, Don't Use iMovie.
If you want to just trim off pieces you can do that in QuickTime X, which is already installed on your Mac. Open the movie in the QuickTime X player. Now go to the Edit Menu and choose Trim... At the bottom of the window you will see a series of thumbnail images with a yellow highlighted box around it. To cut off the beginng pull in the handle along the left hand side of the yellow box. Same is true for the end grab the yellow handle on the right side and pull it in as well. Then click the Trim button, now you've got just the piece of the video you want to save. Now go to File > Save As... set the Format to be Movie (to keep it in the same file format as it started as.
This avoids all the transcoding/conversion that goes on when you import a video into iMovie.

How to export a video at a moderate certain format and size in premiere CC 2014: No Huge File Results!

I have the latest version of Adobe Premiere, CC 2014 but whenever I want to render and export a video that is typically 70 megabytes in size I can't seem to always see how big the file will end up being for all the formats and I don't want to end up with a video larger than 250 to 300 megabytes in size just for the extremely best or highest quality even though I want the video to be well polished and in better quality than I usually get thee, These videos are usually imported as mp4. What file version of mp4 or otherwise would give me an originally imported video of 70 megabytes an output of no more than or around 300 megabytes or even less without compromsing video quality? I don't want a huge file size as I am uploading it to my YouTube channel via their web interface on another person's machine and they don't have Premiere and only has a DSL connection. So what video output settings should I use for a good quality video that isn't smaller than the original but the same size or a bit higher and not extremely large like MOVs and sometimes mp4s usually are depending on what you do to it and it is outputted.?

Export
Format: H.264
Preset: YouTube (make sure this matches the dimensions of your sequence)
You could further adjust the Target Bitrate but the preset is already setup for a good quality file for upload.
You can also do further reading on Video Bitrates.

Upload huge file size: "The page cannot be displayed" browser

What are precautions to be take for uploading a huge file
I have an upload file operation in the web application. The web server is OC4J.
//UploadForm.jsp
<FORM NAME="InputForm" ACTION="UploadAction.jsp" METHOD="POST" enctype=multipart/form-data>
<input type="file" name="fileName">
</FORM>
After I deploy the application to the web server, if I upload a small file size, it is fine.
But if i upload huge data ,data is uploading but if any action button is clicked
After 30 seconds, it has error "The page cannot be displayed" shown on the web browser.

Hi All,
I have a similar time out problem, it ll be great if some one provide a solution.
The problem I'm facing is becasue of the time out setting in the proxy server which is 1.5 mins. The app server is sending Response properly but after some 2 mins or so because the processing is a bit complicated (3 API calls done intermittently for different validations).
I cant request for change in proxy server time out setting. So I have to handle this with in my application.
It will be greatful if anyone helps us out of this.
Thanks
Noufal

ITunes, what have you done with my files?

Irrelevant backstory;
I've actually had quite a turbulent time with technology lately- External harddrives blowing up, iTouch screens cracking in my pocket. Not to mention the screen on this ruddy Mac of mine. But I'm fairly competent with technology and so far have been able to fix most problems I've made for myself. This one is a little tougher so I'd really appreciate a little help.
This is a long story, mostly because information tends to yield better solutions. Feel free to skip to the short version below if you'd like to save your eyes the trouble.
Essentially, I replaced my iTouch yesterday. This 9 month old beauty had accumulated thousands of listens, and since it was manually managed from itunes Last.fm refused to scrobble any of them. So I made my mind up that I'd simply sync the new one automatically. That's when things started to go wrong. I had deleted all my music from the harddrive for the purpose of downloading fresh films, confident in the fact I had two recently backed up external harddrives containing all my music (about 3,000 songs I like, and 1,000 songs other people like that they give out to me for not having) and movies. Altogether, this came to around 24 GB. I realise this is not a huge amount in comparison to what some people have, but I'm a relative purist (except when drunk) and some of the music is exceptionally hard to find (ever heard of Ween?). So I'd really like it back. Now, as I've mentioned, one of the harddrives is a glorified paperweight, it fell 3 feet and is now making typically awful beeping/scratching noises and no longer fires up. The second one was working fine, however, so I plugged it in and set to work copying and pasting.
iTunes duly realised that none of the music was where it used to be, so I re-added my C:/Music folder to the library and got plenty of awesome duplicates. After deleting those, iTunes would play the song if I tried to click on it (exclamation point or not). Problem sorted then, except that because iTunes didn't realise it knew the locations of the songs once more, virtually none of those songs were synching with my spanking new iTouch. So I started manually clicking into the exclamated songs, which would play, and the sync. I got halfway through A before realising this would take forever. So I searched for a more automatic way. I tried readding all my songs again, but this just created tons more duplicates, and after deleting those went into the File->Library->Organise Library menu. Things go badly for me from here.
I knew what Consolidate would do and I didn't like it. I could guess what Reorganise would do, but hey, it might help, and it specifically said it wouldn't remove or delete any files. Just move them.
So I figured what the ****.. Why not?
Mistake.
After taking ages to reorganise my already organised music folders, iTunes promptly lost around 1500 songs. Individual songs and albums. It seems everything that wasn't a Compilation from Kings of Leon-Come Around Sundown (of which I now have half an album) was gone. So I resigned myself to locating the songs manually.
But they weren't there. The first 7 songs were there from Come Around Sundown. The remaining ones were not. I thought this pretty odd. Then i noticed that the rest of the songs weren't there either. Alphabetically, from "K" down, most of my music was simply not there. And I now had this funky new C:/Music/Music folder, which was just full off the songs I did have. After tinkering around with the XP search function and finding nothing anywhere, I decided to simply delete and start over. So I wiped my Music folder and recopied everything from the external HD. Re-opened the folder in iTunes and voila, everything was the exact same.
My only thought here is that I've managed to wipe the Music folder in the external HD, and replaced my wanted music with what iTunes has done to my laptop. Highly stupid, absolutely, but not absolutely out of character with deleting the music in the first place, or reorganising my music folder despite misgivings about it's effectiveness. In fact, it was par for the course with how I approached this problem last night.
Short version;
iTunes has, apparently while reorganising my library, deleted a number of songs from my computer. I now have far less music than I should, and it is not anywhere on my computer. The music folder has shrunk from 24GB to 16GB. My movies have been renamed into oblivion. This was not what I thought reorganising did.
Reorganising = File->Library->Organise Library->Reorganise files in the folder "Music"
"Music" is my music folder, containing all of my beloved tracks. It's located at C:/Music
What's interesting now is that when the iTouch synched with iTunes, it did manage to pick up all of my movies, despite them not being there. When the iTouch is plugged into iTunes and i look at the Movies tab (close to Summary on the top), everything is there. When I look at the Movies viewer on the left (below Music) it's empty. I tried to manually sync some movies to save space (plus, I've already watched Team America a dozen times now) it removed all of them. They will no longer go back on, but iTunes does show them in the Movies tab.
They're still there in theory, but neither a manual search for them nor allowing Movies to sync automatically puts them back onto the iPod touch.
Solution wise, I'm stumped. If the computer doesn't let me find them, then they're not there. But Reorganise expressly pointed out that files would be moved and renamed, not deleted. And the Music folder has shrunk in size, which indicates that iTunes was lying. The only ways out I can see are to a) attempt to repair the damaged hard drive which contains all my unharmed music or b) acquire all my missing music again.
Option a isn't bad, since I'd like to get that HDD working again anyway. Option b is going to take a long time and will be incomplete. Folks, after all this text what I really want to know is- Anyone got an option c?
Cheers! And apologies for the length. I tend to ramble.
~SFD
iTunes is fully up to date. The iTouch software is 4.3.2 instead of 4.3.3, but internet is expensive so downloading that update will not be done on Telstra wireless. The laptop is a dualbooted MBP running XP.

tl:dr?
Really?
There was a short version expressly for people who weren't bothered to read a myriad of text.
Meh, not that it matters anymore. I ended up redownloading 70-odd albums, so I have the bulk of my music back. And I still have no idea what iTunes did with my files, but I'll make sure it never does it again.
I would thank you for the help but you were utterly useless. Oh well, carry on.

Working with RAW files in PSE9: Doable or better to use PSE12?

I have PSE 9 and will soon get a camera with RAW capability. I notice that Adobe no longer supports PSE 9. Am I in for trouble if I don't upgrade to PSE 12 since I don't know how to work with RAW files? It sounds as if I will need something to convert RAW files? Where do I get it? Is it a big deal for a beginner with no software skills?

You can download a free DNG Converter from Adobe and then convert your RAWs to DNGs which can be used in PSE9.
You don't need PSE12, but if you want to purchase PSE 12, you get the most current features in the Raw processor (which in my opinion is a huge improvement over what is available in PSE9), and you don't need the extra step of converting to DNG.

Transfer of huge file sizes

We have ~1.5 petabytes of data needs to be uploaded from users' machines to a certain destination. This is based on an ave. of 5.5 GB worth of files for 263k users. We are looking for guidance from a network/bandwidth perspective on the deployment schedule/approach in order to avoid negatively impacting the our current network infrastructure and other services.
Users are located geographically and connected via MPLS/internet.
Assuming upload of this huge file will be perform via web, what should be the best approach in achieving this?
What are the things/areas need to consider/check before proceeding on this?
Do we need to analyze first the exact behaviour of the file when upload is performed?
Best Regards,
Oliver Caluag

Hi Oliver,
One thing you want consider is to make sure the file transfer job does not take over the whole bandwidth, and affect other application's performance. If you have QoS build end to end in the network, make sure this file transfer job is classified in BULK data with 20-30 % of the bandwidth. If it is possible, you can also schedule the job out of business hour, so it will less impact to other applications.
HTH,
Lei Tian

Handle huge file size in file receiver

I have file(NfS) to file(Ftp) scenario the receiver file failing because of my huge data(nearly 20KB) i am using Message mapping functionality i cant use file check machanisam to split my file .could you share some ideas how can i achieve this requirement .
file ---> PI (7.4 java)--->File(Ftp) 20 to 25 KB)
how can i handle huge data in FIle receiver side ?

Hi Raman,
First you need to identify which step is taking more time. Adapter or transformation. Please share more details about your scenario.
If your scenario is pass through (with no transformation), then you can use the chunk mode. Please refer the below blog
File/FTP Adapter - Large File Transfer (Chunk Mode)
PI/XI: PI 7.3 processing of large files - teaser
If your file is having multiple recordset then you can pick the fix no of record set. Please refer the below blog
Night Mare-Processing huge files in SAP XI
regards,
Harish

MappedByteBuffer with huge files.

Similar Messages

Maybe you are looking for