Handling large files for displaying within a JTextPane

I have some (very) large trace files sometimes up to more than 100MB and I want to group the traces which are originally stored unordered into this trace file.
When I tried to read in a just 16MB large trace file and used this command line option: -Xm60m I get an OutOfMemoryError.
Is there a way to handle (very) large (text) files most effectively? For instance, how can I achieve to group (some kind of sortation) a trace file of let's say 100MB most effectively? Maybe by splitting the file?
I want to display the text of this/these text files in a JTextPane coloring some particular parts of the traces.
Thank you in advance,
Dirk
Berlin, Germany
[email protected]

I don't know of any way to display 100MB of text effectively. I'm sure the user isn't going to sit there and scroll through 100MB of data to find what they are looking for. I would probably load the data into a database and create some queries the user can invoke to filter the data.

Similar Messages

Handling large files in scope of WSRP portlets

Hi there,
just wanted to ask if there are any best practices in respect to handling large file upload/download when using WSRP portlets (apart from by-passing WebCenter all-together for these use-cases, that is). We continue to get OutOfMemoryErrors and TimeoutExceptions as soon as the file being transfered becomes larger than a few hundred megabytes. The portlet is happily streaming the file as part of its javax.portlet.ResourceServingPortlet.serveResource(ResourceRequest, ResourceResponse) implementation, so the problem must somehow lie within WebCenter itself.
Thanks in advance,
Chris

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Java proxies for handling large files

Dear all,
Kindly let me know handle the same in step by step explanation as i do not know much about java.
what is advantage of using the java proxies here.Do we implement the split logic in java code for mandling 600mb file?
please mail me the same to [email protected]

Hi !! Srinivas
Check out this blog....for Large file handling issue
/people/sravya.talanki2/blog/2005/11/29/night-mare-processing-huge-files-in-sap-xi
This will help you
Please see the documents below. This might help you
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/a068cf2f-0401-0010-2aa9-f5ae4b2096f9
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/f272165e-0401-0010-b4a1-e7eb8903501d
/people/prasad.ulagappan2/blog/2005/06/27/asynchronous-inbound-java-proxy
/people/rashmi.ramalingam2/blog/2005/06/25/an-illustration-of-java-server-proxy
We can also find them on your XI/PI server in folders:
aii_proxy_xirt.jar
j2eeclusterserver0 inextcom.sap.aii.proxy.xiruntime
aii_msg_runtime.jar
j2eeclusterserver0 inextcom.sap.aii.messaging.runtime
aii_utilxi_misc.jar
j2eeclusterserver0 inextcom.sap.xi.util.misc
guidgenerator.jar
j2eeclusterserver0 inextcom.sap.guid
Java Proxy
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/a068cf2f-0401-0010-2aa9-f5ae4b2096f9
https://www.sdn.sap.com/irj/sdn/go/portal/prtroot/docs/library/uuid/f272165e-0401-0010-b4a1-e7eb8903501d
Pls reward if useful

Hardware Question – Handling large files in Photoshop

I'm working with some big TIFF files (~1GB) for large-scale hi-res printing (60" x 90", 10718 x 14451), and my system is lagging hard like never before (Retina MacBook Pro 2012 2.6GHz i7 /8 GB RAM/ 512GB HD).
So far I've tried:
1) converting to .psd and .psb
2) changing the scratch disk to an external Thunderbolt SSD
3) allocating all available memory to the program within photoshop preferences
4) closing all other applications
In general I'm being told that I don't have enough RAM. So what are the minimum recommended system requirements to handle this file size more comfortably? Newest Retina Pro with 16GB RAM? Or switch to iMac w/ 32? Mac Pro?
Thanks so much!

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Handling Large files in PI scenarios?

Hello,
We have lot of scenarios (almost 50) where we deal with file interfaces atleast in receiver or sender side. Some of them are just file transfers where we use AAE and some are where we have to do message mapping (sometimes very complex ones).
the interfaces work perfectly fine will a normal file which dont have much records but recently we started testing big files with over 1000 records and its taking a lot of time to process. It is also causing other messages which gets lined up in the same queue to wait in the queue for the amount of time it takes for the first message to process.
This must be a very practical scenario where PI has to process large files specially files coming from banks. What is the best way to handle its processing? Apart from having a better system hardware (we are currently in the test environment. Production environment will definetely be better) is there any technique which might help us improve the processing of large files without data loss and without interrupting other message?
Thanks,
Yash

Hi Yash,
Check this blogs for the strcuture you are mentioning:
/people/shabarish.vijayakumar/blog/2006/02/27/content-conversion-the-key-field-problem
/people/shabarish.vijayakumar/blog/2005/08/17/nab-the-tab-file-adapter
Regards,
---Satish

Handling large files with FTP in XI

Hi All,
I have a file scenario where i have to post the file with size of more than 500MB and with the fields upto 700 in each line.
The other scenario worked fine if the file size is below 70MB and less number of fields.
Could anyone help me in handling such scenario with large file size and without splitting the file.
1) From your previous experience did you use any Tools to help with the development of the FTP interfaces?
2) The client looked at ItemField, but is not willing to use it, due to the licensing costs. Did you use any self-made pre-processors?
3) Please let me know the good and bad experiences you had when using the XI File Adapter?
Thanks & Regards,
Raghuram Vedagiri.

500 MB is huge. XI will not be able to handle such a huge payload for sure.
Are you using XI as a mere FTP or are you using Content Conversion with mapping etc?
1. Either use a splitting logic to split the file outside XI ( using Scripts ) and then XI handles these file.
2. or Quick Size your hardware ( Java Heap etc ) to make sure that XI can handle this file ( not recommended though). SAP recommends a size of 5 MBas the optimum size.
Regards
Bhavesh

Large files not displaying clearly

I have just stitched some photos and the resulting files do not display clearly outside of CS4. They were files - up to 800MB I have then resized them and put them on the web and found the files resized from the larger files do not give a good resolution on the screen. I have tried saving them up to 150 ppi instead of 75 but still get the same result. The original large file has incredible resolution when zooming in with CS4. But terrible when trying to resize put it on the web. For an example check out the gallery. http://gallery.me.com/suethomo#100207&bgcolor=white
The stitched ones are Thomo_1. Thomo_2 Thomo_3 Thomo_4. The sharpest ones are taken with a point and shoot! The stitched pics are with Canon 5D Mk!! and processed from RAW I have sharpened them in CS4 and they look good until I view them in Bridge or elsewhere. Not sure where I am going wrong.
Any suggestions welcome

A couple of things come to mind...
I noticed that your web gallery scales the images according to the viewer's browser window size. Shrink the browser window and the image shrinks. Enlarge the browser window and the image enlarges. I would suspect that scaling the image beyond the actual image size would cause pixelation and possible aliasing.
I would also check that you are sizing your images properly for online viewing. The resolution should likely be 72 ppi at a suitable viewing size. I would also ensure that you are using 'Bicubic Sharper' when scaling images down in PS.

How does Time Machine handle large files?

I'm relatively new at the whole Time Capsule / Time Machine process and have learned that large files (eg aperture library) are backed up each time there is a change and this can lead to the TC filling up quicker than normal.
How does this work with daily and weekly backups?
For example, if my aperture library is, say 1Gb and I import a load of photos from my camera and this goes up to 2Gb. I've learned that I should disable time machine while I'm in Aperture (or at least before 10.6...not sure now). So given I've done that, imported the files to Aperture but want to edit them later and ultimately move them into iPhoto to keep the Aperture album small.
When I turn back on Time Machine, the next hourly backup will know the library has changed and will back it up, this will go on until a day backup has been taken - this deletes the 24 hourly backups? or does it merge them?
If I then do the editing the following week, then export the photos and the library is now back to 1Gb again....backed up hourly/daily/weekly etc what am I left with??
Do I have an original, the 2GB version and the new 1Gb version...ie 4Gb......is there a cunning way I can work to change the files within a week so only one of the changes is in the backup?

Orpheus999 wrote:
When I turn back on Time Machine, the next hourly backup will know the library has changed and will back it up, this will go on until a day backup has been taken - this deletes the 24 hourly backups? or does it merge them?
The Time Machine panel of System Preferences says this:
Time Machine keeps
- Hourly backup for the past 24 hours
- Daily backups for the past month
- Weekly backups until your backup disk is full
Each time Time Machine runs it creates what appears to be an entirely new backup set, although it does this in a way that doesn't require it to copy files that have already been copied. So merging isn't necessary. Another effect of how it operates is that each unique version of a file (as opposed to packages of files) only exists on the backup volume once.
According to the contents of my Time Machine backup file, hourly backups are literally kept for 24 hours, not until the next "daily" backup. For a "daily" backup, it seems to keep the oldest "hourly" backup for a day.
If I then do the editing the following week, then export the photos and the library is now back to 1Gb again....backed up hourly/daily/weekly etc what am I left with??
Do I have an original, the 2GB version and the new 1Gb version...ie 4Gb......is there a cunning way I can work to change the files within a week so only one of the changes is in the backup?
You might be able to exclude those files from being backed up at certain times, but I can't be sure this would result in older copied of those files being retained.

Handling source files for multiple versions

Ok, I've looked around on the web and multiple discussions and have yet to find an answer. We currently have two authors working on the same help project that we have handled relatively successfully under RoboSource. Our development environment has changed so that we need to be able to support multiple versions of the help at once ie- release branch vs development branch that we want documented but it is not yet slated to be in the release.
I've looked at the help for RoboSource and there is an idea of branching but not much help on how this is done or if it really maintains two sets of the files, etc. I've also seen some postings on the forum for this that have gone unanswered. Makes me suspect that perhaps RoboSource doesn't really support multiple versions of help source files. I would be happy to admit that I'm wrong here.
Another option to solve our challenge is by committing our source files to the development source system, SVN. I've noticed some threads discussing this. I'm unsure from reading these posts if RoboHelp alerts you when you are about to modify a file, such fpj or hhk, that is being used by someone else like RoboSource does. Since we still have two authors playing in the same help project it would be most efficient if it would notify us before letting us write over one another's work.
Can anyone shed some light on:
1) does RoboSource really support multiple versions of the help source files and how it is done?
2) does RoboHelp notice when a file is locked in SVN and notify the other author to keep hands off until it is committed again?
3) is there somewhere better for me to research this?
Thanks you for any assistance you can provide,
Nita

We're using RSC 3.1 with RH 8 and with RH X5, and we've found that this is actually a two-step process:
Branching is the method for copying a snapshot of a folder (to establish two branches of the same source).
Reconciling (AKA forward/back patching) is the method for synchronizing edits between branches (or not, as the case may be).
For example, a "V12.3" branch of the "main" source has been prepared for one customer's specific needs. All edits to "main" will be forward patched to "V12.3," but edits to specific areas of "V12.3" will not be back patched to "main."
Tim, my boss, prepared the tutorials below, as much for himself as for us, and I have only stripped the screenshots to protect our intellectual property. Note also that we reference two publish locations, one for internal reviewing, and one for Release Engineering to include in our nightly product builds. Your environment might only have a single publish destination. Note also that the output files are never added to RSC.
Pay special attention to your folder naming process before you start, by anticipating any potential future branching needs. Folder renaming after the fact can be problematic in RH and RSC.
Good luck,
Leon
===================================
Branching Projects in RoboSource 3.1
Since the Help for RoboSource and Robohelp lack a proper explanation for branching and updating configuration files to support proper branching, this document has been created to assist in creating future branches.
Creating New Local Directories
Individuals must create new local path directories on their systems. We use the following conventions:
Ex. C:\902_imsmerge\projects (source folder)
Ex. C:\902_imsgenerate\mergedProjects (output folder)
Create New Server Directories
New server directories need to be created in two locations for holding the output files.
One is for the Robohelp output that gets put in the nightly .war file and checked into Perforce.
The other is in the wwwroot of our IIS server for our doc web site.
They are defined as follows:
\\docsql01\Documentation\RoboHelp\9.0.3\imsgenerate
\\docsql01\Inetpub\wwwroot\help_sys\9.0.3
Create new directories with the correct branch number.
Using RoboSource View Manager to Create a Branch
You use the View Manager to create a new branch. In RoboSource 3.1, a branch is just a different type of view.
Assume the version_903 branch has already been created. To create a new branch, click the Create button and name the new branch (version_910), leaving a blank root folder.
Click on Set Root to define the root folder for the new branch.
Select the root for the branch and click OK.
Select the view (branch) and click Properties. In the Properties window, select Branch as the Type, then click Close.
In the View Manager, check the check box to enable the view.
Setting the Local Path for the New Branch
Select the new branch and right-click.
Type in or browse to the correct local path, check the Apply Recursively box, click Apply. Close the Properties window.
Editing .xpj Files
The .xpj files in the new branch need to be edited to point to the new branch. THIS IS VERY IMPORTANT. IF NOT DONE ROBOHELP WILL CHECK IN CHANGES TO BOTH THE BRANCH AND THE ROOT.
One line in each project’s .xpj needs to change. See below.
This value must match the branch the project lives in. In our above new branch example this would be changes to %version_910.
In a merged project, you can change all these at once with the FAR tool.
Edit the .ssl and .sss files
These files need to be edited to redefine the generating and publishing paths. Note that this can also be accomplished with the FAR tool, or by having each writer do it through the GUI.
ssl Files
The .ssl files contain the data for the generate path. Although there might be seven .ssl files for each project, you only need to edit the ones for your Primary Layout and one or two others that you might also use.
Make sure the generate path is defined correctly.
sss Files
The .sss file defines the publishing paths. This edit can also be done through the GUI if necessary.
====================================
Forward and Back Patching in RoboSource
This document describes the procedure of forward or back patching files (.htm or .gif) between branches in the RoboSource system. The RoboSource product uses a feature called Reconcile Changes to accomplish patching between branches.
Example Scenario
Throughout this document we will use a real life example from the refwindows project.
We have edited the version 9.0.3 security_master_folder.htm file in RoboHelp.
Now, we need to backpatch it to version 9.0.2.
Selecting Folders or Files to Reconcile
You can select either:
A single file for reconciliation
The source folder, or
The destination folder
If you have just one file to reconcile, you will probably want to use the single file method. If you have several files to reconcile, you will need to use the folder method.
Working from the Source Folder
Select the source folder (folder that contains the updated file) in source control. In our case, %version_903/refwindows.
Select Reconcile Changes … from the Action menu. This opens the Reconcile window.
Select the appropriate option in the Reconcile window.
In our example we select the first option to reconcile from %version_903 because that is where our updated file is. Make sure the correct project is selected in the drop-down list. ($/version_902/refwindows).
Other Options
Reconcile changes made in another branch *into %verison_903/refwindows. This is the option you would use if you have an updated file in another branch that you want to put in 903.
Reconcile changes made locally (in c:\903_imsmerge\projects\refwindows) and %version_903/refwindows. We would most likely never use this option.
Reconcile changes made in another database *into* this database. Do not use.
None of the above, let me select what to Reconcile. This one sounds like it would be a good option, but it gets even more complicated.
Wildcard Filter. Use this to display only certain file types. For example, *.htm;*.gif;*.jpg would filter on those file types.
Click OK to open the Difference Reconciliator window.
Notice that this window shows all files in the project that have differences such as the .ssl and .xpj files.
IMPORTANT: NEVER reconcile these files, they are supposed to be different.
You can prevent these files from displaying by using the Wildcard Filter discussed above.
Select the appropriate file (security_master_folder.htm). This window will allow you to replace the file in the version_902 project unfortunately, it will also allow you to replace the updated file with the older one. Be Careful Here.
You can also View Differences from this window by clicking the View Differences link.
Click the Replace button in the $/version_902 pane.
You will get a confirmation dialog. Before clicking OK, take the time to read the question. This is your opportunity to catch potential mistakes!
You will then get the Reconcile Checkout dialog.
Be sure Perform the action directly on the Server is checked, click OK.
You see that the Difference Reconciliator window now indicates No changes to merge from %version_903/rewindows/security_master_folder.htm in the $version_902 pane.
Close the window.
Check the Destination Folder
In the source control window go to the destination branch and check the project. You can see in this shot that the security_master_folder.htm file now shows the date it was replaced, and that the server version is now different from your local version.
Working from the Destination Folder
If you had chosen the destination folder and select Reconcile Changes, the same window would have opened but with different options.
Here you would have selected the second option to reconcile into $/version_902.
You can do it either way, just pay careful attention to the options. The rest of the process is the same.

Exporting Large Files for Web Use

Most of my images are around 3000x2000 in filesize and between 10MB and 15MB. They are for use on the web so I don't want them to be too large (in MB's.)
What is the best option:
- keep original size and lower the quality-setting from 100% to -let's say- between 75% and 90%
- Resize them -let's say- to half size.
The large files have more pixels but are lower quality, or the resized files are higher quality but have less pixels - what is the best?

You don't need to but it is advisable as scaling always makes images soft.
You have to scale to display an image on a monitor as a monitor is a very
low resolution device. Luckily you can do the sharpening straight in the
export dialog and you don't really need to think about it. Sharpen for
screen at standard setting is usually just fine. So try using the following
settings in the export dialog:
File settings: jpeg, quality 75, Color Space sRGB
Image Sizing: resize to fit long edge, 800 pixels
Output sharpening: Sharpen for screen, amount Standard
Metadata: Minimize embedded metadata (preserves your copyright info
but omits all the edit and
camera info)
Then play
around with the sharpening amount and compare the exports to see what
you prefer. There is very little difference between High and Standard.

Splitting large files for upload

I need to upload a very large file to an FTP site, 9.8G. This takes a hugely long time with my bandwidth and invariablly fails somewhere before it finishes. Does anyone know if there is software that will take a large file like this, split it up, perhaps into disk images, so that they can be uploaded separately and then reassembled?
Thanks very much.

If you perform a web search there are some file splitters available for the Mac, I don't use them.
Hopefully you will find a suitable one.

Handling Large File

Hi all,
We need to handle a large file 88omb .is there any provision at the adapter level to break the file in to smaller chunks.
Can we avoid using the shell scripts ansd OS level commands.
Thanks,
Srinivas

Hi Srinivas,
if it is a text file then you could break up the file into multiple recordsets,
e.g.,
[Converting Text Format in the Sender File/FTP Adapter to XML |http://help.sap.com/saphelp_nwpi711/helpdata/en/44/658ac3344a4de0e10000000a1553f7/frameset.htm]
and
[#821267 File Adapter FAQ|http://service.sap.com/sap/support/notes/821267]
14. Memory Requirements
Regards
Kenny

How to send (larger files) for 'Shared Review'?

We are unable to get PDF files above 5.5MB to successfully 'Send for Shared Review' to a WebDav server (using Acrobat Pro 9.3.0 on OSX 10.5.8).
The 'save' process fails at the final stage & the PDF is not uploaded to the server. Files below that size can be sent for Shared Review without any problems. Have tested this on different WebDav servers & domains - and with various PDF files.
Has anyone been able to get PDF files above 5/6MB to successfully 'Send for Shared Review' to a WebDave server???
Is there a file size limit for individual PDF files for 'Shared Review' using a WebDav server???
Thanks for your help!!!

Try creating a review without sending the file and then send the link in a seperate email.
I used the "Do not send email invitation, just save localy" option and didn't add any reviewers in the wizard. Afterwards I sent a email with a link to the file to all reviewers and it worked.

Handling large files

Hi,
how to process large files in SAP XI?
Thanks,
Manogna

hi seshagiri
check the below blogs
huge processing of files
Night Mare-Processing huge files in SAP XI
Step-by-Step Guide in Processing High-Volume Messages Using PI 7.1's Message Packaging
Step-by-Step Guide in Processing High-Volume Messages Using PI 7.1's Message Packaging
SAP XI acting as a (huge) file mover
The specified item was not found.
Managing bulky flat messages with SAP XI (tunneling once again) - UPDATED
The specified item was not found.
regards
kummari
Edited by: Craig Cmehil on Jul 9, 2008 8:58 AM

Which format on a flash drive for large files for use by Mac and PC

I need to copy large files (9GB of movie file exported from iMovie of school graduation) onto 16GB flash drives so they can be used by school parents who may have Mac, PC or even TV.
My first attempt says the files are too large.
A lot of googling tells me this is a recognised problem but I am confursed by which advice to take.
I am on a 2012 iMac running OSX version 10.7.4.
Do I need to download some software to run NTFS. It all sounds so confusing!
I ended up in this predicament because the quality of my photo slideshows I copied to my DVD-R was so bad. Thought a flash drive would be easy. Ha, not at all.
Please answer in laymans terms - I could barely follow some of the advice I found.....

Format the flash drives with Disk Utility in ExFAT format.

Handling large files for displaying within a JTextPane

Similar Messages

Maybe you are looking for