Very Large String

I have a String that is suppose to hold info of about 200 MB, but when the string grows my program crashes giving a OutOfMemory Exception.
Is there some JVM configuration that will help me solve this problem?
Or will shifting to Java5.0 help solve this sort of problems.
I understand that I have 128MB RAM which is lesser than 200MB what my program demands but I dont expect it to crash.

I have a one-gallon tank of water, which is less than
the five gallons I want to fill into it, but I don't expect it to overflowI very well understand what I am asking for.... I
expect my system (program) to become slow because I
guess we all know what Virtual memory stands for?Yes, very slow memory. o_O
I dont expect the JVM to be decision maker of when to
stop my application.It isn't.
You did not tell the JVM to allocate enough heap to hold objects that large.
You are free to configure your system with a large swap area
and tell the JVM to allocate a large heap.
Not sure if that will succeed though
(AFAIK the JVM requests a contiguous memory block for the heap - your success might depend on your OS).
Note, if you are building a large String, you should probably be using the StringBuffer class.
If your largest object is expected to be in the order of 200MB in size, you'll probably need a heap size in the order of half a gig.
Or will shifting to Java5.0 help solve this sort of problems.No, Java 1.5 will not alleviate your resource / configuration difficulties.

Similar Messages

To count number of occurances of a char in a very large string

Hi All,
I like to count the no of occurances of a char in a a very large string.
for example
char ch - 'c'
string str - "practical example is always needed"
c occured 2 times in the above string.
Thanks,
J.Kathir

> string str - "practical example is always needed"
Try to finish this:
        String str = "practical example is always needed";
        char search = 'c';
        int occurrence = 0;
        for(int i = 0; i < str.length(); i++) {
            // Use a method from the String-class to get a
            // char from a specific location in the String.
            // See: http://java.sun.com/j2se/1.5.0/docs/api/java/lang/String.html
        System.out.println("Occerrence of char "+search+" in \""+str+"\" is: "+occurrence);

Newbie question: Binding very large strings ( 4k) to OLAP query

Hi there guys
I am using Oracle 10.2 and I am trying to bind string variables to olap literals in my query string:
SELECT TRUNC(IND-TO_DATE('01011900','ddmmyyyy')+2),VAL1
               FROM (
                    SELECT *
                    FROM TABLE(
                         OLAP_TABLE(
                              'OLAPFAME.MAG_CRD DURATION SESSION',
                              :1 ))
                         MODEL
                              DIMENSION BY(IND)
                              MEASURES(VAL1,R2C1)
                              RULES UPDATE SEQUENTIAL ORDER())
               WHERE OLAP_CONDITION( R2C1, :2, 1 )=1
I try and setString() to params 1 and 2, but it doesnt seem to work. The reason I am trying this is because when I pass large (>4K) DML string literals inline, I get ORA-01704. Hence me trying to bind to the variables via prepared statements (not even sure if this will get me past this 4k problem)!
I have also read that I can store these large strings in variables and use those as a OLAP DML string via the ampersand syntax (not sure how to do this, ensuring that they would just have statement scope).
Sorry if this is obtuse, but I'm at my wits end after a long days "trying stuff" :(
Thanks in advance
Adam

Once again, I answer my own question. -_-;;
The second process (fetch_comments) seems to be unnecessary. The Automated Row Fetch seems to be able to handle the 32K by itself. I'm still curious why the second process didn't work though....

Very large string to internal text table

Hi,
I have a string that contains all the information read from a flat file and I want to put all that information in an internal table where each row is just 1024 char in order to contain a row of the original flat file.
In the string I can recognize the special character "carriage return" in order to separate identify each row but now my problem is that I don't know can I parse the string text.
Can you help me please?

hi marshal ,
do like this first read the internal table and replace the all occurencse of carriage return with '#' and split the string
at '#'.
constant : c_enter type c ,
              c_hash type c value '#' ..
data:
BEGIN OF wa_text ,
          text(1024) TYPE c,
END OF wa_text.
i_text LIKE STANDARD TABLE OF wa_text.
REPLACE ALL OCCURRENCES OF c_enter
            IN wa_text WITH c_hash.
SPLIT wa_text AT c_hash INTO: str1 str2 str3,
                          TABLE i_text.
regards,
sandeep
Edited by: Sandeep patel on Jul 14, 2008 12:43 PM

Checking a Sub String in a String of a very large file ?

Hi All,
I am having a 20mb file and i am coverting that 20mb file to a string.now i like to search whether the following substring appears in the string
a) One
b) Two
c) Three
I am just checking with String.indexOf() for each .
Is there is any other efficient way of searching this rather than the above method String.indexOf
please give me some suggestion how to perform searching in a very large string.
Thanks,
J.Kathir

We have to read line and check for the three strings
right ?
Is there is any other way to check the substring in
the whole string ?
Reading line by line and searching for the three
strings or Reading the entire file as string and
searching for the substring
which is better ?Depends on your definition of "better" - reading line by line saves memory, but is probably slower. Reading the entire file into memory then searching it is probably faster but takes more memory.
As a third option, if you're using version 1.5, you can use the Scanner class to read the file and search for all 3 strings at the same time...something like this:
Scanner scn = new Scanner(new File("myfile.txt"));
String regExp = "a|b|c";
String s;
while ((s = scn.findWithinHorizon(regExp)) != null) {
System.out.println("Found: " + s);
scn.close();I'm pretty sure this is more efficient as it looks for anything matching the regular expression as it goes through the String...but I haven't done any timings on each approach.

Passing a large string in URL

Hi
The scenario is as follows
There are two servers A and B on different locations. From Server A i want to send a request to a url on server B. The request contains a string parameter of very large value (more than 255 characters). The problem is that I cant use a get because of the restriction.
How can I do the above in a jsp page. The string is generated dynamically when the user clicks on a link and also contains special characters like (+%)
I know i need to encode the url. But how do i post to a url on a different server
It would be nice if i can get the complete code. Short of time and cant read much thrugh the docs :)
Thanks
Sairam

That depends, what type of field are you inserting the string into? What you need to do is check the size of the field and the length of the string. If the length exceeds the field size, then you either need to increase the field size or, if you know you will be generally inserting very large strings, change the field to a clob. Hope this helps.

Very large XML String parameters

Hi !
I'm using AXIS 1.x, websphere 5 --- The problem is - when i call webservice with xml (String) parameter upto size of 10kb-400kb.. it works fine..
But my application could genrate very large xml, like 900kb-1000kb even more.. When this large XML is sent as String parameter.. no reply is recieved back..
Can some body throw some light.. what is going wrong... and which approch to be followed.
Thanks a lot
@mit

Maybe this example on the XDB forum will be helpful...
XMLType view of Relational Content
XML type questions are best asked in that forum.
;)

Read very very large excel file into 2d array (strings or varient)

Hi all,
Long time user, first time poster on these boards.
Looking at the copius amounts of great info related to reading Excel data from .xls files into labview, i've found that every one i've found from various people use the ActiveX method (WorkSheet.Range) which two strings are passed, namely excel's LetterNumber format to specify start and end.
However, this function does not work when trying to query huge amounts of information. The error returned is "Type Mismatch, -2147352571" I have a very large excel sheet i need to read data from and then close the excel file (original file remains unchanged). However this file is gigantic (don't ask me, I didn't make it and I can't convince them to use something more appropriate) with over 165 columns at 1000 rows of data.I can read a large number of columns, but only a handful of rows, or vice versa.
Aside from creating a loop to open and close the excel file over and
over reading pieces of it at a time, is there a better way to read more
data using ActiveX? Attached is code uploaded by others (with very minor modification) as an example.
Thanks,
Attachments:
Excel Get Data Specified Field (1-46col).vi ‏23 KB

Hi Maddox731,
I've only had a very quick glance through your thread, and I must admit I haven't really thought it through properly yet. Sounds like you've come up with your own solution anyway. That said I thought I'd take a bit of a scatter gun approach and attach some stuff for you regradless. Please forgive my bluntness.
You'll find my ActiveX Excel worksheet reader, which may or may not contain the problem you've come across. I've never tried it with the data size you are dealing with, so no promises. I've also attached my ADO/SQL approach to the problem. This was something I moved onto when I realised the limitations of AX. One thing I have noticed is that ADO/SQL is much faster than AX, so there may be some gains for you there with large data sets if you can implement it.
I should add that I'm a novice to all this and my efforts are down to bits I've gleamed from MSDN and others' LV examples. I hope it's of some use, if only to spark discussion. Your ctiticism is more than welcome, good or bad.
Regards.
Attachments:
Database Table Reading Stuff.zip ‏119 KB

Slow Performance or XDP File size very large

There have been a few reports of people having slow performance in their forms (tyically for Dynamic forms) or file sizes of XDP files being very large.
These are the symptoms of a problem with cut and paste in Designer where a Process Instruction (PI) used to control how Designer displays a specific palette is repeated many many times. If you look in your XDP source and see this line repeated more than once then you have the issue:
The problem has been resolved by applying a style sheet to the XDP and removing the instruction (until now). A patch has been released that will fix the cut and paste issue as well as repair your templates when you open them in a designer with the patch applied.
Here is a blog entry that describes the patch as well as where to get it.
http://blogs.adobe.com/livecycle/2009/03/post.html

My XDP file grow up to 145mb before i decided to see what was actually happening.
It appears that the LvieCycle Designer ES program sometimes writes alot of redundant data... the same line millions of times over & over again.
I wrote this small java program which reduced the size up to 111KB !!!!!!!!!!!!!!!!!! (wow what a bug that must have been!!!)
Here's the sourcecode:
import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.FileWriter;
import java.io.IOException;
public class MakeSmaller {
private static final String DELETE_STRING = "                           <?templateDesigner StyleID aped3?>";
public static void main(String... args) {
BufferedReader br = null;
BufferedWriter bw = null;
try {
   br = new BufferedReader(new FileReader(args[0]));
   bw = new BufferedWriter(new BufferedWriter(new FileWriter(args[0] + ".small")));
   String line = null;
   boolean firstOccurence = true;
   while((line = br.readLine()) != null) {
    if (line.equals(DELETE_STRING)) {
     if (firstOccurence) {
      bw.write(line + "\n");
      firstOccurence = false;
    } else {
     bw.write(line + "\n");
     firstOccurence = true;
} catch (FileNotFoundException e) {
   e.printStackTrace();
} catch (IOException e) {
   e.printStackTrace();
} finally {
   if (br != null) {
    try {
     br.close();
    } catch (IOException e) {
     e.printStackTrace();
   if (bw != null) {
    try {
     bw.close();
    } catch (IOException e) {
     e.printStackTrace();
File that gets generated is the same as the xdp file (same location) but gets the extension .small. Just in case something goes wrong the original file is NOT modified as you can see in the source code. And yes Designer REALLY wrote that line like a gazillion times in the .xdp file (shame on the programmers!!)
You can also see that i also write the first occurrence to the small file just in case its needed...

Import very large csv files into SAP

Hi
We're not using PI, but have middleware called Trading Networks. Our design is fixed (not my decision) to not upload files into Application Server and import it from there. Our design dictates that we must write RFCs and Trading Networks will call the RFC per interface with very large file sent as table of strings. This takes 14 minutes to import into SAP plain Z-table from where we'll integrate. As a test we uploaded the file to Application Server and integrated into Z-table from there. This took 4 minutes. However our architect is not impressed that we'll stretch available Application Server to it's limits.
I want to propose that the large file be split in e.g. 4 parts at Trading Networks level and call 4 threads of the RFC which could reduce integration time to e.g. 3 minutes 30 seconds. Is there someone that has suggestions in this regard especially about a proposed, working, elegant solution for integrating large files with our current environment? This will form the foundation of our project.
Thank you and best regards,
Adrian

Zip compression can be tried. The RFC will receive zip stream which will be decompressed using CL_ABAP_ZIP.

Any suggestions on calculating with very large or small numbers?

It seems that double values are about 17 decimal places (10e17) in precision.
Is there a way in iPhone calculations to get more precision for very large and small numbers, like 10e80 and so forth? I know that's more than the entire number of atoms in the universe, but still.
I tried "long double" but that didn't seem to make any difference.
Just a limitation?
Thanks,
doug

Hmmm... maybe I was just having a problem with my formatted string then?
I was using the NSString %g format, which is supposed to print in exponential notation if the number is greater than 1e4 or less than 1e-4, or something like that.
But I was not getting anything greater exponents than 1e17 and then I was apparently getting overflows because the number were having negative mantissas.
All the variables involved were double...
How did you "look at" z?
Thanks,
doug

Numbers file sizes very large

I've noticed that file sizes in Numbers are very large. I was using Excel for a long time and all the files I created were between 20-30KB. The files consisted of an Excel workbook, with 1-2 worksheets in it. The same files in Numbers are 200-300KB and if I save them over the network from another computer to my computer, they jump up to over 1MB. Any ideas on this?

Hello
Nothing to do with graphics items.
An XL file as well as an AppleWorks one is a compiled document in which many components are stored in a very compact shape. One byte was sufficient in AW to represent an operand as they where aboutone hundred.
In Numbers, everything is described in pure text with complementary delimiters.
When a formula uses the operand "COUNTBLANK" this one appears with its 10 letters.
In XL as well as in AW6, a date is stored as a floating number while in Numbers it's stored as the string "mercredi 23 janvier 2008 22:33:19"
Sama thing for every attributes of every cell.
So this results in a huge file store in XML format. To spare space, when we close a document, the XML file is packed in .gz format.
This is the Index.xml.gz file that we may see clicking a Numbers document wit ctrl depressed and selecting the contextual menu item "Show Package's contents".
Double click onto Index.xml.gz will unpack it.
giving the expanded Index.xml file.
I assumes that they are applications dedicated to XML files.
I don't know them so I just drag and drop the XML file onto a free text editor named Bean which I find really interesting.
Doing that, we may examine the file's contents.
If someone knows a correct free application able to open and display correctly the XML files, I'm interested
Yvan KOENIG (from FRANCE mercredi 23 janvier 2008 22:49:23)

Very Large Numbers Question

I am a student with a question about how Java handles very large numbers. Regarding this from our teacher: "...the program produces values that
are larger than Java can represent and the obvious way to test their size does not
work. That means that a test that uses >= rather than < won?t work properly, and you
will have to devise something else..." I am wondering about the semantics of that statement.
Does Java "know" the number in order to use it in other types of mathematical expressions, or does Java "see" the value only as gibberish?
I am waiting on a response from the teacher on whether we are allowed to use BigInteger and the like, BTW. As the given program stands, double is used. Thanks for any help understanding this issue!

You're gonna love this one...
package forums;
class IntegerOverflowTesterator
public static void main(String[] args) {
    int i = Integer.MAX_VALUE -1;
    while (i>0) {
      System.out.println("DEBUG: i="+i);
      i++;
}You also need to handle the negative case... and that get's nasty real fast... A positive plus/times a positive may overflow, but so might a negative plus a negative.
This is decent summary of the underlying problem http://mindprod.com/jgloss/gotchas.html#OVERFLOW.
The POSIX specification also worth reading regarding floating point arithmetic standards... Start here http://en.wikipedia.org/wiki/POSIX I guess... and I suppose the JLS might be worth a look to http://java.sun.com/docs/books/jls/second_edition/html/typesValues.doc.html

How To Get rid of Exponential format in datagridview when the number is very large

When the number is very large like :290754232, I got 2.907542E +08. in datagridview cell
I using vb.net , framework 2.0.
how can I get rid of this format?
Thanks in advance

should I change the type of this column to integer or long ?
The datagridview is binded to binding source and a list ( Of).
Mike,
I'll show you an example that shows the correct way to do this and a another way if you're stuck using strings in exponential format. The latter being the "hack way" I spoke about Friday. I don't like it, it's dangerous, but I'll show both anyway.
In this example, I'm using Int64 because I don't know the range of yours. If your never exceeds Int32 then use that one instead.
First, I have a DataGridView with three columns. I've populated the data just by creating longs starting with the maximum value in reverse order for 100 rows:
The way that I created the data is itself not a great way (there's no encapsulation), but for this example "it'll do".
Notice though that the third column (right-most column) isn't formatted at all. I commented out the part that does that so that I could then explain what I'm doing. If it works, it should look like the first column.
The first column represents an actual Int64 and when I show the code, you can see how I'm formatting that using the DGV's DefaultCellStyle.Format property. That's how it SHOULD be done.
The third column though is just a string and because that string contains a letter in it, Long.TryParse will NOT work. This is where the "hack" part comes in - and it's dangerous, but if you have no other option then ...
You can see that now the third column matches the first column. Now the code:
Option Strict On
Option Explicit On
Option Infer Off
Public Class Form1
Private Sub Form1_Load(ByVal sender As System.Object, _
ByVal e As System.EventArgs) _
Handles MyBase.Load
With DataGridView1
.AllowUserToAddRows = False
.AllowUserToDeleteRows = False
.AllowUserToOrderColumns = False
.AllowUserToResizeRows = False
.AlternatingRowsDefaultCellStyle.BackColor = Color.Aquamarine
.ReadOnly = True
.SelectionMode = DataGridViewSelectionMode.FullRowSelect
.MultiSelect = False
.RowHeadersVisible = False
.RowTemplate.Height = 30
.EnableHeadersVisualStyles = False
With .ColumnHeadersDefaultCellStyle
.Font = New Font("Tahoma", 9, FontStyle.Bold)
.BackColor = Color.LightGreen
.WrapMode = DataGridViewTriState.True
.Alignment = DataGridViewContentAlignment.MiddleCenter
End With
.ColumnHeadersHeightSizeMode = DataGridViewColumnHeadersHeightSizeMode.DisableResizing
.ColumnHeadersHeight = 50
.DataSource = Nothing
.Enabled = False
End With
CreateData()
End Sub
Private Sub CreateData()
Dim longList As New List(Of Long)
For l As Long = Long.MaxValue To 0 Step -1
longList.Add(l)
If longList.Count = 100 Then
Exit For
End If
Next
Dim stringList As New List(Of String)
For Each l As Long In longList
stringList.Add(l.ToString("e18"))
Next
Dim dt As New DataTable
Dim column As New DataColumn
With column
.DataType = System.Type.GetType("System.Int64")
.ColumnName = "Actual Long Value (Shown Formated)"
dt.Columns.Add(column)
End With
column = New DataColumn
With column
.DataType = System.Type.GetType("System.String")
.ColumnName = "String Equivalent"
dt.Columns.Add(column)
End With
column = New DataColumn
With column
.DataType = System.Type.GetType("System.String")
.ColumnName = "Formated String Equivalent"
dt.Columns.Add(column)
End With
Dim row As DataRow
For i As Integer = 0 To longList.Count - 1
row = dt.NewRow
row("Actual Long Value (Shown Formated)") = longList(i)
row("String Equivalent") = stringList(i)
row("Formated String Equivalent") = stringList(i)
dt.Rows.Add(row)
Next
Dim bs As New BindingSource
bs.DataSource = dt
BindingNavigator1.BindingSource = bs
DataGridView1.DataSource = bs
With DataGridView1
With .Columns(0)
.DefaultCellStyle.Format = "n0"
.Width = 150
End With
.Columns(1).Width = 170
.Columns(2).AutoSizeMode = DataGridViewAutoSizeColumnMode.Fill
.Enabled = True
End With
End Sub
' The following is what I commented
' out for the first screenshot. ONLY
' do this if there is absolutely no
' other way though - the following
' casting operation is NOT ADVISABLE!
Private Sub DataGridView1_CellFormatting(ByVal sender As Object, _
ByVal e As System.Windows.Forms.DataGridViewCellFormattingEventArgs) _
Handles DataGridView1.CellFormatting
If e.ColumnIndex = 2 AndAlso e.Value.ToString IsNot Nothing Then
' NOTE! The following is dangerous!
' I'm going to use coercion to force the
' string into a type long. TryParse will
' NOT work here. This can easily throw an
' exception if the string cannot be cast
' to a type long. I'm "depending on" the
' the string to cast. At the very least
' you might put this in a Try/Catch but
' that won't stop it from failing (if
' it doesn't work).
Dim actualValue As Long = CType(e.Value.ToString, Long)
Dim formattedValue As String = actualValue.ToString("n0")
e.Value = formattedValue
End If
End Sub
End Class
Like I said, only use that hack way if there's no other option!
I hope it helps. :)
Still lost in code, just at a little higher level.

Sorting very large file

Hi,
I've a very large file (1.3G, more than 10 million records) that I need to sort, what is the best possible way to do that, without using a database?
Thanks,
Gary

Just a suggestion:
If you create a sorted red/black tree of record numbers (i.e. offsets in the data file), you could get an equivalent sort as follows (probably without a huge amount of memory...):
Here's the strategy:
Create a comparitor that compares two Long objects. It performs a seek in the data file and retrieves the records pointed to by those two Longs and compares them using whatever record comparitor you were going to use.
Create a sorted list based on that comparitor (a tree based list will probably be best)
Run a for loop going from 0 to the total records in the data file, create Long objects for each value, and insert that value into the list.
When you are done, the list will contain the record numbers in sorted order.
Create an iterator for the list
For each element in the list, grab the corresponding data file record from the source file and write it to the destination file.
voila
Some optimizations may be possible (and highly desirable, actually):
When you read records from the file, place them into an intermediate cache (linked list?) of limited length (say 1000 records). When a record is requested, scan the cache for that record ID - if found, move it to the top of the list, then return the associated data. If it is not found, read from the file, then add to the top of the list. If the list has more than X records, remove the last item from the list.
The upper pieces of the sort tree are going to get hammered, so having a cache like this is a good idea.
Another possibility: Instead of inserting the records in order (i.e. 0 through X), it may be desirable to insert them in pseudo-random order - I seem to remember that there was a really nice algorithm for doing this (without repeating any records) in an issue of Dr. Dobbs from awhile back. You need to get a string of random numbers between 0 and X without repeats (like dealing from a card deck).
If your data is already in relatively sorted order, the cache idea will not provide significant bennefit without randomizing the inputs.
One final note: Try the algorithm without the cache first and see how bad it is. The OSes disk caching may provide sufficient performance. If you encapsulate the data file (i.e. an object that has a getRecordBytes(int recNumber) method), then it will be easy to implement the cache later if it is needed.
I hope this provides some food for thought!
- K

Very Large String

Similar Messages

Maybe you are looking for