ResultSet too large with huge query

Is there a way to 'flush' a java result set? I am retrieving approx. one million rows using a fetch size of 10. At about row #130,000 I receive an out of memory error. Is there a way to tell the result set to discard from memory all rows it has already retrieved?
Any ideas? The solution must be scalable, and breaking the queries up into smaller queries is not an option. In addition, simply increasing the jvm memory is not a permanent solution.
Thanks in advance!

>>
where does it say the result set and complementing
driver must load all the rows into memoryfor
any type of scrolling? it may be true that in your
experience there is less consumption of resouces bya
forward only cursor but i would certainly hope, nay
expect, that a JDBC driver by scalable regardlessof
this option.
i think a driver that "breaks" under certian load
conditions has got a bug.
This is the reverse of your previous point. Where
does it say that a driver must load a million rows?i have re-read this several times now and i'm not sure what you are saying. i think my point is that in my opinion a well-designed driver would throw a SQLException if due to limitations in it's implementation it cannot perform the request rather than attempt to do everything and cause a run time error. i think if i can make a piece of code cause an error and not an exception than that is a bug.
my original statement was that there is no spec in JDBC that states that a result set must implement a certain type of cursor in a given way other than what is part of the interface so i think it could be misleading to say that all drivers load all the rows of a result set if the cursor is scrollable. in fact i think if a cursor is TYPE_SCROLL_SENSITIVE it is highly likely that the result set would not be loading all the rows into memory as this caching defeats the purpose of a cursor senstive to changes made by others.
also again while the original poster blames thedriver
i am not convinced that indeed that is the point of
failure in their code. i suspect there is some
Collection in which they are attempting to storeall
the rows that is causing the error.
Could be their code - that is the source of most
programming problems. But it could be a limitation of
the driver or even the database.again this is what Exceptions are for no? i think such limitations and conditions are "reasonable" for an application to catch.

Similar Messages

JDBC Mysql Resultset too large!

Hello everybody,
I'm currently building a large database retrieval system(JAVA and mysql). Everything works fine, but I have a big problem. The returning resultset is way higher then it should be!!
The table has the following structure:
Probe_set_id unsigned mediumint(6)
Expressionvalue float(5,1)
Dataset unsigned tinyint(2)
I'm trying to retrieve a dataset with 6.5 million rows with the following query.
Select Probe_Set_ID, Expressionvalue from Test5 where dataset=1
This results in JAVA taking around 500 megabyte retrieving the Resultset!!! If my calculations are right it should take around the following:
Float 4 bytes * 6500000 = 26000000 bytes
unsigned mediumint = long 8 bytes * 6500000 = 52000000 bytes
That would be 78000000 bytes, which is around 75 megabyte and not 500 megabyte!!!!!
Code sample:
               String dataBaseName = "testDB";
               String dataBaseUser = "java";
               String dataBasePass = "clustering";
               String url = "jdbc:mysql://localhost/" + dataBaseName;
               Class.forName ("org.gjt.mm.mysql.Driver").newInstance ();
               Connection conn = DriverManager.getConnection (url, dataBaseUser, dataBasePass);
               conn.setReadOnly(true);
               Statement s = conn.createStatement();
               s.execute("Select Probe_Set_ID, Expressionvalue from Test5 where dataset=1");
               ResultSet rs = s.getResultSet();
This results in a giant resultset and I can't see the answer. I hope someone can help me. I'm using the newest JDBC driver of mysql!

Yes the JAVA application really takes up this amount! It is really akward. The DB doesn't take a real big amount of memory because of the buffer sizes I've set. I execute the java application as follow:
./java -server -XX:MaxHeapFreeRatio=5 -XX:MinHeapFreeRatio=20 -Xmx1512M dbTest
I'm really trying to keep the heap small because the JAVA application ( a gene expression processing application/server) takes big amounts of memmory when downloading data from the mysql database and the server is accesible by multiple users parallel. Though I'm still testing alone, so the memmory abuse is not a cause by other users.

File Size TOO LARGE with linked files

Recently, I am working with illustrator CS6.
After inserting quite a lot of linked images (without embadding), the file size becomes much larger, up too nearly 400MB.
I saved without pfd competatble funcion already, but the file size is sill verry large
I did not face the same problem working with CS5 before, is that a bug for CS6?
or are there any suggested ways to solve?
THANKS

Are you "Placing" the image files or are you copying and pasting the files as so-called smart objects where they come in on their own layer? See if you can flatten the file. Another option is build the images in Photoshop and bring them in as one image.

Print is too large with hp envy 5530

I have the Hp Envy 5530. After upgrading to windows 10 when I try to print online the print is huge. It takes about 3 pages to print something that normally would print on one page or less. This only happens when I am printing online. Any suggestions?

Hello , I would like to see if I can help with printing from the internet on Windows 10. Are you using the new web browser Microsoft Edge? How does the print preview show? This video goes through the print settings in Microsoft Edge, maybe it might help you find the desired print settings. Maybe changing the scale settings?
I also found this video; Please let me know if any of these steps resolved your issue, or if there is anything else I can do to help. I look forward to hearing from you! Thanks,

IPhoto says print too large on cover

on the cover of my picture book on iPhoto it keeps giving me the red triangle exclamation point even when there is no print in the box. I am trying to print the cover title in Nahum brush script in 24 or smaller even and it won't let me. I deleted all print from the one picture cover in the box and it still says too large with no print. I am ready to order right now and don't know how to fix this. Please help ASAP.
I want the largest print I can get in the box under the one cover picture in journal style with nanum brush script. Hope this makes sense.
thank you,
JP

That is a warning - not a fatal error
BUT if you print and the results are bad there is no guarantee since you went ahead
Before ordering your book preview it using this method - http://support.apple.com/kb/HT1040 - and save the resulting PDF for reference - the delivered book will match it.
My best guess is that you will have to do a much smaller point size - start at 12 and see if that works - it is it ok increase slowly until you get the warning and then back down one step
LN

Query is allocating too large memory error in OBIEE 11g

Hi ,
We have one pivot table(A) in our dashboard displaying , revenue against a Entity Hierarchy (i.e we have 8 levels under the hierarchy) And we have another pivot table (B) displaying revenue against a customer hierarchy (3 levels under it) .
Both tables running fine under our OBIEE 11.1.1.6 environment (windows) .
After deploying the same code (RPD&catalog) in a unix OBIEE 11.1.1.6 server , its throwing the below error ,while populating Pivot table A :
Error Codes: OPR4ONWY:U9IM8TAC:OI2DL65P
*State: HY000. Code: 10058. [NQODBC] [SQL_STATE: HY000] [nQSError: 10058] A general error has occurred. [nQSError: 43113] Message returned from OBIS. [nQSError: 96002] Essbase Error: Internal error: Query is allocating too large memory ( > 4GB) and cannot be executed. Query allocation exceeds allocation limits. (HY000)*
But , pivot table B is running fine . Help please !!!!!
data source used : essbase 11.1.2.1
Thanks
sayak

Hi Dpka ,
Yes ! we are hitting a seperate essbase server with Linux OBIEE enviorement .
I'll execute the query in essbase and get back to you !!
Thanks
sayak

Query Error Information: Result set is too large; data retrieval ......

Hi Experts,
I got one problem with my query information. when Im executing my report and drill my info in my navigation panel, Instead of a table with values the message "Result set is too large; data retrieval restricted by configuration" appears. I already applied "Note 1127156 - Safety belt: Result set is too large". I imported Support Package 13 for SAP NetWeaver 7. 0 BI Java (BIIBC13_0.SCA / BIBASES13_0.SCA / BIWEBAPP13_0.SCA) and executed the program SAP_RSADMIN_MAINTAIN (in transaction SE38), with the object and the value like Note 1127156 says... but the problem still appears....
what Should I be missing ?????? How can I fix this issue ????
Thank you very much for helping me out..... (Any help would be rewarded)
David Corté

You may ask your basis guy to increase ESM buffer (rsdb/esm/buffersize_kb). Did you check the systems memory?
Did you try to check the error dump using ST22 - Runtime error analysis?
Edited by: ashok saha on Feb 27, 2008 10:27 PM

Bex Report Error -- Query is Too Large

Hello I am Using Hierarchies in Rows and in Columns Company Wise, Quarter Month Wise Values
I am Using 310 Rows and 100 columns As this is Summary Report I cannot go for Filters and I cannot
Decrease Keyfigures is there any solution for this , Please give me the valuable suggestion for this query.

If the query is too large and you are running out of memory, then either you should run the report with smaller selection (may be year -wise) or increase your server parametres like memory space etc.

Is there a reliable method for detecting that a query is too large?

I am writing some code (that uses OCI) to properly detect when a query string is too long for OCI and/or the Oracle database server. I can't find any specific error code information in the docs, so I just started firing off large queries to see what would happen.
The queries I am sending are >2MB in size, up to 16MB. If the queries are above ~10MB, I get the error "ORA-03113 'End-of-file on communications channel'" after a fairly short amount of time (i.e. not enough for a timeout to expire). If the queries are below ~10MB, but above ~2.5MB, it either just sits there and does not do anything (for more than 15 hours). So watching for ORA-03113 when executing large queries does not seem like a very reliable method for detecting the queries that are too large.
Does anyone know of a reliable way of detecting that a query is too large for the OCI client and/or the Oracle database server?
I am using version 10.2.0.1 for both the OCI client and the Oracle database server, but I'm getting similar errors for combinations of 10.2.0.1 and 9.2.0.7 for both the OCI client and the Oracle database server.
These large queries need to be handled properly (i.e. distinguished from some generic failure) because the server handles requests from users, which could be programs that generate SQL queries (and have constructed huge ones in the past).
Thanks for any information!

The ORA-03113 means that the Oracle server process has died trying to satisfy your request. In almost all cases it is strictly correct to call that a bug, and we shouldn't easily forgive the server process when it happens. But in the case of multimegabyte statements my anger and disapointment turns to sympathy, for in my heart I can't bring myself to blame it. Can you?

Query is allocating too large memory Error ( 4GB) in Essbase 11.1.2

Hi All,
Currently we are preparing dashboards in OBIEE from the Hyperion Essbase ASO (11.1.2) Cubes.When are trying to retrieve data with more attributes we are facing the below error
"Odbc driver returned an error (SQLExecDirectW).
Error Details
Error Codes: OPR4ONWY:U9IM8TAC:OI2DL65P
State: HY000. Code: 10058. [NQODBC] [SQL_STATE: HY000] [nQSError: 10058] A general error has occurred. [nQSError: 43113] Message returned from OBIS. [nQSError: 96002] Essbase Error: Internal error: Query is allocating too large memory ( > 4GB) and cannot be executed. Query allocation exceeds allocation limits. (HY000)"
Currently we have data file size less than 2GB so we are using "Pending Cache Size=64MB".
Please let me know which memory I have to increase to resolve this issue
Thanks,
SatyaB

Hi,
Do you have any dynamic hierarchies? What is the size of the data set?
Thanks,
Nathan

SAP BI Query 'too large'

Hi,
I have a problem with creating a special query in Query Designer. The query has 2 structures, one structure has 100 selections, the second structure has 132 selections. This query is to show 10 years of plan data per period and a detailed account structure. The customer is not willing to reduce the number of rows.
When I try to check the query or execute the query, I get an error message:
Query is too large. Query contains 8192 differing selection cells. However, only 8191 selection cells can be processed in a query.
In fact the query has about 13000 selection cells!
I can not set up the query in a different way by using a drilldown by attribute period or something similar as the query should show actuals to a certain period in a certain year and from that period on the rest of the ten years should show plan data for an entered plan version. The row structure should not show all accounts and nodes of a hierarchy but only certain nodes. Therefore I can not use a hierarchy here.
Please could you tell me if there is a way to exceed the limit of '8192 selection cells' in a query ?
many thanks for your help in advance
Arndt Fritzsche-Marx

Hi Arndt,
have never had this problem before. However you can customize the amount of cells in a query yourself. Have you tried editing this number (as explained below), perhaps it helps?? For I do not have access to a BI 7.0 right now I can not check if there are more settings which can be customized.
You can customize the maximum of rows which can be displayed in web. This setting can be customized as follows in Bex Web Analyzer for the query (execute your query from Query Designer and it will ope in Bex Web Analyzer):
1. Settings --> Data Provider
2. Size Restriction for Result Sets
3. Set the setting to "Custom-Defined No. of Cells" and enter your required "Number of cells"
However be aware that the display of such a big amount of rows has a great impact on the performance of the query.
Brgds,
Marcel

Financial Reports 9.3.1 - Query is too large and cannot be executed

Hi,
I'm trying to pull a Financial report from one of my ASO cube, version 9.3.1 and received the following error.
"Query is too large and cannot be executed. The product of member counts across all dimensions in the query exceeds 2^64..."
The report is pulling from a combination of 6 different cross dimensions and drill down to the bottom level 0. I have read a couple of blogs with trying to change the buffer and sort buffer size from 10 KB to like 1000 KB within the database settings as well as the Pending cache size limit setting from like 32 MB to 64 MB on the Application setting. I've tried all combinations but nothing seems to work.
However, I was able to retrieve the same combinations through Essbase Excel Add-in by addding the MAXFORMULACACHESIZE in the Essbase config setting. But this didn't work for the Financial Reports. I know that in version 11.1.2 has that MDX Query enable through reports which may help with the query. Unfortunately, that MDX Query enable is not available within version 9.3.1.
Please help.
Thanks

John,
The Database has data that is rolled up to all levels.
I can open the report to Edit in the Financial Reporting Studio. This allows me to click on members in the grid, however, when I search for members I am unable to do so. I can drill down to the member and select it, but when I run the report in FR studio, I get the same error message 'Error: Invalid Report Object'.
Thanks and Regards
Kunal Tripathi

I created a 23 minute slideshow with music. how can I share it? youtube and smugmug both say it is too large?

I created a 23 minute slideshow with music of family vacation. Youtube and smugmug both said it was too large to post for my family.
suggetions please?? I use to use Mobileme...

A few different ways about this. You could split the slide show into smaller sections that are withi the limits for posting and then post each section.
You could try a different encoding to make the file smaller. What settings did you use when you created and exported the slide show?

Query throwing Exception as 'Query too large'

HI,
AM working on BI Content 7.
I have created a query and the query name is TEST_RA_0064.
when i am executing this query in the Analyser it is throwing the below mentioned errors.
1. Query TEST_RA_0064 is too large.
2. Program error in class SAPMSSY1 method:UNCAUGHT_EXCEPTION.
Please help me.
Thanks,
Rajesh janardanan

Execute this query via RSRT - for investigation purpose..
Check for the detailed error message here.
Hope it Helps
Chetan
@CP..

Query is too large

Hi,
Can any one give me the solution to develop this report.
I have 12 columns under each vertical.Like that i have 12 verticals for my Gross margin Report.
Then 12* 12=144 coloumns.total are 144.
But in Bi-Bex it will allow only limited columns. When i execute the report i'm getting the error that query is too large.Kow to overcome this.
Note Points :
Vertical1 : ITG
This Vertical consists of 20 Profit centers
Vertical2 : UTG
                This Vertical consists of 60 Profit centers
Vertical3 : SBI
                This Vertical consists of 50 Profit centers
Vertical4 : SERVICEDIVISIONS
                This Vertical consists of 10 Profit centers
Vertical5 : ENGG
                This Vertical consists of 50 Profit centers
Vertical6 : HITECH
                This Vertical consists of 10 Profit centers
u2022     Data Before Eliminations is Total of All the Verticals Data.
u2022     Eliminations Contains Values For Each Month for any of the above profit centers.
u2022     Data After Eliminations is Data Before Eliminations + Eliminations Data.
I searched in SDN but no luck..........
Thanks

hi sony,
one way to go about this is to increase the "Size Restriction for Result Sets". This is pretty tricky. I wouldn't really recommend this since this can have performance issues.
you can increase the size for result sets by running the abap report SAP_RSADMIN_MAINTAIN , Again be carefull when you are doing this.
Follow the below steps.
1)     Transaction Code SA38 or SE38
2)     In the program field, enter the report name SAP_RSADMIN_MAINTAIN and choose Execute.
3)     For OBJECT, enter one of the following parameters;
        BICS_DA_RESULT_SET_LIMIT_DEF
        BICS_DA_RESULT_SET_LIMIT_MAX
For VALUE, enter the value for the size of the result set, and then execute the program:
Also, for your information, the default setting for the result set of Web applications is 500,000 cells. If the result set exceeds the size specified using the report, a message is displayed to inform the user.
I would again ask you to be careful when you are modifying these values.
Best would be to make sure that you check the table RSADMIN, whether you have the above entries (BICS_*), if you don't u can update these entries in the system. via the above said abap report.
Hope this helps.
regards,
Sree.
Edited by: Sree Nair on Aug 18, 2009 1:14 PM
Edited by: Sree Nair on Aug 18, 2009 1:23 PM

ResultSet too large with huge query

Similar Messages

Maybe you are looking for