How to save memory when processing large result set

I need to dump multi millions of rows of data into excel files
I query the tables and open excel to write in
The problem is even I chopped the result into hundred files, close excel completely after 65536 rows, the memory usage keeps going up as the result set is looped and at one point hit the heap size
How can I release the memory has been used in the result set?
Thank you

mycoffee wrote:
936517 wrote:
I think resultSet.close() will do what you want (you shouldn't have to set resultSet=null when you're done with it).
You can't force the garbage collector to run and reclaim memory. It uses an intelligent algorithm to do so .
I question why your project is sending millions of records to excel. Who is going to read a 10,000 page excel document(s)?
Instead, I suggest you provide a (intelligent) filter mechanism to allow users to get a subset of data to send to an excel document rather than all data. For example: instead of sending him the entire telephone book, have him search for results based on lastName and/or firstName. That will cut down on the number of records returned. Next, does the user really need all the columns of data in each record? That will cut it down further.
You can search Google for 'java heap size' to increase the memory for your program. However, your 65536 limit is probably due to Excel's limitation and not your Java program.Sorry I could not explain the need,
No. That is not issue here. I already use max heap size I can
but I can handle it now. Open files, write directly to the file instead of holding the data and dumping all at once. I save all the overhead and it works fine even the result set still consumes almost all the memory.is it possible you are using mysql? the mysql jdbc driver has a terrible default setup in that it keeps all results for the result set in memory ! i think some of the latest drivers finally allow you to stream results sensibly, but you have to use the correct options.

Similar Messages

  • I-bot not emailing when report returns large result set..

    Hi,
    I am trying to set up an i-bot to run daily and email the results to the user. Assuming the report in question is Report_A.
    Report_A returns around 60000 rows of data without any filter condition. When I tried to set up thei-bot for Report_A (No filter conditions on the report) the ibot is publishing results to dashboard but is not delivering via email. When I introduce a filter in Report_A to reduce the data returned then everything works fine and email is being sent out successfully.
    So
    1) Is there a size limit for i-bots to deliver by email?
    2) Is there a way to increase the limits if any so the report can be emailed even when returning large result sets?
    Please let me know.

    Sorry for late reply
    Below is the log file for one of the i-bots. Now I am getting an error message "***kmsgPortalGoRequestHasBeenCancelled: message text not found ***" and the i-bot alert message shows as "Cancelled".
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:04.551
    [nQSError: 77006] Oracle BI Presentation Server Error: A fatal error occurred while processing the request. The server responded with: ***kmsgPortalGoRequestHasBeenCancelled: message text not found ***
    Error Codes: YLKKAV7S
    Error Codes: AGEGTYVF
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:04.553
    iBotID: /shared/_ibots/common/TM/Claims Report
    ...Trying iBot Get Response Content loop again.
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:04.554
    ... Sleeping for 8 seconds.
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:12.642
    [nQSError: 77006] Oracle BI Presentation Server Error: A fatal error occurred while processing the request. The server responded with: ***kmsgPortalGoRequestHasBeenCancelled: message text not found ***
    Error Codes: YLKKAV7S
    Error Codes: AGEGTYVF
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:12.644
    iBotID: /shared/_ibots/common/TM/Claims Report
    ...Trying iBot Get Response Content loop again.
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:12.644
    ... Sleeping for 6 seconds.
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:18.730
    [nQSError: 77006] Oracle BI Presentation Server Error: A fatal error occurred while processing the request. The server responded with: ***kmsgPortalGoRequestHasBeenCancelled: message text not found ***
    Error Codes: YLKKAV7S
    Error Codes: AGEGTYVF
    +++ ThreadID: f3c6cb90 : 2010-12-17 23:55:18.734
    iBotID: /shared/_ibots/common/TM/Claims Report
    Exceeded number of request retries.

  • How to save message when getting "email not responding"

    how to save message when getting "email not responding"

    How could you even know if you have a message if the email account isn't responding?
    Quit the mail app and reboot your iPad.
    Tap the home button once. Then tap the home button twice and the recents tray will appear at the bottom of the screen. Tap and hold down on any app icon until it begins to wiggle. Tap the minus sign in the upper left corner of the mail icon. Tap the home button twice.
    Reboot the iPad by holding down on the sleep and home buttons at the same time for about 10-15 seconds until the Apple Logo appears - ignore the red slider if it appears on the screen - let go of the buttons. Let the iPad start up.

  • How to handle large result set of a SQL query

    Hi,
    I have a question about how to handle large result set of a SQL query.
    My query returns more than a million records. However, the Query Template has a "row count" parameter. If I don't specify it, it by default returns only 100 lines of records in the query result. If I specify it, then it's limited to a specific number.
    Is there any way to get around of this row count issue? I don't want any restriction on the number of records returned by a query.
    Thanks a lot!

    No human can manage that much data...in a grid, a chart, or a direct-connected link to the brain. 
    What you want to implement (much like other customers with similar requirements) is a drill-in and filtering model that helps the user identify and zoom in on data of relevance, not forcing them to scroll through thousands or millions of records.
    You can also use a time-based paging model so that you only deal with a time "slice" at one request (e.g. an hour, day, etc...) and provide a scrolling window.  This is commonly how large datasets are also dealt with in applications.
    I would suggest describing your application in more detail, and we can offer design recommendations and ideas.
    - Rick

  • How to handle large result sets?

    Hi All,
    I have a large result set to be displayed to user using jsp's. Problem is that result set is too big, so I can't display all the records in a single push. I want to show the results page by page say 25 per page. Now for every page I have to fetch data from database, means there are going to be many database calls which is not advisable. Or i can cache data in a CachedRowSet to reduce database calls, but in this case you have to store all the data in memory which is not a good solution in case you have very large data sets. Can anybody suggest me a solution to this problem?

    The best thing for you to do is to implmeneting paging logic in conjunction with a scrollable resultset (JDBC 2.0+).
    The logic would go like this assuming 30 rows per page:
    - keep track of which page the user is on (e.g. page 3)
    - issue the full sql
    - scroll thru only the rows in the current page (e.g. rows 90-120)
    - copy the page's rows to value objects
    - close the resultset, statement, and connection
    In the above example, you would scroll to row 90 using rs.absolute(90).
    The efficiency comes from the fact that you're using a scrollable resultset. By using this, only the rows that you scroll thru are extracted out from the database. I performed some simple testing and with my data, and the scrollable resultset was about 10x in performance.
    Good luck!

  • Web Services with Large Result Sets

    Hi,
    We have an application where in a call to a web service could potentially yield a large result set. For the sake of argument, lets say that we cannot limit the result set size, i.e., by criteria narrowing or some other means.
    Have any of you handled paging when using Web Services? If so can you please share your experiences considering Web Services are stateless? Any patterns that have worked? I am aware of the Value List pattern but am looking for previous experiences here.
    Thanks

    Joseph Weinstein wrote:
    Aswin Dinakar wrote:
    I ran the test again and I removed the ResultSet.Fetch_Forward and it
    still gave me the same error OutOfMemory.
    The problem to me is similar to what Slava has described. I am parsing
    the result set in memory storing the results in a hash map and then
    emptying the post processed results into a table.
    The hash map turns out to be very big and jvm throws a OutOfMemory
    Exception.
    I am not sure how I can turn this around -
    I can partition my query so that it returns smaller chunks or "blocks"
    of data each time(say a page of data or two pages of data). Then I can
    store a page of data in the table. The problem with this approach is
    that it is not exactly transactional. Recovery would be very difficult
    in this approach.
    I could do this in a try catch block page by page and then the catch
    could go ahead and delete the rows that got committed. The question then
    becomes what if that transaction fails ?It sounds like you're committing the 'cardinal performance sin of DBMS processing',
    of shovelling lots of raw data out of the DBMS, processing it in some small way,
    and sending it (or some of it) back. You should instead do this processing in
    a stored procedure or procedures, so the data is manipulated where it is. The
    DBMS was written from the ground up to be a fast efficient set-based processor.
    Using clever SQL will pay off greatly. Build your saw-mills where the trees are.
    JoeYes we did think of stored procedures. Like I mentioned yesterday some of the post
    processing depends on unicode and specific character sets. Java seemed ideally suited
    to this since it handles these unicode characters very well and has all these libraries
    we can use. Moving this to DBMS would mean we would make that proprietary (not that we
    wont do it if it became absolutely essential) but its one of the reasons why the post
    processing happens in java. Now that you mention it stored procedures seem the best
    option.

  • How to iterate the webservice Data control result set?

    Hi all,
    How to iterate the webservice Data control result set? I have an jsff page where I am displaying the single UserDetails by webservice DataContol. As per my design requirement I should iterate the DataControl resultSet to get the user details and push the same in to Managed bean. Please help me how to do this, any sample code please to iterate the resultset and get the data from it.
       <?xml version='1.0' encoding='UTF-8'?>
       <jsp:root xmlns:jsp="http://java.sun.com/JSP/Page" version="2.1"
              xmlns:af="http://xmlns.oracle.com/adf/faces/rich"
              xmlns:f="http://java.sun.com/jsf/core"
              xmlns:c="http://java.sun.com/jsp/jstl/core">
       <c:set var="uiBundle" value="#{adfBundle['edu.syr.oim.uiBundle']}"/>
       <af:pageTemplate viewId="/templates/jsffTemplate.jspx">
        <f:facet name="fTop"/>
        <f:facet name="fCenter">
          <af:panelGroupLayout layout="scroll" inlineStyle="width:100.0%;">
       <af:panelTabbed id="pt1">
        <af:showDetailItem text="#{uiBundle.PRIVACYFLAGS}" id="sdi4">
                <af:panelGroupLayout id="pgl4" layout="scroll">
                  <af:spacer width="10" height="10" id="s3"/>
                  <af:panelFormLayout id="pfl6">
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTEMAIL}"
                                             id="plam35">
                      <af:outputText value="#{bindings.stuEmail.inputValue}"
                                     id="ot42"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTHOMEADDRESS}"
                                             id="plam39">
                      <af:outputText value="#{bindings.stuPermAddr.inputValue}"
                                     id="ot35"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTHOMEPHONE}"
                                             id="plam40">
                      <af:outputText value="#{bindings.stuPermPhn.inputValue}"
                                     id="ot37"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTCURRENTPHONE}"
                                             id="plam42">
                      <af:outputText value="#{bindings.stuCurrAddr.inputValue}"
                                     id="ot40"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTCURRENTPHONE}"
                                             id="plam36">
                      <af:outputText value="#{bindings.stuCurrPhn.inputValue}"
                                     id="ot38"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.STUDENTACAINFO}"
                                             id="plam41">
                      <af:outputText value="#{bindings.stuAcad.inputValue}"
                                     id="ot36"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.EMPHOMEADDRESS}"
                                             id="plam38">
                      <af:outputText value="#{bindings.empPermAddr.inputValue}"
                                     id="ot39"/>
                    </af:panelLabelAndMessage>
                    <af:panelLabelAndMessage label="#{uiBundle.EMPHOMEPHONE}"
                                             id="plam37">
                      <af:outputText value="#{bindings.empPermPhn.inputValue}"
                                     id="ot41"/>
                    </af:panelLabelAndMessage>
                  </af:panelFormLayout>
                </af:panelGroupLayout>
              </af:showDetailItem>
       </af:panelTabbed> Above is my jsff code. Here how/where to add the phase listener to paopulate the managed bean while page render. Do I need to iterate the DC to get and push the each parameter in to ManagedBean or is there any easy way to do this by EL mapping directly at jsff. Please clarify.
    Thanks
    kln

    That is what exactly I am trying right now. I am binding each of my page fragment outputText item in to backing bean and by that way trying to populate the values.
    But the issue here is, the backing bean method doesn't getting any value until I hit any of the link or the button in the fragment. While loading the page the bean set and get is null. If i hit any link or button it is filled up with the vaule. As per my design, I would like to populate the bean method along with page load before any user action. But cant able to do this!! :(
    Below is my sample code what I am trying right now
            <af:panelLabelAndMessage label="#{uiBundle.NETID}" id="plam13">
                      <af:outputText value="#{bindings.netid.inputValue}" id="ot4" binding="#{UserDataBean.netId}"/>
           </af:panelLabelAndMessage>
    backing bean ex:
    private RichOutputText netId;
    static String netidVal;
        public void setNetId(RichOutputText netId) {
           netidVal= netId.getValue() == null? "":netId.getValue().toString();
           this.netId = netId;
        public RichOutputText getNetId() {
           return netId;
        public String getNetIdVal() {
           return netidVal;
        }Thanks
    kln

  • ALV: how to save context space for large tables ?

    Dear collegues,
    We are displaying an ALV table that is quite large. Therefore, the corrsponding DDIC structure and the WD context is large. This has an impact on performance and the load size of the program. Now we will enhance the ALV table again.
    Example: for an icon and its explaining tooltip that are displayed in the ALV: there is are context fields required like "SOURCE_FIELDNAME" for the tooltip as well as for for the icon. They need a lot of characters for each tooltip and icon).
    Question: do you have an idea, how to save context space for those ALV fields ?
    Best regards,
    Christian

    >We are displaying an ALV table that is quite large.
    Do you mean quite large as in a large number of columns or as in a large number of rows (or both)?  I assume that the problem is probably more related to a large number of rows.  For very large tables, you should consider using the table instead of the ALV. For very large tables you can even use a technique called context paging to keep only a subset of the data in the context memory at a time.  Here is a recent blog that I created on the topic with demonstrations of different techniques for table sharing, shared memory, and context paging when dealing with large tables in Web Dynpro ABAP:
    Web Dynpro ABAP: How Fast Can You Consume 1 Million Rows?

  • New to Photoshop;Viewing TV and want to save to MY Library to view later. Can't figure out how to save. When I open or click on My Library it shows I need to sign in and within a few seconds it shows my sign in but nothing changes on page.

    Trying to figure out how to save Adobe TV episode as I am new to Photoshop and need insruction... trying to open & save to My Library to view later. It instructs me to sign in but I am already signed in. Anyone know how?

    Those are the instructions for your computer's iTunes (i.e. using the Store menu at the top of your computer's iTunes and selecting 'View My Account' on it). If you are doing it on your iPad then are you following the instructions for iOS devices :
    To unhide your purchases
    Depending on the content type, open the App Store, iBooks, or iTunes Store on your device.
    Scroll to the bottom of the main page.
    If you're not already signed in, tap Sign In.
    Tap your Apple ID name, then tap View Account.
    Scroll down on the Account pane, and tap Hidden Purchases.
    Locate the item you would like to unhide then tap the Unhide button.

  • Displaying large result sets in Table View u0096 request for patterns

    When providing a table of results from a large data set from SAP, care needs to be taken in order to not tax the R/3 database or the R/3 and WAS application servers.  Additionally, in terms of performance, results need to be displayed quickly in order to provide sub-second response times to users.
    This post is my thoughts on how to do this based on my findings that the Table UI element cannot send an event to retrieve more data when paging down through data in the table (hopefully a future feature of the Table UI Element).
    Approach:
    For data retrieval, we need to have an RFC with search parameters that retrieves a maximum number of records (say 200) and a flag whether 200 results were returned. 
    In terms of display, we use a table UI Element, and bind the result set to the table.
    For sorting, when they sort by a column, if we have less than the maximum search results, we sort the result set we already have (no need to go to SAP), but otherwise the RFC also needs to have sort information as parameters so that sorting can take place during the database retrieval.  We sort it during the SQL select so that we stop as soon as we hit 200 records.
    For filtering, again, if less than 200 results, we just filter the results internally, otherwise, we need to go to SAP, and the RFC needs to have this parameterized also.
    If the requirement is that the user must look at more than 200 results, we need to have a button on the screen to fetch the next 200 results.  This implies that the RFC will also need to have a start point to return results from.  Similarly, a previous 200 results button would need to be enabled once they move beyond the initial result set.
    Limitations of this are:
    1.     We need to use custom RFC function as BAPI’s don’t generally provide this type of sorting and limiting of data.
    2.     Functions need to directly access tables in order to do sorting at the database level (to reduce memory consumption).
    3.     It’s not a great interface to add buttons to “Get next/previous set of 200”.
    4.     Obviously, based on where you are getting the data from, it may be better to load the data completely into an internal table in SAP, and do sorting and filtering on this, rather than use the database to do it.
    Does anyone have a proven pattern for doing this or any improvements to the above design?  I’m sure SAP-CRM must have to do this, or did they just go with a BSP view when searching for customers?
    Note – I noticed there is a pattern for search results in some documentation, but it does not exist in the sneak preview edition of developer studio.  Has anyone had in exposure to this?
    Update - I'm currently investigating whether we can create a new value node and use a supply function to fill the data.  It may be that when we bind this to the table UI element, that it will call this incrementally as it requires more data and hence could be a better solution.

    Hi Matt,
    i'm afraid, the supplyFunction will not help you to get out of this, because it's only called, if the node is invalid or gets invalidated again. The number of elements a node contains defines the number of elements the table uses for the determination of the overall number of table rows. Something quite similar to what you want does already exist in the WD runtime for internal usage. As you've surely noticed, only "visibleRowCount" elements are initially transferred to the client. If you scroll down one or multiple lines, the following rows are internally transferred on demand. But this doesn't help you really, since:
    1. You don't get this event at all and
    2. Even if you would get the event, since the number of node elements determines the table's overall rows number, the event would never request to load elements with an index greater than number of node elements - 1.
    You can mimic the desired behaviour by hiding the table footer and creating your own buttons for pagination and scrolling.
    Assume you have 10 displayed rows and 200 overall rows, What you need to be able to implement the desired behaviour is:
    1. A context attribute "maxNumberOfExpectedRows" type int, which you would set to 200.
    2. A context attribute "visibleRowCount" type int, which you would set to 10 and bind to table's visibleRowCount property.
    3. A context attribute "firstVisibleRow" type int, which you would set to 0 and bind to table's firstVisibleRow property.
    4. The actions PageUp, PageDown, RowUp, RowDown, FirstRow and LastRow, which are used for scrolling and the corresponding buttons.
    The action handlers do the following:
    PageUp: firstVisibleRow -= visibleRowCount (must be >=0 of course)
    PageDown: firstVisibleRow += visibleRowCount (first + visible must be < maxNumberOfExpectedRows)
    RowDown/Up: firstVisibleRow++/-- with the same restrictions as in page "mode"
    FirstRow/LastRow is easy, isn't it?
    Since you know, which sections of elements has already been "loaded" into the dataSource-node, you can fill the necessary sections on demand, when the corresponding action is triggered.
    For example, if you initially display elements 0..9 and goto last row, you load from maxNumberOfExpected (200) - visibleRows (10) entries, so you would request entries 190 to 199 from the backend.
    A drawback is, that the BAPIs/RFCs still have to be capable to process such "section selecting".
    Best regards,
    Stefan
    PS: And this is meant as a workaround and does not really replace your pattern request.

  • How to use stored procedure which returns result set in OBIEE

    Hi,
    I hav one stored procedure (one parameter) which returns a result set. Can we use this stored procedure in OBIEE? If so, how we hav to use.
    I know we hav the Evaluate function but not sure whether I can use for my SP which returns result set. Is there any other way where I can use my SP?
    Pls help me in solving this.
    Thanks

    Hi Radha,
    If you want to cache the results in the Oracle BI Server, you should check that option. When you run a query the Oracle BI Server will get its results from the cache, based on the persistence time you define. If the cache is expired, the Oracle BI Server will go to the database to get the results.
    If you want to use caching, you should enable caching in the nqsconfig.ini file.
    Cheers,
    Daan Bakboord

  • How do you report from a multiple-result-set stored procedure?

    Team,
    I am writing a report against a canned stored procedure which returns eight result sets.  That is, inside the stored procedure, there are eight SELECT statements which pump to the output. In ADO, you would read the first one, then use SqlDataReader.NextResult to advance the reader to the next one until you have done all eight.
    Q1: Have you ever written a Crystal Report against such a sproc? How?
    Q2: I see that I could pull this data using ADO.NET if I knew this: Have you ever read such a sproc into a dataset? When I drag-and-drop the sproc into the dataset designer, it only makes a table schema for the first result set. I desire to have a table schema for all eight.
    Thanks,
    ~ Shaun

    Q1: Have you ever written a Crystal Report against such a sproc? How?
    - best answered in the Crystal Reports Design forum, so please post there:
    SAP Crystal Reports
    Q2: I see that I could pull this data using ADO.NET if I knew this: Have you ever read such a sproc into a dataset? When I drag-and-drop the sproc into the dataset designer, it only makes a table schema for the first result set. I desire to have a table schema for all eight.
    - not a Crystal reports question. Perhaps better posted to some MS / .NET developer forum(?)
    Ludek

  • How do I show more information for result set, I need company name, company address and company phone number to show. Now it is just company name and company address no telephone number, I need the number to show as well in the first result set

    The result set I get for a search now only shows the company name and address, if I want the telephone number of the company in the result set, I have to click on the link to go the site information to get the number. I want the company name, address and phone number to show up in the result set without having to click on each hyperlink to get the telephone number...

    ???
    See my profile at the left for information you are missing.
    Who, when, why, how, how much, what is the URL

  • Calling a service via url when it has result set INPUT

    So i am trying to call this service via a javascript
    # Add users
    @Properties LocalData
    IdcService=ADD_USER
    dName=jsmith
    dUserAuthType=Local
    dFullName=Jennifer Smith
    dPassword=password
    dEmail=[email protected]
    @end
    @ResultSet UserAttribInfo
    2
    dUserName
    AttributeInfo
    jsmith
    role,contributor,15
    @end
    <<EOD>>
    so i make a call using
    /idc/?IdcService=ADD_USER&dName=jsmith&dUserAuthType=Local&dFullName=Jennifer Smith&dPassword=password&dEmail=[email protected]
    but how to get the result set UserAttribInfo added to the end of this

    Jason,
    Thanks for the good info. Based on your technique I was able to figure out how to do a pure json post of a result set. You just need to send a Content-Type of text/json, set IsJson=1 and send the data encoded in the request body in the following format:
        LocalData: {
            dName: 'user',
            dUserAuthType: 'Local'
        ResultSets: {
            UserAttribInfo: {
                fields: ['dName', 'AttributeInfo'],
                rows: [['user', 'role, guest, 15']]
    }

  • How to Colour the fields in the Result Set of the query

    Hi all,
    I am having a requirement wher the user want to view the queries results with some colour to the char in the result set.
    suppose customer is the row he want to view some customers with one color and some with different color.
    Is it possible in Bi.
    If so Plz let me know
    Regards

    Hi Priya,
    for the macro thing, you will have to create a workbook and store your query into it, because macros are attached to only workbooks and not queries.
    in the workbook screen (i.e. excel), go to tools --> Macro --> Visual Basic Editor. or simply press Alt + F11. this will take you to macro editor screen.
    you can use code that may look something like below to color the columns, you can also give constant column if they are fixed.
    sub ColorColumns(rngTarget As Range, _
        intColor As Integer)
    Dim c As Long
        With rngTarget
            .Interior.ColorIndex = intColour
            Next c
        End With
    End Sub
    You will find some easy and short snippets for this.
    Regards,
    Purvang

Maybe you are looking for