Working with large number of search results in struts/jsp application

I'm developing a struts/jsp web application that returns upwards of
30000 results obtained from a very complex schema and multiple queries
to the database.
We want the user to be able to paginate through the results, sort by
several fields, add/drop columns and download in several formats. We
don't expect heavy traffic but we do want it to be scalable.
Up to now, I have been storing the results in a session ArrayList
which can eat up a lot of memory. The queries take long to perform so
I do not want to repeat them with every request.
I would like to store in a single mysql temp table but I have read
that when the connection ends, the table is dropped. I understand that
creating a database Connection as a session variable is not a good
practice.
This is not a unique problem - I was hoping someone else has some
insight into the best way to proceed.
Thanks

glwinsor wrote:
I'm developing a struts/jsp web application that returns upwards of
30000 results obtained from a very complex schema and multiple queries
to the database.Who will want to deal with 30,000 rows? Not me. When Google returns a result to me, I get it in chunks of 10 at a time.
We want the user to be able to paginate through the results, sort by
several fields, add/drop columns and download in several formats. We
don't expect heavy traffic but we do want it to be scalable.That's a separate, complex issue.
Up to now, I have been storing the results in a session ArrayList
which can eat up a lot of memory. The queries take long to perform so
I do not want to repeat them with every request.Maybe your schema has issues. Do you have appropriate indexes? Has your DBA checked EXPLAIN PLAN to be sure that the queries are optimal?
I would like to store in a single mysql temp table but I have read
that when the connection ends, the table is dropped. I understand that
creating a database Connection as a session variable is not a good
practice.Not scalable.
This is not a unique problem - I was hoping someone else has some
insight into the best way to proceed.The paging part has been dealt with by Hibernate and other frameworks. Something like the Flex grid UI component might help. Caching and such can help you out. It's not easy, as you already know, and there isn't a canned solution that is one size fits all.
%

Similar Messages

  • How to limit the number of search results returned by oracle text

    Hello All,
    I am running an oracle text search which returned the following error to my java program.
    ORA-20000: Oracle Text error:
    DRG-51030: wildcard query expansion resulted in too many terms
    #### ORA-29902: Fehler bei der Ausführung von Routine ODCIIndexStart()
    ORA-20000: Oracle Text error:
    DRG-51030: wildcard query expansion resulted in too many terms
    java.sql.SQLException: ORA-29902: Fehler bei der Ausführung von Routine ODCIIndexStart()
    ORA-20000: Oracle Text error:
    DRG-51030: wildcard query expansion resulted in too many terms
    at oracle.jdbc.dbaccess.DBError.throwSqlException(DBError.java:169)
    When i looked up the net, one suggestion that was given is to narrow the wildcard query, which i cannot in my search page. Hence I am left with the only alternative of limiting the number of results returned by oracle text search query.
    Please let me know how to limit the number of search results returned by oracle text so that this error can be avoided.
    Thanks in advance
    krk

    Hi,
    If not set explicitly, the default value for WILDCARD_MAXTERMS is 5000. This is set as a wordlist preference. This means that if your wildcard query matches more than 5000 terms the message is returned. Exceeding that would give the user a lot to sift through.
    My suggestion: trap the error and return a meaningful message to the user indicating that the search needs to be refined. Although it is possible to increase the number of terms before hitting maxterms (increase wildcard_maxterms preference value for your wordlist), 5000 records is generally too much for a user to deal with anyway. The search is not a good one since it is not restricting rows adequately. If it happens frequently, get the query log and see the terms that are being searched that generate this message. It may be necessary to add one or more words to a stoplist if they are too generic.
    Example: The word mortgage might be a great search term for a local business directory. It might be a terrible search term for a national directory of mortgage lenders though (since 99% of them have the term in their name). In the case of the national directory, that term would be a candidate for inclusion in the stoplist.
    Also remember that full terms do not need a wildcard. Search for "car %" is not necessary and will give you the error you mentioned. A search for "car" will yield the results you need even if it is only part of a bigger sentence because everything is based on the token. You may already know all of this, but without an example query I figured I'd mention it to be sure.
    As for limiting the results - the best way to do that is to allow the user to limit the results through their query. This will ensure accurate and more meaningful results for them.
    -Ron

  • Change default value "Number of Search Results" - Component PPM_DYNSEARCH

    Hi Gurus,
    I am new to SAP CRM WebUI development and hope you can help me on following problem.
    In Pipeline & Performance Management we want to change the default value for the search results from 100 to 1000.
    SAP CRM 7.0
    Component: PPM_DYNSEARCH
    View: PPM_DYNSEARCH/Filter
    I have already tried to solve this via following solution:
    1) Assign parameter SEARCH_MAX_HITS to a parameter profile in the following IMG and set his value to 1000:
    CRM >> UI Framework >> Technical role definition >> Define parameters.
    2) Then, assign your parameter profile to your business role:
    CRM >> UI Framework >> Business role >> Define business role.
    However this works for all initial search screens but not for Pipeline and Performance Management.
    Can anyone give me a short hint how I can set the max number of search results from 100 to 1000 in the PPM Screen.
    Many Thanks
    Michael

    Hi Michael,
    Seems like this is a normal edit page, not a search page.
    Keep a break point at this code in the HTML page.
    <thtmlb:inputField id        = "maxHitsInput"
                               width     = "80%"
                               value     = "<%= lv_display_max_hits %>"
                               maxlength = "8"
                               design    = "DEFAULT"
                               type      = "INTEGER" />\
    Change lv_display_max_hits in debugging and see if that brings in the value on the UI.
    Normally for a search page there will be a THTMLB attribute maxHits where we pass the value. But in your case it is a normal view.
    Regards,
    Masood Imrani S.

  • Working with Large List in sharepoint 2010

    Hi All
    I have a list with almost 10k records in my sharepoint list and based on some business requirement i am binding (almost 6k records) the data to asp.net grid view and this will visible on the home page of the portal where most of the users will access. Can
    someone please guide the best method to reduce the performance inorder the program to hit the SP list everytime the page loads...
    Thanks & Regards
    Rakesh Kumar

    Hi,
    If you are Working with large data retrieval from the content database (SharePoint list), the points below for your reference:
    1. Limit the number of returned items.
    SPQuery query = new SPQuery();
    query.RowLimit =6000; // we want to retrieve2000 items
    query.ListItemCollectionPosition = prevItems.ListItemCollectionPosition; // starting at a previous position
    SPListItemCollection items = SPContext.Current.List.GetItems(query);
    2. Limit the number of returned columns.
    SPQuery query = new SPQuery();
    query.ViewFields = "";
    3. Query specific items using CAML (Collaborative Markup Language).
    SPQuery query = new SPQuery();
    query.Query = "15";
    4.Use ContentIterator class
    https://spcounselor-public.sharepoint.com/Blog/Post/2/Querying-a--big-list--with-ContentIterator-and-multiple-filters
    5. Create a Stored Procedure in Database to get the special data, create a web service to get the data, when create a web part to show the data in home page.
    Best Regards
    Dennis Guo
    TechNet Community Support

  • Is anyone working with large datasets ( 200M) in LabVIEW?

    I am working with external Bioinformatics databasesa and find the datasets to be quite large (2 files easily come out at 50M or more). Is anyone working with large datasets like these? What is your experience with performance?

    Colby, it all depends on how much memory you have in your system. You could be okay doing all that with 1GB of memory, but you still have to take care to not make copies of your data in your program. That said, I would not be surprised if your code could be written so that it would work on a machine with much less ram by using efficient algorithms. I am not a statistician, but I know that the averages & standard deviations can be calculated using a few bytes (even on arbitrary length data sets). Can't the ANOVA be performed using the standard deviations and means (and other information like the degrees of freedom, etc.)? Potentially, you could calculate all the various bits that are necessary and do the F-test with that information, and not need to ever have the entire data set in memory at one time. The tricky part for your application may be getting the desired data at the necessary times from all those different sources. I am usually working with files on disk where I grab x samples at a time, perform the statistics, dump the samples and get the next set, repeat as necessary. I can calculate the average of an arbitrary length data set easily by only loading one sample at a time from disk (it's still more efficient to work in small batches because the disk I/O overhead builds up).
    Let me use the calculation of the mean as an example (hopefully the notation makes sense): see the jpg. What this means in plain english is that the mean can be calculated solely as a function of the current data point, the previous mean, and the sample number. For instance, given the data set [1 2 3 4 5], sum it, and divide by 5, you get 3. Or take it a point at a time: the average of [1]=1, [2+1*1]/2=1.5, [3+1.5*2]/3=2, [4+2*3]/4=2.5, [5+2.5*4]/5=3. This second method required far more multiplications and divisions, but it only ever required remembering the previous mean and the sample number, in addition to the new data point. Using this technique, I can find the average of gigs of data without ever needing more than three doubles and an int32 in memory. A similar derivation can be done for the variance, but it's easier to look it up (I can provide it if you have trouble finding it). Also, I think this funtionality is built into the LabVIEW pt by pt statistics functions.
    I think you can probably get the data you need from those db's through some carefully crafted queries, but it's hard to say more without knowing a lot more about your application.
    Hope this helps!
    Chris
    Attachments:
    Mean Derivation.JPG ‏20 KB

  • Lookups with large number of records do not return the page

    Hi,
    I am developing an application using Oracle JHeadstart 10.1.3 Preview Version 10.1.3.0.78
    In my application I created a lookup under domains and used that lookup for an attribute (Display Type for this attribute is: dropDownList) in a group to get the translation fro this attribute. The group has around 14,800 records and the lookup has around 7,400 records.
    When I try to open this group (Tab), the progress shows that it is progressing but it does not open even after a long time.
    If I change the Display Type for the attribute from dropDownList to textInput then it works fine.
    I have other lookups with lower number of records. Those lookups work fine with dropDownList Display Type.
    Only I have this kind of problem when I have a lookup with large number of records.
    Is there any limitation of record number for lookups under Domains?
    How I can solve this?
    I need to translate the attribute (get the description from another table using the code).
    Your help would be appreciated.
    Thanks
    Syed

    We have also faced similar issue, but us, it was happening when we were using the dropDownList in a table, while the same dropDownList was working in table format. In our case the JVM is just used to crash and after google'ing it here in forums, found that it might be related to some JVM issue on Windows XP machines without Service Pack 2.
    Anyway... the workaround that we taken to get around the issue is to use LOV instead of a dropDownList in your jHeadStart.
    Hope this helps...
    - rutwik

  • Working with large Artboards/Files in Illustrator

    Hello all!
    I'm currently designing a full size film poster for a client. The dimensions of the poster are 27" x 40" (industry standard film poster).
    I am a little uncertain in working with large files in Illustrator, so I have several problems that have come up in several design projects using similar large formats.
    The file size is MASSIVE. This poster uses several large, high-res images that I've embedded. I didn't want them to pixelate, so I made sure they were high quality. After embedding all these images, along with the vector graphics, the entire .ai file is 500MB. How can I reduce this file size? Can I do something with the images to make the .ai file smaller?
    I made my artboard 27" x 40" - the final size of the poster. Is this standard practice? Or when designing for a large print format, are you supposed to make a smaller, more manageable artboard size, and then scale up after to avoid these massive file sizes?
    I need to upload my support files for the project, including .ai and .eps - so it won't work if they're 500MB. This would be good info to understand for all projects I think.
    Any help with this would be appreciated. I can't seem to find any coherent information pertaining to this problem that seems to address my particular issues. Thank you very much!
    Asher

    Hi Asher,
    It's probably those high-res images you've embedded. Firstly, be sure your images are only as large as you need them Secondly, a solution would be to use linked images while you're working instead of embedding them into the file.
    Here is a link to a forum with a lot of great discussion about this issue, to get you started: http://www.cartotalk.com/lofiversion/index.php?t126.html
    And another: http://www.graphicdesignforum.com/forum/archive/index.php/t-1907.html
    Here is a great list of tips that someone in the above forum gave:
    -Properly scale files.  Do not take a 6x6' file then use the scaling tool to make it a 2x2'  Instead scale it in photoshop to 2x2 and reimport it.  Make a rule like over 20%, bring it back in to photoshop for rescaling.
    -Check resolutions. 600dpi is not going to be necessary for such and such printer.
    -Delete unused art.  Sloppy artists may leave old unused images under another image.  The old one is not being used but it still takes up space, therefore unecessarily inflating your file.
    -Choose to link instead of embedd.  This is your choice.  Either way you still have to send a large file but many times linking is less total MB then embedding.  Also embedding works well with duplicated images. That way multiple uses link to one original, whereas embedding would make copies.
    -When you are done, using compression software like ZIP or SIT (stuffit)
    http://www.maczipit.com/
    Compression can reduce file sizes alot, depending on the files.
    This business deals with alot of large files.  Generally people use FTP's to send large files, or plain old CD.  Another option is using segmented compression.  Something like winRAR/macRAR or dropsegment (peice of stuffit deluxe) compresses files, then breaks it up into smaller manageble pieces.   This way you can break up a 50mb file into say 10x 5mb pieces and send them 5mb at a time. 
    http://www.rarlab.com/download.htm
    *make sure your client knows how to uncompress those files.  You may want to link them to the site to download the software."
    Good luck!

  • I had an iphone 4s and it worked nice with facetime and imessage using my phone number but  it was stolen on December so I buy a new iphone 4s using the same cellphone number  but now imessage and facetime does not work with my number, it only works with

    I had an iphone 4s and it worked nice with facetime and imessage using my phone number but  it was stolen on December so I buy a new iphone 4s using the same cellphone number  but now imessage and facetime does not work with my number, it only works with my apple ID.   Please Help me I speak Spanish so  if my English is not ok  I´m sorry about it.
    Do you think that apple has to   reset in their database of the old serial number attached with my phone number and that’s why I can´t activate imessage and facetime with my number in the new iphone 4s?? 

    I understand all of this Meg; that is why I bought an Iphone; but never expected my phone not even give a at least a 24 or even 12 hours....I work 12 hour shifts and also would expect to have to charge each night but not twice a day or more.  I am not always somewhere I can charge my phone.
    Your points are true; however it doesn't help me......

  • Working with Large Numbers

    Hi there,
    I am currently doing a school assignment and not looking for answers but just a little guidance.
    I am working with large numbers and the modulo operator.
    I might have some numbers such as :
    int n = 221;
    int e = 5;
    int d = 77;
    int message = 84;
    int en = (int) (Math.pow(message, e) % n);
    int dn = (int) (Math.pow(en, d) % n);Would there be a better way to do this kind of calculation. The dn value should come out the same as message. But I always get something different and I think I might be losing something in the fact that an int can only hold smaller values.

    EJP wrote:
    It might make sense in some contexts to have a positive and negative infinity.
    Yes, perhaps that's a better name. Guess I was harking back to old COBOL days :-).(*)
    But the reason these things exist in FP is because the hardware can actually deliver them. That rationale doesn't apply to BIgInteger.Actually, it does. All I'm talking about is a value that compares higher or lower than any other. That could be done either by a special internal sign value (my slight preference) or by simply adding code to compareTo(), equals() and hashCode() methods that takes the two constants into account (as they already do with ZERO and ONE).
    Don't worry, I'm not holding my breath; but I have come across a few situations in which values like that would have been useful.
    Winston
    Edited by: YoungWinston on Mar 22, 2011 9:07 AM
    (*) Actually, '&plusmn;infinity' tends to suggest a valid arithmetic value, and I wasn't thinking of changing existing BigInteger/BigDecimal maths (except perhaps to throw an exception if either value is involved).

  • Speed up Illustrator CC when working with large vector files

    Raster (mainly) files up to 350 Mb. run fast in Illustrator CC, while vector files of 10 Mb. are a pain in the *blieb* (eg. zooming & panning). When reading the file it seems to freeze around 95 % for a few minutes. Memory usage goes up to 6 Gb. Processor usage 30 - 50 %.
    Are there ways to speed things up while working with large vector files in Illustrator CC?
    System:
    64 bit Windows 7 enterprise
    Memory: 16 Gb
    Processor: Intel Xeon 3,7 GHz (8 threads)
    Graphics: nVidia Geforce K4000

    Files with large amounts vector points will put a strain on the fastest of computers. But any type of speed increase we can get you can save you lots of time.
    Delete any unwanted stray points using  Select >> Object >> stray points
    Optimize performance | Windows
    Did you draw this yourself, is the file as clean as can be? Are there any repeated paths underneath your art which do not need to be there from live tracing or stock art sites?
    Check the control panel >> programs and features and sort by installed recently and uninstall anything suspicious.
    Sorry there will be no short or single answer to this, as per the previous poster using layers effectively, and working in outline mode when possible might the best you can do.

  • Photoshop CS6 keeps freezing when I work with large files

    I've had problems with Photoshop CS6 freezing on me and giving me RAM and Scratch Disk alerts/warnings ever since I upgraded to Windows 8.  This usually only happens when I work with large files, however once I work with a large file, I can't seem to work with any file at all that day.  Today however I have received my first error in which Photoshop says that it has stopped working.  I thought that if I post this event info about the error, it might be of some help to someone to try to help me.  The log info is as follows:
    General info
    Faulting application name: Photoshop.exe, version: 13.1.2.0, time stamp: 0x50e86403
    Faulting module name: KERNELBASE.dll, version: 6.2.9200.16451, time stamp: 0x50988950
    Exception code: 0xe06d7363
    Fault offset: 0x00014b32
    Faulting process id: 0x1834
    Faulting application start time: 0x01ce6664ee6acc59
    Faulting application path: C:\Program Files (x86)\Adobe\Adobe Photoshop CS6\Photoshop.exe
    Faulting module path: C:\Windows\SYSTEM32\KERNELBASE.dll
    Report Id: 2e5de768-d259-11e2-be86-742f68828cd0
    Faulting package full name:
    Faulting package-relative application ID:
    I really hope to hear from someone soon, my job requires me to work with Photoshop every day and I run into errors and bugs almost constantly and all of the help I've received so far from people in my office doesn't seem to make much difference at all.  I'll be checking in regularly, so if you need any further details or need me to elaborate on anything, I should be able to get back to you fairly quickly.
    Thank you.

    Here you go Conroy.  These are probably a mess after various attempts at getting help.

  • Can i make this work with a number specified in the constructor?

    Hi everyone
    I posted a couple of days ago and i have managed to advance a bit since then.
    Here i have a class called "flower" and it draws a flower with two leaves and 12 petals using the classes "petal and "leaf".
    However, ideally i would like to be able to make this work with a number of petals specified in the constructor, not just 12.
    is this possible or do i need to rewrite the entire class?
    import java.awt.*;
    import java.awt.geom.*;
    public class flower {
        private Shape circle;
        private Shape stem;
        public flower()   {
            circle = new Ellipse2D.Double(-25, -25, 50, 50);
            stem = new Line2D.Double(0.0, 0.0, 0.0, -200.0);
        public void paint(Graphics g) {
        Graphics2D g2 = (Graphics2D)g;
        g2.setPaint(Color.green);
        g2.setStroke(new BasicStroke(5)); // Line width 8 pixels
        g2.draw(stem);
        // Add the petals
        petal P = new petal(Color.red);
        // Scale, translate and rotate the petal into place.
        AffineTransform orig  = g2.getTransform();
        AffineTransform T = AffineTransform.getTranslateInstance(0.0, -25);
        for (int k = 0; k < 12; k++) {
          double angle = k*Math.PI/6.0;
          AffineTransform A =  AffineTransform.getScaleInstance(1.0, 1.0);
          A.preConcatenate(T);     // Scale, then rotate
          AffineTransform R = AffineTransform.getRotateInstance(angle);
          A.preConcatenate(R);     // Scale, translate and then rotate
          g2.transform(A);
          P.paint(g);
          g2.setTransform(orig);     // Restore original transform
          g2.setPaint(Color.yellow);
          g2.fill(circle);
          leaf L = new leaf();
          AffineTransform B = AffineTransform.getTranslateInstance(0.0, 0.0);
          B.translate(340, 425);
          B.scale(1.0, 1.0);
          B.rotate (Math.PI/6);
          g2.setTransform(B);     
          L.paint(g);
          g2.setTransform(orig);     // Restore original
          leaf L1 = new leaf();
          AffineTransform C = AffineTransform.getTranslateInstance(0.0, 0.0);
          C.translate(285, 430);
          C.scale(1.0, 1.0);
          C.rotate (Math.PI/-6);
          g2.setTransform(C);     
          L1.paint(g);
          g2.setTransform(orig);     // Restore original
    }I am tempted to make "flower into a class that extends "petal" would this make it easier for me to achieve my goal?
    Any help would be great
    Thanks Carl

    Your NewFlower class extends petal. You NewFlower constructor calls super( color, petnum ), so the petal constructor is invoked. In the petal constructor, you ignore petNum. You do nothing else in the NewFlower constructor, so petNum is ignored there as well.
    What makes you think that in this orgy of ignoring petNum, that any petNum variable will magically get set, or that your flower will now magically have the right number of petals?
    � {�                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   

  • How to Capture a Table with large number of Rows in Web UI Test?

    HI,
    Is there any possibility to capture a DOM Tabe with large number of Rows (say more than 100) in Web UI Test?
    Or is there any bug?

    Hi,
    You can try following code to capture the table values.
    To store the table values in CSV :
    *web.table( xpath_of_table ).exportToCSVFile("D:\exporttable.csv", true);*
    TO store the table values in a string:
    *String tblValues=web.table( xpath_of_table ).exportToCSVString();*
    info(tblValues);
    Thanks
    -POPS

  • How does the sync functionality work with large libraries on small devices?

    How does the sync functionality work with large libraries?
    Say I sync 100gb of photos with the new Photos app and turn on sync on a 16gb iphone. Will it fill the device up to 16gb? Can I tell it to limit to xgb so I leave room for music and apps? How does this work? Will it slow down my phone if its trying to sync 100gb across smaller devices?

    "Will the Apple TV now read directly from the Time Capsule?" ATV does not 'read' from the TC. It connects to the Mac and itunes library associated with the Mac. Of course the itunes program can 'point' to a library on the TC. You'll still need to have itunes open and the Mac powered on and not sleeping with the TC mounted to the Desktop to use ATV properly. So bottom line the use of TC just adds one more step to view files on the ATV.

  • My full search window shows the search results with large space between each result

    I have updated to Maverick on my iMac, my full search window now shows my results with massive gaps between each result, how do I change this?

    This is a bug.

Maybe you are looking for