Speed up data retrieval

I have around millions of data in oracle inventory which i need to show in the website. currently we r by running a procedure that get this information and display the information to user. but this take lots of time. is there any other method which would reduce the time take to fetch the data.

Nazoor
First, is a 10 minute runtime for your query really a problem? How many times/day do you run it? It may be better to wait 10 mins here than to spend time and effort to get it down to a few seconds - at the same time as making your critical load slower.
What is the range of values for the number you are selecting on? is it random, or is it something like a status value? Is it skewed?
How many rows are you loading per day? Is each partition a single day, or several days. How much data do you have overall (ie what range of dates).
A possible solution might be:
1) split your data in two
- latest data (ie today) - lets call that A
- historic data (everything else - your existing partitioned table - lets call B (with partitions B1, B2, ... )
2) Index B by your number column (lets call it X). DON'T index A.X
3) create a UNION ALL view over A and B which you can use for queries that need to include today's latest data
4) When you query by the number, the CBO should divide into two queries
- on A, which can do a (small) tablescan)
- on B which can use the index(es)
5) Loading into A is unaffected (no additional index)
6) At end of each day, create an index on A.X (creating it once is quicker than loading with an index many times) and then:
7) exchange partition A into B, and create a new A for tomorrow. (or simply copy A into B using INSERT INTO / SELECT FROM - that would make sure that any "late" data in A got sorted into the appropriate partition in B - but does mean you are double-handling that data).
Would something like that work for you?
Regards Nigel

Similar Messages

  • Data Retrieval Speed in Oracle Spatial vs. ESRI ArcSDE

    I would appreciate any opinions regarding data retrieval
    performance between Oracle Spatial and ESRI ArcSDE. Would an end-
    user (using ESRI software) experience significant differences in
    data retrieval speed depending on how the data were stored in
    Oracle (MDSYS.SDO_GEOMETRY verses ESRI Binary/Blob formats).
    Knowing that the ESRI binary formats are tailored to their
    software front-end apps (ArcGIS, ArcMap, ArcCatalog, and
    ArcInfo), wouldn't this be a "non-issue" until the spatial
    dataset gets "large", and even then, wouldn't performance be
    (almost) equal if the spatial indexes were created properly?
    Thanks for your inputs,
    Bruce

    John,
    You can't do that type of query in sql from sql*plus using
    SDEBINARY. HOwever, you can perform spatial queries in ArcMap
    if you are using SDEBINARY.
    You can use the query builder to perform point-in-polygon type
    queries.
    Hope that helps.
    For my two cents, I think SDO_GEOMETRY gives you a more robust
    database to work with, because you have the added power of
    Oracle Spatial functions. If you are using SDEBINARY you are
    limited to only what you can do thru ArcGIS.
    If you are concerned more about performance than accessibility,
    especially with a large number of users, then SDEBINARY might
    be the better choice.
    I love Oracle Spatial and am hoping that the performance issue
    will not be a serious one when we start putting ArcIMS developed
    apps into production.
    Dave

  • Data Retrieval speed from DIAdem Database using Data Finder Toolkit in LabVIEW

    I have developed a LabVIEW code using NI Data Finder toolkit for extracting the data from the DIAdem. I have formed queries for extraction the data. But the data retrieval takes more time for extracting the data.
    I have attached my LabVIEW code image along with this post. In that i have circled the VI "Results to Waveform", which is Data Finder toolkit VI which takes more time for extraction. From my Testing, query formation is taking only few millisecond to execute, but "Results to Waveform.vi" takes around 35 seconds. Also i want to execute similar kind of search for 5 times. Altogether it will taken around 1.5 mins to 2 mins for execution.
    Is there any other alternative way of doing this search? Kindly help me in sorting this out.
    Thanks,
    Alagar 
    Solved!
    Go to Solution.
    Attachments:
    Report Gen.JPG ‏91 KB

    Hi Alagar,
    You say that the "Results to Waveform.vi" takes "more" time, but what are you comparing this to?  More than what?  More than loading the contents of the same file with the TDMS VIs or with the Storage VIs?  What file format are you loading data from after querying-- TDMS files, ASCII files, binary files?
    What information do you want to retrieve?  Do you really need ALL the values of ALL the channels in each data file?  Are you interested only in the channel values, or all you interested in scalar properties stored along with the data?  If you want only some of the channels loaded or if you want only scalar properties loaded, there are indeed faster ways of doing this.
    Brad Turpin
    DIAdem Product Support Engineer
    National Instruments

  • Multi server data retrieval performance

    Hi experts,
    I have a question regarding the data retrieval performance (EVDRE) on a multi server installation environment on Microsoft SQL Server 2008.
    We have succesfully migrated Outlooksoft 4.2 SP03 to SAP BPC 7.0 SP07 for a customer. During this project we have also set up a complete new server environment consisting of:
    Development server: dedicated single server, Windows 2003 Standard SP2 32 bit, SQL Server 2008 SP1 with cumulative update package 6, SAP BPC 7.0 SP07, 2 quad core processors, 4 GB RAM
    QA server: dedicated multi servers - 1 database server (SQL/OLAP), Windows 2003 Standard SP2 64 bit, SQL Server 2008 SP1 with cumulative update package 6, 2 quad core processors, 32 GB RAM - 1 dedicated application/web server, Windows 2003 Standard SP2 32 bit, SQL Server 2008 SP1 with cumulative update package 6 (shared components / reporting services), SAP BPC 7.0 SP07, 2 quad core processors, 4 GB RAM
    Production server: dedicated multi servers - 1 database server (SQL/OLAP/Reporting services), Windows 2003 Standard SP2 64 bit, SQL Server 2008 SP1 with cumulative update package 6, 2 quad core processors, 32 GB RAM - 2 dedicated application/web server, Windows 2003 Standard SP2 32 bit, SQL Server 2008 SP1 with cumulative update package 6 (shared components), SAP BPC 7.0 SP07, 2 quad core processors, 4 GB RAM
    Furthermore, two terminal servers with the SAP BPC client.
    All servers have good performancve and we have great times on cube processing and SQL processing. However, to our great surprise we find that the single development server is much faster with a single user to retrieve data using EVDRE than the multi-server environment. About 2x as fast. A reporting book with more then 10 sheets and about 25 EVDRE's takes about 42 seconds on the development server and 93 seconds on the multi server.
    It seems that EVDRE is taking up a lot of time to communicate between the application server and the database server in a multi server environment while being much faster on a single server. This is not what we want :-). The network speed in the domain consist of all 1 GB lines so that should not be the issue.
    Do you have any experience with this? How can we upgrade the speed of the multi server, are there specific settings?
    Hope the get some useful answers. Thanks in advance.
    Damien
    Edited by: DWiegman on Feb 20, 2010 4:15 PM

    Hi,
       You have to activate also the EVDRE logs on the client and server level, just to understand from where is coming the problem (appserver-db comuncication or client-appserver communication). You have to check also if there is any proxy firewall between client and application server.
        In case you are using NLB, please verify if afinity is setup to true.
        The performance problems can be come from db level. Did you verify how many records do you have into WB table for the specific application? Are you keeping the DB in full mode? How big is the log of the databse?
        The are a lot of things that can have impact of this, but it looks to be a setup problem.
    Hope this can help you,
    Mihaela

  • BPC 10 - EPM data retrieval very slow!

    Hi BPCers,
    We are using an Excel EPM Input Schedules as a Resource Management tool - using VBA to provide the functionality we need.
    Performance is generally good, but quickly deteriorates when handling larger data sets - even 500-600 rows of transactional data is enough to slow data retrieval from our BPC Cube to EPM to an unusable speed. This is in relative terms pretty small so  there should be some option for optimisation.
    Does anybody have any experience with this? All suggestions welcome. We are operating on EPM Service Pack 7 Patch 1, but I'm not sure that EPM is necessarily the problem here.
    Thanks,
    Tom

    Thanks Gersh,
    Had a look through fiddler and have identified the job that is causing the delay - some rooting around in the ABAP debugger produced the answer as to why adding more data slows processing speed so dramatically.
    When we take data from the back end, we select a couple of parameters which limit the range of data that we are pulling through - a certain set of people, and a certain range of days. Once this is pulled through, allocations are made to any combination of person and day within this range, which generates and extra two properties - a project ID and a work status.
    This makes 4 properties, and when BPC pulls data it attempts to find every combination of every one of the properties that exist within this range - so the more allocations made the more this slows down as it dramatically increases the number of combinations.
    The result is that BPC runs through a couple of hundred thousand generated tables, most of which are nonsense.
    Not sure what to do from here. This is how BPC reads data so approaching a fix could be difficult.
    Tom

  • Query Error Information: Result set is too large; data retrieval ......

    Hi Experts,
    I got one problem with my query information. when Im executing my report and drill my info in my navigation panel, Instead of a table with values the message "Result set is too large; data retrieval restricted by configuration" appears. I already applied "Note 1127156 - Safety belt: Result set is too large". I imported Support Package 13 for SAP NetWeaver 7. 0 BI Java (BIIBC13_0.SCA / BIBASES13_0.SCA / BIWEBAPP13_0.SCA) and executed the program SAP_RSADMIN_MAINTAIN (in transaction SE38), with the object and the value like Note 1127156 says... but the problem still appears....
    what Should I be missing ??????  How can I fix this issue ????
    Thank you very much for helping me out..... (Any help would be rewarded)
    David Corté

    You may ask your basis guy to increase ESM buffer (rsdb/esm/buffersize_kb). Did you check the systems memory?
    Did you try to check the error dump using ST22 - Runtime error analysis?
    Edited by: ashok saha on Feb 27, 2008 10:27 PM

  • WAD : Result set is too large; data retrieval restricted by configuration

    Hi All,
    When trying to execute the web template by giving less restiction we are getting the below error :
    Result set is too large; data retrieval restricted by configuration
    Result set too large (758992 cells); data retrieval restricted by configuration (maximum = 500000 cells)
    But when we try to increase the number of restictions it is giving output. For example if we give fiscal period, company code ann Brand we are able to get output. But if we give fical period alone it it throwing the above error.
    Note : We are in SP18.
    Whether do we need to change some setting in configuration? If we yes where do we need to change or what else we need to do to remove this error
    Regards
    Karthik

    Hi Karthik,
    the standard setting for web templates is to display a maximum amount of 50.000 cells. The less you restrict your query the more data will be displayed in the report. If you want to display more than 50.000 cells the template will not be executed correctly.
    In general it is advisable to restrict the query as much as possible. The more data you display the worse your performance will be. If you have to display more data and you execute the query from query designer or if you use the standard template you can individually set the maximum amount of cells. This is described over  [here|Re: Bex Web 7.0 cells overflow].
    However I do not know if (and how) you can set the maximum amount of cells differently as a default setting for your template. This should be possible somehow I think, if you find a solution for this please let us know.
    Brgds,
    Marcel

  • Result set is too large; data retrieval restricted by configuration

    Hi,
    While executing query for a given period, 'Result set is too large; data retrieval restricted by configuration' message is getting displayed. I had searched in SDN and I had referred the following link:
    http://www.sdn.sap.com/irj/scn/index?rid=/library/uuid/d047e1a1-ad5d-2c10-5cb1-f4ff99fc63c4&overridelayout=true
    Steps followed:
    1) Transaction Code SE38
    2) In the program field, entered the report name SAP_RSADMIN_MAINTAIN and Executed.
    3) For OBJECT, entered the following parameters: BICS_DA_RESULT_SET_LIMIT_MAX
    4) For VALUE, entered the value for the size of the result set, and then executed the program:
    After the said steps, the below message is displayed:
    OLD SETTING:
    OBJECT =                                VALUE =
    UPDATE failed because there is no record
    OBJECT = BICS_DA_RESULT_SET_LIMIT_MAX
    Similar message is displayed for Object: BICS_DA_RESULT_SET_LIMIT_DEF.
    Please let me know as to how to proceed on this.
    Thanks in advance.

    Thanks for the reply!
    The objects are not available in the RSADMIN table.

  • How to improve speed of data acquisition? Help needed urgently.

    I want to convert analog signals to digital signals and simultaneously perform some search on the data acquired and this whole process has to be done continuously for few hours.
    So I tried writing two programs in Matlab, one acquires the analog data and converts it to digital and saves the data in small chunks on hard disk (like file1, file2, file3,...) continuously. The other program performs the search operation in those chunks of data continuously. I run both the programs at a time by opening two mat lab windows.
    But the problem Iam facing is that the data acquisition is slow. As a result I get an error message in the second program saying that
    "??? Error using ==> load
    Unable to read file file4.mat: No such file or directory."
    Iam unable to synchronize the two programs. I cannot use timers in search program because I cannot add any delays.
    Iam using a NI PCI-6036E ,16 Bit Resolution ,200 KS/s Sampling Rate A/D board.
    Should I switch to some other series such as M series having sampling rate of the order MS/s?
    Can anyone please tell me how to improve the speed of data acquisition?
    Thanks.

    Gayathri wrote:
    I want to convert analog signals to digital signals and simultaneously perform some search on the data acquired and this whole process has to be done continuously for few hours.
    So I tried writing two programs in Matlab, one acquires the analog data and converts it to digital and saves the data in small chunks on hard disk (like file1, file2, file3,...) continuously. The other program performs the search operation in those chunks of data continuously. I run both the programs at a time by opening two mat lab windows.
    But the problem Iam facing is that the data acquisition is slow. As a result I get an error message in the second program saying that
    "??? Error using ==> load
    Unable to read file file4.mat: No such file or directory."
    Iam unable to synchronize the two programs. I cannot use timers in search program because I cannot add any delays.
    Iam using a NI PCI-6036E ,16 Bit Resolution ,200 KS/s Sampling Rate A/D board.
    Should I switch to some other series such as M series having sampling rate of the order MS/s?
    Can anyone please tell me how to improve the speed of data acquisition?
    Thanks.
    Hi gayathri,
    well my email is [email protected]
    if ur from india mail me back.
    Regards
    labview boy

  • Real tough data retrieval - assistance needed

    Late 2011 Macbook Pro with 500GB hard drive
    Lion 10.7
    One morning out of absolutely nowhere I get this grey screen with a flashing question mark folder. I take it to the geniuses at the Apple store and they tell me my hard drive has failed (no explanation). My mac is under warranty so they gave me a new hard drive for free, bagged my old hard drive and told me "good luck" retrieving the data.
    I'm on a mission to retrieve the data without paying for services. I have never retrieved data before but I've been doing a lot of forum reading and I have been getting protips from an IT friend who has saved my PC data before.
    As of now, I have been unable to even access the hard drive and so I am reaching out to the community to help me conquer this project.
    the problem is not the OS (according to Apple store)
    when hooking up with the dongle, neither Finder nor Disk Utility detect the bad drive
    the drive will spin when forced by the dongle (so I've ruled out the freezer method)
    I was advised (by friends and forums alike) to download so powerful data retrieval software:
    Data Rescue [did not detect bad external drive]
    Disk Drill [did not detect bad external drive]
    TestDisk (http://www.cgsecurity.org/wiki/TestDisk) [I can get it up and running but I have no idea how to use this software]
    So that seems to be the big problem, when I hook up my failed drive as an external hard drive, there is no acknowledgement from my MBP that it is connected and as such data recovery programs cannot access it. When I still had it installed in my computer, there was no clicking sound and it doesn't seem like any of the components are jammed up as it still spins.
    Where do I go from here? Please keep in mind that I am new to resolving my own technical problems but I'm willing to learn. Tired of being one of those people who look at computer parts and get anxious.
    NOTE: I haven't tried Target Mode as I do not have a firewire cable or access to another mac (yet). If you think I should try this, please let me know.

    A hard drive that will not divulge BOTH its Make&Model and a reasonable size/capacity to the likes of Disk Utility and data rescue programs has died, and connot be repaired with any software.
    Target Disk mode will not improve anything. The drive is read as a Mac Volume by software. If it won't mount under Mac OS X, it won't mount under Target Disk Mode.

  • Should I use a data retrieval service or software to recover data ?

    Please pardon the length of this post.
    Here's the setup for our office:
    Computer 1:
    10.4.8 OS X
    1 GHZ PowerPC G4: silver grey tower, grey apple
    1MB L3 Cache
    256 MB DDR SDRAM
    Computer 2:
    10.4.8 OS X
    Dual 450 MHZ PowerPC G4: blue grey tower, blue apple
    256 MB SDRAM
    Computer 3:
    10.4.8 OS X
    1 GHZ PowerPC G4 IMac Flat Screen:
    256 MB DDR SDRAM
    I have 2 LaCie Big Disk d2 Extremes daisy chained and connected to the IMac. We use the first to store all of our data to keep our local disks free. The second d2 is the backup to the first. The other 2 computers connect to the d2's via an ethernet hub. The d2's are each partitioned into 4 compartments.
    A couple of days ago I started the system up when I got in in the morning, and the main d2 would not open. I ran disk utility, but it said that the drive was damaged beyond it's ability to repair. I ran DiskWarrior, and it gave me this message:
    "The directory of disk 'G4' cannot be rebuilt. The disk was not modified. The original directory is too severely damaged. It appears another disk utility has erased critical directory information. (3004, 2176)."
    I contacted Disk Warrior tech support and after a series of exchanges that had me send him dated extracted from the terminal function, he said this:
    "It appears that the concatenated RAID inside your LaCie
    drive has failed (your 500GB drive is actually 2 250GB
    hard drives). That is why we only saw 2 partitions
    on "disk3". A possible cause could be a failed bridge
    in the case.
    You may be looking at sending this drive to a data recovery service.
    However, it is possible that we may be able to recover data
    from the partitions that we CAN see. What we would be doing would cause no damage to your data
    unless the hard drives were having a mechanical failure (ie, the
    head crashed and was contacting the platters, similar to scratching
    a record). But from what I've seen, I don't feel that is the case.
    I believe the piece of hardware that 'bridges' the two drives to
    make them act as one has failed. that's why we can only see data
    about 1 of the 2 drives in the case.
    We would only be attempting to gather data off the drive. Since
    data recovery services sometimes charge for amount of data retrieved,
    it's up to you how you want to proceed."
    Most of the data from the past 5 years for our business stands at being lost. Only some of it had been properly backed up on the second drive, due to some back up software issues. I want to do whatever I can to retrieve all of, or at least some of the data. From what the Alsoft technician said, do you think that the data recovery software available to the consumer is going to be robust enough to retrieve at least the data from the one disk in the drive that is recognizable (there are 2 250gig disks in the d2X. Only one is responding at all). If so, do these Softwares further damage the disks? Or should I just send the drive to a data recovery service?
    I'd like to try to extract some of it myself via over the counter retrieval software, but I don't know whether to trust these programs?
    Any advice would be greatly appreciated.
    Thanks in advance.
    Peter McConnell
    1 GHZ PowerPC G4 IMac Flat Screen   Mac OS X (10.4.8)   Posted

    Peter
    My 2 cents:
    I have used FileSalvage
    http://www.subrosasoft.com/OSXSoftware/index.php?mainpage=product_info&productsid=1
    to recover files from damaged disks. It works as advertised, within limits. Some files may be too damaged to recover. More importantly yo get to scan the disk before actually recovering, and it will give you a list of what it thinks it can recover.
    My experience was that it recovered approx 85% of the data.
    YMMV but they do have a trial.
    Regards
    TD

  • Report Developed in Webi Rich Client Consuming more time in Data Retrieval

    Dear All,
    I am a BO Consultant, recently in my project I have developed one report in Webi Rich Client., at the time of development and subsequent days the report was working fine (taking Data Retrieval time less than 1 minute), but after some days its taking much time (increasing day by day and now its taking more than 11 minutes).
    Can anybody point out what could be the reason?????
    We are using,
    1. SAP BI 7.0
    2. SAP BO XI 3.1 Edge
    3. Webi Rich Client Version :12.3.0 and Build 601
    This report is made on a Multiprovider (Sales).
    What are the important points that should be considered so that we can improve the performance of Webi Reports????
    Waiting for a suitable solution.....................
    Regards,
    Arun Krishnan.G
    SAP BO Consultant
    Edited by: ArunKG on Oct 11, 2011 3:50 PM

    Hi,
    Please come back here with a copy/paste of the 2 MDX statements from the MDA.log to compare the good/bad runtimes.
    & the 2 equivalent DPCOMMANDS clauses (good and bad) from the WebI trace logs.
    Can u explain what u really mean in the bold text above..................Actually I didn't get you..........
    Pardon, I have only 3 months experience in BO.
    Regards,
    Arun
    Edited by: ArunKG on Oct 11, 2011 4:28 PM

  • Data retrieval for Sony handycam with 60G harddrive.

    Does BestBuy do data retrieval?  I have a handycam with work video on it that accidentaly got formated.  I know with a standard harddrive you can still retrieve it if there has not been recording since the formatting but I am not sure about camera harddrives.  If BestBuy does not do it, does anyone know where it can be done and approximate cost.  I live in the New Orleans area.  
    I may try the same question in computers if nobody knows.  Thanks in advance for any guidance at all. 

    Best Buy sends out for data retrieval. You can have it sent out through geek squad to get the estimate of cost.
    Crystal
    Superuser
    Forum Guidelines | Terms & Conditions | Community Guidelines | What is a Superuser?
    *Remember to mark your questions solved and click the star to give kudos to show your thanks!*
    While I used to be a Best Buy Employee, I no longer have any affiliation with Best Buy.
    My opinions do not in any way shape or form represent Best Buy's Official decisions.

  • Passing values to subreport in SSRS throwing an error - Data Retrieval failed for the report, please check the log for more details.

    Hi,
    I have the subreport calling from the main report. The subreport is based on MDX query agianst the SSAS cube. some dimensions in cube has values 0 and 1.
    when I try to pass '0' to the sub report as the parameter value, it gives an error "Data Retrieval failed for the report, please check the log for more details".
    Actually I am using table for storing these parameter values. In the main report I am calling this table (dataset) and passing these values to subreport.
    so I have given like [0],[1] and this works fine. when I give only either [0] or [1] then it is throwing an error.
    Could you please advise on this.
    Appreciate all and any help.
    Thanks,
    Divya

    Hi Divya,
    Based on the current description, I understand that there is no issue if you pass two values from main report to subreport, while the issue occurs when passing one value to subreport.
    To narrow down the issue, I want to confirm whether the subreport can run if there is only [0] or [1] in the subreport. If so, it indicates the query statements exist error in the subreport. If it’s not the case, this shows the issue occurs during passing
    values from main report to subreport. To make further analysis, please post the details of query statements of the subreport to the forum.
    Regards,
    Heidi Duan
    Heidi Duan
    TechNet Community Support

  • Slow data retrieval on 8i

    I am using oracle 8i database on Windows 2000 server on compaq
    proliant 350 server machine. The problem is that sometimes
    connecting from a client is very slow and after connection data
    retrieval is also slow. I am usring TCP/IP and net8. I came to
    know from internet that there is a patch which is actually a
    work around to solve this problem. Can anybody help me to locate
    this patch?
    Thanks in advance.
    G. Rajan.

    You can probably prove that this is the issue by creating a retrieve using the excel addin or smart view to replicate the form and see how long it takes to retrieve.
    You will also see in the essbase app logs how long it is taking to perform the retrieve.
    Cheers
    John
    http://john-goodwin.blgspot.com/

Maybe you are looking for

  • Using TEDS as storage for precision standards instead of transducers

    My company makes and calibrates precision resistance, capacitance and inductance standards. These are not transducers, yet they posess calibration data that would benefit from 'internal storage'. Some data could event be represented as a polynomial o

  • Kernel panic after installing additional ram

    Hi there. (I think this should go here, not in laptop issues because it seems to be more hardware-related) I have a laptop Acer Aspire 2920Z with 2x512 Mb stock ram and I recently bought an memory upgrade 2x2gb. Any of them works when inserted separa

  • No disk image of Ipod touch

    When I connect my Ipod touch to my Mac, Itunes recognizes it, and I'm able to sync it , etc. However, there is no disk image of it on my desktop? I opened Disk Utilities, and it's not shown there either? How can I create a disk image of it that will

  • Webi Server Performance Tuning

    We have Webi Server in a clustered environment. Our environment doesn't have much reports for Webi server to handle.. But, webi server goes down often and i have to restart the physical server to get webi running. Is there a way to tune webi performa

  • Receipt taken in wrong subinventory

    hi, Suppose i put a receipt in a wrong subinventory and if i want to correct it how to do it? Regards sudharshan