Large data sets and table partitioning

I have an Oracle 9i database and I need to store huge volume of data in a table,
so I use a composite partitioning.
Is there a maximum size for my table to keep performances (if it's possible, I would
like to store several Terra (1000 Mega) of data)?
I would like to know if somebody works on a similar project
and which tools and environment it uses (server, OS, memory
size..).
Thanks for your help.

Yes, users can update data.
I don't join data.
This is an OLTP system.
To estimate the size of 5 Tb, I just use the number of inserts per day and how many days I keep data in the table and I have an average size of one line (taken on an existing first version table).
I have another question on partitioning : someone says me it is more efficient to drop a whole partition (or subpartition if I use a composite partition) than to delete a line at the same time, for the performances.
He says me that data access (in my partition) should be very bad if I delete lines progressively (in this partition) instead of keeping the lines and deleting the whole partition when all its lines are no more used.
What do you think about it?
Thanks
Sandrine

Similar Messages

Large data sets and table partitioning : removing data

Hi,
I have to delete lines into a big table using partition.
Someone says me it is more efficient for the performances to drop a whole partition (or subpartition if I use a composite partition) than to delete a line at the same time.
He says me that data access (in my partition) should be very bad if I delete lines progressively (in this partition) instead of keeping the lines and deleting the whole partition when all its lines are no more used.
What do you think about it?
Thanks
Sandrine

Hi Sandrine,
I agree with what you're being told. I'll be much more efficient to "clone" the data you want to keep from your patition somewhere (clone table, ...) and then drop the whole partition.
Main thing is if you drop an object there's no BEFORE IMAGE stored in your UNDO structures (UNDO TS / RBS). SO you'll have way less disk I/Os.
Hope this helps ^^

Large data sets and key terms

Hello, I'm looking for some guidance on how BI can help me. I am a business analyst in a health solutions firm, but not proficient in SQL. However, I have to work with large data sets that just exceed the capabilities of Excel.
Basically, I'm having to use Excel to manaully search for key terms and apply a values to those results. For instance, I have a medical claims file, with Provider Names, Tax ID, Charges, etc. It's 300,000 records long and 15-25 columsn wide. I need to search for key terms in the provider name like Ambulance, Fire Dept, Rescue, EMT, EMS, etc. Anything that resembles an ambulance service. Also, need to include abbreviations of them such as AMB, FD, or variations like EMT, E M T, EMS, E M S, etc. Each time I do a search, I have filter and apply an "N/A" flag.
That's just one key term. I also have things like Dentists or DDS, Vision, Optomemtry and a dozen other Provider Types that need to be flagged as "N/A".
Is this something that can be handled using BI? I have access to a BI group, but I need to understand more about the capabilities of what can be done. As an analyst, I'm having to deal with poor data inegrity. So, just cleaning up the file can be extremely taxing and cumbersome.
Some insight would be very helpful. Thanks.

I am not sure if you are looking for an explanation about different BI products? If so, may be this forum is not the place to get a straight answer.
But, Information Discovery product suite might be useful in your case. Regarding the "large date set" you mentioned, searching and analyzing 300,000 records may not be considered a large data set at least in Endeca standards :).
All your other requests, could also be very easily implemented using Endeca's product suite. Please reach out to Oracle's Endeca product team and they might guide you on how this product suite would help you.

Just in case any one needs a Observable Collection that deals with large data sets, and supports FULL EDITING...

the VirtualizingObservableCollection does the following:
Implements the same interfaces and methods as ObservableCollection<T> so you can use it anywhere you’d use an ObservableCollection<T> – no need to change any of your existing controls.
Supports true multi-user read/write without resets (maximizing performance for large-scale concurrency scenarios).
Manages memory on its own so it never runs out of memory, no matter how large the data set is (especially important for mobile devices).
Natively works asynchronously – great for slow network connections and occasionally-connected models.
Works great out of the box, but is flexible and extendable enough to customize for your needs.
Has a data access performance curve so good it’s just as fast as the regular ObservableCollection – the cost of using it is negligible.
Works in any .NET project because it’s implemented in a Portable Code Library (PCL).
The latest package can be found on nugget. Install-Package VirtualizingObservableCollection. The source is on github.

Good job, thank you for sharing
Best Regards,
Please remember to mark the replies as answers if they help

Need to load large data set from Oracle table onto desktop using ODBC

I don't have TOAD nor any other tool for querying the database. I'm wondering how I can load a large data set from an Oracle table onto my desktop using Excel or Access or some other tool using ODBC or not using ODBC if that's possible. I need results to be in a .csv file or something similar. Speed is what is important here. I'm looking to load more than 1 million but less than 10 million records at once. Thanks.

hillelhalevi wrote:
I don't have TOAD nor any other tool for querying the database. I'm wondering how I can load a large data set from an Oracle table onto my desktop using Excel or Access or some other tool using ODBC or not using ODBC if that's possible. I need results to be in a .csv file or something similar. Speed is what is important here. I'm looking to load more than 1 million but less than 10 million records at once. Thanks.
Use Oracle's free Sql Developer
http://www.oracle.com/technetwork/developer-tools/sql-developer/downloads/index.html
You can just issue a query like this
SELECT /*csv*/ * FROM SCOTT.EMP
Then just save the results to a file
See this article by Jeff Smith for other options
http://www.thatjeffsmith.com/archive/2012/05/formatting-query-results-to-csv-in-oracle-sql-developer/

How to handle large data sets?

Hello All,
I am working on a editable form document. It is using a flowing subform with a table. The table may contain up to 50k rows and the generated pdf may even take up to 2-4 Gigs of memory, in some cases adobe reader fails and "gives up" opening these large data sets.
Any suggestions?

On 25.04.2012 01:10, Alan McMorran wrote:
> How large are you talking about? I've found QVTo scales pretty well as
> the dataset size increases but we're using at most maybe 3-4 million
> objects as the input and maybe 1-2 million on the output. They can be
> pretty complex models though so we're seeing 8GB heap spaces in some
> cases to accomodate the full transformation process.
Ok, that is good to know. We will be working in roughly the same order
of magnitude. The final application will run on a well equipped server,
unfortunately my development machine is not as powerful so I can't
really test that.
> The big challenges we've had to overcome is that our model is
> essentially flat with no containment in it so there are parts of the
We have a very hierarchical model. I still wonder to what extent EMF and
QVTo at least try to let go of objects which are not needed anymore and
allow them to be garbage collected?
> Is the GC overhead limit not tied to the heap space limits of the JVM?
Apparently not, quoting
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html:
"The concurrent collector will throw an OutOfMemoryError if too much
time is being spent in garbage collection: if more than 98% of the total
time is spent in garbage collection and less than 2% of the heap is
recovered, an OutOfMemoryError will be thrown. This feature is designed
to prevent applications from running for an extended period of time
while making little or no progress because the heap is too small. If
necessary, this feature can be disabled by adding the option
-XX:-UseGCOverheadLimit to the command line."
I will experiment a little bit with different GC's, namely the parallel GC.
Regards
Marius

Displaying large result sets in Table View u0096 request for patterns

When providing a table of results from a large data set from SAP, care needs to be taken in order to not tax the R/3 database or the R/3 and WAS application servers. Additionally, in terms of performance, results need to be displayed quickly in order to provide sub-second response times to users.
This post is my thoughts on how to do this based on my findings that the Table UI element cannot send an event to retrieve more data when paging down through data in the table (hopefully a future feature of the Table UI Element).
Approach:
For data retrieval, we need to have an RFC with search parameters that retrieves a maximum number of records (say 200) and a flag whether 200 results were returned.
In terms of display, we use a table UI Element, and bind the result set to the table.
For sorting, when they sort by a column, if we have less than the maximum search results, we sort the result set we already have (no need to go to SAP), but otherwise the RFC also needs to have sort information as parameters so that sorting can take place during the database retrieval. We sort it during the SQL select so that we stop as soon as we hit 200 records.
For filtering, again, if less than 200 results, we just filter the results internally, otherwise, we need to go to SAP, and the RFC needs to have this parameterized also.
If the requirement is that the user must look at more than 200 results, we need to have a button on the screen to fetch the next 200 results. This implies that the RFC will also need to have a start point to return results from. Similarly, a previous 200 results button would need to be enabled once they move beyond the initial result set.
Limitations of this are:
1.     We need to use custom RFC function as BAPIs dont generally provide this type of sorting and limiting of data.
2.     Functions need to directly access tables in order to do sorting at the database level (to reduce memory consumption).
3.     Its not a great interface to add buttons to Get next/previous set of 200.
4.     Obviously, based on where you are getting the data from, it may be better to load the data completely into an internal table in SAP, and do sorting and filtering on this, rather than use the database to do it.
Does anyone have a proven pattern for doing this or any improvements to the above design? Im sure SAP-CRM must have to do this, or did they just go with a BSP view when searching for customers?
Note I noticed there is a pattern for search results in some documentation, but it does not exist in the sneak preview edition of developer studio. Has anyone had in exposure to this?
Update - I'm currently investigating whether we can create a new value node and use a supply function to fill the data. It may be that when we bind this to the table UI element, that it will call this incrementally as it requires more data and hence could be a better solution.

Hi Matt,
i'm afraid, the supplyFunction will not help you to get out of this, because it's only called, if the node is invalid or gets invalidated again. The number of elements a node contains defines the number of elements the table uses for the determination of the overall number of table rows. Something quite similar to what you want does already exist in the WD runtime for internal usage. As you've surely noticed, only "visibleRowCount" elements are initially transferred to the client. If you scroll down one or multiple lines, the following rows are internally transferred on demand. But this doesn't help you really, since:
1. You don't get this event at all and
2. Even if you would get the event, since the number of node elements determines the table's overall rows number, the event would never request to load elements with an index greater than number of node elements - 1.
You can mimic the desired behaviour by hiding the table footer and creating your own buttons for pagination and scrolling.
Assume you have 10 displayed rows and 200 overall rows, What you need to be able to implement the desired behaviour is:
1. A context attribute "maxNumberOfExpectedRows" type int, which you would set to 200.
2. A context attribute "visibleRowCount" type int, which you would set to 10 and bind to table's visibleRowCount property.
3. A context attribute "firstVisibleRow" type int, which you would set to 0 and bind to table's firstVisibleRow property.
4. The actions PageUp, PageDown, RowUp, RowDown, FirstRow and LastRow, which are used for scrolling and the corresponding buttons.
The action handlers do the following:
PageUp: firstVisibleRow -= visibleRowCount (must be >=0 of course)
PageDown: firstVisibleRow += visibleRowCount (first + visible must be < maxNumberOfExpectedRows)
RowDown/Up: firstVisibleRow++/-- with the same restrictions as in page "mode"
FirstRow/LastRow is easy, isn't it?
Since you know, which sections of elements has already been "loaded" into the dataSource-node, you can fill the necessary sections on demand, when the corresponding action is triggered.
For example, if you initially display elements 0..9 and goto last row, you load from maxNumberOfExpected (200) - visibleRows (10) entries, so you would request entries 190 to 199 from the backend.
A drawback is, that the BAPIs/RFCs still have to be capable to process such "section selecting".
Best regards,
Stefan
PS: And this is meant as a workaround and does not really replace your pattern request.

XML Solutions for Large Data Sets

Hi,
I'm working with a large data set (9 million records comprising 36 gigabytes) and am exploring the use of XML with it.
I've experimented with a JDBC app (taken straight from Steve Muench's excellent <i>Oracle_XML_Applications</i>) for writing to CLOBS, but achieve throughputs of much less than 40k/s (the minimum speed required to process the data in < 10 days).
What kind of throughputs are possible loading XML records from CLOBs into multiple tables (using server-side Java apps)?
Could anyone comment whether XML is a feasible possibility for this size data set?
Regards,
Mike

Just would like to identify myself (I'm the submitter):
Michael Driscoll <[email protected]>.
null

Download using open data set and close data set

can any body please send some sample pgm using open data set and close data set .the data should get downloaded in application server
very simple pgm needed

Hi Arun,
See the Sample code for BDC using OPEN DATASET.
report ZSDBDCP_PRICING no standard page heading
line-size 255.
include zbdcrecx1.
*--Internal Table To hold condition records data from flat file.
Data: begin of it_pricing occurs 0,
key(4),
f1(4),
f2(4),
f3(2),
f4(18),
f5(16),
end of it_pricing.
*--Internal Table To hold condition records header .
data : begin of it_header occurs 0,
key(4),
f1(4),
f2(4),
f3(2),
end of it_header.
*--Internal Table To hold condition records details .
data : begin of it_details occurs 0,
key(4),
f4(18),
f5(16),
end of it_details.
data : v_sno(2),
v_rows type i,
v_fname(40).
start-of-selection.
refresh : it_pricing,it_header,it_details.
clear : it_pricing,it_header,it_details.
CALL FUNCTION 'UPLOAD'
EXPORTING
FILENAME = 'C:\WINDOWS\Desktop\pricing.txt'
FILETYPE = 'DAT'
TABLES
DATA_TAB = it_pricing
EXCEPTIONS
CONVERSION_ERROR = 1
INVALID_TABLE_WIDTH = 2
INVALID_TYPE = 3
NO_BATCH = 4
UNKNOWN_ERROR = 5
GUI_REFUSE_FILETRANSFER = 6
OTHERS = 7.
WRITE : / 'Condition Records ', P_FNAME, ' on ', SY-DATUM.
OPEN DATASET P_FNAME FOR INPUT IN TEXT MODE.
if sy-subrc ne 0.
write : / 'File could not be uploaded.. Check file name.'.
stop.
endif.
CLEAR : it_pricing[], it_pricing.
DO.
READ DATASET P_FNAME INTO V_STR.
IF SY-SUBRC NE 0.
EXIT.
ENDIF.
write v_str.
translate v_str using '#/'.
SPLIT V_STR AT ',' INTO it_pricing-key
it_pricing-F1 it_pricing-F2 it_pricing-F3
it_pricing-F4 it_pricing-F5 .
APPEND it_pricing.
CLEAR it_pricing.
ENDDO.
IF it_pricing[] IS INITIAL.
WRITE : / 'No data found to upload'.
STOP.
ENDIF.
loop at it_pricing.
At new key.
read table it_pricing index sy-tabix.
move-corresponding it_pricing to it_header.
append it_header.
clear it_header.
endat.
move-corresponding it_pricing to it_details.
append it_details.
clear it_details.
endloop.
perform open_group.
v_rows = sy-srows - 8.
loop at it_header.
perform bdc_dynpro using 'SAPMV13A' '0100'.
perform bdc_field using 'BDC_CURSOR'
'RV13A-KSCHL'.
perform bdc_field using 'BDC_OKCODE'
'/00'.
perform bdc_field using 'RV13A-KSCHL'
it_header-f1.
perform bdc_dynpro using 'SAPMV13A' '1004'.
perform bdc_field using 'BDC_CURSOR'
'KONP-KBETR(01)'.
perform bdc_field using 'BDC_OKCODE'
'/00'.
perform bdc_field using 'KOMG-VKORG'
it_header-f2.
perform bdc_field using 'KOMG-VTWEG'
it_header-f3.
**Table Control
v_sno = 0.
loop at it_details where key eq it_header-key.
v_sno = v_sno + 1.
clear v_fname.
CONCATENATE 'KOMG-MATNR(' V_SNO ')' INTO V_FNAME.
perform bdc_field using v_fname
it_details-f4.
clear v_fname.
CONCATENATE 'KONP-KBETR(' V_SNO ')' INTO V_FNAME.
perform bdc_field using v_fname
it_details-f5.
if v_sno eq v_rows.
v_sno = 0.
perform bdc_dynpro using 'SAPMV13A' '1004'.
perform bdc_field using 'BDC_OKCODE'
'=P+'.
perform bdc_dynpro using 'SAPMV13A' '1004'.
perform bdc_field using 'BDC_OKCODE'
'/00'.
endif.
endloop.
*--Save
perform bdc_dynpro using 'SAPMV13A' '1004'.
perform bdc_field using 'BDC_OKCODE'
'=SICH'.
perform bdc_transaction using 'VK11'.
endloop.
perform close_group.
Hope this resolves your query.
Reward all the helpful answers.
Regards

Spry Data Sets and IE8

Hi, I Currently have a glossary using the Spry Tabbed Panels with Spry Data Sets embedded in each tab. I had it working great in pre IE8.
Once we upgraded to IE8 it just wont load or display the data sets in the tabs. Below is the page code. (I used the JS straight from the Spry Framework no alterations, except in the CSS files). Any help would be great =).
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />

<title>Glossary</title>

<script src="/composition/SpryAssets/xpath.js" type="text/javascript"></script>
<script src="/composition/composition002/profile_processes/SpryAssets/SpryTabbedPanels.js" type="text/javascript"></script>
<script src="/composition/SpryAssets/SpryData.js" type="text/javascript"></script>
<link href="/composition/composition002/profile_processes/SpryAssets/SpryTabbedPanels.css" rel="stylesheet" type="text/css" />
<link href="/composition/SpryAssets/SpryStackedContainers.css" rel="stylesheet" type="text/css" />
<script type="text/javascript">
var ds1 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defnum", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds2 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defa", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds3 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defb", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds4 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defc", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds5 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defd", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds6 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defe", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds7 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/deff", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds8 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defg", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds9 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defh", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds10 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defi", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds11 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defj", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds12 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defk", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds13 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defl", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds14 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defm", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds15 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defn", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds16 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defo", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds17 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defp", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds18 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defq", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds19 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defr", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds20 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defs", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds21 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/deft", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds22 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defu", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds23 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defv", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds24 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defw", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds25 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defx", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds26 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defy", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
var ds27 = new Spry.Data.XMLDataSet("/composition/composition002/profile_processes/profile_glossary/glos sary.xml", "glossary/def/defz", {sortOnLoad: "word", sortOrderOnLoad: "ascending"});
</script>

<link href="../../../../composition/css/profile_sp1.css" rel="stylesheet" type="text/css" media="screen" />
<link href="../../../../composition/css/print1.css" rel="stylesheet" type="text/css" media="print" />
</head>
<body>
<div id="top1">
<table width="100%" height="75px" border="0" cellpadding="3" cellspacing="0">
    <tr bgcolor="#000000">
      <th width="6%" height="36" align="left" valign="middle" scope="col"><img src="/composition/composition002/Profile_Processes/graphics/globe2.gif" alt="globe" width="70" height="68" border="0" /></th>
      <th width="94%" align="left" valign="top"><span class="whitebanner">Composition Learning Center</span><br />
                     <span class="whitebanner2"><strong>Learn. Grow. Succeed.</strong></span></th>
    </tr>
</table>
</div>
<div id="Layer4">
<h1 align="center">Glossary</h1>
<div id="TabbedPanels1" class="TabbedPanels">
<ul class="TabbedPanelsTabGroup">
    <li class="TabbedPanelsTab" style="display:none" tabindex="0"></li>
    <li class="TabbedPanelsTab" tabindex="0">A</li>
    <li class="TabbedPanelsTab" tabindex="0">B</li>
    <li class="TabbedPanelsTab" tabindex="0">C</li>
    <li class="TabbedPanelsTab" tabindex="0">D</li>
    <li class="TabbedPanelsTab" tabindex="0">E</li>
    <li class="TabbedPanelsTab" tabindex="0">F</li>
    <li class="TabbedPanelsTab" tabindex="0">G</li>
    <li class="TabbedPanelsTab" tabindex="0">H</li>
    <li class="TabbedPanelsTab" tabindex="0">I</li>
    <li class="TabbedPanelsTab" tabindex="0">J</li>
    <li class="TabbedPanelsTab" tabindex="0">K</li>
    <li class="TabbedPanelsTab" tabindex="0">L</li>
    <li class="TabbedPanelsTab" tabindex="0">M</li>
    <li class="TabbedPanelsTab" tabindex="0">N</li>
    <li class="TabbedPanelsTab" tabindex="0">O</li>
    <li class="TabbedPanelsTab" tabindex="0">P</li>
    <li class="TabbedPanelsTab" tabindex="0">Q</li>
    <li class="TabbedPanelsTab" tabindex="0">R</li>
    <li class="TabbedPanelsTab" tabindex="0">S</li>
    <li class="TabbedPanelsTab" tabindex="0">T</li>
    <li class="TabbedPanelsTab" tabindex="0">U</li>
    <li class="TabbedPanelsTab" tabindex="0">V</li>
    <li class="TabbedPanelsTab" tabindex="0">W</li>
    <li class="TabbedPanelsTab" tabindex="0">X</li>
    <li class="TabbedPanelsTab" tabindex="0">Y</li>
    <li class="TabbedPanelsTab" tabindex="0">Z</li>
    <li class="TabbedPanelsTab" tabindex="0">0 to 9</li>
</ul>
<div class="TabbedPanelsContentGroup">
    <div class="TabbedPanelsContent">
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds2" class="StackedContainers">
        <div spry:repeat="ds2" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds3" class="StackedContainers">
        <div spry:repeat="ds3" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds4" class="StackedContainers">
        <div spry:repeat="ds4" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds5" class="StackedContainers">
        <div spry:repeat="ds5" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds6" class="StackedContainers">
        <div spry:repeat="ds6" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds7" class="StackedContainers">
        <div spry:repeat="ds7" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds8" class="StackedContainers">
        <div spry:repeat="ds8" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds9" class="StackedContainers">
        <div spry:repeat="ds9" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds10" class="StackedContainers">
        <div spry:repeat="ds10" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds11" class="StackedContainers">
        <div spry:repeat="ds11" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds12" class="StackedContainers">
        <div spry:repeat="ds12" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds13" class="StackedContainers">
        <div spry:repeat="ds13" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds14" class="StackedContainers">
        <div spry:repeat="ds14" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds15" class="StackedContainers">
        <div spry:repeat="ds15" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds16" class="StackedContainers">
        <div spry:repeat="ds16" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds17" class="StackedContainers">
        <div spry:repeat="ds17" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds18" class="StackedContainers">
        <div spry:repeat="ds18" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds19" class="StackedContainers">
        <div spry:repeat="ds19" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds20" class="StackedContainers">
        <div spry:repeat="ds20" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds21" class="StackedContainers">
        <div spry:repeat="ds21" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds22" class="StackedContainers">
        <div spry:repeat="ds22" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds23" class="StackedContainers">
        <div spry:repeat="ds23" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds24" class="StackedContainers">
        <div spry:repeat="ds24" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds25" class="StackedContainers">
        <div spry:repeat="ds25" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds26" class="StackedContainers">
        <div spry:repeat="ds26" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
      <div spry:region="ds27" class="StackedContainers">
        <div spry:repeat="ds27" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
    <div class="TabbedPanelsContent">
    <div spry:region="ds1" class="StackedContainers">
        <div spry:repeat="ds1" class="RowContainer"> <div id="w1d" class="RowColumn">{word}</div>
          <span class="RowColumn">{definition}</span>
        </div>
      </div>
    </div>
</div>
</div>
<script type="text/javascript">
var TabbedPanels1 = new Spry.Widget.TabbedPanels("TabbedPanels1", {defaultTab:0});
</script>
</div>
</body>
</html>

Thanks Gramps.
I totally forgot about the Xpath filtering.
Trusty Hammer
altruistic gramps <[email protected]>
01/11/2011 05:06 PM
Please respond to
[email protected]
To
patrick dennis <[email protected]>
cc
Subject
Spry Data Sets and IE8
A quick fix: add the following to your document
A better fix: reduce the number of datasets to just one and use Xpath
filtering triggered by an onclick event, as in clicking the relevant tab.
Gramps
For a more specific answer, please supply a link to your site.

Spry XML data set and dynamic post variables

Hi,
I am trying to create an XML data set that has dynamic post
variables.
Everytime something is pressed on the page a variable changes
and I then want to reload the XML data set using the new variable.
I know I can just pull in an XML with all possible variables
and filter client side but this would make it way too large.
Does anyone know what I may need to do.
I tried this:
var myVar = 0;
var dss = new Spry.Data.XMLDataSet (
'../../cgi-bin/server_details.pl' , 'top' , { method: 'POST' ,
postData: sid=ajaja21&ip=127.0.0.1&cid=' . myVar ,
subPaths: [ "auth" , "plugins" , "plugins/plugin" ] , keepSorted:
"true", sortOnLoad: "plugins/plugin/order", sortOrderOnLoad:
"descending", useCache: false, loadInterval: 10000 } );
onclick="myVar=1";
But the script doesn't understand the post variables sent (it
does when I remove the . myVar part and put in a static value). I
think it isn't sending that dynamic variable with the post
variables.
Any ideas anyone?
Thanks

Well I had it working when I stripped back everything and
just had the dss data set and a single onclick function, but now
that I put it back together it hash foobared again.
Here are the relevant bits of code that I've changed.
The function to change server id:
//function to run when changing the server id
function changeServer ( sid ) {
//set the url to use the current server id
dss.setURL = ( '../../cgi-bin/server_details.pl' , { method:
'POST' , postData:
'sid=7gv1m3vjvagfl7h7qeefb8iodj8evhmb&ip=127.0.0.1&cid='+sid
//force a reload of the server data
dss.loadData();
The inital load of the data set
var dss = new Spry.Data.XMLDataSet (
'../../cgi-bin/server_details.pl' , 'yams' , { method: 'POST' ,
postData:
'sid=7gv1m3vjvagfl7h7qeefb8iodj8evhmb&ip=127.0.0.1&cid=0' ,
subPaths: [ "auth" , "plugins" , "plugins/plugin" ] , keepSorted:
"true", sortOnLoad: "plugins/plugin/order", sortOrderOnLoad:
"descending", useCache: false, loadInterval: 10000 } );
And the part that changes the server id
<td align="left" style="cursor:default; width:174px;"
onclick="changeServer({dsv::servers/server/@id})">{dsv::servers/server/name}</td>
I checked that the function is receiving the correct server
id and I even tried hard coding the cid variable to 2 in the change
function but it still wasn't changing on the server side.
Any ideas?
Thanks

Working with Large data sets Waveforms

When collection data at a high rate ( 30K ) and for a long period (120 seconds) I'm unable rearrange the data due to memory errors, is there a more efficient method?
Attachments:
Convert2Dto1D.vi ‏36 KB

Some suggestions:
Preallocate your final data before you start your calculations. The build array you have in your loop will tend to fragment memory, giving you issues.
Use the In Place Element to get data to/from your waveforms. You can use it to get single waveforms from your 2D array and Y data from a waveform.
Do not use the Transpose and autoindex. It is adding a copy of data.
Use the Array palette functions (e.g. Reshape Array) to change sizes of current data in place (if possible).
You may want to read Managing Large Data Sets in LabVIEW.
Your initial post is missing some information. How many channels are you acquiring and what is the bit depth of each channel? 30kHz is a relatively slow acquisition rate for a single channel (NI sells instruments which acquire at 2GHz). 120s of data from said single channel is modestly large, but not huge. If you have 100 channels, things change. If you are acquiring them at 32-bit resolution, things change (although not as much). Please post these parameters and we can help more.
This account is no longer active. Contact ShadesOfGray for current posts and information.

64-bit LabVIEW - still major problems with large data sets

Hi Folks -
I have LabVIEW 2009 64-bit version running on a Win7 64-bit OS with Intel Xeon dual quad core processor, 16 gbyte RAM. With the release of this 64-bit version of LabVIEW, I expected to easily be able to handle x-ray computed tomography data sets in the 2 and 3-gbyte range in RAM since we now have access to all of the available RAM. But I am having major problems - sluggish (and stoppage) operation of the program, inability to perform certain operations, etc.
Here is how I store the 3-D data that consists of a series of images. I store each of my 2d images in a cluster, and then have the entire image series as an array of these clusters. I then store this entire array of clusters in a queue which I regularly access using 'Preview Queue' and then operate on the image set, subsets of the images, or single images.
Then enqueue:
I remember talking to LabVIEW R&D years ago that this was a good way to do things because it allowed non-contiguous access to memory (versus contigous access that would be required if I stored my image series as 3-D array without the clusters) (R&D - this is what I remember, please correct if wrong).
Because I am experiencing tremendous slowness in the program after these large data sets are loaded, and I think disk access as well to obtain memory beyond 16 gbytes, I am wondering if I need to use a different storage strategy that will allow seamless program operation while still using RAM storage (do not want to have to recall images from disk).
I have other CT imaging programs that are running very well with these large data sets.
This is a critical issue for me as I move forward with LabVIEW in this application. I would like to work with LabVIEW R&D to solve this issue. I am wondering if I should be thinking about establishing say, 10 queues, instead of 1, to address this. It would mean a major program rewrite.
Sincerely,
Don

First, I want to add that this strategy works reasonably well for data sets in the 600 - 700 mbyte range with the 64-bit LabVIEW.
With LabVIEW 32-bit, I00 - 200 mbyte sets were about the limit before I experienced problems.
So I definitely noticed an improvement.
I use the queuing strategy to move this large amount of data in RAM. We could have used other means such a LV2 globals. But the idea of clustering the 2-d array (image) and then having a series of those clustered arrays in an array (to see the final structure I showed in my diagram) versus using a 3-D array I believe even allowed me to get this far using RAM instead of recalling the images from disk.
I am sure data copies are being made - yes, the memory is ballooning to 15 gbyte. I probably need to have someone examine this code while I am explaining things to them live. This is a very large application, and a significant amount of time would be required to simplify it, and that might not allow us to duplicate the problem. In some of my applications, I use the in-place structure for indexing
data out of arrays to minimize data copies. I expect I might have to
consider this strategy now here as well. Just a thought.
What I can do is send someone (in US) via large file transfer a 1.3 - 2.7 gbyte set of image data - and see how they would best advise on storing and extracting the images using RAM, how best to optimize the RAM usage, and not make data copies. The operations that I apply on the images are irrelevant. It is the storage, movement, and extractions that are causing the problems. I can also show a screen shot(s) of how I extract the images (but I have major problems even before I get to that point),
Can someone else comment on how data value references may help here, or how they have helped in one of their applications? Would the use of this eliminate copies? I currently have to wait for 64-bit version of the Advanced Signal Processing Toolkit for LabVIEW 2010 before I can move to LabVIEW 2010.
Don

What is open data set and close data set

what is open data set and close data set,
how to use the files in sap directories ?

hi,
Open Dataset is used to read or write on to application server ... other than that i am not sure that there exists any way to do the same ... here is a short description for that
FILE HANDLING IN SAP
Introduction
Files on application server are sequential files.
Files on presentation server / workstation are local files.
A sequential file is also called a dataset.
Handling of Sequential file
Three steps are involved in sequential file handling
OPEN
PROCESS
CLOSE
Here processing of file can be READING a file or WRITING on to a file.
OPEN FILE
Before data can be processed, a file needs to be opened.
After processing file is closed.
Syntax:
OPEN DATASET <file name> FOR {OUTPUT/INPUT/APPENDING}
IN {TEXT/BINARY} MODE
This statement returns SY_SUBRC as 0 for successful opening of file or 8, if unsuccessful.
OUTPUT: Opens the file for writing. If the dataset already exists, this will place the cursor at the start of the dataset, the old contents get deleted at the end of the program or when the CLOSE DATASET is encountered.
INPUT: Opens a file for READ and places the cursor at the beginning of the file.
FOR APPENDING: Opens the file for writing and places the cursor at the end of file. If the file does not exist, it is generated.
BINARY MODE: The READ or TRANSFER will be character wise. Each time n characters are READ or transferred. The next READ or TRANSFER will start from the next character position and not on the next line.
IN TEXT MODE: The READ or TRANSFER will start at the beginning of a new line each time. If for READ, the destination is shorter than the source, it gets truncated. If destination is longer, then it is padded with spaces.
Defaults: If nothing is mentioned, then defaults are FOR INPUT and in BINARY MODE.
PROCESS FILE:
Processing a file involves READing the file or Writing on to file TRANSFER.
TRANSFER Statement
Syntax:
TRANSFER <field> TO <file name>.
<Field> can also be a field string / work area / DDIC structure.
Each transfer statement writes a statement to the dataset. In binary mode, it writes the length of the field to the dataset. In text mode, it writes one line to the dataset.
If the file is not already open, TRANSFER tries to OPEN file FOR OUTPUT (IN BINARY MODE) or using the last OPEN DATASET statement for this file.
IF FILE HANDLING, TRANSFER IS THE ONLY STATEMENT WHICH DOES NOT RETURN SY-SUBRC
READ Statement
Syntax:
READ DATASET <file name> INTO <field>.
<Field> can also be a field string / work area / DDIC structure.
Each READ will get one record from the dataset. In binary mode it reads the length of the field and in text mode it reads each line.
CLOSE FILE:
The program will close all sequential files, which are open at the end of the program. However, it is a good programming practice to explicitly close all the datasets that were opened.
Syntax:
CLOSE DATASET <file name>.
SY-SUBRC will be set to 0 or 8 depending on whether the CLOSE is successful or not.
DELETE FILE:
A dataset can be deleted.
Syntax:
DELETE DATASET <file name>.
SY-SUBRC will be set to 0 or 8 depending on whether the DELETE is successful or not.
Pseudo logic for processing the sequential files:
For reading:
Open dataset for input in a particular mode.
Start DO loop.
Read dataset into a field.
If READ is not successful.
Exit the loop.
Endif.
Do relevant processing for that record.
End the do loop.
Close the dataset.
For writing:
Open dataset for output / Appending in a particular mode.
Populate the field that is to be transferred.
TRANSFER the filed to a dataset.
Close the dataset.
Regards
Anver
if hlped pls mark points

Purchase Order released date field and table name

Hi,
Please tell me the Purchase Order released date field and table name.
Regards
Deepak

Hi,
Table : EKKO.
Fields : FRGKE,FRGZU.
You can get release date of PO from CDHDR table.
give object value as your po number and check.dont forget to give leading zeros.
The udate will be date field and transaction will be me29n for relaesed po.

Large data sets and table partitioning

Similar Messages

Maybe you are looking for