Interfacing a large set of data mysql

My Flex app is currently interfacing a mySQL table of about
3Million rows. Flex is used just to view and slice the data;
nothing is ever written back to the database.
We just moved from e4x to using amfphp and experienced a good
performance increase but still performance is quite slow. Paging or
"splitting" is not a real option since the charts and tables on the
Flex front are already aggregating.
What do you think is the fastest way for Flex to read a MYsql
database? Is amfphp the best method or am I missing a trick? I
presume there is a much better way but just need a nudge in the
right direction!

Similar Messages

Problem while having a large set of data to work on!

Hi,
I am facing great problem with processing large set of data. I have a requirement in which i'm supposed to generate a report.
I have a table and a MView, which i have joined to reduce the number of records to process. The MView holds 200,00,000 records while the table 18,00,000. Based on join conditions and where clause i'm able to break down the useful data to approx 4,50,000 and i'm getting 8 of my report columns from this join. I'm dumping these records into the table from where i'll be generating the report by spooling.
Below is the block which takes 12mins to insert into the report table MY_ACCOUNT_PHOTON_DUMP:
begin
dbms_output.put_line(to_char(sysdate,'hh24:mi:ss'));
insert into MY_ACCOUNT_PHOTON_DUMP --- Report table
(SUBSCR_NO, ACCOUNT_NO, AREA_CODE, DEL_NO, CIRCLE, REGISTRATION_DT, EMAIL_ID, ALT_CNTCT_NO)
select crm.SUBSCR_NO, crm.ACCOUNT_NO, crm.AREA_CODE, crm.DEL_NO, crm.CIRCLE_ID,
aa.CREATED_DATE, aa.EMAIL_ID, aa.ALTERNATE_CONTACT
from MV_CRM_SUBS_DTLS crm, --- MView
(select /*+ ALL_ROWS */ A.ALTERNATE_CONTACT, A.CREATED_DATE, A.EMAIL_ID, B.SUBSCR_NO
from MCCI_PROFILE_DTLS a, MCCI_PROFILE_SUBSCR_DTLS b
where A.PROFILE_ID = B.PROFILE_ID
and B.ACE_STATUS = 'N'
) aa --- Join of two tables giviing me 18,00,000 recs
where crm.SUBSCR_NO = aa.SUBSCR_NO
and crm.SRVC_TYPE_ID = '125'
and crm.END_DT IS NULL;
INTERNET_METER_TABLE_PROC_1('MCCIPRD','MY_ACCOUNT_PHOTON_DUMP'); --- calling procedure to analyze the report table
COMMIT;
dbms_output.put_line(to_char(sysdate,'hh24:mi:ss'));
end; --- 12 min 04 secFor the rest of the 13 columns required i am running a block which has a FOR UPDATE cursor on the report table:
declare
cursor cur is
select SUBSCR_NO, ACCOUNT_NO, AREA_CODE, DEL_NO,
CIRCLE, REGISTRATION_DT, EMAIL_ID, ALT_CNTCT_NO
from MCCIPRD.MY_ACCOUNT_PHOTON_DUMP --where ACCOUNT_NO = 901237064
for update of
MRKT_SEGMNT, AON, ONLINE_PAY, PAID_AMNT, E_BILL, ECS, BILLED_AMNT,
SRVC_TAX, BILL_PLAN, USAGE_IN_MB, USAGE_IN_MIN, NO_OF_LOGIN, PHOTON_TYPE;
v_aon VARCHAR2(10) := NULL;
v_online_pay VARCHAR2(10) := NULL;
v_ebill VARCHAR2(10) := NULL;
v_mkt_sgmnt VARCHAR2(50) := NULL;
v_phtn_type VARCHAR2(50) := NULL;
v_login NUMBER(10) := 0;
v_paid_amnt VARCHAR2(50) := NULL;
v_ecs VARCHAR2(10) := NULL;
v_bill_plan VARCHAR2(100):= NULL;
v_billed_amnt VARCHAR2(10) := NULL;
v_srvc_tx_amnt VARCHAR2(10) := NULL;
v_usg_mb NUMBER(10) := NULL;
v_usg_min NUMBER(10) := NULL;
begin
dbms_output.put_line(to_char(sysdate,'hh24:mi:ss'));
for rec in cur loop
begin
select apps.TTL_GET_DEL_AON@MCCI_TO_PRD591(rec.ACCOUNT_NO, rec.DEL_NO, rec.CIRCLE)
into v_aon from dual;
exception
when others then
v_aon := 'NA';
end;
SELECT DECODE(COUNT(*),0,'NO','YES') into v_online_pay
FROM TTL_DESCRIPTIONS@MCCI_TO_PRD591
WHERE DESCRIPTION_CODE IN(SELECT DESCRIPTION_CODE FROM TTL_BMF_TRANS_DESCR@MCCI_TO_PRD591
WHERE BMF_TRANS_TYPE
IN (SELECT BMF_TRANS_TYPE FROM
TTL_BMF@MCCI_TO_PRD591 WHERE ACCOUNT_NO = rec.ACCOUNT_NO
AND POST_DATE BETWEEN
TO_DATE('01-'||TO_CHAR(SYSDATE,'MM-YYYY'),'DD-MM-YYYY') AND SYSDATE
AND DESCRIPTION_TEXT IN (select DESCRIPTION from fnd_lookup_values@MCCI_TO_PRD591 where
LOOKUP_TYPE='TTL_ONLINE_PAYMENT');
SELECT decode(count( *),0,'NO','YES') into v_ebill
FROM TTL_CUST_ADD_DTLS@MCCI_TO_PRD591
WHERE CUST_ACCT_NBR = rec.ACCOUNT_NO
AND UPPER(CUSTOMER_PREF_MODE) ='EMAIL';
begin
select ACC_SUB_CAT_DESC into v_mkt_sgmnt
from ttl_cust_dtls@MCCI_TO_PRD591 a, TTL_ACCOUNT_CATEGORIES@MCCI_TO_PRD591 b
where a.CUST_ACCT_NBR = rec.ACCOUNT_NO
and a.market_code = b.ACC_SUB_CAT;
exception
when others then
v_mkt_sgmnt := 'NA';
end;
begin
select nvl(sum(TRANS_AMOUNT),0) into v_paid_amnt
from ttl_bmf@MCCI_TO_PRD591
where account_no = rec.ACCOUNT_NO
AND POST_DATE
BETWEEN TO_DATE('01-'||TO_CHAR(SYSDATE,'MM-YYYY'),'DD-MM-YYYY')
AND SYSDATE;
exception
when others then
v_paid_amnt := 'NA';
end;
SELECT decode(count(1),0,'NO','YES') into v_ecs
from ts.Billdesk_Registration_MV@MCCI_TO_PRD591 where ACCOUNT_NO = rec.ACCOUNT_NO
and UPPER(REGISTRATION_TYPE ) = 'ECS';
SELECT decode(COUNT(*),0,'PHOTON WHIZ','PHOTON PLUS') into v_phtn_type
FROM ts.ttl_cust_ord_prdt_dtls@MCCI_TO_PRD591 A, ttl_product_mstr@MCCI_TO_PRD591 b
WHERE A.SUBSCRIBER_NBR = rec.SUBSCR_NO
and (A.prdt_disconnection_date IS NULL OR A.prdt_disconnection_date > SYSDATE )
AND A.prdt_disc_flag = 'N'
AND A.prdt_nbr = b.product_number
AND A.prdt_type_id = b.prouduct_type_id
AND b.first_level LIKE 'Feature%'
AND UPPER (b.product_desc) LIKE '%HSIA%';
SELECT count(1) into v_login
FROM MCCIPRD.MYACCOUNT_SESSION_INFO a
WHERE (A.DEL_NO = rec.DEL_NO or A.DEL_NO = ltrim(rec.AREA_CODE,'0')||rec.DEL_NO)
AND to_char(A.LOGIN_TIME,'Mon-YYYY') = to_char(sysdate-5,'Mon-YYYY');
begin
select PACKAGE_NAME, BILLED_AMOUNT, SERVICE_TAX_AMOUNT, USAGE_IN_MB, USAGE_IN_MIN
into v_bill_plan, v_billed_amnt, v_srvc_tx_amnt, v_usg_mb, v_usg_min from
(select rank() over(order by STATEMENT_DATE desc) rk,
PACKAGE_NAME, USAGE_IN_MB, USAGE_IN_MIN
nvl(BILLED_AMOUNT,'0') BILLED_AMOUNT, NVL(SRVC_TAX_AMNT,'0') SERVICE_TAX_AMOUNT
from MCCIPRD.MCCI_IM_BILLED_DATA
where (DEL_NUM = rec.DEL_NO or DEL_NUM = ltrim(rec.AREA_CODE,'0')||rec.DEL_NO)
and STATEMENT_DATE like '%'||to_char(SYSDATE,'Mon-YY')||'%')
where rk = 1;
exception
when others then
v_bill_plan := 'NA';
v_billed_amnt := '0';
v_srvc_tx_amnt := '0';
v_usg_mb := 0;
v_usg_min := 0;
end;
-- UPDATE THE DUMP TABLE --
update MCCIPRD.MY_ACCOUNT_PHOTON_DUMP
set MRKT_SEGMNT = v_mkt_sgmnt, AON = v_aon, ONLINE_PAY = v_online_pay, PAID_AMNT = v_paid_amnt,
E_BILL = v_ebill, ECS = v_ecs, BILLED_AMNT = v_billed_amnt, SRVC_TAX = v_srvc_tx_amnt,
BILL_PLAN = v_bill_plan, USAGE_IN_MB = v_usg_mb, USAGE_IN_MIN = v_usg_min, NO_OF_LOGIN = v_login,
PHOTON_TYPE = v_phtn_type
where current of cur;
end loop;
COMMIT;
dbms_output.put_line(to_char(sysdate,'hh24:mi:ss'));
exception when others then
dbms_output.put_line(SQLCODE||'::'||SQLERRM);
end;The report takes >6hrs. I know that most of the SELECT queries have ACCOUNT_NO as WHERE clause and can be joined, but when i joining few of these blocks with the initial INSERT query it was no better.
The individual queries within the cursor loop dont take more then 0.3 sec to execute.
I'm using the FOR UPDATE as i know that the report table is being used solely for this purpose.
Can somebody plz help me with this, i'm in desperate need of good advice here.
Thanks!!
Edited by: user11089213 on Aug 30, 2011 12:01 AM

Hi,
Below is the explain plan for the original query:
select /*+ ALL_ROWS */ crm.SUBSCR_NO, crm.ACCOUNT_NO, ltrim(crm.AREA_CODE,'0'), crm.DEL_NO, >crm.CIRCLE_ID
from MV_CRM_SUBS_DTLS crm,
        (select /*+ ALL_ROWS */ A.ALTERNATE_CONTACT, A.CREATED_DATE, A.EMAIL_ID, B.SUBSCR_NO
        from MCCIPRD.MCCI_PROFILE_DTLS a, MCCIPRD.MCCI_PROFILE_SUBSCR_DTLS b
        where A.PROFILE_ID = B.PROFILE_ID
        and   B.ACE_STATUS = 'N'
        ) aa
where crm.SUBSCR_NO    = aa.SUBSCR_NO
and   crm.SRVC_TYPE_ID = '125'
and   crm.END_DT IS NULL
| Id | Operation              | Name                     | Rows | Bytes |TempSpc| Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT       |                          | 1481K|   100M|       |   245K (5)| 00:49:09 |
|* 1 | HASH JOIN             |                          | 1481K|   100M|    46M|   245K (5)| 00:49:09 |
|* 2 |   HASH JOIN            |                          | 1480K|    29M|    38M| 13884   (9)| 00:02:47 |
|* 3 |    TABLE ACCESS FULL   | MCCI_PROFILE_SUBSCR_DTLS | 1480K|    21M|       | 3383 (13)| 00:00:41 |
|   4 |    INDEX FAST FULL SCAN| SYS_C002680              | 2513K|    14M|       | 6024   (5)| 00:01:13 |
|* 5 |   MAT_VIEW ACCESS FULL | MV_CRM_SUBS_DTLS_08AUG   | 1740K|    82M|       |   224K (5)| 00:44:49 |
Predicate Information (identified by operation id):
   1 - access("CRM"."SUBSCR_NO"="B"."SUBSCR_NO")
   2 - access("A"."PROFILE_ID"="B"."PROFILE_ID")
   3 - filter("B"."ACE_STATUS"='N')
   5 - filter("CRM"."END_DT" IS NULL AND "CRM"."SRVC_TYPE_ID"='125')Whereas for the modified MView query, the plane remains the same:
select /*+ ALL_ROWS */ crm.SUBSCR_NO, crm.ACCOUNT_NO, ltrim(crm.AREA_CODE,'0'), crm.DEL_NO, >crm.CIRCLE_ID
from    (select * from MV_CRM_SUBS_DTLS
         where SRVC_TYPE_ID = '125'
         and   END_DT IS NULL) crm,
        (select /*+ ALL_ROWS */ A.ALTERNATE_CONTACT, A.CREATED_DATE, A.EMAIL_ID, B.SUBSCR_NO
        from MCCIPRD.MCCI_PROFILE_DTLS a, MCCIPRD.MCCI_PROFILE_SUBSCR_DTLS b
        where A.PROFILE_ID = B.PROFILE_ID
        and   B.ACE_STATUS = 'N'
        ) aa
where crm.SUBSCR_NO = aa.SUBSCR_NO
| Id | Operation              | Name                     | Rows | Bytes |TempSpc| Cost (%CPU)| Time     |
|   0 | SELECT STATEMENT       |                          | 1481K|   100M|       |   245K (5)| 00:49:09 |
|* 1 | HASH JOIN             |                          | 1481K|   100M|    46M|   245K (5)| 00:49:09 |
|* 2 |   HASH JOIN            |                          | 1480K|    29M|    38M| 13884   (9)| 00:02:47 |
|* 3 |    TABLE ACCESS FULL   | MCCI_PROFILE_SUBSCR_DTLS | 1480K|    21M|       | 3383 (13)| 00:00:41 |
|   4 |    INDEX FAST FULL SCAN| SYS_C002680              | 2513K|    14M|       | 6024   (5)| 00:01:13 |
|* 5 |   MAT_VIEW ACCESS FULL | MV_CRM_SUBS_DTLS_08AUG   | 1740K|    82M|       |   224K (5)| 00:44:49 |
Predicate Information (identified by operation id):
   1 - access("CRM"."SUBSCR_NO"="B"."SUBSCR_NO")
   2 - access("A"."PROFILE_ID"="B"."PROFILE_ID")
   3 - filter("B"."ACE_STATUS"='N')
   5 - filter("CRM"."END_DT" IS NULL AND "CRM"."SRVC_TYPE_ID"='125')Also took your advice and tried to merge all the queries into single INSERT SQL, will be posting the results shortly.
Edited by: BluShadow on 30-Aug-2011 10:21
added {noformat}{noformat} tags. Please read {message:id=9360002} to learn to do this yourself

Best Practices for loading large sets of Data

Just a general question regarding an initial load with a large set of data.
Does it make any sense to use a materialized view to aid with load times for an initial load? Or do I simply let the query run for as long as it takes.
Just looking for advice on what is the common approach here.
Thanks!

Hi GK,
What I have normally seen is:
1) Data would be extracted from APO Planning Area to APO Cube (FOR BACKUP purpose). Weekly or monthly, depending on how much data change you expect, or how critical it is for business. Backups are mostly monthly for DP.
2) Data extracted from APO planning area directly to DSO of staging layer in BW, and then to BW cubes, for reporting.
For DP monthly, SNP daily
You can also use the option 1 that you mentioned below. In this case, the APO cube is the backup cube, while the BW cube is the one that you could use for reporting, and this BW cube gets data from APO cube.
Benefit in this case is that we have to extract data from Planning Area only once. So, planning area is available for jobs/users for more time. However, backup and reporting extraction are getting mixed in this case, so issues in the flow could impact both the backup and the reporting. We have used this scenario recently, and yet to see the full impact.
Thanks - Pawan

Perl API: growing memory problem in loops over large sets of data

Hi,
When going through all XmlResults like this:
while ($results->next($val)) {
print $val->asString, "\n";
The process size keeps growing. It does not when I comment $val->asString method out, but then I have no way of getting the results.
This becomes a significant problem when the number of results is huge. I am doing this on a database of over a million short XML documents (400-800 bytes each).
The more complete code is here:
eval {
$env = new DbEnv();
$env->open($dbDir, Db::DB_JOINENV | Db::DB_INIT_LOCK
| Db::DB_INIT_MPOOL | Db::DB_CREATE, 0);
my $mgr = new XmlManager($env, DbXml::DBXML_ADOPT_DBENV);
my $db = $mgr->openContainer(undef, $dbName, Db::DB_RDONLY);
my $context = $mgr->createQueryContext(XmlQueryContext::LiveValues,
XmlQueryContext::Lazy);
my $lookup = $mgr->createIndexLookup($db, "", $nodeName,
"node-$nodeType-equality-$syntax",
new XmlValue($types{$nodeType}, $value), XmlIndexLookup::GTE);
my $results = $lookup->execute(undef, $context);
my $val = new XmlValue();
while ($results->next($val)) {
print $val->asString, "\n";
if (my $e = catch std::exception) {
die $e->what() . "\n";
The process size just grows until the system limit is reached, then the process quits saying 'Out of memory'.
I suspect the problem is with the std::string result returned by C++ XmlValue::asString() const.
The (left-hand-side) result string is likely allocated by new std::string and receives the value by calling the string copy operator. Then the Perl scalar result is prepared, but when it gets returned to my code, the C++ string is not deleted.
Moving the Sleepycat::XmlValue Perl object inside the loop does not help either:
while ($results->hasNext()) {
my $val = new XmlValue();
$results->next($val);
print $val->asString, "\n";
In fact, the process seems to grow faster, possibly because the old $val instances do not get destroyed by Perl at the end of the loop. Where is Perl's garbage collection?
I am using DB XML version: 2.2.13; BDB version: 4.4.20.2; OS: FreeBSD 6-STABLE. However the problem seems to be common for any OS or BDB XML version as it involves Perl-to-C++ interface.
Has anyone experienced similar problems?
Thanks,
Konstantin.
Konstantin @ Chuguev.com

Good catch - you found a memory leak. Luckily the fix is very straightforward. Edit the file
dbxml/src/perl/common.h
and find this line
#define newSVfromString(str) newSVpvn(str.c_str(), str.length())
Change it to this
#define newSVfromString(str) sv_2mortal(newSVpvn(str.c_str(), str.length()))
and recompile the module.
Paul

VO: Large Number of Data

Hi,
i have an issue with large set of data, one of my requirement is to select all row in the table, so ctrl + A will quickly select them all, but if i want to do operation on the selected record, the server crashes because it have like 2,00,000+ rows and while(rowSet.hasNext()) will take a very huge memory.
can any one please help me with this situation.
Thanks
jdev: 11.1.1.3.0

Thanks Timo for the reply, our applications has all most all tables with multiselect and CTRL + A works for all, and operations depends, user can select all and delete, export programmatically, loop through each selected row to find something, so i am wondering how can we do it.
I normally end up doing a virtual select all, meaning that I set a flag which gets interpreted in method (in the application module or the vo) and the method acts on all rows. i am not clear on this. Please can you give an example.
and i am also not clean, how can we do select all visible row.
Thank you so much.

Breakdown a largeish set of data

Hi, I am working with a large set of data that I would like to break down into smaller sets. I can filter the data in the sheet but I can't find a way to copy just the filter results into a new numbers document. All the data seems to be copied when selecting just the visible rows. Is this possible, and how is it done?
Many thanks in advance for any help with this.

koan22,
You can create another table with the filtered rows by adding an auxiliary column to the original table. This new column contains formulas that identify the rows which meet the conditions used in the filter and numbers them so they can be tied to rows in the Copy Items table:
=IF(ISBLANK(Names),"",IF(LEFT(Names)="B",MAX($C$1 :INDIRECT(ADDRESS(ROW()-1,3)))+1,""))
The Names formula in the "Copied Items" table becomes:
=IF(ROW()-1>MAX(Original :: Aux),"",LOOKUP(ROW()-1,Original :: Aux,Original :: Names))
If additional columns are to be listed, a similar formula for each is needed.
pw

***What is your favorite VI or other software to view large sets of waveform data?

What is your favorite VI or other software to view large sets of
waveform data?
I have to look at waveforms with several sets of y data of 10000 or so
points. Should I put this into Excel? There are no cursors, or zoom
capabilities, and so on in Excel.
Using a graph in LV is very slow when loaded down with this much data.
Frankly a scope is easier for viewing waveforms. A scopelike utility
for Windows?
MathCAD?
Is there any neat shareware out there?
Thanks for your opinion,
Mike

There are a large number of analysis packages on the market that can display very large datasets. The one to choose depends largely on the nature of the data and what you want to do with it after you have loaded it in. Wandering down the software aisle at your local Staples you will find at least two or three.
With datsets the size you are contemplating, a major feature to look for is a zoom function that initially shows the entire waveform but lets you narrow in on the data you want to see.
DIGRESSION ALERT
Actually, this sort of algorithm isn't to hard to code. I wrote a viewer once that allowed you to view datasets of arbitrarily large size (in V3 I think... though it may have been in V4). The code initally presented an overview of the entire dataset
by decimating it down to a size that LV could display rapidly. As the user zoomed in using the cursors (not LV's graph zoom) the code would extract the selected portion of the data, redecimate it if necessary and redraw the display. (Which by the way is all the fancy packages really do anyway.)
The tricky part is selecting a decimation function that retains the overall shape of the data.
ENDOF DIGRESSION
Several years ago I wrote an article for a magazine (Personal Engineering and Instrumentation News) that dealt with graphing packages that could handle a million datapoints at that time (late eighties) there were over a dozen programs--and they were all better than Excel...
Mike...
Certified Professional Instructor
Certified LabVIEW Architect
LabVIEW Champion
"... after all, He's not a tame lion..."
Be thinking ahead and mark your dance card for NI Week 2015 now: TS 6139 - Object Oriented First Steps

Processing large volumes of data in PL/SQL

I'm working on a project which requires us to process large volumes of data on a weekly/monthly/quarterly basis, and I'm not sure we are doing it right, so any tips would be greatly appreciated.
Requirement
Source data is in a flat file in "short-fat" format i.e. each data record (a "case") has a key and up to 2000 variable values.
A typical weekly file would have maybe 10,000 such cases i.e. around 20 million variable values.
But we don't know which variables are used each week until we get the file, or where they are in the file records (this is determined via a set of meta-data definitions that the user selects at runtime). This makes identifying and validating each variable value a little more interesting.
Target is a "long-thin" table i.e. one record for each variable value (with numeric IDs as FKs to identify the parent variable and case.
We only want to load variable values for cases which are entirely valid. This may be a merge i.e. variable values may already exist in the target table.
There are various rules for validating the data against pre-existing data etc. These rules are specific to each variable, and have to be applied before we put the data in the target table. The users want to see the validation results - and may choose to bail out - before the data is written to the target table.
Restrictions
We have very limited permission to perform DDL e.g. to create new tables/indexes etc.
We have no permission to use e.g. Oracle external tables, Oracle directories etc.
We are working with standard Oracle tools i.e. PL/SQL and no DWH tools.
DBAs are extremely resistant to giving us more disk space.
We are on Oracle 9iR2, with no immediate prospect of moving to 10g.
Current approach
Source data is uploaded via SQL*Loader into static "short fat" tables.
Some initial key validation is performed on these records.
Dynamic SQL (plus BULK COLLECT etc) is used to pivot the short-fat data into an intermediate long-thin table, performing the validation on the fly via a combination of including reference values in the dynamic SQL and calling PL/SQL functions inside the dynamic SQL. This means we can pivot+validate the data in one step, and don't have to update the data with its validation status after we've pivoted it.
This upload+pivot+validate step takes about 1 hour 15 minutes for around 15 million variable values.
The subsequent "load to target table" step also has to apply substitution rules for certain "special values" or NULLs.
We do this by BULK collecting the variable values from the intermediate long-thin table, for each valid case in turn, applying the substitution rules within the SQL, and inserting into/updating the target table as appropriate.
Initially we did this via a SQL MERGE, but this was actually slower than doing an explicit check for existence and switching between INSERT and UPDATE accordingly (yes, that sounds fishy to me too).
This "load" process takes around 90 minutes for the same 15 million variable values.
Questions
Why is it so slow? Our DBAs assure us we have lots of table-space etc, and that the server is plenty powerful enough.
Any suggestions as to a better approach, given the restrictions we are working under?
We've looked at Tom Kyte's stuff about creating temporary tables via CTAS, but we have had serious problems with dynamic SQL on this project, so we are very reluctant to introduce more of it unless it's absolutely necessary. In any case, we have serious problems getting permissions to create DB objects - tables, indexes etc - dynamically.
So any advice would be gratefully received!
Thanks,
Chris

We have 8 "short-fat" tables to hold the source data uploaded from the source file via SQL*Loader (the SQL*Loader step is fast). The data consists simply of strings of characters, which we treat simply as VARCHAR2 for the most part.
These tables consist essentially of a case key (composite key initially) plus up to 250 data columns. 8*250 = 2000, so we can handle up to 2000 of these variable values. The source data may have 100 any number of variable values in each record, but each record in a given file has the same structure. Each file-load event may have a different set of variables in different locations, so we have to map the short-fat columns COL001 etc to the corresponding variable definition (for validation etc) at runtime.
CASE_ID VARCHAR2(13)
COL001 VARCHAR2(10)
COL250 VARCHAR2(10)
We do a bit of initial validation in the short-fat tables, setting a surrogate key for each case etc (this is fast), then we pivot+validate this short-fat data column-by-column into a "long-thin" intermediate table, as this is the target format and we need to store the validation results anyway.
The intermediate table looks similar to this:
CASE_NUM_ID NUMBER(10) -- surrogate key to identify the parent case more easily
VARIABLE_ID NUMBER(10) -- PK of variable definition used for validation and in target table
VARIABLE_VALUE VARCHAR2(10) -- from COL001 etc
STATUS VARCHAR2(10) -- set during the pivot+validate process above
The target table looks very similar, but holds cumulative data for many weeks etc:
CASE_NUM_ID NUMBER(10) -- surrogate key to identify the parent case more easily
VARIABLE_ID NUMBER(10) -- PK of variable definition used for validation and in target table
VARIABLE_VALUE VARCHAR2(10)
We only ever load valid data into the target table.
Chris

Large Amount of Data in JSF

Hello,
I am using the Table Group component for displaying data in my application designed in Java Studio Creator.
I have enabled paging on the component. I use CachedRowSet on the bean for the page for getting the data. This works very well at the moment in my development environment. At the moment I am testing on small amount of data.
I was wondering how does this component perform with very large amounts of data (>75,000 rows). I noticed that there is a button available for users to retrieve all the rows. So I was wondering apart from that instance, when viewing in a paged mode does the component get all the results from the database everytime ?
Which component would be best suited for displaying large amounts of data in a table format?
Thanks In Advance!!

Thanks for your reply. The table control that I use does have paging as a feature and I have enabled it. It still takes time to load the data initially.
I wonder if it is got to do with the logic of paging. How do you specify which set of 20 records to extract from SQL.
Thanks for your help!!

Bex Report Designer - Large amount of data issue

Hi Experts,
I am trying to execute (on Portal) report made in BEx Report Designer, with about 30 000 pages, and the only thing I am getting is a blank page. Everything works fine at about 3000 pages. Do I need to set something to allow processing such large amount of data?
Regards
Vladimir

Hi Sauro,
I have not seen this behavior, but it has been a while since I tried to send an input schedule that large. I think the last time was on a BPC NW 7.0 SP06 system and it worked OK. If you are on a recent support package, then you should search for relevant notes (none come to mind for me, but searching yourself is always a good idea) and if you don't find one then you should open a support message with SAP, with very specific instructions for recreating the problem from a clean input-schedule.
Good luck,
Ethan

Looking for ideas for transferring large amounts of data between systems

Hello,
I am looking for ideas based on best practices for transferring Large Amounts of Data in and out of a Netweaver based application.
We have a new system we are developing in Netweaver that will utilize both the Java and ABAP stack, and will require integration with other SAP and 3rd Party Systems. It is a standalone product that doesn't share any form of data store with other systems.
We need to be able to support 10s of millions of records of tabular data coming in and out of our system.
Since we need to integrate with so many different systems, we are planning to use RFC for our primary interface in and out of the system. As it turns out RFC is not good at dealing with this large amount of data being pushed through a single call.
We have considered a number of possible ideas, however we are not very happy with any of them. I would like to see what the community has done in the past to solve problems like this as well as how SAP currently solves this problem in other applications like XI, BI, ERP, etc.

Primoz wrote:Do you use KDE (Dolphin) 4.6 RC or 4.5?
Also I've noticed that if i move / copy things with Dolphin they're substantially slower than if I use cp/mv. But cp/mv works fine for me...
Also run Dolphin from terminal to try and see what's the problem.
Hope that help at least a bit.
Could you explain why Dolphin should be slower? I'm not attacking you, I'm just asking.
Cause I thought that Dolphin is just a „little" wrapper around the cp/mv/cd/ls applications/commands.

Dealing with large volumes of data

Background:
I recently "inherited" support for our company's "data mining" group, which amounts to a number of semi-technical people who have received introductory level training in writing SQL queries and been turned loose with SQL Server Management
Studio to develop and run queries to "mine" several databases that have been created for their use. The database design (if you can call it that) is absolutely horrible. All of the data, which we receive at defined intervals from our
clients, is typically dumped into a single table consisting of 200+ varchar(x) fields. There are no indexes or primary keys on the tables in these databases, and the tables in each database contain several hundred million rows (for example one table
contains 650 million rows of data and takes up a little over 1 TB of disk space, and we receive weekly feeds from our client which adds another 300,000 rows of data).
Needless to say, query performance is terrible, since every query ends up being a table scan of 650 million rows of data. I have been asked to "fix" the problems.
My experience is primarily in applications development. I know enough about SQL Server to perform some basic performance tuning and write reasonably efficient queries; however, I'm not accustomed to having to completely overhaul such a poor design
with such a large volume of data. We have already tried to add an identity column and set it up as a primary key, but the server ran out of disk space while trying to implement the change.
I'm looking for any recommendations on how best to implement changes to the table(s) housing such a large volume of data. In the short term, I'm going to need to be able to perform a certain amount of data analysis so I can determine the proper data
types for fields (and whether any existing data would cause a problem when trying to convert the data to the new data type), so I'll need to know what can be done to make it possible to perform such analysis without the process consuming entire days to analyze
the data in one or two fields.
I'm looking for reference materials / information on how to deal with the issues, particularly when a large volumn of data is involved. I'm also looking for information on how to load large volumes of data to the database (current processing of a typical
data file takes 10-12 hours to load 300,000 records). Any guidance that can be provided is appreciated. If more specific information is needed, I'll be happy to try to answer any questions you might have about my situation.

I don't think you will find a single magic bullet to solve all the issues. The main point is that there will be no shortcut for major schema and index changes. You will need at least 120% free space to create a clustered index and facilitate
major schema changes.
I suggest an incremental approach to address you biggest pain points. You mention it takes 10-12 hours to load 300,000 rows, which suggests there may be queries involved in the process which require full scans of the 650 million row table. Perhaps
some indexes targeted at improving that process is a good first step.
What SQL Server version and edition are you using? You'll have more options with Enterprise (partitioning, row/page compression).
Regarding the data types, I would take a best guess at the proper types and run a query with TRY_CONVERT (assuming SQL 2012) to determine counts of rows that conform or not for each column. Then create a new table (using SELECT INTO) that has strongly
typed columns for those columns that are not problematic, plus the others that cannot easily be converted, and then drop the old table and rename the new one. You can follow up later to address columns data corrections and/or transformations.
Dan Guzman, SQL Server MVP, http://www.dbdelta.com

Azure Cloud service fails when sent large amount of data

This is the error;
Exception in AZURE Call: An error occurred while receiving the HTTP response to http://xxxx.cloudapp.net/Service1.svc. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being
aborted by the server (possibly due to the service shutting down). See server logs for more details.
Calls with smaller amounts of data work fine. Large amounts of data cause this error.
How can I fix this??

Go to the web.config file, look for the <binding> that is being used for your service, and adjust the various parameters that limit the maximum length of the messages, such as
maxReceivedMessageSize.
http://msdn.microsoft.com/en-us/library/system.servicemodel.basichttpbinding.maxreceivedmessagesize(v=vs.100).aspx
Make sure that you specify a size that is large enough to accomodate the amount of data that you are sending (the default is 64Kb).
Note that even if you set a very large value here, you won't be able to go beyond the maximum request length that is configured in IIS. If I recall correctly, the default limit in IIS is 8 megabytes.

What is the need for setting property data inside the JMSMesage

Hi
Could anybody please let me know
*What is the need for setting property data inside the JMSMesage??
For example i have a seen a similar example as shown ??
I have seen a
Message.setStringProperty("Sport","Basketball");
and also please tell me how can the MDB recievies this property data ??
Thanks in advance .

raviprivate wrote:
Could anybody please let me know
*What is the need for setting property data inside the JMSMesage??
For example i have a seen a similar example as shown ??
I have seen a
Message.setStringProperty("Sport","Basketball"); Look at the detail JMS documentation on [Message Properties|http://java.sun.com/j2ee/sdk_1.3/techdocs/api/javax/jms/Message.html] .
>
and also please tell me how can the MDB recievies this property data ?? MDB onMessage method argument is the Message object and if you look at the documentation, Message interface has getter methods to retrieve the properties.

Any easier way to set due dates?

By and large, I'm pretty happy with the functionality of iCal, even if it's a bit basic. But setting due dates on to dos and events is very frustrating!
Let's say you add a new to do, then click the due date box. Today's date is filled in, say, 26/08/2007 in my part of the world. I want to schedule the to do for next month. To change it you click on the day part, enter a number (say, 08), then tab or click on the month part. At this time the iCal display jumps to 08/08/2007 (in the past!). Now I get to manually type in the new month, 9, and press Return or tab and things are good again. It's all kind of long-winded and messy. What's sorely needed is a drop-down calendar picker, or the ability to click on the existing calendar at the left to pick a date.
Does anyone know of any easier way to enter due dates? I've even looked for external programs that handle to do functionality, but can't find any that automatically sync changes to iCal for syncing to my Palm Treo. Anyone know if iCal in Leopard is any better in this regard?
Thanks in advance!

I couldn't agree more. I can't believe that this has escaped Apple for so long, and didn't even make it into Leopard. Let's say its the middle of November and I want to schedule a to-do due date for the end of the week, say Friday, second week of December. I do this all the time. Sorry, but I can't do the math this quickly, so I need to open up a calendar, or scroll down the calendar, then enter the exact date manually.
Just give me a quick pop-up. Travel web sites do this. Now Up To Date in 1992 had drop down calendars. So does Entourage, which is what I use ONLY because of this one glaring pain. Entering due dates, reminders, etc, is just a pain in iCal, and even worse now in Mail.

Interfacing a large set of data mysql

Similar Messages

Maybe you are looking for