SQLJ for Data Intensive Application?
Hi!
I am thinking about writing a program that is very data-intensive data-mining application. It will process millions of records where each record is a 80 byte character. I will need to do a lot of comparisons and sub-string operations. I wonder if anyone out here has done some similar project in Java before. Any information would help. I am particularly interested in looking for information pertain to reading all data into memory for Java processing. Also, should I use PL/SQL or Oracle C instead?
Thanks.
Aaron
it depends what you mean by "millions" of (80 byte)records
if you mean about 5 million, then you will be dealing with 400 Megs or info at one time, and if you have a fancy computer with a whole bunch of ram you probably won't have any problem just loading all the info into ram at once. but if your talking 200 million well then you will probably have some redesigning to do.
Other then that the processing time really just depends on what you are doing to the data, and since you haven't been specific about it it is hard to tell how the performance of your program will be.
Similar Messages
-
Options for Bandwidth intensive application
I appreciate any recommendations, I have a remote office needing to access a very bandwidth intensive application (scanned TIF images) currently we are looking at a 100Mb metro ethernet or DS3 type of link to our office where the DB resides. Are there any content caching products or new technology that would help me improve access to this application?
in order to accomodate content caching as you've described you need, for an enterprise class environment, you'll want to check out the Cisco CSS line of content switches.
a quick view of the CSS abilities for content caching includes:
(taken from cco documentation)
* examining network traffic for web content requests
* bypassing the cache automatically for non-cachable content
* distributing content requests to maximize cache hits
* bypassing the cache or redistributing requests among the remaining cache services if a cache service fails
* proxy caching capabilities.
among other capabilities, this device would allow you to create a 'farm' of web servers where your content can be housed utilizing faster response times and high availability.
please see the following link for more info on the CSS and its abilities:
(there are other 'model' versions of the CSS than the one linked here)
http://www.cisco.com/en/US/products/hw/contnetw/ps792/products_data_sheet0900aecd800f851e.html -
What goes where when using separate drives for data and applications?
Hello people!
In Mac Guru's "Guide to Macintosh Performance Acceleration" it states " Separating your "User" data from the Operating System and Applications is the simplest and most cost effective way to enjoy a significant performance boost at
minimal cost"...
Now how exactly do I do that?
I have a 40 GB HD with Panther on it and a 55 GB drive I want to use for the "data" drive. What goes where?
Thanks for reading!Hello Danny:
Welcome to Apple discussions.
I disagree with the premise in that article - "significant" is relative and depends on where you are. The only method I am aware of to "separate" would be to create partitions. OS X uses sophisticated algorithms to manage memory - both real and virtual. There are very few good reasons to partition an internal HD - and several that suggest you should not do that - absent a "good" reason.
Barry -
I have an iPhone 4S that I purchased new from Verizon back in January 2012. I've started to notice that the 3G data coverage appears and disappears according to whether or not I'm actually using the data. For instance: When the phone is stationary and not using the data, it indicates that I have 3G data coverage. When I click on an app like Netflix (without moving the phone from it's stationary position), data coverage either immediately or soon thereafter switches to a 2G data coverage connection (indicated with an "o" for Verizon). Has anyone else had this same problem? I can't say that this happens everywhere, but in particular, it happens at my work every single time I would like to be able to watch episodes of tv shows on my lunch break.
Yeah I'll need to call and ask because it's very possible that the tower is "congested" as I work in a large city. Hardly seems fair that I would get a diminished data speeds due to congestion of a large city that Verizon indicates it has full coverage on all of their "data coverage" maps. I may have made a different carrier choice had I known that this was going to happen. I wonder if this happens with their 4G towers as well...
-
Query tuning for data-warehousing application in Oracle 8i.
We have to pick up 24 months old data. Each month data is kept in a different partition.
2007-May month data is kept in PRESC200705 partition
SELECT r.account_id,
p.presc_num,
spm.product_id,
p.month,
t.best_call_state,
sum(p.trx_count)
FROM rlup_assigned_account r,
temp_presc_num_TEST t,
retail.prescrip_retail partition (PRESC200705) p,
sherlock.sherlock_product_mapping spm
WHERE spm.product_id like '056%'
and spm.mds6 = p.product_id
and t.CLIENT_ID = p.presc_num
and r.ndc_pyr_id = p.payer_plan
and t.best_call_state = r.ST
GROUP BY r.account_id,
p.presc_num,
spm.product_id,
p.month,
t.best_call_state
Q This Query is to be tuned
SQL> SELECT table_name,
2 partition_name,
3 high_value,
4 num_rows
5 FROM user_tab_partitions
6 ;
no rows selectedI have the following task:
Requirement:
According to the client, new partitions are created every month.
So the query should contain only the 24 most recent partions leaving one old partition every month.
So, the query becomes dynamic.
The query will have to leave one old partion every month and move ahead with the new partion created.
The total partitions accessed should not cross 24.
Is this possible?
Partition# for OCt 2007
1
2
3
4
5
6
7
24
Partition# for Nov 2007
Old|New
1
2..1
3..2
4..3
5..4
6..5
7..6
24..23
..24
Secondly, with one month data (in a partitioned table), the query takes about one hour to run.
with 24 month data accessed by the query, the query will take 24 hours to run.
I am sure that Oracle can be tuned to run on such huge data with ease and the query output can
come within seconds.. Otherwise, nobody would use oracle for datawarehousing applicaions.
Q. How do I write a dynamic query that references 24 recent partitions, using the query provided:
abc>SELECT r.account_id,
2 p.presc_num,
3 spm.product_id,
4 p.month,
5 t.best_call_state,
6 sum(p.trx_count)
7 FROM rlup_assigned_account r,
8 temp_presc_num_TEST t,
9 retail.prescrip_retail partition (PRESC200705) p,
10 sherlock.sherlock_product_mapping spm
11 WHERE spm.product_id like '056%'
12 and t.CLIENT_ID='934759'
13 and spm.mds6 = p.product_id
14 and t.CLIENT_ID = p.presc_num
15 and r.ndc_pyr_id = p.payer_plan
16 and t.best_call_state = r.ST
17 GROUP BY r.account_id,
18 p.presc_num,
19 spm.product_id,
20 p.month,
21 t.best_call_state
22 ;
retail.prescrip_retail partition (PRESC200705) p,
Partition name, PRESC200705 cannot be hardcoded into the sql.
Sql should take a range of 24 recent partitions.
And the query should execute fast too.
Now, is that what is called a challenge?
+++++++++++++++++++++++++++++++++++++++++++++++
Here are the index/constraints/explain_plan output on prescrip_retail table (which is partitioned)
as well as other tables to which prescrip_retail table is joined
This is what prescrip_retail looks like. This is the table having partitions.
It does not seem to have a primary key!
SQL> desc prescrip_retail
Name Null? Type
PRESC_NUM NUMBER
PFIER_NUM CHAR(8)
RELID NOT NULL CHAR(9)
ME_NUM NOT NULL CHAR(10)
PRODUCT_ID NOT NULL CHAR(6)
PRODUCT_FRMSTR NOT NULL CHAR(1)
PAYER_PLAN NOT NULL CHAR(6)
MONTH NOT NULL DATE
PYMT_CODE NOT NULL CHAR(1)
NRX_COUNT NOT NULL NUMBER(7)
NRX_QUANTITY NOT NULL NUMBER(9)
NRX_DOLLARS NOT NULL NUMBER(13,2)
TRX_COUNT NOT NULL NUMBER(7)
TRX_QUANTITY NOT NULL NUMBER(9)
TRX_DOLLARS NOT NULL NUMBER(13,2)
Table Size of Prescrip_Retail...
1 select table_name,tablespace_name,pct_free,pct_used,num_rows,avg_space
2 from all_tables
3* where table_name='PRESCRIP_RETAIL'
SQL> /
TABLE_NAME TABLESPACE_NAME PCT_FREE PCT_USED NUM_ROWS AVG_SPACE
PRESCRIP_RETAIL 2806673860 360
Explain Plan for the query to be tuned...
22:32:31 SQL> explain plan set statement_id='vista_query'
22:43:33 2 for
22:43:35 3 SELECT r.pfier_account_id,
22:43:41 4 p.presc_num,
22:43:41 5 spm.product_id,
22:43:41 6 p.month,
22:43:41 7 t.best_call_state,
22:43:41 8 sum(p.trx_count)
22:43:41 9 FROM rlup_assigned_account r,
22:43:41 10 temp_presc_num_TEST t,
22:43:41 11 retail.prescrip_retail partition (PRESC200705) p,
22:43:41 12 sherlock.sherlock_product_mapping spm
22:43:41 13 WHERE spm.product_id like '056%'
22:43:41 14 and spm.mds6 = p.product_id
22:43:41 15 and t.CLIENT_ID = p.presc_num
22:43:41 16 and r.ndc_pyr_id = p.payer_plan
22:43:41 17 and t.best_call_state = r.ST
22:43:41 18 GROUP BY r.pfier_account_id,
22:43:41 19 p.presc_num,
22:43:41 20 spm.product_id,
22:43:41 21 p.month,
22:43:41 22 t.best_call_state;
Explained.
SQL> select statement_id,operation,options,object_name
2 from plan_table
3 where statement_id='vista_query';
22:46:03 SQL> /
STATEMENT_ID OPERATION OPTIONS OBJECT_NAME
vista_query SELECT STATEMENT
vista_query SORT GROUP BY
vista_query HASH JOIN
vista_query TABLE ACCESS FULL TEMP_PRESC_NUM_TEST
vista_query HASH JOIN
vista_query TABLE ACCESS FULL RLUP_ASSIGNED_ACCOUNT
vista_query HASH JOIN
vista_query TABLE ACCESS FULL SHERLOCK_PRODUCT_MAPPING
vista_query TABLE ACCESS FULL PRESCRIP_RETAIL
9 rows selected.
Partition Pruning: This is supposed to provide an insight to the partitions oracle
vists internally...
I guess we can use "month>= add_months(sysdate,-24)" instead of partions too.
I don't think Oracle is visiting any partitions.
I'll also search into all_tab_partitions to verify this.
Explain_Plan for what partitions oracle visits internally (partition pruning):
SQL> ed
Wrote file afiedt.buf
1 explain plan set statement_id='vista'
2 for select * from retail.prescrip_retail
3* where month>= add_months(sysdate,-24)
SQL> /
Explained.
Elapsed: 00:00:00.05
22:13:56 SQL> select statement_id,operation,options,object_name
22:14:28 2 from plan_table
22:14:30 3 where statement_id='vista';
STATEMENT_ID OPERATION
OPTIONS OBJECT_NAME
vista SELECT STATEMENT
vista PARTITION RANGE
ITERATOR
vista TABLE ACCESS
FULL PRESCRIP_RETAIL
Elapsed: 00:00:01.00
Indexes/Constraints on PRESCRIP_RETAIL table:
SQL> ED
Wrote file afiedt.buf
1 SELECT TABLE_NAME,TABLE_TYPE,INDEX_NAME,INDEX_TYPE,PCT_FREE,STATUS,PARTITIONED
2 FROM ALL_INDEXES
3* WHERE TABLE_NAME IN ('PRESCRIP_RETAIL')
SQL> /
TABLE_NAME TABLE INDEX_NAME INDEX_TYPE PCT_FREE STATUS PAR
PRESCRIP_RETAIL TABLE BX6_PRESC_RELID BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX7_PRESC_ME BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX1_PRESC_PROD BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX2_PRESC_PAYER BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX3_PRESC_PAYERCD BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX4_PRESC_PRESC BITMAP N/A YES
PRESCRIP_RETAIL TABLE BX5_PRESC_PFIER BITMAP N/A YES
7 rows selected.
SQL> ed
Wrote file afiedt.buf
1 SELECT TABLE_NAME,CONSTRAINT_NAME,CONSTRAINT_TYPE,STATUS,DEFERRABLE
2 FROM ALL_CONSTRAINTS
3* WHERE TABLE_NAME IN ('PRESCRIP_RETAIL')
SQL> /
TABLE_NAME CONSTRAINT_NAME C STATUS DEFERRABLE
PRESCRIP_RETAIL SYS_C001219 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001220 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001221 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001222 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001223 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001224 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001225 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001226 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001227 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001228 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001229 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001230 C ENABLED NOT DEFERRABLE
PRESCRIP_RETAIL SYS_C001231 C ENABLED NOT DEFERRABLE
13 rows selected.
In all_tables:
NUM_ROWS:2806673860
AVG_SPACE:360
Here is the data size in the table.
SQL> select count(*) from PRESCRIP_RETAIL;
COUNT(*)
4602980312
Again, here is the partition information and the amount of data in each partition:
SQL> ed
Wrote file afiedt.buf
1 select
2 partition_name,SUBPARTITION_COUNT,PARTITION_POSITION,TABLESPACE_NAME,
3 NUM_ROWS
4 from all_tab_partitions
5 where table_name='PRESCRIP_RETAIL'
6* order by partition_name desc
SQL> /
PARTITION_NAME SUBPARTITION_COUNT PARTITION_POSITION TABLESPACE_NAME NUM_ROWS
PRESC200705 0 36 PRESC_PARTITION_29 141147085
PRESC200704 0 35 PRESC_PARTITION_28 140299317
PRESC200703 0 34 PRESC_PARTITION_27 140703128
PRESC200702 0 33 PRESC_PARTITION_26 132592733
PRESC200701 0 32 PRESC_PARTITION_25 145832356
PRESC200612 0 31 PRESC_PARTITION_24 136702837
PRESC200611 0 30 PRESC_PARTITION_23 137421767
PRESC200610 0 29 PRESC_PARTITION_22 140836119
PRESC200609 0 28 PRESC_PARTITION_21 131273578
PRESC200608 0 27 PRESC_PARTITION_20 134967317
PRESC200607 0 26 PRESC_PARTITION_19 130785504
PRESC200606 0 25 PRESC_PARTITION_18 131472696
PRESC200605 0 24 PRESC_PARTITION_17 138590581
PRESC200604 0 23 PRESC_PARTITION_16 126849798
PRESC200603 0 22 PRESC_PARTITION_15 137164667
PRESC200602 0 21 PRESC_PARTITION_14 126938544
PRESC200601 0 20 PRESC_PARTITION_13 135408324
PRESC200512 0 19 PRESC_PARTITION_12 123285100
PRESC200511 0 18 PRESC_PARTITION_11 121245764
PRESC200510 0 17 PRESC_PARTITION_10 122112932
PRESC200509 0 16 PRESC_PARTITION_09 119137399
PRESC200508 0 15 PRESC_PARTITION_08 123372311
PRESC200507 0 14 PRESC_PARTITION_07 112905435
PRESC200506 0 13 PRESC_PARTITION_06 119581406
PRESC200505 0 12 PRESC_PARTITION_05 123977315
PRESC200504 0 11 PRESC_PARTITION_04 118975597
PRESC200503 0 10 PRESC_PARTITION_03 125782688
PRESC200502 0 9 PRESC_PARTITION_02 117448839
PRESC200501 0 8 PRESC_PARTITION_01 122214436
PRESC200412 0 7 PRESC_PARTITION_36 124799998
PRESC200411 0 6 PRESC_PARTITION_35 125471042
PRESC200410 0 5 PRESC_PARTITION_34 118457422
PRESC200409 0 4 PRESC_PARTITION_33 119537488
PRESC200408 0 3 PRESC_PARTITION_32 121319137
PRESC200407 0 2 PRESC_PARTITION_31 115226621
PRESC200406 0 1 PRESC_PARTITION_30 119143031
36 rows selected.
Data in induvidual partition of PRESCRIP_RETAIL.
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200704);
COUNT(*)
140299317
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200703);
COUNT(*)
140703128
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200702);
COUNT(*)
132592733
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200701);
COUNT(*)
145832356
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200606);
COUNT(*)
131472696
SQL> SELECT COUNT(*) FROM PRESCRIP_RETAIL PARTITION(PRESC200605);
COUNT(*)
138590581
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Other tables info:
Index of other tables related to PRESCRIP_REATIL:
SQL> SELECT TABLE_NAME,TABLE_TYPE,INDEX_NAME,INDEX_TYPE,PCT_FREE,STATUS,PARTITIONED
2 FROM ALL_INDEXES
3 WHERE TABLE_NAME IN ('RLUP_ASSIGNED_ACCOUNT','TEMP_PRESC_NUM_TEST','SHERLOCK_PRODUCT_MAPPING');
SQL> /
TABLE_NAME TABLE INDEX_NAME INDEX_TYPE PCT_FREE STATUS PAR
SHERLOCK_PRODUCT_MAPPING TABLE SHERLOCK_PRODUCT_MAPPING_PK NORMAL 10 VALID NO
SHERLOCK_PRODUCT_MAPPING TABLE SHERLOCK_PRODUCT_MAPPING_X1 NORMAL 0 VALID NO
SHERLOCK_PRODUCT_MAPPING TABLE SHERLOCK_PRODUCT_MAPPING_BX1 BITMAP 0 VALID NO
SHERLOCK_PRODUCT_MAPPING TABLE SHERLOCK_PRODUCT_MAPPING_BX2 BITMAP 0 VALID NO
SHERLOCK_PRODUCT_MAPPING TABLE SHERLOCK_PRODUCT_MAPPING_BX3 BITMAP 0 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX1_RLUP_ASSIGNED_ACCT_PYR BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX2_RLUP_ASSIGNED_ACCT_TOPLVL BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX3_RLUP_ASSIGNED_ACCT_PBM BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX4_RLUP_ASSIGNED_ACCT_AA_FLAG BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX5_RLUP_ASSIGNED_ACCT_AA_CHD BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX6_RLUP_ASSIGNED_ACCT_PBM_FLG BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE BX7_RLUP_ASSIGNED_ACCT_ACCTID BITMAP 10 VALID NO
RLUP_ASSIGNED_ACCOUNT TABLE PK_RLUP_ASSIGNED_ACCOUNT NORMAL 10 VALID NO
13 rows selected.
Constraints in other tables:
SQL> SELECT TABLE_NAME,CONSTRAINT_NAME,CONSTRAINT_TYPE,STATUS,DEFERRABLE
2 FROM ALL_CONSTRAINTS
3 WHERE TABLE_NAME IN ('RLUP_ASSIGNED_ACCOUNT','TEMP_PRESC_NUM_TEST','SHERLOCK_PRODUCT_MAPPING');
TABLE_NAME CONSTRAINT_NAME C STATUS DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637753 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637754 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637755 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637756 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637757 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637758 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637759 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637760 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637761 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT SYS_C00637762 C ENABLED NOT DEFERRABLE
RLUP_ASSIGNED_ACCOUNT PK_RLUP_ASSIGNED_ACCOUNT P ENABLED NOT DEFERRABLE
TEMP_PRESC_NUM_TEST SYS_C00640536 C ENABLED NOT DEFERRABLE
12 rows selected.
TEMP_PRESC_NUM_TEST does not contain any constraints or index. -
Which MacBook for graphically intensive applications?
Hey all
Im planning to get a MacBook for my GCSE's, and im leaning towards the Air.
I will be running applications such as Adobe Photoshop CS5, After Effects CS5, Lightroom CS5, and Final Cut Express regularly, so would a 13" 1.86ghz MBA C2D Processor with 4gb of RAM be able to run these apps fast and efficiently with no lag? Or should I opt for the baseline MBP 13"? Im worried that it may be laggy as my cureent Core i3 HP can not handle AE well, but then again, it does have many problems such as slow HDD and only 3gb of Ram
Any help on which i should get? The 13" MBA 1.86ghz 4GB or 13" MBP 2.4ghz?
ThanksFor your needs the described MBA would perform worse than your I3 which is a faster processor. Stick to the 15" MBPs - I5 or I7 which have the better GPUs.
-
How to handle lengthy data-intensive operations in a web app?
Hello,
I have a highly data-intensive application which has to be kick-started from a web page (which could take anywhere between 15-20 mnts even with threads and optimizations). Usually, I prefer having these as cronjobs but this one needs to be under the user's control. I was thinking of submitting the job and sending a response back with some message like "This may take 15-20 minutes. Please check back later to see the status." etc..but I would like to have some sort of notification to the user once it is finished (no mails etc.) instead of the user checking back everytime. Can anybody suggest a good solution for this?
Thanks in advance!There's also a Javascript standard function called init() , called on the HTML body tag ,
for example <body onload="init()"> , and the function gets called as soon as the HTML DOM finishes loading.
I've come accross Javascript code that displays a DIV tag that is tall, and displays a message like "Please wait until this page finishes loading" - and as soon as the page finishes loading the DIV tag is hidden, thus revealing the rest of the page, that was at the bottom of the tall div tag.
This will only work for the entire duration of the page, and can't poll the wait with a given frequency. I don't think it can estimate how long it will take for the page to load.
<div class="loading" id="please_wait_loading" >
<span style="font-size:xx-large;font-family:'Comic Sans MS'">Loading your request. Please Wait!</span>
<br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/><br/>
<span style="font-size:large;"><a href="#startcontent" style="text-decoration:underline">skip this screen</a></span><br/>
</div>
<script type="text/javascript">
var ld;
ld = document.getElementById("please_wait_loading").style;
function init()
ld.visibility = "hidden";
ld.display = "none";
</script> -
Data Access Object for Data Warehouse?
Hi,
Does anyone know how the DAO pattern looks like when it is used for a data warehouse rather than a normal transactional database?
Normally we have something like CustomerDAO or ProductDAO in the DAO pattern, but for data warehouse applications, JOINs are used and multiple tables are queried, for example, a query may contains data from the Customer, Product and Time table, what should the DAO class be named? CustomerProductTimeDAO?? Any difference in other parts of the pattern?
Thanks in advance.
SKIn my opinion, there are no differences in the Data Access Object design pattern which have any thing to do with any characteristic of its implementation or the storage format of the data the pattern is designed to function with.
The core pupose of the DAO design pattern is to encapsulate data access code and separate it from the business logic code of the application. A DAO implementation might vary from application to application. The design pattern does not specify any implementation details. A DAO implementation can be applied to group of XML data files, an Excel-based CSV file, a relational database, or an OS file system. The design is the same for all these, it is the implementation that varies.
The core difference between an operational database and a strategic data warehouse is the purpose of why and how the data is used. It is not so much a technical difference. The relational design may vary however, there may be more tables amd ternary relationships in a data warehouse to support more fine-tuned queries; there may be less tables in a operational database to support insert/add efficiencies.
The DAO implementation for a data warehouse would be based on the model of the databases. However the tables are set up, that is how the DAO is coded. -
Hello,
We have a research project for restructructuring our data warehouse system.
I would like to get some opinions about whether RAC architecture can be
a good solution for Data Warehouse application.
We have using parallel queries massively. Does running these kind of queries
on different servers on RAC with multiple server result in performance degradation rather than
running on single monolithic server with multiple CPUs
I will appreciate any comments using RAC architecture for Data Warehouse
systems?
Regards,Maurice Muller wrote:
Just keep in mind that during the last 4 years (I guess your current system is about 4 years old) the CPUs became much faster.
A cpu can't work without data which means that the I throughput has to be fast enough to feed all your cores with data.
The main bottleneck of all DWHs I have seen during the last 8 years was allways the IO never the CPUs. And not just data warehousing Maurice, but a basic principle for any data processing platform - the slowest layer is always the I/O layer.. and can be the most expensive one to solve too.
Which is why newer technology like Infiniband is exciting as it can also serve as the I/O layer. Instead of using the traditional HBA which is typically configured with 2Gb fibre channels to the storage layer, using HCA cards you can wire this directly into an Infiniband storage array... and this can run at up to speeds of 40Gb. Dual connections means a total theoretical pipe size of 80Gb. I do not know of any other standard technology (like GigE) that can provide any similar bandwidth speed.
Back to RAC though - with RAC, when you add a new server that comes with a new set of I/O pipes.. plus of course more RAM and more CPU cores. SMP server architecture does not scale like this at all. You only have x number of slots for PCI cards, CPUs and RAM. A very specific ceiling that cannot be moved. With MPP this ceiling is a a lot higher and more flexible.
You can also replace a dual core dual CPU nodes with a 6 core AMD Istanbul CPUs next year.. and possibly 12 core CPUs year after that. So even a smallish 4 node cluster with 16 cores in total can be grown significantly and remain a 4 node cluster. Together with advances in HPC (High Performance Computing) like Infiniband.
I'm not seeing much use of non-RAC RDBMS architecture in the future. Databases are getting ever bigger because we have the technology to crunch more data, and crunch it a lot more intelligently than ever before. My first production database was 4MB in size, and ran on a Novell File Server with two 20MB disks. I'm currently testing a 24TB array for use for a single database.
Technology is inevitable, as is the growth in data volumes. And I cannot see a non-RAC architecture rising to that challenge. Especially not in something like data warehousing. -
No data in Application failure Analysis report for a particular application alone.
Dear All,
We are not able to get the data in Application Failure Analysis report from Appadvisor for a particular application(say ABC) alone, whereas it works fine for different application. Also,we could get the data from other reports(such
as Application status,Problem distribution analysis,etc) for the same application(ABC). PFB snapshot for your reference.
Thanks in Advance.
Regards,
Rajesh Kumar CHi,
Is there any error message when you run Application Failure Analysis report for the specific application?
Does this application have any failure alerts?
You may also look into operation manager even logs to check is there any errors or warnings.
Regards,
Yan Li
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected] -
No Data to display in Application failure report for a specific Application
Hi ,
For a specific application,Application failure report showing no data to display but for the same application other reports shows value.
Application failure reports seems to work fine for some other application except few.Need to know the root cause.
Kindly help us to sort this issue.
Thanks & Regards,
Preethi SHi,
Is there any error message when you run Application Failure Analysis report for the specific application?
Does this application have any failure alerts?
You may also look into operation manager even logs to check is there any errors or warnings.
Regards,
Yan Li
Please remember to mark the replies as answers if they help and unmark them if they provide no help. If you have feedback for TechNet Subscriber Support, contact [email protected] -
Issue in confirmation links for personal data ESS application (CA)
Hi All,
We have implemented EXIT button configuration for ESS applications and the Exit button along with 'Go to Home page' link is navigating to custom created page in portal.
The configuration done earlier for ESS applications like Address Update, Family Member/Dependents works fine.
We got a new service added (Personal data) & tried configuring the Exit button for this application also.
Some how the exit button doesnt seem working for Canada besides it working for US.
Please help in figuring out if there is any cause of the Exit button not working for personal data application for molga 07 specially. Adding to it, the links for confirmation screen are also not working for the same application
Appreciate your help.
ThanksHi,
As per your Req.....
Its seems to be the Services exits that are county specific Services .........mease the Molga ( country Code ) is set for this services is "US"
That Y its working fine .....
But as per your REq you want this Services working for the Molga 07 ...
I thikn for this you need to set the Molga 07 in the contry specific home pages......
Then it will work fine -
Attached document for data column in BPS Web application
Hi,
I have created layout for a BPS application, the layout has been configured to have the ability to allow the used to create the attached documentd for the data column. when the user open the layout through the planning profile, the icon for those documents is displayed and can be open by click those icons. After we generated the Web applicatios, those icons are gone and the user can not open them any more.
I notice that there is document subcomponent under BPS-WB, I added that subcomponent to my web layout, it still not work.Anybody know how to display the attached documents on the Web, Is this doable?Hi Marc,
I did exactly what you mention in your responce e-mail, but it still not work coreectly.
following is my propperty configuration for layout and documents:
Planning layout (layout_lay01)
Only ready for output: True
Row/Cell Selection: Cell
Event on selection: true
Document on planning layout (Document1)
Layout: = layout_lay01
Only ready for output: false
When I launch the application through the Tcode BPS_WIF0
There is no document icon display for each data columns which can been in UPSPL. Instead there is a generic document input/display space under the lauout,however there is no any document display within the box, and I can not input anything within the document box. I changes some of the property for the layout and document, but it not work.
Did I forget some thing? -
Oracle version of the Microsoft Application Blocks for Data Access
Where can I find an Oracle version of the Microsoft Application Blocks for Data Access? Is it tied to any specific version of the Oracle Client software? I currently have version 8.1.7.3 installed.
I think that DAAB 3.0 only has support for ODBC and OLEDB, which indeed can be used for Oracle. The Nile 3.0 program has an OracleHelper file, but this one uses the Microsoft Dataprovider for Oracle RDBMS. With some minor changes you can make this version work with the ODP.NET that Oracle provides.
The issue I'm trying to convey is, that it seems hard to use the Microsoft Application Blocks using ODP.NET. As I described earlier, the LOBs and REF Cursors are reference types that actually point to data in the Database. This means, that the Connection to the database must be open to get to the Data.
In the DataHelper, you open the Connection and get the data you need. The DataHelper should then clean up and close the Connection. Then you pass the data to the caller. However, since we're dealing with Reference types, a closed connection means no data.
So I guess the only way to make this easy to use and not having to assume the caller/developer will open and close connections, you must copy the data in an instance of a DataSet, before you close the connection and then pass the Dataset back to the caller.
I wonder if people have delt with this issue and what their solution looks like.
thnx -
I'd like to move an on-premesis SQL Server Database to SQL Azure. I've used SQL Mgmt Studio to Extract Data Tier Application and save my db as a dacpac file. Now I'm connected to my Azure server and I've chosen to Deploy Data Tier Application. I select my
dacpac and the deploy starts but then on the last step "Registering metadata for database" it times out. I've tried it a couple of times and each time the deployed database is there and appears to be fully populated, but I'm not sure if I can ignore
that error and continue. What is supposed to happen in that step, and should I expect it to fail when deploying to SQL Azure?
I'm following the steps here http://msdn.microsoft.com/en-us/library/hh694043.aspx in the Using Migration Tools > Data-tier Application DAC Package section, except that to deploy there's no SQL Mgmt Studio > Object Explorer [server]
> Management >"Data Tier Applications" node, so I'm deploying by right-clicking on the server name and choosing "Deploy Data-tier Application".
My (total) guess here is that it's deployed the database fine and it's doing whatever magic happens when you register a data tier application, except that it's not working for SQL Azure.
I'm running against a server created for the new Azure service tiers, not against a Web/Business edition server.
The full details of error I get are below.
thanks,
Rory
TITLE: Microsoft SQL Server Management Studio
Could not deploy package.
Warning SQL0: A project which specifies SQL Server 2008 as the target platform may experience compatibility issues with SQL Azure.
(Microsoft.SqlServer.Dac)
ADDITIONAL INFORMATION:
Unable to register data-tier application: Unable to reconnect to database: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. (Microsoft.Data.Tools.Schema.Sql)
Unable to reconnect to database: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. (Microsoft.Data.Tools.Schema.Sql)
Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding. (Microsoft SQL Server, Error: -2)
For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft%20SQL%20Server&ProdVer=11.00.9213&EvtSrc=MSSQLServer&EvtID=-2&LinkId=20476
The wait operation timed out
BUTTONS:
OKHello,
The registration process creates a DAC definition that defines the objects in the database, and register the DAC definition in the master system database in Windows Azure SQL Database.
Based on the error message, there is timeout error when connect to SQL Database. Did you deploy a large database? When move large data to Azure SQL Database, it is recommended to use SQL Server Integration Services (SSIS) and BCP utility.
Or you can try to create a client application with the Data-Tier Application Framework (DACFx) client tool to import database and handle connection loss it by re-establish the connection.
Reference: http://sqldacexamples.codeplex.com/
Regards,
Fanny Liu
If you have any feedback on our support, please click here.
Fanny Liu
TechNet Community Support
Maybe you are looking for
-
What program do I use to burn from iMovie to a DVD?
I don't have iDVD and I can't seem to find it for purchase on the App Store. I need to burn a created project on iMovie to a blank DVD.
-
CS3 JS Running an JS within an Action?
I realize that I can create a script that will through an alert as shown below and can be created as an action. So the process would be to run the action and the script will display the alert. If the user clicks the Yes button the script would run th
-
Can Apple TV fade from one music title to the next?
Playing my mysic with iTunes I can set it to fade softly from one title to the next, without pausing between the tracks. I can't find this setting in my new Apple TV and the Apple TV seems unaffected by my settings in my iTunes. How can I tell my App
-
Creating custom scopes in SharePoint 2013
Guys, Environment : SharePoint 2013 I want to provide the user with the ability to search from a custom search scope which has been deprecated in sharepoint 2013. How can i do this in 2013, also i would like to have the custom scopes in a dropdown ne
-
The download was complete and successful, but when i go to extract the file i get this error messages in succession I have plenty of disc space, and i have also tried to redownload twice... same issue Anyone help???