External Table Performance and Sizing
Hi,
Can anyone tell me anything about best practices for external tables?
I have an application that writes structured log data to flat files. The size of these files can be configured and when the size limit is reached, they are rolled over. The data itself is queriable via an external table in oracle. Every so often the data is migrated (materialized) to a normal database table so it can be indexed, etc. and to keep the external file size down.
My questions are:
<ol><li> is there an optimum file size for an external table (overall size / number of rows) - by that, I suppose I mean, is there a limit where performance degrades significantly rather than constantly?
</li>
<li>is it better to have one large file mapped to the external table or multiple smaller ones mapped to the same table? e.g. does oracle do some parallel work on multiple smaller files at the same time which might improve things?
</li>
</ol>
If there are any resources discussing these issues, that would be great - or if there is any performance data for external tables in this respect, I would love to see it.
Many thanks,
Dave
Hi Dave
is there an optimum file size for an external table (overall size / number of rows) - by
that, I suppose I mean, is there a limit where performance degrades significantly rather
than constantly?AFAIK there is no such limit. In other words, access time is proportional to the size (number of rows).
is it better to have one large file mapped to the external table or multiple smaller ones
mapped to the same table? e.g. does oracle do some parallel work on multiple smaller
files at the same time which might improve things?The DOP of a parallel query on an external table is limited by the number of files. Therefore, to use parallel processing, more than one file is needed.
HTH
Chris Antognini
Troubleshooting Oracle Performance, Apress 2008
http://top.antognini.ch
Similar Messages
-
Issue in External Table Authentication and Authorization in OBIEE11G
Hello Gurus,
Can anyone help me how to configure External Table Authentication and Authorization in OBIEE11g through weblogic server not like in 10g style(Through INIT Blocks).
I've followed the (Doc ID 1338007.1) document. But when i'm restart the Managed servers and Admin servers after configuring the SQLAuthenticator all my services are showing down.
I already raised the SR (SR 3-6286054151) on this issue. But still i didn't get any reply from them.
Can anyone help me out on this issue or can anyone me send the document for "how to configure External Table Authentication and Authorization in OBIEE11g" . It's really appreciate for your quick response.
my mail ID [email protected]
Thanks,
Syam.
Edited by: 942658 on Oct 13, 2012 10:55 AMHi John,
Thanks for your quick response.
We configured "ReadOnlySQL Provider" by following the Oracle's white paper(Doc ID 1338007.1) Please find the below steps what we configured in weblogic console.
1. Created the Data Source
2. In the data source specified the Database driver--> *Oracle's Driver Thin for service connections: Versions:9.0.1 and later.
3. Defined the connection Properties .
4. Selected targets as Admin server and bi_server.
Then Activate changes
5. Created new provider by using ReadOnlySQL Authenticator
6. In the provider specific tab we given the SQL statements and saved it.
7. Restarted the Admin and Managed servers.
After restarted the services when we open the Enterprise Manager page all the services are showed as Undefined - means red.
Apart from that we followed your suggested link http://askjohnobiee.blogspot.com/2012/09/how-to-oid-authentication-with-groups.html
For External table authentication do we need to configure BISQLAuthenticator or ReadOnlySQLAuthenticator ?
If we configure BISQLAuthenticator we just import Groups from database to Console application. Then how can it Authenticated to the User ?
Please let me know your ideas on this.
Thanks,
Syam -
Need performance and sizing document
Hi Experts,
Pls provide me some materials regarding performance and sizing topic in Input Enabled Queries and planning applications Competency to know abt it .BW Query Performance
Query Executime time ?
Precalculated Value Set
BW Performance Tuning Knowledge Center - SAP Developer Network (SDN)
Business Intelligence Performance Tuning
performance docs on query
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/3f66ba90-0201-0010-ac8d-b61d8fd9abe9
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/cccad390-0201-0010-5093-fd9ec8157802
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/ce7fb368-0601-0010-64ba-fadc985a1f94
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/c8c4d794-0501-0010-a693-918a17e663cc
https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/064fed90-0201-0010-13ae-b16fa4dab695
weblog
Query Creation Checklist
Query Optimization
https://www.sdn.sap.com/irj/sdn/directforumsearch?threadid=&q=cube+size&objid=c4&daterange=all&numresults=15
cube size
Message was edited by:
hari kv -
Unload data with DATAPUMP external tables performance
Hi,
I archive some tables with external tables ORACLE_DATAPUMP.
How can I increase the performance of the table creation?
I try PARALLEL, and I want to know if some others parameters or init parameters can increase the performance.
Thanks for your feedback.
ElodieHi,
take a look here:
http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14215/dp_perf.htm
You will find some hints.
Acr -
External table log and bad files
Hi, i have defined the following access parameters for my external table and set REJECT LIMIT to 0:
ACCESS PARAMETERS
RECORDS DELIMITED by NEWLINE
BADFILE BAD_DIR:'CARDS.bad'
LOGFILE LOG_DIR:'CARDS.log'
NODISCARDFILE
FIELDS TERMINATED BY ","
OPTIONALLY ENCLOSED BY '"'
READSIZE 1048576
LRTRIM
MISSING FIELD VALUES ARE NULL
REJECT ROWS WITH ALL NULL FIELDS
I want to know everytime i query the external table, my log file and bad file will be overwritten or appended? i want them to be overwritten.Hi,
Yeah, well, be in for a surprise here:
"The external tables feature is a complement to existing SQL*Loader functionality."
http://download.oracle.com/docs/cd/B19306_01/server.102/b14220/utility.htm#sthref1800
http://www.oracle.com/pls/db102/search?remark=quick_search&word=external+table&tab_id=&format=ranked
Have you actually read/tried anything?
You can download an XE DB for free and play with that. -
Access Manager FAQ on performance and sizing, policy, et al
Check out the ninth segment of the Sun Java System Access Manager FAQ on Sun Developer Network at http://developers.sun.com/identity/overview/faq/perf-policy-failover-agents.jsp. These latest Q&As focus on performance and policy, session failover, deployment, and agents, with tips and guidelines for configurations and best practices.
Hi Steve,
Of the three components relevent in this case :
1) the core server itself.
2) the webapplication - also referred to as client resources.
3) Multiplexors.
The webapplication which gets hosted on the app/web server - contains only static information.
That is, it has the client jars, jnlp's for fetching these, a few jsp and launcher html files for endusers to launch the im client over webstart.
So, this webclient can be moved to any webcontainer of your choice - and has no restrictions or dependencies on anything.
Just install the client resources on any server of your choice (which is accessible by end users) and configure/deploy it to your webserver - it needs no access to the backend in any way at all.
The IM server uses the access manager sdk (amsdk) to talk to access manager - hence, you will need to modify AMConfig.properties appropriately in case you are making the deployment changes after installation and configuration of IM and amsdk on IM server (please refer to IM and AM documentation for more info on how to do this).
Since what you are attempting looks like a AM supported config, you should not have any issues in configuring IM with that for auth and policy enforcement.
The multiplexor is totally oblivious to all this - just needs to be accessible by clients and should be able to talk to server - thats all.
Hope this helps.
Regards,
Mridul -
Light Table: aligning and sizing photos
I am looking for a comfortable possibility to make single photo presentation sheets to give them to my portrait clients. One sheet should contain several photos in different sizes. Until yet I use the photo book layouts - but this is not a really good solution as it is annoying to make a book when I just need a single sheet with some aligned photos in different sizes on it. Aligning position and sizes of photos on the light table does not work really fine.
any ideas how I could solve my problem?I am looking for a comfortable possibility to make
single photo presentation sheets to give them to my
portrait clients. One sheet should contain several
photos in different sizes. Until yet I use the photo
book layouts - but this is not a really good solution
as it is annoying to make a book when I just need a
single sheet with some aligned photos in different
sizes on it. Aligning position and sizes of photos on
the light table does not work really fine.
any ideas how I could solve my problem?
Well, there are a number of alignment options in the light table when you Control-Click (right click) on an image - you can select multiple images on the light table holding down Shift or Command, and then use the context menu option to align or redistribute images.
ABout the only thing that is not there is to make two images exactly the same size, for that you'll need to make use of the grid - if you zoom in it should be easier to get two images almost exactly the same size. -
Error while selecting date from external table
Hello all,
I am getting the follwing error while selecting data from external table. Any idea why?
SQL> CREATE TABLE SE2_EXT (SE_REF_NO VARCHAR2(255),
2 SE_CUST_ID NUMBER(38),
3 SE_TRAN_AMT_LCY FLOAT(126),
4 SE_REVERSAL_MARKER VARCHAR2(255))
5 ORGANIZATION EXTERNAL (
6 TYPE ORACLE_LOADER
7 DEFAULT DIRECTORY ext_tables
8 ACCESS PARAMETERS (
9 RECORDS DELIMITED BY NEWLINE
10 FIELDS TERMINATED BY ','
11 MISSING FIELD VALUES ARE NULL
12 (
13 country_code CHAR(5),
14 country_name CHAR(50),
15 country_language CHAR(50)
16 )
17 )
18 LOCATION ('SE2.csv')
19 )
20 PARALLEL 5
21 REJECT LIMIT UNLIMITED;
Table created.
SQL> select * from se2_ext;
SQL> select count(*) from se2_ext;
select count(*) from se2_ext
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04043: table column not found in external source: SE_REF_NO
ORA-06512: at "SYS.ORACLE_LOADER", line 19It would appear that you external table definition and the external data file data do not match up. Post a few input records so someone can duplicate the problem and determine the fix.
HTH -- Mark D Powell -- -
External Table Authentication - Not Able to Login to Presentation services
Hi ,
I am trying to setup External table Authentication and in Rpd file I have setup session Variable as described in the OBIEE Server Admin Guide (http://download.oracle.com/docs/cd/E10415_01/doc/bi.1013/b31770.pdf -- Page 326 ) ..
But when I try to logon to Presentation services Iget the following error.
State: 08004. Code: 10018. [NQODBC] [SQL_STATE: 08004] [nQSError: 10018] Access for the requested connection is refused. [nQSError: 13024] Successful completion of init block 'TableLDAP' is required. (08004)
what am I doing wrong ? Please adivce.
Thanks
SSHello,
I Have setup system session User and group variables in a Initialazation block. No LDAP is tied. I have just created a table and added bunch of users and their passwords and their groups they belongs to.When I test my initialization block in rpd by suppling Uid/Pwd I see correct group they belongs to.
But When I try in Presentation layer it doesn't work and throwing same error I mentioned.I tried various things but no luck .Not sure why intialization block is not firing off.
Thanks
SS -
IGNORE COMMENTS IN EXTERNAL TABLE FILE
I currently have a partitioned table, split by month going back some 7 years.
The business have now agreed to archive off some of the data but to have it
available if required. I therefore intend to select month by month, the data
to a flat file and make use of external tables to provide the data for the
business if required. Then to remove the months partition thus releaseing
space back to the database. Each months data is only about 100bytes long, but
there could be about 15million rows.
I can successfully create the external table and read the data.
However I would like to place a header in the spooled file but the only method
I have found to ignore the header is to use the "SKIP" parameter. I would
rather not use this as it hard codes the number of lines I reserve for a
header.
Is there another method to add a comment to a external tables file and have the
selects on the file ignore the comments?
Could I use "LOAD WHEN ( column1 != '--' )" as each comment line begins with a
double dash?
Also, is there a limit to the size of an external tables file?When I added the lines to attempt to
exclude the Header lines to be:
RECORDS DELIMITED BY newline
NOBADFILE
NODISCARDFILE
NOLOGFILE
SKIP 0
LOAD WHEN ( cvt_seq_num LIKE '--%' )
FIELDS
TERMINATED BY ","
OPTIONALLY ENCLOSED BY '"'
MISSING FIELD VALUES ARE NULL
REJECT ROWS WITH ALL NULL FIELDS
( cvt_seq_num INTEGER EXTERNAL(12)
I got:
select * from old_cvt_external
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPENcallout
ORA-29400: data cartridge error
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "identifier": expecting one of: "equal,notequal"
KUP-01008: the bad identifier was: LIKE
KUP-01007: at line 6 column 33
ORA-06512: at "SYS.ORACLE_LOADER", line 14
ORA-06512: at line 1
When I moved the line I got:
RECORDS DELIMITED BY newline
NOBADFILE
NODISCARDFILE
NOLOGFILE
SKIP 0
FIELDS
TERMINATED BY ","
OPTIONALLY ENCLOSED BY '"'
MISSING FIELD VALUES ARE NULL
REJECT ROWS WITH ALL NULL FIELDS
LOAD WHEN ( cvt_seq_num LIKE '--%' )
I got:
select * from old_cvt_external
ERROR at line 1:
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-00554: error encountered while parsing access parameters
KUP-01005: syntax error: found "load": expecting one of: "exit, ("
KUP-01007: at line 11 column 11
ORA-06512: at "SYS.ORACLE_LOADER", line 14
ORA-06512: at line 1
So it appears that the "LOAD WHEN" does not work.
Can you please suggest anything else? -
Hi,
Oracle Version:10.2.0.1
Operating system:Linux
Here i created the external table successfully and when fetching the records for the text file i am getting some problem.
Here is my external table creation syntax.
create table sample_ext1 (DETAIL_SEQ_NUM number(15,0),
REPAY_IDENTIFIER varchar(20),
REPAY_STATUS varchar(20),
VISA_CODE VARCHAR2(30),
NCN_CODE VARCHAR2(10 ),
AUTH_NUMBER VARCHAR2(10 ),
EXTENDED_STATUS VARCHAR2(20 )
organization external
( default directory APD_EXTRACTS
access parameters( RECORDS DELIMITED BY NEWLINE
FIELDS TERMINATED BY 0X'09'
MISSING FIELD VALUES ARE NULL) location ('sample.txt'));
and my sample.txt file data is
1000 james anderson 51 3 630-033 75188553
1001 james anderson 52 3 630-034 75188554
1002 james anderson 53 3 630-035 75188555
1003 james anderson 54 3 629-959 75188556
1004 james anderson 50 2 75188552
1005 james anderson 55 2 75188557
1006 james anderson 56 2 75188558
1007 james anderson 57 2 75188559
1008 james anderson 58 2 75188560
1009 james anderson 59 2 75188561and the output is
1000 james anderson 51 3 630-033 75188553
1001 james anderson 52 3 630-034 75188554
1002 james anderson 53 3 630-035 75188555
1003 james anderson 54 3 629-959 75188556
1004 james anderson 50 2
1005 james anderson 55 2
1006 james anderson 56 2
1007 james anderson 57 2
1008 james anderson 58 2
1009 james anderson 59 2 here my problem is in the sixth column i am having some null values in sample.txt file as shown above but in the out put i am getting null values in last column also but there is data .
Please help me how to solve my problem.
Thanks & Regards,
Poorna Prasad.Hi All,
My problem was solved her what i did wrong is at first i made some changer manually to my sample.txt file.But after loading the original fiile as it is with editing it i get the correct output .
Thanks & Regards,
Poorna Prasad. -
Problem in external table & sql*loader
hi all
i want to transter data between flat file to oracle database through external table option and sql*loader Utility.
1. how u make .dat file in external table.
2. how u make .ctl file in sql*loader utility.
if u know any good site plz. tell me.
thanx
Mohammadi52Hi,
Use this link to search any Oracle documentation:
http://tahiti.oracle.com/
Also please have a look on the below link:
http://download-uk.oracle.com/docs/cd/B19306_01/server.102/b14220/utility.htm
Regards -
Hi,
I have created an external table. And it has been run successfully. When i view the log file i am not able to find out the counts like
no of records loaded,
no of records rejected.
Kindly advice me on this.
Cheers
Ramkannan.AHi,
ORGANIZATION EXTERNAL
( type oracle_loader
default directory PARAGON_FEED_DIR
access parameters
records delimited by newline skip 1
BADFILE PARAGON_FEED_LOG:'pgon_mapunmap.bad'
LOGFILE PARAGON_FEED_LOG:'pgon_mapunmap.log'
date_cache = 650000
STRING SIZES ARE IN CHARACTERS
CHARACTERSET = WE8ISO8859P1
fields terminated by "," OPTIONALLY ENCLOSED BY '"' LDRTRIM
missing field values are null
In the Log file
Field Definitions for table PGON_MAPUNMAP_LOAD
Record format DELIMITED BY NEWLINE
Data in file has same endianness as the platform
Rows with all null fields are accepted
Fields in Data Source:
SOURCESYSTEMNAME CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SOURCESYSTEMLOCATION CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SOURCEID CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SOURCESHORT CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SOURCELEGAL CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
SOURCECOUNTRY CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ORGID CHAR (255)
Terminated by ","
Enclosed by """ and """
Trim whitespace same as SQL Loader
ORGSHORT CHAR (255)
I want to know how many records are loaded & how many records are rejected in Log file.
But i am not able to see that -
IGNORE COMMENTS IN EXTERNAL TABLE
I currently have a partitioned table, split by month going back some 7 years.
The business have now agreed to archive off some of the data but to have it
available if required. I therefore intend to select month by month, the data
to a flat file and make use of external tables to provide the data for the
business if required. Then to remove the months partition thus releaseing
space back to the database. Each months data is only about 100bytes long, but
there could be about 15million rows.
I can successfully create the external table and read the data.
However I would like to place a header in the spooled file but the only method
I have found to ignore the header is to use the "SKIP" parameter. I would
rather not use this as it hard codes the number of lines I reserve for a
header.
Is there another method to add a comment to a external tables file and have the
selects on the file ignore the comments?
Could I use "LOAD WHEN ( column1 != '--' )" as each comment line begins with a
double dash?
Also, is there a limit to the size of an external tables file?
Thanks
JDHi Gurus..
any help will be highly appreciated. -
Helllo Everyone.
I am new to external tables concept and I have only used using flat files.
Is there a complete reference on on external tables somewhere?
Also, instead of .csv or text based datafiles, can I use XML as data source for these external tables? If so, how?
Thanks,
RThe link to the documentation has been already posted.
About the xml as a possible datasource, in Oracle9i only the ORACLE_LOADER driver is supported, it allows you to read data using SQL*Loader, that is to say only plain text files.
In Oracle10g a new driver has been added: ORACLE_DATAPUMP, it allows you to read with an external table a dump file created from other Oracle tables using the same ORACLE_DATAPUMP driver.
Unfortunately the ORACLE_DATAPUMP driver doesn't support the XMLTYPE datatype:
SQL> desc tbl_xml
Nome Nullo? Tipo
XML PUBLIC.XMLTYPE
SQL> CREATE TABLE xml_ext
2 ORGANIZATION EXTERNAL
3 (
4 TYPE ORACLE_DATAPUMP
5 DEFAULT DIRECTORY DATA_PUMP_DIR
6 LOCATION ('xml_ext.dmp')
7 )
8 AS SELECT * FROM tbl_xml;
CREATE TABLE xml_ext
ERRORE alla riga 1:
ORA-30656: tipo di colonna non supportato in una tabella organizzata esterna
SQL>Max
[My Italian Oracle blog|http://oracleitalia.wordpress.com/2009/12/18/table-elimination-oppure-join-elimination-lottimizzatore-si-libera-della-zavorra/]
Maybe you are looking for
-
Hi All, I am using BI report for plan and actual cost deviation of Work order(maintenance order) of PM(tcode iw33>cost tab>plan actual cost to check), question there is 2 query for this plan actual cost deviation, one on 0pm_c01(Maintenance Orders -
-
Hi, I'm hoping someone can help me find a solution to my problem. When a backup is attempted, it starts off as if it's working, then stops and the following error message appears: "iTunes could not backup iPhone because the iPhone disconnected". But,
-
I have a iphone 4s with ios 8.3 and I used to be able to sync only a week or two of emails from outlook email. Now it won't let me do that and it syncs all my inbox. How can I change it back to sync less. I can't find the right place.
-
Connecting exchange to hp eprint
I am unable to connect my work outlook account to eprint on my ipad. When i go to add email account. I can enter my email address, domain, password and hostname. There is no place to enter my username that is required for my serve to accept. Please a
-
How to display currency values in indian format
hi all, When I am displaying Currency values as output , those are displaying in U.S. format (ie.1,234,000.00) , But I need to display those in Indian Rupee format (i.e 12,34,000.00). Plz any one can help me that how to display this thank you, reg