External Table Performance and Sizing

Hi,
Can anyone tell me anything about best practices for external tables?
I have an application that writes structured log data to flat files. The size of these files can be configured and when the size limit is reached, they are rolled over. The data itself is queriable via an external table in oracle. Every so often the data is migrated (materialized) to a normal database table so it can be indexed, etc. and to keep the external file size down.
My questions are:
<ol><li>     is there an optimum file size for an external table (overall size / number of rows) - by that, I suppose I mean, is there a limit where performance degrades significantly rather than constantly?
</li>
<li>is it better to have one large file mapped to the external table or multiple smaller ones mapped to the same table? e.g. does oracle do some parallel work on multiple smaller files at the same time which might improve things?
</li>
</ol>
If there are any resources discussing these issues, that would be great - or if there is any performance data for external tables in this respect, I would love to see it.
Many thanks,
Dave

Hi Dave
is there an optimum file size for an external table (overall size / number of rows) - by
that, I suppose I mean, is there a limit where performance degrades significantly rather
than constantly?AFAIK there is no such limit. In other words, access time is proportional to the size (number of rows).
is it better to have one large file mapped to the external table or multiple smaller ones
mapped to the same table? e.g. does oracle do some parallel work on multiple smaller
files at the same time which might improve things?The DOP of a parallel query on an external table is limited by the number of files. Therefore, to use parallel processing, more than one file is needed.
HTH
Chris Antognini
Troubleshooting Oracle Performance, Apress 2008
http://top.antognini.ch

Similar Messages

  • Issue in External Table Authentication and Authorization in OBIEE11G

    Hello Gurus,
    Can anyone help me how to configure External Table Authentication and Authorization in OBIEE11g through weblogic server not like in 10g style(Through INIT Blocks).
    I've followed the (Doc ID 1338007.1) document. But when i'm restart the Managed servers and Admin servers after configuring the SQLAuthenticator all my services are showing down.
    I already raised the SR (SR 3-6286054151) on this issue. But still i didn't get any reply from them.
    Can anyone help me out on this issue or can anyone me send the document for "how to configure External Table Authentication and Authorization in OBIEE11g" . It's really appreciate for your quick response.
    my mail ID [email protected]
    Thanks,
    Syam.
    Edited by: 942658 on Oct 13, 2012 10:55 AM

    Hi John,
    Thanks for your quick response.
    We configured "ReadOnlySQL Provider" by following the Oracle's white paper(Doc ID 1338007.1) Please find the below steps what we configured in weblogic console.
    1. Created the Data Source
    2. In the data source specified the Database driver--> *Oracle's Driver Thin for service connections: Versions:9.0.1 and later.
    3. Defined the connection Properties .
    4. Selected targets as Admin server and bi_server.
    Then Activate changes
    5. Created new provider by using ReadOnlySQL Authenticator
    6. In the provider specific tab we given the SQL statements and saved it.
    7. Restarted the Admin and Managed servers.
    After restarted the services when we open the Enterprise Manager page all the services are showed as Undefined - means red.
    Apart from that we followed your suggested link http://askjohnobiee.blogspot.com/2012/09/how-to-oid-authentication-with-groups.html
    For External table authentication do we need to configure BISQLAuthenticator or ReadOnlySQLAuthenticator ?
    If we configure BISQLAuthenticator we just import Groups from database to Console application. Then how can it Authenticated to the User ?
    Please let me know your ideas on this.
    Thanks,
    Syam

  • Need performance and sizing document

    Hi Experts,
      Pls provide me  some materials regarding performance and sizing topic in Input Enabled Queries and planning applications Competency to know abt it .

    BW Query Performance
    Query Executime time ?
    Precalculated Value Set
    BW Performance Tuning Knowledge Center - SAP Developer Network (SDN)
    Business Intelligence Performance Tuning
    performance docs on query
    https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/3f66ba90-0201-0010-ac8d-b61d8fd9abe9
    https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/cccad390-0201-0010-5093-fd9ec8157802
    https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/ce7fb368-0601-0010-64ba-fadc985a1f94
    https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/c8c4d794-0501-0010-a693-918a17e663cc
    https://www.sdn.sap.com/irj/servlet/prt/portal/prtroot/docs/library/uuid/064fed90-0201-0010-13ae-b16fa4dab695
    weblog
    Query Creation Checklist
    Query Optimization
    https://www.sdn.sap.com/irj/sdn/directforumsearch?threadid=&q=cube+size&objid=c4&daterange=all&numresults=15
    cube size
    Message was edited by:
            hari kv

  • Unload data with DATAPUMP external tables performance

    Hi,
    I archive some tables with external tables ORACLE_DATAPUMP.
    How can I increase the performance of the table creation?
    I try PARALLEL, and I want to know if some others parameters or init parameters can increase the performance.
    Thanks for your feedback.
    Elodie

    Hi,
    take a look here:
    http://download-east.oracle.com/docs/cd/B19306_01/server.102/b14215/dp_perf.htm
    You will find some hints.
    Acr

  • External table log and bad files

    Hi, i have defined the following access parameters for my external table and set REJECT LIMIT to 0:
    ACCESS PARAMETERS
    RECORDS DELIMITED by NEWLINE
    BADFILE BAD_DIR:'CARDS.bad'
    LOGFILE LOG_DIR:'CARDS.log'
    NODISCARDFILE
    FIELDS TERMINATED BY ","
    OPTIONALLY ENCLOSED BY '"'
    READSIZE 1048576
    LRTRIM
    MISSING FIELD VALUES ARE NULL
    REJECT ROWS WITH ALL NULL FIELDS
    I want to know everytime i query the external table, my log file and bad file will be overwritten or appended? i want them to be overwritten.

    Hi,
    Yeah, well, be in for a surprise here:
    "The external tables feature is a complement to existing SQL*Loader functionality."
    http://download.oracle.com/docs/cd/B19306_01/server.102/b14220/utility.htm#sthref1800
    http://www.oracle.com/pls/db102/search?remark=quick_search&word=external+table&tab_id=&format=ranked
    Have you actually read/tried anything?
    You can download an XE DB for free and play with that.

  • Access Manager FAQ on performance and sizing, policy, et al

    Check out the ninth segment of the Sun Java System Access Manager FAQ on Sun Developer Network at http://developers.sun.com/identity/overview/faq/perf-policy-failover-agents.jsp. These latest Q&As focus on performance and policy, session failover, deployment, and agents, with tips and guidelines for configurations and best practices.

    Hi Steve,
    Of the three components relevent in this case :
    1) the core server itself.
    2) the webapplication - also referred to as client resources.
    3) Multiplexors.
    The webapplication which gets hosted on the app/web server - contains only static information.
    That is, it has the client jars, jnlp's for fetching these, a few jsp and launcher html files for endusers to launch the im client over webstart.
    So, this webclient can be moved to any webcontainer of your choice - and has no restrictions or dependencies on anything.
    Just install the client resources on any server of your choice (which is accessible by end users) and configure/deploy it to your webserver - it needs no access to the backend in any way at all.
    The IM server uses the access manager sdk (amsdk) to talk to access manager - hence, you will need to modify AMConfig.properties appropriately in case you are making the deployment changes after installation and configuration of IM and amsdk on IM server (please refer to IM and AM documentation for more info on how to do this).
    Since what you are attempting looks like a AM supported config, you should not have any issues in configuring IM with that for auth and policy enforcement.
    The multiplexor is totally oblivious to all this - just needs to be accessible by clients and should be able to talk to server - thats all.
    Hope this helps.
    Regards,
    Mridul

  • Light Table: aligning and sizing photos

    I am looking for a comfortable possibility to make single photo presentation sheets to give them to my portrait clients. One sheet should contain several photos in different sizes. Until yet I use the photo book layouts - but this is not a really good solution as it is annoying to make a book when I just need a single sheet with some aligned photos in different sizes on it. Aligning position and sizes of photos on the light table does not work really fine.
    any ideas how I could solve my problem?

    I am looking for a comfortable possibility to make
    single photo presentation sheets to give them to my
    portrait clients. One sheet should contain several
    photos in different sizes. Until yet I use the photo
    book layouts - but this is not a really good solution
    as it is annoying to make a book when I just need a
    single sheet with some aligned photos in different
    sizes on it. Aligning position and sizes of photos on
    the light table does not work really fine.
    any ideas how I could solve my problem?
    Well, there are a number of alignment options in the light table when you Control-Click (right click) on an image - you can select multiple images on the light table holding down Shift or Command, and then use the context menu option to align or redistribute images.
    ABout the only thing that is not there is to make two images exactly the same size, for that you'll need to make use of the grid - if you zoom in it should be easier to get two images almost exactly the same size.

  • Error while selecting date from external table

    Hello all,
    I am getting the follwing error while selecting data from external table. Any idea why?
    SQL> CREATE TABLE SE2_EXT (SE_REF_NO VARCHAR2(255),
      2        SE_CUST_ID NUMBER(38),
      3        SE_TRAN_AMT_LCY FLOAT(126),
      4        SE_REVERSAL_MARKER VARCHAR2(255))
      5  ORGANIZATION EXTERNAL (
      6    TYPE ORACLE_LOADER
      7    DEFAULT DIRECTORY ext_tables
      8    ACCESS PARAMETERS (
      9      RECORDS DELIMITED BY NEWLINE
    10      FIELDS TERMINATED BY ','
    11      MISSING FIELD VALUES ARE NULL
    12      (
    13        country_code      CHAR(5),
    14        country_name      CHAR(50),
    15        country_language  CHAR(50)
    16      )
    17    )
    18    LOCATION ('SE2.csv')
    19  )
    20  PARALLEL 5
    21  REJECT LIMIT UNLIMITED;
    Table created.
    SQL> select * from se2_ext;
    SQL> select count(*) from se2_ext;
    select count(*) from se2_ext
    ERROR at line 1:
    ORA-29913: error in executing ODCIEXTTABLEOPEN callout
    ORA-29400: data cartridge error
    KUP-04043: table column not found in external source: SE_REF_NO
    ORA-06512: at "SYS.ORACLE_LOADER", line 19

    It would appear that you external table definition and the external data file data do not match up. Post a few input records so someone can duplicate the problem and determine the fix.
    HTH -- Mark D Powell --

  • External Table Authentication - Not Able to Login to Presentation services

    Hi ,
    I am trying to setup External table Authentication and in Rpd file I have setup session Variable as described in the OBIEE Server Admin Guide (http://download.oracle.com/docs/cd/E10415_01/doc/bi.1013/b31770.pdf -- Page 326 ) ..
    But when I try to logon to Presentation services Iget the following error.
    State: 08004. Code: 10018. [NQODBC] [SQL_STATE: 08004] [nQSError: 10018] Access for the requested connection is refused. [nQSError: 13024] Successful completion of init block 'TableLDAP' is required. (08004)
    what am I doing wrong ? Please adivce.
    Thanks
    SS

    Hello,
    I Have setup system session User and group variables in a Initialazation block. No LDAP is tied. I have just created a table and added bunch of users and their passwords and their groups they belongs to.When I test my initialization block in rpd by suppling Uid/Pwd I see correct group they belongs to.
    But When I try in Presentation layer it doesn't work and throwing same error I mentioned.I tried various things but no luck .Not sure why intialization block is not firing off.
    Thanks
    SS

  • IGNORE COMMENTS IN EXTERNAL TABLE FILE

    I currently have a partitioned table, split by month going back some 7 years.
    The business have now agreed to archive off some of the data but to have it
    available if required. I therefore intend to select month by month, the data
    to a flat file and make use of external tables to provide the data for the
    business if required. Then to remove the months partition thus releaseing
    space back to the database. Each months data is only about 100bytes long, but
    there could be about 15million rows.
    I can successfully create the external table and read the data.
    However I would like to place a header in the spooled file but the only method
    I have found to ignore the header is to use the "SKIP" parameter. I would
    rather not use this as it hard codes the number of lines I reserve for a
    header.
    Is there another method to add a comment to a external tables file and have the
    selects on the file ignore the comments?
    Could I use "LOAD WHEN ( column1 != '--' )" as each comment line begins with a
    double dash?
    Also, is there a limit to the size of an external tables file?

    When I added the lines to attempt to
    exclude the Header lines to be:
    RECORDS DELIMITED BY newline
    NOBADFILE
    NODISCARDFILE
    NOLOGFILE
    SKIP 0
    LOAD WHEN ( cvt_seq_num LIKE '--%' )
    FIELDS
    TERMINATED BY ","
    OPTIONALLY ENCLOSED BY '"'
    MISSING FIELD VALUES ARE NULL
    REJECT ROWS WITH ALL NULL FIELDS
    ( cvt_seq_num INTEGER EXTERNAL(12)
    I got:
    select * from old_cvt_external
    ERROR at line 1:
    ORA-29913: error in executing ODCIEXTTABLEOPENcallout
    ORA-29400: data cartridge error
    KUP-00554: error encountered while parsing access parameters
    KUP-01005: syntax error: found "identifier": expecting one of: "equal,notequal"
    KUP-01008: the bad identifier was: LIKE
    KUP-01007: at line 6 column 33
    ORA-06512: at "SYS.ORACLE_LOADER", line 14
    ORA-06512: at line 1
    When I moved the line I got:
    RECORDS DELIMITED BY newline
    NOBADFILE
    NODISCARDFILE
    NOLOGFILE
    SKIP 0
    FIELDS
    TERMINATED BY ","
    OPTIONALLY ENCLOSED BY '"'
    MISSING FIELD VALUES ARE NULL
    REJECT ROWS WITH ALL NULL FIELDS
    LOAD WHEN ( cvt_seq_num LIKE '--%' )
    I got:
    select * from old_cvt_external
    ERROR at line 1:
    ORA-29913: error in executing ODCIEXTTABLEOPEN callout
    ORA-29400: data cartridge error
    KUP-00554: error encountered while parsing access parameters
    KUP-01005: syntax error: found "load": expecting one of: "exit, ("
    KUP-01007: at line 11 column 11
    ORA-06512: at "SYS.ORACLE_LOADER", line 14
    ORA-06512: at line 1
    So it appears that the "LOAD WHEN" does not work.
    Can you please suggest anything else?

  • Problem in external table

    Hi,
    Oracle Version:10.2.0.1
    Operating system:Linux
    Here i created the external table successfully and when fetching the records for the text file i am getting some problem.
    Here is my external table creation syntax.
    create table sample_ext1 (DETAIL_SEQ_NUM number(15,0),
    REPAY_IDENTIFIER varchar(20),
    REPAY_STATUS varchar(20),
    VISA_CODE VARCHAR2(30),
    NCN_CODE VARCHAR2(10 ),
    AUTH_NUMBER VARCHAR2(10 ),
    EXTENDED_STATUS VARCHAR2(20 )
    organization external
    ( default directory APD_EXTRACTS
      access parameters( RECORDS DELIMITED BY NEWLINE
    FIELDS TERMINATED BY 0X'09'
    MISSING FIELD VALUES ARE NULL) location ('sample.txt'));
              and my sample.txt file data is
    1000      james      anderson      51      3      630-033      75188553     
    1001      james      anderson      52      3      630-034      75188554     
    1002      james      anderson      53      3      630-035      75188555     
    1003      james      anderson      54      3      629-959      75188556     
    1004      james      anderson      50      2                75188552     
    1005      james      anderson      55      2                75188557     
    1006      james      anderson      56      2                75188558     
    1007      james      anderson      57      2                75188559     
    1008      james      anderson      58      2                75188560     
    1009      james      anderson      59      2                75188561and the output is
    1000     james      anderson      51      3      630-033      75188553
    1001     james      anderson      52      3      630-034      75188554
    1002     james      anderson      53      3      630-035      75188555
    1003     james      anderson      54      3      629-959      75188556
    1004     james      anderson      50      2           
    1005     james      anderson      55      2           
    1006     james      anderson      56      2           
    1007     james      anderson      57      2           
    1008     james      anderson      58      2           
    1009     james      anderson      59      2           here my problem is in the sixth column i am having some null values in sample.txt file as shown above but in the out put i am getting null values in last column also but there is data .
    Please help me how to solve my problem.
    Thanks & Regards,
    Poorna Prasad.

    Hi All,
    My problem was solved her what i did wrong is at first i made some changer manually to my sample.txt file.But after loading the original fiile as it is with editing it i get the correct output .
    Thanks & Regards,
    Poorna Prasad.

  • Problem in external table & sql*loader

    hi all
    i want to transter data between flat file to oracle database through external table option and sql*loader Utility.
    1. how u make .dat file in external table.
    2. how u make .ctl file in sql*loader utility.
    if u know any good site plz. tell me.
    thanx
    Mohammadi52

    Hi,
    Use this link to search any Oracle documentation:
    http://tahiti.oracle.com/
    Also please have a look on the below link:
    http://download-uk.oracle.com/docs/cd/B19306_01/server.102/b14220/utility.htm
    Regards

  • Log files  in external table

    Hi,
    I have created an external table. And it has been run successfully. When i view the log file i am not able to find out the counts like
    no of records loaded,
    no of records rejected.
    Kindly advice me on this.
    Cheers
    Ramkannan.A

    Hi,
    ORGANIZATION EXTERNAL
    ( type oracle_loader
    default directory PARAGON_FEED_DIR
    access parameters
    records delimited by newline skip 1
    BADFILE PARAGON_FEED_LOG:'pgon_mapunmap.bad'
    LOGFILE PARAGON_FEED_LOG:'pgon_mapunmap.log'
    date_cache = 650000
    STRING SIZES ARE IN CHARACTERS
    CHARACTERSET = WE8ISO8859P1
    fields terminated by "," OPTIONALLY ENCLOSED BY '"' LDRTRIM
    missing field values are null
    In the Log file
    Field Definitions for table PGON_MAPUNMAP_LOAD
    Record format DELIMITED BY NEWLINE
    Data in file has same endianness as the platform
    Rows with all null fields are accepted
    Fields in Data Source:
    SOURCESYSTEMNAME CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    SOURCESYSTEMLOCATION CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    SOURCEID CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    SOURCESHORT CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    SOURCELEGAL CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    SOURCECOUNTRY CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    ORGID CHAR (255)
    Terminated by ","
    Enclosed by """ and """
    Trim whitespace same as SQL Loader
    ORGSHORT CHAR (255)
    I want to know how many records are loaded & how many records are rejected in Log file.
    But i am not able to see that

  • IGNORE COMMENTS IN EXTERNAL TABLE

    I currently have a partitioned table, split by month going back some 7 years.
    The business have now agreed to archive off some of the data but to have it
    available if required. I therefore intend to select month by month, the data
    to a flat file and make use of external tables to provide the data for the
    business if required. Then to remove the months partition thus releaseing
    space back to the database. Each months data is only about 100bytes long, but
    there could be about 15million rows.
    I can successfully create the external table and read the data.
    However I would like to place a header in the spooled file but the only method
    I have found to ignore the header is to use the "SKIP" parameter. I would
    rather not use this as it hard codes the number of lines I reserve for a
    header.
    Is there another method to add a comment to a external tables file and have the
    selects on the file ignore the comments?
    Could I use "LOAD WHEN ( column1 != '--' )" as each comment line begins with a
    double dash?
    Also, is there a limit to the size of an external tables file?
    Thanks
    JD

    Hi Gurus..
    any help will be highly appreciated.

  • Oracle 9i External Tables

    Helllo Everyone.
    I am new to external tables concept and I have only used using flat files.
    Is there a complete reference on on external tables somewhere?
    Also, instead of .csv or text based datafiles, can I use XML as data source for these external tables? If so, how?
    Thanks,
    R

    The link to the documentation has been already posted.
    About the xml as a possible datasource, in Oracle9i only the ORACLE_LOADER driver is supported, it allows you to read data using SQL*Loader, that is to say only plain text files.
    In Oracle10g a new driver has been added: ORACLE_DATAPUMP, it allows you to read with an external table a dump file created from other Oracle tables using the same ORACLE_DATAPUMP driver.
    Unfortunately the ORACLE_DATAPUMP driver doesn't support the XMLTYPE datatype:
    SQL> desc tbl_xml
    Nome                                      Nullo?   Tipo
    XML                                                PUBLIC.XMLTYPE
    SQL> CREATE TABLE xml_ext
      2    ORGANIZATION EXTERNAL
      3    (
      4      TYPE ORACLE_DATAPUMP
      5      DEFAULT DIRECTORY DATA_PUMP_DIR
      6      LOCATION ('xml_ext.dmp')
      7    )
      8    AS SELECT * FROM tbl_xml;
    CREATE TABLE xml_ext
    ERRORE alla riga 1:
    ORA-30656: tipo di colonna non supportato in una tabella organizzata esterna
    SQL>Max
    [My Italian Oracle blog|http://oracleitalia.wordpress.com/2009/12/18/table-elimination-oppure-join-elimination-lottimizzatore-si-libera-della-zavorra/]

Maybe you are looking for