Use of external tables
We have a new scada and database system being delivered which has a section for Alarm Data. We have been told that without extensive re-writing of the ActiveX components of the scada, then we cannot put this alarm data information into Oracle (10.2). Instead, aswell as the Oracle database (which resides on a RAID5 server alongside the application server, the flash recovery area (which is our database backup mechanism), the domain controller, the forms and reports - getting off track with a rant, but they're taking the mick...), they are giving us SQL Express to store the alarm data, which is ported out as a flat file.
I am not yet too familiar with anything above 7.3.4, but my limited research suggests to me that using the external tables feature of oracle would solve this problem and allow us to use the flat file to import the data into an oracle instance and manage it alongside the rest of the database.
Am I right? Seems too obvious a solution that they wouldn't have thought of it, but this is just one of the many, many problems we're having that I feel we could provide a better solution for. Comments??
hi,
u mention the Alarm data.
whether the data is in XML format or CSV format.
if it is CSV format use External Tables.
CSV format means store in a flat file.
i was worked on External Table.
External Tables r u used to store the data into oracle tables..
Regards
SATYA
Similar Messages
-
Use of External tables to load XML data.
Hi,
I have used external table definitions to load various XML files to the database, usually splitting the XML into seperate records - 1 per major element tag, and using PL/SQL to parse out a primary key to store in a relational table with all of the XML relevant to that primary key value stored as an XMLTYPE coumn in a row of the table. This has worked fine for XML with a single major entity (element tag) .
However, I now have an XML file that contains two "major" elements (both children of the root) that I would like to split out and store in seperate tables.
The XML file is of the following basic format:-
<drugs>
<drug>drug 1...</drug>
<drug>drug 2...</drug>
<partners>
<partner>partner 1</partner>
<partner>partner 2</partner>
</partners>
</drugs>
The problem is there are around 18000 elements of the first type, followed by several thousand of the 2nd type. I can create two seperate external tables - one for each element type, but how do I get the external table for the 2nd to ignore all the elements of the first type? My external table definition is :-
CREATE TABLE DRUGBANK_OWNER.DRUGBANK_PARTNERS_XML_EXTERNAL
DRUGBANK_XML CLOB
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY DRUGBANK_DIR
ACCESS PARAMETERS
( records delimited by "</partner>" SKIP 100000
characterset al32utf8
badfile extlogs:'drugbank_partners_xml.bad'
logfile extlogs:'drugbank_partners_xml.log'
discardfile extlogs:'drugbank_partners_xml.dis'
READSIZE 52428800
fields
drugbank_xml CHAR(50000000) terminated by '</partners>'
LOCATION (DRUGBANK_DIR:'drugbank.xml')
REJECT LIMIT UNLIMITED
PARALLEL ( DEGREE 8 INSTANCES 1 )
NOMONITORING;
The problem is that before the first <partners> element the 1800 or so <drugs> elements cause a data cartrdige error:-
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04020: found record longer than buffer size supported, 52428800
This happens regardless of the value of the SKIP or the size of the drugbank_xml field.
I have tried using an OR on the "records delimited by" access parameter, to 'delimit by "</partner>" OR "</drug>"', with the intention of filtering out the <drug> elements but this leads to a syntax error.
Anyone ever tried anything similar and got it to work?
Any other suggestions?
Thanks,
Sid.No, the content inside quotes is spanned across multiple lines....there are line breaks after every html tag.
"What's the error message you are getting?"
Iam not getting any error while selecting from external table , but I am getting tose rows in BAD file and log file has the following entries
KUP-04021: field formatting error for field TKBS_DSCN
KUP-04036: second enclosing delimiter not found
Message was edited by:
user627610 -
Reject the footer using the external table
Hi,
I have a flat file with fixed length which have a header and a footer.
My file is something like this:
HADF.TXT0309
D12345ABCD
D22345ABCD
FOOTERHJ
I want to create an external table based on that file, but
I don't want to have the header and the footer in my table. To eliminate the header I used skip 1, but I don't know how to eliminate the footer.
Any example and suggestions will be appreciated
Thank you for your time and consideration
CatalinHi,
This problem may be due to several reasons. I am aware of few reasons.
1)Have you deployed the external table first?
2) If you do not have your data base client and server running in the same machine, you should place the csv file in the database server machine's 'c:\CSV' folder in order to create the external table through database. Then do a select count(*) statement.
3)Another reason may be as gerardnico said the file name you refered may be wrong.
I don't know your requirement. If you could create the external table succefully and if you get value for the select count(*) from <external_table_name>, then try to import the external table into the Design Center and map it with the Table you need.
If you are doing it purely with OWB then,
Do you have the file Export_WithHeaders.csv in the Server machine's 'c:\CSV_FILE' folder?
Because while importing the metadata of the CSV file the OWB will point to your local machine's 'c:\CSV_FILE' that is why your Validation and Deployment is success without errors.
But while executing the map it will take the data from Server machine. It will search for the Location 'c:\CSV_FILE' in the server machine and will look for the file Export_WithHeaders.csv there. So create athe same folder setup which you have in your client machine and run this again.
Try this if you do not get any better answers in this thread -
Stuck on using an External Table in OWB 11.
I am using Oracle 11, and OWB 11.
I am trying to get some information from a CSV file, into an Oracle table. Here is what I have currently done:
1. I have setup a location called CSV_FILE and this is pointing to my local drive in a folder called c:\CSV
2. I have added the file to the Files tab, imported the metadata and the file is called Export_WithHeaders_CSV.csv
3. I have setup a new external table called CSV and pointed it to the above File and set the default llocation to CSV_FILE
4. I then right clicked on the external table and created the Data File pointing it to the CSV_FILE, and filling in the Data File Name with: Export_WithHeaders_CSV.csv
I am trying to do 2 things, the first is to just view the data in the external table through Design Center, and the 2nd is too add it to an OWB map to import into an Oracle Database. When I try to view the data (Right click on CSV, and select Data) I get the following error:
ORA-29913: error in exectuing
ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04040: file
Export_WithHeaders.csv in
CSV_FILE not found
I am able to generate and validate the external table without an erro, but I am not able to see any data, and I know the file contains close to 120 records?
When I bring the external table into a simple OWB map, just mapping it directly to a table, the map validates fine, but when I run the map, the test data shows a row_count of 0? I have been beating my head against the wall for a good 2.5 days now, and any help would be greatly appreciated....Hi,
This problem may be due to several reasons. I am aware of few reasons.
1)Have you deployed the external table first?
2) If you do not have your data base client and server running in the same machine, you should place the csv file in the database server machine's 'c:\CSV' folder in order to create the external table through database. Then do a select count(*) statement.
3)Another reason may be as gerardnico said the file name you refered may be wrong.
I don't know your requirement. If you could create the external table succefully and if you get value for the select count(*) from <external_table_name>, then try to import the external table into the Design Center and map it with the Table you need.
If you are doing it purely with OWB then,
Do you have the file Export_WithHeaders.csv in the Server machine's 'c:\CSV_FILE' folder?
Because while importing the metadata of the CSV file the OWB will point to your local machine's 'c:\CSV_FILE' that is why your Validation and Deployment is success without errors.
But while executing the map it will take the data from Server machine. It will search for the Location 'c:\CSV_FILE' in the server machine and will look for the file Export_WithHeaders.csv there. So create athe same folder setup which you have in your client machine and run this again.
Try this if you do not get any better answers in this thread -
Error while calling an procedure using an external table with C#.
Hello,
I am developping a scheduler application with Visual Studio 2010 (C#) to start my PL/SQL procedures.
Everything works fine with every procedure but one who is reading the content of an external table.
Strange thing is when i launch the same procedure with Toad, i have no problem, but when i launch it with my C# code :
OracleCommand cmdMET = new OracleCommand();
cmdMET.CommandText = "STG_AE.M_MET_S_EXT_DEFECT";
cmdMET.CommandType = CommandType.StoredProcedure;
cmdMET.Connection = con;
//OracleParameter retvalMET = new OracleParameter("retval", OracleDbType.Varchar2, 50);
//retvalMET.Direction = ParameterDirection.ReturnValue;
//cmdMET.Parameters.Add(retvalMET);
cmdMET.ExecuteNonQuery();
I've got this error :
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-01722: invalid number
ORA-06512: at "STG_AE.M_MET_S_EXT_DEFECT", line 8
ORA-06512: at "STG_AE.MET_SRC", line 10
ORA-06512: at "STG_AE.MET", line 14
ORA-06512: at line 1
I can't figure out why it works when i launch it with Toad and why i get this error when launching it with C#... Any advice would be great !
Thanks!You're right!
Adding
string sql = "ALTER SESSION SET NLS_NUMERIC_CHARACTERS = \",.\"";
OracleCommand cmd = new OracleCommand(sql, con);
cmd.CommandType = CommandType.Text;
OracleDataReader dr = cmd.ExecuteReader();
solved the problem !
Thanks a lot =) -
Hi, i am getting an error while using the External Table.
Hi,
I am getting the below error. External table has been created successfully and when I read the data from the external table i get the below error. Can anyone give solution of this error.
ORA-29913: error in executing ODCIEXTTABLEOPEN callout
ORA-29400: data cartridge error
KUP-04063: unable to open log file EXT_ZPP0A871_6738.log
OS error Permission denied
ORA-06512: at "SYS.ORACLE_LOADER", line 19
Thanks,
HimanshuKUP-04063: unable to open log file EXT_ZPP0A871_6738.log
OS error Permission deniedThe error seems quite clear... what's you OS and Oracle version ? -
Problem with file used by external table
Hi all,
following situation:
I am creating my file on unix/solaris and ftp it to oracle server which is running on windows.
I am getting following error when I try to do select on the external table I get:
ORA-29913: error in executing ODCIEXTTABLEFETCH callout
ORA-29400: data cartridge error
KUP-04020: found record longer than buffer size supported, 524288, in D:\corona\flat_files\input\CORE.PROGRESS.TXT
ORA-06512: at "SYS.ORACLE_LOADER", line 52
But it is working when I do following:
File->Conversions->UNIX/MAC to DOS-> Save it and ftp to windows server.
I have already tried to call unix2dos in my shell script, but I keep getting the same error.
Here is the CREATE TABLE:
CREATE TABLE EXT_TRACKING_DATA_PROGRESS
TRACKING_ID VARCHAR2(19 BYTE),
COMPANY_ID VARCHAR2(19 BYTE),
SUPPLIER_ID VARCHAR2(1 BYTE),
TRACKING_ID_TYPE VARCHAR2(2 BYTE),
START_DATE VARCHAR2(10 BYTE),
END_DATE VARCHAR2(10 BYTE),
BLANK VARCHAR2(100 BYTE)
ORGANIZATION EXTERNAL
( TYPE ORACLE_LOADER
DEFAULT DIRECTORY DIR_INPUT
ACCESS PARAMETERS
( fields terminated BY '@|@' )
LOCATION (DIR_INPUT:'CORE.PROGRESS.TXT')
REJECT LIMIT 0
NOPARALLEL
NOMONITORING;Are you FTP-ing the file to the Windows machine in ASCII mode? Or in binary mode? Binary mode won't convert the line breaks from Unix to Windows, which would appear to be the problem.
Justin -
Creation of External table by using XML files.
I am in the process of loading of XML file data into the database table. I want to use the External Table feature for this loading. Though we have the external table feature with Plain/text file, is there any process of XML file data loading into the Database table by using External table?
I am using Oracle 9i.
Appreciate your responses.
Regards
Edited by: user652422 on Dec 16, 2008 11:00 PMHi,
The XML file which U posted is working fine and that proved that external table can be created by using xml files.
Now My problem is that I have xml files which is not as the book.xml, my xml file is having some diff format. below is the extracts of the file ...
<?xml version="1.0" encoding="UTF-8" ?>
- <PM-History deviceIP="172.20.7.50">
<Error Reason="" />
- <Interface IntfName="otu2-1-10B-3">
- <TS Type="15-MIN">
<Error Reason="" />
- <PM-counters TimeStamp="02/13/2008:12:15">
<Item Name="BBE-S" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="BBE-SFE" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="ES-S" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="ES-SFE" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="SES-S" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="SES-SFE" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="CSES-S" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="CSES-SFE" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="UAS-S" Direction="Received" Validity="ADJ" Value="135" />
<Item Name="UAS-SFE" Direction="Received" Validity="ADJ" Value="0" />
<Item Name="SEF-S" Direction="Received" Validity="ADJ" Value="135" />
</PM-counters>
<PM-counters TimeStamp="03/26/2008:12:30">
<Item Name="BBE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="BBE-FE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="ES" Direction="Received" Validity="OFF" Value="0" />
<Item Name="ES-FE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="SES" Direction="Received" Validity="OFF" Value="0" />
<Item Name="SES-FE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="CSES" Direction="Received" Validity="OFF" Value="0" />
<Item Name="CSES-FE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="UAS" Direction="Received" Validity="OFF" Value="0" />
<Item Name="UAS-FE" Direction="Received" Validity="OFF" Value="0" />
<Item Name="PSC" Direction="Received" Validity="OFF" Value="0" />
</PM-counters>
</TS>
</Interface>
</PM-History>
My problem is the Item Name and Direction the value of both(ex PSCReceived or UASReceived) will be treated as the coulmn name of the table and '0' would be the value of that column. I am confused how to create the external table creation program for that.
I would really appreciate your responses.
Regards -
Group Permissions using External Table
I have a problem with using an external table for user group permissions.
I am using OBI authentication but need to use an external table to manage the user’s group permissions. I created two RPD groups, GROUP1 and GROUP2. GROUP1 has access to TABLE1. GROUP2 has access to TABLE2. I created the initialization block with the following SQL:
Select ‘GROUP’, groupname from groups_tab where username = ‘:USER’
I also turned on row-wise initialization.
I created a user, USER1, with access to both RPD groups. I also created corresponding Catalog Group (Settings Administration Manage Presentation Catalog Groups and Users Create a new Catalog Group). I have two dashboard pages PAGE1 and PAGE2. GROUP1 has access to PAGE1 and GROUP2 has access to PAGE2. When I log in as USER1, I have a quick test on the My Dashboard page that displays the GROUP session variable (@{biServer.variables[‘NQ_SESSION.GROUP’]}). The variable displays that USER1 belongs to GROUP1; GROUP2. I still cannot see the dashboard pages PAGE1 and PAGE2. When I go to Answers I cannot see TABLE1 or TABLE2.
Obviously, I must be missing a step somewhere. Any ideas?
I have tried the Rittman Mead post (http://www.rittmanmead.com/2007/05/21/using-initialization-blocks-with-ldap-and-database-queries-to-control-authentication-and-authorization/) and I am still not getting the right results.
Edited by: Canz on Feb 25, 2009 4:39 PMIt's likely to be a permissioning setup issue rather than your Init Block setup which seems to be working. Start by granting your test user full permissions on the object you want and then start removing them gradually to see where you don't see the dashboard any more. I think you might be missing a Traverse privilege in your dashboard shared folders but I can't check all the possible conditions with seeing your web catalog. Also check the case of your Web Catalog groups and the ones you populate on the Init block.
-
Accesing windows files using oracle unix external tables.
Hi, I'm trying to access files on windows2k using oracle external tables on unix:
Oracle server on unix: oracle ver. is 9.2.0.6 and OS is 5.8 Generic_117000-03 sun4u sparc SUNW,Sun-Fire
1. Created dir object on oracle unix pointing to windows path
2. created user
3. Assigned read and write privs on dir to user
4. created external table using dir step 1
5. select on external table
When I FTP the files into unix and recreate dir object, it works, but if I point dir object into windows path, it fails.
Is possible to do this?
Tks
GAPHi
The directory object must be accesible to your Database server, so if the Database is in unix you must create the directory in unix.
I don´t know software that allow unix systems to see a windows file system. Maybe windows can see unix from a samba server or NFS. -
External Tables using Discoverer
I am mapping the contents of a complex query to an external table. I am mapping the SUM() of each field to the external table. BUT... when I create a workbook using that complex folder, Discoverer correctly uses the external table but when I look at the actual query that is generated it performs a 'GROUP BY' on the the summary table (i.e my external table) prior to doing the query that I originally wanted.
This is different than if Discoverer was managing the summary table. A very simple example is:
External Table Query
SELECT
SUM(alias_avar)
FROM
(select id, SUM(avar) alias_avar
FROM
summarytable
group by id)
Discoverer Managed
Select SUM(avar) from summarytable
Thank you in advance for helping me with any ideas or workarounds.
MikeJust a word...I have had NO LUCK with Oracle9i external tables accessing network drive with Windows Server 2003 spk1.
I have read all the tips from Metalink and forums and nothing that works on Windows 2000 or NT work for 2003..Has anyone had any luck with this?
UNC or mapped letters..The best I could do was GIVE EVERYPERMISSION and create a SUPER users with ALL groups and still then all external tables could do was move it to the error folder with a OS errors saying it could not open the file. I have the same setup locally and it opens fine.
In fact UNC was the only way I could even touch the files..mapped drive letter did nothing ... -
Using users from an external table in the BI Publisher add-in for Word
After installing the BI Publisher add-in in Word, I find that I can connect to BI Publisher (http://hostname:9704/xmlpserver) only using IDs that are defined within WLS. I'm using an Oracle database table and external authentication to get my users logged in to the Answers environment, and they use the BI Publisher features of Analytics as well. But if I try to connect from the add-in to xmlpserver as one of those externally authenticated users, it fails.
Do I have to create new users in WebLogic Security for those folks who need to build BIP templates in Word? If not, how can I get the add-in to recognize and use the external table authentication?
I'm in 11.1.1.6.2 BP1, but I've noticed the problem for a long time, possibly back to 10g.As you probably noticed, your formulas won't import into Project Siena - only the data.
I've created a sample application for you to demonstrate how you create formulas in Project Siena.
Here's the link to the project:
https://onedrive.live.com/redir?resid=D64FC13017F614FD!47783&authkey=!AN3UxGDTP1vNbbU&ithint=folder%2cjpg
In this example, I'm doing the heavy lifting on the Calculate button.
Here's what I did to create the project.
I added two input text boxes for height and weight
I added a calculate button.
I added a label to display the result. In the Default Text I added
BMI which is a context variable. I could have called it BMIResult - it's up to you.
Here's the formula for OnSelect of the Calculate button:
UpdateContext({BMI: Round((Value(txtWeight!Text) / Value(txtHeight!Text * txtHeight!Text)), 2)*703})
Note: If you haven't done so already, click 'Express View' in the lower right hand corner to see the properties.
UpdateContext is a formula that is used to change results - in this case the variable, BMI.
I used the Round function so it displays 2 digits for the decimal.
I used the Value formula to convert the input boxes to numbers.
This should get you started in the right direction - let me know if you need further explanation.
One more thing to add - when creating a context variable, you will see a yellow exclamation mark until you have a formula that references it.
Thor -
Is there a way to know how many rows were rejected using external tables?
Hi,
I'm developing a package that needs to load different kinds of text files into the DB.
I was wondering whether there's a way of knowing in SQL or PL/SQL how many rows were rejected if my external table is defined with "reject limit unlimited".
I'm using the external tables in "insert into ... (select ... from external_table where...)"
sql%rowcount returns the number of inserted rows
I was hoping for something similar/easy to get the number of rejected rows as well.
I know I can try to analyze the bad file for the number of records if there's no other way...
Thanks in advance.Pyrocks wrote:
I was afraid of this answer...
this is a certain "no there's no other way" or should i wait for other responses (no disrespect - just want to make sure :) )Well the difference would be that this external table will have no complex logic. You would simple read in all the lines from the bad file. The chances that this is broken are therefore very slim compared with your original external table.
And tehre are many other ways possible. For example you can first copy the file into a real internal (temp) table without using any processing logic but with using an external table. Then do the processing / logic checking of the data and move the data from this temp table into your final data structure. Since you move the data two times the whole process will be slower then the original version. -
It's possible to use the external table with SYSDATE? Here is the create external table syntax I am using and it's not working.
create TABLE T_MASTER_EXT
CUSTOMER_NAME VARCHAR2(255),
AUDIT_TIMESTAMP DATE
organization external
type oracle_loader
default directory MY_DIRECTORY
access parameters
records delimited by newline
skip 1
fields terminated by ','
OPTIONALLY ENCLOSED BY '"'
lrtrim
missing field values are null
(CUSTOMER_NAME,
AUDIT_TIMESTAMP DATE SYSDATE))
location ('customers.csv')
reject limit unlimited;
Thank you.
HWell, this discussion is rather academic since there is no facility to supply a virtual column with a default value of SYSDATE or anything else in an external table, but as I understand it "oracle_loader" external tables currently work like this:
1. You issue a query against an external table
2. The database starts an external loader process to open and read the file
3. The loader process parses the file contents and returns the results to your query.
Possibly I have oversimplified, but I just don't see where "The SYSDATE changes when the table is written" comes in. What process is this?
PS:
@VG - this thread would be a lot easier to read for those of us with small monitors if you could make that AskTom link a bit shorter ;) -
Error importing CSV files with 'hidden' characters using External Table
Hi Folks
Bit of a strange one here.
We're well used to using the External Table method of loading data from CSV files into the database but a recent event has presented us with a problem.
We have received some CSV files that 'look' like regular CSV files but Oracle will not load them.
When we examined the CSV file using VIM on a UNIX box we saw the following 'hidden' characters between every regular character in the file.
^@So a string that looks like this when opened in Excel/Wordpad etc
"TEST","TEXT"Looks like this when exmained with VIM
^@"^@T^@E^@S^@T^@"^@,^@"^@T^@E^@X^@T^@"Has anyone come across this before?
Many thanks
Simon Gadd
Oracle 11g 11.2.0.1.0Hi Simon,
^@ represents the NUL character (0x00).
So, most likely, you've got a Unicode-encoded file.
You'll have to specify the character set in the record specification (and if necessary the byte order mark), for instance :
CREATE TABLE ext_table
col1 VARCHAR2(10),
col2 VARCHAR2(10)
ORGANIZATION EXTERNAL
TYPE ORACLE_LOADER
DEFAULT DIRECTORY dump_dir
ACCESS PARAMETERS
RECORDS DELIMITED BY '
' CHARACTERSET 'UTF16'
FIELDS TERMINATED BY ','
LOCATION ('dump.csv')
REJECT LIMIT UNLIMITED;http://download.oracle.com/docs/cd/E11882_01/server.112/e16536/et_params.htm#i1009499
Maybe you are looking for
-
Problem in accounting document
Dear Guru's, we have created a credit memo and at that time DG document is released.After that we have reversed the credit memo,it reversed by DG document only. But in some of the cases ..RV document is coming in case of reversale accounting doc fo
-
Hey, can anyone help me? Some of my albums have songs missing. I tried to download purchases but it says that all songs have been downloaded! Help
-
Cannot bring up firefox from the tray. it's minimized and unusable.
This problem started yesterday. I cannot bring firefox to a full screen, and when I try to read an email attachment, it opens to firefox which I cannot read.
-
Inventory : MTL_SERIAL_NUMBERS table
Hi All, In inventory we have table MTL_SERIAL_NUMBERS. From the eTRM, we can see there are so many other module tables depending on this table. Is deleting data from this table is advisable? If it is advisable, is there any conditions to be verified
-
OBIEE 11.1.1.7 Exporting to Excel 2003 and 2007 difference
Hi I have a simple analysis, a couple of measures, some formatted so tat if they are negative show them as Red, minus, 1 decimal and thousand separator. Another measure is not formatted to be Red, just black, minus, 1 decimal and thousand separator.