SQL and Spool File: low performance (Oracle 8)
Hi,
I've got some difficulties with the spool function: it works very slowly when I have to write on a file.
In details, a shell script starts the PL/SQL procedure, which starts the spool function that produces a file.dat .
Oracle solves the query and sends out the data immediately. This with 850 million of records for a 20gb final file. The total time to write this file is 8 hours.
May it be a solutions with could to improve this file writing procedure, trying not to use the spool function, like a C program which could work on the memory? In this way it could take fewer time.
Thank you for your suggestions and solutions.
A
Oracle RDBMS does not write to the spool file. The client does (SQL*Plus in this case).
SQL*Plus uses standard kernel calls.
To open/create the file and write data to it:
- open call with flags O_WRONLY|O_CREAT|O_TRUNC (see open (C System Call) - Code Wiki)
- write call to file descriptor (see write (C System Call) - Code Wiki)
These are standard calls supported by all modern kernels (Unix, Linux, Windows, OS/X, etc). The performance of which is entirely dependant on how fast the kernel can service the call. Which in turn is dependant on the speed of the file system on which these calls operate.
Speeding up a spool type process typically means using async I/O (called overlap I/O in the Windows kernel). The approach being to fetch the next set of data from the database cursor, while writing the previous fetch's data to spool file, at the same time.
Of course, better and faster hardware can also improve the performance of the I/O subsystem and make such a spool process faster.
Similar Messages
-
Using archivelog and control file from other Oracle server
I am still bothered with my backup process.
I have 2 AIX boxes (same model, say A and B); both have BAAN 5 and Oracle 10g R2 on. Right now my colleague insists to use the export pump (cold backup) from Prod Oracle server (A) to restore the Oracle server on Box B. The Prod server has the archivelog mode turn on. But it will miss any transaction data from import pump till the crash point of Box A. So this is my confusion.
Can I pass the control files and archivelog files form Box A (prod server) to Box B and use them to restore the Box B as the latest Prod server? How?
I tried to convince them to use the RMAN backup? But not successful?
I think the best way is probably to use the Oracle Data guard. However, there is always one concern to my manager and colleague, that such process will cause the data on the restored server (failover, Box B) not recognizable by the BAAN, which define the objects (tables).
ThanksPerforming a logical backup is not useful to restore to the point of failure. The only valid and available option is a hot backup/archivelog mode. Your recovery manager backup perform a controlfile and redologfile backup, so those can be restored at the destination. You must take care of the way you perform the backup, and ensure the paths where your backup is being deposited are visible by the second node. A shared storage with same mount points is suitable in this case. A tape robot configured at both nodes is also a suitable solution.
Recovery manager perfoms a controlfile and spfile restore, too. This rman command perform the action:
SET DBID <DBID of the database,for which you want to restore the controlfile>;
RESTORE CONTROLFILE FROM <name_of_backupiece_which_contains_the_controlfile backup>';
I don't see any problem on the recovery manager side, and technically speaking, on the Oracle side it is perfectly possible to restore your database at a remote location. I don't know what happens on the BAAN side, if you are required to have it configured to be operative on the target node. You could try to clone your database at the node B, configure BAAN and prepare the proceduere in case of failure.
Configuring a dataguard is also a recomended action. So it is to think about Cold Failed Over clusters. I have recently performed a CFC configuration with BAAN. No problem it works smoothly.
~ Madrid. -
Hello.(sorry for my english)
I have a legacy system , ported to opencobol that uses Berkeley DB. I can successfully open the database files using java both in windows and linux. However i want to create a web front end for the application.
Data is stored using strange cobol structures (most numbers are stored as text or 4 bits per digit) and other fancy stuff.. so custom parsers should be written. (dont rember BDB classes name for that)
I have been reading the documentation of Berkeley DB 12 hours totaly today but couldnt find answers to some questions...
Is it possible to attach the database to dbsql.exe (sqlite) server ?? I tried to do "attach "..pathtofile" as NULL ,but i get multiple databases specified and not supported and other stupid messages....
I think that what i tried propably doesnt make sense as collumns arent specified and nothing is in order this thing to work ..It is a simple Key value database.
So i am a little stucked here.
The only solution i can think is to synchronize with a relational Database. But i dont have triggers or nothing that helps....
Note that i want one way synchronization. The "other way" will have very limited tasks.
It seems to me that checking every x minutes if something changed using a cursor is a demanding task. So i am wondering.. Is there a way to track changes???
Thank you :)
Edited by: 843912 on 12 Μαρ 2011 7:13 μμHello,
I am not sure I completely understand your question. If you are asking about importing
and exporting data from a Berkeley DB database into an Oracle DB Table in Berkeley DB
releases prior to 5.*, then this can be done with the Oracle OCI interface. If I am
misunderstanding the question, please let me know.
Thanks,
Sandra -
Parse JSON records using PL/SQL and store the output in Oracle Database.
We have JSON data/record stored in a Oracle table as CLOB(database version 10g). We need to read/parse this JSON data and recognize the relevant tags and corresponding values in this JSON data and then insert these id/values into a separate Oracle staging table.
Please let me know if there is any re-usable code available for this.
We are looking for solution that does this manipulation in pl/sql. If this is not feasible, even Java code approach is fine.
Regards,
Kiranmayi.Hello,
PL/JSON | Free software downloads at SourceForge.net
ERASME - pl/sql library for JSON
Regards
Marcus -
HII,
I am trying to develop an SQL query that will produce an Excel File that can be posted on a WEB site for x amount of transaction.
I would like to know which ways are there in which I can do this tast.
If anyone has a suggestion that would be greatly appreciated.
Thanks
PanktiHii Justin,
I was just reading up on UTL_File package althought I am still unable to conclude as to how am I to go about using this method. I hope you don't mind I am a bit new to this enviorment so I have lots of question. Let me explain in detail: I have columns x, y, and z in a perticular table. I want to develop an SQL query which will give data in Excel file.
How do I use the UTL_File method to acchive this task?? -
Catproc.sql and catalog.sql files
I am using oracle 10g recently i have runned two files catproc.sql and catalog files by connecting to sys
after running those files in sys when i select
select * from tab;
no rows were returned
1) these files will show any eeect on data base
2)From toad when i connect to data base in toad it is showing all the tables in sys user
is there any effect of running those files933900 wrote:
I am using oracle 10g recently i have runned two files catproc.sql and catalog files by connecting to sys
after running those files in sys when i select
select * from tab;
no rows were returned
1) these files will show any eeect on data base
2)From toad when i connect to data base in toad it is showing all the tables in sys user
is there any effect of running those filesHave you created database newly? so you ran catalog.sql & catproc.sql?
Of course you can able to see from TOAD but not from SQL, its weird.
Can you post
SQL> select * from dba_registry;
Also any errors from alert log file, when you ran These scripts? if so post here. -
Is anyone aware of why we should not delete audit and trace files under the oracle admin directory that are over 30 days old? I don't know that there is anything in place to do this and they are building up. Tuldcorpadb01:/oracle is at 96%. These files aren't that big, but there are a lot of them. See counts below.
oracle:tuldcorpadb01:ecmd> cd admin
oracle:tuldcorpadb01:ecmd>find . -name \*.aud -type f -ctime +30 -ls|wc -l
25149
oracle:tuldcorpadb01:ecmd>find . -name \*.trc -type f -ctime +30 -ls|wc -
2426Move older audit files to a different filesystem until you find someone being able to define what files can be deleted.
Regards
Gustavo Restuccia -
Hi,
I don't want the output of the spool file to be written to a log file while executing the script. My script looks like this,
#/bin/ksh
sqlplus test/test 1>>/test/abc.log 2>>/test/abc.log <<EOF
set termout off
set feedback off
set flush off
spool /test/test.dat
select 'a' || 'b'|| 'c' from dual;
spool off
disconnect;
exit;
EOF!
I want the output to be written only into the spooled file. now it writes to both the log file and spool file. how can i avoid it from writting to the log file(1>>/test/abc.log). pl helpActually. Why do you want to write to both anyway?
You are using the general unix format to write to a file so why do you need another file? -
How do you extract SQL from Oracle Forms and Reports files?
I am developing an "as is" data model for a government client for a 14 year old system that has three databases, 20 schemas, over 1500 tables, and over 23,000 columns. Needless to say, I do not plan to perform a manual mapping of data to screens and reports.
Most of the system has been developed in Oracle Forms and Reports. I am trying to map the live tables and columns to forms and reports.
The process here has been to save the forms and reports files as .fmb and .rdf files. The client does not have an available copy of Oracle Designer, which I understand could be used to extract the SQL.
Is there a utility somewhere that can parse the .fmb and .rdf files to extract the SQL?
Thanks,
Jim GearingJim,
I don't know of any utility that will do this. You can convert and save each fmb as a fmt so you can view/search the contents , but I don't recommend that approach.
On the other hand, you download a copy of Oracle Designer:
http://www.oracle.com/technology/software/index.html
It usually is included with Oracle Forms and Reports. -
C# application to execute both oracle and sql server script files.
Hi All,
I am suppose to develop an application using c# which can execute oracle script files as well as sql server script files based on some user's choice.
Both the script files(oracle/sql server) contains scripts to create tables, create views and create functions.
What is the best way to do this and which existing c# functions should is use.
Thanks in advance.I am suppose to develop an application using c# which can execute oracle script files as well as sql server script files based on some user's choice.
You can use SQL Plus for Oracle. You can have a C# program that issues commands for scripts for table, views, triggers creation and the whole 10 yards to SQL Plus,
http://docs.oracle.com/cd/B19306_01/server.102/b14357/qstart.htm
https://www.google.com/#q=sql+plus
I started writing a program to use SQL Plus to run the scripts, but cI ame to the conclusion that it was a waste of my time.
A tool called OSM, an Oracle, tool gives a list of script files that need to be run do to a change in a script file based on file timestamp. You can copy the list doing a copy/past operation and drop it into SQL Plus's Command line, and SQL Plus runs the
scripts.
For MS SQL Server, you have this, which you can run from a C# program too.
http://www.webfactory-world.de/wfknowledgebase/wfknowledgebase_wf3.3/Content/AdditionalWindowsSettings/Tutorials/UpdatingSQLDBFromCommandLine.htm
I would consider using the above from a C# program a waste of time too.
I look at it this way. If the user doesn't have database admin experience and know how to use DB admin tools, then they have no business doing anything with a database - period.
BTW, the Invoke statement in VB or C# will run SQL Plus or SQL Server command line tool. -
Extraction of oracle pkg and pkg bodies into spool file??
Hi,
I need to extract all packages and its bodies into spool file. can you give me script or guide how to do this.. i have more than 200 packages.. need to create spool file for each package. great thanks for help.
/mr
Edit/Delete MessageFrom 10g, DBMS_METADATA is designed for ! :)
[http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14258/d_metada.htm#i1019414]
Edited by: Leo Anderson on 1 sept. 2008 14:22 -
Date and time in spool file name
Hi,
can anyone show me how i can write the code below so that the spool file will automatically take the system time and date?
thanks again.
set term off;
set echo off;
set heading off;
set linesize 1500;
set pagesize 9999;
set feedback off;
spool /home/oracle/ATM_Upload/files/CRD000119MMDDYYHHMISS.DAT;
SELECT rec_ind||source_type||bbdsa_code||seq_num||dt_ti_ext||ver_num
as atm_header
FROM atm_dc_hd;
spool off;Hi,
I do something similar to this in a file called ed.sql which I use in SQL*Plus to give me a history of SQL*Plus buffers in separate files which follow the naming convention $DATABASE_$SID_$TIMESTAMP.sql. I use it from SQL*Plus when editing, rather than typing "ed" I type "@ed" and it does the naming for me:
-- Turn off terminal output
set termout off
-- Save current buffer to a temp file
save tmp.txt replace
-- Set editfile name to $DATABASE_$SID_$TIMESTAMP.sql
column fname new_value fname
column sid new_value sid
select trim(sid) sid
from v$mystat
where rownum = 1;
select global_name||'_&sid'||'_'||to_char(sysdate, 'YYYYMMDDHH24MISS')||'.sql' fname
from global_name;
set editfile '&fname'
-- Retrieve our initial buffer
get tmp.txt
-- Turn terminal output back on
set termout on
-- Fire up the editor
edYou could probably adapt this in the following way for your needs:
column fname new_value fname
select 'CRD000119'||to_char(sysdate, 'MMDDYYHH24MISS')||'.dat' fname
from dual;
spool '&fname'I don't know how you get the 'CRD' part of the filename but you may be able to generate that too using the above as a guide.
cheers,
Anthony -
Hello, How do I tell sql+ to spool output file from windows to Unix server?
Hello, How do I tell sql+ to spool output file from windows to Unix server?
I am new to SQL+ and just learned how to spool the file. But file is saved in my local windows enviroment and since it's 2GB in size...I want to spool it directly to another remote unix server.
Pls answer in detail... I have been to most of the thread and didn't see relevant answer to above question.
Am I suppose to develope some script which FTP the spool file directly to the server I want to
or
i Have to use UTL_FILE Package ?
Thanks for replyYou may not be able to...
SQL*Plus can only spool to the local machine. If you have mapped a directory on the Unix server from your Windows machine, you can specify that directory in your SPOOL command.
You could telnet to the Unix server, run SQL*Plus there, and spool the file to a local (Unix) directory.
If the Unix server is also the Oracle database server, you could use the UTL_FILE package rather than using SQL*Plus to spool a file.
If the Unix server is also an FTP server, you could also FTP the file from your local machine to the server.
Of course, I would tend to re-examine a requirement to regularly generate a 2 GB text file. It seems likely that there is a better way...
Justin -
Seek help to format spool file from SQL*PLUS
I am running a Unix shell script to call a Oracle 11g SQL script from a Oracle database. In this SQL script, I need to connect to many different remote databases to select data, then sool these records as one big text file to a directory. Then email the file to related Group users. In the spool file, there is a line on the top of each page like this:
DUMMY
DB_NAME
I know this is caused by connect to remote database in SQL*PLUS. My connection string is like this:
Conn system/password@Oracle_SID
How can I remove these lines or how to skip these lines into spool file? Please advise. Thanks in advnce. Wish all of you Happy New Year!!!Hi,
It sounds like you have some kind of formatting (such as SQL*Plus TTITLE) producing the output you don't want. If that's the case, temporarily stopping the spooling might not help you. Find out what is causing the output that you don't want. You say that you know it is caused by the CONNECT statements, but it must be more than that. I've written scripts with CONNECT statements that don't have anything like what you reported at the top of each page; in fact, they don't even have pages: the output is one continuous stream. Find out what's putting the unwanted output there, and that will be a big clue as to how you can stop it.
You say that you know the unwanted titles are there because of the CONNECT statements. If so, use database links instead of CONNECT. You don't have to use dbms_scheduler or utl_file; just eliminate the CONNECT statements. (I'm not saying that there's anything wrong with dbms_scheduler or utl_file; you should definitely investigate those tools. I'm just saying that using database links is independent of them.)
What would happen if you did all your connecting to different databases at the OS level? Can you write a shell script that connects to each database in turn, and runs a SQL*Plus script in each one. Each SQL*Plus script would have a SPOOL or SPOOL ... APPEND command, or maybe you could build the SPOOL into a LOGIN.SQL script. -
Spool file contains spool off; statement and a / in it .how to remove these
Hi ,
I am new to using forums.
I am a basic beginner user of oracle .
I need to spool the output of 2 select queries into a file.
I could achieve it.
But the file also contains ,
SPOOL OFF;
in it . How to avoid these in the file.
Any help is appreciated .
thanksuser11768287 wrote:
Hi
I checkout and created a .sql file .
But the spool file , contains an empty line after every record is printed. These are the heading i am using.
SET HEADING OFF
SET ECHO OFF
SET TERMOUT OFF
SET VERIFY OFF
SET FEEDBACK OFF
The spool file starts with a blank line followed by the record and then blank line after every record .
What to do to get all records one after the other without blank lines in between the records in the spool file.
In addition to Toni's mention of PAGESIZE and TRIMSPOOL, look at LINESIZE. Also, what are you using to view the spooled file? Are you sure you aren't just seeing line wrap within that application - like Notepad?
Thanks ...
Maybe you are looking for
-
I thought that I understood that once music uploads or matches to iTunes match, that my library would no longer be stored (using space) on my iPhone. Before the purchase, my library took 4GB space. Since the purchase, I haven't downloaded/listened
-
Creation of a table with static data(not a DB table)
Hi I need to display a table with some static data in an OAF page. This table is not a DB table. It is like an HTML table I have placed the following query SELECT 'A1','B1' FROM DUAL UNION SELECT 'A2',B2' FROM DUAL in the VO. But it is returning a si
-
Z report for BED, ECS VAT values
Hello Experts, Here I have an issue we have developed the zreport for fetching the BED, ECS and Basic amount values for the particular Purchase orders with invoice posting dates as intial parameters. The report is giving error for each line items...t
-
Creating a Two Dimensional Array in Xcode and populating it with a values f
Whats the easiest way to declare a two dimensional array in Xcode. I am reading an matrix of numbers from a text file from a website and want to take the data and place it into a 3x3 matrix. Once I read the URL into a string, I create an NSArray and
-
Is it possible to import one spark component without importing the entire spark library?
I want to create an actionscript mobile application project because the total size will be smaller than a flex mobile application - the SDK is smaller because it doesnt contain unnecessary libraries e.g. the spark, mobilecomponents, framework etc. ar