Extracting data into different flat files

I am using oracle 10g.
I've to run a very complex query that will extract/spool around 300 million records. It is so expensive to run the query multiple times. I want to run the query once but spool all the jobs into chunks of different files
so for 300 million records, I want to split the records into 10 different files of 30 million records each. Is it possible to do so??
Thanks

Hello,
Technically it doesn't make any sense and make it poor solution over spooling a file and then splitting. But if O/P comes back and says table is partitioned, now I can use 2 different solution (utl_file or spooling using sqlplus). I will then rely on shell script which will lauch query based on partition and spool a file, once done it close the file and opens a new one for next partition and so on.
Regards.

Similar Messages

Error Loading Data into a flat file

I am recieving the following error when loading data into a flat file from a flat file. SQL 2005 is my back end DB. If I cut the file iin half approx 500K rows my ODI interface works fine. Not sure what to look at.. I rebuit the interface which before was just dying giving no error and now I am getting this.
Thanks
java.lang.Exception
     at com.sunopsis.dwg.dbobj.SnpSessTaskSql.treatTask(SnpSessTaskSql.java)
     at com.sunopsis.dwg.dbobj.SnpSessStep.treatSessStep(SnpSessStep.java)
     at com.sunopsis.dwg.dbobj.SnpSession.treatSession(SnpSession.java)
     at com.sunopsis.dwg.cmd.DwgCommandSession.treatCommand(DwgCommandSession.java)
     at com.sunopsis.dwg.cmd.DwgCommandBase.execute(DwgCommandBase.java)
     at com.sunopsis.dwg.cmd.e.i(e.java)
     at com.sunopsis.dwg.cmd.g.y(g.java)
     at com.sunopsis.dwg.cmd.e.run(e.java)
     at java.lang.Thread.run(Unknown Source)

Figured it out, found similar post that stated changing the HEAP size
Increase the page size in odiparams.bat in the bin folder and restart Designer.
For eg:
set ODI_INIT_HEAP=128m
set ODI_MAX_HEAP=1024m

How to Extract data to a Flat File?

Hello,
Im using OWB 10g and Oracle 9.2 version of database. I want to know the complete steps of creating the flat file as my target module. I need to extract the data (source) from the oracle tables and load it into the flat files. Pls let me know your suggestions. Pls note except the flat files, I do not have any oracle tables as my targets.
Thanks,
Kishan

Hi, here's an excerpt from some notes I wrote a few months ago, they shpoud be clear enough.
0) the file will be created on the server using the utl_file package. So the path where the file will be created must be added to the utl_file_dir Oracle Server parameter. Request your DBA to complete this step.
-In OWB client:
1) create a file location (or use existing one) under "Files/Locations" in project tree
2) create a new Flat File module (or use existing one) under "Files" ; associate it with the desired location (not really necessary)
3) create a new mapping in an Oracle module
4) add to mapping a table operator bound to the table to dump
5) add to mapping un unbound Flat File (FF) Operator, give it a meaningful name
6) drag the desired attributes from table to FF Operator. This will instruct OWB to feed the file with comma-separated values from the source table using the utl_file package. Be careful to format attributes accordingly to the format you want.
7) reconcile outbound the FF Operator, creating a new flat file in the module created during step 2
8) close mapping, right-click on it an from menu "Configure..."->"Sources&Targets"-><YOURTARGETFILE>, specify the desired output location and the output file name
9) Open deployment manager and connect to runtime rep. If the FF location is new, register it by right-clicking+menu "Register" and specify parameters: operating system user/passawrd of the DB server, host and path where the file will be stored (must be the same added to utl_file_dir). ATTENTION: path info is stored in the generated mapping so be sure to redeploy mappings if changes occur
10) deploy mapping, test execution
11) OPTIONAL you can specify a different delimiter (TAB,etc) for the target file directly in its editor. But be careful then to reconcile inbound the FF op in the mapping before deploying.
Cheers, Antonio

How to extract data from multiple flat files to load into corresponding tables in SQL Server 2008 R2 ?

Hi,
          I have to implement the following scenario in SSIS but don't know how to do since I never worked with SSIS before. Please help me.
          I have 20 different text files in a single folder and 20 different tables corresponding to each text file in SQL Server 2008 R2 Database. I need to extract the data from each text file and
load the data into corresponding table in Sql Server Database. Please guide me in how many ways I can do this and which is the best way to implement this job. Actually I have to automate this job. Few files are in same format(with same column names
and datatypes) where others are not.
1. Do I need to create 20 different projects ?
               or
    Can I implement this in only one project by having 20 packages?
             or
    Can I do this in one project with only one package?
Thanks in advance.

As I said I don't know how to use object data type, I just given a shot as below. I know the following code has errors can you please correct it for me.
Public
Sub Main()
' Add your code here
Dim f1
As FileStream
Dim s1
As StreamReader
Dim date1
As
Object
Dim rline
As
String
Dim Filelist(1)
As
String
Dim FileName
As
String
Dim i
As
Integer
i = 1
date1 =
Filelist(0) =
"XYZ"
Filelist(1) =
"123"
For
Each FileName
In Filelist
f1 = File.OpenRead(FileName)
s1 = File.OpenText(FileName)
rline = s1.ReadLine
While
Not rline
Is
Nothing
If Left(rline, 4) =
"DATE"
Then
date1 (i)= Mid(rline, 7, 8)
i = i + 1
Exit
While
End
If
rline = s1.ReadLine
End
While
Next
Dts.Variables(
"date").Value = date1(1)
Dts.Variables(
"date1").Value = date1(2)
Dts.TaskResult = ScriptResults.Success
End
Sub

How to Extract data from a flat file and how to upload it into table

Hi
i want to extract the data from a web page, that data i want to keep in text file.
from the text file i want to read it and i want to upload into table
can any one help me its very urgent

here is my 20min try without debugging (needs a rs232 and loopback connector )
use the probe and highlight the dataflow to see how it works.
as always: there are many ways to solve your problem
Greetings from Germany
Henrik
LV since v3.1
“ground” is a convenient fantasy
'˙˙˙˙uıɐƃɐ lɐıp puɐ °06 ǝuoɥd ɹnoʎ uɹnʇ ǝsɐǝld 'ʎɹɐuıƃɐɯı sı pǝlɐıp ǝʌɐɥ noʎ ɹǝqɯnu ǝɥʇ'
Attachments:
read write file to RS232.vi ‏69 KB

Extract PSA Data to a Flat File

Hello,
I would like to download PSA data into a flat file, so I can load it into SQL Server and run SQL statements on it for analysis purposes. I have tried creating a PSA export datasource; however, I can find way to cleanly get the data into a flat structure for analysis and/or download.
Can anyone suggest options for doing this?
Thanks in advance for your help.
Sincerely,
Sonya

Hi Sonya,
In teh PSA screen try pressing Shift and F8. If this does not bring up the file types then you can try the following: Settings > Chnage Display Variants > View tab > Microsoft Excel > Click SAP_OM.xls and then the green check amrk. The data will be displayed in excel format which you can save to your PC.
Hope this helps...

XML data (Output) into a flat file

I need to convert XML data into a flat file, which is having "|" as delimiter and ftp the same to a server. The XML data is an output of BPEL activity. The XML data is employee records. Each employee records should come as one line in the flat file.
Is it possible to do this? if so, what will be the steps
Thanks

Hi,
You need to use file adapter to write your XML payload as a flat file.
Following are the steps to write XML payload into a flat file:
1. Create a sample flat file and add few rows in which fields should be delimited with pipe symbol. See the record sample given below:
Name|Age|Address|City|State|Zip
Sam|32|2788 Viking Dr|Green Bay|WI|54304
Here first row is the header row, you can remove the header row if it is not required.
2. Create a file adapter service using adapter configuration wizard. Give file name and folder location where you want to write flat file. On step 5 of the adapter configuration wizard you have to define or select message schema. Click on Define Schema for Native Format button, then Native Format Builder Wizard will be opened, here you can define message schema for your flat file.
2.1 Here you need to select Delimited for file type.
2.2 Give details about your flat file, select your sample flat file here, which you have created in step 1 above.
2.3 Click Next button twice and go to step 4, here give namespace and element names you want and click on Next.
2.4 In step 5 you need to specify delimiters. Wizard doesn’t shows *|* as a delimiter so you need to enter *|* in the Delimited By combo box and click on Next button.
2.5 Complete remaining steps and click on Finish in the wizard.
3. Finish "Adapter Creation Wizard". Add BPEL activities (assign, invoke) to pass XML payload to file adapter and file adapter will generate *|* delimited flat file and write it to the given folder.
See the following article for more details on Native Format Builder Wizard:
http://download-uk.oracle.com/docs/cd/B31017_01/integrate.1013/b28994/nfb.htm#CCHCIGCA
Regards,
Dharmendra
http://soa-howto.blogspot.com

Sqlplus – spool data to a flat file

Hi,
Does any oracle expert here know why the sqlplus command could not spool all the data into a flat file at one time.
I have tried below command. It seems like every time I will get different file size :(
a) sqlplus -s $dbUser/$dbPass@$dbName <<EOF|gzip -c > ${TEMP_FILE_PATH}/${extract_file_prefix}.dat.Z
b) sqlplus -s $dbUser/$dbPass@$dbName <<EOF>> spool.log
set feedback off
set trims on
set trim on
set feedback off
set linesize 4000
set pagesize 0
whenever sqlerror exit 173;
spool ${extract_file_prefix}.dat

For me, this is working. What exactly are you getting and what exactly are you expecting?
(t352104@svlipari[GEN]:/lem) $ cat test.ksh
#!/bin/ksh
TEMP_FILE_PATH=`pwd`
extract_file_prefix=emp
dbUser=t352104
dbPass=t352104
dbName=gen_dev
dataFile=${TEMP_FILE_PATH}/${extract_file_prefix}.dat
sqlplus -s $dbUser/$dbPass@$dbName <<EOF > $dataFile
set trims on
set trim on
set tab off
set linesize 7000
SET HEAD off AUTOTRACE OFF FEEDBACK off VERIFY off ECHO off SERVEROUTPUT off term off;
whenever sqlerror exit 173;
SELECT *
FROM emp ;
exit
EOF
(t352104@svlipari[GEN]:/lem) $ ./test.ksh
(t352104@svlipari[GEN]:/lem) $ echo $?
0
(t352104@svlipari[GEN]:/lem) $ cat emp.dat
      7369 SMITH      CLERK           7902 17-DEC-80        800                    20
      7499 ALLEN      SALESMAN        7698 20-FEB-81       1600        300         30
      7521 WARD       SALESMAN        7698 22-FEB-81       1250        500         30
      7566 JONES      MANAGER         7839 02-APR-81       2975                    20
      7654 MARTIN     SALESMAN        7698 28-SEP-81       1250       1400         30
      7698 BLAKE      MANAGER         7839 01-MAY-81       2850                    30
      7782 CLARK      MANAGER         7839 09-JUN-81       2450                    10
      7788 SCOTT      ANALYST         7566 09-DEC-82       3000                    20
      7839 KING       PRESIDENT            17-NOV-81       5000                    10
      7844 TURNER     SALESMAN        7698 08-SEP-81       1500          0         30
      7876 ADAMS      CLERK           7788 12-JAN-83       1100                    20
      7900 JAMES      CLERK           7698 03-DEC-81        950                    30
      7902 FORD       ANALYST         7566 03-DEC-81       3000                    20
      7934 MILLER     CLERK           7782 23-JAN-82       1300                    10
(t352104@svlipari[GEN]:/lem) $

Export multiple tables into one flat file

I have data in multiple tables on a processing database that I need to move up to a production database. I want to export the data into a flat file, ftp it to the production server and have another job pick up the file and process it. I am looking for
design suggestions on how to get multiple tables into one flat file using SSIS?
Thank You.

Hey,
Without a bit more detail, as per Russels response, its difficult to give an exact recommendation.
Essentially, you would first add a data flow task to your control flow. Create a source per table, then direct the output of each into an union all task. The output from the union all task would then be directed to a flat file destination.
Within the union all task you can map the various input columns into the appropriate outputs.
If the sources are different, it would probably be easiest to add a derived column task in-between the source and the union all, adding columns as appropriate and setting a default value that can be easily identified later (again depending on your requirements).
Hope that helps,
Jamie

Loading CLOB data to a flat file in owb 11.2.0.3

Hi,
Also,I have one more question.I want to load a table with clob data into a flat file using my owb client 11.2.0.3.But when I run this simple mapping ,I am getting this error:
ORA-06502:pl/sql:NUMERIC OR VALUE ERROR:CHARACTER STRING BUFFER TOO SMALL.
Please suggest what needs to be done.

I was up all night as I had a dead line doing this....
Turned out my Mapping was corrupted.
If you do a series of synchronizations by object name, position, id the mapping would end up corrupted.
I had to delete my staging table operator within the mapping, drag and drop and reconnect and now my loads work just fine..
Thanks

Export table data in a flat file without using FL

Hi,
I am looking for options where I can export table data into a flat file without using FL(File Layout) i.e., by using App Engine only.
Please share your experience if you did anything as this
Thanks

A simple way to export any record (table/view) to an csv fiel, is to create a rowset and loop through all record fields, like below example code
Local Rowset &RS;
Local Record &Rec;
Local File &MYFILE;
Local string &FileName, &strRecName, &Line, &Seperator, &Value;
Local number &numRow, &numField;
&FileName = "c:\temp\test.csv";
&strRecName = "PSOPRDEFN";
&Seperator = ";";
&RS = CreateRowset(@("Record." | &strRecName));
&RS.Fill();
&MYFILE = GetFile(&FileName, "W", %FilePath_Absolute);
If &MYFILE.IsOpen Then
   For &numRow = 1 To &RS.ActiveRowCount
      &Rec = &RS(&numRow).GetRecord(@("RECORD." | &strRecName));
      For &numField = 1 To &Rec.FieldCount
         &Value = String(&Rec.GetField(&numField).Value);
         If &numField = 1 Then
            &Line = &Value;
         Else
            &Line = &Line | &Seperator | &Value;
         End-If;
      End-For;
      &MYFILE.WriteLine(&Line);
   End-For;
End-If;
&MYFILE.Close(); You can of course create an application class for generic calling this piece of code.
Hope it helps.
Note:
Do not come complaining to me on performance issues ;)

What is the best way to export the data out of BW into a flat file on the S

Hi All,
We are BW 7.01 (EHP 1, Service Pack Level 7).
As part of our BW project scope for our current release, we will be developing certain reports in BW, and for certain reports, the existing legacy reporting system based out of MS Access and the old version of Business Objects Release 2 would be used, with the needed data supplied from the BW system.
What is the best way to export the data out of BW into a flat file on the Server on regular intervals using a process chain?
Thanks in advance,
- Shashi

Hello Shashi,
some comments:
1) An "open hub license" is required for all processes that extract data from BW to a non-SAP system (including APD). Please check with your SAP Account Executive for details.
2) The limitation of 16 key fields is only valid when using open hub for extracting to a DB table. There's no such limitation when writing files.
3) Open hub is the recommended solution since it's the easiest to implement, no programming is required, and you don't have to worry much about scaling with higher data volumes (APD and CRM BAPI are quite different in all of these aspects).
For completeness, here's the most recent documentation which also lists other options:
http://help.sap.com/saphelp_nw73/helpdata/en/0a/0212b4335542a5ae2ecf9a51fbfc96/frameset.htm
Regards,
Marc
SAP Customer Solution Adoption (CSA)

Data Extraction or IDOC to flat file

hi,
I have a project to create a flat file from SAP, for an external legacy system. There are 3 requirement.
What approach should I take. Simple data extraction OR Idoc to flat file.
There are 4 requirements:
1. first time extract all data.
2. on subsequent run, extract only changed & new records
3. if in SAP table, a record is deleted, then marked deleted in flat file.
What approach should I take if I use data extraction.
Thanks.

I read your question, my first thought would be to look at where the data is going?
What are the data requirements of the legacy system. IDOCs can speed up the development related to pushing the data out from SAP. Using ALE and change pointers you can automatically pass out the delta with a limited amount of development.
However, the receiving system then needs to parse the IDOC data. depending on the IDOC you are working with this can be a challenge especially if the legacy developer doesn't get IDOCs.
Sometimes its easier to collect and write the data from SAP using "simple data extraction". The data is more readily organized into a format the receiving system is expecting.
You can also pass the idoc to a middleware maping application if one is available and do the SAP to legacy mapping there.
Cheers

Uploading the data from a flat file into ztable

Hi,
I have a requirement where I have to upload the data from 2 flat files into 2 z tables(ZRB_HDR,ZRB_ITM).From the 1st flat file only data for few fields have to be uploaded into ztable(ZRB_HRD) .Fromthe 2nd flat file data for all the fields have to me uploaded into ztable(ZRB_ITM). How can I do this?
Regards,
Hema

hi,
declare two internal table with structur of your tables.
your flat files should be .txt files.
now make use of GUI_UPLOAD function module to upload your flatfile into internal tables.
CALL FUNCTION 'GUI_UPLOAD'
    EXPORTING
      filename            = 'c:\file1.txt'
      has_field_separator = 'X'
    TABLES
      data_tab            = itab1
    EXCEPTIONS
      OTHERS              = 1.
use this function twice for two tables.
then loop them individually and make use of insert command.

CUNIT error in data loading from flat file after r/3 extraction

Hi all
After R/3 business content extraction, if i load data from flat file to info cube, I am getting Conversion exit CUNIT error, what might be the reason, the data in the flat file 0unit column is accurate, and mapping rules are also correct, but still i am getting error with CUNIT.?

check your unit if you are loading amount or quantities what mapping you have and what you are loading from flat files.
BK

Extracting data into different flat files

Similar Messages

Maybe you are looking for