Is it possible incremental load using sqlloader

Hi
I am working on datawarehousing projects and every day loading 3 lakhs record to target server using sqlloader.Is it possible incremental loading using sqlloader
Example First day loaded 3 lakhs record into target using sqlloader ,when next day need to load another 2 lakhs records .Using sqlloader how to do the incremental load?
Thanks inadvance
Mohan

Hi
The sql loader has three options
Append
Replace
Truncate
The first option will help you and append the data and will not reject the duplicate . FOr rejecting duplicate record rejection make sure table has constraints.
Prashant_Arvind

Similar Messages

Incremental Load using Do Not Process Processing Option

Hi,
I have an SSAS Tabular model which is set to Do Not Process. How do I refresh and add new data to the model without changing the processing option
me

Hi Liluthcy,
In a SQL Server Analysis Service tabular model, the process has the following options:
Default – This setting specifies Analysis Services will determine the type of processing required. Unprocessed objects will be processed, and if required, recalculating attribute relationships, attribute hierarchies,
user hierarchies, and calculated columns. This setting generally results in a faster deployment time than using the Full processing option.
Do Not Process This setting specifies only the metadata will be deployed. After deploying, it may be necessary to run a process operation on the deployed model to update and recalculate data.
Full – This setting specifies that both the metadata is deployed and a process full operation is performed. This assures that the deployed model has the most recent updates to both metadata and data.
So you need run a process operation to update the data.
Reference:
http://www.sqlbi.com/articles/incremental-processing-in-tabular-using-process-add
Regards,
Charlie Liao
TechNet Community Support

Incremental Load - is it possible ?

Hi,I am new to Essbase and trying to understand if it is possible to incrementally load data from an OLTP system. My requirement is this:I have a Relational OLTP database where Service Orders are processed.I'd like to extract data from the OLTP database into Essbase. Because of large volumes of data in the OLTP database, I want to load data that has changed or is new on a regular basis, and update the Essbase Cube inrementally.My question is:How can I do this ?Where can I find documentation related to incremental loads ?Thanks and RegardsDibyendu

Hi Dibyendu,It is possible to use MaxL, Esscmd, or the Essbase API to automate dataloads. We have been using Esscmd to load data from a MSSQL/Domino Web application for years, and it works well. The web app periodically performs a 'transfer' which entails writing a data load file, an organization dim build file, a scenario dimension build file, two calc scripts (clear and calc), an Esscmd script, and a dos batch file to the Essbase server disk, then executes the batch file. The Esscmd script has error handling, and returns an error condition if an error occured.Jeff McAhrenDallas, Texas

Is it possible to load a non-standard image using some Java API?

Hi,
My "problem" is:
1. I have an image called "mediterranean_sea.IMG" (non-standard image format)
2. I need to process it (histogram, palette, etc).
3. I'm wondering if it is possible to load this image an process it using some Java API.
4. I've tried to do this using JAI but I think that this API only works with TIFF, PNG, JPEG, etc.
Any idea?
Thanks in advance,
Roger

[url http://forum.java.sun.com/thread.jsp?thread=468188&forum=31]Cross-post

Is it possible to load more dictionaries to ibooks? French, german ando so on? I have books in severl languages and it would be easire to use a built in dictionary like the one it already has.

is it possible to load more dictionaries to ibooks? French, german and so on? I have books in severl languages and it would be easirer to use a built in dictionary like the one it already has.

At the moment no - http://support.apple.com/kb/HT4059 says :
iBooks allows you to look up the definition of words using a built-in English or Japanese language dictionary
So it looks like just the two languages are currently supported

Is it possible to know flat files names loading using particular info pack

hi experts ,
is it possible to know flat files names loading using particular info package ??
in our project we hav flat file extraction manually ...as i am replacing old sap bi consultant i must contine pending loads ..
as ther s no document about loads i must check and ensure about loads left out ..'
for every flatfile they used load with differnt name ... if am able to get those files name i can list out pending ones
and can load
is it possible to those details ??
pls help me .....
relative answer will rearded with points ............
thanks in advance
regards
harry

I AM REALLY THANK FUL FOR ALL.. your people are really quick .and giving relavtive answers ...
they loaded from desktop only ....not from application server ..
i got details from info cube mange screen .they used to load data monthly wise ,so i hav given month in selection and viewed data..
Regards ,
Harry

Unable to use sqlloader to load large data into a varchar2 column..plz.help

Hi,
Currently the column in the database is a varchar2(4000) coulmn that stores data in the form of xml and data that has many carrriage returns.
Current I am trying to archive this column value and later truncate the data in this column.
I was able to create a .csv file but when using the sqlloder ,the utility fails as the data I am about to load has data stored both in the form of "xml" sometimes and sometimes as a list of attributes separated by newline character.
After several failed attempts... I would like to know if it can be achieved using sqlloader or if sqlloader is a bad choice and i should go for an import/export instead?
Thanks in advance...
-Kevin

Currently the column in the database is a
varchar2(4000) coulmn that stores data in the form of
xml and data that has many carrriage returns. Can I ask why your XML data has carriage returns in it? The nature of XML dictates that the structure is defined by the tags. Of course you can have CR's in your actual data between those tags, but if someone is hard coding CR's into the XML to make it look pretty this in not how XML was intended to be used.
I was able to create a .csv file but when using the
sqlloder ,the utility fails as the data I am about to
load has data stored both in the form of "xml"
sometimes and sometimes as a list of attributes
separated by newline character.It probably can be (can you provide a sample of data so we can see the structure) but you would probably be best revisiting the code that generates the CSV and ensure that it is output in a simpler format.

Table to load dynamically using sqlloader.

how to specify which table to load dynamically using sqlloader. We are using sqlldr in a script as given below...
sqlldr user/pass control=/control/loader.ctl log=/log/logger.log bad=/bad/badrec.bad data=/data/d1.txt

Hello, you'd need to build the necessary control files through a shell script, here is an example:
# Write sqlplus commands to control_file.sql
echo "set head off
set feedback off
set pagesize 0
set termout off" > control_file.sql
# read table and column list from table.dat
more table.dat |awk '{print $1}' |
while read TAB
do
        echo "Table ${TAB}"
        read COLUMNS
        STR="nothing"
        if test "$TABLE_LIST" = "empty"
        then
                TABLE_LIST="'$TAB'"
        else
                TABLE_LIST="$TABLE_LIST,'$TAB'"
        fi
# construct select to load data - creating SQL*Loader control files
        echo "set head off\n
        set feedback off\n
        set pagesize 0\n
        select column_name||' '||
        decode(data_type,'VARCHAR2',' char ('||DATA_LENGTH||')',
        'CHAR',' char ('||DATA_LENGTH||')','DATE',' date (20) \"DD-MON-YYYY HH24:MI:SS\"','')
        from user_tab_columns
        where table_name ='$TAB'
        order by 1;\n" | $ORACLE_HOME/bin/sqlplus -s $USR | awk '{print $1" "$2" "$3" "$4" "$5}' |
# read the columns in and concatenate them together to form the control file select
        while read COLUM
        do
                if test "$STR" = "nothing"
                then
                        STR="\nspool $TAB.CTL\nselect 'LOAD DATA\nINFILE \"$TAB.DMP\"\nBADFILE \"$TAB.BAD\""
                        STR="$STR\nDISCARDFILE \"$TAB.DIS\"\nDISCARDMAX 99\nTRUNCATE"
                        STR="$STR\nCONTINUEIF LAST != \"$FIELD_ENCLOSURE2\"\nINTO TABLE $TAB"
                        #STR="$STR INTO TABLE $TAB"
                        STR="$STR\nFIELDS TERMINATED BY \"$FIELD_DELIMITER\""
                        STR="$STR ENCLOSED BY \"$FIELD_ENCLOSURE\" AND \"$FIELD_ENCLOSURE2\""
                        STR="$STR\n TRAILING NULLCOLS"
                        STR="$STR\n($COLUM"
                else
                        STR="$STR,\n$COLUM"
                fi
        done
        STR=" $STR)'\nFROM DUAL;\nspool off\n"
        echo "$STR" >> control_file.sql
done

Incrementally Loading Data Using a Data Load Buffer

Hi
I am using Essbase 9.3.1. I am trying to use the feature "Replacing Database or Incremental Data Slice Contents" for my data loads to the ASO Cube.
I have 2 data sets, one of them is 2 years history. And another is last 3 months which would be changing on a daily basis. I looked at that DBAG and they have exact same scenario as an example for this feature. But I am not able to overwrite my valatile data set with my new file.
Here is what I do
alter database ${1}.${2} initialize load_buffer with buffer_id ${6} resource_usage 0.3 property ignore_missing_values, ignore_zero_values ;
import database ${1}.${2} data from data_file '${3}' using server rules_file '${4}' to load_buffer with buffer_id ${6} add values create slice on error write to '${5}' ;
alter database {1}.{2} merge incremental data;
alter database ${1}.${2} clear aggregates ;
execute aggregate process on database ${1}.${2} ;
In fact my data from my new (incremental file) does not even make it to the database. I checked that it does get rejected.
Am I doing something wrong over here. How do I use the concept of "data slice" and its incremental load feature.
Can anyone please explain ?
Thanks
Mandar Joshi

Hi,
Just wondering if anyone had any inputs or feedback on my query. Or is my question a really stupid one and does not deserve any attention :)
Can someone explain how the "data slice" concept works ??
Thanks
Mandar Joshi.

Loading a falt file into table using sqlloader

I have a flat file (csv/txt) which has a header and a trailer record
I want to load it using SQLLoader , i want to skip this trailer/footer record.
Please suggest , due to some business rules i cannot use WHEN clause
I am on Oracle 11g

Well, maybe you'll get better help in theses spaces then, ODI is not really my cup of tea:
Data Integrator
https://forums.oracle.com/community/developer/english/oracle_database/export_import_sql_loader_%26_external_tables

Loads using DTP - manual update possible?

Hi,
I have a question regarding loads done using DTP in BW 7.0.
In 3.X loads using infopackages, sometimes a data package will get runtime errors such as DBIF_RSQL_SQL_ERROR (deadlocks). Usually, we would change the status to red and do a manual update of the affected data package in the data load monitor.
However, I cannot find this option for DTP and I have to reload the whole request even only 1 data package has failed because of the SQL error. Does anyone know how to repair this in a DTP load?
Any help will be appreciated, thanks in advance!

Hi CK,
Not found a way of doing this at current but I know where you are coming from, you're thinking of the old set manual status or even try a sm58 and trigger the idocs. If a DTP fails you might have the option of using the error stack but only if this has been configured. A failed load will prompt you to do this. Other ways of trying to fix the SQL errors are to check what the SQL problems are, most of our problems have been tablespace issues, etc.
Daily monitoring by basis/bw admin should sort these.
Cheers,
Pom

Incremental Loads and Refresh Date

Hi all,
Thank you for taking the time to review this post.
Environment
Oracle BI Applications 7.9.6 (Financial & Project Analytics)
Oracle E-Business Suite 11.5.10
Question
I have a Test BI Apps 7.9.6 in a Test environment that is connected to a static EBS 11.5.10 data source. As part of my testing phase I'd like to do multiple Incremental Loads to get an accurate performance impact and timing study for the final pre-approval before migrating to Production. I can get a refresh of EBS which has a week's worth of transactions after my Initial Full Load. What I'd like to do is change Refresh Dates to "trick" the Incremental Load into only loading one days worth of data at a time, rather than the full week's worth of data in the Incremental load. Is this possible, and if so, how?
Example timeline:
Today - Initial Full load using Test EBS as of today
1 week later - Refresh static Test EBS from Production with a week of transactions
Post Refresh - Run daily Incremental jobs using static Test EBS
First Incremental Load - Today's position + 1 day,
Second " " - Today's position + 2 days,
Third " " - Today's position + 3 days, etc
As always all comments and solutions greatly appreciated.
Kind Regards,
Gary.

Say on the 01st of the month, you did a Load.
Then on the 08th of the month, the source EBS system was itself refreshed.
What you want to do is to run a single day refresh on the 08th for all data from the 01st to the 02nd of the month), and then another single day referesh -- whether on the 08th or on the 09th , you don't care -- for all data from the 03rd to the 04th.
Unfortunately, the refresh is from last refresh date to current date. You can't define "refresh upto date". Therefore, your first 'incremental' refresh on the 08th would refresh all data from the 02nd to the 08th in one shot. What you could try to do is
a. After the first load on the 01st, shutdown the BI DWH.
b. When the EBS test source is refresh on the 08th, reset your SYSTEM CLOCK ON THE BI DWH DATABASE SERVER to the 2nd (or 3rd) of the month.
c. Now, when you run a refresh, BI will extract all data from the 01st to the 02nd or 03rd (even though EBS is as of the 08th).
d. Once this is done, shutdown BI DWH.
e. Reet the SYSTEM CLOCK ON THE BI DWH DATABASE SERVER to the 3rd or 4th of the month.
f. Run another Incremental Refresh.
... and so on ...
Hemant K Chitale
http://hemantoracledba.blogspot.com

Incremental Load with ODI

Hi All,
I have a few questions related to ODI.
1. ODI can be used to migrate data from DB2/4000 to Oracle 10g on AIX. This is possible right?
2. My DB2 is a core system. Can ODI handle incremental loads to the target source? For example, if a particular tables sees updates every 10 minutes, will ODI be able to handle the changes on this table and reflect the same on the target database?
Any suggestions would be appreciated.
Thanks

Yes , you can by using trigger but the trade off is the source table must be installed a trigger , and several temp tables are created in source database .
CDC by using native journal , we are on the way of testing it .
Edited by: oracletruedb on Oct 7, 2008 2:33 AM

Incremental loading in owb

hi all,
IT IS URGENT
i came to know that we can use filter on date field.but i cannt uderstand which condition to be placed in filter because source table is having fields like
prod_id,prod_desc,created_date,etl_process_date.
target table is having attributes like prod_id,prod_desc,created_date,modified_date.And also i have to perform TYPE2 mapping for any modification on prod_desc field.
SO my requirement is type2 with incremental loading in OWB
pls let me know incremental loading part,i.e. how could i develop using filter condition

You ca do it by various methods:
1.Load date time store the latest load date time in a table and while loading the target table make sure that data date is after the load date time. (joiner can be used to specify condition).
2. Create an mlog and mview to pick incremental data from source table and use this set as source of your map.
3.If possible add a column which indicates the status of processed records (in source table).Say for example add column named processed records and set it status as YES in the pre map process only those records with status as YES and at the end of the map set this status as DONE.Then again in the new cycle set the records with null in processed records column as YES and repeat the fore mentioned process.

Duplicate rows in Hierarchy Table created when running incremental load

I copied an out of the box dimension and hierarchy mapping to my custom folders (task hierarchy) this should create the same wids from dimension to hierarchy table and on full load does this using the sequence generator. The problem I am getting is whenever I run a incremental load instead of updating, a new record is created. What would be the best place to start looking at this and testing. A full load runs with no issues. I have also checked the DAC and run the SDE trunc always and SIL trunc for full load only.
Help appreciated

Provide the query used for populating the child records. Issue might be due to caching.
Thanks
Shree

Is it possible incremental load using sqlloader

Similar Messages

Maybe you are looking for