Capturing changes - efficent way?
Hi,
We want to capture DML changes to tables in a given db schema. Oracle offers several features to do this namely, Audit, Triggers, DBMS_CHANGE_NOTIFICATIOn, Flash Back Queries, Log Mining, etc. Some more work than the others.
I dont mind doing more work, I just want to know what is the EFFICIENT way of doing this. I am afraid that each of these have a weak point some where in a high throughput transactional databases. I dont mind allocating disk space but CPU cycles must be minimal to identify the row (and column in case of update). I am worried that Flash back queries generate lots of data and requires twice the log space ?. Can we delete the flash back logs once i figure out the records that changed - assuming I use these logs for this purpose only ?
Thanks for any help
Vissu
I'm not sure exactly what sort of 10 minute window we're talking about... Is that the maximum amount of time between a change being applied on the database and that change getting reflected in the application's cache?
If we're talking about a J2EE app, presumably we have multiple servers involved. On which servers are you concerned about consuming CPU cycles? Are we trying to make things easy on the database? Or easy on the middle tier?
The simplest solution (assuming your application has already been built) would likely be to use Streams (which is built on top of LogMiner) to capture the changes, package them up, and send them to the middle tier via JMS, then to have the middle tier process those changes. Assuming you're concerned about the load on the database server, you can offload the LogMiner task to another system, putting very little load on the database (assuming it's already in ARCHIVELOG mode, which all production OLTP systems should be)
If you're just in the design stages, I'd be tempted to deploy TimesTen to the app servers and use that to manage the caching. It probably puts a bit more strain on the database server, but it also simplifies the architecture tremendously which probably pays dividends down the line.
Justin
Similar Messages
-
Change the way month or year page flips in Lion?
As nits go, this is pretty nitty :-) In Lion, iCal changed the way month (and year) views changed the way they moved. Instead of just a straight transition to the next month or year, some wise guy decided to transition with a lovely (not) animation that sets off my vertigo whenever the darn page moves from month to month.
Is there a way to revert to the previous behavior without any animation? I want a straight transition, with no flipping. I would love a classic view of iCal. I realize I am a traditionalist, a Luddite, if you will, but I dislike the colors and the current UI. I had more control over what I saw with the old iCal. I have less control now. I really dislike animation I can't control. Anyone know what I can do about that?
Thanks, JohannaThose dates are read from the Capture Date filed that is in the acutal file. You can make the changes you want if you export those files to a folder on the desktop, run a 3rd party application on them and then reimport into iPhoto. Then delete the original, bad dated files.
The application that I use to batch change the dates is PhotoInfo. It will allow you to change any one or more of the date and time values by a given amount. -
Trigger based approach to capture changes in OLTP DB
I have this scenario where we need to capture changes (updates/deletes) to transaction tables in our OLTP database. The changes will be populated to temp tables and then moved to Hadoop for analysis. The OLTP table record huge volume of data on a daily basis, some tables having thousands of updates every day and some having a few million.
We have thought of using a trigger-based approach, where a trigger in the OLTP tables will take care of inserting records in the temp tables. It will be a "After Update and Delete" trigger as:
IF updating THEN
BEGIN
insert into temp table // insert statement here
EXCEPTION
WHEN others THEN
null;
END;
ELSIF DELETING THEN
BEGIN
insert into temp table //insert statement here
EXCEPTION
WHEN others THEN
null;
END;
END IF;
END;Please let me know if this is an acceptable approach. Are there any better ways of doing the same?
I am using Oracle 11g r1.>
So, as a test, I created a table called test, with a PK on col1.
Then, I created a MV on it,
Next, I inserted 2 records, committed, and deleted 1 record, and did commit.
The RUPD$_TEST is empty, but the MLOG$_TEST has data.
>
The RUPD$_TEST is a temporary table that Oracle creates and uses with primary key based MV logs. As a temporary table you won't see data in it unless your session puts data there.
>
But how do I read this data?
>
What data? You didn't log any data to read.
Here is an example that doesn't even use a primary key - just a plain old table with familiar data.
create table emp_copy as select * from emp
create materialized view log on emp_copy
with rowid, sequence (empno, ename, job, mgr, hiredate, sal, comm, deptno)
including new values;
insert into emp_copy (empno, ename) values (9999, 'test')
delete from emp_copy where empno = 7369
update emp_copy set job = 'NEW JOB' where empno = 7499
select * from mlog$_emp_copy
EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO M_ROW$$ SEQUENCE$$ SNAPTIME$$ DMLTYPE$$ OLD_NEW$$ CHANGE_VECTOR$$ XID$$
9999 test AAAUayAAEAAAAJVAAO 1 1/1/4000 I N FEFF 844454994905658
7369 SMITH CLERK 7902 12/17/1980 30 20 AAAUayAAEAAAAJVAAA 2 1/1/4000 D O 0000 844454994905658
7499 ALLEN SALESMAN 7698 2/20/1981 1600 300 30 AAAUayAAEAAAAJVAAB 3 1/1/4000 U U 0800 844454994905658
7499 ALLEN NEW JOB 7698 2/20/1981 1600 300 30 AAAUayAAEAAAAJVAAB 4 1/1/4000 U N 0800 844454994905658SNAPTIME$$ shows the SEQUENCE of events.
XID$$ shows the transaction id
DMLTYPE$$ I/D/U for Insert/Update/Delete
For the Insert you can see the new EMPNO and ENAME
For the Delete you can see the values that were in the deleted row
For the Update you can see the old job 'SALESMAN' in sequence #3 and the new job 'NEW JOB' in sequence #4
Just pull the data that you haven't mined/processed yet based on SNAPTIME$$ and then delete those rows when you are done with them.
It took a few minutes to create that example and start collecting change data. Oracle will collect the changes automatically. All the user needs to do is manage the MV Log to keep it from growing indefinitely.
For some uses you may not need the actual column values; maybe you just want to know what ROWS in the table changed. This modified example uses ROWID but you could use primary key instead.
create materialized view log on emp_copy
with rowid, sequence;
insert into emp_copy (empno, ename) values (9999, 'test')
delete from emp_copy where empno = 7369
update emp_copy set job = 'NEW JOB' where empno = 7499
select * from mlog$_emp_copy
M_ROW$$ SEQUENCE$$ SNAPTIME$$ DMLTYPE$$ OLD_NEW$$ CHANGE_VECTOR$$ XID$$
AAAUa3AAEAAAR+mAAA 5 1/1/4000 I N FEFF 1.68898729922029E15
AAAUa3AAEAAAR+jAAA 6 1/1/4000 D O 0000 1.68898729922029E15
AAAUa3AAEAAAR+jAAB 7 1/1/4000 U U 0800 1.68898729922029E15There are the ROWIDs for the rows that have changed. The row for the delete MAY not be in the table anymore. The original row with that ROWID was deleted (SNAPTIME$$ = 6) but there could be a later INSERT whose row now has that ROWID. And if ROW MOVEMENT is enabled the rowid of a row could change.
For that last example you would access the MASTER table (emp_copy) and pull rows based on ROWID. Those are the rows that have 'changed' since the last time you mined the MV log.
I don't know how it can get any simpler than that! ;) -
Capture change on certain columns only in ODI,
Hi experts:
Need you help to suggest the proper way to meet the following requirements.
Requirements”
(a) Source– Oracle 11g, need to track changes to 2 columns only - [Table A, Column A1] & [Table B, Column B1]. These 2 columns may be changed many times during a day.
(b) Every 4 hours, based on what are captured from (a), find out the distinct rows changed (i.e. based on primary keys of Table A, Table B) , and only ship the most current row images for these rows to target
(c) Target – Sybase, meet to perform transformations on the 2 column values (i.e. Column A1, Column B1) to map to target table [Table C, Column C1]
From my limited understanding, I think trigger-based CDC is to be used. However, not sure how to do this , especially (a) and (b). For example, where do we configure trigger logic to achieve (a) from the ODI studio ?
Your help is much appreciated.
RegardsHi,
887999 wrote:
Hi experts:
Need you help to suggest the proper way to meet the following requirements.
Requirements”
(a) Source– Oracle 11g, need to track changes to 2 columns only - [Table A, Column A1] & [Table B, Column B1]. These 2 columns may be changed many times during a day.
(b) Every 4 hours, based on what are captured from (a), find out the distinct rows changed (i.e. based on primary keys of Table A, Table B) , and only ship the most current row images for these rows to target-- Check out the JV$ and JVD$ views that are created when you start the ODI Journal, they do exactly what you have described in (b) - You get the latest update based on the SCN number when they occured.
(c) Target – Sybase, meet to perform transformations on the 2 column values (i.e. Column A1, Column B1) to map to target table [Table C, Column C1]To be honest the target is irrelvant, you just need to decide if you want to do the transformation on the source on the way out of oracle (set staging area different to target and choose your Source logical schema).
I would design an interface that uses Table A and Table B as the sources, do the join , transformation on Oracle and map to Target Table C, choose your staging area based on where you want the joins / transformation to take place, pick a Knowlege module based on Technology and how you want to update the target.
>
From my limited understanding, I think trigger-based CDC is to be used. However, not sure how to do this , especially (a) and (b). For example, where do we configure trigger logic to achieve (a) from the ODI studio ?You can use Synchronous (Trigger based) or Asynchronous (Logminer / Streams based) to perform what you want, see this nice guide on setting up CDC and consuming the changes :
http://soainfrastructure.blogspot.ie/2009/02/setting-up-oracle-data-integrator-odi.html
If your friendly with your source system DBA then I prefer Asynchronous CDC , its less intrusive than triggers, it does however need a bit of knowledge on how to monitor it, Metalink / Support has plenty of info.
>
Your help is much appreciated.You welcome, have a play with it in a demo environment and get a feel for how you consume the captured changes (Lock Subscriber, Extend Window, Consume, Purge + Unlock, Loop) etc.
The guide I've linked to uses an ODI WAit for Data to trigger the consumption of changes, you have stated every 4 hours so I would skip the ODI wait for data and simply schedule your package to run every 4 hours. -
Capture Changes from Sql Server using Oracle Streams - Destination Oracle
Is it possible to capture changes made to tables in Sql Server database and propagate the changes to Oracle Database using Oracle Streams and Heterogeneous Gateway. I see plenty of information about pushing data from Oracle to Sql server, but I haven't been able to find much information about going the other way. Currently we are using sql server 2005 replication to accomplish this. We are looking into the possibility of replacing it with streams.
the brief understanding i have is that there is nothing out of the tin that Oracle provides to stream between SQL Server and Oracle. The senario is documented in Oracle docs however and says you need to implement the SQL Server side to grabe changes and submit to Oracle stream queues.
i'm sure i've seen third parties who sell software to do this.
If you know otherwise please let me know. Also wasn;t aware one could push from SQL Server to Oracle. Is this something only avail in SQL Server 2005 or does 200 also have it? How are you doing this?
Cheers -
Hello all,
I have written a program such that, output of program is table which captures changes for a table say J_3avasso from cdhdr and cdpos .
Now my problem is , new record has been created and changes has been done in same client, but i could nt able to see
values for it in CDHDR.
Could you please suggest solution for this
Regards
BalaHi Bala,
athough peoplem may condier this beeing a basic question, let me ask:
What did your [GOOGLE SEARCH|http://www.google.cn/search?hl=de&source=hp&q=CDHDRnotcapturing+changes&btnG=Google-Suche&meta=] result tell you?
OK, check table in dictionary, technical settings, Log data changes flag, press F1.
The other way is if a change object has been created and the transaction creates change documents.
Regards,
Clemens -
They changed the way we see our appointments. That's awful! Is there a way to chnage back to the "old" way when you could see all your appointments at a glance????
Maybe I'm fool, but how I get the "day view"? I have three options on the top line, an icon that seems a computer (a square and two lines), a magnifying glass to search a especific item and a "plus" to add appointments. The "day view" isn't any more there - that's the point!!! Using the first icon a get the full month-that uses 2/3 of the screen- and two lines of appointments...To see the rest I have to slide up and down the two lines taking care to not touch the month...
-
Hi
I have backed up itunes by copying my itunes file to an external hard drive. I try to keep my large media files (like TV shows) in a separate folder so that I can keep my hard drive on my macbook from filling up. Problem is, apple has changed the way Itunes is organized and backs up over the years. Now I am having a hard time keeping straight what is where. When I go to support, it has different instructions for the various versions of itunes. I am using version 10 now but it seems like I can't see all my files within the itunes music or media folder. I want to get everything in one place so that I can easily back up, regardless of which version of itunes I am running at the time.
I have selected to let itunes organize my files. It keeps all the files I have purchased in one place apparently, but items I have imported don't seem to always be there.
How can I best sort this out?I have, but if I don't have that particular external hard drive connected when my time machine backup hard drive is connected, then I don't think it is all backed up together. It is getting too complicated to get out all this equipment every time I want to buy some music.
-
How do I change the way that my company name appears when googled?
How do I change the way that my company name appears when it is googled? Currently it only shows up as part of my company name followed by the website listed below and then the start of a description. I would like to make it so my entire company name show up as the title when searched. I used iWeb to create the site. Please Help! Thanks:)
Put the whole company name in the title tag...
http://www.iwebformusicians.com/Search-Engine-Optimization/Tags.html -
How to capture changed data at ALV screen
Hi ALV Experts,
I am using Following method for ALV display :-
CALL METHOD g_alv_grid_0200->set_table_for_first_display
EXPORTING
i_bypassing_buffer = 'X'
is_layout =
it_toolbar_excluding =
CHANGING
it_outtab =
it_fieldcatalog =
CALL METHOD g_alv_grid_0200->set_ready_for_input
EXPORTING
i_ready_for_input = '1'.
Now I am changing few values in ALV screen and capturing through following Method :-
CALL METHOD g_alv_grid_0200->check_changed_data
IMPORTING
e_valid = l_valid.
THIS ABOVE METHOD STRANGELY CAPTURES CHANGES IN QUANTITY AND DATE FIELDS ONLY AND NOT IN CHARATER FIELDS.
Can somebody suggest how to capture changes of character fields also.
Thanks in Advance,
ChandanHi,
Check whether the following logic helps.
p_er_data_changed TYPE REF TO cl_alv_changed_data_protocol.
DATA :
lwa_mod_cell TYPE lvc_s_modi,
lwa_mod TYPE ty_mod.
LOOP AT p_er_data_changed->mt_good_cells INTO lwa_mod_cell.
lwa_mod-row = lwa_mod_cell-row_id.
APPEND lwa_mod TO i_mod.
ENDLOOP.
Regards,
Lohitha -
Fixing bluetooth in windows 8.1 and changing the way windows update works
Hello, this is a message for Microsoft, can you please fix the Bluetooth in the setting in Windows 8.1. Because I can not connect my devices on it to play music. Also, my second suggestion would be for Windows update. First of all,
Windows tells me that it will install update automatically but It's not true. Everyday I check to see if there is an update on the pc. So change the way windows update works because It is a bit confusing. In other words, try to do like windows 7 if you
can, or if not It is not a big deal. But a least try to have a solution for this issue and the Bluetooth also.
Thank you.
Hi Ariel,
The MSDN Windows Store apps forums are for developers to discuss writing their own Windows Store apps.
For help with using Windows please post in the appropriate forum for your OS on
http://answers.microsoft.com .
--Rob -
Help! Batch Capture Changes My Logged Clips' Media Start & Media End
I'd be very grateful for any help on this.
I log a tape with about 40 clips ranging from 30 seconds to 4 minutes. I select the clips and start a batch capture. Everything appears to be capturing normally and I receive no error messages or dropped frame warnings - after my last clip is captured I get a "Successfully Captured" message.
However, as soon as I click "Finished," all of my clip media start's, media end's, and durations in the log window arbitrarily change to weird times. All my media start points are 26 frames too early (I've tripple-checked that "Add Handles" is indeed deactivated), and the media end points make no sense at all.
A clip that was once 44 seconds long is now only 12 seconds long, the media end point moved a whopping 34:03 too early. Another clip will become 2 seconds longer, the media start point still shifted back 26 frames, but with the media end now 31 frames too late. Checking the original files on my drive confirms that only these weird, unusable segments are what has been captured.
What complicates things is that I just discovered today that if I capture locally to my internal hard disk, the clips capture normally (the times remain unchanged). But if I set my capture scratch to my external hard drive (a LaCie F800 2-TB RAID-5 using Firewire-800), that's when the clips start changing themselves. I ran disk utility and found no errors on the drive, and have never had a problem with it or any part of my configuration until the last month or so... the only changes to my system being the regular use of Software Update.
On top of that, my colleague is experiencing a similar problem with batch capture changing his clip times, even though he is using a different computer (single-processor G5), a different video deck (Sony DHR-1000), and a different storage medium (Apple Xserve RAID).
So my question is: What is happening! The only thing I can think of is that perhaps some update has a bug - but I can't find anyone with a similar problem online.
Any help on this would be extremely appreciated! Thank you in advance for your advice.
Dual 2.3GHz PowerPC G5 Mac OS X (10.4.4) Deck: JVC CU-VH1 / Storage: LaCie F800 2-TB RAID-5
Dual 2.3GHz PowerPC G5 Mac OS X (10.4.4)Here's an update on this. Both my colleague and I have confirmed that if we capture to an internal or external drive that is not a RAID FCPro batch captures just fine.
But, if the drive we are capturing to is a RAID (in my case a LaCie Biggest F800 and his case an Apple Xserve), then we get dropped frame reports and/or clip in's and out's changing after the batch capture is complete.
We even tried reformatting one of the xServe's and rebuilding it as non-journaled, it didn't solve the problem.
It seems impossible that we're the only ones to be experiencing this problem... does anybody have any help or suggestions!? -
Recent software upgrades has changed the way my iPhone syncs with my macbook and items entered on my iPhone are no longer uploaded to my macbook. I've tried to fix it and seem to have made things worse, my iPhone now has a long list of calanders that I don't want and can't delete and nothing is uploading to my macbook.
You need to look at Mac's iTunes, check > System Preferences > iCloud.
If you use iCloud for Calendar, Contacts then turn off those items in iTunes, Info tab with your iPhone connected and selected. -
Has Mozilla changed the way Firefox displays @font-face enbeded fonts? The site I'm codeing works in Safari, Chrome, and IE, but Firefox 8.0.1 doesn't display the embeded font.
No problems here with the H1 headers.
Reload web page(s) and bypass the cache.
* Press and hold Shift and left-click the Reload button.
* Press "Ctrl + F5" or press "Ctrl + Shift + R" (Windows,Linux)
* Press "Cmd + Shift + R" (MAC) -
Is sql server using trigger to capture change data in SQL server CDC?
hi all,
what is the Architecture of SQL server change data capture(CDC)? is sql server using trigger to capture change data in SQL server CDC? for example Change Data Capture (CDC) in SQL Server 2008 using we can capture change Data records INSERTs, UPDATEs, and
DELETEs applied to SQL Server tables.so my question is SQL server using triggers to capture change data like INSERT TRIGGER, UPDATE TRIGGER and DELETED TRIGGER using each tables after enabled CDC...?
Thanks
Tharindu Dhaneenja
Tharindu Dhaneenja http://dhaneenja.blogspot.comIt is not using triggers. The source is the transaction log.
Related link:
Basics of Change Data Capture
...The source of change data for change data capture is the SQL Server transaction log. As inserts, updates, and deletes are applied to tracked source tables, entries that describe those
changes are added to the log. The log serves as input to the change data capture capture process. This reads the log and adds information about changes to the tracked table’s associated change table. Functions are provided to enumerate
the changes that appear in the change tables over a specified range, returning the information in the form of a filtered result set. The filtered result set is typically used by an application process to update a representation of the source in some external
environment. ...
Kalman Toth, SQL Server & Business Intelligence Training; SQL Server 2008 Training
Maybe you are looking for
-
Attachments not opeing defaulting to video
attachments not opeing defaulting to video what can i do to open my email attachments
-
How to make like Spry Products demo using Flash CS4 / xml?
hello there! i wonder is it possible to make something similar to Spry Products Demo using Flash CS4 for front end and xml for back end data. take a look at this draft image. If it can be done, can someone guide me please. I am super new to AS2/3 - P
-
IOS 7 iTunes Store keeps crashing any advice to help?
My iTunes keeps crashing every time I try to open it I have rebooted it by holding the home button and lock button but it will not do anything to help and I have tried to turn it off and on about 10 times and plugged it into my computer and updated m
-
Macbook pro retina running suddently running slow
Problem : Suddently my Retina MBP is extremely slow! (I see the spinning rainbow cursor very often, in the start i never saw it. There is generally a lot of delays.) What i`ve recently changed: I think it actually all started after i had bought a WD
-
Satellite L300-22L - Which RAM for upgrade?
Hello. I have Satellite L300-22L notebook. Can you tell me which is the best RAM for my system in speed and capacity?