Moving the 80 Million records from Conversion database to System Test database (Just for one transaction table) taking too long.

Hello Friends,
The background is I am working as conversion manager and we move the data from oracle to SQL Server using SSMA and then we will apply the conversion logic and then move the data to system test ,UAT and Production.
Scenario:
Moving the 80 Million records from Conversion database to System Test database (Just for one transaction table) taking too long. Both the databases are in the same server.
Questions are…
What is best option?
IF we use the SSIS it’s very slow and taking 17 hours (some time it use to stuck and won’t allow us to do any process).
I am using my own script (Stored procedure) and it’s taking only 1 hour 40 Min. I would like know is there any better process to speed up and why the SSIS is taking too long.
When we move the data using SSIS do they commit inside after particular count? (or) is the Microsoft is committing all the records together after writing into Transaction Log
Thanks
Karthikeyan Jothi

http://www.dfarber.com/computer-consulting-blog.aspx?filterby=Copy%20hundreds%20of%20millions%20records%20in%20ms%20sql
Processing
hundreds of millions records can be done in less than an hour.
Best Regards,Uri Dimant SQL Server MVP,
http://sqlblog.com/blogs/uri_dimant/
MS SQL optimization: MS SQL Development and Optimization
MS SQL Consulting:
Large scale of database and data cleansing
Remote DBA Services:
Improves MS SQL Database Performance
SQL Server Integration Services:
Business Intelligence

Similar Messages

Select Data from BSIS table taking too long

Hi
I have to develop a report to give the details of Extended Withholding Tax (EWT) for a list of Expenses GL.
Each expense GL is linked to another Gl which is the EWT Tax GL Account. This is maintained in a Ztable.
I havae wirtten the following code. It takes a lot of time to extract a data.
This give me the GL i require
SELECT * FROM ZSECCO_GL_EWT INTO CORRESPONDING FIELDS OF TABLE IT_GL
WHERE BUKRS = P_BUKRS
AND WT_QSCOD = P_QSCOD.
Then I select only the distinct documents No, from BSIS table fro the hkonts in the above internal table
SELECT DISTINCT BUKRS GJAHR BELNR FROM BSIS INTO CORRESPONDING FIELDS OF
TABLE IT_BSIS_GL
FOR ALL ENTRIES IN IT_GL
WHERE BUKRS = P_BUKRS AND HKONT = IT_GL-HKONT AND GJAHR = P_GJAHR.
Here I once again select the document details based on the document No. from above internal table
This query takes a lot of time
SELECT * FROM BSIS INTO CORRESPONDING FIELDS OF TABLE IT_BSIS
FOR ALL ENTRIES IN IT_BSIS_GL
WHERE BUKRS = P_BUKRS AND GJAHR = P_GJAHR
AND BELNR = IT_BSIS_GL-BELNR.
Please Help

Hi,
Check note 992803; it could be that there is insufficient or missing index for BSIS table.
Regards,
Eli

How to handle an update of the same record from two different user JSP

how do you handle an update of the same record from two different users in JSP... if one person updates the record from one drop downs should be updated as well.. is the possible.

I'm not sure whether I understand your question. If one user changes a record then you would like other users to see those changes. Normally those changes will be visible to other users when they refresh their browser, after the first user has committed changes to the record. I you want to be sure that the same row isn't updated at the same time by to different user, you need to configure some locking (pessimistic locking) - this depends of what technology you use. In EJB3 pessimistic locking is performed by adding a version to every entity object. If you are using ADF, the framework is able to handle either pessimistic or even optimistic locking.
If you want the changed row to be updated in other users browsers without any user interaction by these users, you should take a look at Reverse Ajax (ex. DWR http://ajaxian.com/archives/reverse-ajax-with-dwr) - or Ajax in general. But you will never get a realtime solution, where changes is visible in other users browsers right after each record update.

How to handle an update of the same record from two different user in JSP

how to handle an update of the same record from two different user
how do you handle an update of the same record from two different users in JSP... if one person updates the record from one drop downs should be updated as well.. is the possible.

Usually, if two users try to update the same row at the same time, you want the first to commit to succeed, and when the second commits, they should fail with the error that the row was being concurrently updated. They you may want to show them the new row values and give them the opportunity to merge their values with the new row values...
How can you achieve this? Google optimistic locking.

Selecting the last record from a database table

In my ABAP Program, I have to use a select statement to retrieve the last record from the database table with the same key. In other words, the Program will get more than one hit on the database table for the selected keys and I need to retrieve values from only the last record and not the first. I know I can use an internal table to sort the records first and then retrieve the right value.   But to make things easier, is there a SELECT statement keyword than I can use to do this in one single step? Thanks!

hi,
tables:mara.
    data: begin of it_mara occurs 0,
            matnr like mara-matnr,
            meins like mara-meins,
            mtart like mara-mtart,
            end of it_mara.
select-options:s_matnr for mara-matnr.
select matnr
          meins
          mtart
from mara
into table it_mara
where matnr in s_matnr.
if not it_mara[] is initial.
sort it_mara by matnr descending.
read table it_mara index 1.
endif.
then you get the last record of the select statement.
reward points if useful,
venkat.

How to handle the deleted records from R3

Hello,
We have created a generic data source on a database table in R3 side and now we have a case where there is a huge volume of data gets deleted and new records get updated every day.
By doing a delta load we are able to load the New records and also the changed ones but we are unable to identify the deleted records from that table and move them to BI for deleteing those records in BI also.
Can any one please suggest a solution for the handling the deleted records.
Thanks,
Ravindra.

we had the same requirement some time ago and had two option:
1. ask the R/3 development team add a deletion indicator in the table (and thus not actually deleting the record). this deletion indicator could then be used like for any other standard datasource
this option was however refused, due to huge data volume after a while
2. at the end of the load we copied the ZTABLE1 to ZTABLE2. then in the begin of the load (day after) we compare the data of table1 to table2. entries available in table2 but not in table1 are deleted, and we put a 'D'. in deletion indicator; as we only keep the deleted entries for one day, the volume of the new table is acceptable.
M.

How to delete the single record from the ODS

HI,
I want to DELETE the particular record from ODS.. how to delete that..
plz give me the solution.. it's urgent..
with regards
@jay

Hi Prakash,
You can delete a particular record from ODS by specifying the key field name in Selective deletion button available in Contents tab in the manage screen of ODS. But it will only delete the record from the active data table and not from the Change log and new data table . You should be very carefull if you are doing in Production system as there is chance of deleting other records accidentally.
Go to Selective deletion button - > Click on delete selection -> Enter the key field values of the record you want to delete-> Press save button to save as Variant -> Go back and click on selection button -> Chosse the background job type -> Click on start button.
Regards,
Prakash

Not moving the .pdf files from Drop Off Library to my custom document library in sharepoint 2013

Hi Team,
I am trying to move the .pdf files from drop-off library to custom library.
For the above requirement i am writing content organizer rule condition like (On Group: Document content types & Type: Document)
Property: Name
Operator: Ends with
value: .pdf
and Setting value in target location like
/sites/MS/ProductGuides
How to solve the issue
If help appriciated
Regards,
Dhaya

Hi Dhaya,
I created a Content Organizer rule like this:
When uploading a .pdf file to the Drop Off library, it is moved to the destination library immediately:
I suggest you create a similar rule as above for another test to see if the issue still persists.
Thanks
Patrick Liang
TechNet Community Support
Please remember to mark the replies as answers if they help, and unmark the answers if they provide no help. If you have feedback for TechNet Support, contact
[email protected]

Inserting records from a txt file to a database table

I would like to know how to insert records from a file (txt) to a database table through a java application?

[BufferedReader |http://java.sun.com/javase/6/docs/api/java/io/BufferedReader.html] and PreparedStatement
IO and JDBC

Lost bookmarks, cookies and history when I moved the Folder 'Users' from the C disk to D disk. What can I do to retrieve them intact?

Lost bookmarks, cookies and history when I moved the Folder 'Users' from the C disk to D disk. What can I do to retrieve them intact?

Hi voyant
I would put the Users folder back... moving it will cause a lot of issues with applications.
Instead you can just copy individual files/folders from C:\Users\your_username\Documents,Pictures,Music.... to your D drive to save space on C.
-Curtis

Security Agent: Could not get the user record from OpenDirectory

Hi,
Two months ago I had to change my SSD due to a damage at the old one. In the Apple shop they installed OS X Lion with a image from their image-server.
After that the Mac was quite slow in booting. With the old SSD (128 GB) booting takes 25 sec now it takes 75 sec.
I tried different things to solve the problem without result. First I asked in the shop, then I googled the problem all with no real solution.
Today now I searched in the log files. I found the following:
09.02.12 11:03:07.861 SecurityAgent: Could not get the user record from OpenDirectory.
09.02.12 11:03:07.861 SecurityAgent: Will sleep 3 seconds and try again (retryCount = 10)
09.02.12 11:03:34.907 SecurityAgent: Could not get the user record from OpenDirectory.
09.02.12 11:03:34.907 SecurityAgent: Will sleep 3 seconds and try again (retryCount = 1)
09.02.12 11:03:34.907 SecurityAgent: Will sleep 3 seconds and try again (retryCount = 1)
09.02.12 11:03:38.653 WindowServer: kCGErrorFailure: Set a breakpoint @ CGErrorBreakpoint() to catch errors as they are logged.
09.02.12 11:03:58.245 SecurityAgent: User info context values set for andi
09.02.12 11:03:58.246 SecurityAgent: User info context values set for andi
09.02.12 11:03:58.356 SecurityAgent: Login Window login proceeding
09.02.12 11:03:58.356 SecurityAgent: Login Window login proceeding
This operation takes 30 secs during the boot process. Can somebody help me, to solve this problem?
Regards
Andreas
MacBook Pro, Mid 2009
2.66 GHz Intel Core 2 Duo
8GB 1067 MHz DDR3
SAMSUNG SSD 830 Series , 256 GB
OS X 10.7.3

cece2bali wrote: "it seems that your mac tries to connect to a server while booting. This action will erase all your network configuration..."
Where can I go to learn more about this 'erasure' of network configuration if the mac tries to connect to a server while booting?
I get the same error message as the OP. My account is an Admin on the machine. It's configured to automatically log in on boot -- except sometimes it fails. It has a login-item which runs a bash script that makes an https request to a nearby machine. Maybe I should make this script sleep a while so the https action happens after the boot sequence is done? (unlike SmashHuevo, my user needs to be an Admin)

[svn:fx-trunk] 10607: Moved the 'breakOpportunity' style from BasicInheritingTextStyles.as to AdvancedInheritingTextStyles.as , so it is declared on fewer components now.

Revision: 10607
Author:   [email protected]
Date:     2009-09-25 16:08:48 -0700 (Fri, 25 Sep 2009)
Log Message:
Moved the 'breakOpportunity' style from BasicInheritingTextStyles.as to AdvancedInheritingTextStyles.as, so it is declared on fewer components now. As far as I can tell, it is mainly useful in rich text.
QE notes: None
Doc notes: None
Bugs: SDK-22474
Reviewer: Ryan
Tests run: ant checkintests
Is noteworthy for integration: No
Ticket Links:
    http://bugs.adobe.com/jira/browse/SDK-22474
Modified Paths:
    flex/sdk/trunk/frameworks/projects/spark/src/spark/components/Label.as
    flex/sdk/trunk/frameworks/projects/spark/src/spark/styles/metadata/AdvancedInheritingText Styles.as
    flex/sdk/trunk/frameworks/projects/spark/src/spark/styles/metadata/BasicInheritingTextSty les.as

Gordon, it looks like its been a while since you made this post. Not sure how valid it is now...   I am particularly interested in the LigatureLevel.NONE value. It seems that it is no longer supported.
How do I turn of ligatures in the font rendering?
My flex project involves trying to match the font rendering of Apache's Batik rendering of SVG and ligatures have been turned off in that codebase. Is there any way (even roundabout) to turn ligatures off in flash?
Thanks,
Om

How do you sync contacts with gmail, not other place on computer? I don't know where the contacts came from on that are now on my new phone. There are way too many and it is not my list of contacts I have on gmail. Where else is this from?

I don't know where the contacts came from on that are now on my new phone. There are way too many and it is not my list of contacts I have on gmail. Where else is this from? I spent a LONG time on gmail going through my list of contacts. There were about 400 and I got it down to about 90. When I went to sync my new phone, ALL 400 synced.
Thank you for any help!

Sorry for the repeat...new here. OBVIOUSLY.

I have the converter to change pdf to word. Works sometime, other time says conversion is taking too long. Can I get what it has translated so I can use the parts?

I have the converter to change pdf to word. Works sometime, other time says conversion is taking too long. Can I get what it has translated so I can use the parts?

I would ask in the ExportPDF forum, Adobe ExportPDF (read only) (assuming that's the service to which you have subscribed). This is the Reader one.

HT201274 i tried to erase all content and reset settings from the device option.it is taking too long and its been 15 hours but not yet done.its showing a skeleton sigh with buffering and sometimes flipping to apple sign.

i tried to erase all content and reset settings from the device option.it is taking too long and its been 15 hours but not yet done.its showing a skeleton sigh with buffering and sometimes flipping to apple sign.

nabidhadi wrote:
.its showing a skeleton sigh with buffering and sometimes flipping to apple sign.
You iPod has been jail broken. Unfortunately you are on your own here. I can offer that you need to find out which method and software was used to JB.
Good luck...you're gonna need it....

Moving the 80 Million records from Conversion database to System Test database (Just for one transaction table) taking too long.

Similar Messages

Maybe you are looking for