Purging the data older than 1 year

I have a requirement to purge data older than 1 year.
Table has 5 billion rows and delete about 4 billion rows in batches.
there is no primary key or indexes on table.
There is column Datetime and i created nonclustered index on table and planning to purge data based on that column.
Any help in preparing the script to purge data in batches older than 1 year.
Thanks,
Ron.

As noted above, it is better to build a new table for what you want to keep using SELECT-INTO:
http://www.sqlusa.com/bestpractices/select-into/
Kalman Toth Database & OLAP Architect
SELECT Video Tutorials 4 Hours
New Book / Kindle: Exam 70-461 Bootcamp: Querying Microsoft SQL Server 2012

Similar Messages

Purge data older than an year from multiple tables

Hello all,
I have a requirement to purge data older than 1 year.
These are multiple related table with 300 million rows in each and need to delete about 1 million rows. I need to archive the data to a different database for some tables and then delete and for the other tables I dont need to archive but just delete.
All these tables are being used in transactional replication.
I need to join these tables to delete data as some of them wont have a timestamp column
I was thinking to run the purge job based off 2 sql agent jobs one which will archive and the delete and the other which would only delete where archiving is not required. All these tables use a lot of guid based columns.
What would be the best strategy to purge the data? My concern is the transaction replication. Any help in preparing the script to purge data in batches older than 1 year.
Thanks

Hopefully this example will help you:
DECLARE @transactions TABLE (transactionID INT IDENTITY, transactionDate DATE, clientID INT)
DECLARE @clients TABLE (clientID INT IDENTITY, clientName VARCHAR(30))
INSERT INTO @clients (clientName) VALUES ('James'),('Jean-Luc'),('Jonathon'),('Katheryn'),('Ben')
INSERT INTO @transactions (transactionDate, clientID) VALUES ('2000-01-01',1),('2000-01-01',2),('2000-01-01',3),('2000-01-01',4),('2000-01-01',5),('2015-01-01',4),('2015-01-01',5)
DELETE c FROM @clients c
LEFT OUTER JOIN @transactions t
ON c.clientID = t.clientID
AND t.transactionDate > DATEADD(YEAR,-1,CURRENT_TIMESTAMP)
WHERE t.clientID IS NULL
DELETE t FROM @transactions t
WHERE t.transactionDate < DATEADD(YEAR,-1,CURRENT_TIMESTAMP)
SELECT *
FROM @clients
SELECT *
FROM @transactions
Of course, you're going to want to use similar logic to move the records before you delete them, but you can specify the table to delete by its alias in the delete statement.

How to delete data older than three years from DSO

Hi all,
We needs to keep historical data in the DSO only for three years. One way of doing this is by selective deletion we can delete data older than three years but we want to make it authomated. Is it possible to add a process or ABAP program to delete data older than three years.
If it is p[ossible by ABAP code then can you please give me sample code.
Many Thanks in advance.
Harman

Some time for in case of huge volume of data selective deletion is not works, Try for "prunning" option.
It's work as follows
1)     Create DSO which is same replica of your Source DSO.
2)     Create two transformation
     a) from Source DSO to Copy DSO
     b) Another is from Copy DSO to Source DSO
3)     Transprort Last 3 years records to Copy of Source DSO using DTP Filter range for time.
4)     Once data loading from Source DSO to Copy DSO get completed delete all data from Source DSO.
5)     Now load data from Copy DSO to Source DSO.
6)     After complete loading delete entire data from Source DSO.
I use prunning process to one of my client and it's work perfectly.
Regards,
Sushant
+919820849482

Query taking long time for EXTRACTING the data more than 24 hours

Hi ,
Query taking long time for EXTRACTING the data more than 24 hours please find the query and explain plan details below even indexes avilable on table's goe's to FULL TABLE SCAN. please suggest me.......
SQL> explain plan for select a.account_id,round(a.account_balance,2) account_balance,
2 nvl(ah.invoice_id,ah.adjustment_id) transaction_id,
to_char(ah.effective_start_date,'DD-MON-YYYY') transaction_date,
to_char(nvl(i.payment_due_date,
to_date('30-12-9999','dd-mm-yyyy')),'DD-MON-YYYY')
due_date, ah.current_balance-ah.previous_balance amount,
decode(ah.invoice_id,null,'A','I') transaction_type
3 4 5 6 7 8 from account a,account_history ah,invoice i_+
where a.account_id=ah.account_id
and a.account_type_id=1000002
and round(a.account_balance,2) > 0
and (ah.invoice_id is not null or ah.adjustment_id is not null)
and ah.CURRENT_BALANCE > ah.previous_balance
and ah.invoice_id=i.invoice_id(+)
AND a.account_balance > 0
order by a.account_id,ah.effective_start_date desc; 9 10 11 12 13 14 15 16
Explained.
SQL> select * from table(dbms_xplan.display);
PLAN_TABLE_OUTPUT
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)|
| 0 | SELECT STATEMENT | | 544K| 30M| | 693K (20)|
| 1 | SORT ORDER BY | | 544K| 30M| 75M| 693K (20)|
|* 2 | HASH JOIN | | 544K| 30M| | 689K (20)|
|* 3 | TABLE ACCESS FULL | ACCOUNT | 20080 | 294K| | 6220 (18)|
|* 4 | HASH JOIN OUTER | | 131M| 5532M| 5155M| 678K (20)|
|* 5 | TABLE ACCESS FULL| ACCOUNT_HISTORY | 131M| 3646M| | 197K (25)|
| 6 | TABLE ACCESS FULL| INVOICE | 262M| 3758M| | 306K (18)|
Predicate Information (identified by operation id):
2 - access("A"."ACCOUNT_ID"="AH"."ACCOUNT_ID")
3 - filter("A"."ACCOUNT_TYPE_ID"=1000002 AND "A"."ACCOUNT_BALANCE">0 AND
ROUND("A"."ACCOUNT_BALANCE",2)>0)
4 - access("AH"."INVOICE_ID"="I"."INVOICE_ID"(+))
5 - filter("AH"."CURRENT_BALANCE">"AH"."PREVIOUS_BALANCE" AND ("AH"."INVOICE_ID"
IS NOT NULL OR "AH"."ADJUSTMENT_ID" IS NOT NULL))
22 rows selected.
Index Details:+_
SQL> select INDEX_OWNER,INDEX_NAME,COLUMN_NAME,TABLE_NAME from dba_ind_columns where
2 table_name in ('INVOICE','ACCOUNT','ACCOUNT_HISTORY') order by 4;
INDEX_OWNER INDEX_NAME COLUMN_NAME TABLE_NAME
OPS$SVM_SRV4 P_ACCOUNT ACCOUNT_ID ACCOUNT
OPS$SVM_SRV4 U_ACCOUNT_NAME ACCOUNT_NAME ACCOUNT
OPS$SVM_SRV4 U_ACCOUNT CUSTOMER_NODE_ID ACCOUNT
OPS$SVM_SRV4 U_ACCOUNT ACCOUNT_TYPE_ID ACCOUNT
OPS$SVM_SRV4 I_ACCOUNT_ACCOUNT_TYPE ACCOUNT_TYPE_ID ACCOUNT
OPS$SVM_SRV4 I_ACCOUNT_INVOICE INVOICE_ID ACCOUNT
OPS$SVM_SRV4 I_ACCOUNT_PREVIOUS_INVOICE PREVIOUS_INVOICE_ID ACCOUNT
OPS$SVM_SRV4 U_ACCOUNT_NAME_ID ACCOUNT_NAME ACCOUNT
OPS$SVM_SRV4 U_ACCOUNT_NAME_ID ACCOUNT_ID ACCOUNT
OPS$SVM_SRV4 I_LAST_MODIFIED_ACCOUNT LAST_MODIFIED ACCOUNT
OPS$SVM_SRV4 I_ACCOUNT_INVOICE_ACCOUNT INVOICE_ACCOUNT_ID ACCOUNT
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_ACCOUNT ACCOUNT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_ACCOUNT SEQNR ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_INVOICE INVOICE_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_ADINV INVOICE_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_CIA CURRENT_BALANCE ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_CIA INVOICE_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_CIA ADJUSTMENT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_CIA ACCOUNT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_LMOD LAST_MODIFIED ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_ADINV ADJUSTMENT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_PAYMENT PAYMENT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_ADJUSTMENT ADJUSTMENT_ID ACCOUNT_HISTORY
OPS$SVM_SRV4 I_ACCOUNT_HISTORY_APPLIED_DT APPLIED_DATE ACCOUNT_HISTORY
OPS$SVM_SRV4 P_INVOICE INVOICE_ID INVOICE
OPS$SVM_SRV4 U_INVOICE CUSTOMER_INVOICE_STR INVOICE
OPS$SVM_SRV4 I_LAST_MODIFIED_INVOICE LAST_MODIFIED INVOICE
OPS$SVM_SRV4 U_INVOICE_ACCOUNT ACCOUNT_ID INVOICE
OPS$SVM_SRV4 U_INVOICE_ACCOUNT BILL_RUN_ID INVOICE
OPS$SVM_SRV4 I_INVOICE_BILL_RUN BILL_RUN_ID INVOICE
OPS$SVM_SRV4 I_INVOICE_INVOICE_TYPE INVOICE_TYPE_ID INVOICE
OPS$SVM_SRV4 I_INVOICE_CUSTOMER_NODE CUSTOMER_NODE_ID INVOICE
32 rows selected.
Regards,
Bathula
Oracle-DBA

I have some suggestions. But first, you realize that you have some redundant indexes, right? You have an index on account(account_name) and also account(account_name, account_id), and also account_history(invoice_id) and account_history(invoice_id, adjustment_id). No matter, I will suggest some new composite indexes.
Also, you do not need two lines for these conditions:
and round(a.account_balance, 2) > 0
AND a.account_balance > 0
You can just use: and a.account_balance >= 0.005
So the formatted query isselect a.account_id,
       round(a.account_balance, 2) account_balance,
       nvl(ah.invoice_id, ah.adjustment_id) transaction_id,
       to_char(ah.effective_start_date, 'DD-MON-YYYY') transaction_date,
       to_char(nvl(i.payment_due_date, to_date('30-12-9999', 'dd-mm-yyyy')),
               'DD-MON-YYYY') due_date,
       ah.current_balance - ah.previous_balance amount,
       decode(ah.invoice_id, null, 'A', 'I') transaction_type
from account a, account_history ah, invoice i
where a.account_id = ah.account_id
   and a.account_type_id = 1000002
   and (ah.invoice_id is not null or ah.adjustment_id is not null)
   and ah.CURRENT_BALANCE > ah.previous_balance
   and ah.invoice_id = i.invoice_id(+)
   AND a.account_balance >= .005
order by a.account_id, ah.effective_start_date desc;You will probably want to select:
1. From ACCOUNT first (your smaller table), for which you supply a literal on account_type_id. That should limit the accounts retrieved from ACCOUNT_HISTORY
2. From ACCOUNT_HISTORY. We want to limit the records as much as possible on this table because of the outer join.
3. INVOICE we want to access last because it seems to be least restricted, it is the biggest, and it has the outer join condition so it will manufacture rows to match as many rows as come back from account_history.
Try the query above after creating the following composite indexes. The order of the columns is important:create index account_composite_i on account(account_type_id, account_balance, account_id);
create index acct_history_comp_i on account_history(account_id, invoice_id, adjustment_id, current_balance, previous_balance, effective_start_date);
create index invoice_composite_i on invoice(invoice_id, payment_due_date);All the columns used in the where clause will be indexed, in a logical order suited to the needs of the query. Plus each selected column is indexed as well so that we should not need to touch the tables at all to satisfy the query.
Try the query after creating these indexes.
A final suggestion is to try larger sort and hash area sizes and a manual workarea policy.alter session set workarea_size_policy = manual;
alter session set sort_area_size = 2147483647;
alter session set hash_area_size = 2147483647;

Routine to convert the date into month/year

Hi friends,
I have requirement like I have 0billdate, but i want only month and year in that. So i have created a z object for that taking reference as 0calmonth. I think the format is yyyymm.
So now I want to write a routine which converts the date to month/year. If anybody know this please let me know.
Regards
San Reddy

Hi
in the trnasfer rule or in the update rule select the Zinfo object to which you need to do conversion
in details select formula and create.
Select all functions and follow the steps below.
use the conversion function DATE_MONTH( )
Short Text
Function: Date -> Month
Functionality
Calculates the month from a specified date.
Requirements
The date must appear in the format yyyymmdd.
Parameters
DATE_MONTH (date)
Result
The result is calculated in the format yyyymm.
Regards
RaM

Purging the data in exchange tables in IM 3.2

Hi All,
Please sugest me on what basis we can purge the data of Exchange tables for SAP IM 3.2
Regards-
Garima
Tags edited by: Michael Appleby

Hi All,
Please sugest me on what basis we can purge the data of Exchange tables for SAP IM 3.2
Regards-
Garima
Tags edited by: Michael Appleby

Post PO & GR on the date of one year back

I want to post PO & GR on the date of one year back, is it possible?

hi
You can only post in OPEN PERIOD i.e. current open period and previous open period
If you try to post in privious year it will give you error saying posing possible only in period 09/2008 and 08/2008
Regards

How to sync calendar data older than 1 month?

How to sync data from calendar older than 1 month? (from computer to phone) Impossible to get my data from beginning of november on calendar on Iphone. What did I miss?

If you are syncing calendar events direct with Outlook via the iTunes sync process, there is a setting for this under the Info tab for your iPhone sync preferences with iTunes below the sync calendars section.

Working with Dates older than 1900-01-01

I'm curious about working with old historical dates. Crystal doesn't seem to like working with them as a valid date as far as displaying and working with them in formulas.
For simplicity, instead of using a database I tried a simple one cell input Excel spreadsheet with a combination of dates in it, some prior to 1900-01-01. The newer dates display fine, the older dates (any 1899 and before) display as blank. As well, they don't return expected results when used in formulas doing < or > comparisons to more current dates.
What's the best work-around if a person should need to create a reports with historical data containing pre-1900 dates?
Thanks! Russ.

Hi Russ,
To clarify, is this issue happening with Excel as a datasource or within Crystal Reports as well? I've found articles specific to Excel where it considers 1900-01-01 as a leap year.
I have also tested this in Crystal Reports by creating a formula:
date(1900/01/01)
Regards,
Wallie
Edited by: Wallie Cheng on Sep 24, 2008 1:15 AM
Actually my syntax was incorrect. Crystal Reports displays the date correctly with the date formatted as 1900,1,1

SRM: SC 1000017291 (PO 9000003133): FI 099 booking date older than due date

In SRM-Monitor as well as in RZ20 i got the message:
>>Shopping cart 1000017291 (PO 9000003133): FI 099 Buchungsdatum älter als Fälligkeitsdatum<<
no further description. Did you have experience with such message? Can any help.
Thanks, Isam

Hi Isam,
if you convert from german to Buchungsdatum älter als Fälligkeitsdatum english the meansing is Posting date earlier than the due date. It means that your posting date is before the due date. I think it's warning only......
Thanks
Ravi

Bug: Duplicates not detected when importing photos older than year 2001

If you import photos taken before 1st of January 2001 00:00:00, duplicates will not be detected even if you select the "Ignore duplicates" option on import. The files will be imported twice into the same location with a "-2" postfix added to their name.
Technical details (to help developers):
To detect duplicates, Lightroom uses a so called import hash (AgLibraryFile.importHash in the database) that has the following form:
<capture_time><original_filename><file_size>,
where:
<capture_time> - interval in seconds between 1st Jan, 2001 00:00:00 and capture time recorded in Exif,
<original_filename> - name of source file before import,
<file_size> - file size in bytes.
The problem here is the <capture_time> part, since it gets negative for files older than 2001, since 1st Jan 2001 is sort of considered the beginning of time (called "epoch" in computing). Lightroom seems to get confused on negative timestamps.

I will report this bug. Thanks!

Iphoto is seeing the date as a year behind

i'm using a quad with iphoto 6 and have noticed that since installing the latest patch 10.4.6, iphoto thinks its 6/8/05. the date/time is correct in date/time app, but iphoto doesnt seem to pick up the correct date. and its a pain having to manually correct this. i can't believe i'm the only one with this problem.

leQ:
Welcome to the Apple Discussions. First open one of
the images in Preview and go to the Tools->Get Info
menu option. Check the date there to make sure it
didn't get changed somehow by the system.
what's strange is that the photo was created 6/8/2005, but modified 6/8/2006.
If all's
well the try rebuilding the library (launch iPhoto
with the Command+Option keys depressed and follow the
instructions to rebuild the library) and just select
the option to rebuild the database.
i've rebuilt the library, and it still shows my 6/*/2006 pictures as being 6/*/2005 created dates.
If that doesn't clear it up the rebuild the library
with iPhoto Library Manager. It doesn't
touch your current library but creates a new one
entirely. See if it won't get iPhoto to read the
correct dates. Here's how:
Using iPhoto Library Manager to Rebuild
Your iPhoto Library
Download iPhoto Library Manager and launch.
Click on the Add Library button, navigate to
o your User/Pictures folder and select your iPhoto
Library folder.
Now that the library is listed in the left hand
d pane of iPLM, click on your library and go to the
File->Rebuild Library menu option
In the next window name the new library and
d select the location you want it to be placed.
Click on the Create button.
Note: It can take some time to create the new
library if you have a lot of photos. One user
reported that with a library of about 5,000 images it
took about 12 hours so plan ahead.
when i did what you said, i got errors with this:
MERGE:
An error has occurred while attempting to import the roll "Pictures". (* -[NSNull objectEnumerator]: selector not recognized [self = 0xa07c1218]) If you would like to continue and try to import the roll "Pictures" again, click "Continue". If you would like to continue, but skip the roll "Pictures", click "Skip". If you would like to stop the merge, click "Cancel"
clicking cancel, i got this:
* -[NSNull objectEnumerator]: selector not recognized [self = 0xa07c1218]
what's wrong with my photo library?

After purging the data..is need to rebuild the index?

Hi All,
In our production DB,Application level incidents Purging got finished.(Mean first 3 years data got purged,which is not usable by client.This is done for performance issue)
Now in this situation,Is necessary to rebuild the index?
Before that FYI,
oracle 9i(Enterprise Edition)
hp-ux b11
dbsize=30G
Application:
tool:HP-ov
Please throw your input.
Thanks in advance :-)
Regards,
DB.
Edited by: DB on May 21, 2013 11:26 PM

>
In our production DB,Application level incidents Purging got finished.(Mean first 3 years data got purged,which is not usable by client.This is done for performance issue)
Now in this situation,Is necessary to rebuild the index?
>
Please clarify.
You said you did the purge for 'performance issue' and now, after the purge, you still don't know if your performance issue went away?
If you no longer have whatever (unknown to us since you didn't post it) 'performance issue' you had then why do you need to do anything?
If you do still have a performance issue then you need to follow the same steps you should follow whenever you have a performance issue:
1. validate that you REALLY have an issue
2. determine the cause of the issue
3. identify solutions to mitigate/eliminate the issue
4. test one or two of the possible solutions
5. implement your 'best' tested solution from step #4
6. go back to step #1
Based on what you posted it sounds like you implemented a 'purge' without really knowing that it was even necessary or was even causing a performance issue.
And you make no mention at all of whether you recollected statistics after the purge was done.

Can you sort albums by release date, rather than year, in album view?

I love the new album view in iTunes 11, but the only problem I have with it is sometimes albums are in the wrong order.
For example, last year Green Day released a trilogy of albums called ¡Uno!, ¡Dos! and ¡Tré! but since they were released in the same year, they're ordered as ¡Dos!, ¡Tré, ¡Uno! when sorted by artist and then year which really doesn't sit right with me.
In short, the sorting options for albums view are very limited, is there any way to extend them?
Also I'm aware that I could, as a cheap fix, change the sorting in album to the release date backwards, but I'd rather not do that as it would make it a bit more troublesome for me to navigate my albums on my iPhone.
Thanks!

I agree that Apple should improve the year field to be an actual date.
I am not 100% happy doing this, but when you have multiple albums from a single artist for the same release year, a temporary solution is to chage the album title by adding an outline type letter before album titles for that year, so that they appear in the correct order:
David Bowie
1977
a. Low
David Bowie
1977
b. "Heroes"
etc
This could help in the short term until Apple changes that field.

Purging the data older than 1 year

Similar Messages

Maybe you are looking for