Best way to delete large number of records but not interfere with tlog backups on a schedule

Ive inherited a system with multiple databases and there are db and tlog backups that run on schedules.  There is a list of tables that need a lot of records purged from them.  What would be a good approach to use for deleting the old records?
Ive been digging through old posts, reading best practices etc, but still not sure the best way to attack it.
Approach #1
A one-time delete that did everything.  Delete all the old records, in batches of say 50,000 at a time.
After each run through all the tables for that DB, execute a tlog backup.
Approach #2
Create a job that does a similar process as above, except dont loop.  Only do the batch once.  Have the job scheduled to start say on the half hour, assuming the tlog backups run every hour.
Note:
Some of these (well, most) are going to have relations on them.

Hi shiftbit,
According to your description, in my opinion, the type of this question is changed to discussion. It will be better and 
more experts will focus on this issue and assist you. When delete large number of records from tables, you can use bulk deletions that it would not make the transaction log growing and runing out of disk space. You can
take the table offline for maintenance, a complete reorganization is always best because it does the delete and places the table back into a pristine state. 
For more information about deleting a large number of records without affecting the transaction log.
http://www.virtualobjectives.com.au/sqlserver/deleting_records_from_a_large_table.htm
Hope it can help.
Regards,
Sofiya Li
Sofiya Li
TechNet Community Support

Similar Messages

  • What is the best practice of deleting large amount of records?

    hi,
    I need your suggestions on best practice of deleting large amount of records of SQL Azure regularly.
    Scenario:
    I have a SQL Azure database (P1) to which I insert data every day, to prevent the database size grow too fast, I need a way to  remove all the records which is older than 3 days every day.
    For on-premise SQL server, I can use SQL Server Agent/job, but, since SQL Azure does not support SQL Job yet, I have to use a Web job which scheduled to run every day to delete all old records.
    To prevent the table locking when deleting too large amount of records, in my automation or web job code, I limit the amount of deleted records to
    5000 and batch delete count to 1000 each time when calling the deleting records stored procedure:
    1. Get total amount of old records (older then 3 days)
    2. Get the total iterations: iteration = (total count/5000)
    3. Call SP in a loop:
    for(int i=0;i<iterations;i++)
       Exec PurgeRecords @BatchCount=1000, @MaxCount=5000
    And the stored procedure is something like this:
     BEGIN
      INSERT INTO @table
      SELECT TOP (@MaxCount) [RecordId] FROM [MyTable] WHERE [CreateTime] < DATEADD(DAY, -3, GETDATE())
     END
     DECLARE @RowsDeleted INTEGER
     SET @RowsDeleted = 1
     WHILE(@RowsDeleted > 0)
     BEGIN
      WAITFOR DELAY '00:00:01'
      DELETE TOP (@BatchCount) FROM [MyTable] WHERE [RecordId] IN (SELECT [RecordId] FROM @table)
      SET @RowsDeleted = @@ROWCOUNT
     END
    It basically works, but the performance is not good. One example is, it took around 11 hours to delete around 1.7 million records, really too long time...
    Following is the web job log for deleting around 1.7 million records:
    [01/12/2015 16:06:19 > 2f578e: INFO] Start getting the total counts which is older than 3 days
    [01/12/2015 16:06:25 > 2f578e: INFO] End getting the total counts to be deleted, total count:
    1721586
    [01/12/2015 16:06:25 > 2f578e: INFO] Max delete count per iteration: 5000, Batch delete count
    1000, Total iterations: 345
    [01/12/2015 16:06:25 > 2f578e: INFO] Start deleting in iteration 1
    [01/12/2015 16:09:50 > 2f578e: INFO] Successfully finished deleting in iteration 1. Elapsed time:
    00:03:25.2410404
    [01/12/2015 16:09:50 > 2f578e: INFO] Start deleting in iteration 2
    [01/12/2015 16:13:07 > 2f578e: INFO] Successfully finished deleting in iteration 2. Elapsed time:
    00:03:16.5033831
    [01/12/2015 16:13:07 > 2f578e: INFO] Start deleting in iteration 3
    [01/12/2015 16:16:41 > 2f578e: INFO] Successfully finished deleting in iteration 3. Elapsed time:
    00:03:336439434
    Per the log, SQL azure takes more than 3 mins to delete 5000 records in each iteration, and the total time is around
    11 hours.
    Any suggestion to improve the deleting records performance?

    This is one approach:
    Assume:
    1. There is an index on 'createtime'
    2. Peak time insert (avgN) is N times more than average (avg). e.g. supposed if average per hour is 10,000 and peak time per hour is 5 times more, that gives 50,000. This doesn't have to be precise.
    3. Desirable maximum record to be deleted per batch is 5,000, don't have to be exact.
    Steps:
    1. Find count of records more than 3 days old (TotalN), say 1,000,000.
    2. Divide TotalN (1,000,000) with 5,000 gives the number of deleted batches (200) if insert is very even. But since it is not even and maximum inserts can be 5 times more per period, set number of deleted batches should be 200 * 5 = 1,000.
    3. Divide 3 days (4,320 minutes) with 1,000 gives 4.32 minutes.
    4. Create a delete statement and a loop that deletes record with creation day < today - (3 days ago - 3.32 * I minutes). (I is the number of iterations from 1 to 1,000)
    In this way the number of records deleted in each batch is not even and not known but should mostly within 5,000 and even you run a lot more batches but each batch will be very fast.
    Frank

  • Lookups with large number of records do not return the page

    Hi,
    I am developing an application using Oracle JHeadstart 10.1.3 Preview Version 10.1.3.0.78
    In my application I created a lookup under domains and used that lookup for an attribute (Display Type for this attribute is: dropDownList) in a group to get the translation fro this attribute. The group has around 14,800 records and the lookup has around 7,400 records.
    When I try to open this group (Tab), the progress shows that it is progressing but it does not open even after a long time.
    If I change the Display Type for the attribute from dropDownList to textInput then it works fine.
    I have other lookups with lower number of records. Those lookups work fine with dropDownList Display Type.
    Only I have this kind of problem when I have a lookup with large number of records.
    Is there any limitation of record number for lookups under Domains?
    How I can solve this?
    I need to translate the attribute (get the description from another table using the code).
    Your help would be appreciated.
    Thanks
    Syed

    We have also faced similar issue, but us, it was happening when we were using the dropDownList in a table, while the same dropDownList was working in table format. In our case the JVM is just used to crash and after google'ing it here in forums, found that it might be related to some JVM issue on Windows XP machines without Service Pack 2.
    Anyway... the workaround that we taken to get around the issue is to use LOV instead of a dropDownList in your jHeadStart.
    Hope this helps...
    - rutwik

  • Best way to handle large number video files for a project..

    Hey, I was looking at getting some insight from the community here. Bascially there is a project that is being worked on that requires large amount of footage to be sifted through that only a small percentage will be used. These are mostly HD files and while most of the footage has been watched on Quicktime with notes taken, my question is this.
    What is the best way to take only small portions of each file without having to load everything into final cut and without any loose of quality. Should I just trim and rename from Quicktime or is there an easier way?
    Reason this needs to be done this way is the smaller segments will each be sent to other editors and rather then send huge files we want to split it to smaller amounts for each editor to use.
    Thank you so much for any input regarding this, I look forward to what you have to say

    Open the clip into the viewer. Mark In and Out points on the section you want. Make it a subclip Cmd-U. Drag the subclip into the bin for the editor who needs it. Repeat.
    If you batch export from a clip there is a selection to choose whether to export the whole clip or check box to export the marked I/O.
    This does not sound like a good project on which to being learning FCP.

  • Deleting large number of records

    Hi Gurus,
    I need to delete millions of records from a table of a specific column.
    This cloumn is of type BLOB and text data in column looks like
    #29/07/2010 2:20 PM#Eevent#MyEvent#MyClass_classToSchedule=abcd
    I need to delete records based on partial value of above column(based on 'MyClass_classToSchedule=abcd' ).
    Please help me to write a query for this.
    Should I need to convert BLOB to VARCHAR2 or is there any other optimized way ?
    Note: I dont want to use 'like' and 'nvl' in the query ( as those are very costly operation).
    Thanks and Regards,
    Nitesh

    How many rows are in the table and how many do you want to delete aproximately?
    Sometimes it is better to copy the remaining rows to a temporary table and truncate the original table instead of deleting it.
    This is if the delete itself is the slow process.
    In your case also the search logic inside the blob might be slow. Is it really a blob or is it a clob?
    Some ways to speed this up have already been suggested.
    I think the oracle text index using the contains comparison is a very promising one.

  • Is there a way to delete everything off the Ipod but not off itunes?

    I have a 4g Ipod nano, and it is full so it wont let me add any new songs. I have gone into itunes and deleted songs from my playlist and tried to update my ipod but even when i delete playlists and songs it still says that i dont have enough room to add any more. Is there any way i can delete everything off my ipod and but still have all my music in itunes so i can basically start over?

    Try restoring your iPod. Restoring will erase the iPod's hard drive, reload the software and put it back to default settings. it won't remove anything from your iTunes library. Once the restore is complete follow the on screen instructions to name the iPod and automatically sync your songs and videos onto the fresh installation. Press Done and the iPod will appear in iTunes and start to sync. If you want to update manually or using selected playlists uncheck the box beside the sync automatically instruction and press Done, it will default to manual mode and you can choose whatever setting you like: Restoring iPod to factory settings with iTunes 7

  • Problem fetch large number of records

    Hi
    I want to fetch large number of record from database.and I use secondary index database for improve performance for example my database has 100000 records and query fetch 10000 number of records from this database .I use secondary database as index and move to secondary database until fetch all of the information that match for my condition.but when I move to this loop performance terrible.
    I know when I use DB_MULTIPLE fetch all of the information and performance improves but
    I read that I can not use this flag when I use secondary database for index.
    please help me and say me the flag or implement that fetch all of the information all to gether and I can manage this data to my language
    thanks alot
    regards
    saeed

    Hi Saeed,
    Could you post here your source code, that is compiled and ready to be executed, so we can take a look at the loop section ?
    You won't be able to do bulk fetch, that is retrieval with DB_MULTIPLE given the fact that the records in the primary are unordered by master (you don't have 40K consecutive records with master='master1'). So the only way to do things in this situation would be to position with a cursor in the secondary, on the first record with the secondary key 'master1' retrieve all the duplicate data (primary keys in the primary db) one by one, and do the corresponding gets in the primary database based on the retrieved keys.
    Though, there may be another option that should be taken into consideration, if you are willing to handle more work in your source code, that is, having a database that acts as a secondary, in which you'll update the records manually, with regard to the modifications performed in the primary db, without ever associating it with the primary database. This "secondary" would have <master> as key, and <std_id>, <name> (and other fields if you want to) as data. Note that for every modification that your perform on the std_info database you'll have to perform the corresponding modification on this database as well. You'll then be able to do the DBC->c_get() calls on this database with the DB_MULTIPLE flag specified.
    I have other question.is there any way that fetch information with number of record?
    for example fetch information that located third record of my database.I guess you're refering to logical record numbers, like the relational database's ROW_ID. Since your databases are organized as BTrees (without the DB_RECNUM flag specified) this is not possible directly.You could perform this if use a cursor and iterate through the records, and stop on the record whose number is the one you want (using an incrementing counter to keep track of the position). If your database could have operated with logical record numbers (BTree with DB_RECNUM, Queue or Recno) this would have been possible directly:
    http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/logrec.html
    http://www.oracle.com/technology/documentation/berkeley-db/db/ref/am_conf/renumber.html
    Regards,
    Andrei

  • Best way to copy large folders to external drive?

    What is the best way to transfer large folders across to external drives?
    I'm trying to clean up my internal hard drive and want to copy a folder (including its sub folders) of approx 10GB of video files to an external firewire drive.
    I've tried drag and drop in finder (both copy and move) and it gets stuck on a particular file and then aborts the whole process.
    1. Surely it should copy everything it can, and simply not copy the problem file.
    2. Is there a way I can do copy and verify, a bit like when I burn a disk so I can be sure the video files have transferred safely before I delete them from my internal drive?
    Many thanks in advance for any advice.

    What you are trying to do makes perfect sense to me and I have done the same prior to getting myself a Time Machine system in place.
    1. Surely it should copy everything it can, and simply not copy the problem file.
    The fact that it is getting stuck on a particular file suggests that there is a problem with it. Try to identify which one it is and deal with that file on it's own. It could be that there is a disk error where that file is stored.
    2. Is there a way I can do copy and verify....
    The copy process you are using does that implicitly as I understand it.
    Chris

  • What is the Best way to move large mailboxes between datacenters?

    What is the Best way to move large mailboxes between datacenters?

    Hi, 
     Are you asking with regards to on-premises Exchange? With Microsoft Online SaaS services (aka Exchange Online) there is no control and no need to control which data center a mailbox resides in.
     With regard to on-premises Exchange, you have two choices: you can move it over the WAN in which case you would either do a native mailbox move (assuming you have Exchange 2010 or later you can suspend the move after the copy so you can control the
    time of the cutover) or create a database copy in the second data center and once the database copies have synchronized change the active copy.
    The other choice is to move is out of band which would usually involve an offline seed of the database (you could conceivably move via PST file but that would disrupt access to the mailbox and is not really the 'best way').
    In general, Exchange on-premises questions are best asked on the Exchange forum: http://social.technet.microsoft.com/Forums/office/en-US/home?category=exchangeserver
    Thanks,
    Guy 

  • Best way to delete transcoded and proxy media?

    Hello,
    I am trying to free up some disc space and am curious what is the best way to delete transcoded and proxy media in fcpx?  I know I can open up a library bundle and delete the folder, but is this safe?  There must be a better option...
    Thanks,
    Austin

    Austin Samuel wrote:
    Hello,
    I am trying to free up some disc space and am curious what is the best way to delete transcoded and proxy media in fcpx?  I know I can open up a library bundle and delete the folder, but is this safe?  There must be a better option...
    Thanks,
    Austin
    I don't know of any other way.
    Proxy and Transcoded media can always be re-generated if the original files are in place.
    If this question refers to your system drive then a BIG suggestion is to aquire a fast dedicated media drive.
    Al

  • What is the BEST way to delete an app on an iPhone 4S so it will NOT be added back when I sync with iTunes?

    What is the BEST way to delete an app on an iPhone 4S so it will NOT be added back when I sync with iTunes?

    To completely remove an app and all of its data, do this: First delete the app on your phone, then in iTunes, under Library, click on Apps, right-click the app you want to delete & select delete, when prompted, move all files to trash, empty your trash. Connect your phone and sync. The app and all of its data will be gone.

  • I own CS4, upgraded to CS5, then joined CS6, CC... and now I installed CC14. Which will be the best way to delete or uninstall the programs I don't need keeping only CC14 without running into issues? I have an Alienware Area 51 PC running Windows 7.

    I own CS4, upgraded to CS5, then joined CS6, CC... and now I installed CC14. Which will be the best way to delete or uninstall the programs I don't need keeping only CC14 without running into issues? I have an Alienware Area 51 PC running Windows 7.

    Which will be the best way to delete or uninstall the programs I don't need keeping only CC14 without running into issues?
    There is none. Whatever you do to get rid of the older versions will inevitably do damage to shared components and other stuff of the new versions as well. That's just how it is with this approach... Uninstall the older versions, run the cleaner tool then reinstall CC 2014.
    Use the CC Cleaner Tool to solve installation problems | CC, CS3-CS6
    Mylenium

  • What is the best way to delete MacKeeper?

    I installed MacKeeper last year in an attempt to free up some hard drive space on my former hard drive. I now have a new SSD and have no need for MacKeeper. In addition, I see that this programme attracts a lot of negative press.
    What is the best way to delete MacKeeper from my hard drive and Time machine?

    Remove the "MacKeeper" crapware as follows. First, back up all data.
    "MacKeeper" has only one useful feature: it deletes itself.
    Note: These instructions apply to the version of the product that I downloaded and tested in early 2012. I can't be sure that they apply to other versions.
    IMPORTANT: "MacKeeper" has what the developer calls an “encryption” feature. In my tests, I didn't try to verify what this feature really does. If you used it to “encrypt” any of your files, “decrypt” them before you uninstall, or (preferably) restore the files from backups made before they were “encrypted.” As the developer is not trustworthy, you should assume that the "decrypted" files are corrupt unless proven otherwise.
    In the Finder, select Go ▹ Applications from the menu bar, or press the key combination shift-command-A. The "MacKeeper" application is in the folder that opens. Quit it if it's running, then drag it to the Trash. You'll be prompted for your login password. Then a dialog will pop up asking why you want to uninstall it. You don't have to provide that information. Enter anything you like in the text box, then click the Uninstall MacKeeper button. All the functional components of the software will be deleted. Reboot.
    Do not try to uninstall by dragging the MacKeeper icon in the Dock or the LaunchPad to the Trash.
    Here are some general suggestions. If you want your computer to be usable, don't install crapware, such as “themes,” "haxies," “add-ons,” “toolbars,” “enhancers," “optimizers,” "tune-ups," “accelerators,” “extenders,” “cleaners,” “defragmenters,” “firewalls,” "barriers," “guardians,” “defenders,” “protectors,” most “plugins,” commercial "virus scanners,” "disk tools," or "utilities." With very few exceptions, this kind of material is useless, or worse than useless.
    The more actively promoted the product, the more likely it is to be garbage. The most extreme example is the “MacKeeper” scam.
    The only software you should install is that which directly enables you to do the things you use a computer for — such as creating, communicating, and playing — and does not modify the way other software works. Never install any third-party software unless you know how to uninstall it.

  • What is the best way to delete applications?

    I am told that my Mac came with everything I need in the way of programs for basic functions.  What is the best way to delete applications making sure there are no threads left in Library and other places?

    Yes, but not always. An example of what I call "System" files/folders would be Quicktime or things like pre-installed print and scanner drivers. You have to be pretty confident to go rummaging through files in either /System or /Library. Most of the time there is no reason to be in there.
    An example of a file that is in those system folders, that does not come pre-installed as of Lion, is Flash plug-ins. That gets installed in the /Library/Internet Plug-Ins
    But your average App downloaded from the MAS, most of it's files will be in the User Folder   ~/username

  • Given ipod from someone best way to delete their music and add to my apple id

    Have been given ipod from someone what is best way to delete their music and then sync to my itumes and apple id

    Hi Caz0,
    You might want to try to factory reset the iPod to wipe it completely. Here's a link on how to do it: http://support.apple.com/kb/ht1339
    As far as I'm aware an iPod Classic isn't tied to an Apple ID, therefore all you need to do is factory reset the iPod, sync your music library and plug in headphones, and you're ready to go!
    Good Luck,
    Stefan

Maybe you are looking for