Long Term Storage with VTL (Firestreamer)

Hi All
I'm currently in the planning stages of DPM 2012 R2 and am looking at eliminating tapes for long term/archival purposes by using a VTL solution. I have a trial of Firestreamer and am testing it out. The thought is that we would output to to file media and
then somehow replicate this offsite to a 3rd party for long term storage.
The thinking around removing physical tapes from daily life would save manual labour, tape transfers and the general chore of physical media but I'm wary of the virtual side of things becoming a time consuming monster to manage.
The "tape" management side of it would need to be very carefully thought out to keep things under control and relatively simple. Things like max media size would need to be kept fairly low compared to modern LTO capacities to allow easy WAN transfer
but at the same time I'm wary of ending up with a myriad of files from one large backup let alone months/years worth.
For those "tapes" that will need to rotate (say weeklies on a 5 week rotation) we could simply reuse the tape files and then allow them to overwrite at the offsite end to preserve space and keep things simple.
I'm backing up around 12TB pre compression using traditional tape methods currently and this is set to rise steeply with the onset of new projects, so depending on the compression rate the transfer of full backups over the wire could be large.
There is the added issue that DPM either compresses or encrypts and Firestreamer only compresses. The issue here is that I have a requirement for encrypted backups so therefore encryption will have to be performed at DPM level negating any form of compression
from there on.
Does anyone have any experience of this, or any other VTL's or non tape practices, if so I'd be very interested in how you did it.
Thanks

Hi,
I would say it depends - DPM encrypts the data in chunks before writing the data to the tape. If most of the chunks are the same for each version of the backup, then the disk dedupe would be efficient. If data was inserted or removed from files in
such a way that the chunks are no longer aligned, then the encrypted data would not match previous chunks and would not be deduped since it didn't match previous backup data. But overall, I think you have a pretty good chance of getting some
fairly decent dedupe saving over time.
You may want to inquiry about that with the OEM to see if they have any test results using DPM backup with software encryption enabled.
Please remember to click “Mark as Answer” on the post that helps you, and to click “Unmark as Answer” if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread. Regards, Mike J. [MSFT]
This posting is provided "AS IS" with no warranties, and confers no rights.

Similar Messages

Long Term Storage ideas for Data in an Oracle Database

We have data that we no longer need access to and want to archive it on tapes for long term storage.
We want to save the data using datapump or rman from time to time and and store the exports/backups on tapes. The issue we don't want to face in the future is , if Oracle deprecates the methodology we used to save the data in 2012, we no longer have a way to retrieve the data into a database 10 years from now.
This may sound crazy, but hear me out. Before datapump existed everyone was using vanilla exp to export the data. Now we have datapump as exp's upgrade and we have rman too. What if in 2012 we took an export using datapump or a backup using rman and save it off on a tape. 10 years go by and in 2022 management wants to look at the 2012 data. Lets say by this time Oracle's mothodology changes and they have a much more efficient/robust export/backup method. One way for us to keep the old backups up to date is to import the data back before the method we used in 2012 is deprecated and then export/backup the data using the new method to keep the exports/tapes in sync with then current technology.
My question is, what methods of saving large amounts of data do you recommend/use to save data to a tape so that you don't have to worry about changing times.
An idea I heard is to save the data as insert statements. This would work for most of the data except for blobs.
Any help would be appreciated.
Thanks in advance.

>
Won't an intelligent compression algorithm take care of a lot of the overhead for that?
>
That's certainly a valid point to make and one that the OP can test. And because the duplicated metadata is identical and in the same physical row location there may be compression algorithms that can reduce it to just one set.
There were two points I was trying to make.
1. Don't just save the data - save the metadata also.
2. The metadata for each row will be the same so if you use an INSERT format that duplicates it for every row you are potentially wasting a lot of space.
Your other point was probably more important than these or anything previously mentioned.
>
perform restores of the entire database every couple of years or so to make sure that your
database isn't degrading
>
Regardless of the format and media used to backup the data it is important to verify the integrity of the backup periodically.
There are a lot of ways to do this. One way is a simple MD5 (or other) checksum of data at some level of granularity and then periodically read the data and confirm you still get the same checksum.
In my opinion the more generic the format that the data is saved in the better the odds that you can recover the data later no matter how technology may have changed.

Best method for converting 8 mm cassettes for long term storage

I have approximately one hundred 8 mm video cassette tapes that I would like to convert to another format for long term storage. I captured my first tape today from my camera Sony DCR VTR 520 which is a digital 8 mm camera. I was not planning to convert the files to DVDs at this time but just keep the avi file on the computer.   The video was just over 2 hours long but the file ends up being 27 GB.   I expecting it to be around 8 GB like a 2 hr DVD but I guess its not the same.
The 8 mm tape I converted was from an very old 8 mm camera which I no longer have (1990). The camera used to capture from is a VTR 520 which is actually a Sony digital 8 mm, which can play back standard 8 mm. I used CS3 on my old computer with standard project settings for DV NTSC 29.97 f/s.   I currently use PE10 so I could use either program.
What do other people use to convert and store videos on??
If I want to keep them as avi files will I have to live with these large file sizes?
I did a search on the forum and found titles that talk about this subject from a couple of years ago but when I click on them they don't come up.   How do you see these older threads?
Any help would be appreciated.

JMJarige,
The only potential problems with this workflow are:
DVR units (I have a Panasonic, but this is pretty much across the board) seldom produce a 100% DVD-compliant disc, and as most add a rudimentary Menu & navigation system to the disc, produce a first VOB (where the Menus & navigation are located), that is tough, if not impossible to directly Import.
In the production to DVD, there is MPEG2 compression applied. If one edits that material, and then they output to any compressed CODEC, the quality loss is easily seen
When I convert from VHS for a client, I create a DVR DVD disc, for their archives (in case the tapes die), but then Capture to DV-AVI to be edited to a DVD-Video. The client gets two sets of discs - one that is produced by the DRV, and then one that is the finished edit of their tapes. For me, it's a two step process, but gives them a backup, plus a finished product.
Good luck,
Hunt

Can I delete renders and previews when putting projects into long term storage?

I want to shrink the size of my longe term storage folders. Would it be ok to just delete the previews and renders?
Any project opened in the future is going to rebuild them anyway..
Thanks

I agree with Joe (nothing unusual there), and, as stated, how you do the mechanics might well depend on how you establish your folder structure, when you start the Project. I keep everything (including copies of any Assets), in a folder structure below the Project's name. This article goes into more detail: http://forums.adobe.com/message/4028199#4028199
With such a structure, it is easy to Delete, say Media Cache files, etc., before using Project Manager, or just setting it to NOT copy those files over. It also makes a final Deleting easy.
I do sacrifice a bit of performance, by not spreading, say my Render files, my Audio and my Video, over separate physical drives, but then, when the Project is done, tracking down everything is easy, as I just Delete the root folder, [Project Name], and everything goes with that.
Good luck, and glad that you asked, before archiving a bunch of Projects, or Deleting a bunch of files.
Hunt

G5 - Long Term Storage Hints

Any tips for long term storage & overseas shipping of my G5 ?
All advise much appreciated
Thanks in advance

P.s
It was rather tricky to remove the PRAM battery - as I encountered some difficulty in getting access with my hand into such a tight space, without touching the logic board !!

Using "Deleted Items" folder for long term storage.

I have a user that is using the "Deleted Items" folder in Outlook for long-term storage. He has unchecked the box in the options which would remove the items in the Deleted Items folder at shutdown
and is having problems with his Outlook freezing up several times a day. He was using a Microsoft Surface as his primary computer and has since changed to a Dell Latitude, but Outlook still freezes up. He has an Office 365 Pro Plus license.
I have been looking for articles to explain all the reasons for not using "Deleted Items" for storage, but in order to get him to change, This is not a .pst, it is the Outlook default “Deleted Items”.
. I really need some "ammunition" to let him know the consequences of keep that much mail in Deleted Items.
Could you tell me when Outlook starts, does it enumerate the Deleted Items folder as it does the Inbox?

I have to agree with Robert's point here. The question of whether or not the user should be doing this answers itself: of course he shouldn't. On a lighter note there's a Dilbert cartoon from 2003 which covers this very point:
http://dilbert.com/strips/comic/2003-11-13/
I'd suggest you ask him to create a sub-folder of his inbox and get him to move everything over, then he still has his history and can use deleted items the way it was intended.
The question I would ask is are you sure that this has any bearing on his Outlook freezing issue? I have encountered people who never empty their recycle bin and I have encountered issues with Outlook freezing, but I don't recall ever having the two together.
If you still have issues after he's moved everything then it's a general Outlook freezing issue, so you have to look at those sort of things. For a start, is it bang up to date with Microsoft Updates? If so how big is his mailbox? Presumably he has an ost
file, so how big and fragmented is that? I'd probably try creating a new Outlook profile. If not then what add-ins does he have, could they all be disabled for a while to see if it's one of them, then add them back one at a time? If not is it worth running
a repair in Office? I've had that fix a fair few issues.
Just a few thoughts. I don't know if any of them will help.

Long term Storage

I have to put some iPod Touches in long term storage (1+ years). Any recommendations on what should be done to prepare them for this?

Might it not be better to sell them now and buy something newer (and better) in a year or so?

Lumia 720 long term storage

Hi, I bought my 720 as a spare phone and to test a WP8 first hand. What would be the proper way to store this phone for long term when not used? With other phones, I would usually take the battery out, but this is not an option with the 720. So what should I do here, charge the battery fully then turn the device off? Thanks.

If you do not wanna lose battery capacity too much, charge it up to 40% and turn the phone off.
Permanent Capacity Loss versus Storage Conditions
Storage Temp. 40% Charge 100% Charge
0 °C (32 °F) 2% loss after 1 year 6% loss after 1 year
25 °C (77 °F) 4% loss after 1 year 20% loss after 1 year
40 °C (104 °F) 15% loss after 1 year 35% loss after 1 year
60 °C (140 °F) 25% loss after 1 year 40% loss after 3 months
http://uterc.org/files/LipoStorageTips.pdf
http://batteryuniversity.com/learn/article/how_to_store_batteries
The silence will fall

Long-term storage solution

This isn't "immediate", per se, and won't be "soon". I'm still at the lots-of-planning stage here with this, so you don't need to get into a race or anything to quickly help me before I make my decision just so you know.
Now, for my question: what would make a good, long-term, backup/archival system that isn't read only?
I want something fast (so not tapes), long-term (so not hard disks) and not fragile (so probably not CDs). What else do I have available?
I could use CDs if something akin to the CD changer one finds in more elaborate hi-fi systems and car stereos existed for data discs - so, therefore, a device that was equivalent to a clean-room environment and could hold like 500 discs. One that accepted standard 5.25" burners would be awesome, because I could switch it to bluray once prices went down and popularity and Linux support went up
I'm basically looking for something that lets me access data like 10 years after storing it, maybe make a couple of changes, etc. 10 years would be the absolute maximum I'd probably need though
Oh, and the medium I want to use needs to scale high. Very high. Into the order of more than a decent amount of TBs... like, say, 40 or 50. Of course that nice, round figure wouldn't pop up out of nowhere the day I have my plans in order, it'd grow - slowly - over time. So not only do I need something scalable, I need something that can scale slowly.
So kinda like a NAS or external HDD type of solution, except one that doesn't use failure-prone media like HDDs. Unless there's a kind of hard disk out there that doesn't fail very "easily" and lasts in an order of years?
-dav7
Last edited by dav7 (2008-10-08 03:34:28)

molom wrote:
dav7 wrote:
Oh, and the medium I want to use needs to scale high. Very high. Into the order of more than a decent amount of TBs... like, say, 40 or 50. Of course that nice, round figure wouldn't pop up out of nowhere the day I have my plans in order, it'd grow - slowly - over time. So not only do I need something scalable, I need something that can scale slowly.
So kinda like a NAS or external HDD type of solution, except one that doesn't use failure-prone media like HDDs. Unless there's a kind of hard disk out there that doesn't fail very "easily" and lasts in an order of years?
-dav7
I believe there is nothing that exists in the world, unless you pay huge money for Solid State Hard Drives. The only solution I see is using a bunch of rewritable double layer Blu-ray discs. I don't think there is any proper solution for what your looking for. Your looking for the perfect that doesn't exist.
I'm not sure how long SSDs will last and I don't think there is any reliable information out there, wether an SSD I plugin now will still work in ten years. Plus you would need one hell of a lot SSDs to store Terabytes of data. BlueRay is probably worse than DVD regarding reliability, until the manufacturers get more experience with them.
Am I the only one to see a pattern with dav7 opening threads looking for things which don't exist?
Last edited by Garns (2008-10-08 13:39:29)

How to preserve calendar on calDAV in iCal for long term storage?

Is there a way one can transfer/preserve calDAV/GoogleCalendar entries to iCal so that all the entries are preserved even when iCal no longer subscribes to those feeds?
Message was edited by: SoGood

Hi John and Austin,
I think both of you are correct (when looking at iCal "Version 6.0 (1639)" with 10.8.1/ML).
Thanks!
The "very subtle grey in Month view" is visible on my iMac as well, but ONLY in Month view.
This (almost invisible) highlighting is not shown in Week view. In my initial question, I was addressing Week view (without mentioning).
This is inconsistent - one may hope that Apple will fix this (and also the "invisibility").
Maybe, some extension programmer has already generated a tool to modify these settings of iCal?
(And at the same time removes the terrible "leather look" of iCal? Yes, I know there are descriptions how to change the resources. But I would prefer a tool that allows to modify the "skin" of iCal and Contacts via a GUI.)
Bests, BW
Added: check here: http://tweaksapp.com/app/mountain-tweaks/
The tools allows to remove the "leather skin", but does nothing to highlight sundays or weekends in iCal's Week view.
Message was edited by: bwadma

I receive songs or photos from friends on email, and when I download them, they stay in my Downloads file. How can I move these to my hard drive for long-term storages and organization?

How can I transfer song or photo files from Downloads on Firefox to my own hard drive for permanent storage?

If the hard drive that crashed is the one that stored the itunes library, be sure to restore the library from Time Machine (mac) or whatever backup you use on windows. That will get back your music. I suspect that the replacement drive has an empty itunes library and thus, perhaps, the itunes match used that to sync to its servers.
Try posting in the iTunes Match forum, you'll probably find more knowledgeable folks there.
https://discussions.apple.com/community/itunes/itunes_match

Printing for long term storage

I'm in the process of writing a personal history. It will consist of text and black & white and colour photos and reproductions of other documents. I have both HP6122 & 5652 printers and use genuine HP cartridges. I have a supply of acid & lignin free paper and intend to store the finished items in acid & lignin free binders. Has anyone done something similar? I'm interested in knowing whether the ink jet printing will last, or whether there is a better way.

You can save the scanned TIFF's to DNG from Adobe Camera RAW (and I think Lightroom too). You can't save TIFF's to DNG from the stand-a-lone DNG converter though.
People will say that the best thing to do is keep them as TIFF's, however I choose not to. Basically, all my original files are converted to DNG, whether they are RAW's from my DSLR, JPEG's from my compact camera, or TIFF's from my scanned negatives. The reason I do this is as follows:
Firstly, with the JPEG's and TIFF's, by converting them to DNG format means that I can't accidently overwrite the original during post processing (for example, imagine resizing an image for the web and accidently overwriting the original would be a nightmare).
Secondly, it means that I can easily identify each file by type. DNG's are always the original and TIFF's are always the full size edited file, with layers, etc. These are the only two you really need to keep, but I also keep smaller JPEG versions too, which I have converted for web/email/fast browsing use.
Thirdly, by converting RAW's and TIFF's to DNG, it lowers the file size of my originals. With JPEG's it increases the file size, but is not too much of a problem as this is way off-set by the savings of my scanned negatives (my negative scans are approx. 115MB each, but as DNG's they are approx 84MB).
TIFF is the number one archival format, but if your edited 'working' file is saved as a TIFF anyway, there is no need for the original to be too because it would just be more confusing, take up more space and create the potential to accidently over-write the original.
Also, if you open the images from Adobe Camera Raw into Photoshop as Smart Objects, you willl still have a version of the original in the TIFF file anyway, as it uses non destructive editing. So in 50 years time, you will have not only DNG versions, but also TIFF versions that can also be used as originals if need be.

ODS as a Long-Term Storage Area

I am involved in an initial implementation of BW 3.3. Customer has heard that ODS is not only a good ( i.e. traditional ) means for staging and/or combining data, but also, and more importantly, a means of storing data to facilitate timely reloads of data in the event one becomes necessary.
Cant such an approach result in the ODS becoming too large and consequently, adversely impact processing, as well as depleting system resources ? Are there circumstances when such an approach is cost effective ? If data can be re-loaded from the source system, ODS seems like an expensive storage area. Current area of interest is Cost Center Accounting ( full cost, not line item ) additional areas are likely to be considered.
Customer indicated that this approach has been advocated at several SAP conferences.
I would appreciate any input, pro or con, regarding this.

I say store it. P.S. what company do they buy storage from?

Long term storage help 2 months

I am going away for two months. How should I store my powerbook?

Personally, I would just shut it off, unplug everything from it, and put it in a safe place. I would guess that a fully charged main battery will last 2 months if the computer is off. If not, no big deal, plug in the charger when you get home. If you are ultra careful, I'd take out the battery and store it separately. Your data will be safe on the hard drive, although the PRAM data such as date and time will reset when the PRAM battery dies in a few weeks. It might take some extra effort to restart it, then make sure to have it plugged into the power for several days to charge the PRAM battery.
I don't think I would leave it for 2 months with the power plugged in.

Long term storage of itunes libraries on Apple TV

I would like to know if audio and video media in my itunes library can be sync’ed to appletv and remain playable even if my computer is off or does itunes have to be on and sync’ed everytime I want to listen or watch via appletv?

Definitely agree with Brody. Your synced files will remain on ATV. You will not need a computer or itunes on or open. Once the ATV gets synced AND you removed one or more of those files then they'll be gone from ATV. Easy to fall into this scenario.

Long Term Storage with VTL (Firestreamer)

Similar Messages

Maybe you are looking for