Advice for keeping high quality files for archiving

Question

I'm contemplating archiving here. I love reference files because they are pretty small in file size and don't compress. However, I have to worry that all my files will remain in the same exact place forever. I have found that is not always the case for whatever reason. Then there is self-contained reference files. A great idea...I don't have to worry about moved files but the file size is enormous. I do long videos (sometimes an hour or so) so file size is really important to me.
Is there anyway I can do a self-contained reference file but somehow reduce the file size or something? I know I'm throwing a bone here but I figured I would ask because my current scenario is not improving daily. I know I can put my hour or whatever to tape but that essentially is compressing it and I'm using valuable time. Any advice?

Answer 1

HI Lowsky, I been down this road some time back and my opinion is that if you are working in any workflows associated with CMFs (contemporary media format) in a tapeless workflow then storing your LONG TERM content as CMF's on spinning disk will eventually lead to misery and a hopeless state of anger and heartache. Spinning disk is not the long term answer.
refer here for lots of detail: I have post this detail in another forum at
http://www.reduser.net/forum/showthread.php?t=21284&page=6
*Archiving FCP Media Manager Projects, essence, import/transfer material.. etc*
Aside from many testimonials about DISK being faster (more accessible?) and convenient, storing content on spinning disk is a like as a child having all the toys you could imagine but only playing with your 3-4 favourites. (replace with any analogy). Simply you only need the stuff around on disk that you actually use a lot or currently.
I (and I'm not alone) don't trust and disk storage system unless you buy enterprise level disk arrays with some fault tolerance and smart disk array controllers AND that you have at least multiple instances of the content. I qualify ENTERPRISE disk systems - these at $USD5.00-$USD10.00/GB+. I mean those disk arrays from I,T. vendors, and M&E guys like Isolon, DDN, Apple, HP, Copan and their 40+ ilk.
The affordable stuff most of us is around $USD0.25-$USD0.75/GB - SATA and lowest end HBA's or simple controllers. Basically just spinning +power soaking heat dissipating buckets+ to put our content into. (USB | FW or the external JBOD disk enclosues with an HBA or higher FC disk arrays)
Most of these are enterprise disk storage systems that are out of the price range of many on this forum...
Devices like DROBO™ are cute but they are way way too slow (29MB/s when not busy consolidating itself ) and are very limited on their architecture for VIDEO work (useless). I gave mine away after a week to a photographer.
Alternatives like many sets of external and detachable and mobile spinning disks, leaving them spun down until use offer a great odea however unless the disk controller is savvy, you may as well have a bank of el cheapo Le Cie 1TB death disks and get on your knees and face the moon when you power them up.. (Good = luck with that!) mode=:rant
*Viable Archive Solutions for CMFs?*
After looking for viable solutions to by content storage problem after moving to a tapeless workflow with my HVX200 and P2, AND losing on two separate occasions a whole 1TB external disk (Le Cie).. losing most of the data of two drives (2 x 1TB over 4 months).. I have had it with disks!
For me, Disk storage should only be used like a +kitchen table+ for immediate and current and frequently accessed work. Like the +toy analogy+, hen not in use, put the toys away somewhere where they will not be damaged (the content objects) (like a large toy box or cupboard) that you can access with ease. (not the garden shed out the back nor the attic else it will never be used). This is workflow is simple enough..
My first thought for managing FCP media manager archives, P2 footage and final composites was to use BD's.. however the cost for the content that was wanted to archive was not cost effective. I had literally 100's of old video tapes, DV , HDV and ye old video8 rubbish that I needed to move to a CMF, all accessible via proxies and metadata (a la CATDV™ a mam from squarebox.co.uk). The restriction was COST of BD or DVD-DL media and worse the time of write and then access the material ESSENCE when I needed it.
I even looked for a while at many 100's of DVD-5's which in Hong Kong are as cheap as free beer coasters in a bar. I would buy them in canisters of 100's at a time. Simply these are cost effective using TOAST to make multivolumes self extracting archives, but are SOOOooooo slow to write and then access.
So I turned to looking at the economies and workflows of using LTO4 Ultrium data tape drive. I was quite surprised at the reasonable cost.
The cost of Ultrium LTO4 data tape media of to 800GB native for prices that are as cheap as $USD1.40/GB as pure media (the tape cartridge itself).It depend where you buy them and at what quantity. Of course there are the necessary offset/initial costs of the data tape drive AND the SAS HBA you will need for your mac. (LTO4 Ultrium @ $USD2700 and $USD150+ for SAS HBA). Much cheaper fo Ultrium LTO3 but lower data rates and capacities.
NExt year HP et al will have Ultrium LTO5 out at 1.5TB RAW/native capacity as their roadmap suggests.
THe uncompressed / RAW data rates of LTO4 Ultrium data tape drives for ARCHIVING material is for me between 105MB-109MB/sec (that's megabytes).. simply BLOWS any firewire 800 interface out of the water. There are many variables however essentially for people like us whose business is content creation or even the part timer who doesn't want to lose stuff, this is really worth a look. Again it will depend on the software you utilise. I use Tolisgroups BRU.
Just do the sums on the time it takes you to archive (or copy to another disk system [BD or DVD-5, DVD 9, et or a spindle) lets say 4 hours of DVCPROHD 1080P (110Mbs or just over 1GB/minute). You can see that this is a lot of content and will take a great deal of time.
This works out at roughly 240GB.
I think this will take over 500 minutes (6 hours? @ 2min/GB)+ to COPY on a FW400 interface and slightly less on a FW800 interface.
However it takes a mere 2200 second or 36 minutes using an LTO4 SAS Ultrium tape drive (with no tape mount/dismount timeloadlocate)!
On a lesser scale, a look at DDS tapes for the low end. These are despite rumours/myths are quite stable and work well with the right software on the mac and the mac laptop.
I am having great success with this archive and recall system using Tolisgroups BRU that is NATIVE for OSX.
I have posted some very recent information on the two SAS PCIe HBAs I am using (ATTOTECH and LSI) and the two tape drives I am using (HP and QUantum).
For LTO4 Ultrium SAS tape drives I would recommend only HP and ATTOTECH EXPRESSSAS as this is reliable and very stable and very fast for OSX.
I am very happy with this archive/recall workflow using LTO4 Ultrium data tape for FCP.
I have post this detail in another forum at
http://www.reduser.net/forum/showthread.php?t=21284&page=6
hth
w
HK