I/O Sub-System - What is That?

Question

I use the term “I/O sub-system” quite often, and some wonder what that is. Usually, I will give a bit of detail, but probably need to take things to a higher level, and also explain why the I/O sub-system is important to video editing. The term I/O is Input/Output, and the concept of the “sub-system” encompasses everything about the HDD’s (Hard Disk Drives). Basically, used in these fora, the I/O sub-system refers to the HDD’s, and all the specs. of those HDD’s.
It starts with the HDD’s themselves. The main considerations are:
The number of physical HDD’s (not partitons)*
HDD capacity/size
HDD space used, and remaining (defragmented)
Format Type: FAT-32, or NTSF**
The speed of the HDD, i.e. the RPM performance of the HDD
The controller type, say SATA II, and this will specify the connection speed
The allocation of each HDD, i.e. where one has the OS, the programs, the Project files, the Scratch Disks, the Exported files, etc.
For Video editing, it is just about a minimum to have 2 physical HDD’s, splitting the I/O load over the separate, physical HDD’s. Having 3, and splitting the I/O load over all 3 is even better. For even greater performance, one can add additional HDD’s, and allocate them for even more spread.
With a 2 HDD I/O sub-system, one would allocate in this manner:
C:\ OS, programs and probably the Windows Virtual Memory [Page File]
D:\ Project files, with Assets, Scratch Disks and Exports
With a 3 HDD I/O sub-system, one would split D:\ by perhaps the Project files and Scratch Disks and then the Assets on the separate Disk. Note: one can also split the Scratch Disks between the D:\ and E:\ for even better performance.
The more that one can separate the needs for reads/writes to/from the HDD’s, the greater the performance will be, as the computer will not be waiting for those reads and writes.
Why is proper allocation of the HDD’s, and spreading the I/O load important? One of the first laws of physics is that an object cannot be in more than one place at the same time. When running a video editing program, there are several programs, and processes telling the computer to do many things at the same time. First, the OS is demanding reads from the HDD, and in the case of the Page File, writes, as well. The OS gets first dibs on these reads and writes. Next, the program, our NLE (Non Linear Editor) here, requires reads from the HDD, and will also require writes for many operations. This is just to run. When one is working on a Project, there are many writes to the Scratch Disks, to create, or work from large working files. The NLE will also need to gather information from the Assets in the Project, hence plenty of reads. When playing back the Timeline, the NLE will require many high-speed reads. When Exporting, the NLE first writes working files, and then reads from them, to create the Export file - more writes. If the computer is requested to do reads and writes for the OS and the NLE at the same time, someone must wait. If these reads and writes are requested on a single HDD, the wait is much, much longer. If one has spread the I/O load over several HDD’s, the wait is far, far shorter. The better you spread this load, the greater your performance and the fewer lags you will have. This is especially noticeable with playback of the Video and the Audio.
One problem with HDD’s is that as they fill up, the heads have to move greater distances to do the reads and writes. Most HDD’s began to slow down at about 70% capacity, as their read/write times go up. This also causes mechanical wear and tear on the head actuating mechanism, which can lead to premature failure. The exact same thing happens, when an HDD is fragmented. The heads have to hunt all over the platters, to get the pieces of a file, which will then need to be “assembled.” Some applications and operations will also not work well, if at all, if there is not adequate defragmented free space available. Also, as video-editing requires very large working files, just to function, having adequate free space is very important.
One might think only in terms of say the space required to write to a DVD-5 (approximately 4.7GB), but first the program creates working files, that can be much larger, and then the program will write a separate Video and a separate Audio file, and finally combine them into one muxed MPEG-2. file. Last, the program gathers up this MPEG-2 file, plus the navigation and Menus, and bundle everything into the IFO, BUP and VOB files. In PrE, and Encore, a module, the Vobulator does this last function, and will need plenty of working files, to produce that 4.7GB DVD. If the I/O sub-system cannot provide adequate free space, the process will fail. This shows why one must have adequate defragmented free space available.
This brings us to additional reading, and to a Q & A session:
* See Partitions, a No-No for Video Editing
See Harm on HDD Setup for Video Editing
See Harm on To Raid, or Not to Raid
See Windows Virtual Memory, as the Page File requires I/O interaction, and how you have this set up, can affect things on several levels.
** See Converting FAT-32 to NTSF
Question: Can I use external HDD’s in my I/O sub-system?
Answer: yes, but the connection type will be important. USB 2.0 connections are very, very slow, and because of the speed of most computers, and the great deal of read/write activity required by an NLE, the connection is often too slow just to perform, leading to read/write failures, including the dreaded Delayed Write Failure, which can totally corrupt the external, requiring full low-level reformatting and the loss of ALL data on that HDD. USB 2.0 HDD’s are great for archiving data, but not much more. FireWire 400 is better, but is still very slow. FW-800 is adequately fast for editing to/from, and is stable. The use of eSATA is even better, and will equal in real-world terms, about what one gets with internal SATA II HDD’s.
Question: What about SSD drives?
Answer: They are fast, but still cost a lot for the storage, and that is still limited. They really help with boot-up speed, but then how often does one boot-up the system. In time, and as the capacity goes up, and the cost comes down, they will very likely replace the mechanical HDD’s. They do bear watching, but beyond the boot disk, are not that well-suited for additional drives for video editing. That will likely change in the future.
Question: Do I really need to frequently defragment my HDD’s?
Answer: Absolutely - see above. After every editing session, I will run a defragmentation program, and will do so weekly, even if I have not been editing.
Question: Can I edit video on my laptop with a 40GB HDD?
Answer: Perhaps, but you will want to have only the OS (and pared down to the minimum), and the NLE program, plus you will need to keep the Projects very small, and not have any more Assets, than will absolutely necessary. Then, you will need to be prepared to wait, and possibly deal with crashes, when that wait-time becomes too much for the OS, or the NLE to bear. You will want to defragment the HDD very, very often. This holds true, to a degree, with any single-disk I/O, regardless of the size, speed or controller type. There are many reads/writes, and they will bottleneck, waiting for that one poor HDD to provide them.
Question: Does the HDD’s speed really matter?
Answer: Yes. For Video and Audio, one needs at least a 7200 RPM HDD, to play back smoothly under all load conditions. A 5400 RPM unit might be able to keep up under otherwise ideal conditions, but lags, and jumpy playback is very common. A 10K RPM unit will likely not help much in playback, over a 7200 RPM drive, but will greatly improve other read/write functions, meaning that one might not notice that extra speed just with playback, but that other functions will be improved, speed-wise.

Answer 1

Dear Mr. Hunt,
I'm an experienced Oracle DBA.
The tech staff in organizing VLDB systems (very large databases) is quiet similar to what you have showed here.
May I suggest, Perhaps, for your readers benefits ( especially those who do professional large scale editing projects) I think it will be a good idea to elaborate about some modern storage systems architectures like NetApp , EMC etc.
Using a modern storage systems can solve much of the I/O problems, and it seems to me after reading this article that me and you are dealing with the same IO problems: large amounts of data who needs to be wriiten and accessed very fast.
Thank you for your effort.
Sincerely Yours,
shimon.