Mounting the Root File System into RAM

Question

Hi,
I had been wondering, recently, how can one copy the entire root hierarchy, or wanted parts of it, into RAM, mount it at startup, and use it as the root itself.  At shutdown, the modified files and directories would be synchronized back to the non-volatile storage. This synchronization could also be performed manually, before shutting down.
I have now succeeded, at least it seems, in performing such a task. There are still some issues.
For anyone interested, I will be describing how I have done it, and I will provide the files that I have worked with.
A custom kernel hook is used to (overall):
Mount the non-volatile root in a mountpoint in the initramfs. I used /root_source
Mount the volatile ramdisk in a mountpoint in the initramfs. I used /root_ram
Copy the non-volatile content into the ramdisk.
Remount by binding each of these two mountpoints in the new root, so that we can have access to both volumes in the new ramdisk root itself once the root is changed, to synchronize back any modified RAM content to the non-volatile storage medium: /rootfs/rootfs_{source,ram}
A mount handler is set (mount_handler) to a custom function, which mounts, by binding, the new ramdisk root into a root that will be switched to by the kernel.
To integrate this hook into a initramfs, a preset is needed.
I added this hook (named "ram") as the last one in mkinitcpio.conf. -- Adding it before some other hooks did not seem to work; and even now, it sometimes does not detect the physical disk.
The kernel needs to be passed some custom arguments; at a minimum, these are required: ram=1
When shutting down, the ramdisk contents is synchronized back with the source root, by the means of a bash script. This script can be run manually to save one's work before/without shutting down. For this (shutdown) event, I made a custom systemd service file.
I chose to use unison to synchronize between the volatile and the non-volatile mediums. When synchronizing, nothing in the directory structure should be modified, because unison will not synchronize those changes in the end; it will complain, and exit with an error, although it will still synchronize the rest. Thus, I recommend that if you synch manually (by running /root/Documents/rootfs/unmount-root-fs.sh, for example), do not execute any other command before synchronization has completed, because ~/.bash_history, for example, would be updated, and unison would not update this file.
Some prerequisites exist (by default):
    Packages: unison(, cp), find, cpio, rsync and, of course, any any other packages which you can mount your root file system (type) with. I have included these: mount.{,cifs,fuse,ntfs,ntfs-3g,lowntfs-3g,nfs,nfs4}, so you may need to install ntfs-3g the nfs-related packages (nfs-utils?), or remove the unwanted "mount.+" entires from /etc/initcpio/install/ram.
    Referencing paths:
        The variables:
            source=
            temporary=
        ...should have the same value in all of these files:
            "/etc/initcpio/hooks/ram"
            "/root/Documents/rootfs/unmount-root-fs.sh"
            "/root/.rsync/exclude.txt"    -- Should correspond.
        This is needed to sync the RAM disk back to the hard disk.
    I think that it is required to have the old root and the new root mountpoints directly residing at the root / of the initramfs, from what I have noticed. For example, "/new_root" and "/old_root".
Here are all the accepted and used parameters:
    Parameter                       Allowed Values                                          Default Value        Considered Values                         Description
    root                                 Default (UUID=+,/dev/disk/by-*/*)            None                     Any string                                      The source root
    rootfstype                       Default of "-t <types>" of "mount"           "auto"                    Any string                                      The FS type of the source root.
    rootflags                         Default of "-o <options>" of "mount"        None                     Any string                                      Options when mounting the source root.
    ram                                 Any string                                                  None                     "1"                                                  If this hook sould be run.
    ramfstype                       Default of "-t <types>" of "mount"           "auto"                     Any string                                      The FS type of the RAM disk.
    ramflags                         Default of "-o <options>" of "mount"        "size=50%"           Any string                                       Options when mounting the RAM disk.
    ramcleanup                    Any string                                                   None                     "0"                                                  If any left-overs should be cleaned.
    ramcleanup_source       Any string                                                   None                     "1"                                                  If the source root should be unmounted.
    ram_transfer_tool          cp,find,cpio,rsync,unison                            unison                   cp,find,cpio,rsync                           What tool to use to transfer the root into RAM.
    ram_unison_fastcheck   true,false,default,yes,no,auto                    "default"                true,false,default,yes,no,auto        Argument to unison's "fastcheck" parameter. Relevant if ram_transfer_tool=unison.
    ramdisk_cache_use        0,1                                                              None                    0                                                      If unison should use any available cache. Relevant if ram_transfer_tool=unison.
    ramdisk_cache_update   0,1                                                              None                    0                                                     If unison should copy the cache to the RAM disk. Relevant if ram_transfer_tool=unison.
This is the basic setup.
Optionally:
    I disabled /tmp as a tmpfs mountpoint: "systemctl mask tmp.mount" which executes "ln -s '/dev/null' '/etc/systemd/system/tmp.mount' ". I have included "/etc/systemd/system/tmp.mount" amongst the files.
    I unmount /dev/shm at each startup, using ExecStart from "/etc/systemd/system/ram.service".
Here are the updated (version 3) files, archived: Root_RAM_FS.tar (I did not find a way to attach files -- does Arch forums allow attachments?)
I decided to separate the functionalities "mounting from various sources", and "mounting the root into RAM". Currently, I am working only on mounting the root into RAM. This is why the names of some files changed.
Of course, use what you need from the provided files.
Here are the values for the time spend copying during startup for each transfer tool. The size of the entire root FS was 1.2 GB:
    find+cpio:  2:10s (2:12s on slower hardware)
    unison:      3:10s - 4:00s
    cp:             4 minutes (31 minutes on slower hardware)
    rsync:        4:40s (55 minutes on slower hardware)
    Beware that the find/cpio option is currently broken; it is available to be selected, but it will not work when being used.
These are the remaining issues:
    find+cpio option does not create any destination files.
    (On some older hardware) When booting up, the source disk is not always detected.
    When booting up, the custom initramfs is not detected, after it has been updated from the RAM disk. I think this represents an issue with synchronizing back to the source root.
Inconveniences:
    Unison needs to perform an update detection at each startup.
    initramfs' ash does not parse wild characters to use "cp".
That's about what I can think of for now.
I will gladly try to answer any questions.
I don't consider myself a UNIX expert, so I would like to know your suggestions for improvement, especially from who consider themselves so.
Last edited by AGT (2014-05-20 23:21:45)

Answer 1

How did you use/test unison? In my case, unison, of course, is used in the cpio image, where there are no cache files, because unison has not been run yet in the initcpio image, before it had a chance to be used during boot time, to generate them; and during start up is when it is used; when it creates the archives. ...a circular dependency. Yet, files changed by the user would still need to be traversed to detect changes. So, I think that even providing pre-made cache files would not guarantee that they would be valid at start up, for all configurations of installation. -- I think, though, that these cache files could be copied/saved from the initcpio image to the root (disk and RAM), after they have been created, and used next time by copying them in the initcpio image during each start up. I think $HOME would need to be set.
Unison was not using any cache previously anyway. I was aware of that, but I wanted to prove it by deleting any cache files remaining.
Unison, actually, was slower (4 minutes) the first time it ran in the VM, compared to the physical hardware (3:10s). I have not measured the time for its subsequent runs, but It seemed that it was faster after the first run. The VM was hosted on a newer machine than what I have used so far: the VM host has an i3-3227U at 1.9 GHz CPU with 2 cores/4 threads and 8 GB of RAM (4 GB ware dedicated to the VM); my hardware has a Pentium B940 at 2 GHz CPU with 2 cores/2 threads and 4 GB of RAM.
I could see that, in the VM, rsync and cp were copying faster than on my hardware; they were scrolling quicker.
Grub, initially complains that there is no image, and shows a "Press any key to continue" message; if you continue, the kernel panics.
I'll try using "poll_device()". What arguments does it need? More than just the device; also the number of seconds to wait?
Last edited by AGT (2014-05-20 16:49:35)