"No such file or directory" errors on Time Machine backup volume

Question

I remotely mounted the Time Machine backup volume onto another Mac and was looking around it in a Terminal window and discovered what appeared to be a funny problem. If I "cd" into some folders (but not all) and do a "ls -la" command, I get a lot of "No such file or directory" errors for all the subfolders, but all the files look fine. Yet if I go log onto the Mac that has the backup volume mounted as a local volume, these errors never appear for the exact same location. Even more weird is that if I do "ls -a" everything appears normal on both systems (no error messages anyway).
It appears to be the case that the folders that have the problem are folders that Time Machine has created as "hard links" to other folders which is something that Time Machine does and is only available as of Mac OS X 10.5 and is the secret behind how it avoids using up disk space for files that are the same in the different backups.
I moved the Time Machine disk to the second Mac and mounted the volume locally onto it (the second Mac that was showing the problems), so that now the volume is local to it instead of a remote mounted volume via AFP and the problem goes away, and if I do a remote mount on the first Mac of the Time Machine volume the problem appears on the first Mac system that was OK - so basically by switching the volume the problem switches also on who shows the "no such file or directory" errors.
Because of the way the problem occurs, ie only when the volume is remote mounted, it would appear to be an issue with AFP mounted volumes that contain these "hard links" to folders.
There is utility program written by Amit Singh, the fella who wrote the "Mac OS X Internals" book, called hfsdebug (you can get it from his website if you want - just do a web search for "Mac OS X Internals hfsdebug" if you want to find it ). If you use it to get a little bit more info on what's going on, it shows a lot of details about one of the problematic folders. Here is what things look like on the first Mac (mac1) with the Time Machine locally mounted:
mac1:xxx me$ pwd
/Volumes/MyBackups/yyy/xxx
mac1:xxx me$ ls -a
. .DS_Store D2
.. Documents D3
mac1:xxx me$ ls -lai
total 48
280678 drwxr-xr-x 5 me staff 204 Jan 20 01:23 .
282780 drwxr-xr-x 12 me staff 442 Jan 17 14:03 ..
286678 -rw-r--r--@ 1 me staff 21508 Jan 19 10:43 .DS_Store
135 drwxrwxrwx 91 me staff 3944 Jan 7 02:53 Documents
729750 drwx------ 104 me staff 7378 Jan 15 14:17 D2
728506 drwx------ 19 me staff 850 Jan 14 09:19 D3
mac1:xxx me$ hfsdebug Documents/ | head
<Catalog B-Tree node = 12589 (sector 0x18837)>
path = MyBackups:/yyy/xxx/Documents
# Catalog File Record
type = file (alias, directory hard link)
indirect folder = MyBackups:/.HFS+ Private Directory Data%000d/dir_135
file ID = 728505
flags = 0000000000100010
. File has a thread record in the catalog.
. File has hardlink chain.
reserved1 = 0 (first link ID)
mac1:xxx me$ cd Documents
mac1:xxx me$ ls -a | head
.DS_Store
.localized
.parallels-vm-directory
.promptCache
ACPI
ActivityMonitor2010-12-1710p32.txt
ActivityMonitor2010-12-179pxx.txt
mac1:Documents me$ ls -lai | head
total 17720
135 drwxrwxrwx 91 me staff 3944 Jan 7 02:53 .
280678 drwxr-xr-x 5 me staff 204 Jan 20 01:23 ..
144 -rw-------@ 1 me staff 39940 Jan 15 14:27 .DS_Store
145 -rw-r--r-- 1 me staff 0 Oct 20 2008 .localized
146 drwxr-xr-x 2 me staff 68 Feb 17 2009 .parallels-vm-directory
147 -rwxr-xr-x 1 me staff 8 Mar 20 2010 .promptCache
148 drwxr-xr-x 2 me staff 136 Aug 28 2009 ACPI
151 -rw-r--r-- 1 me staff 6893 Dec 17 10:36 A.txt
152 -rw-r--r--@ 1 me staff 7717 Dec 17 10:54 A9.txt
So you can see from the first few lines of the "ls -a" command, it shows some file/folders but you can't tell which yet. The next "ls -la" command shows which names are files and folders - that there are some folders (like ACPI) and some files (like A.txt and A9.txt) and all looks normal. And the "hfsdebug" info shows some details of what is really happening in the "Documents" folder, but more about that in a bit.
And here are what a "ls -a" and "ls -al" look like for the same locations on the second Mac (mac2) where the Time Machine volume is remote mounted:
mac2:xxx me$ pwd
/Volumes/MyBackups/yyy/xxx
mac2:xxx me$ ls -a
. .DS_Store D2
.. Documents D3
mac2:xxx me$ ls -lai
total 56
280678 drwxr-xr-x 6 me staff 264 Jan 20 01:23 .
282780 drwxr-xr-x 13 me staff 398 Jan 17 14:03 ..
286678 -rw-r--r--@ 1 me staff 21508 Jan 19 10:43 .DS_Store
728505 drwxrwxrwx 116 me staff 3900 Jan 7 02:53 Documents
729750 drwx------ 217 me staff 7334 Jan 15 14:17 D2
728506 drwx------ 25 me staff 806 Jan 14 09:19 D3
mac2:xxx me$ cd Documents
mac2:Documents me$ ls -a | head
.DS_Store
.localized
.parallels-vm-directory
.promptCache
ACPI
ActivityMonitor2010-12-1710p32.txt
ActivityMonitor2010-12-179pxx.txt
mac2:Documents me$ ls -lai | head
ls: .parallels-vm-directory: No such file or directory
ls: ACPI: No such file or directory
... many more "ls: ddd: No such file or directory" error messages appear - there is a one-to-one
correspondence between the "ddd" folders and the "no such file or directory" error messages
total 17912
728505 drwxrwxrwx 116 me staff 3900 Jan 7 02:53 .
280678 drwxr-xr-x 6 me staff 264 Jan 20 01:23 ..
144 -rw-------@ 1 me staff 39940 Jan 15 14:27 .DS_Store
145 -rw-r--r-- 1 me staff 0 Oct 20 2008 .localized
147 -rwxr-xr-x 1 me staff 8 Mar 20 2010 .promptCache
151 -rw-r--r-- 1 me staff 6893 Dec 17 10:36 A.txt
152 -rw-r--r--@ 1 me staff 7717 Dec 17 10:54 A9.txt
If you look very close a hint as to what is going on is obvious - the inode for the Documents folder is 152 on the local mounted case (the first set of code above for mac1), and it's 728505 in the remote mounted case for mac2. So it appears that these "hard links" to folders have an extra level of folder that is hidden from you and that AFP fails to take into account, and that is what the "hfsdebug" shows even better as you can clearly see the REAL location of the Documents folder is in something called "/.HFS+ Private Directory Data%000d/dir_135" that is not even visible to the shell. And if you look closely in the remote mac2 case, when I did the "cd Documents" I don't go into the inode 135, but into the inode 728505 (look close at the "." entry for the "ls -la" commands on both mac1 and mac2) which is the REAL problem, but have no idea how to get AFP to follow the extra level of indirection.
Anyone have any ideas how to fix this so that "ls -l" commands don't generate these "no such file or folder" messages?
I am guessing that the issue is really something to do with AFP (Apple File Protocol) mounted remote volumes. The TimeMachine example is something that I used as an example that anyone could verify the problem. The real problem for me has nothing to do with Time Machine, but has to do with some hard links to folders that I created on another file system totally separate from the Time Machine volume. They exhibit the same problem as these Time Machine created folders, so am pretty sure the problem has nothing to do with how I created hard links to folders which is not doable normally without writing a super simple little 10 line program using the link() system call - do a "man 2 link" if you are curious how it works.
I'm well aware of the issues and the conditions when they can and can't be used and the potential hazards. I have an issue in which they are the best way to solve a problem. And after the problem was solved, is when I noticed this issue that appears to be a by-product of using them.
Do not try these hard links to folders on your own without knowing what they're for and how to use them and not use them. They can cause real problems if not used correctly. So if you decide to try them out and you loose some files or your entire drive, don't say I didn't warn you first.
Thanks ...
-Bob

Answer 1

The problem is Mac to Mac - the volume that I'm having the issue with is not related in any way to Time Machine or to TimeCapsule. The reference to TIme Machine is just to illustrate the problem exists outside of my own personal work with hard links to folders on HFS Extended volumes (case-sensitive in this particular case in case that matters).
I'm not too excited about the idea of snooping AFP protocol to discover anything that might be learned there.
The most significant clue that I've seen so far has to do with the inode numbers for the two folders shown in the Terminal window snippets in the original post. The local mounted case uses the inode=728505 of the problematic folder which is in turn linked to the hidden original inode of 135 via the super-secret /.HFS+... folder that you can't see unless using something like the "hfsdebug" program I mentioned.
The remote mounted case uses the inode=728505 but does not make the additional jump to the inode=135 which is where lower level folders appear to be physically stored.
Hence the behavior that is seen - the local mounted case is happy and shows what would be expected and the remote mounted case shows only files contained in the problem folder but not lower-level folders or their contents.
From my little knowledge of how these inode entries really work, I think that they are some sort of linked list chain of values, so that you have to follow the entire chain to get at what you're looking for. If the chain is broken somewhere along the line or not followed correctly, things like this can happen. I think this is a case of things not being followed correctly, as if it were a broken chain problem then the local mounted case would have problems also.
But the information for this link in the chain is there (from 728505 to the magic-135) but for some reason AFP doesn't make this extra jump.
Yesterday I heard back from Apple tech support and they have confirmed this problem and say that it is a "implementation limitation" with the AFP client. I think it's a bug, but that will have to be up to Apple to decide now that it's been reported. I just finished reporting this as a bug via the Apple Bug Reporter web site -- it's bug id 8926401 if you want to keep track it.
Thanks for the insights...
-Bob