Insane XML Import, Huge Project, Duplicate file names work around...

Question

I planned on kicking off my journey attempting to edit a massive multi year documentary on FCPX with a nice introduction of the blog I'm going to keep about the experience, but I've run into a bit of a roadblock, or maybe major speed bump at least before even getting to that point. I wanted to share what is working as a work around for me and you guys can tell me how I'm doing it wrong.
Ok, I will try to explain this as succinctly as possible. I'll write in somewhat stream of consciousness just to try and get through it quicker... Basically, after discovering the work around below, I am now utterly confused on how FCPX handles the relationship between its own database of where media is stored, and the actual media itself. I have plenty experience on FCPX now, probably done 30-40 pro commercial jobs on it over the last year since XML became doable as I'm also a Resolve Colorist and all the FCPX projects where hardcore coloring product spots. For commercial work, I never needed to worry about splitting up footage up over multiple Events. Everything, all in one, FCPX handled it no problem. (well the occasional beach ball, but that seems to be a thing with FCPX in general)
This will be my 10th feature documentary as an Editor. Every one before it was either on Avid's many flavors over the last 12 years or FCP Studio. When this new film came along, I made the decision a few months ago to use FCPX for a few reasons, but mostly because I'm insane and I like to try to mix it up for myself in a career that can get stale quick if you aren't willing to be that way. The film was shot over 2+ years, every shoot was multi cam 5D (yes i know, looks great, but please kill me), I haven't done the math on length, but there is over 10,000 clips of video (this is actually medium in size compared to what I've dealt with before). Its 5D, so theres external audio for everything. FCPX's syncing is great, but I've learned that theres an unsaid window of heads and tales clips must fall within to sync properly with the nearby clips, if they are too far apart FCPX gives up. One shoot day could have 3 cams, 50 clips each, and 2 audio files to sync to, FCPX simply cannot handle this, so off to Plural eyes they went, no problems.
Ok, all this is relevant eventually I swear! Again, in the past, all in one event, no problem. I tried for fun to bring all media into one Event on this film. It worked, but there is a 10+ second spinning beach ball for every single move you make, so thats no good. Ok, I'll separate the Events out to, lets say, each shoot month. Well that's dumb, in verite documentary, any shot could be the first, any shot could be the last, you need a command over all searchable footage at all times. Shift selecting all events to search *****, and it actually takes longer for FCP to reload each event each time than it does to just have the one massive one. So no go there. Next hair brained idea... What if make a new Event that is just Compound Clips of all the other Event's contents and do more with Markers and Favorites in logging that I was planning to parse it all out. That way I'm working with and FCPX is dealing with 50-60 clips instead of 10,000+ Quick test, Cmd-A, Opt-G, boom, boom, boom, move all to dedicated to Event, hide huge Event, BEHOLD, that works! FCPX chokes a little bit on the insane length of some of the clips, but searching, and general performance is back on par!
So your saying to yourself "Ok *********, sounds like you figured it out, what's the problem." Well to you I say, "Not so fast!" Remember, that was just a quick test using the media I had imported into the massive 10,000+ clip Event. To do this project proper, I am having to import Multicam sync'd XMLs from Plural Eyes. And this is where it all starts to fall apart. A little foreshadowing for your eager eyes. 10,000+ files all shot with multiple 5D's over the course of years. What does that mean? many, many duplicate file names!
FCPX as well all know irritatingly imports XML's as new Events, not into existing ones. This obviously takes a lot of burden off media management because with a new Event comes a new database referencing its own version of the raw media. All well and good, and I'm betting its starting to click for some if you advanced users where I'm finally going with this. So I have 50 or so XMLs to bring in, all done no problem. Now I want to replicate that singular Event like I did with the Compound Clip test and have it all in one place to be my master as extensive logging begins for easy searching once editing begins. Highlight the Events, click Merge Events. NOPE. I get a new "Kill Yourself Now" error (a term I coined for Out of Memory and General Error messages in FCP Legacy meaning there ain't much you can do about it): "Two or more files have the same name. Change the names and try again, because I don't know what the **** to do with them." Ok I made up that last part but that's basically what it's saying. Just take the variable out of the equation, this happens with every which way you could try to get the clips together. Merge Events, dragging events on top of each other, dragging just the Multicam clip alone, nothing gets passed that message. What's worse is that while Batch Renaming seems like a solution, the renames do not populate inside the created clips and there is no way to Batch Rename those. Renaming everything at the finder level isn't so great because then I'd have to resync and theres an offline/online thing going here where the film has to be reconformed eventually.
Basically, I've found that FCPX handles media management in completely different ways depending on whether you are importing into one Event yourself or doing essentially what is a new import with FCPX moving or merging Events. If you bring in all the media to one Event, on a macro level FCPX goes through file by file making aliases referencing the master file. If it hits a duplicate, it makes a parenthesis counter, and keeps going. And with the genius of FCPX metadata, that file name doesn't even matter, you can change it at the Finder level and FCPX will keep the link intact. BUT for some reason if you try to do this outside the realm of a single Event and combine files after the fact a different process takes over in creating this database system and can't handle the duplicates. I can't totally figure the reason other than it probably is scared to change the originally referenced alias to something else (which it shouldn't be scared of since Merge Events deletes the original anyway).
This is where it gets INSANE and where I lose all understanding of what is going on, and believe me it was a delicate understanding to begin with. It comes in to play with the work around I figured out. I make the master Event with the 10,000+ clips. Then I import all the XMLs as dedicated Events. Now, I then drag the Multicam clips into the master Event, it WORKS! just takes it, no "Kill Yourself Now" error. Stranger still, now with the switched referenced Event, it even takes into account which aliased duplicate file name it's referencing in the Original Media folder. Somehow, it's taking into account the original file path and saying "Ok, I see 5 instances of MVI_5834.mov. Based on the original file path or maybe creation date or god knows what, it must be MVI_5834 (fcp3).mov." It connects perfectly. I can even remove the old XML imported Event with no problem. Crazier yet, I can now move those again to the dedicated Event I wanted originally that only contains those Multicam or Compound Clips.
So instead of going straight from A to C, going from A to B to C works even though that actually seems way more complicated. Why can't FCPX handle Merge Events and dragging clips the same way it handles media imported into a single Event. And weirder still, why can't FCPX handle the (fcp1,2,3...) appending in the same scenario. But if the appended links are already there, No Problem. And for the love of god, it'd be nice to important XML's into existing Events and make the correct referencing happen right from the get go, which is really the source of all the above headache. I'd have no problem helping FCPX with a little manual pointing in the right direction just like any other NLE needs.
Ok, having said all of that crap above, my question is, have I missed something completely simple I should have done instead? Now that I have everything in place how I originally envisioned, I think I will still play around a little bit more to make sure FCPX is really going to be able to handle this project. I'm at a stage right now where going another direction is still an option, although the dare devil in me wants to make this work. Media management aside, once you start editing on a FCPX timeline, its hard to go back to anything else. Apple is going to have to figure out some way not to access to everything at all times to work fluidly or big projects like this are never going to be practical in FCPX.
Sorry for the long confusing post....

Answer 1

I'm having the exact same problem, but I know nothing of ruby scripts. I've exhausted my resources with tech support, and after literally hours of work with them it seems I now am faced with either re-rating each individual song, or pointing iTunes to each song that it can't locate. Is yours a solution that could help me? How can I find out more about Ruby Scripts? Thanks for your help, hellolylo (or anyone else who might also be able to help...)
Kenn