ICloud disaster destroyed all my Pages work on the iPad!

Question

I have an IPad 2 with Pages that I have used for months to write business articles and research papers for publication. I have been using iCloud to synch it through a single user account.
Last week, I was forced to upgrade my iMac to Lion because corrupted account permissions locked me out of all my home folders as well as my two different external drives and even my Time Capsule and Time Machine --- something supposedly impossible. But it all inexplicably happened. It took over a week for assorted sympathetic Apple care specialists and engineers to try to sort it out. The original problem was still never diagnosed or solved. Instead, my hard drive was erased, OSX 10.6 reinstalled, a date back up restored, and then Lion loaded as new. 
As per Apple care instruction, iCloud in Lion was immediately activated on the iMac -- but no document files were ever shared.
Four days later, after synching my iPad to iCloud, when I next opened Pages I watched in horror as in a mere second more than 30 documents vanished en masse right before my eyes.
Somehow all these files never got backed up in my iTunes even though I had made several recent saves to the hard drive as well as an iCloud back up!
And as for the files or even the mobile documents library folder being save in my recent Time Machine back ups, they never were. The back ups contain no iOS Pages text nor any deleted folder files to restore. 
Despite all the Time Machine and iTunes back ups, and despite all the free space on the iPad where I actually did all the work without an internet connection, as soon as I synched they were mysteriously deleted by iCloud. 
Pages on iPad was left empty and blank. Replacing the mobile documents folder on my iMac from a back up did at least managed to provide Pages file some icons with names dates, but they are ghost documents, mere skeleton files that can’t be opened. All my actual text content is now apparently nowhere; lost in Apple’s iCloud ether!
This is an iPad which for months I used for Pages work without any internet connection, 3G or WiFi, and yet in an instant the iCloud on its own disintegrated everything it contained!  Apple essentially reached down into my device, into my personal property that I own, and without warning, notice, or permission and simply erased all my work; all my personal labor and effort that belonged to me. They essentially committed sabotage and theft.
I have nothing saved anywhere despite being told how the iCloud was a trustworthy safety net, that Time Machine provided worry free back up, and that saving through iTunes could restore iPad content. Garbage. I did everything right and still got screwed.
Further…
In just the past two weeks alone I have suffered: my second iPhone -- a month old replacement for a previous defective iPhone itself only a month old -- experience a “software corruption” that caused it to continuously shut down if unplugged; my iMac and Time Capsule failing; and now this catastrophic iPad disaster.
If this is not resolved with my fiels returned to me I simply will never use iCloud and I will never trust Apple again.
I am disgusted and furious beyond words.
JC
Atlanta

Answer 1

Hi JC,
My iPad is in a similar state, although I arrived in the bizarro state by a different path.  I am also waiting for the apple experts to get back with me.  I have an iPad 1 with ios 5.1.1 and Pages 1.7.1.
In the mean time, while I was waiting, I found a way to extract the TEXT from a large Pages document I did not want to lose;  it was the result of many years of notes.  This might save you some time instead of sifting through the raw index.db. 
The index.db file is a sqlite3 file.  If you are on mac you can open it from a terminal with the command:
sqlite3 index.db.
I suggest you move the file out of the iTunes directory before you start working with it, just in case something unexpected happens, you don't want to lose the only copy of the data you are trying to recover. 
1) Once the file is open via 'sqlite3 index.db' from a terminal command line, type ".tables" at the 'sqlite>" prompt and press enter.  It should show five tables:
cullingState
dataStates
document
objects
relationships
2) The file with your data is named dataStates, but it has many records in it.  I found my data by looking for the "largest" record.  Let me explain.  There are two fields in the dataStates table:
identifier
state
Your data is stored in the 'state' field and it's data type is 'blob'. (I know that sounds stupid, but stick with me.)
3) I found the largest record by typing  this query at the 'sqlite>' prompt:
select identifier, length(state) from dataStates order by length(state) asc;
My 'dataStates' table had 599 records, so many lines fly by, but the largest state field is at the bottom of the list with it's identifier.  Mine was:
3|173212
That identified a dataStates record with identifier '3' and 173,212 bytes of data in the 'state' field. Most of the bytes where printable ascii.  It WAS the text of my original Pages document.
DISCLAIMER:  This method does not retrieve ANY formatting, pagination, tables bullets or numbering, images or other document rendering information.  It is just the TEXT.
4)  Type this SQL command at the 'sqlite>' prompt to pull out the blob data:
select hex(state) from dataStates where identifier=X;
In my case the identifier was 3 so MY command was:
select hex(state) from dataStates where identifier=3;
5) Now is when the funs begins.   The large block of gibberish is 'ascii encoded binary' brought to you courtesy of the hex function. 
To store it in a file, use the the sqlite3 '.output' command and type the preceding sql select.  Like the following:
.output binacii.txt
select hex(state) from dataStates where identifier=3;
.output stdout
This takes the large block of 'binary encoded ascii' and writes it to file named 'binascii.txt' rather than displaying it in the terminal window.  After the select statement ends, the output is switched back to the terminal window (stdout).    The 'binascii.txt' file is located in the same folder/directory that was the current directory when you launched the sqlite3 command.
6) I moved my 'binascii.txt' file to a windows computer at this point and wrote a simple conversion routine to decode the 'ascii coded binary' into something usable with a work processor.  In "ascii coded binary", each 8-bit byte of data is encoded as two ascii characters, it enables sending binary data via the internet, etc, similar to uuencode, but NOT the same.
7) Here is a small python program to convert the 'binascii.txt' file to the text from your Pages document on a Mac:
import sys
p = sys.stdin.read()
s = []
for location in range(0, len(p), 2):
   value = int(p[location:location+2], 16)
   if value >=32 and value <=126 or value==10:
       s.append(chr(value))
sys.stdout.write(''.join(s))
8) (NOTE: The last line has two single quotes (') not a double quote (")). I am assuming at this point that you know how to create a text file using a terminal window.   Add the python script to a text file, make sure you honor the indentations!  I named my python file 'convert.py'.  To use the python script, type the following command:
cat binascii.txt | python convert.py > your.txt
This method also assumes that the files 'binascii.txt', 'convert.py' and 'your.txt' are all in the same folder/directory.
9) The resulting file, 'your.txt' can be opened by Word or Pages (you can name the file 'your.txt' to whatever you want).  In my case, I opened the file 'your.txt' with MS Word.  It had some rudimentary formatting in the form of paragraph breaks, single lines, some headings etc.  I did lose the data stored in tables, although I expect the text is still lurking somewhere in the index.db file.
There were some ascii characters that were not part of my original document, but they were pretty easy to find and remove.  In my case the errant ascii characters were outside my document's large text block.   Also numbered and bulleted blocks run together.  Again the formatting information is still buried within the index.db file.
The iOS Pages file format is more cryptic than the simple zip/XML format from the desktop OS X version of Pages.  
Hope this helps a little, although you seem to be OK with the approach of sifting through the raw index.db file.  The Pages document  I wanted to recover was 63 pages long and manually sifting through all the crap in the index.db  drove me on a quest to find the real text within the index.db file, which i found.
Sorry for the length of this post,
Cheers