Loading a large number of strings into memory quickly

Question

Hello,
I'm working on an iPhone application where I need to load a large number of strings into memory. Currently I'm simply reading from a file where each string is stored in plain text on a single line. I read the file contents into a string using stringWithContentsOfFile and then I create an NSSet object using NSSet setWithArray:[string componentsSeparatedByString:@"\n"]];
This works like a charm but takes around 8 seconds to load on the iPhone. I'm looking for ways to speed this up. I've already tried a few things which weren't any faster:
1) I used [NSKeyedArchiver archiveRootObject:myList toFile:appFile]; to store the NSSet data structure. Then instead of reading the text file storage. I read this file using [NSKeyedUnarchiver unarchiveObjectWithFile:appFile]; This was actually very slow and created a strings file that was about 2x the size of the original plain text.
2) Instead of using an NSSet, I used and NSDictionary and used writeToFile and dictionaryWithContentsOfFile. This was also no faster.
3) Finally I tried using the NSDictionary to write to a binary file format using NSPropertyListSerialization. This was also not any faster.
I've been thinking about using SQLite instead of the flat file read, but I haven't had a chance to prototype that out to see if it would be faster. It's important that I can do fast searches for specific strings, which is why I originally used a set.
Does any one else have any ideas how to load this into memory faster? If all else fails, I'm simply going to load the strings into memory using a separate thread on application launch so I don't prevent the user from getting to the main menu for 8 seconds.
Thanks,
-Keith

Answer 1

I'd need to know more about what you're doing, but from what you're describing I think you should try to change your algorithm.
For example: Instead of distributing one flat file, split your list of strings into 256 files, based on the first two hex digits of their MD5 hashes*. (Two digits might not be enough--you might need three or four. You may also want to use folders, especially if you need more than two digits.) When testing if a string exists, first calculate its MD5 hash and extract the necessary number of digits, then load that file into memory and scan its list. (You can cache these lists in memory so that you only have to load each file once--just make sure that a didReceiveMemoryWarning message will empty those caches.)
Properly configured, SQLite may be faster than the eight second load time you talk about, especially if you ensure it indexes the column you store the strings in. But it's probably overkill for this application.
\* A hash is a numeric code calculated from a string; on average, changing a single bit anywhere in the string should change half the bits in the hash, so even very similar strings should generate very different hashes. I suggest using MD5 instead of -\[NSString hash\] because the hash method is not guaranteed to return the same results on Mac OS and iPhone OS, or on different releases of either OS. You could also use a different algorithm, like a CRC; these are faster but I'm not as familiar with them. This thread discusses calculating MD5 hashes on iPhone OS: http://discussions.apple.com/thread.jspa?messageID=7362074
Message was edited by: Brent Royal-Gordon