Workarounds for 2.0.1 app install crashes

Question

Hi All,
This will be a long post, but hopefully of help to those suffering from the infuriating application related crashes and instability of 2.0 and 2.0.1.
For those just looking for the workarounds, jump to the bottom of this message. For those interested in a technical discussion of what's going on behind the scenes, keep reading.
After running a very stable iPod Touch on 1.1.3 for quite some time complete with jailbroken apps, I decided it was time to give 2.0 and the official third party apps a go.
At first I loved the new features especially the WPA Enterprise support and MS Exchange support which I think are brilliant, but then I started to notice all kinds of problems with crashes like most other people in this forum as soon as I had a few apps installed.
Basically the thing was horribly unstable to the point of being unusable, and I did about 10 full restores in the first two weeks, just to keep the thing going, and experienced dozens of cases of being stuck on the apple boot screen.
The crashes could be narrowed down into two types of crashes - one is crashes while trying to RUN applications, and the second is crashes trying to INSTALL or UN-INSTALL applications.
The first type - crashes while running applications would take the form of either an app crashing back to the home screen when launched, or causing a total system freeze or a spontaneous reboot either when launching an app, or sometimes while running an app.
Usually when this would happen most large apps (hint: memory hungry apps) would fail to launch, good examples are super monkey ball, cromag rally and enigmo. Repeated attempts to launch these apps would fail and eventually the device would spontaneously reboot.
In all cases, a full reboot of the device, (hold sleep button for 5 seconds, swipe to power off, then power on again) would allow the app to run again. The reviews of some apps on the app store are littered with complaints from people that the app crashes during launch - the irony is the app is NOT to blame, in all cases I have seen.
The second type of crash is a lockup and/or spontaneous reboot during app installation or removal either through the built in appstore application or via syncing with itunes. More often than not this type of crash would render the device unbootable with you stuck at the apple logo - you could tell this was happening because after a few minutes the apple logo would go darker, and the device is no longer pingable over wifi.
First the good news - for me anyway, 2.0.1 FIXES the first type of crash - apps crashing back to the home screen, spontaneous reboots and lockups when launching apps or using them. I haven't had super monkey ball fail to launch, or crash during play a single time since updating to 2.0.1 for example. Wahoo Apple.
When not adding/removing/updating applications, I now find 2.0.1 to be quite stable, apart from the usual Safari crashes, which seem to happen a bit more often than 1.1.3.
Now the bad news - the app install / un-install related crashes are NOT fixed in 2.0.1 or even alleviated, so these are what I will focus on in this message describing workarounds I have found and the reasoning behind them.
Ok here's the deal, with a bit of analysis the root cause of the problem is LOW MEMORY conditions that occur during the app install / un-install process. It seems to be a combination of increased system memory requirements of the 2.0 firmware in general (there is simply more stuff running than there was in 1.1.x, so there is less free memory) as well as what look like memory leaks (or at least poor coding) in both Springboard and mobileinstallationproxy. More on those in a minute.
In addition the emergency low memory handling of the system (maybe the kernel) seems to be broken in 2.0.x - once free memory gets below a certain point the system will hard freeze - this never happened in 1.1.x, at most it would cause a reboot if memory became critical and nothing could be done to free memory.
Ok, what is mobileinstallationproxy ? This is a program whose job it is to install and uninstall applications, basically, and did not exist in 1.1.x firmware. When an application is to be installed or removed either directly on the device or by syncing with itunes, this is the process that is launched to do it.
Problem number one - sometimes, this processes memory usage will grow enormously - there is typically only 60MB ram free no additional apps are running and background mailcheck is disabled. mobileinstallationproxy's memory usage will often climb to reach or exceed this amount causing an extreme low memory condition which will cause the Springboard to try to kill off unnecessary processes, but if it can't find enough memory the device will either freeze or spontaneously reboot - the reboot in the middle of application install / un-install corrupts the application state, causing more trouble later on.
Ok, what is Springboard ? This is the main application on an ipod touch / iphone which gives you your home screen, and manages the launching of applications, provides touchscreen input to applications, and also does additional memory management by asking applications to reduce their memory usage if free memory is tight, and will even go so far as to quit background processes to free up memory when memory gets really tight.
Problem number two - sometimes Springboard starts allocating large amounts (over 70-80MB) of memory itself causing a low memory condition - and it cant free up memory because it can't ask itself to quit! At least, not without rendering the device unusable, because you lose the ability to use the touchscreen......
This typically happens when Springboard is "regenerating the application map" which basically means it is looking at newly installed applications to find out details about them and display them on the home screen. This regenerating happens both immediately after application install, and during bootup (before the lock screen) if it wasn't previously finished. BIG PROBLEM.
What happens if the device crashes if you are half way through installing an app ? The answer is that when you try to boot up again Springboard starts regenerating the application map before you are presented with the lock screen - it looks like nothing is happening, but it is. Corrupt applications can cause Springboard to crash during this phase which means you are stuck in an INFINITE LOOP of rebooting showing the Apple logo, which is what many people have experienced.
Problem number 3 - if you have a LOT of applications installed, and just one of them gets corrupted during a crash, when the device reboots, Springboard will start rebuilding the application maps, but this can take a LONG time, and there is a TIME LIMIT on how long Springboard is allowed to run in this state! A watchdog process waits for Springboard to respond during bootup but because Springboard is busy regenerating the application map it doesn't respond. If it doesn't respond in a certain amount of time (approx 5 minutes) the watchdog will KILL Springboard, thus not only preventing it from finishing the job and letting you get to the lock screen, but possibly further corrupting data, and causing the device to freeze, as Springboard is an essential system process.
This is the reason why the system becomes prone to getting stuck on the Apple screen after just a single application becomes corrupted, if you have MORE than a certain number of applications installed. If you have few applications installed (less than approx 10) the situation of one app becoming corrupted is recoverable, and self healing, if you have many applications installed (over 20) there is never enough time for Springboard to finish the job before the watchdog terminates it - thus being stuck at the Apple Logo with no escape other than a system restore.
For anyone who wonders how I am gathering the data to form these conclusions there are two tools that I've used - one is freely available from Apple, and that is the 'iPhone Configuration Utility' available from here:
http://www.apple.com/downloads/macosx/apple/application_updates/iphoneconfigurat ionutility10formacosx.html
Which has the ability to show the local system log (console) of the device in real time, showing things such as:
Sat Aug 9 20:54:28 unknown configd[15] <Notice>: kernel memory event (93), free: 376, active: 1179, inactive: 770, purgeable: 0, wired: 5400
Sat Aug 9 20:54:30 unknown configd[15] <Notice>: kernel memory event (90), free: 376, active: 1398, inactive: 788, purgeable: 0, wired: 5417
Sat Aug 9 20:54:32 unknown SpringBoard[16] <Warning>: Memory level is urgent (10), but there are no apps to warn!
Sat Aug 9 20:54:33 unknown configd[15] <Notice>: kernel memory event (90), free: 376, active: 1455, inactive: 1204, purgeable: 0, wired: 5419
Sat Aug 9 20:54:33 unknown SpringBoard[16] <Warning>: Memory level is urgent (10), but there are no apps to warn!
A great deal of information about what is going on can be gained from watching this log file, and the program can be left monitoring even during an itunes sync.
The second way I have gathered data is by running top over ssh on a jailbroken version of the firmware. I don't endorse or encourage jailbreaking of firmware, but in this instance it has allowed me to gather invaluable data on the problem, and I would stress that the crashes that I've been chasing occur equally on standard and jailbroken firmware.
In particular ssh has allowed me to prove one hypothesis - when the device gets stuck at the Apple logo after an app is corrupted, and is in an unrecoverable state that requires yet another restore, if I quickly ssh in as soon as I can during the bootup phase, and delete all the Appstore applications from /var/mobile/Applications the device will almost immediately spring to life and has been "saved" from requiring another restore - this proves that Springboard is getting bogged down scanning the applications and once they are removed it will boot normally again and the apps can be reinstalled.
WORKAROUNDS AND RECOMMENDATIONS
The biggest problems here are (a) too many apps installed preventing bootup recovery once an app gets corrupted, and (b) low memory situations being encountered during app install / uninstall causing the corruption in the first place.
1) Don't install or update Apps using the built in Appstore application on the ipod/iphone. For reasons not entirely clear this is a very hit and miss proposition and there is a VERY high chance that you will get a freeze or a crash from doing this if you have more than a few apps installed, and end up in an unrecoverable state. Just don't do it, always sync apps across from itunes.
2) Free up as much memory as possible BEFORE syncing applications with itunes, and this applies both to installing and un-installing applications. The best way to do this is disable MobileMail from running in the background. If you have automatic message checking of any kind enabled MobileMail launches at bootup and runs in the background ALL the time, consuming approximately 14MB of ram, and this is often the straw that broke the camels back when memory gets tight.
To do so, disable PUSH, and change Fetch new Data to Manually, then either do a full reboot (sleep button for 5 seconds then swipe to turn off, then press to turn on again) OR launch mobile mail, wait for it's mail check to finish, then hold down the home button for 5 seconds to do a forced quit. It's imperative that automatic checking of mail is disabled, or it will immediately launch in the background again.
This step of disabling MobileMail to free up a bit more memory makes the most difference of all the things I have tried in preventing crashes during application syncing, so don't skimp. You can enable automatic mail checking again afterwards.
3) As much as it pains me to say, having downloaded over 30+ apps, if you want your ipod/iphone to be stable, until Apple fixes this issue, install as few apps as you possibly can - just the ones you really need, and not lots of gimmicky things you don't really need. Try to keep it under 20 at any one time, and you can do this on the Application tab in iTunes telling it to sync only selected applications. Keeping the number of synced applications down to a minimum makes a big difference, and means that even if you do get a crash during app installation it will usually recover. If you have lots of apps installed it will NOT recover by itself.
Good luck all, and roll on 2.0.2.

Answer 1

If you DO get in the situation where the iPhone won't load and is stuck at the Apple logo, there is an alternative to Restoring: renaming the Applications folder momentarily.
Of course this assumes you have a way of getting to the Applications folder before the phone has fully booted. I am lucky as I have OpenSSH installed and when this crash occurs the WiFi has already connected, so I can (for example) log in and rename Applications to Applications2 and then back again.
The Springboard will start about 5 seconds after you rename the folder. This at least gives you an opportunity to back stuff up or try to remove the offending application instead of restoring.
I have heard that you can use iPhoneBrowser on Windows to do the same but not tested this.