How Apple caused our iPhone app to crash for 3 days

Like many app developers, we updated our app when iOS 8 was released. We threw in some Swift code and fixed a few bugs. Our new version worked flawlessly in Testflight for iOS7/8, and everything looked good. On Wednesday (October 1st) we submitted the new binary to Apple, sat back and waited.

There is always a fair bit of nail biting that takes place when we submit an app. Your app goes into a black box and comes out in an indeterminate number of days after that. There were a few bug fixes (the version of Stripe we were using had a critical SSL bug—the only solution was to upgrade) in our new version so we asked for an expedited review, which was granted.

When the e-mail comes that says "The Status of Your App is now Ready for Sale," it is usually a series of joyous moments. This time, however, was different: when we downloaded the new version of the app from the store, it immediately crashed on iOS 8. The first user complaints came within hours. People started ringing the support line, e-mailing us, tweeting at us and leaving messages on our Facebook page:

What followed was 3 days of hair-tearing. Was it the Swift code that had been added? Was it a runtime error being caused from values retrieved from NSKeyedarchiver? Why did it work in development but not in production? Why did the new version of the app work on iOS 7 but not iOS 8? Was anybody else experiencing a similar problem? Why would Apple post a version that crashes on launch, when this is pretty much the first thing they look for?

I was unable to reproduce the crash in development, and Apple wasn't showing any crash logs in iTunes Connect. I had a hunch that Apple was dropping the ball when processing the binary for sale in the app store because of a similar situation Marco Arment experienced with Instapaper in 2012.

To make matters worse, there is no such thing as a quick fix with Apple, even if you request an expedited review. So we turned to Stack Overflow and forums, quickly noticing that other people were experiencing this exact problem.

One suggestion, which nobody had confirmed as working (due to the long app submission cycle), was to switch the flag "Embedded Content Contains Swift Code" from NO to YES. With little else to do we resubmitted the app with this one change. This was ultimately a red herring, as inclusion of Swift in the build had nothing to do with the crash.

Finally, in the Apple Developer Forum things started to become more clear despite no response from Apple.

We decided to wait it out, and indeed, 8 hours later I woke up to a working app build, with my re-submission not yet having been reviewed.

The closest we got to an officail acknowledgement was a retweet by Apple's Developer Evangelist Dave DeLong.

Some developers reported 1-star reviews were removed from their App's listing in the App Store.
I had also created a Developer Technical Support ticket with Apple, which was later resolved and not charged to my account, so they've done their best to make the situation better. In future more transparency and clear communication would be ideal.

P.S. I make Gymsurfing, we sell discounted day passes to gyms. Use the promo code "dustin" after signup for $5 of free credit.