We think so!
When developing in the mobile space, we are always chasing that next device being released. For iOS we have the huge iPad Pro all the way down to the, still viable, iPhone 4s (or older if the client requires it). We are also trying to stay on top of the regular OS updates. There are other factors also in play across so many devices -- processor speed, firmware iterations, retina/non-retina, etc.
Apple provides software simulators in Xcode but as the name denotes, they attempt a close approximation of how your App will behave on a target device. The simulator is not the same as having a real device for testing.
Our library of devices has surpassed 35 devices with the introduction of the iPad Pro. When I type that number, I'm realizing the level of investment we have in all this hardware. Has it been worth it? We believe so and here are some of the reasons why:
- The simulator can't emulate hardware specific features (e.g. bluetooth, camera, microphone, etc.).
- The simulator can't give you a good read on performance because the simulator can be as fast as the Mac you are running it on. Or in some cases a very slow. The iPhone 5 is still a very viable device, but is much slower than the newer 6's. There have been code specific considerations implemented to accommodate slower devices. The simulator would not show us these limitations. So, our older iPhone 5's are used during development.
- Memory usage is not necessarily the same. The simulator does not enforce the same memory limitations as the device. Low memory testing needs to be on a device.
- OS versions are limited on the simulator. Apple pushes for current OS versions. We understand that QA is easier when you have fewer OS versions to test against. We have found that users don't always keep up with maintenance releases and there are times when significants changes (the Bluetooth stack of example) happen. We want to know that our Apps are running well in OS versions a few versions back. Now that I've typed that, if there are significant enhancements to newer OS versions (the networking aspecting of iOS 8 over iOS 7 was substantial) we'll strongly recommend that a client require a newer version of the OS.
Also, if a client has a limited budget, we may suggest that the App be released on the current OS version. This is a substantial savings in QA time. That said, we'll do a cursory test pass on an older OS version just to make sure nothing too heinous is going on.
- Seems kinda of silly, but finger testing vs. mouse testing is important. The simulator recognizes the 1-pixel point of your mouse cursor. The last time I checked, my fingers are a little bigger than that. Gestures are also cumbersome in the simulator. Sometimes a gesture isn't possible in the simulator - 3-finger tap.
There is no substitute for using your App in a real-world environment. Just this week we took several devices into a basement to test spotty cell, gps and wi-fi services. We want to make sure our Apps fail gracefully in these sorts of situations.
Our clients may have their own personal iOS devices, but that's it. As a full service development house our clients have come to expect that we can cover a wide spectrum of device environments so the end product will be something that both we and our clients will be proud of.
We are confident that we can reproduce all the mainstream device environments for our projects. We also pride ourselves in covering some of the edge cases that clients invariably encounter once an App is "out in the wild."
July 13 Follow-up: Looks like Facebook finds value in a device library also! https://techcrunch.com/2016/07/13/facebook-lifts-the-veil-on-its-mobile-device-lab/