Common native mobile app test errors, what they mean and how to resolve them
Troubleshooting Native Mobile Tests
This document lists some common problems and errors that can occur when running mobile tests and how to troubleshoot them.
It's possible to test mobile apps using real devices (via AWS Device Farm) or our mobile VMs. In both cases, customers should create for each app an App URL (Settings > Sites > Add App) pointing to the address from where the app will be downloaded.
The App URL should lead to a direct download, which means it should not redirect, require credentials or have access limited to some network or IP.
At the beginning of each Run, the app will be download from the specified App URL and automatically installed on the real devices or VMs. The Run will be errored if for any reason the download or the app installation fails.
Also, the apps should be correctly build and configured to run on the real devices or VMs. For more information, please check the following support docs:
- Test Native iOS Application on a Real Device
- Test Native iOS App on a Virtual Machine
- Test Native Android Apps on a Virtual Machine
- Test Native Android Apps on a Real Device
Common Error Messages
Runs failing with "Couldn't download file…"
This error occurs when the application fails to download from the provided app URL.
To resolve this, follow the below steps
- Check if the test is pointing to an App URL and not a Site URL on all environments.
- Check if you can download the app using only the specified App URL without a redirect, providing any credentials or having to click on any link/button to start the download.
- Apps hosted on Dropbox should specify dl=1 as a query parameter on the App URL to force the app to download.
Runs failing with "Please check that your app is configured properly."
This error usually occurs when it was not possible to install or launch the app on the real device or VM.
- First, check the App/Device Logs.
- Check with the customer if he have provided a valid app file:
iOS real devices:
- Should be an .ipa file.
- build with a valid "Minimum OS Version".
- Apps build with iOS 10.0 as the "Minimum OS Version" can't run on iOS 9 devices!!
- Should be an .zip file with the .app file included.
- Should be compiled for x86.
- The following line in the App Logs indicates that an app was not build to run in the simulators: Program specified by service does not contain one of the requested architectures
- When creating the .app file the user must inform the -sdk iphonesimulator param in the xcodebuild command.
- Users can test the .app file locally by following the troubleshooting instructions provided here.
- Should be an .apk file.
- Should be compiled to run on x86.
Android real devices:
- Should be an .apk file.
- Should be compiled to run on x86 and ARM.
- Samsung Galaxy Tab 3 10.1" runs on x86!
Runs failing with "There are not available xxx devices at the moment…"
This error means that at this moment there are not available phones of the requested type due to maintenance issues on AWS Device Farm. It can also shows up when the app was not build to support the device's OS version.
- Verify that it is possible to reproduce the same error on preview. If this can be reproduced, this means that this device is on maintenance on Device Farm.
- When this occurs let us at firstname.lastname@example.org or through Intercom so that we can monitor this. Usually, AWS will make the device available again after some hours.
- If the preview works, please check that the app was built to run on the device's OS version. It is particular important to check the app's "Minimum OS Version".
Testers reporting black or frozen terminals on iOS real devices
This problem can happen when the app crashes in the middle of the test or because a problem on our script to dismiss permission dialogs.
- First check the Device Logs; usually the error/crash will be in the second log file.
- Check the device video. In the moment where the testers are reporting the problem. If the app opens Safari or any other external application, please change the test as testers can only work within the app in a sand.
- If in the video a system permission dialog appears, please let us know at email@example.com and we will investigate with our engineering team.
Browser loaded instead of the App in the beginning of the test
If and when the testers report that the browser was showed instead of the app this usually means that the app crashed right after be launched. If this occurs, please check the App Logs for the error event and watch the tester video.
App not installed on Android real devices
On Android tests using real phones (via Device Farm) the app will not be installed if it does not support the Android version installed on the devices.
- Ask the customer to check the App Logs + Videos.
- Usually, the following error message will appears in the logs when the app requires a newer Android version: "Requires newer sdk version #XX (current version is #YY)"
If you encounter error that is not detailed above, please let us know at firstname.lastname@example.org.