Cucumber running on real devices in the cloud
October 06, 2022

Cucumber running on real devices in the cloud

Anonymous | TrustRadius Reviewer
Score 9 out of 10
Vetted Review
Verified User

Modules Used

  • Sauce Labs Real Device Cloud

Overall Satisfaction with Sauce Labs

As part of our development build pipelines, we are using Sauce Labs to perform our Cucumber automation tests on real devices in Sauce Labs. Specifically, we're using the command line tool for running tests both from our local dev machines as well as within our Bitrise CICD. We have a dedicated integration automation testing team who are working using Selenium to build a set of automation tests for both Android and iOS The hands-on QA team also use real devices for ad-hoc testing on specific devices or OS versions

Pros

  • Selection and availability of a multitude of mobile devices
  • Simple-to-use command line tools and configuration files have proved to be quick to setup
  • Test results with videos and bookmarks make it easy to view failing tests

Cons

  • The website UI isn't always the easiest to navigate around, and some of the menus aren't always obvious.
  • Cancelling test runs from the command line interface doesn't seem to work, and it doesn't appear possible to stop a running test.
  • I'd like the ability to pull data files off the devices at the end of the test runs.
  • We are currently too early in the process to answer this
Firebase is pay per use and so was difficult to work out the true costings, it also felt more developer focussed whereas Sauce Labs had better Appium support for our Automation team. AWS Device Farm appeared to require either Android or iOS and we wanted both, so that wasn't a great fit. It was also not possible to run Cucumber tests on Android in AWS but this may no longer be the case, but it was another unknown for us and not worth the risk. Sauce Labs explicitly mentioned Cucumber support.

Do you think Sauce Labs delivers good value for the price?

Yes

Are you happy with Sauce Labs's feature set?

Yes

Did Sauce Labs live up to sales and marketing promises?

Yes

Did implementation of Sauce Labs go as expected?

Yes

Would you buy Sauce Labs again?

Yes

Having used some of the competitor's tools over the year I would say that SauceLabs provides a lot of value for money if you plan to run long sets of tests with high frequencies. Paying for a single slot means you can run tests whenever you want without creeping costs but it does make running tests in parallel require an extra slot. Currently, our test suite takes over three hours to run and at the moment it is cost prohibitive to purchase an extra slot. However, having access to live testing and integration with Appium is great.

Comments

More Reviews of Sauce Labs

  1. Home
  2. Functional Testing Tools
  3. Sauce Labs Reviews
  4. User Review of Sauce Labs