Sauce Labs Customer Idea Portal

Submit an idea and make Sauce Labs even better!

Time out queued tests within a minute

The Problem/s

  1. Customer runs more tests then their concurrency allows
  2. Tests queue for more than a minute
  3. The customer's http library times out the connection, throwing an inscrutable http exception that looks like a Sauce Labs error
  4. The error on the Sauce Labs job is "Browser request was cancelled before a Sauce Labs virtual machine was found" which also sounds like a Sauce Labs error
  5. Customers have no idea what's wrong

OR

  1. Customer runs more tests then their concurrency allows
  2. Tests queue for more than a minute
  3. Tests show as particularly slow on either the Sauce Labs UI or the customers test report
  4. Customer gets real mad about Sauce Labs being slow

 

Suggested Solution

Don't let customers queue for so long

Lower the time tests are allowed to queue before we time them out to under a minute.  We consider it an antipattern for customers to queue multiple tests, yet by default we let them do it for about 10 minutes.  Customers could specifically opt into a longer queuing time; making it clear they've asked to violate best practise.

Make it more apparent when tests have been cancelled by the user

Change the error message to reflect that the users test runner cancelled the browser request, not Sauce Labs.

Notify Customers when this happens

Either through popup or email, notify customers that their trying to run too many tests and it's freaking their test runner out.

 

Impact to Customers

Customers without queuing problems will be unaffected.

New Customers with queuing problems will discover them more easily and have a more satisfying onboarding experience.

Existing Customers with queuing problems will start to have more test errors until they stop over-queuing.

 

Impact to Sauce Support

Less tickets.

 

Potential Problems

Existing suites with substantial queuing will start to fail.

Customers who don't want to manage their own concurrency will get even madder.  This is an occasional complaint, despite concurrency limits not changing that often.

  • Guest
  • Jul 7 2015
  • Future consideration
  • Attach files
  • Guest commented
    7 Jul, 2015 07:38am

    Uh... Not sure why this got created in Web or how to move it -_-

  • +1