BUG: SauceLabs + Safari Flake / Bug (?)

[ryan-roemer]

BrowserStack + Safari is failing like 3/4 times or something.

Now Safari + SauceLabs is too which means there may be a poor test / bad adapter / bad library issue with how we deal with maybe client teardown and the browser window not being in focus / available ??

Different SL issues:

• ERROR: https://saucelabs.com/tests/98141cc403c647d996a0f70e79f210dc
• FAILURE: https://saucelabs.com/tests/6bdc969a2bad43d5b79da123837ece0e

/cc @Maciek416

Using worker: worker-linux-docker-f8ef7611.prod.travis-ci.org:travis-linux-2
## SNIPPED ##
$ export ROWDY_OPTIONS='{ "client":{ "logger":true }, "server":{ "logger":true } }'
$ export SAUCE_USERNAME=[secure]
$ export SAUCE_ACCESS_KEY=[secure]
$ export BROWSER_STACK_USERNAME=[secure]
$ export BROWSER_STACK_ACCESS_KEY=[secure]
$ export ROWDY_SETTINGS="sauceLabs.safari_7_OS_X_10_9_Desktop"
## SNIPPED ##
-----
0.01s$ if [ "$ROWDY_CHECK" ]; then npm run check; else true; fi
The command "if [ "$ROWDY_CHECK" ]; then npm run check; else true; fi" exited with 0.
0.01s$ if [ "$ROWDY_CHECK" ]; then ROWDY_OPTIONS="{ \"client\":{ \"logger\":true }, \"server\":{ \"logger\":true,\"port\":4321 } }" npm run test; else true; fi
The command "if [ "$ROWDY_CHECK" ]; then ROWDY_OPTIONS="{ \"client\":{ \"logger\":true }, \"server\":{ \"logger\":true,\"port\":4321 } }" npm run test; else true; fi" exited with 0.
192.42s$ [ "$ROWDY_CHECK" ] || npm test
> rowdy@0.3.1 test /home/travis/build/FormidableLabs/rowdy
> npm run test-wd && npm run test-webdriverio
> rowdy@0.3.1 test-wd /home/travis/build/FormidableLabs/rowdy
> npm run test-wd-default-config && npm run test-wd-manual-config && npm run test-wd-per-test-client
> rowdy@0.3.1 test-wd-default-config /home/travis/build/FormidableLabs/rowdy
> cd examples/mocha/wd && mocha
  notes - wd.js
  ✓ adds a note and deletes it (4650ms)
  ✓ adds a note and edits it (4493ms)
    with refreshed (new) client
  1) "before each" hook
  2) "after all" hook
  2 passing (3m)
  2 failing
  1) notes - wd.js with refreshed (new) client "before each" hook:
     Error: timeout of 60000ms exceeded. Ensure the done() callback is being called in this test.

  2)  "after all" hook:
     Error: timeout of 60000ms exceeded. Ensure the done() callback is being called in this test.

npm ERR! Linux 3.13.0-40-generic
npm ERR! argv "/home/travis/.nvm/versions/node/v0.12.7/bin/node" "/home/travis/.nvm/versions/node/v0.12.7/bin/npm" "run" "test-wd-default-config"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! code ELIFECYCLE
npm ERR! rowdy@0.3.1 test-wd-default-config: `cd examples/mocha/wd && mocha`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the rowdy@0.3.1 test-wd-default-config script 'cd examples/mocha/wd && mocha'.
npm ERR! This is most likely a problem with the rowdy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     cd examples/mocha/wd && mocha
npm ERR! You can get their info via:
npm ERR!     npm owner ls rowdy
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/travis/build/FormidableLabs/rowdy/npm-debug.log
npm ERR! Linux 3.13.0-40-generic
npm ERR! argv "/home/travis/.nvm/versions/node/v0.12.7/bin/node" "/home/travis/.nvm/versions/node/v0.12.7/bin/npm" "run" "test-wd"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! code ELIFECYCLE
npm ERR! rowdy@0.3.1 test-wd: `npm run test-wd-default-config && npm run test-wd-manual-config && npm run test-wd-per-test-client`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the rowdy@0.3.1 test-wd script 'npm run test-wd-default-config && npm run test-wd-manual-config && npm run test-wd-per-test-client'.
npm ERR! This is most likely a problem with the rowdy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm run test-wd-default-config && npm run test-wd-manual-config && npm run test-wd-per-test-client
npm ERR! You can get their info via:
npm ERR!     npm owner ls rowdy
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /home/travis/build/FormidableLabs/rowdy/npm-debug.log
npm ERR! Test failed.  See above for more details.
The command "[ "$ROWDY_CHECK" ] || npm test" exited with 1.
Done. Your build exited with 1.
$ export DISPLAY=:99.0######################################################################## 100.0%

[ryan-roemer]

More failures:

  notes - wd.js
  1) "before each" hook
  0 passing (11s)
  1 failing
  1)  "before each" hook:
     Error: [get("http://backbone-testing.com/notes/app/")] Error response status: 23, NoSuchWindow - A request to switch to a different window could not be satisfied because the window could not be found. Selenium error: The driver is not focused on a window. You must switch to a window before proceeding. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 19 milliseconds
Build info: version: '2.40.0', revision: 'fbe29a9', time: '2014-02-19 20:54:28'
System info: host: 'itako46144.prod.miso', ip: '172.20.46.144', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9.5', java.version: '1.7.0_75'
Session ID: null
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{platform=MAC, cssSelectorsEnabled=true, javascriptEnabled=true, secureSsl=true, browserName=safari, takesScreenshot=true, version=7.0.6}]
      at _stream_readable.js:908:16

[ryan-roemer]

Local reproduction of issue:

  2 passing (14s)
  1 failing

  1) notes - wd.js adds a note and deletes it:
     Error: [element.type("Delete Test")] Error response status: 10, StaleElementReference - An element command failed because the referenced element is no longer attached to the DOM. Selenium error: Element does not exist in cache (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 106 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/stale_element_reference.html
Build info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'
System info: host: 'L-SB80X4FFT4-M.local', ip: '192.168.2.34', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9.5', java.version: '1.8.0_45'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{browserName=safari, takesScreenshot=true, javascriptEnabled=true, version=7.1.7, cssSelectorsEnabled=true, platform=MAC, secureSsl=true}]
Session ID: null
      at _stream_readable.js:944:16

[ryan-roemer]

Seeing multiple tabs locally when things fail. May be an issue of:

1. Not finished tearing down previous
2. Get two tabs and focus on wrong one...

[ryan-roemer]

Some screenshots after a hang with:

  1 passing (38s)
  1 failing

  1)  "before each" hook:
     Error: [get("http://backbone-testing.com/notes/app/")] Error response status: 28, ScriptTimeout - A script did not complete before its timeout expired. Selenium error: Timed out awaiting response to command "get" after 30004 ms (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 30.01 seconds
Build info: version: '2.45.0', revision: '5017cb8', time: '2015-02-26 23:59:50'
System info: host: 'L-SB80X4FFT4-M.local', ip: '192.168.2.34', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.9.5', java.version: '1.8.0_45'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{browserName=safari, takesScreenshot=true, javascriptEnabled=true, version=7.1.7, cssSelectorsEnabled=true, platform=MAC, secureSsl=true}]
Session ID: null
      at _stream_readable.js:944:16

two tabs:

[mdwragg]

Hi, we're getting the same thing... This bug is reeeaaalllyyy old, was there any solution to this?

Thanks!

https://travis-ci.org/PredixDev/px-view-header/builds/302764184

[ryan-roemer]

@mdwragg -- Unfortunately, never came back to this to reenable it. Kind of scary that Safari remains this flaky on SL!

[mdwragg]

Thanks for getting back to me Ryan! Yeah... worrying right? :-(

[uchuugaka]

So, SafariDriver with selenium 2.x doesn't like (and doesn't support) multiple tabs and windows really.
The code doesn't work well.
It also doesn't help that selenium 2.x launches Safari in a separate process ignoring if it is currently running, and then kills it without closing tabs and windows.
Basically, you need to ensure a clean session.
Before a session, you need this set of steps:
Launch Safari, close tabs and windows, quit Safari.
(I do this in my personal test systems via UI automation with the accessibility APIs, not Selenium. AppleScript can do it too, it wraps those in "System Events").
Then, you can launch and get connected.

If it is simply NoSuchWindowException but you have a window, you need error handling at launch that looks at the array/list of window_handles objects and then switch_to_window(window_handles[0])
(pseudo code, exats vary by language bindings.).