Use playwright for desktop smoke tests

[joaomoreno]

We are using playwright for our browser tests (#128061) and should explore to use this library also for our Electron based smoke tests.

Update (Nov 2021, @bpasero)
We now have a branch up and running with playwright for Electron smoke tests: #136367

However, there are blocking issues:

• we see extension host connection hanging making the overall execution flaky ([1])
• we should explore to reduce or drop window.driver and rely purely on playwright APIs for the driver
• we need to investigate how others are using our current Electron based driver and whether they could adopt playwright too

[1]
When the extension host connection hangs, messages that are passed between window process and extension host process (node.js socket / named pipe) are not fully transmitted. Since messages are not fully delivered, our protocol that is implemented on top does not work. Either the extension host is not receiving the initialization data from the window or the extension host is unable to acknowledge that it is ready.
We believe this may be related to how playwright enables remote-debugging-port which is something we never have enabled otherwise. Possibly this results in unknown runtime behaviour when the browser window has node.js enabled. We should revisit this when we have sandbox enabled, though it might still not be an option given the shared process will remain node.js enabled for the time being.

[bpasero]

Simple code to bring up Code:

const { _electron } = require('playwright');

const electronApp = await _electron.launch({  executablePath: '/Users/bpasero/Desktop/Visual Studio Code - Insiders.app/Contents/MacOS/Electron' });

The executablePath property allows us to use an existing Electron, as opposed to Playwright picking one.

[joaomoreno]

Yeah, that the option I mentioned a couple months ago when we discussed this. I’m glad to know it just works, fantastic! Shouldn’t be too hard to get the existing Playwright driver to run our smoketests. Just some glue code, hopefully.

[bpasero]

Did a quick exploration in a hello world sample and was not able to get screenshots from the tracing, filed microsoft/playwright#13180

[bpasero]

Nevermind, I might have validated wrong, according to Pavel they should work. I think we need to revisit this when we pick up the work again to see.

[christian-bromann]

@bpasero I guess I am a bit too late for this but just so you and the team is aware we @stateful have invested time to build an integration into WebdriverIO that allows to run browser and Electron tests on Windows, Mac and Ubuntu. As part of this integration we will maintain a set of page objects for VSCode that avoids breaking builds due to changing locators in the app.

[bpasero]

Thats cool, but I am not sure exactly what the ask is? We will stay with playwright for our testing needs for the foreseeable future, probably even expand it to Unit tests too.

[christian-bromann]

No ask here. Just wanted to let you know that there is a web standard based alternative 😉

[bpasero]

Cool, but wouldn't it make sense to let playwright team know?