Page MenuHomePhabricator
Create Task
Maniphest T179190

Run Selenium Cucumber tests in CI
Closed, ResolvedPublic
Actions

Description

Status

Blocked on T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.

Description

Needed for T190710: Minerva Ruby and Node.js browser tests running side by side. The initial idea was to run both core and extension/skin tests from core, using either mocha or cucumber (or possibly both). I do not think that is a good idea any more. Looks like the way forward is to run tests for each repository (core, extension, skin) from the repository.

TODO

🚀 T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests

Merged

🏝 428040 Run npm run selenium-test instead of grunt webdriver:test
🏝 424567 Selenium: use default value for waitforTimeout
🏝 424592 Run npm run selenium instead of grunt webdriver:test
🏝 424763 selenium: Remove Jenkins configuration file
🏝 427362 Selenium: Jenkins configuration file no longer needed

🏝 424764 Selenium: move logic for running tests from package.json to selenium.sh
🏝 426881 Selenium: move logic for running tests from package.json to selenium.sh
🏝 426883 Selenium: move logic for running tests from package.json to selenium.sh
🏝 426884 Selenium: move logic for running tests from package.json to selenium.sh

🏝 429018 selenium: Abstract test command behind 'npm run'
🏝 428344 selenium: Abstract test command behind 'npm run'
🏝 428373 selenium: Abstract test command behind 'npm run'
🏝 428374 selenium: Abstract test command behind 'npm run'
🏝 428375 selenium: Abstract test command behind 'npm run'

Abandoned

⚰️ 391546 WIP Run Cucumber+Selenium+Node.js in CI
⚰️ 391547 WIP Run Cucumber+Selenium+Node.js in CI
⚰️ 391566 WIP Run Cucumber+Selenium+Node.js in CI
⚰️ 424609 WIP Selenium: run both Mocha and Cucumber tests

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
zeljkofilipin raised the priority of this task from Low to High.Apr 6 2018, 10:59 AM
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
zeljkofilipin added projects: JavaScript, MediaWiki-Core-Tests.
zeljkofilipin moved this task from Inbox to Selenium on the MediaWiki-Core-Tests board.
zeljkofilipin updated the task description. (Show Details)Apr 6 2018, 11:02 AM
gerritbot added a comment.Apr 6 2018, 11:08 AM

Change 424567 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: use default value for waitforTimeout

https://gerrit.wikimedia.org/r/424567

gerritbot added a project: Patch-For-Review.Apr 6 2018, 11:08 AM
zeljkofilipin updated the task description. (Show Details)Apr 6 2018, 12:12 PM
zeljkofilipin updated the task description. (Show Details)Apr 6 2018, 1:22 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.Apr 6 2018, 1:54 PM

Change 424592 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] WIP Run Cucumber+Selenium+Node.js in CI

https://gerrit.wikimedia.org/r/424592

gerritbot added a comment.Apr 6 2018, 2:45 PM

Change 424609 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Selenium: run Cucumber tests

https://gerrit.wikimedia.org/r/424609

zeljkofilipin renamed this task from Run Cucumber+Selenium+Node.js in CI to Run Selenium Cucumber tests in CI.Apr 6 2018, 2:46 PM
zeljkofilipin mentioned this in T190710: Minerva Ruby and Node.js browser tests running side by side.
gerritbot added a comment.Apr 7 2018, 2:36 PM

Change 424764 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] Selenium: cun npm run selenium instead of grunt webdriver:test in CI

https://gerrit.wikimedia.org/r/424764

gerritbot added a comment.Apr 7 2018, 2:49 PM

Change 424763 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@master] WIP Selenium: remove Jenkins configuration file

https://gerrit.wikimedia.org/r/424763

gerritbot added a comment.Apr 7 2018, 4:03 PM

Change 424567 merged by jenkins-bot:
[mediawiki/core@master] Selenium: use default value for waitforTimeout

https://gerrit.wikimedia.org/r/424567

ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)).Apr 7 2018, 5:00 PM
zeljkofilipin mentioned this in T191982: Blog posts about new Selenium framework features.Apr 11 2018, 2:53 PM
zeljkofilipin added a comment.Apr 11 2018, 5:50 PM

Tests are now running from core! But, something is wrong:

TypeError: feature.getFailureException is not a function
at SauceService.afterStep (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sauce-service/build/sauce-service.js:87:25)
at execHook (/Users/z/Documents/gerrit/mediawiki/vagrant/mediawiki/node_modules/wdio-sync/build/index.js:166:35)
zeljkofilipin added a comment.Apr 11 2018, 5:54 PM

424609/11 removes sauce labs configuration and removes error messages! :D

gerritbot added a comment.Apr 14 2018, 12:11 AM

Change 424764 merged by Krinkle:
[mediawiki/core@master] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/424764

gerritbot added a comment.Apr 14 2018, 12:11 AM

Change 424763 merged by Krinkle:
[mediawiki/core@master] selenium: Remove Jenkins configuration file

https://gerrit.wikimedia.org/r/424763

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)); removed MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)).Apr 14 2018, 1:01 AM
gerritbot added a comment.Apr 16 2018, 9:49 AM

Change 426881 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@REL1_30] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426881

gerritbot added a comment.Apr 16 2018, 9:49 AM

Change 426883 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@REL1_29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426883

gerritbot added a comment.Apr 16 2018, 9:52 AM

Change 426884 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[mediawiki/core@wmf/1.31.0-wmf.29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426884

gerritbot added a comment.Apr 18 2018, 11:46 AM

Change 427362 had a related patch set uploaded (by Zfilipin; owner: Zfilipin):
[integration/config@master] Selenium: Jenkins configuration file no longer needed

https://gerrit.wikimedia.org/r/427362

gerritbot added a comment.Apr 20 2018, 12:42 AM

Change 426881 merged by jenkins-bot:
[mediawiki/core@REL1_30] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426881

gerritbot added a comment.Apr 20 2018, 12:43 AM

Change 426883 merged by jenkins-bot:
[mediawiki/core@REL1_29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426883

ReleaseTaggerBot added projects: MW-1.30-release-notes, MW-1.29-release-notes.Apr 20 2018, 1:00 AM
gerritbot added a comment.Apr 21 2018, 1:26 AM

Change 427362 merged by jenkins-bot:
[integration/config@master] Selenium: Jenkins configuration file no longer needed

https://gerrit.wikimedia.org/r/427362

gerritbot added a comment.Apr 21 2018, 1:28 AM

Change 428040 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/quibble@master] Run npm run selenium instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/428040

gerritbot added a comment.Apr 21 2018, 2:14 AM

Change 424592 merged by Krinkle:
[integration/config@master] Run npm run selenium instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/424592

gerritbot added a comment.Apr 21 2018, 7:32 AM

Change 426884 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.29] Selenium: move logic for running tests from package.json to selenium.sh

https://gerrit.wikimedia.org/r/426884

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)); removed MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)).Apr 21 2018, 8:00 AM
gerritbot added a comment.Apr 23 2018, 2:58 PM

Change 428344 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@master] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428344

gerritbot added a comment.Apr 23 2018, 4:41 PM

Change 428373 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_31] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428373

gerritbot added a comment.Apr 23 2018, 4:41 PM

Change 428374 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428374

gerritbot added a comment.Apr 23 2018, 4:41 PM

Change 428375 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[mediawiki/core@REL1_29] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428375

gerritbot added a comment.Apr 23 2018, 4:53 PM

Change 428344 merged by jenkins-bot:
[mediawiki/core@master] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428344

gerritbot added a comment.Apr 23 2018, 4:57 PM

Change 428375 merged by jenkins-bot:
[mediawiki/core@REL1_29] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428375

gerritbot added a comment.Apr 23 2018, 4:59 PM

Change 428374 merged by jenkins-bot:
[mediawiki/core@REL1_30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428374

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-04-24 (1.32.0-wmf.1)).Apr 23 2018, 5:00 PM
gerritbot added a comment.Apr 23 2018, 5:08 PM

Change 428373 merged by jenkins-bot:
[mediawiki/core@REL1_31] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/428373

ReleaseTaggerBot edited projects, added MW-1.31-release-notes; removed MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)).Apr 23 2018, 6:00 PM
gerritbot added a comment.Apr 25 2018, 8:36 PM

Change 429018 had a related patch set uploaded (by Hashar; owner: Krinkle):
[mediawiki/core@wmf/1.31.0-wmf.30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/429018

gerritbot added a comment.Apr 25 2018, 8:39 PM

Change 428040 merged by jenkins-bot:
[integration/quibble@master] Run npm run selenium-test instead of grunt webdriver:test

https://gerrit.wikimedia.org/r/428040

gerritbot added a comment.Apr 25 2018, 8:52 PM

Change 429018 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.30] selenium: Abstract test command behind 'npm run'

https://gerrit.wikimedia.org/r/429018

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-04-17 (1.31.0-wmf.30)); removed MW-1.31-release-notes.Apr 25 2018, 9:01 PM
Krinkle subscribed.May 2 2018, 2:35 AM
mediawiki/core:/package.json
+     "chai": "4.1.2",

@zeljkofilipin In the interest of not requiring as intense maintenance and up-keep for this new stack, I would recommend focussing on making the selenium test run from the repository rather than from core.

The above diff in particular stood out, because it reminded me of when we first introduced JSLint in Wikimedia's CI (back in 2012 I think?). We globally installed a version of JSHint and started running it on different repositories. All was well. It supported repo-local override settings as well. Then, it came time to upgrade to a newer version of JSHint and it proved extremely disruptive because the version was in one place only. The only right way to upgrade that "one" version would've been to somehow atomically update all repositories to conform to new rules. This is a problem because we don't maintain all repositories ourselves. And even the repositories we do maintain, you don't want to force a single point in time for one person (nor everyone together) to have to make an upgrade work.

We solved this by instead using the npm test entry point where repositories decide individually which version of what tool to use.

I suspect the idea to run from core may've been inspired by how PHPUnit tests are run, but I think the Selenium test stack is in a unique position to do it better.

Especially because you've already designed and implemented in a way that is (almost) independently runnable. The framework comes from npm. The configuration is stateless from CLI or environment variables. The only thing that (most) Selenium tests need from core is the page objects, which per T193088#4165721 could easily be published and pulled in locally.

I haven't spoken up before now (aside from T193088) because it was just a handful of files, which are already pulled in automatically by Jenkins, and are actually versions correctly for older branches thanks to Zuul. It seemed pretty minimal and easy to fix later. But now when adding Chai and Cucumber and other abstractions to it, it starts to become harder to fix later on. Let me know if you're interested in trying to get this working over the next week, so that the Cucumber addition can make use of it from the start.

Krinkle mentioned this in T193943: Selenium test job should install local dependencies before starting tests.May 5 2018, 5:46 PM
zeljkofilipin added a comment.May 10 2018, 3:34 PM
In T179190#4173431, @Krinkle wrote:

Let me know if you're interested in trying to get this working over the next week, so that the Cucumber addition can make use of it from the start.

My plan for the Barcelona hackathon is to spend about 80% of the time pairing with people (T190687: Pair on writing Selenium tests in JavaScript/Node.js). Would you like to pair on this?

In the meantime, I will try to review (and merge) all existing commits.

zeljkofilipin moved this task from Deep work 🌊 to Q2 👔 on the User-zeljkofilipin board.May 18 2018, 1:09 PM
zeljkofilipin edited parent tasks, added: T198389: Q1 Selenium framework improvements; removed: T164721: Run Selenium tests in CI per-patch for extensions.Jun 28 2018, 3:30 PM
zeljkofilipin mentioned this in T198389: Q1 Selenium framework improvements.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Jun 28 2018, 5:06 PM
zeljkofilipin moved this task from In-progress to Backlog on the Release-Engineering-Team (Kanban) board.
zeljkofilipin added a subscriber: hashar.Jul 6 2018, 4:20 PM
In T179190#4173431, @Krinkle wrote:

@zeljkofilipin In the interest of not requiring as intense maintenance and up-keep for this new stack, I would recommend focussing on making the selenium test run from the repository rather than from core.

Agreed. I have talked with @hashar about it, and I think he agrees too. 😅

zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:22 PM
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:26 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:30 PM
zeljkofilipin updated the task description. (Show Details)Jul 6 2018, 4:36 PM
zeljkofilipin updated the task description. (Show Details)
gerritbot added a comment.Jul 6 2018, 4:38 PM

Change 424609 abandoned by Zfilipin:
WIP Selenium: run both Mocha and Cucumber tests

Reason:
Tests for a repo will run from the repo, not from core.

https://gerrit.wikimedia.org/r/424609

zeljkofilipin claimed this task.Jul 6 2018, 4:39 PM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin removed a project: Patch-For-Review.
zeljkofilipin moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
zeljkofilipin moved this task from Q2 👔 to Deep work 🌊 on the User-zeljkofilipin board.Jul 6 2018, 4:42 PM
zeljkofilipin added a parent task: T190710: Minerva Ruby and Node.js browser tests running side by side.
zeljkofilipin mentioned this in T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.Jul 9 2018, 2:22 PM
zeljkofilipin changed the task status from Open to Stalled.Jul 17 2018, 8:57 AM
zeljkofilipin updated the task description. (Show Details)

Blocked on T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.

zeljkofilipin removed zeljkofilipin as the assignee of this task.Jul 17 2018, 9:04 AM
zeljkofilipin updated the task description. (Show Details)
zeljkofilipin moved this task from Deep work 🌊 to Backlog 🪒 on the User-zeljkofilipin board.
zeljkofilipin moved this task from In-progress to Backlog on the Release-Engineering-Team (Kanban) board.
zeljkofilipin added a subtask: T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests.Jul 17 2018, 9:09 AM
zeljkofilipin changed the status of subtask T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests from Open to Stalled.Sep 19 2018, 10:13 AM
zeljkofilipin awarded a token.Sep 25 2018, 10:34 AM
zeljkofilipin mentioned this in T206624: Q2 Selenium framework improvements.Oct 10 2018, 12:22 PM
Jdlrobson changed the status of subtask T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests from Stalled to Open.Apr 5 2019, 4:49 PM
greg added a project: Release-Engineering-Team-TODO.Jul 1 2019, 9:28 PM
greg moved this task from Should be empty (use Release-Engineering-Team) to Soon-ish on the Release-Engineering-Team-TODO board.Jul 1 2019, 9:30 PM
greg removed a project: Release-Engineering-Team (Kanban).Jul 1 2019, 9:31 PM
greg added a project: Release-Engineering-Team (Unit & Int & System Tooling).Jul 6 2019, 5:52 AM
hashar closed subtask T199116: Quibble should run `npm install` and `npm run selenium-test` for each extension/skin that has Selenium tests as Resolved.Jul 25 2019, 11:20 PM
hashar closed this task as Resolved.Jul 25 2019, 11:24 PM

Quibble now detects whether a repository has a selenium-test npm script. If so, it runs npm install && npm run-script selenium-test with ChromeDriver enabled.

wdio-cucumber-framework seems to work on MobileFrontend

CirrusSearch needs some bot to be upgraded to NodeJS10 T229033

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits