Page MenuHomePhabricator
Create Task
Maniphest T189567

Phaseout CI mediawiki config / extensions_load.txt to load extensions
Closed, ResolvedPublic
Actions

Description

integration/jenkins.git has a MediaWiki configuration snippet to detect and load MediaWiki extensions (mediawiki/config.d/50_mw_ext_loader.php) . The same mechanism exists in MediaWiki core itself and CI should rely on it instead.

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build. The config snippet is instructed by Zuul for a list of extensions to include via a file extensions_load.txt, the others present on the disk were not loaded. The then detects whether there is an extension.json or an expected entry point then wfLoadExtensions() or require_once.

Nowadays we start with a clean workspace so that system is no more needed.

We would need:

Note: in case an extension lacks extension.json or the expected PHP entry point, LocalSettingsGenerator::generateExtEnableLine forges a require_once with the extension name nonetheless even if the file does not exist. That will later causes an error when failing to require the file. That covers the abort / exit(1); we have in 50_mw_ext_loader.php

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

hashar created this task.Mar 13 2018, 9:35 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 13 2018, 9:35 AM
hashar updated the task description. (Show Details)Mar 13 2018, 9:35 AM
gerritbot subscribed.Mar 13 2018, 9:36 AM

Change 419130 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419130

gerritbot added a project: Patch-For-Review.Mar 13 2018, 9:36 AM
hashar claimed this task.Mar 13 2018, 9:41 AM
hashar edited projects, added Continuous-Integration-Infrastructure (shipyard), Release-Engineering-Team (Kanban); removed Continuous-Integration-Infrastructure.
hashar moved this task from Backlog to In progress on the Continuous-Integration-Infrastructure (shipyard) board.
hashar moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
hashar moved this task from Unsorted to Migrate / Replace on the Technical-Debt board.
Legoktm added a comment.Mar 13 2018, 4:05 PM

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build

Are you planning to get rid of this?

hashar added a comment.Mar 14 2018, 8:10 AM
In T189567#4047054, @Legoktm wrote:

For the context, we used to have Jenkins build reusing workspaces, hence differents set of extensions were left behind being build

Are you planning to get rid of this?

Yes definitely. That is causing too much maintenance troubles and get build to fail randomly because of the dirty state.

gerritbot added a comment.Mar 14 2018, 8:52 AM

Change 419130 merged by jenkins-bot:
[mediawiki/core@master] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419130

hashar updated the task description. (Show Details)Mar 14 2018, 8:54 AM
ReleaseTaggerBot added a project: MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)).Mar 14 2018, 9:00 AM
gerritbot added a comment.Mar 14 2018, 10:29 AM

Change 419384 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_30] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419384

gerritbot added a comment.Mar 14 2018, 10:56 AM

Change 419391 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_29] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419391

gerritbot added a comment.Mar 14 2018, 11:20 AM

Change 419393 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419393

hashar updated the task description. (Show Details)Mar 14 2018, 11:21 AM
gerritbot added a comment.Mar 14 2018, 11:21 AM

Change 419394 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@wmf/1.31.0-wmf.24] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419394

gerritbot added a comment.Mar 14 2018, 11:21 AM

Change 419396 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@wmf/1.31.0-wmf.25] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419396

gerritbot added a comment.Mar 14 2018, 4:37 PM

Change 419394 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.24] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419394

gerritbot added a comment.Mar 14 2018, 4:39 PM

Change 419396 merged by jenkins-bot:
[mediawiki/core@wmf/1.31.0-wmf.25] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419396

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)); removed MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)).Mar 14 2018, 5:00 PM
hashar updated the task description. (Show Details)Mar 14 2018, 8:48 PM
gerritbot added a comment.Mar 14 2018, 8:53 PM

Change 419526 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@fundraising/REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419526

gerritbot added a comment.Mar 14 2018, 10:14 PM

Change 419605 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/core@master] Let install.php detect and inject extensions [2]

https://gerrit.wikimedia.org/r/419605

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)); removed MW-1.31-release-notes (WMF-deploy-2018-03-06 (1.31.0-wmf.24)).Mar 15 2018, 3:00 AM
hashar added a parent task: T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker.Mar 23 2018, 10:43 AM
gerritbot added a comment.Apr 2 2018, 4:27 PM

Change 419384 merged by jenkins-bot:
[mediawiki/core@REL1_30] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419384

gerritbot added a comment.Apr 2 2018, 4:27 PM

Change 419391 merged by jenkins-bot:
[mediawiki/core@REL1_29] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419391

gerritbot added a comment.Apr 2 2018, 4:32 PM

Change 419393 merged by jenkins-bot:
[mediawiki/core@REL1_27] Let install.php detect and inject extensions

https://gerrit.wikimedia.org/r/419393

ReleaseTaggerBot added projects: MW-1.29-release-notes, MW-1.27-release-notes, MW-1.30-release-notes.Apr 2 2018, 5:00 PM
gerritbot added a comment.Apr 4 2018, 12:34 PM

Change 419605 merged by jenkins-bot:
[mediawiki/core@master] Let install.php detect and inject extensions [2]

https://gerrit.wikimedia.org/r/419605

ReleaseTaggerBot edited projects, added MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)); removed MW-1.31-release-notes (WMF-deploy-2018-03-20 (1.31.0-wmf.26)).Apr 4 2018, 1:00 PM
gerritbot added a comment.Apr 5 2018, 4:32 PM

Change 424352 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Make MediaWiki install.php detects extensions

https://gerrit.wikimedia.org/r/424352

gerritbot added a comment.Apr 5 2018, 4:53 PM

Change 424352 merged by jenkins-bot:
[integration/quibble@master] Make MediaWiki install.php detects extensions

https://gerrit.wikimedia.org/r/424352

hashar added a comment.Apr 20 2018, 8:00 AM

I havent worked on this for the last two weeks or so. I have been focusing on polishing up Quibble to run against mediawiki/core. Will come back to this next week.

hashar updated the task description. (Show Details)Apr 29 2018, 7:07 PM

mediawiki/core and mediawiki/vendor are using the Quibble based jobs. There are still some jobs using Nodepool though.

I triggered the Quibble job for extensions and skins over the week-end:

SUCCESSFAILURE
Skins3211
Extensions619277

That is encouraging.

gerritbot added a comment.Jul 11 2018, 3:19 PM

Change 419526 abandoned by Reedy:
Let install.php detect and inject extensions

Reason:
https://gerrit.wikimedia.org/r/#/c/mediawiki/core/ /444628/-1..2 brought it in :)

https://gerrit.wikimedia.org/r/419526

hashar removed hashar as the assignee of this task.Aug 1 2018, 3:02 PM
hashar moved this task from In progress to Backlog on the Continuous-Integration-Infrastructure (shipyard) board.

It is still used here and there but I lack time to audit the remaining use cases. So it is back to the backlog.

dduvall moved this task from In-progress to Backlog on the Release-Engineering-Team (Kanban) board.Oct 22 2018, 4:37 PM
Legoktm added a comment.Oct 23 2018, 5:47 PM

I think the last two places this is being used is in the legacy Wikibase jobs (T188717), and some fundraising jobs.

Legoktm added a subtask: T188717: Move Wikibase to using the normal mediawiki extension (quibble) jobs.Oct 23 2018, 5:47 PM
hashar closed subtask T188717: Move Wikibase to using the normal mediawiki extension (quibble) jobs as Resolved.Nov 15 2018, 7:24 PM
hashar moved this task from Backlog to Ready on the Continuous-Integration-Infrastructure (shipyard) board.Dec 31 2018, 3:11 PM
hashar edited projects, added Continuous-Integration-Config; removed Continuous-Integration-Infrastructure (shipyard).Dec 31 2018, 3:19 PM
Krinkle removed projects: MW-1.31-release-notes (WMF-deploy-2018-04-10 (1.31.0-wmf.29)), Patch-For-Review.Jan 1 2019, 12:12 AM
hashar added a parent task: T220199: Quibble: deprecate and phase out EXT_DEPENDENCIES / SKIN_DEPENDENCIES.Apr 12 2019, 12:34 PM
hashar added a comment.Apr 12 2019, 12:37 PM

I am back on this and the usage has been extended to the phan jobs unfortunately.

dockerfiles/ci-src-setup/setup-mwext.sh:echo -e "${EXT_DEPENDENCIES:-}" >> /tmp/extensions_load.txt

# Use a docker image to clone for a MediaWiki extension or skin
- builder:
    name: docker-ci-src-setup-mwext
        image: 'docker-registry.wikimedia.org/releng/ci-src-setup:0.2.9'
        args: '/srv/setup-mwext.sh'

And that macro docker-ci-src-setup-mwext is used by:

  • mwext-php70-phan-docker
  • mwext-php70-phan-seccheck-docker
  • mwext-php70-phan-seccheck-docker-non-voting
  • mwskin-php70-phan-docker
  • mwskin-php70-phan-seccheck-docker
  • mwskin-php70-phan-seccheck-docker-non-voting
hashar added a comment.Apr 12 2019, 12:55 PM

We should be able to use Quibble to clone the repositories / vendor.git and run composer:

quibble --packages-source=vendor --db=sqlite --git-cache=/home/hashar/projects --skip=all -- mediawiki/extensions/3D

But that doesn't run composer install --dev in the extension/skin repository. Then the mediawiki-phan/run.sh entry point tries to look for extensions/XXX/vendor/mediawiki/mediawiki-phan-config/composer.json to get the phan version that should be used.

I guess we can adjust that entry point to install mediawiki/mediawiki-phan-config first.

hashar added a comment.Apr 12 2019, 1:20 PM

Taking mediawiki/extensions/3D as an example:

First prepare the repository using Quibble:

ZUUL_PROJECT=mediawiki/extensions/3D \
  quibble --packages-source=vendor --db=sqlite \
  --git-cache=/home/hashar/projects \
  --skip=all
  -c 'cd extensions/3D && composer update'

Then run the magic container:

docker run \
  -e THING_SUBNAME=extensions/3D \
  --volume=/home/hashar/src:/mediawiki \
  --rm -it \
  docker-registry.wikimedia.org/releng/mediawiki-phan:0.1.11 \
  /mediawiki/extensions/3D -m checkstyle

Result:

+ umask 002
+ cd /mediawiki/extensions/3D
+ '[' '!' -f /mediawiki/extensions/3D/.phan/config.php ']'
+ jq -e '.extra."phan"' composer.json
null
+ CFG_COMPOSER=vendor/mediawiki/mediawiki-phan-config/composer.json
+ jq -e '.extra."phan"' vendor/mediawiki/mediawiki-phan-config/composer.json
"1.2.6"
++ jq -r '.extra."phan"' vendor/mediawiki/mediawiki-phan-config/composer.json
+ PHAN_VERSION=1.2.6
+ '[' -f .phan/config.php ']'
+ export PHP_ARGS=-dextension=ast_101.so
+ PHP_ARGS=-dextension=ast_101.so
+ install_phan
+ cd /srv/phan
+ composer require phan/phan:1.2.6
./composer.json has been created
...
  - Installing phan/phan (1.2.6): Downloading (100%)         
...
+ exec /srv/phan/vendor/bin/phan -d . -p
RecursiveDirectoryIterator::__construct(includes/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(maintenance/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(tests/phan/stubs/): failed to open dir: No such file or directory
RecursiveDirectoryIterator::__construct(../../tests/phan/stubs): failed to open dir: No such file or directory
<< progress bar showing up >>
$ echo $?
0
$

Seems easier than expected :]

Note that Quibble still runs MediaWiki install.php as well as npm install for mediawiki/core. That is because --command is intended to have a fully working install.

gerritbot added a comment.Apr 12 2019, 2:07 PM

Change 503362 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] phan: use Quibble to prepare MediaWiki sources

https://gerrit.wikimedia.org/r/503362

gerritbot added a project: Patch-For-Review.Apr 12 2019, 2:08 PM
gerritbot added a comment.Apr 12 2019, 2:20 PM

Change 503364 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: remove setup-mwext.sh entry point

https://gerrit.wikimedia.org/r/503364

hashar claimed this task.May 3 2019, 12:29 PM
gerritbot added a comment.May 3 2019, 3:23 PM

Change 503362 merged by jenkins-bot:
[integration/config@master] phan: use Quibble to prepare MediaWiki sources

https://gerrit.wikimedia.org/r/503362

gerritbot added a comment.May 3 2019, 3:33 PM

Change 503364 merged by jenkins-bot:
[integration/config@master] docker: remove setup-mwext.sh entry point

https://gerrit.wikimedia.org/r/503364

hashar closed this task as Resolved.May 6 2019, 8:40 AM

The last use case of EXT_DEPENDENCIES was for Phan which I have removed.

For Quibble, that will be done as part of T220199

gerritbot added a comment.May 6 2019, 1:44 PM

Change 508320 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] phan: use composer and mysql

https://gerrit.wikimedia.org/r/508320

gerritbot added a comment.May 6 2019, 2:31 PM

Change 508320 merged by jenkins-bot:
[integration/config@master] phan: use composer and mysql

https://gerrit.wikimedia.org/r/508320

Urbanecm_WMF mentioned this in T373114: Enable CommunityConfiguration Example in CI.Aug 23 2024, 5:46 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits