Could someone point me to how to make the probe not appear in access.log for perl5.36 webservice?

I see, thank you for the explanation. So if the /healthz endpoint is set correctly and probe is pointed to that endpoint using the parameter, this task can be considered solved, right? As the periodical health check happens inside the webservice, there should be no need for any health check script or bash running periodically on grid cronjob, correct?

If not passed, a simple TCP check will be used instead.

Sorry for the confusion, of course I've had a typo in my code. That's why the /healthz endpoint was working just for several seconds after restart.

Also, shouldn't /healthz endpoint be generated automatically by lighttpd, same as mod_status? Or maybe is it a good strategy to use mod_status's endpoint also for probe? Also can I use just root for probe as the whole webapp is just in root?

Not sure what exactly you want me to do. If I start webservice with the probe, the /healthz endpoint is created somehow automatically (why?) - there is a blank page giving code 200. Just the root starts to give 503 somehow, which is super weird.

No, I just deactivated the probe as it broke the root of kmlexport webapp (I had to revert back to webservice --backend kubernetes -m 1536Mi perl5.36 start so the users can use it)

Oh, I see. With the flag, https://kmlexport.toolforge.org/ gives 503. Without the flag, it gives 200, so the flag breaks root :/

@dcaro Seems like it is not deployed yet? toolforge webservice: error: unrecognized arguments: --health-check-url /healthz

I there a peak memory usage stat somewhere for k8s toolforge-jobs?

I forgot about this one, but the new one I created is much more broadly described and is linked to more related tasks

Therefore marking this as declined as this is not an issue with kmlexport and has to be fixed elsewhere (frwiki? kartographer? maplink?)

I investigated further and it seems fr's Module:Coordinates cann't correctly assign parameter name to geohack url parameter title. Not sure if this is module's fault or kartographer's/maplink's fault though.

Hi, as mentioned before, KMLexport is just a webscraper. It scans wiki page for geohack links/urls and parses parameters from these urls. One of the geohack url parameters is "title", which is directly used as a name. Therefore if coord/maplink adds this parameter to the geohack url, then KMLexport is using that automatically.

Simplified a bit (not using venv branch, rather creating venv from scratch every time) and deployed

BTW solved

No need for that, per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Jobs I can just create venv prior and the recreate it every time I need it using a toolforge job

It seems this is the issue with Firefox blocking tracking cookies by default

I'll check how it currently behaves at Jan 1st (scheduled task) and close this if working. I'll check if it takes 2-3days as it did in 2021 and before and we'll see

(BTW it always took 2-3 days, last time in January 2022. In May 2022 it took 8 days. Now it takes 6 days and still is woking on categories starting with B)

The patch seems to help a little bit, it doesn't fail after a minute, but after few hours. And if I pass all exceptions, it is now extremely slow. It takes 6 days for one letter of the alphabet of cswiki categories, previously the whole cswiki was scanned in 2-3 days

It seems this commit broke the behavior: rPWBC76cbce788c6db42b5285140d100245c8bdc3cd87, before it it still works as expected

Why does Category.members() calls Category.subcategories()? This seems to be completely wrong as subcategories() calls members() internally:

Reverting subcategories method to its state from May 2022 fixes the issue:

    def subcategories(self,
	                      recurse: Union[int, bool] = False,
	                      total: Optional[int] = None,
	                      content: bool = False):
	        """
	        Iterate all subcategories of the current category.
	
	        :param recurse: if not False or 0, also iterate subcategories of
	            subcategories. If an int, limit recursion to this number of
	            levels. (Example: recurse=1 will iterate direct subcats and
	            first-level sub-sub-cats, but no deeper.)
	        :param total: iterate no more than this number of
	            subcategories in total (at all levels)
	        :param content: if True, retrieve the content of the current version
	            of each category description page (default False)
	        """
	        if not isinstance(recurse, bool) and recurse:
	            recurse = recurse - 1
	        if not hasattr(self, '_subcats'):
	            self._subcats = []
	            for member in self.site.categorymembers(
	                    self, member_type='subcat', total=total, content=content):
	                subcat = Category(member)
	                self._subcats.append(subcat)
	                yield subcat
	                if total is not None:
	                    total -= 1
	                    if total == 0:
	                        return
	                if recurse:
	                    for item in subcat.subcategories(
	                            recurse, total=total, content=content):
	                        yield item
	                        if total is not None:
	                            total -= 1
	                            if total == 0:
	                                return
	        else:
	            for subcat in self._subcats:
	                yield subcat
	                if total is not None:
	                    total -= 1
	                    if total == 0:
	                        return
	                if recurse:
	                    for item in subcat.subcategories(
	                            recurse, total=total, content=content):
	                        yield item
	                        if total is not None:
	                            total -= 1
	                            if total == 0:
	                                return

Also there is python3-requests missing in Python pods (I know I can avoid that using venv, but this is https requests library, which is usually preferred to be installed locally installed due to security reasons). Would have to relocate my bot's environment to venv

Also there is gzip missing (Pywikibot needs that as well)

It is a blocker for me as I use ssh to git push on one of my projects

I see, should be mentioned at the image page and perhaps the redirect from commons should be changed too

Last time in May 7th. Result:

Last time in May 7th. Result:

If I remember correctly, last time I tried there was git not working properly in K8s, which is a blocker for DvorapaBot

Thanks @Multichill. Anyway, I check periodically every 3-6 months if the transition is possible, but there are still some issues with K8s and it waits for them to be resolved.

It waits for some issues with K8s to be resolved. I check periodically every 3-6 months if the transition is possible.

It looks like rPWBC935f33205b730677a2be46200e021868995cb98a introduced this issue

Per wrapper.py docs, it should still be possible:
https://phabricator.wikimedia.org/diffusion/PWBC/browse/master/pywikibot/scripts/wrapper.py$9

Hi, just a reminder to update the schema in https://commons.wikimedia.org/w/index.php?title=File:MediaWiki_database_schema_latest.svg&redirect=no when the work is finished (or perhaps also after each of the normalizations?)

I think you've summarized my thoughts pretty well. From my point of view, there came nothing good from mentoring novices, who never saw Pywikibot before and never used it on a wiki. Just losing time with those basically, as no one stayed with Pywikibot long-term and for me personally it was just too much energy for almost nothing. I had to teach them basics - how to use Pywikibot's scripts, Gerrit, Phabricator, Git, command line, wiki, Python, ... and what for?
I remember more interest at the 2019 Prague Wikimedia Hackathon came from people already using it / planning to use it on their task/project/wiki.
I am now starting to think that this might be better audience to reach as they are already familiar / getting familiar with Pywikibot and also might be more willing to stay contributing to Pywikibot, once they first try. Me and @matej_suchanek are good examples of such Pywikibot users, who started to contribute to the project as well. Such people might even need less mentoring as they dig in Pywikibot docs by themselves. So I would propose targetting such people / communities.

So my options are https or OAuth. Neither convenient :/

Oh, I found the issue, you have already solved it in the new commit :)

@Xqt no problem. Pretty weird, I think there might have been an issue within command line parameter parsing code section. Have you found the cause already?

It seems that the commit broke the basic functionality of the replace.py script.

Reverted in https://gerrit.wikimedia.org/r/c/pywikibot/core/+/760553 (see T301185)

https://gerrit.wikimedia.org/r/c/pywikibot/core/+/760553

Identified rPWBC15b2db3906276030b8b475c66c3d2ad4ce8eddc0 as a source of trouble. Reverting for now

We can either ignore dependency check in Toolforge for pymysql, or wait until Debian devs add 0.7.11 to Stretch (never gonna happen), or wait until Toolforge updates to Buster (hopefully soon), or go back and revert the change, or release 7.0.0 after Toolforge updates to Buster. No solution is ideal but every solution is better than the current state.

In T216741#7531673, @Xqt wrote:

Solved upstream already.

In T284939#7158375, @Dvorapa wrote:

As I said earlier, we can do a workaround and "install" setuptools into a folder inside pywikibot folder, like we do with mwparserfromhell. I'll test my suggestion and if it will work, we don't have to revert anything.

In T284299#7156384, @Dvorapa wrote:

As a workaround we could temporarily install newer setuptools into /shared/pywikibot/setuptools, like we do with mwparserfromhell. Not sure if it will work this way though.

As I said earlier, we can do a workaround and "install" setuptools into a folder inside pywikibot folder, like we do with mwparserfromhell. I'll test my suggestion and if it will work, we don't have to revert anything.

As a workaround we could temporarily install newer setuptools into /shared/pywikibot/setuptools, like we do with mwparserfromhell. Not sure if it will work this way though.