I've checked crawler stats and they look fine as far as I can tell. I also tried fetching the site as robot and it succeeded:

I think we'll just monitor it for a bit first while watching analytics / crawler stats to understand the impact. It's unclear to me what could be responsible for this.

The only somewhat relevant changes we made recently were: #21665 and #21689.

There there are #21712 #21763 but those didn't make it to prod yet.

I also got a notification that we now have new search console available for our site, but that seems unrelated.

The index status is looking fine:

but we should note that the data processing for this graph appears to have latency of several days, so the graph ends with status as of Tuesday (3 days ago).

EDIT: the data in this comment is wrong - I was looking at angularjs.org - I think the data is incomplete yet and that's why we see less traffic on that site. Ignore this comment. I'm leaving it here for archival purposes.

Sessions are 4-20% down for today (ignore the sharp drop - we don't have complete data for today yet)

sessions were already down by 4% down at 6am:

But the number of users is down sharply - 20-60% for today:

Started around 7pm PT yesterday:

It looks like these pages have a Googlebot noindex meta tag on them -- this prevents Google from indexing the page. Using JavaScript to remove limitations like these is not recommended, and can result in those pages being dropped from indexing before they're rendered.

Thanks @softplus. In that case the culprit PR is #21665. I find it however very odd that the search console didn't warn us about this.

So we should remove the pre-emptive hard coded tag?

@petebacondarwin yeah. I have a PR: #21816 out for review

The fix in #21816 is now in production.

@IgorMinar - Is this now resolved?

looks like we recovered.

and interestingly, I just got an alert from google that we were doing it wrong. I wish it didn't take 9 days for these alerts to arrive:

note that the date when the crawling errors occurred are around the same time as we noticed the the decreased traffic from google.com results.

I'm going to close this issue now. Just as a related note we are experimenting with adding noindex dynamically as a preemptive measure against crawling a screen that hasn't been completely rendered. more info in #21941 and #21992

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}