[01:52:57] Hey, I'm working on Commons, and it seems like there's something wrong with caching. Some newly created files/categories aren't appearing in the categories I've placed them in. I've tried purging and making blank edits, but no dice, even fifteen minutes later (started around 01:35). [01:53:02] example: this image https://commons.wikimedia.org/wiki/File:Sam_Esmail_at_SXSW_2016.jpg won't show up in this category: https://commons.wikimedia.org/wiki/Category:Sam_Esmail [02:58:27] hey, i was in here an hour or so ago reporting an issue with caching on Commons. it's still going on. [02:59:11] category pages aren't updating, even if i purge or make an empty edit. this has been going on since ~01:35 (it's now 02:59) [03:00:37] example: I uploaded this image (https://commons.wikimedia.org/wiki/File:Sam_Esmail_at_SXSW_2016.jpg) at 01:35 and placed it in Category:Sam_Esmail. it still won't actually appear in the category page. [03:08:19] are there any admins or techs here who can look into this? [04:43:22] https://phabricator.wikimedia.org/T135801 [04:43:44] categorylinks rows missing. :-( [04:46:49] the jobs backlog on commons is really overfilled -- currently at 150k, and rising fast: https://commons.wikimedia.org/w/api.php?action=query&meta=siteinfo&siprop=statistics [04:47:12] category updates have been at a standstill since ~01:35 as a result. [04:50:30] I doubt mine issue is a job queue issue. [04:50:51] https://grafana.wikimedia.org/dashboard/db/job-queue-health [04:51:41] Mine issue, heh. [04:55:21] huh, well something's up with the job queue -- categorization is completely broken https://commons.wikimedia.org/wiki/Commons:Village_pump#The_October_2015_categorization_lag_is_back [04:57:03] Is there a Phabricator task about it? [05:32:11] don't think so, but Ori Livneh just said they'd look into it [05:33:42] yes, something is up. for example, the rate at which thumbnail rendering jobs are executed dropped to zero at shortly after 01:15 UTC: https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1463722410.177&from=-12hours&target=MediaWiki.jobqueue.acks.ThumbnailRender.rate [05:40:25] huh. that sounds like what the category pages have done. they haven't slowed down, they've just outright stopped as far as i can tell. [05:42:29] that is, the very first category I saw fail to update is still not updating (Category:Sam_Esmail still doesn't show File:Sam_Esmail_at_SXSW_2016.jpg). if it were just lag, you'd think that one would have gotten through by now [05:44:14] yes, agreed [05:51:48] maintenance problem on all wikis? [05:52:30] ankry: yes, I'll update the channel status [05:54:17] aaaand our amsterdam datacenter is having an outage [06:03:54] ori: wikis seem to be available from Europe already [06:04:26] yeah, TechOps just depooled it [06:04:33] meaning traffic is getting routed to other datacenters [06:08:40] <_joe_> ankry: normally an european/african user would be served from the datacenter in amsterdam; while at the moment everyone is being served either out of dallas or aushburn, virginia [06:09:03] <_joe_> while we understand what is wrong with the link between the datacenters [06:11:31] <_joe_> IagoQnsi: still seeing problems with thumbnail rendering? [06:12:14] it looks like pages are starting to update [06:12:51] <_joe_> that is what I see from the metrics too [06:13:02] awesome :) [06:41:40] hehe, it's still struggling a bit -- i renamed a category and moved all the photos to the new name, and the old category briefly showed as negative ten files: https://imgur.com/xrgDtCn [07:08:24] hmm [07:08:39] what happened to https://vote.wikimedia.org/wiki/ ? I get arabic mainpage [07:34:20] revi: it was changed for the poll that is currently being run iirc [07:35:37] revi: https://phabricator.wikimedia.org/T132667 [18:45:05] hi guys [18:45:10] what happend on wmf labs? [18:45:10] https://tools.wmflabs.org/video2commons-test/ [18:45:14] 503 error [19:08:08] The_Photographer: not for me or it's gone now [19:08:25] see a login for commons [19:09:29] if it happens again, report directly on -labs too if you can