[00:53:12] PROBLEM - Postgres Replication Lag on maps2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 28607984 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:53:40] PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 44611768 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:55:02] RECOVERY - Postgres Replication Lag on maps2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 1720 and 56 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [00:55:30] RECOVERY - Postgres Replication Lag on maps2002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 613176 and 83 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [01:14:28] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [01:16:16] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [01:25:55] 10Operations, 10MediaWiki-extensions-Score, 10Security-Team, 10Wikimedia-General-or-Unknown, and 2 others: Extension:Score / Lilypond is disabled on all wikis - https://phabricator.wikimedia.org/T257066 (10Krinkle) [01:37:16] PROBLEM - Varnish traffic drop between 30min ago and now at ulsfo on icinga1001 is CRITICAL: 31.61 le 60 https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [01:39:08] RECOVERY - Varnish traffic drop between 30min ago and now at ulsfo on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Varnish%23Diagnosing_Varnish_alerts https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1 [02:28:10] !log reedy@deploy1001 Synchronized php-1.35.0-wmf.39/extensions/Score/includes/Score.php: Short circuit lilypond version check to allow usage of cached files T257066 (duration: 00m 55s) [02:28:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:28:15] T257066: Extension:Score / Lilypond is disabled on all wikis - https://phabricator.wikimedia.org/T257066 [02:47:18] (03CR) 10JJMC89: Pywikibot container (031 comment) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/603652 (https://phabricator.wikimedia.org/T249787) (owner: 10BryanDavis) [05:21:16] (03CR) 10Ammarpad: Rename WPBSkinBlacklist to WPBSkinDisabled (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609490 (https://phabricator.wikimedia.org/T254675) (owner: 10Peter.ovchyn) [05:26:56] (03PS1) 10DannyS712: Interwiki: Remove `spore` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609304 (https://phabricator.wikimedia.org/T257076) [05:33:55] (03PS2) 10DannyS712: Interwiki: Remove `spore` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609304 (https://phabricator.wikimedia.org/T257076) [05:34:23] (03CR) 10DannyS712: "This change is ready for review." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609304 (https://phabricator.wikimedia.org/T257076) (owner: 10DannyS712) [05:37:16] sukhe are you available? Emergency deployment might be needed [05:37:21] T257076 [05:37:21] T257076: Unable access to frwiki page "Spore : Aventures_galactiques" - https://phabricator.wikimedia.org/T257076 [05:37:45] The interwiki map needs to be updated to remove an entry than is breaking links - https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/609304 [07:00:04] Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20200704T0700) [07:12:16] PROBLEM - MediaWiki exceptions and fatals per minute on icinga1001 is CRITICAL: cluster=logstash job=statsd_exporter level=ERROR site=eqiad https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [07:15:58] RECOVERY - MediaWiki exceptions and fatals per minute on icinga1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Application_servers https://grafana.wikimedia.org/d/000000438/mediawiki-alerts?panelId=2&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops [08:22:13] (03PS2) 10JMeybohm: add chartmuseum[12]001 to dhcp netboot.cfg and site.pp [puppet] - 10https://gerrit.wikimedia.org/r/609449 (https://phabricator.wikimedia.org/T256970) [08:22:15] (03PS2) 10JMeybohm: Add cumin alias for chartmuseum hosts [puppet] - 10https://gerrit.wikimedia.org/r/609450 (https://phabricator.wikimedia.org/T256970) [08:25:48] (03PS3) 10JMeybohm: Add cumin alias for chartmuseum hosts [puppet] - 10https://gerrit.wikimedia.org/r/609450 (https://phabricator.wikimedia.org/T256970) [08:25:50] (03PS3) 10JMeybohm: add chartmuseum[12]001 to dhcp netboot.cfg and site.pp [puppet] - 10https://gerrit.wikimedia.org/r/609449 (https://phabricator.wikimedia.org/T256970) [08:32:13] (03PS3) 10Privacybatm: Transferer.py: Calculate source checksum parallel to the data transfer [software/transferpy] - 10https://gerrit.wikimedia.org/r/608640 (https://phabricator.wikimedia.org/T254979) [08:32:21] (03CR) 10jerkins-bot: [V: 04-1] Transferer.py: Calculate source checksum parallel to the data transfer [software/transferpy] - 10https://gerrit.wikimedia.org/r/608640 (https://phabricator.wikimedia.org/T254979) (owner: 10Privacybatm) [08:35:54] (03PS1) 10VulpesVulpes825: Change the remaining namespace default aliases from Chinese to English for Zh Projects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609508 (https://phabricator.wikimedia.org/T257101) [08:45:24] (03CR) 10Hamish: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609508 (https://phabricator.wikimedia.org/T257101) (owner: 10VulpesVulpes825) [08:48:27] (03PS1) 10Hazem: Revert "Add a maintenance script to get all LY files" [extensions/Score] (wmf/1.35.0-wmf.39) - 10https://gerrit.wikimedia.org/r/609529 [08:48:29] (03CR) 10Welcome, new contributor!: "Thank you for making your first contribution to Wikimedia! :) To learn how to get your code changes reviewed faster and more likely to get" [extensions/Score] (wmf/1.35.0-wmf.39) - 10https://gerrit.wikimedia.org/r/609529 (owner: 10Hazem) [08:49:52] (03Abandoned) 10Hazem: Revert "Add a maintenance script to get all LY files" [extensions/Score] (wmf/1.35.0-wmf.39) - 10https://gerrit.wikimedia.org/r/609529 (owner: 10Hazem) [08:50:28] (03Restored) 10Hazem: Revert "Add a maintenance script to get all LY files" [extensions/Score] (wmf/1.35.0-wmf.39) - 10https://gerrit.wikimedia.org/r/609529 (owner: 10Hazem) [08:50:46] (03Abandoned) 10Hazem: Revert "Add a maintenance script to get all LY files" [extensions/Score] (wmf/1.35.0-wmf.39) - 10https://gerrit.wikimedia.org/r/609529 (owner: 10Hazem) [09:04:47] T257076: Unable access to frwiki page "Spore : Aventures_galactiques" - https://phabricator.wikimedia.org/T257076 [09:04:47] T257076: Unable access to frwiki page "Spore : Aventures_galactiques" - https://phabricator.wikimedia.org/T257076 [09:04:52] pointed to me by DannyS712 [09:06:16] There was a `Spore:` prefix added to the interwiki map, which I have since removed on meta. However, it is still in the interwiki config, meaning that links to pages with `Spore:` are broken on multiple wikis. The hotfix patch removes the prefix from the config until the next full update of the interwiki config can occur [09:08:41] DannyS712: yeah then I dont qualify that as an unbreak now on a saturday. Looks that is merely a few pages on frwiki for now [09:09:11] and for sure, I am not going to touch to anything related to interwiki. I probably killed the site last time I did, that stuff is wayy toooooo scaaaaryyyy [09:09:34] what I am wondering is what would happens to page that had 'Spore' used as an interwiki link [09:09:42] its not just frwiki - https://meta.wikimedia.org/wiki/Talk:Interwiki_map#spore [09:09:57] I've already removed all uses of it as an intentional interwiki link (as of 27 June) [09:10:15] ahhhh great [09:10:31] so any remaining uses are meant for pages with `Spore` in the title [09:10:50] if you look at the patch is pretty simple - don't think there is much that can be broken :) [09:10:51] https://gerrit.wikimedia.org/r/c/609304 [09:11:02] yeah [09:11:11] you underestimate interwikis ;]]] [09:12:29] do you agree that this warrants a deployment, and are just uncomfortable doing it yourself? or do you think this should wait until monday? [09:13:27] DannyS712: wait for monday [09:13:42] cause even if it is super annoying and has a slight user impact, it is merely just a few pages [09:13:42] ack oh well. Will note that on the task [09:14:10] and I know for sure that interwikis are painful to deal with when something explodes [09:14:21] so even if updating the interwiki cache is probably straightforward (run whatever command) [09:14:35] that might well explode in the most unexpected way possible :^] [09:14:40] well not that it is a high risk [09:14:57] but monday morning no worries, I will be happy to do it [09:14:58] ;) [09:16:32] There is a chance I might not be available during that deployment window - can I answer any questions that you have now? [09:16:50] DannyS712: it will be fine. [09:17:27] DannyS712: I mean on monday we will get WMF / WMDE folks around that can assist ;] [09:18:27] and I guess we will need some test to ensure that namespaces and interwikis do not conflict with each other [09:18:53] namespaces shouldn't conflict, just individual titles? [09:18:59] I'll write a script [09:19:13] OH [09:19:18] I misread it on fr wiki bah [09:19:19] ;) [09:19:43] Also, can anyone delete https://wikitech.wikimedia.org/wiki/User:Lilpimpn86 please? [09:22:20] I don't have rights there :/ [09:25:51] I've tagged it for deletion [09:26:02] Do you know where in core the logic for processing interwiki prefixes is? [09:26:29] well [09:26:53] it is surely one of the oldest part of mediawiki, I am pretty sure that was there even before I started [09:27:04] so that must come from the perl wiki or something very old [09:27:36] the idea in the late 1990's was to link to each other website to share traffic [09:27:49] (I mean, before Google became a thing or blog existed) [09:28:12] so one could be a member of a webring, you would put a small banner automagically linking to other members participating [09:28:21] and in exchange you had your site linked from other members websites [09:28:27] that is what awesome :] [09:28:45] I guess the first Wiki engines borrowed that idea and went with interwikis to easily link to each others [09:28:59] cause the web is all that: links [09:29:09] if there are no links, there is no web per definition [09:29:14] anyway [09:29:30] our software went to copy paste the list from who knows what origin and we carried that since then [09:30:00] the internal would be in the parser somewhere I guess. But I would rather not touch or even look at that code anymore [09:30:03] DannyS712: ^^ ;) [09:30:07] a bit long story sorry [09:30:42] Nevermind, I manually tested on wiki to get the answer - prefix matches case insensitive and regardless of whitespace before the `:` - https://en.wikipedia.org/wiki/SPORE%20%20%20%20%20:e [09:31:28] DannyS712: re "There is a chance I might not be available during that deployment window": I can deploy it for you if you're not around then [09:31:47] also no objections for stealing my spot on the window :D [09:31:54] possibly MediaWiki should be able to recognize that [[Spore:_Galactic_Adventures]] is an existing article and thus should not try to make it an interwiki [09:32:23] but the parser probably doesn't know about that fact [09:32:57] but the parser knows which links are red and which are blue, so why it wouldn't know that [[Spore:_Galactic_Adventures]] exists [09:33:18] no idea ;] [09:33:31] it might be worth filing a task about it for whoever maintains the parser nowadays [09:34:38] thank you for the ping DannyS712 ;) [09:34:51] I am heading out, I have housework to do [09:35:08] it's not only the parser, pretty much all of mediawiki makes links [09:41:37] okay, I can't seem to get sql to convert the selected column to lowercase - any ideas? [09:41:41] +--------------+---------+----------------------+ [09:41:41] | page_title | page_id | LOWER(pg.page_title) | [09:41:41] +--------------+---------+----------------------+ [09:41:41] | FooBar_:_baz | 216779 | FooBar_:_baz | [09:41:41] +--------------+---------+----------------------+ [09:43:08] depends on the collation... you probably need to specify it manually [09:43:24] this is on the betacluster [10:27:34] okay. For any given new prefix `foobar`, the following should return any pages that would be broken: ` SELECT page_id, page_title, CONVERT(page_title USING latin1) FROM page WHERE page_namespace = 0 AND (CONVERT(page_title USING latin1) LIKE 'foobar\_:%' ESCAPE '\\' OR CONVERT(page_title USING latin1) LIKE 'foobar:%');` [12:40:20] (03PS1) 10Reedy: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609516 [12:40:22] (03CR) 10Reedy: [C: 03+2] Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609516 (owner: 10Reedy) [12:41:01] (03Merged) 10jenkins-bot: Update interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609516 (owner: 10Reedy) [12:42:29] !log reedy@deploy1001 Synchronized wmf-config/interwiki.php: Update interwiki cache (duration: 02m 24s) [12:42:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:42:55] (03Abandoned) 10DannyS712: Interwiki: Remove `spore` [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609304 (https://phabricator.wikimedia.org/T257076) (owner: 10DannyS712) [13:08:59] DannyS712: I am available now if I can still be helpful [13:09:16] Nope, no longer needed, thanks though [13:10:12] np. I am around the rest of the day if required, hopefully not :) [14:05:16] !log Disable puppet on gerrit1002 (gerrit-test) to deploy Gerrit UI updates there to gather feedback [14:05:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:29] (03PS1) 10QChris: gerrit: Add table for Zuul CI results underneath the commit message [puppet] - 10https://gerrit.wikimedia.org/r/609519 (https://phabricator.wikimedia.org/T256575) [14:09:31] (03PS1) 10QChris: gerrit: Add SonarQube results to CI table [puppet] - 10https://gerrit.wikimedia.org/r/609520 [14:10:22] (03PS2) 10QChris: gerrit: Add table for Zuul CI results underneath the commit message [puppet] - 10https://gerrit.wikimedia.org/r/609519 (https://phabricator.wikimedia.org/T256575) [14:10:24] (03PS2) 10QChris: gerrit: Add SonarQube results to CI table [puppet] - 10https://gerrit.wikimedia.org/r/609520 (https://phabricator.wikimedia.org/T256575) [14:25:35] (03CR) 10QChris: "I've put this change live on gerrit-test.wikimedia.org to allow reviewing:" [puppet] - 10https://gerrit.wikimedia.org/r/609520 (https://phabricator.wikimedia.org/T256575) (owner: 10QChris) [14:25:57] (03CR) 10QChris: "I've put this change live on gerrit-test.wikimedia.org to allow reviewing." [puppet] - 10https://gerrit.wikimedia.org/r/609519 (https://phabricator.wikimedia.org/T256575) (owner: 10QChris) [18:21:49] (03PS2) 10Peter.ovchyn: Rename WPBSkinBlacklist to WPBSkinDisabled [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609490 (https://phabricator.wikimedia.org/T254675) [18:22:55] (03CR) 10Peter.ovchyn: Rename WPBSkinBlacklist to WPBSkinDisabled (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/609490 (https://phabricator.wikimedia.org/T254675) (owner: 10Peter.ovchyn) [19:17:33] (03PS1) 10QChris: Bump gitiles plugin to v3.2.2-1-g00c5ca0-with-0e3b533 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/609547 (https://phabricator.wikimedia.org/T256571) [19:21:58] (03CR) 10QChris: [V: 03+2 C: 03+2] Bump gitiles plugin to v3.2.2-1-g00c5ca0-with-0e3b533 [software/gerrit] (deploy/wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/609547 (https://phabricator.wikimedia.org/T256571) (owner: 10QChris) [19:23:30] !log qchris@deploy1001 Started deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 [19:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:23:39] !log qchris@deploy1001 Finished deploy [gerrit/gerrit@b78914b]: Bump gitiles to v3.2.2-1-g00c5ca0-with-0e3b533 on gerrit1002 (duration: 00m 08s) [19:23:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:39:55] 10Operations, 10observability, 10User-MoritzMuehlenhoff: Fix paniclog alert to only sent mails once - https://phabricator.wikimedia.org/T257016 (10herron) One idea to avoid causing alerts during spamassassin restarts is have the two MXes act as failback spamd servers for each other. This way when the local... [19:43:31] (03PS1) 10Urbanecm: Add sysop_itwiki to private_wikis [puppet] - 10https://gerrit.wikimedia.org/r/609548 (https://phabricator.wikimedia.org/T256545) [19:44:25] (03PS1) 10Urbanecm: Update urbanecm's .gitconfig [puppet] - 10https://gerrit.wikimedia.org/r/609549 [19:46:39] (03PS1) 10Urbanecm: Create sysop-it.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/609550 (https://phabricator.wikimedia.org/T256545) [19:52:32] (03CR) 10QChris: [C: 04-1] "I'd not mess with Jenkins at this point, because after some massaging the plugin mentioned" [puppet] - 10https://gerrit.wikimedia.org/r/608296 (https://phabricator.wikimedia.org/T256575) (owner: 10Hashar) [19:58:51] (03Abandoned) 10Urbanecm: Create sysop-it.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/609550 (https://phabricator.wikimedia.org/T256545) (owner: 10Urbanecm) [20:02:41] (03PS1) 10Urbanecm: Create sysop-it.wikipedia.org [dns] - 10https://gerrit.wikimedia.org/r/609551 (https://phabricator.wikimedia.org/T256545) [20:22:39] (03CR) 10Krinkle: "The SonarQubeBot links are all the same but are duuplicated 5 times, that seems confusing and also perhaps a bit too prominent." [puppet] - 10https://gerrit.wikimedia.org/r/608296 (https://phabricator.wikimedia.org/T256575) (owner: 10Hashar) [20:23:52] (03CR) 10Krinkle: "Typical core commit:" [puppet] - 10https://gerrit.wikimedia.org/r/608296 (https://phabricator.wikimedia.org/T256575) (owner: 10Hashar) [20:28:19] (03CR) 10Krinkle: [C: 04-1] "The Sonar links all point to the same link, it seems confusing to duplicate that 5 or 6 times. Maybe limit it it to one entry for "Sonar" [puppet] - 10https://gerrit.wikimedia.org/r/609520 (https://phabricator.wikimedia.org/T256575) (owner: 10QChris) [20:32:31] (03CR) 10Krinkle: gerrit: Add table for Zuul CI results underneath the commit message (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/609519 (https://phabricator.wikimedia.org/T256575) (owner: 10QChris) [20:33:42] (03CR) 10Krinkle: [C: 04-1] gerrit: Add table for Zuul CI results underneath the commit message (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/609519 (https://phabricator.wikimedia.org/T256575) (owner: 10QChris) [22:51:40] (03CR) 10Bmansurov: Add recommendation-api helmfile stanzas (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/602527 (https://phabricator.wikimedia.org/T241230) (owner: 10Bmansurov)