[00:04:08] 10Operations, 10Reading-Infrastructure-Team-Backlog, 10SRE-Access-Requests, 10Patch-For-Review: Add Michael Holloway (Reading Infrastructure) to maps admin groups - https://phabricator.wikimedia.org/T194404#4201910 (10Mholloway) @Dzahn Hmm, good question. Maybe that would be @Gehel? [00:06:30] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1967 bytes in 0.086 second response time [00:48:46] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#3503912 (10Dzahn) You have successfully created the mailing list toolforge-standards-committee and notification has been sent... [00:50:39] 10Operations, 10Wikimedia-Mailing-lists: Enable reCAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201943 (10Huji) [00:50:50] 10Operations, 10Wikimedia-Mailing-lists: Enable reCAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201954 (10Huji) p:05Triage>03High [00:53:15] 10Operations, 10Wikimedia-Mailing-lists: Enable reCAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201956 (10Huji) [00:54:38] 10Operations, 10Wikimedia-Mailing-lists: Enable reCAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201943 (10Huji) [00:56:13] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#4201958 (10Dzahn) ``` [fermium:~] $ sudo /usr/local/sbin/rename_list --help usage: /usr/local/sbin/rename_list (03PS1) 10Dzahn: lists: rename tool-labs-standards-committee to toolforge-standards-committee [puppet] - 10https://gerrit.wikimedia.org/r/432709 (https://phabricator.wikimedia.org/T127624) [01:05:21] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4201965 (10Dzahn) ``` [helium:~] $ sudo bconsole Connecting to Director helium.eqiad.wmnet:9101 .. When at a prompt, entering a period... [01:07:11] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4201980 (10Dzahn) ``` When at a prompt, entering a period cancels the command. *status Status available for: 1: Director 2:... [01:08:24] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418#4201981 (10Dzahn) 05Open>03Resolved [01:09:17] (03PS2) 10Dzahn: lists: rename tool-labs-standards-committee to toolforge-standards-committee [puppet] - 10https://gerrit.wikimedia.org/r/432709 (https://phabricator.wikimedia.org/T127624) [01:10:12] (03PS3) 10Dzahn: lists: rename tool-labs-standards-committee to toolforge-standards-committee [puppet] - 10https://gerrit.wikimedia.org/r/432709 (https://phabricator.wikimedia.org/T127624) [01:10:23] (03CR) 10Dzahn: [C: 032] lists: rename tool-labs-standards-committee to toolforge-standards-committee [puppet] - 10https://gerrit.wikimedia.org/r/432709 (https://phabricator.wikimedia.org/T127624) (owner: 10Dzahn) [01:14:40] (03PS1) 10Dzahn: mailman: update outdated path in list_rename script [puppet] - 10https://gerrit.wikimedia.org/r/432711 [01:15:21] (03CR) 10Dzahn: [C: 032] mailman: update outdated path in list_rename script [puppet] - 10https://gerrit.wikimedia.org/r/432711 (owner: 10Dzahn) [01:18:33] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#3503912 (10Dzahn) https://gerrit.wikimedia.org/r/#/c/432709/ [01:18:50] (03CR) 10Dzahn: [C: 032] "linked wrong ticket. belongs to https://phabricator.wikimedia.org/T172624" [puppet] - 10https://gerrit.wikimedia.org/r/432709 (https://phabricator.wikimedia.org/T127624) (owner: 10Dzahn) [01:28:58] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#4202007 (10Dzahn) > Please create a new list, and transfer all current settings/subscribers/admins over. done! https://list... [01:29:26] 10Operations, 10Toolforge-standards-committee, 10Wikimedia-Mailing-lists: Rename (recreate) mailing list for Toolforge-standards-committee - https://phabricator.wikimedia.org/T172624#4202009 (10Dzahn) 05Open>03Resolved a:03Dzahn [01:34:38] 10Operations, 10Africa-Wikimedia-Developers, 10Wikimedia-Mailing-lists: Rename project mailing list for Africa Wikimedia Developers project - https://phabricator.wikimedia.org/T183832#4202014 (10Dzahn) Hello @D3r1ck01 Are you still interested in a rename of the list? We meanwhile have a script for that so it... [01:39:05] 10Operations, 10Community-Liaisons, 10Developer-Relations, 10Wikimedia-Mailing-lists: Rename (create anew) the TC team mailing list - https://phabricator.wikimedia.org/T155683#4202019 (10Dzahn) Let us know if you are still interested in list renaming. A renaming script can be used (https://wikitech.wikime... [01:50:20] 10Operations, 10Africa-Wikimedia-Developers, 10Wikimedia-Mailing-lists: Rename project mailing list for Africa Wikimedia Developers project - https://phabricator.wikimedia.org/T183832#4202045 (10Dzahn) p:05Triage>03Low [01:50:40] 10Operations, 10SRE-Access-Requests: Give Seddon access to the analytics cluster - https://phabricator.wikimedia.org/T194445#4202046 (10Dzahn) p:05Triage>03Normal [01:56:24] (03PS1) 10Krinkle: mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) [01:56:54] (03CR) 10jerkins-bot: [V: 04-1] mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) (owner: 10Krinkle) [01:57:39] (03PS2) 10Krinkle: mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) [01:58:06] (03CR) 10jerkins-bot: [V: 04-1] mtail: Add xcachestatus to varnishrls [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) (owner: 10Krinkle) [01:58:28] (03CR) 10Krinkle: "Not sure how to update rule_ops. Ideally not by renaming the metric, but also, the front portion should describe the 'by' clause. I suppos" [puppet] - 10https://gerrit.wikimedia.org/r/432712 (https://phabricator.wikimedia.org/T190978) (owner: 10Krinkle) [03:26:00] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1972 bytes in 0.095 second response time [03:31:11] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1962 bytes in 0.126 second response time [03:34:22] PROBLEM - High load average on labstore1003 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [24.0] https://grafana.wikimedia.org/dashboard/db/labs-monitoring [03:36:11] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 855.73 seconds [03:36:52] RECOVERY - High load average on labstore1003 is OK: OK: Less than 50.00% above the threshold [16.0] https://grafana.wikimedia.org/dashboard/db/labs-monitoring [03:44:20] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1972 bytes in 0.090 second response time [04:07:50] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 278.66 seconds [04:26:20] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1966 bytes in 0.112 second response time [05:29:07] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4202129 (10MZMcBride) [05:30:02] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201943 (10MZMcBride) I changed "reCAPTCHA" to "CAPTCHA" in the task title because we've typically avoided reCAPTCHA specifically, in favor of using our own CAPTCHA on the wikis. [05:42:30] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1969 bytes in 0.093 second response time [05:53:00] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1948 bytes in 0.120 second response time [06:00:40] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1974 bytes in 0.108 second response time [06:16:30] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1965 bytes in 0.088 second response time [06:50:13] !log rm 2FA from User:Fatemi (at 6:38 PM UTC, Friday May 11) [06:50:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:51:05] (sorry) [07:11:30] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1968 bytes in 0.092 second response time [07:26:14] 10Operations, 10Wikimedia-Mailing-lists: Reset admin password for Wikipedia-FR-Wikimag mailinglist - https://phabricator.wikimedia.org/T194466#4202174 (10Romainhk) It worked. Thanks :) [07:27:10] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1946 bytes in 0.106 second response time [08:21:33] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201943 (10Vogone) Although the idea sounds nice I believe it would not prevent all bot-generated subscription requests since sending a -request@lists.wikimedia.org mail as well ca... [08:26:01] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4202207 (10Vogone) Sorry, I meant -join@… as per http://www.list.org/mailman-member/node13.html. [08:32:49] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4201943 (10MarcoAurelio) I support reCAPTCHA. Our CAPTCHA is broken (cf. the ammount of spambots we lock everyday) in adition to other measures such as blocking subscription requests sent to... [08:56:38] (03CR) 10Mobrovac: [C: 031] Kafka: increase group.initial.rebalance.delay.ms to 10s. [puppet] - 10https://gerrit.wikimedia.org/r/432615 (https://phabricator.wikimedia.org/T189618) (owner: 10Ppchelko) [09:20:36] (03PS2) 10Merlijn van Deen: labs/db: create basic integration test for maintain-meta_p [puppet] - 10https://gerrit.wikimedia.org/r/432698 [11:13:20] PROBLEM - HHVM jobrunner on mw1300 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time [11:14:40] RECOVERY - HHVM jobrunner on mw1300 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.013 second response time [13:12:53] 10Operations, 10Wikimedia-Mailing-lists: Enable CAPTCHA on mailman instances - https://phabricator.wikimedia.org/T194558#4202287 (10Huji) >>! In T194558#4202205, @Vogone wrote: > Although the idea sounds nice I believe it would not prevent all bot-generated subscription requests since sending a -requ... [14:24:00] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1972 bytes in 0.107 second response time [14:29:11] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1943 bytes in 0.117 second response time [14:34:54] (03PS9) 10MarcoAurelio: idwikimedia: initial configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/429385 (https://phabricator.wikimedia.org/T192726) [14:39:27] (03PS3) 10Gergő Tisza: Add techadmin to privileged groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421122 (https://phabricator.wikimedia.org/T190015) [14:39:29] (03PS4) 10Gergő Tisza: Temporarily preserve sysops' JS editing ability [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421123 (https://phabricator.wikimedia.org/T190015) [14:39:31] (03PS5) 10Gergő Tisza: Remove sitewide and user CSS/JS editing from old groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421124 (https://phabricator.wikimedia.org/T190015) [14:39:33] (03PS6) 10Gergő Tisza: Enforce that techadmin is the only group that can edit non-own CSS/JS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421125 (https://phabricator.wikimedia.org/T190015) [14:42:40] tgr|away: new userright? [14:43:03] s/userright/user group? [14:43:19] Hauskatze: o/ [14:43:44] was that a question or are you saying I mixed up those words somewhere? [14:44:13] Szia! [14:44:29] no, a question for all those changes in general [14:44:45] want to know what'll be happening, etc :) [14:45:21] the linked task has the story [14:45:38] T190015 that is [14:45:40] T190015: Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default - https://phabricator.wikimedia.org/T190015 [14:45:41] * Hauskatze will check after fixing a patch [14:47:30] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1971 bytes in 0.081 second response time [14:49:27] tgr: then T152296 is unnaceptable to keep as I've advocated [14:49:28] T152296: Review the 'botadmin' group at mlwiktionary and mlwikisource - https://phabricator.wikimedia.org/T152296 [14:51:08] Hauskatze: well, to the extent of 'botadmin' giving the same permissions as 'sysop', that would not be affected, as both would lose JS editing powers [14:51:37] so sysops will need an aditional userright if I'm reading rightly [14:51:39] the last patch in that chain enforces on the software level that there is no way to edit sitewide JS without being techadmin [14:51:55] they will need an additional user group [14:52:14] well, the proposal could work but I feel it'll just move the attack vector to those holders [14:52:36] smaller group -> smaller attack surface [14:53:09] and users who are able to edit JS tend to be more technically apt and probably have better password security and password practices [14:53:20] yep, just sayin' :) [14:53:52] who will be able to add you to those groups? 'crats? [14:53:58] yeah [14:54:12] well, some projects grant sysop+'crat together :) [14:54:41] can't fix all the broken things in one strike [14:55:43] thanks for taking care of this, I hope it works :) [15:00:33] 10Operations, 10Wikimedia-Mailing-lists: Archive "wiki-offline-reader-l" - https://phabricator.wikimedia.org/T194575#4202356 (10Kelson) [15:06:03] 10Operations, 10Wikimedia-Mailing-lists: Archive "wiki-offline-reader-l" - https://phabricator.wikimedia.org/T194575#4202371 (10MarcoAurelio) Is deletion of the list an option or is there anything to preserve? [15:13:41] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1950 bytes in 0.131 second response time [15:25:14] 10Operations, 10Wikimedia-Mailing-lists: Archive "wiki-offline-reader-l" - https://phabricator.wikimedia.org/T194575#4202381 (10Kelson) @MarcoAurelio I would be more comfortable if the messages were kept in an web available archive. [15:26:02] 10Operations, 10Wikimedia-Mailing-lists: Archive "wiki-offline-reader-l" - https://phabricator.wikimedia.org/T194575#4202382 (10MarcoAurelio) Sure. Making the list "locked" is also posible. No problem. [15:27:18] 10Operations, 10Reading-Infrastructure-Team-Backlog, 10SRE-Access-Requests, 10Patch-For-Review: Add Michael Holloway (Reading Infrastructure) to maps admin groups - https://phabricator.wikimedia.org/T194404#4202383 (10Gehel) I'm not sure I'm the project lead for maps, and I'm not sure we do have one. @jmat... [17:52:10] 10Operations, 10Wikimedia-Mailing-lists, 10User-revi: Close arbcom-ko mailing list - https://phabricator.wikimedia.org/T160892#4202510 (10revi) [17:52:13] 10Operations, 10Wikimedia-Mailing-lists: Remove darkbluesky from being mailinglist admin on arbcom-ko (cannot access anymore) - https://phabricator.wikimedia.org/T187174#4202508 (10revi) @Dzahn That list should have been closed: {T160892}. [17:55:41] 10Operations, 10Wikimedia-Mailing-lists, 10User-revi: Remove darkbluesky from being mailinglist admin on arbcom-ko (cannot access anymore) - https://phabricator.wikimedia.org/T187174#4202515 (10revi) [17:57:40] 10Operations, 10Wikimedia-Mailing-lists: wikitech-l is mangling my PGP/MIME emails, causing signature validation to fail - https://phabricator.wikimedia.org/T186311#4202523 (10Legoktm) I haven't, though I encountered the same problem on mediawiki-l recently. Both wikitech-l admins are subscribed to this ticket... [18:06:01] 10Operations, 10Wikimedia-Mailing-lists: wikitech-l is mangling my PGP/MIME emails, causing signature validation to fail - https://phabricator.wikimedia.org/T186311#4202526 (10Aklapper) Confirming. I could reproduce with https://lists.wikimedia.org/pipermail/wikitech-l/2018-February/089475.html after importin... [21:34:00] PROBLEM - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1967 bytes in 0.081 second response time [21:39:10] RECOVERY - wikidata.org dispatch lag is higher than 300s on www.wikidata.org is OK: HTTP OK: HTTP/1.1 200 OK - 1973 bytes in 0.095 second response time [23:37:13] (03PS2) 10Subramanya Sastry: Enable RemexHtml on wikis with < 100 ns0 errors in high priority cats [mediawiki-config] - 10https://gerrit.wikimedia.org/r/432621 (https://phabricator.wikimedia.org/T193685)