[00:40:27] [00:40:31] On https://design.wikimedia.org/ [00:41:02] Sigh. [00:42:26] Oh, this isn't the channel I really wanted, but whatevs. [03:27:53] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 907.89 seconds [03:57:34] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 288.69 seconds [06:19:22] (03PS7) 10ArielGlenn: generate and email a few dump-related stats each month [puppet] - 10https://gerrit.wikimedia.org/r/303707 (https://phabricator.wikimedia.org/T142435) [06:20:35] (03CR) 10ArielGlenn: [C: 032] generate and email a few dump-related stats each month [puppet] - 10https://gerrit.wikimedia.org/r/303707 (https://phabricator.wikimedia.org/T142435) (owner: 10ArielGlenn) [06:22:44] (03PS1) 10ArielGlenn: commit the changes to previous commit that I left lying around [puppet] - 10https://gerrit.wikimedia.org/r/447219 [06:23:03] (03PS2) 10ArielGlenn: commit the changes to previous commit that I left lying around [puppet] - 10https://gerrit.wikimedia.org/r/447219 [06:23:07] (03CR) 10jerkins-bot: [V: 04-1] commit the changes to previous commit that I left lying around [puppet] - 10https://gerrit.wikimedia.org/r/447219 (owner: 10ArielGlenn) [06:25:21] (03PS3) 10ArielGlenn: commit the changes to previous commit that I left lying around [puppet] - 10https://gerrit.wikimedia.org/r/447219 [06:26:21] (03CR) 10ArielGlenn: [C: 032] commit the changes to previous commit that I left lying around [puppet] - 10https://gerrit.wikimedia.org/r/447219 (owner: 10ArielGlenn) [06:30:15] PROBLEM - puppet last run on labvirt1014 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/apt2xml] [06:51:42] (03PS1) 10ArielGlenn: move hewiki to 'big wikis' list for xml/sql dumps [puppet] - 10https://gerrit.wikimedia.org/r/447220 (https://phabricator.wikimedia.org/T200146) [06:55:54] RECOVERY - puppet last run on labvirt1014 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [07:10:13] PROBLEM - Disk space on elastic1025 is CRITICAL: DISK CRITICAL - free space: /srv 51835 MB (10% inode=99%) [07:42:04] RECOVERY - Disk space on elastic1025 is OK: DISK OK [08:35:43] PROBLEM - toolschecker: tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 20 seconds [08:36:33] RECOVERY - toolschecker: tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1015 bytes in 0.031 second response time [09:03:24] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [09:04:02] grrr... what made the toolforge landing page flap? [09:04:14] * bd808 looks in his email for clues [09:04:33] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [09:13:37] (03PS5) 10Gergő Tisza: Add techadmin to privileged groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421122 (https://phabricator.wikimedia.org/T190015) [09:13:39] (03PS6) 10Gergő Tisza: Temporarily preserve sysops' JS editing ability [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421123 (https://phabricator.wikimedia.org/T190015) [09:13:41] (03PS2) 10Gergő Tisza: Configure group management for techadmin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440676 [09:13:43] (03PS7) 10Gergő Tisza: Remove sitewide and user CSS/JS editing from old groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421124 (https://phabricator.wikimedia.org/T190015) [09:13:45] (03PS8) 10Gergő Tisza: Enforce that techadmin is the only group that can edit non-own CSS/JS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421125 (https://phabricator.wikimedia.org/T190015) [09:14:24] (03CR) 10jerkins-bot: [V: 04-1] Temporarily preserve sysops' JS editing ability [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421123 (https://phabricator.wikimedia.org/T190015) (owner: 10Gergő Tisza) [09:14:59] (03CR) 10BryanDavis: [C: 031] gridengine: Add package information for stretch exec nodes [puppet] - 10https://gerrit.wikimedia.org/r/447089 (https://phabricator.wikimedia.org/T199276) (owner: 10Bstorm) [09:15:02] (03CR) 10jerkins-bot: [V: 04-1] Enforce that techadmin is the only group that can edit non-own CSS/JS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421125 (https://phabricator.wikimedia.org/T190015) (owner: 10Gergő Tisza) [09:51:27] (03PS6) 10Gergő Tisza: Add interface-admin to privileged groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421122 (https://phabricator.wikimedia.org/T190015) [09:51:29] (03PS7) 10Gergő Tisza: Temporarily preserve sysops' JS editing ability [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421123 (https://phabricator.wikimedia.org/T190015) [09:51:31] (03PS3) 10Gergő Tisza: Configure group management for interface-admin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440676 [09:51:33] (03PS8) 10Gergő Tisza: Remove sitewide and user CSS/JS editing from old groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421124 (https://phabricator.wikimedia.org/T190015) [09:51:35] (03PS9) 10Gergő Tisza: Enforce that interface-admin is the only group that can edit non-own CSS/JS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421125 (https://phabricator.wikimedia.org/T190015) [09:56:48] 10Operations, 10Cloud-Services, 10hardware-requests, 10Patch-For-Review, 10cloud-services-team (Kanban): decom silver (was silver has trouble rebooting) - https://phabricator.wikimedia.org/T168559 (10Jdforrester-WMF) 05duplicate>03Open As this is in the tree, making this the target. [09:57:01] 10Operations, 10Cloud-Services, 10hardware-requests, 10Patch-For-Review, 10cloud-services-team (Kanban): decom silver (was silver has trouble rebooting) - https://phabricator.wikimedia.org/T168559 (10Jdforrester-WMF) [09:57:27] 10Operations, 10decommission: Reclaim/Decommission Silver.wikimedia.org - https://phabricator.wikimedia.org/T190085 (10Jdforrester-WMF) [09:57:29] 10Operations, 10Cloud-Services, 10hardware-requests, 10Patch-For-Review, 10cloud-services-team (Kanban): decom silver (was silver has trouble rebooting) - https://phabricator.wikimedia.org/T168559 (10Jdforrester-WMF) [09:57:45] 10Operations, 10cloud-services-team, 10Epic: replace all Ubuntu (trusty) hosts in production with Debian - https://phabricator.wikimedia.org/T186288 (10Jdforrester-WMF) [10:15:45] (03CR) 10Gergő Tisza: "PS 5: rebase" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421122 (https://phabricator.wikimedia.org/T190015) (owner: 10Gergő Tisza) [10:16:35] (03CR) 10Gergő Tisza: "PS6-7: rebase, group name change" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/421123 (https://phabricator.wikimedia.org/T190015) (owner: 10Gergő Tisza) [11:19:44] (03PS1) 10Jcrespo: move_replica.py: A script to do replica topology changes [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447225 [11:20:06] (03CR) 10jerkins-bot: [V: 04-1] move_replica.py: A script to do replica topology changes [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447225 (owner: 10Jcrespo) [11:46:57] (03CR) 10MarcoAurelio: Configure group management for interface-admin (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440676 (owner: 10Gergő Tisza) [11:50:17] (03CR) 10MarcoAurelio: "Also, if you change the name to interface-admin here, you should also change the name elsewhere. WikimediaMessages do have already several" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440676 (owner: 10Gergő Tisza) [11:53:23] (03CR) 10MarcoAurelio: "Scratch the latests. I see that https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/421121/ does that (for core, not WikimediaMessages; go" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440676 (owner: 10Gergő Tisza) [11:58:19] (03PS1) 10Addshore: More Wikimania throttle exceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/447228 (https://phabricator.wikimedia.org/T198288) [11:58:52] (03CR) 10Legoktm: [C: 031] More Wikimania throttle exceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/447228 (https://phabricator.wikimedia.org/T198288) (owner: 10Addshore) [11:59:10] (03CR) 10Addshore: [C: 032] More Wikimania throttle exceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/447228 (https://phabricator.wikimedia.org/T198288) (owner: 10Addshore) [12:00:19] (03Merged) 10jenkins-bot: More Wikimania throttle exceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/447228 (https://phabricator.wikimedia.org/T198288) (owner: 10Addshore) [12:00:35] (03CR) 10jenkins-bot: More Wikimania throttle exceptions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/447228 (https://phabricator.wikimedia.org/T198288) (owner: 10Addshore) [12:02:50] *twiddles thumbs* [12:03:32] !log addshore@deploy1001 Synchronized wmf-config/throttle.php: T198288 More Wikimania throttle rules (duration: 01m 18s) [12:03:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:03:37] T198288: Increase account creation at Wikimania 2018 July 18-22 - https://phabricator.wikimedia.org/T198288 [12:08:01] !log addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 196.208.95.30 [12:08:01] !log addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.68.116 [12:08:01] !log addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.150 [12:08:01] !log addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.159 [12:08:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:08:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:18:24] PROBLEM - toolschecker: check mtime mod from tools cron job on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/toolscron - 185 bytes in 0.013 second response time [12:20:43] RECOVERY - toolschecker: check mtime mod from tools cron job on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.007 second response time [13:15:23] PROBLEM - Disk space on maps1001 is CRITICAL: DISK CRITICAL - free space: /srv 54815 MB (3% inode=99%) [13:26:53] RECOVERY - Disk space on maps1001 is OK: DISK OK [13:40:01] 10Operations, 10Wikimedia-Mailing-lists: Web SourceContent Control - https://phabricator.wikimedia.org/T200161 (10Sourcecontent1) [13:55:41] 10Operations, 10Wikimedia-Mailing-lists: Web SourceContent Control - https://phabricator.wikimedia.org/T200161 (10Legoktm) 05Open>03Invalid If you want to be a mass message sender on the English Wikipedia you should file a request at https://en.wikipedia.org/wiki/Wikipedia:Requests_for_permissions#rperm-ma... [13:58:24] 10Operations, 10Wikimedia-Mailing-lists: Web SourceContent Control - https://phabricator.wikimedia.org/T200161 (10Aklapper) a:05Sourcecontent1>03None [16:14:54] (03PS1) 10Jcrespo: mariadb: Added functionality to perform arbitrary topology changes [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447233 [16:15:21] (03CR) 10jerkins-bot: [V: 04-1] mariadb: Added functionality to perform arbitrary topology changes [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447233 (owner: 10Jcrespo) [16:15:57] !log rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix) [16:16:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:16:01] mobrovac: --^ [17:12:36] (03CR) 10Framawiki: "> Would this change make unable to login those [unlikely] accounts not meeting the new password guidelines after implementing? If so, an a" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440834 (https://phabricator.wikimedia.org/T197577) (owner: 10MarcoAurelio) [17:40:32] (03CR) 10Marostegui: "I will most likely be stopping replicas in sync in the next few days so I'm happy to be a beta tester for this script whenever you think i" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447225 (owner: 10Jcrespo) [18:36:34] PROBLEM - Disk space on maps1001 is CRITICAL: DISK CRITICAL - free space: /srv 54942 MB (3% inode=99%) [18:41:13] RECOVERY - Disk space on maps1001 is OK: DISK OK [18:45:24] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is CRITICAL: CRITICAL - failed 20 probes of 306 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [18:50:33] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is OK: OK - failed 19 probes of 306 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [19:47:11] (03PS2) 10Andrew Bogott: labservices: help pdns talk to the local database [puppet] - 10https://gerrit.wikimedia.org/r/447105 [19:52:45] (03CR) 10Andrew Bogott: [C: 032] labservices: help pdns talk to the local database [puppet] - 10https://gerrit.wikimedia.org/r/447105 (owner: 10Andrew Bogott) [22:46:53] (03PS1) 10Jcrespo: replication: Add additional replica migration functionalities [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447341 [22:47:20] (03CR) 10jerkins-bot: [V: 04-1] replication: Add additional replica migration functionalities [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/447341 (owner: 10Jcrespo) [23:04:36] (03PS1) 10Andrew Bogott: designate: allow second_region_designate_host to talk to keystone [puppet] - 10https://gerrit.wikimedia.org/r/447343 [23:12:58] (03PS2) 10Andrew Bogott: designate: allow second_region_designate_host to talk to keystone [puppet] - 10https://gerrit.wikimedia.org/r/447343 [23:13:58] (03CR) 10Andrew Bogott: [C: 032] designate: allow second_region_designate_host to talk to keystone [puppet] - 10https://gerrit.wikimedia.org/r/447343 (owner: 10Andrew Bogott) [23:22:18] (03PS1) 10Andrew Bogott: labtestn: no longer assume labtest and labtestn use the same designate [puppet] - 10https://gerrit.wikimedia.org/r/447352 [23:23:04] (03CR) 10Andrew Bogott: [C: 032] labtestn: no longer assume labtest and labtestn use the same designate [puppet] - 10https://gerrit.wikimedia.org/r/447352 (owner: 10Andrew Bogott) [23:26:51] (03PS1) 10Andrew Bogott: labtestn designate: update reverse dns zone ID [puppet] - 10https://gerrit.wikimedia.org/r/447353 [23:28:33] (03CR) 10Andrew Bogott: [C: 032] labtestn designate: update reverse dns zone ID [puppet] - 10https://gerrit.wikimedia.org/r/447353 (owner: 10Andrew Bogott)