[01:29:08] PROBLEM - High lag on wdqs1003 is CRITICAL: 3635 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [02:04:56] (03PS9) 10MacFan4000: Phabricator: Block vandalism IP addresses [puppet] - 10https://gerrit.wikimedia.org/r/440510 (owner: 10Aklapper) [02:05:20] (03CR) 10MacFan4000: [C: 031] Phabricator: Block vandalism IP addresses [puppet] - 10https://gerrit.wikimedia.org/r/440510 (owner: 10Aklapper) [02:16:58] RECOVERY - High lag on wdqs1003 is OK: (C)3600 ge (W)1200 ge 1197 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [02:23:29] (03CR) 10Subramanya Sastry: [C: 04-1] "Need to make sure this new PS doesn't break references $wgTidyConfig in CommonSettings.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [02:34:20] !log l10nupdate@deploy1001 scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 39s) [02:34:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:44:40] !log l10nupdate@deploy1001 ResourceLoader cache refresh completed at Mon Jul 2 02:44:40 UTC 2018 (duration 10m 21s) [02:44:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:08:21] 10Operations, 10Analytics, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837 (10CommunityTechBot) [03:08:24] 10Operations, 10Analytics, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837 (10CommunityTechBot) a:03MoritzMuehlenhoff [03:08:27] 10Operations, 10Analytics, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837 (10CommunityTechBot) p:05High>03Normal [03:08:30] 10Operations, 10Analytics, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837 (10CommunityTechBot) 05Open>03Resolved [03:08:33] 10Operations, 10Analytics, 10SRE-Access-Requests, 10Patch-For-Review, 10Performance-Team (Radar): Requesting access to analytics-privatedata-users for gilles - https://phabricator.wikimedia.org/T195837 (10CommunityTechBot) [03:18:23] I can't access Phabricator... [03:18:25] Lol [03:18:35] Got blocked apparently. Too many refreshes haha [03:18:58] I got the dreaded "TOO MANY REQUESTS" [03:26:39] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 780.40 seconds [03:32:09] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 299.34 seconds [04:24:47] !log applied security patch on phab1001, source in /srv/patches [04:24:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:25:25] !log Bsadowski1 The throttle is averaged over 5 minutes, you should be unblocked shortly after being blocked [04:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:25:30] err [04:25:35] !unlog [04:25:38] lol [04:50:33] (03PS1) 10Marostegui: db-eqiad.php: Depool db1087 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) [04:52:20] (03PS2) 10Marostegui: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) [04:53:12] (03PS3) 10Marostegui: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) [04:55:43] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [04:56:58] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [04:58:02] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1109 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443345 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [04:58:14] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 00m 56s) [04:58:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:59:10] !log Deploy schema change on db1109 T191316 T192926 T89737 T195193 [04:59:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:59:16] T89737: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 [04:59:16] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [04:59:16] T195193: Schema change for ct_tag_id field to change_tag" - https://phabricator.wikimedia.org/T195193 [04:59:17] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [05:01:48] (03PS1) 10KartikMistry: apertium-separable: New upstream release [debs/contenttranslation/apertium-separable] - 10https://gerrit.wikimedia.org/r/443346 (https://phabricator.wikimedia.org/T191404) [05:02:17] (03CR) 10jerkins-bot: [V: 04-1] apertium-separable: New upstream release [debs/contenttranslation/apertium-separable] - 10https://gerrit.wikimedia.org/r/443346 (https://phabricator.wikimedia.org/T191404) (owner: 10KartikMistry) [05:09:48] 10Operations, 10ops-eqiad: Degraded RAID on db1063 - https://phabricator.wikimedia.org/T196918 (10Marostegui) 05Open>03Resolved p:05High>03Normal [05:10:00] Lol [05:12:58] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 224, down: 1, dormant: 0, excluded: 0, unused: 0 [05:45:39] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [05:46:48] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [05:50:03] ACKNOWLEDGEMENT - Device not healthy -SMART- on db2056 is CRITICAL: cluster=mysql device=cciss,0 instance=db2056:9100 job=node site=codfw Marostegui T198048 - The acknowledgement expires at: 2018-07-06 05:49:51. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db2056&var-datasource=codfw%2520prometheus%252Fops [05:53:36] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418 (10CommunityTechBot) [05:53:38] 10Operations, 10Graphite, 10Patch-For-Review, 10Performance-Team (Radar): Certain graphite data directories should be backed up - https://phabricator.wikimedia.org/T194418 (10CommunityTechBot) [06:11:06] 10Operations, 10ops-eqiad: anaytics1032's BBU is not working correctly - https://phabricator.wikimedia.org/T194234 (10CommunityTechBot) [06:11:13] 10Operations, 10ops-eqiad: anaytics1032's BBU is not working correctly - https://phabricator.wikimedia.org/T194234 (10CommunityTechBot) [06:16:25] 10Operations, 10Discovery, 10Discovery-Search, 10Wikidata, and 4 others: rack/setup/install wdqs10[09|10].eqiad.wmnet - https://phabricator.wikimedia.org/T194184 (10CommunityTechBot) 05Open>03Resolved [06:16:28] 10Operations, 10Discovery, 10Discovery-Search, 10Wikidata, and 4 others: rack/setup/install wdqs10[09|10].eqiad.wmnet - https://phabricator.wikimedia.org/T194184 (10CommunityTechBot) [06:24:56] 10Operations, 10Traffic: Setup a new PKI software as an alternative to the puppet CA for managing services certificates - https://phabricator.wikimedia.org/T194031 (10elukey) p:05High>03Normal a:03Joe [06:29:48] PROBLEM - puppet last run on labvirt1014 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/nova/policy.json],File[/usr/local/bin/apt2xml] [06:30:49] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:56:19] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:00:29] RECOVERY - puppet last run on labvirt1014 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:00:32] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install backup2001 - https://phabricator.wikimedia.org/T196477 (10CommunityTechBot) [07:00:44] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install backup2001 - https://phabricator.wikimedia.org/T196477 (10CommunityTechBot) [07:00:53] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install backup2001 - https://phabricator.wikimedia.org/T196477 (10CommunityTechBot) a:03Papaul [07:00:56] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install backup2001 - https://phabricator.wikimedia.org/T196477 (10CommunityTechBot) p:05High>03Normal [07:00:58] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install backup2001 - https://phabricator.wikimedia.org/T196477 (10CommunityTechBot) [07:20:05] (03CR) 10Volans: [V: 032 C: 032] Improve validation on host package updates [software/debmonitor] - 10https://gerrit.wikimedia.org/r/442876 (https://phabricator.wikimedia.org/T191299) (owner: 10Volans) [07:21:05] (03CR) 10Volans: [V: 032 C: 032] Add custom Django management command [software/debmonitor] - 10https://gerrit.wikimedia.org/r/442901 (owner: 10Volans) [07:24:04] (03PS2) 10Muehlenhoff: Add library hint for xerces-c [puppet] - 10https://gerrit.wikimedia.org/r/443078 [07:26:05] (03PS1) 10Jcrespo: mariadb: Depool es1018 for upgrade to stretch [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443359 [07:28:11] (03CR) 10Muehlenhoff: [C: 032] Add library hint for xerces-c [puppet] - 10https://gerrit.wikimedia.org/r/443078 (owner: 10Muehlenhoff) [07:32:23] (03Abandoned) 10Elukey: Move the jmxtrans submodule to operations/puppet [puppet] - 10https://gerrit.wikimedia.org/r/437951 (https://phabricator.wikimedia.org/T188377) (owner: 10Elukey) [07:32:36] (03PS1) 10Jcrespo: mariadb: Prepare es1018 for reimage to stretch [puppet] - 10https://gerrit.wikimedia.org/r/443360 [07:32:38] (03Abandoned) 10Elukey: Move the kafkatee submodule to operations/puppet [puppet] - 10https://gerrit.wikimedia.org/r/437950 (https://phabricator.wikimedia.org/T188377) (owner: 10Elukey) [07:32:45] (03Abandoned) 10Elukey: Move the varnishkafka submodule to operations/puppet [puppet] - 10https://gerrit.wikimedia.org/r/437467 (https://phabricator.wikimedia.org/T188377) (owner: 10Elukey) [07:33:16] (03PS2) 10Jcrespo: mariadb: Prepare es1018 for reimage to stretch [puppet] - 10https://gerrit.wikimedia.org/r/443360 [07:33:49] (03PS3) 10Elukey: profile::cache::kafka::alerts: move per host vk alarms to aggregates [puppet] - 10https://gerrit.wikimedia.org/r/443086 (https://phabricator.wikimedia.org/T173492) [07:34:13] (03CR) 10Jcrespo: [C: 032] mariadb: Depool es1018 for upgrade to stretch [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443359 (owner: 10Jcrespo) [07:34:17] sorry in advance for the CR spam (and related -1s ;) ) [07:34:31] (03PS1) 10Volans: Refactor client authentication [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443361 (https://phabricator.wikimedia.org/T198526) [07:34:33] (03PS1) 10Volans: Allow to delete hosts [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443362 (https://phabricator.wikimedia.org/T198526) [07:34:35] (03PS1) 10Volans: Allow to link hosts to external resources [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443363 (https://phabricator.wikimedia.org/T198590) [07:34:37] (03CR) 10Jcrespo: [C: 032] mariadb: Prepare es1018 for reimage to stretch [puppet] - 10https://gerrit.wikimedia.org/r/443360 (owner: 10Jcrespo) [07:34:39] (03PS1) 10Volans: DataTables: save state for the session [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443364 (https://phabricator.wikimedia.org/T198591) [07:34:41] (03PS1) 10Volans: DataTables: cleanup initialization [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443365 (https://phabricator.wikimedia.org/T198591) [07:34:43] (03PS1) 10Volans: Kernels refactor [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443366 (https://phabricator.wikimedia.org/T198592) [07:34:45] (03PS1) 10Volans: DataTables: refactor column grouping [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443367 (https://phabricator.wikimedia.org/T198592) [07:34:47] (03PS1) 10Volans: Add search capability [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443368 (https://phabricator.wikimedia.org/T198592) [07:35:27] (03Merged) 10jenkins-bot: mariadb: Depool es1018 for upgrade to stretch [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443359 (owner: 10Jcrespo) [07:35:31] (03CR) 10jerkins-bot: [V: 04-1] Refactor client authentication [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443361 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [07:35:47] (03CR) 10jerkins-bot: [V: 04-1] DataTables: save state for the session [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443364 (https://phabricator.wikimedia.org/T198591) (owner: 10Volans) [07:35:49] (03CR) 10jerkins-bot: [V: 04-1] Allow to link hosts to external resources [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443363 (https://phabricator.wikimedia.org/T198590) (owner: 10Volans) [07:35:51] (03CR) 10jerkins-bot: [V: 04-1] Allow to delete hosts [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443362 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [07:35:53] (03CR) 10jerkins-bot: [V: 04-1] DataTables: cleanup initialization [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443365 (https://phabricator.wikimedia.org/T198591) (owner: 10Volans) [07:35:56] (03CR) 10jerkins-bot: [V: 04-1] Kernels refactor [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443366 (https://phabricator.wikimedia.org/T198592) (owner: 10Volans) [07:36:20] (03CR) 10jerkins-bot: [V: 04-1] Add search capability [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443368 (https://phabricator.wikimedia.org/T198592) (owner: 10Volans) [07:36:28] (03CR) 10jerkins-bot: [V: 04-1] DataTables: refactor column grouping [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443367 (https://phabricator.wikimedia.org/T198592) (owner: 10Volans) [07:38:19] (03CR) 10jenkins-bot: mariadb: Depool es1018 for upgrade to stretch [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443359 (owner: 10Jcrespo) [07:41:20] (03Abandoned) 10Elukey: Move the nginx submodule to operations/puppet [puppet] - 10https://gerrit.wikimedia.org/r/440080 (https://phabricator.wikimedia.org/T188377) (owner: 10Elukey) [07:50:18] (03PS1) 10Muehlenhoff: Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) [07:50:41] (03PS4) 10Elukey: profile::cache::kafka::alerts: move per host vk alarms to aggregates [puppet] - 10https://gerrit.wikimedia.org/r/443086 (https://phabricator.wikimedia.org/T173492) [07:50:49] (03CR) 10jerkins-bot: [V: 04-1] Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) (owner: 10Muehlenhoff) [07:51:59] (03CR) 10Elukey: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/11624/" [puppet] - 10https://gerrit.wikimedia.org/r/443086 (https://phabricator.wikimedia.org/T173492) (owner: 10Elukey) [07:52:56] (03PS2) 10Muehlenhoff: Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) [08:04:53] !log powercycle maps-test2003 [08:04:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:05:41] !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool es1018 (duration: 00m 50s) [08:05:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:06:31] (03PS1) 10Jonas Kress (WMDE): Add monthly storage schema for graphite [puppet] - 10https://gerrit.wikimedia.org/r/443370 [08:06:59] (03CR) 10jerkins-bot: [V: 04-1] Add monthly storage schema for graphite [puppet] - 10https://gerrit.wikimedia.org/r/443370 (owner: 10Jonas Kress (WMDE)) [08:07:47] (03PS2) 10Jonas Kress (WMDE): Add monthly storage schema for graphite [puppet] - 10https://gerrit.wikimedia.org/r/443370 (https://phabricator.wikimedia.org/T193641) [08:09:13] RECOVERY - Host maps-test2003 is UP: PING OK - Packet loss = 0%, RTA = 30.35 ms [08:09:32] !log stop and reimage es1018 [08:09:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:23:34] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 226, down: 0, dormant: 0, excluded: 0, unused: 0 [08:24:09] !log installing jasper security updates [08:24:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:25:21] (03PS1) 10Volans: debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) [08:25:57] (03CR) 10jerkins-bot: [V: 04-1] debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [08:27:56] (03PS2) 10Volans: debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) [08:32:34] (03PS3) 10Volans: debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) [08:33:30] (03PS1) 10Vgutierrez: Pin dependencies versions [software/certcentral] - 10https://gerrit.wikimedia.org/r/443372 [08:37:19] (03PS1) 10Volans: Updated src and wheels for the GC fix [software/debmonitor/deploy] - 10https://gerrit.wikimedia.org/r/443373 [08:38:18] (03PS1) 10Gehel: wdqs: increase frequency of update lag check [puppet] - 10https://gerrit.wikimedia.org/r/443374 [08:38:20] (03CR) 10Volans: [V: 032 C: 032] Updated src and wheels for the GC fix [software/debmonitor/deploy] - 10https://gerrit.wikimedia.org/r/443373 (owner: 10Volans) [08:46:39] !log volans@deploy1001 Started deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command [08:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:47:03] !log volans@deploy1001 Finished deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command (duration: 00m 24s) [08:47:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:47:54] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443375 [08:48:00] (03PS2) 10Marostegui: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443375 [08:52:13] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443375 (owner: 10Marostegui) [08:53:24] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443375 (owner: 10Marostegui) [08:53:40] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1109" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443375 (owner: 10Marostegui) [08:54:41] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 00m 50s) [08:54:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:59:10] PROBLEM - SSH on ms-be1039 is CRITICAL: Server answer [08:59:57] !log ppchelko@deploy1001 Started deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462 [09:00:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:00:01] T198462: Rethink pacing the cirrusSearchCheckerJob - https://phabricator.wikimedia.org/T198462 [09:00:10] RECOVERY - SSH on ms-be1039 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u2 (protocol 2.0) [09:00:47] !log ppchelko@deploy1001 Finished deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462 (duration: 00m 50s) [09:00:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:04:33] (03PS1) 10Volans: debmonitor: configure external links [puppet] - 10https://gerrit.wikimedia.org/r/443377 (https://phabricator.wikimedia.org/T198590) [09:05:41] (03CR) 10Volans: [C: 04-1] "Requires that Debmonitor change I244ed7c8ad3ca05fa05456d67a870bdb358dfed3 is merged deployed in production." [puppet] - 10https://gerrit.wikimedia.org/r/443377 (https://phabricator.wikimedia.org/T198590) (owner: 10Volans) [09:05:57] (03CR) 10Vgutierrez: [C: 031] "the new view decorator looks good :)" [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443361 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [09:06:19] PROBLEM - SSH on ms-be1039 is CRITICAL: Server answer [09:07:49] (03PS1) 10Marostegui: db-eqiad.php: Depool db1092 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443379 (https://phabricator.wikimedia.org/T191316) [09:08:37] 10Operations, 10ops-esams, 10Traffic: cp3037 is currently unreachable - https://phabricator.wikimedia.org/T196974 (10ema) [09:08:39] (03CR) 10Volans: "Puppetcompiler fails in a strange way, I'll check with Keith:" [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [09:09:52] (03PS2) 10Marostegui: db-eqiad.php: Depool db1087 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443379 (https://phabricator.wikimedia.org/T191316) [09:12:09] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1087 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443379 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [09:12:29] RECOVERY - SSH on ms-be1039 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u2 (protocol 2.0) [09:12:49] PROBLEM - HP RAID on ms-be1039 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [09:13:36] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1087 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443379 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [09:13:51] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1087 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443379 (https://phabricator.wikimedia.org/T191316) (owner: 10Marostegui) [09:15:07] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 00m 51s) [09:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:16:40] anybody looking at ms-be1039.eqiad.wmnet ? seems acting a bit weirdly [09:17:20] !log Stop replication on db1087 to drop triggers on db1124 - T192926 [09:17:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:17:23] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [09:17:25] (checking the console) [09:18:06] elukey: from grafana the load went skyrocketing, iowait too, I bet broken disk [09:18:08] a ton of ** 19 printk messages dropped ** [10180915.048776] sd 0:1:0:1: rejecting I/O to offline device [09:18:16] :) [09:19:43] didn't manage to log in as root to check more though [09:19:50] PROBLEM - SSH on ms-be1039 is CRITICAL: Server answer [09:19:52] the tty keeps being flooded [09:20:24] one option is to depool and kill it :D [09:21:31] is it pooled somewhere or the swift backends simply need to be shutdown when needed? [09:21:32] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS diskspace is low - https://phabricator.wikimedia.org/T196485 (10CommunityTechBot) [09:21:35] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS diskspace is low - https://phabricator.wikimedia.org/T196485 (10CommunityTechBot) [09:21:38] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS diskspace is low - https://phabricator.wikimedia.org/T196485 (10CommunityTechBot) [09:21:49] apergos: pinging :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/440613 [09:22:18] Amir1: will do today. i'm looking at another pending atm [09:22:52] Thank you very much [09:23:09] RECOVERY - SSH on ms-be1039 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u2 (protocol 2.0) [09:24:00] !log Deploy schema change on db1087 with replication, this will generate lag on s8 on labs T191316 T192926 T89737 T195193 [09:24:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:24:06] T89737: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 [09:24:06] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [09:24:06] T195193: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 [09:24:07] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [09:24:42] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add MSantos to `ldap/wmf` - https://phabricator.wikimedia.org/T196943 (10CommunityTechBot) [09:24:47] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add MSantos to `ldap/wmf` - https://phabricator.wikimedia.org/T196943 (10CommunityTechBot) [09:24:52] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add MSantos to `ldap/wmf` - https://phabricator.wikimedia.org/T196943 (10CommunityTechBot) p:05High>03Normal [09:24:56] 10Operations, 10LDAP-Access-Requests, 10Patch-For-Review: Add MSantos to `ldap/wmf` - https://phabricator.wikimedia.org/T196943 (10CommunityTechBot) 05Open>03Resolved a:03CommunityTechBot [09:25:55] (03CR) 10Vgutierrez: [C: 04-1] "IPv6 record missing" [dns] - 10https://gerrit.wikimedia.org/r/442129 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [09:26:37] (03PS1) 10Jcrespo: mariadb: Reenable notifications on es1018 after reimage to stretch [puppet] - 10https://gerrit.wikimedia.org/r/443382 [09:27:20] (03CR) 10Jcrespo: [C: 032] mariadb: Reenable notifications on es1018 after reimage to stretch [puppet] - 10https://gerrit.wikimedia.org/r/443382 (owner: 10Jcrespo) [09:28:09] (03PS1) 10Jcrespo: Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 [09:28:22] (03PS2) 10Jcrespo: Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 [09:29:39] PROBLEM - SSH on ms-be1039 is CRITICAL: Server answer [09:33:48] 10Operations, 10DNS, 10Traffic: rack/setup/install dns200[12].wikimedia.org - https://phabricator.wikimedia.org/T196493 (10ema) [09:34:09] RECOVERY - SSH on ms-be1039 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u2 (protocol 2.0) [09:35:00] PROBLEM - IPMI Sensor Status on ms-be1039 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [09:37:35] !log SET GLOBAL expire_logs_days = 30 on db1067 [09:37:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:37:38] 10Operations, 10ops-codfw, 10Traffic, 10Patch-For-Review: rack/setup/install LVS200[7-10] - https://phabricator.wikimedia.org/T196560 (10ema) [09:37:47] 10Operations, 10Traffic, 10Patch-For-Review: Enable numa_networking on all caches - https://phabricator.wikimedia.org/T193865 (10ema) [09:38:03] 10Operations, 10Performance-Team, 10Traffic, 10Patch-For-Review: Refactor varnishospital and varnishslowlog - https://phabricator.wikimedia.org/T193489 (10ema) [09:39:18] (03CR) 10Jcrespo: [C: 04-1] "Waiting for buffer pool warmup" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 (owner: 10Jcrespo) [09:43:47] (03PS3) 10Jonas Kress (WMDE): Add monthly storage schema for graphite [puppet] - 10https://gerrit.wikimedia.org/r/443370 (https://phabricator.wikimedia.org/T193641) [09:45:58] (03CR) 10ArielGlenn: "On the genrating hosts we don't need to keep 7 of these; 2 or 3 should be plenty, since we really only use the last one for generating the" [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) (owner: 10Smalyshev) [09:47:27] (03PS1) 10Ladsgroup: Set $wgChangeTagsSchemaMigrationStage to write both for frwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443387 (https://phabricator.wikimedia.org/T194165) [09:47:46] (03PS1) 10Gehel: Revert "elasticsearch: enable G1 garbage collector" [puppet] - 10https://gerrit.wikimedia.org/r/443388 (https://phabricator.wikimedia.org/T156137) [09:49:07] (03PS4) 10Gehel: maps: remove "style" parameter [puppet] - 10https://gerrit.wikimedia.org/r/439543 [09:49:45] 10Operations, 10Traffic, 10Patch-For-Review: Enable numa_networking on all caches - https://phabricator.wikimedia.org/T193865 (10MacFan4000) p:05High>03Normal [09:50:04] (03CR) 10Gehel: [C: 032] maps: remove "style" parameter [puppet] - 10https://gerrit.wikimedia.org/r/439543 (owner: 10Gehel) [09:51:58] PROBLEM - Check systemd state on ms-be1024 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [09:53:29] !log reboot ms-be1039 (bad disk, spike in I/O and load, not reachable via ssh or mgmt console) [09:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:53:38] PROBLEM - tilerator on maps-test2004 is CRITICAL: connect to address 10.192.16.35 and port 6534: Connection refused [09:54:08] PROBLEM - tileratorui on maps-test2004 is CRITICAL: connect to address 10.192.16.35 and port 6535: Connection refused [09:54:14] maps-test is me, reverting... [09:54:51] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443389 (https://phabricator.wikimedia.org/T128546) [09:55:15] (03PS1) 10Gehel: Revert "maps: remove "style" parameter" [puppet] - 10https://gerrit.wikimedia.org/r/443390 [09:55:33] (03CR) 10Gehel: [V: 032 C: 032] Revert "maps: remove "style" parameter" [puppet] - 10https://gerrit.wikimedia.org/r/443390 (owner: 10Gehel) [09:56:48] PROBLEM - Host ms-be1039 is DOWN: PING CRITICAL - Packet loss = 100% [09:57:09] 10Operations, 10ops-codfw, 10DNS, 10Traffic, 10netops: switch port configuration for dns200[1-2] - https://phabricator.wikimedia.org/T197697 (10ema) p:05High>03Normal [09:58:18] RECOVERY - Host ms-be1039 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms [09:59:32] 10Operations, 10Security: Disable agent forwarding to important hosts - https://phabricator.wikimedia.org/T198138 (10CommunityTechBot) [09:59:34] 10Operations, 10Security: Disable agent forwarding to important hosts - https://phabricator.wikimedia.org/T198138 (10CommunityTechBot) p:05High>03Triage [10:00:33] (03CR) 10Jdrewniak: [C: 032] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443389 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:01:44] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443389 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:02:00] (03CR) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443389 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:03:31] 10Operations, 10ops-codfw, 10netops: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) a:03Papaul [10:03:34] 10Operations, 10ops-codfw, 10netops: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) p:05High>03Normal [10:03:37] 10Operations, 10ops-codfw, 10netops: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) [10:03:40] 10Operations, 10ops-codfw, 10netops: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) [10:03:43] 10Operations, 10ops-codfw, 10netops: switch port configuration for graphite2003 - https://phabricator.wikimedia.org/T198119 (10CommunityTechBot) 05Open>03Resolved [10:03:59] RECOVERY - HP RAID on ms-be1039 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4 - Controller: OK - Battery/Capacitor: OK [10:04:49] RECOVERY - IPMI Sensor Status on ms-be1039 is OK: Sensor Type(s) Temperature, Power_Supply Status: OK [10:05:28] !log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:443389|Bumping portals to master (T128546)]] (duration: 00m 51s) [10:05:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:05:31] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [10:06:18] !log jdrewniak@deploy1001 Synchronized portals: Wikimedia Portals Update: [[gerrit:443389|Bumping portals to master (T128546)]] (duration: 00m 50s) [10:06:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:13:48] RECOVERY - tileratorui on maps-test2004 is OK: HTTP OK: HTTP/1.1 200 OK - 305 bytes in 0.100 second response time [10:14:19] RECOVERY - tilerator on maps-test2004 is OK: HTTP OK: HTTP/1.1 200 OK - 305 bytes in 0.097 second response time [10:16:53] (03CR) 10ArielGlenn: "One question: I see" [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) (owner: 10Smalyshev) [10:28:59] jouncebot: next [10:28:59] In 0 hour(s) and 31 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1100) [10:29:09] jouncebot: refresh [10:29:10] I refreshed my knowledge about deployments. [10:29:15] jouncebot: next [10:29:15] In 0 hour(s) and 30 minute(s): Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1100) [10:29:18] hmmm [10:29:48] I see [10:40:24] ^^ uh [10:42:31] (03PS2) 10Giuseppe Lavagetto: Sanitize class names for entities [software/conftool] - 10https://gerrit.wikimedia.org/r/442899 [10:42:33] (03PS6) 10Giuseppe Lavagetto: [WIP] Add a WMF-specific tool for managing db config in MediaWiki [software/conftool] - 10https://gerrit.wikimedia.org/r/441396 (https://phabricator.wikimedia.org/T197126) [10:42:35] (03PS1) 10Giuseppe Lavagetto: Drop python 2.7 support [software/conftool] - 10https://gerrit.wikimedia.org/r/443394 [10:48:52] (03CR) 10Muehlenhoff: [C: 031] "Great idea" [software/debmonitor] - 10https://gerrit.wikimedia.org/r/443363 (https://phabricator.wikimedia.org/T198590) (owner: 10Volans) [10:52:01] (03CR) 10Muehlenhoff: [C: 031] debmonitor: configure external links [puppet] - 10https://gerrit.wikimedia.org/r/443377 (https://phabricator.wikimedia.org/T198590) (owner: 10Volans) [10:52:36] Could someone whitelist the WMDE IP address in the Phabrictatr request throtthe [10:52:51] *Phabricator request throttle [10:52:59] I'm getting the error TOO MANY REQUESTS [10:52:59] You ("87.138.110.76, 10.20.0.108, 10.20.0.108, 10.20.0.107") are issuing too many requests too quickly. [10:55:58] andre__: as the phabricator wizard, anything you could do about the above? [10:57:08] Hi everyone, would it be possible to whitelist my work account on phabricator: Charlie_WMDE i was kicked out of phabricator just now for posting two consecutive comments. At least I have no other explanation of what happened. Thanks in advance! [11:00:04] jan_drewniak: Your horoscope predicts another unfortunate Wikimedia Portals Update deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1100). [11:02:04] !log installing libgcrypt20 security updates [11:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:02:41] (03PS1) 10Aklapper: phabricator weekly project changes email: Ignore disabled new assignees [puppet] - 10https://gerrit.wikimedia.org/r/443401 (https://phabricator.wikimedia.org/T195780) [11:06:56] leszek_wmde: Sorry, no, but I filed https://phabricator.wikimedia.org/T198612 [11:07:15] andre__: cool, thanks! [11:09:25] thanks! [11:09:57] (03CR) 10MarcoAurelio: [C: 031] "Given that for late-minute issues I had to cancel this patch for the previous SWAT window, I have scheduled this one for today." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [11:10:30] (03CR) 10Jcrespo: "Will repool after lunch." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 (owner: 10Jcrespo) [11:10:49] 10Operations, 10MediaWiki-Platform-Team, 10HHVM, 10TechCom-RFC (TechCom-Approved), 10User-ArielGlenn: Migrate to PHP 7 in WMF production - https://phabricator.wikimedia.org/T176370 (10aaron) [11:10:53] 10Operations, 10MediaWiki-Platform-Team, 10Performance-Team, 10MW-1.27-release-notes, and 3 others: php-memcached 3.0 (PHP 7) incompatible with BagOStuff - https://phabricator.wikimedia.org/T196125 (10aaron) 05Open>03Resolved [11:22:36] (03PS1) 10Muehlenhoff: library hint for libgcrypt [puppet] - 10https://gerrit.wikimedia.org/r/443404 [11:24:48] hi apergos! are you around? I'm looking for 2017 dumps, do you know where can I find them? [11:25:24] dsaez: we don't keep dumps that far back. they may have been archived at archive.org. may I ask why you want one that's not relatively current? [11:27:28] apergos, I need to compare the information with some experiments we did on 2017, for sure I can build that from history, but I thought that maybe the old dumps were somewhere [11:27:59] if you didn't save the dumps then, your best bet is archve.org. we only have so much storage :-) [11:28:18] makes sense! thanks! [11:28:27] !log repool cp3037 T196974 [11:28:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:28:31] T196974: cp3037 is currently unreachable - https://phabricator.wikimedia.org/T196974 [12:04:32] (03CR) 10Alexandros Kosiaris: [C: 032] lttoolbox: New upstream release [debs/contenttranslation/lttoolbox] - 10https://gerrit.wikimedia.org/r/442798 (https://phabricator.wikimedia.org/T197559) (owner: 10KartikMistry) [12:10:19] !log T197559 upload lttoolbox_3.4.2-2+wmf2 to apt.wikimedia.org/jessie-wikimedia/main [12:10:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:10:22] T197559: Upgrade lttoolbox - https://phabricator.wikimedia.org/T197559 [12:22:07] akosiaris: thanks [12:22:58] kart_: yw [12:23:40] (03CR) 10KartikMistry: "recheck" [debs/contenttranslation/apertium-separable] - 10https://gerrit.wikimedia.org/r/443346 (https://phabricator.wikimedia.org/T191404) (owner: 10KartikMistry) [12:24:11] !log ppchelko@deploy1001 Started deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386 [12:24:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:24:14] T198386: Move static rerender blacklist from RESTBase to ChangeProp - https://phabricator.wikimedia.org/T198386 [12:24:50] !log ppchelko@deploy1001 Finished deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386 (duration: 00m 39s) [12:24:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:31] (03PS1) 10Arturo Borrero Gonzalez: openstack: glance: update bootstrap seed script [puppet] - 10https://gerrit.wikimedia.org/r/443414 (https://phabricator.wikimedia.org/T196633) [12:31:28] (03PS2) 10Arturo Borrero Gonzalez: openstack: glance: update bootstrap seed script [puppet] - 10https://gerrit.wikimedia.org/r/443414 (https://phabricator.wikimedia.org/T196633) [12:36:27] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: glance: update bootstrap seed script [puppet] - 10https://gerrit.wikimedia.org/r/443414 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [12:37:54] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1087" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443415 [12:38:51] (03CR) 10DCausse: [C: 031] Revert "elasticsearch: enable G1 garbage collector" [puppet] - 10https://gerrit.wikimedia.org/r/443388 (https://phabricator.wikimedia.org/T156137) (owner: 10Gehel) [12:39:03] jouncebot: next [12:39:03] In 0 hour(s) and 20 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1300) [12:39:11] (03PS2) 10Gehel: Revert "elasticsearch: enable G1 garbage collector" [puppet] - 10https://gerrit.wikimedia.org/r/443388 (https://phabricator.wikimedia.org/T156137) [12:40:14] (03CR) 10Gehel: [C: 032] Revert "elasticsearch: enable G1 garbage collector" [puppet] - 10https://gerrit.wikimedia.org/r/443388 (https://phabricator.wikimedia.org/T156137) (owner: 10Gehel) [12:42:58] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1087" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443415 (owner: 10Marostegui) [12:43:23] (03PS2) 10ArielGlenn: snapshot: fix css used to show report cards [puppet] - 10https://gerrit.wikimedia.org/r/440613 (owner: 10Ladsgroup) [12:43:59] (03CR) 10ArielGlenn: [C: 032] snapshot: fix css used to show report cards [puppet] - 10https://gerrit.wikimedia.org/r/440613 (owner: 10Ladsgroup) [12:44:14] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1087" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443415 (owner: 10Marostegui) [12:45:26] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 00m 52s) [12:45:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:47:55] (03CR) 10Gehel: [C: 04-1] "Looks good, but we need to wait for the next release of WDQS which will include janino." [puppet] - 10https://gerrit.wikimedia.org/r/443001 (https://phabricator.wikimedia.org/T197645) (owner: 10Smalyshev) [12:48:02] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1087" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443415 (owner: 10Marostegui) [12:48:42] !log ppchelko@deploy1001 Started deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2 [12:48:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:48:46] T198386: Move static rerender blacklist from RESTBase to ChangeProp - https://phabricator.wikimedia.org/T198386 [12:49:06] !log ppchelko@deploy1001 Finished deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2 (duration: 00m 24s) [12:49:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:49:27] !log Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw T191316 T192926 T89737 T195193 [12:49:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:49:33] T89737: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 [12:49:34] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [12:49:34] T195193: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 [12:49:34] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [12:50:46] !log Stop replication on db2077 to remove triggers from db2095 - T192926 [12:50:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:57:55] !log ppchelko@deploy1001 Started deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621 [12:57:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:58:00] T198621: Scap3 doesn't support non us-ascii characters in config templates for service deployments - https://phabricator.wikimedia.org/T198621 [12:58:00] T198386: Move static rerender blacklist from RESTBase to ChangeProp - https://phabricator.wikimedia.org/T198386 [12:59:21] !log ppchelko@deploy1001 Finished deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621 (duration: 01m 26s) [12:59:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Time to snap out of that daydream and deploy European Mid-day SWAT(Max 6 patches). Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1300). [13:00:05] Hauskatze, kart_, and stephanebisson: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [13:00:11] * kart_ around [13:00:23] Hello [13:00:43] 10Operations, 10Discovery, 10Maps, 10Maps-Sprint: migrate maps servers to stretch with the current style - https://phabricator.wikimedia.org/T198622 (10Gehel) [13:02:11] o/ [13:02:31] kart_: stephanebisson: looking at the patches :] [13:03:17] 10Operations, 10Cloud-Services, 10Cloud-VPS, 10IPv6: Enable ipv6 on labs - https://phabricator.wikimedia.org/T37947 (10Cyberpower678) +1. A lot of granted ACC requests may be slipping through because we only see the user's IPv4 when they are actually suffering from an IPv6 block. For all we know, we coul... [13:03:41] kart_: stephanebisson I should have CR+2 ed the patches before :] [13:06:08] hashar: now :) [13:12:12] (03PS1) 10Giuseppe Lavagetto: Purge the last references to jobqueue redis. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443420 (https://phabricator.wikimedia.org/T198220) [13:13:14] (03CR) 10Ppchelko: [C: 031] "I've also grepped the MW sources for any references to these and couldn't find any" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443420 (https://phabricator.wikimedia.org/T198220) (owner: 10Giuseppe Lavagetto) [13:15:51] !log Optimize viwiki.logging on codfw - T197459 [13:15:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:15:55] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [13:16:37] :( [13:17:00] kart_: stephanebisson ok ready to deploy on mwdebug1001 :) [13:17:33] cool [13:18:27] I'm ready [13:20:35] pulling [13:20:44] kart_: stephanebisson patches are now on mwdebug1001 [13:21:07] !log Optimize frwiktionary'.logging on codfw - T197459 [13:21:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:21:11] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [13:21:12] okay. testing. [13:23:25] hashar: works fine [13:24:52] hashar: looks good for my patch too. [13:25:37] (03PS1) 10Elukey: netboot.cfg: temp remove aqs hosts to allow manual work during d-i [puppet] - 10https://gerrit.wikimedia.org/r/443421 (https://phabricator.wikimedia.org/T196138) [13:25:56] good syncing them [13:26:20] (03CR) 10Elukey: [C: 032] netboot.cfg: temp remove aqs hosts to allow manual work during d-i [puppet] - 10https://gerrit.wikimedia.org/r/443421 (https://phabricator.wikimedia.org/T196138) (owner: 10Elukey) [13:28:09] !log hashar@deploy1001 Synchronized php-1.32.0-wmf.10/extensions/ContentTranslation: Allow non-Main namespace source titles iff given in URL - T198178 (duration: 00m 52s) [13:28:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:28:12] T198178: No longer possible to translate from non-mainspace - https://phabricator.wikimedia.org/T198178 [13:29:41] !log hashar@deploy1001 Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/styles/: RCFilters: Fix highlight container selector in Watchlist overrides - T198445 (duration: 00m 51s) [13:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:29:44] T198445: Markers not appearing - https://phabricator.wikimedia.org/T198445 [13:30:39] hashar: Thanks! [13:34:59] !log reimage aqs1004 to Debian Stretch [13:35:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:33] (03PS1) 10Elukey: Set aqs* PXE boot to Debian Stretch [puppet] - 10https://gerrit.wikimedia.org/r/443422 (https://phabricator.wikimedia.org/T196138) [13:37:20] (03CR) 10Elukey: [C: 032] Set aqs* PXE boot to Debian Stretch [puppet] - 10https://gerrit.wikimedia.org/r/443422 (https://phabricator.wikimedia.org/T196138) (owner: 10Elukey) [13:40:23] !log installing libgcrypt11 updates on trusty [13:40:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:45:08] PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most read articles for January 1, 2016) timed out before a response was received: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most-read articles for January 1, 2016 (with aggregated=true)) timed out before a response was received [13:45:18] PROBLEM - restbase endpoints health on restbase-dev1006 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received [13:45:28] PROBLEM - mobileapps endpoints health on scb2004 is CRITICAL: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most read articles for January 1, 2016) timed out before a response was received: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most-read articles for January 1, 2016 (with aggregated=true)) timed out before a response was received [13:45:29] PROBLEM - Restbase LVS codfw on restbase.svc.codfw.wmnet is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received [13:45:48] PROBLEM - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most read articles for January 1, 2016) timed out before a response was received [13:45:48] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/most-read/{year}/{month}/{day} (retrieve the most-read articles for January 1, 2016 (with aggregated=true)) timed out before a response was received [13:45:48] PROBLEM - Restbase LVS eqiad on restbase.svc.eqiad.wmnet is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received [13:45:49] PROBLEM - restbase endpoints health on restbase-dev1004 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received [13:45:49] PROBLEM - restbase endpoints health on restbase-dev1005 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received [13:46:19] RECOVERY - restbase endpoints health on restbase-dev1006 is OK: All endpoints are healthy [13:46:49] RECOVERY - Restbase LVS eqiad on restbase.svc.eqiad.wmnet is OK: All endpoints are healthy [13:46:58] RECOVERY - restbase endpoints health on restbase-dev1005 is OK: All endpoints are healthy [13:49:09] RECOVERY - restbase endpoints health on restbase-dev1004 is OK: All endpoints are healthy [13:49:48] RECOVERY - mobileapps endpoints health on scb2004 is OK: All endpoints are healthy [13:50:08] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy [13:50:39] RECOVERY - mobileapps endpoints health on scb2003 is OK: All endpoints are healthy [13:50:58] RECOVERY - Memory correctable errors -EDAC- on mw1239 is OK: (C)4 ge (W)2 ge 1 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw1239&var-datasource=eqiad%2520prometheus%252Fops [13:52:18] RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy [13:52:19] RECOVERY - Mobileapps LVS eqiad on mobileapps.svc.eqiad.wmnet is OK: All endpoints are healthy [13:53:38] <_joe_> ij, [13:53:41] <_joe_> *uhm [13:53:46] (03CR) 10Hashar: [C: 032] "I don't think that will work, but that is for beta so we can just try live :]" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [13:55:02] (03Merged) 10jenkins-bot: Fix en-rtl in Special:SiteMatrix in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [13:58:27] (03CR) 10jenkins-bot: Fix en-rtl in Special:SiteMatrix in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [13:58:47] (03PS3) 10Jcrespo: Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 [13:58:59] (03CR) 10Jcrespo: [C: 032] Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 (owner: 10Jcrespo) [14:00:04] kart_: #bothumor My software never has bugs. It just develops random features. Rise for (Dry Run) ContentTranslation Draft Purge. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1400). [14:00:18] (03Merged) 10jenkins-bot: Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 (owner: 10Jcrespo) [14:00:58] Yes. [14:02:47] (03CR) 10jenkins-bot: Revert "mariadb: Depool es1018 for upgrade to stretch" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443383 (owner: 10Jcrespo) [14:04:40] (03CR) 10MarcoAurelio: [C: 031] "Well, en-rtl now displays correctly at but we lack a project name." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [14:05:31] hey hashar thanks for doing that beta config merge, traffic was awful :| [14:06:16] hashar: SWAT is over, right? [14:06:59] Hauskatze: ^ [14:07:27] the 'scap update-interwiki-cache' thing is still missing but I'm not sure if he wants to do it [14:07:33] kart_: yes [14:07:50] Hauskatze: I havent run the scap update-interwiki-cache script [14:07:59] I have lost my time looking at SiteMatrix :( [14:08:17] hashar: left some info in the task but don't worry [14:08:19] but yeah, I am notconfident to run update-interwiki-cache :\ [14:08:32] it seems Reedy is the only one who uses it [14:08:41] interwiki rings to me as ""wide scale troubles of some sort will happen"" [14:08:48] ? [14:08:55] It puts a diff up onto gerrit to review [14:09:07] and +2 it [14:09:35] hashar: Thanks. I'll run my maint script's dry-run as per schedule then. [14:09:57] !log Dry run: ContentTranslation Draft Purge [14:09:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:10:47] I'll reschedule the request to update the interwiki cache [14:11:00] thanks [14:11:36] ACKNOWLEDGEMENT - Device not healthy -SMART- on db2052 is CRITICAL: cluster=mysql device=cciss,1 instance=db2052:9100 job=node site=codfw Marostegui T197146 - The acknowledgement expires at: 2018-07-06 14:11:04. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db2052&var-datasource=codfw%2520prometheus%252Fops [14:13:37] (03PS3) 10Muehlenhoff: Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) [14:15:26] (03CR) 10BBlack: [C: 031] varnish::instance: separate VCLs support [puppet] - 10https://gerrit.wikimedia.org/r/440508 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [14:16:16] !log Finished: Dry run: ContentTranslation Draft Purge [14:16:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:16:40] (03CR) 10Muehlenhoff: [C: 031] Move the nginx submodule to environments/production [puppet] - 10https://gerrit.wikimedia.org/r/442242 (owner: 10Elukey) [14:17:00] (03CR) 10Muehlenhoff: [C: 031] Move the nginx module back from environments/production [puppet] - 10https://gerrit.wikimedia.org/r/442244 (owner: 10Elukey) [14:17:25] (03PS1) 10Herron: puppetdb: change hardcoded nginx client cert setting to ca_path param [puppet] - 10https://gerrit.wikimedia.org/r/443423 [14:18:10] (03PS2) 10Addshore: Load WikibaseLexeme on testwiki (again again again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441890 [14:18:20] (03PS8) 10Addshore: Load WikibaseLexeme on all of group0 (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438006 (https://phabricator.wikimedia.org/T197454) [14:18:30] (03PS8) 10Addshore: Load WikibaseLexeme on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436498 (https://phabricator.wikimedia.org/T195615) [14:18:40] (03PS8) 10Addshore: Load WikibaseLexeme on all wikidata clients [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436499 (https://phabricator.wikimedia.org/T195615) [14:19:32] (03PS4) 10Muehlenhoff: Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) [14:20:18] We have created some new Phabricator tickets earlier and we just noticed that they only appear editable to the author even though we have not created them in a manner which differs from our usual procedure, could this be some overzealous new default settings on new Phabricator tickets? [14:20:55] Here's an example: https://phabricator.wikimedia.org/T198605 [14:21:24] We can't find any settings which are wrong with this one, not sure if we're missing something obvious or if something was in fact changed [14:22:06] (03CR) 10Vgutierrez: [C: 031] "as discussed in IRC, IPv6 records can be added automatically :)" [dns] - 10https://gerrit.wikimedia.org/r/442129 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:22:14] (03PS2) 10Herron: puppetdb: change hardcoded nginx client cert setting to ca_path param [puppet] - 10https://gerrit.wikimedia.org/r/443423 [14:23:19] tim_WMDE hi, try now. [14:23:32] i've added you to "Trusted-Contributors" [14:23:39] https://phabricator.wikimedia.org/project/profile/3104/ [14:23:39] !log jynus@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool es1018 (duration: 00m 52s) [14:23:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:23:41] Works, thanks [14:24:05] (03CR) 10Vgutierrez: [C: 032] DNS: ADD DNS entries for authdns2001 [dns] - 10https://gerrit.wikimedia.org/r/442129 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:24:11] (03PS2) 10Vgutierrez: DNS: ADD DNS entries for authdns2001 [dns] - 10https://gerrit.wikimedia.org/r/442129 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:24:20] (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler02/11630/" [puppet] - 10https://gerrit.wikimedia.org/r/443423 (owner: 10Herron) [14:24:31] (03CR) 10Muehlenhoff: [C: 032] Add component/vp9 to video scalers [puppet] - 10https://gerrit.wikimedia.org/r/443369 (https://phabricator.wikimedia.org/T190333) (owner: 10Muehlenhoff) [14:24:59] (03PS6) 10Ema: varnish::instance: separate VCLs support [puppet] - 10https://gerrit.wikimedia.org/r/440508 (https://phabricator.wikimedia.org/T164609) [14:25:41] (03CR) 10Ema: [C: 032] varnish::instance: separate VCLs support [puppet] - 10https://gerrit.wikimedia.org/r/440508 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [14:29:41] !log copy cassandra-tools-wmf 1.0.2-1 from jessie-wikimedia to stretch-wikimedia [14:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:33:22] (03PS1) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [14:33:24] (03PS1) 10Giuseppe Lavagetto: profile::jobrunner: remove references to mediawiki::jobrunner [puppet] - 10https://gerrit.wikimedia.org/r/443427 (https://phabricator.wikimedia.org/T198220) [14:34:35] (03CR) 10Mobrovac: [C: 031] Purge the last references to jobqueue redis. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443420 (https://phabricator.wikimedia.org/T198220) (owner: 10Giuseppe Lavagetto) [14:34:38] (03CR) 10Vgutierrez: [C: 032] DNS: Add mgmt DNS entries for authdns2001 [dns] - 10https://gerrit.wikimedia.org/r/441921 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:34:44] (03PS2) 10Vgutierrez: DNS: Add mgmt DNS entries for authdns2001 [dns] - 10https://gerrit.wikimedia.org/r/441921 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:37:19] (03PS1) 10Papaul: DHCP: Add MAC address entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443428 (https://phabricator.wikimedia.org/T196664) [14:37:46] (03PS2) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [14:37:48] (03PS2) 10Giuseppe Lavagetto: profile::jobrunner: remove references to mediawiki::jobrunner [puppet] - 10https://gerrit.wikimedia.org/r/443427 (https://phabricator.wikimedia.org/T198220) [14:40:03] (03PS1) 10Addshore: BETA: Remove wmgUseWikibaseLexeme from IS-labs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443429 [14:42:30] (03CR) 10Addshore: Wikibase.php shift around the loading of WikibaseLexeme (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436495 (owner: 10Addshore) [14:42:56] kart_: hows your deploy slot going? :) [14:43:16] addshore: over. Go ahead with your deployment. [14:46:19] kart_: thanks [14:46:24] leszek_wmde: ^^ here yet ?:D [14:46:50] (03CR) 10Vgutierrez: [C: 032] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/443428 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:46:56] addshore: yup [14:47:09] (03PS2) 10Vgutierrez: DHCP: Add MAC address entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443428 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:47:22] leszek_wmde: amazing [14:47:39] (03CR) 10Addshore: [C: 032] Load WikibaseLexeme on testwiki (again again again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441890 (owner: 10Addshore) [14:47:46] well, its working on beta so this should be a breeze.... [14:48:29] addshore: you said that [14:48:34] (03PS1) 10Papaul: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196483) [14:48:54] let episode #4 commence, hopefully the last in the season [14:48:56] (03Merged) 10jenkins-bot: Load WikibaseLexeme on testwiki (again again again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441890 (owner: 10Addshore) [14:49:08] (03CR) 10jenkins-bot: Load WikibaseLexeme on testwiki (again again again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441890 (owner: 10Addshore) [14:49:29] leszek_wmde: its on mwdebug1002 [14:50:40] (03CR) 10Vgutierrez: [C: 04-1] "Fix the commit message and the task number" [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196483) (owner: 10Papaul) [14:50:54] leszek_wmde: got some edit ready to test it with? :) [14:51:31] addshore: yup [14:51:35] something's off... [14:51:45] addshore: https://test.wikipedia.org/wiki/Client_test [14:52:13] (03PS3) 10Ema: cache::text: ship cache_misc VCL [puppet] - 10https://gerrit.wikimedia.org/r/440157 (https://phabricator.wikimedia.org/T164609) [14:53:17] >.> leszek_wmde [14:53:39] *looks for a stacktrace* [14:54:49] i dont see anything... [14:55:03] ACKNOWLEDGEMENT - Memory correctable errors -EDAC- on cp1053 is CRITICAL: 1266 ge 4 Ema T165252 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=cp1053&var-datasource=eqiad%2520prometheus%252Fops [14:55:42] (03PS2) 10Papaul: install_server: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196664) [14:56:15] (03CR) 10jerkins-bot: [V: 04-1] install_server: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [14:56:18] leszek_wmde: coming from LanguageAwareRenderer ? [14:58:39] (03PS2) 10Muehlenhoff: library hint for libgcrypt [puppet] - 10https://gerrit.wikimedia.org/r/443404 [14:58:49] addshore: somewhere deeper I think [14:59:04] addshore: doesn't make much sense if asking me. But revert please [14:59:10] yeh, EntityIdValueFormatter::format [14:59:18] (03CR) 10Muehlenhoff: [C: 032] library hint for libgcrypt [puppet] - 10https://gerrit.wikimedia.org/r/443404 (owner: 10Muehlenhoff) [14:59:32] leszek_wmde: this didnt happen on beta though? O_o [14:59:53] addshore: does not look like it. Was just testing on beta [15:00:04] addshore: My dear minions, it's time we take the moon! Just kidding. Time for (Dry Run) ContentTranslation Draft Purge deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1500). [15:00:22] haha, did i not update the name of the deploy window :P silly Cp [15:00:58] *fixed the cal* [15:01:05] addshore: cool. shame does not go on us [15:01:18] leszek_wmde: I think we need some logging in that exception catching so we can actually see what is happening? [15:02:06] (03PS3) 10Papaul: install_server: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196664) [15:03:38] (03PS1) 10Addshore: Revert "Load WikibaseLexeme on testwiki (again again again)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443435 [15:03:47] (03CR) 10Addshore: [C: 032] Revert "Load WikibaseLexeme on testwiki (again again again)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443435 (owner: 10Addshore) [15:04:04] (03CR) 10Addshore: [C: 04-2] "Still not ready apparently..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436498 (https://phabricator.wikimedia.org/T195615) (owner: 10Addshore) [15:04:09] (03CR) 10Addshore: [C: 04-2] "Still not ready apparently..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436499 (https://phabricator.wikimedia.org/T195615) (owner: 10Addshore) [15:04:13] (03CR) 10Addshore: [C: 04-2] "Still not ready apparently..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438006 (https://phabricator.wikimedia.org/T197454) (owner: 10Addshore) [15:04:35] well leszek_wmde looks like the last episode has a surprise twist, and will have a second part [15:04:37] !log restarted apache2 on phab1001: whitelist WMDE [15:04:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:04:54] addshore: yup [15:05:04] (03Merged) 10jenkins-bot: Revert "Load WikibaseLexeme on testwiki (again again again)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443435 (owner: 10Addshore) [15:05:09] addshore: that's what good series are about [15:05:30] (03CR) 10Vgutierrez: [C: 032] install_server: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [15:05:38] (03PS4) 10Vgutierrez: install_server: Add netboot entry for authdns2001 [puppet] - 10https://gerrit.wikimedia.org/r/443431 (https://phabricator.wikimedia.org/T196664) (owner: 10Papaul) [15:06:00] !log Lexeme deploy window done... [15:06:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:54] !log installing file/libmagic security updates [15:06:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:07:56] (03PS1) 10Addshore: Load WikibaseLexeme on testwiki (again x5(EP5)) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443436 (https://phabricator.wikimedia.org/T197454) [15:08:08] (03CR) 10Addshore: [C: 04-2] "not ready" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443436 (https://phabricator.wikimedia.org/T197454) (owner: 10Addshore) [15:08:18] (03CR) 10jenkins-bot: Revert "Load WikibaseLexeme on testwiki (again again again)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443435 (owner: 10Addshore) [15:08:22] (03PS9) 10Addshore: Load WikibaseLexeme on all of group0 (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438006 (https://phabricator.wikimedia.org/T197454) [15:08:30] (03PS9) 10Addshore: Load WikibaseLexeme on group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436498 (https://phabricator.wikimedia.org/T195615) [15:08:39] (03PS9) 10Addshore: Load WikibaseLexeme on all wikidata clients [mediawiki-config] - 10https://gerrit.wikimedia.org/r/436499 (https://phabricator.wikimedia.org/T195615) [15:08:47] (03PS2) 10Addshore: BETA: Remove wmgUseWikibaseLexeme from IS-labs.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443429 [15:08:48] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install authdns2001.wikimedia.org - https://phabricator.wikimedia.org/T196664 (10Papaul) [15:08:52] (03CR) 10Addshore: [C: 04-2] "not yet" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443429 (owner: 10Addshore) [15:14:28] (03CR) 10jerkins-bot: [V: 04-1] Load WikibaseLexeme on testwiki (again x5(EP5)) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443436 (https://phabricator.wikimedia.org/T197454) (owner: 10Addshore) [15:15:50] (03CR) 10Addshore: [C: 04-2] "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443436 (https://phabricator.wikimedia.org/T197454) (owner: 10Addshore) [15:21:19] (03PS1) 10Vgutierrez: site: add authdns2001 node definition [puppet] - 10https://gerrit.wikimedia.org/r/443442 (https://phabricator.wikimedia.org/T196664) [15:25:33] (03PS1) 10Andrew Bogott: labnodepool1002: turn on wmcs paging [puppet] - 10https://gerrit.wikimedia.org/r/443443 [15:26:46] (03PS2) 10Andrew Bogott: labnodepool1002: turn on wmcs paging [puppet] - 10https://gerrit.wikimedia.org/r/443443 [15:27:26] (03CR) 10Andrew Bogott: [C: 032] labnodepool1002: turn on wmcs paging [puppet] - 10https://gerrit.wikimedia.org/r/443443 (owner: 10Andrew Bogott) [15:30:00] (03PS1) 10Mobrovac: service::node: Expose the MW appservers' host to modules [puppet] - 10https://gerrit.wikimedia.org/r/443444 (https://phabricator.wikimedia.org/T198461) [15:31:44] !log rebooting labnodepool1002 as part of a paging test [15:31:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:33:06] (03CR) 10Mobrovac: [C: 031] "PCC looking good - https://puppet-compiler.wmflabs.org/compiler02/11633/" [puppet] - 10https://gerrit.wikimedia.org/r/443444 (https://phabricator.wikimedia.org/T198461) (owner: 10Mobrovac) [15:33:22] PROBLEM - Host labnodepool1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:34:31] RECOVERY - Host labnodepool1002 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [15:35:33] ^ known alert test [15:36:00] (03PS3) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [15:36:02] (03PS3) 10Giuseppe Lavagetto: profile::jobrunner: remove references to mediawiki::jobrunner [puppet] - 10https://gerrit.wikimedia.org/r/443427 (https://phabricator.wikimedia.org/T198220) [15:37:05] (03CR) 10Volans: [C: 031] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/443423 (owner: 10Herron) [15:38:24] (03PS3) 10Herron: puppetdb: change hardcoded nginx client cert setting to ca_path param [puppet] - 10https://gerrit.wikimedia.org/r/443423 [15:38:26] !log Deploy schema change on dbstore1002:s7 T191316 T192926 T89737 T195193 T197459 [15:38:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:38:33] T89737: Make several mediawiki table fields unsigned ints on wmf databases - https://phabricator.wikimedia.org/T89737 [15:38:34] T192926: Schema change to drop archive.ar_text and archive.ar_flags - https://phabricator.wikimedia.org/T192926 [15:38:34] T195193: Schema change for ct_tag_id field to change_tag - https://phabricator.wikimedia.org/T195193 [15:38:34] T191316: Schema change to make archive.ar_rev_id NOT NULL - https://phabricator.wikimedia.org/T191316 [15:38:34] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [15:39:27] (03CR) 10Herron: [C: 032] puppetdb: change hardcoded nginx client cert setting to ca_path param [puppet] - 10https://gerrit.wikimedia.org/r/443423 (owner: 10Herron) [15:41:49] (03CR) 10Anomie: "> Patch Set 1: Code-Review+1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440543 (https://phabricator.wikimedia.org/T197475) (owner: 10Anomie) [15:42:17] (03PS4) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [15:42:40] (03CR) 10Vgutierrez: [C: 032] site: add authdns2001 node definition [puppet] - 10https://gerrit.wikimedia.org/r/443442 (https://phabricator.wikimedia.org/T196664) (owner: 10Vgutierrez) [15:42:51] (03PS2) 10Vgutierrez: site: add authdns2001 node definition [puppet] - 10https://gerrit.wikimedia.org/r/443442 (https://phabricator.wikimedia.org/T196664) [15:42:53] (03PS1) 10Andrew Bogott: Openstack: page wmcs team for any host failure [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) [15:45:52] (03PS1) 10Andrew Bogott: labnodepool1002: turn off paging [puppet] - 10https://gerrit.wikimedia.org/r/443450 [15:46:13] (03CR) 10Ppchelko: [C: 031] service::node: Expose the MW appservers' host to modules [puppet] - 10https://gerrit.wikimedia.org/r/443444 (https://phabricator.wikimedia.org/T198461) (owner: 10Mobrovac) [15:48:09] (03CR) 10Rush: "I have a suspicion this doesn't do what you want, but puppetcompiler should be able to confirm. If it does then +1" [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) (owner: 10Andrew Bogott) [15:48:11] (03PS5) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [15:50:22] (03CR) 10Herron: "We ran into cert validation issues on the puppet compilers after this change and fixed them with Id985a270140c19b1b8468384952cd38186eff232" [puppet] - 10https://gerrit.wikimedia.org/r/437057 (https://phabricator.wikimedia.org/T194962) (owner: 10Alex Monk) [15:50:59] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/11636/mw1300.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) (owner: 10Giuseppe Lavagetto) [15:51:13] (03PS6) 10Giuseppe Lavagetto: profile::mediawiki::jobrunner: remove the old jobrunner service [puppet] - 10https://gerrit.wikimedia.org/r/443426 (https://phabricator.wikimedia.org/T198220) [15:55:43] (03PS4) 10Volans: debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) [15:55:47] (03PS1) 10Vgutierrez: install_server: Fix authdns2001 FQDN [puppet] - 10https://gerrit.wikimedia.org/r/443454 (https://phabricator.wikimedia.org/T196664) [15:55:55] <_joe_> !log stopping jobrunner, jobchron across all jobrunners T198220 [15:55:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:55:58] T198220: Stop and remove old job runners - https://phabricator.wikimedia.org/T198220 [15:56:00] (03PS2) 10Rush: Openstack: page wmcs team for any host failure [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) (owner: 10Andrew Bogott) [15:56:09] (03CR) 10Vgutierrez: [C: 032] install_server: Fix authdns2001 FQDN [puppet] - 10https://gerrit.wikimedia.org/r/443454 (https://phabricator.wikimedia.org/T196664) (owner: 10Vgutierrez) [15:56:39] (03PS2) 10Vgutierrez: install_server: Fix authdns2001 FQDN [puppet] - 10https://gerrit.wikimedia.org/r/443454 (https://phabricator.wikimedia.org/T196664) [15:57:20] (03CR) 10jerkins-bot: [V: 04-1] debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [15:58:12] <_joe_> elukey, mobrovac https://grafana.wikimedia.org/dashboard/db/redis-jobqueue-elukey?orgId=1&from=now-15m&to=now [16:01:33] \o/ [16:02:20] yaaaaay [16:05:53] 10Operations, 10ops-eqiad, 10CheckUser, 10DC-Ops, and 11 others: 2ucaaaaaaa - https://phabricator.wikimedia.org/T194841 (10Cmjohnson) 05Open>03Invalid [16:10:01] cmjohnson1: ^^ you marked that as invalid before it was restored [16:11:21] sorry [16:25:02] (03CR) 10Anomie: [C: 04-1] "> Need to make sure this new PS doesn't break references $wgTidyConfig in CommonSettings.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [16:30:47] (03PS6) 10Bstorm: wiki replicas - prepare for refactored actor storage [puppet] - 10https://gerrit.wikimedia.org/r/431823 (https://phabricator.wikimedia.org/T188299) [16:45:32] (03PS5) 10MarcoAurelio: Create site striker.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/441817 (https://phabricator.wikimedia.org/T189637) [16:54:04] (03PS2) 10Andrew Bogott: labnodepool1002: turn off paging [puppet] - 10https://gerrit.wikimedia.org/r/443450 [16:54:21] (03PS1) 10Alexandros Kosiaris: docker: Create the configuration before installing the package [puppet] - 10https://gerrit.wikimedia.org/r/443464 [16:57:01] (03PS3) 10Andrew Bogott: labnodepool1002: turn off paging [puppet] - 10https://gerrit.wikimedia.org/r/443450 [16:57:49] (03CR) 10Andrew Bogott: [C: 032] labnodepool1002: turn off paging [puppet] - 10https://gerrit.wikimedia.org/r/443450 (owner: 10Andrew Bogott) [17:00:04] gehel: How many deployers does it take to do Wikidata Query Service weekly deploy deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1700). [17:00:14] jouncebot: o/ [17:00:50] (03PS3) 10Andrew Bogott: Openstack: page wmcs team for host failures [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) [17:04:38] !log gehel@deploy1001 Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only) [17:04:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:05:06] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only) (duration: 00m 28s) [17:05:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:07:26] !log gehel@deploy1001 Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph [17:07:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:08:12] (03PS1) 10Andrew Bogott: Removed obsolete hiera def [puppet] - 10https://gerrit.wikimedia.org/r/443470 [17:08:29] (03PS6) 10Gehel: Enable smater logging for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/443001 (https://phabricator.wikimedia.org/T197645) (owner: 10Smalyshev) [17:11:04] (03CR) 10Andrew Bogott: [C: 032] Removed obsolete hiera def [puppet] - 10https://gerrit.wikimedia.org/r/443470 (owner: 10Andrew Bogott) [17:11:16] (03PS2) 10Alexandros Kosiaris: docker: Create the configuration before installing the package [puppet] - 10https://gerrit.wikimedia.org/r/443464 [17:11:25] (03CR) 10Gehel: [C: 032] Enable smater logging for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/443001 (https://phabricator.wikimedia.org/T197645) (owner: 10Smalyshev) [17:11:55] (03PS7) 10Gehel: Enable smater logging for wdqs [puppet] - 10https://gerrit.wikimedia.org/r/443001 (https://phabricator.wikimedia.org/T197645) (owner: 10Smalyshev) [17:13:01] (03CR) 10Smalyshev: "fortunately, I think once we get daily diffs going for categories, we will not have the disruption that weekly category load is causing an" [puppet] - 10https://gerrit.wikimedia.org/r/443374 (owner: 10Gehel) [17:16:30] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (duration: 09m 04s) [17:16:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:20:03] gehel: https://gerrit.wikimedia.org/r/c/operations/puppet/+/443001 doesn't seem to be merging... [17:20:21] looks like something not working with depends resolution maybe [17:21:01] or something like me taking too long to hit [submit] [17:21:11] oh :) [17:21:23] :/ [17:21:46] fyi, testing on wdqs1004 [17:25:02] looks good to me [17:25:06] SMalyshev: looking good [17:25:27] :) at least we agree :) [17:25:51] gehel: great, let's watch it for a while and see if anything weird happens, but I like logs this way much more [17:26:03] yeah, so far, much easier to read! [17:34:31] (03CR) 10Rush: [C: 031] Openstack: page wmcs team for host failures [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) (owner: 10Andrew Bogott) [17:37:10] (03CR) 10Subramanya Sastry: [C: 04-1] "Okay, I think I still want to separate s/Tidy/RemexHtml/g with cleanup of the tidy config ... so that if there is something we overlook in" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [17:47:40] jouncebot: next [17:47:41] In 0 hour(s) and 12 minute(s): Morning SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1800) [17:51:49] (03PS3) 10Subramanya Sastry: Replace Tidy with RemexHtml everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) [17:58:23] (03PS1) 10C. Scott Ananian: Give a name to en-rtl wiki in Special:SiteMatrix in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443475 (https://phabricator.wikimedia.org/T195675) [18:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: How many deployers does it take to do Morning SWAT (Max 6 patches) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T1800). [18:00:04] Niharika, Hauskatze, tgr, and MatmaRex: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:11] o/ [18:00:11] meow! [18:00:15] I'll SWAT. [18:00:23] I iz ready [18:00:52] musikanimal: mooeypoo: Standby for testing in a moment. [18:01:00] (03CR) 10C. Scott Ananian: "> Well, en-rtl now displays correctly at " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441422 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [18:01:18] Darn they haven't fixed the test failures yet. [18:01:28] (03PS2) 10C. Scott Ananian: Give a name to en-rtl wiki in Special:SiteMatrix in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443475 (https://phabricator.wikimedia.org/T195675) [18:01:55] Niharika: my patch cannot be tested on mwdebug, etc. Just in case. [18:02:37] Hauskatze: I don't see a patch. [18:03:00] Niharika: because scap will create one for us [18:03:31] Hauskatze: Yeah, reading through now. How long do you think it will take? I'll like to schedule it last if it will take a while. [18:03:50] Niharika: very few I think [18:04:12] you just cd to mediawiki-staging and run 'scap update-interwiki-cache' [18:04:23] be sure to have generated an HTTP password for gerrit [18:04:48] I updated the docs today, but let me know if there's something unclear and I may be able to assist [18:05:45] (03PS1) 10Niharika29: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443476 [18:05:47] (03CR) 10Niharika29: [C: 032] Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443476 (owner: 10Niharika29) [18:05:57] also, unless cscott wants to do it himself I'd like to propose https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443475/ as well [18:06:12] Hauskatze: Limit is 6 patches. [18:06:34] do we reached 6 already? idk [18:06:46] Niharika: see, it was easy ;) [18:07:03] Hauskatze: Indeed! Waiting for zuul to merge it now. [18:07:05] i don't have any particular desire to swat myself [18:07:19] (03Merged) 10jenkins-bot: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443476 (owner: 10Niharika29) [18:07:21] i keep myself busy setting parsoid on fire [18:07:26] lol [18:08:06] (03CR) 10jenkins-bot: Updating interwiki cache [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443476 (owner: 10Niharika29) [18:08:16] cscott: in any case, this window is full so we won't be able to do it [18:08:19] Hauskatze: Updating the interwiki cache failed. [18:08:25] oh? [18:08:31] https://www.irccloud.com/pastebin/s7L1R5BL/ [18:08:35] Hauskatze: ^ [18:08:52] damn [18:08:56] Not so easy after all. :P [18:08:57] Reedy: ^^ ? [18:09:09] Do I need to revert the patch now or something? [18:09:24] Niharika: yep, revert, it's not going to do anything [18:09:47] and it'll be erased next scap so better to log this [18:09:56] what was the command? [18:10:05] scap update-interwiki-cache greg-g [18:10:15] Without 'greg-g' [18:10:19] xD [18:10:29] musikanimal: mooeypoo: Test on mwdebug1002 now. [18:10:30] :) [18:10:35] (this is going to stashbot quips now) [18:10:35] okay, looking [18:11:25] mwdebug1002 seems to be going really slow [18:11:27] !log Tried updating interwiki cache but it failed with " 'Namespace' object has no attribute 'force'" [18:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:11:43] musikanimal: Yeah, that happens. :( [18:12:19] (03PS5) 10Volans: debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) [18:13:23] (03PS1) 10Niharika29: Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443478 [18:13:27] (03PS1) 10MarcoAurelio: Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443479 [18:13:42] Niharika: patch appears to work [18:13:47] Hauskatze: no worries, i don't think it's a critical patch at all. [18:13:54] and i just wrote it 10 minutes ago ;) [18:13:59] \o/ [18:14:09] musikanimal: Great! Syncing it out now. [18:14:10] like a fine wine, it needs time to mature and develop its bugginess [18:14:16] (or lack thereof) [18:14:25] Niharika: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443479/ reverts our failed interwiki cache update [18:14:28] (03CR) 10jerkins-bot: [V: 04-1] debmonitor: set up maintenance [puppet] - 10https://gerrit.wikimedia.org/r/443371 (https://phabricator.wikimedia.org/T198526) (owner: 10Volans) [18:14:31] cscott: ... what kind of wine are you drinking O.o [18:14:43] we can wait til the end just in case [18:14:44] Hauskatze: I made the revert patch right before you did. :P [18:15:13] (03Abandoned) 10MarcoAurelio: Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443479 (owner: 10MarcoAurelio) [18:15:26] matter of seconds [18:15:40] (03CR) 10Niharika29: [C: 032] Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443478 (owner: 10Niharika29) [18:16:04] !log niharika29@deploy1001 Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Fix setting of reviewed state based on autopatrol permission T198497 (duration: 00m 52s) [18:16:05] tgr: MatmaRex: You here? [18:16:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:16:07] T198497: Creations by autopatrolled users not marked as autopatrolled in Special:NewPagesFeed - https://phabricator.wikimedia.org/T198497 [18:16:11] woot [18:16:13] yeah [18:16:16] Niharika: o/ [18:16:23] musikanimal: mooeypoo: Shipped. Make sure it works in prod! [18:16:31] okay doing another test [18:16:47] Niharika: https://tools.wmflabs.org/bash/quip/AWRcMR1PrshNCZN1Q3er saved for the posterity :) [18:17:09] all good! thanks Niharika [18:17:22] Hauskatze: :D [18:17:34] (03Merged) 10jenkins-bot: Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443478 (owner: 10Niharika29) [18:17:45] Awesome, thanks musikanimal. I really gotta get you into SWATting stuff now, btw. [18:18:16] (03CR) 10jenkins-bot: Revert "Updating interwiki cache" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443478 (owner: 10Niharika29) [18:18:17] =P [18:19:30] and me [18:19:42] MatmaRex: Should your patches be synced in the order you've put them in? [18:19:45] musikanimal: you and I should get a master class from Niharika [18:19:55] Yep, both of you. [18:20:00] And Hauskatze. :P [18:20:18] Niharika: sounds like we need a full classroom [18:20:40] :D I might charge or that. [18:20:41] * Hauskatze rolls eyes [18:20:42] for* [18:20:54] Niharika: no, it doesn't matter. but, if you want to save time, https://gerrit.wikimedia.org/r/#/c/443471/ and https://gerrit.wikimedia.org/r/#/c/443462/ can be deployed and tested together [18:21:09] they both fix bugs with the rollback feature [18:21:25] MatmaRex: Okay, great. I'll ping you when they're ready for testing. [18:27:55] +1 to musikanimal and mooeypoo SWATing :) [18:30:13] certainly, musikanimal can be a good swatter [18:30:59] Sigh. Zuul is very slow. [18:31:06] *sigh* [18:31:12] have a great swat everyone [18:31:13] * addshore runs [18:33:10] tgr: Your patch is on mwdebug1002. [18:33:15] Kindly test. [18:34:03] (03CR) 10C. Scott Ananian: "> As far as I can tell, the only difference between PS1 and PS2 (with" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [18:34:52] (03CR) 10C. Scott Ananian: "FTR I approve of splitting this into two parts, PS1 which turns on Remex, and a later follow-up to actually remove $wgUseTidy and $wgTidyC" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442142 (https://phabricator.wikimedia.org/T175706) (owner: 10Subramanya Sastry) [18:37:28] tgr: There? [18:37:38] Niharika: yeah [18:37:47] Testing? [18:38:03] thanks y'all, I'll get to that SWATin' sometime soon! [18:38:22] it's supposed to be a performance fix, the numbers are not convincing on mwdebug1002, but maybe it's not a good place for testing that... [18:38:30] did not break anything, at least [18:38:40] Good to sync? [18:38:51] I think it's fine to sync, yeah [18:39:47] mwdebug1002 has had performance issues for the last few weeks when i was testing patches on it [18:41:15] !log niharika29@deploy1001 Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Fix incorrect arguments to prepareContent() call in WikiPage T198483 (duration: 00m 51s) [18:41:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:41:19] T198483: Saving timings up 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 [18:41:21] tgr: Deployed. [18:41:31] thanks Niharika! [18:42:01] MatmaRex: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/443461/ is on mwdebug1002. [18:42:08] (03CR) 10MarcoAurelio: [C: 031] Give a name to en-rtl wiki in Special:SiteMatrix in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443475 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [18:42:23] I liked the older gerrit urls better. [18:43:40] testing [18:45:16] things are taking ages to load [18:45:27] That server has become slow. [18:45:55] looks good though, after it finally loaded [18:46:01] Great. [18:46:18] yeah, i dunno if it's a problem with the specific machine or something? [18:46:53] or some HHVM thing where it takes some time to precompile stuff or warm up its caches or whatever? [18:47:02] it did not use to be slow like this [18:47:11] I could try with the other server for when I swat next. [18:47:34] !log niharika29@deploy1001 Synchronized php-1.32.0-wmf.10/resources/src/: RCFilters: Hide highlight containers when RCFilters is disabled T198440 (duration: 00m 51s) [18:47:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:47:38] T198440: Changes shifted down on Special:RecentChanges and Special:Watchlist - https://phabricator.wikimedia.org/T198440 [18:47:59] MatmaRex: That's deployed. [18:48:06] Waiting for your last two to be merged. [18:54:50] MatmaRex: Both of WikiPage ones are on mwdebug1002. [18:55:20] testing as soon as i manage to edit a page [18:55:31] Thanks. [18:55:38] so about twenty minutes :/ [18:55:42] :( [18:56:47] Niharika: last time i was trying to verify this change on mwdebug1002, on friday, we actually cancelled the deployment because the server was so slow, i was getting timeout errors, and we couldn't be sure they are not caused by the new code [18:57:19] That's annoying. Want me to pull on mwdebug1001? [18:57:22] Can do/ [18:57:39] (03PS4) 10Andrew Bogott: Openstack: page wmcs team for host failures [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) [18:58:10] Niharika: hmm, seems to have gone through this time [18:58:35] and everything actually seems normally fast now [18:58:47] MatmaRex: Good to sync? [18:58:52] so i guess it takes some time to warm up? but it definitely wasn't like that before [18:58:57] Niharika: yeah, everything looks good. thanks [18:59:46] (03CR) 10Andrew Bogott: [C: 032] Openstack: page wmcs team for host failures [puppet] - 10https://gerrit.wikimedia.org/r/443448 (https://phabricator.wikimedia.org/T196252) (owner: 10Andrew Bogott) [19:00:36] !log niharika29@deploy1001 Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Mark rollbacking revision as patrolled T198449; WikiPage: Do not set undid revision ID for rollbacks T190374 (duration: 00m 50s) [19:00:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:00:44] T198449: Rollback from autopatrolled user was marked as not patrolled - https://phabricator.wikimedia.org/T198449 [19:00:45] T190374: Rollback gets both "mw-rollback" and "mw-undo" tags - https://phabricator.wikimedia.org/T190374 [19:00:46] Done! [19:01:11] 10Operations, 10Analytics, 10Cleanup, 10User-Elukey: Archive operations/puppet/jmxtrans repository - https://phabricator.wikimedia.org/T198097 (10hashar) 05Open>03Resolved [19:01:14] 10Operations, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377 (10hashar) [19:01:20] 10Operations, 10Analytics, 10Cleanup: Archive operations/puppet/varnishkafka repository - https://phabricator.wikimedia.org/T197503 (10hashar) 05Open>03Resolved [19:01:23] 10Operations, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377 (10hashar) [19:02:32] thanks Niharika [19:05:48] (03CR) 10Catrope: [C: 031] CORS whitelist chapter wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441096 (https://phabricator.wikimedia.org/T181165) (owner: 10Hagar Shilo) [19:25:08] !log Bringing down Cassandra for hardware testing, restbase2001 - T197477 [19:25:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:25:12] T197477: RESTBase storage capacity planning - https://phabricator.wikimedia.org/T197477 [19:32:14] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [19:37:14] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [19:39:22] (03CR) 10Jforrester: [C: 04-2] "> Patch Set 3:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 (owner: 10Jforrester) [19:39:28] (03PS4) 10Jforrester: Stop loading the MwEmbedSupport extension, part I [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441519 [19:39:30] (03PS4) 10Jforrester: Stop loading the MwEmbedSupport extension, part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 [19:39:32] (03PS4) 10Jforrester: Stop loading the MwEmbedSupport extension, part III [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441520 [19:39:34] (03PS4) 10Jforrester: Stop loading the MwEmbedSupport extension, part IV [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441521 [19:48:35] (03CR) 10Alex Monk: [C: 04-1] "See comments on I692226d3 - something is not working in prod like it is in labs..." [puppet] - 10https://gerrit.wikimedia.org/r/437640 (https://phabricator.wikimedia.org/T194962) (owner: 10Alex Monk) [19:49:13] akosiaris, hey [20:00:04] cscott, arlolra, subbu, bearND, halfak, and Amir1: It is that lovely time of the day again! You are hereby commanded to deploy Services – Parsoid / Citoid / Mobileapps / ORES / …. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T2000). [20:00:23] no mobileapps deploy today [20:02:39] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is CRITICAL: Traceback (most recent call last) [20:04:12] 10Operations, 10ops-eqiad, 10fundraising-tech-ops, 10Patch-For-Review: Rack/setup frmon1001 - https://phabricator.wikimedia.org/T186073 (10cwdent) a:03cwdent [20:04:34] 10Operations, 10fundraising-tech-ops, 10netops: NAT and DNS for fundraising monitor host - https://phabricator.wikimedia.org/T198516 (10cwdent) [20:04:41] 10Operations, 10ops-eqiad, 10fundraising-tech-ops, 10Patch-For-Review: Rack/setup frmon1001 - https://phabricator.wikimedia.org/T186073 (10cwdent) [20:06:25] 10Operations, 10ops-eqiad, 10fundraising-tech-ops: Authentication for grafana - https://phabricator.wikimedia.org/T198648 (10cwdent) p:05Triage>03Normal [20:07:39] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw IPv6 is OK: OK - failed 9 probes of 303 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [20:09:16] (03CR) 10Alex Monk: "It looks like the puppetdb-termini code just calls Puppet::Network::HttpPool.http_instance (use_ssl defaulting to true as it only sets the" [puppet] - 10https://gerrit.wikimedia.org/r/437057 (https://phabricator.wikimedia.org/T194962) (owner: 10Alex Monk) [20:18:25] (03PS4) 10Urbanecm: Allow bcts on private&fishbowl wikis advanced privilege manipulation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440000 (https://phabricator.wikimedia.org/T197024) [20:19:31] any ops around who can look into a prod log for me? [20:20:29] (03PS2) 10Urbanecm: New throttle rule for Wikimania 2018 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442268 (https://phabricator.wikimedia.org/T198288) [20:20:35] (03PS2) 10Urbanecm: Enable SandboxLink on eswikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442345 (https://phabricator.wikimedia.org/T198335) [20:20:43] (03PS3) 10Urbanecm: Add namespace alias on pswikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440987 (https://phabricator.wikimedia.org/T197507) [20:21:13] (03PS3) 10Urbanecm: Clean legacy AddGroups/RemoveGroups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440001 (https://phabricator.wikimedia.org/T197024) [20:21:29] (03PS3) 10Urbanecm: Some wikis bureacurats are able to grant non-grantable groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440002 (https://phabricator.wikimedia.org/T197026) [20:22:49] PROBLEM - cassandra-c CQL 10.192.16.164:9042 on restbase2001 is CRITICAL: connect to address 10.192.16.164 and port 9042: Connection refused [20:22:53] doh [20:23:14] ignore ^^^ I missed scheduling enough downtime by *this* much [20:23:50] RECOVERY - cassandra-c CQL 10.192.16.164:9042 on restbase2001 is OK: TCP OK - 0.036 second response time on 10.192.16.164 port 9042 [20:27:32] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10Performance-Team: test.wp is using test2.wp's message cache - https://phabricator.wikimedia.org/T197450 (10Krinkle) [20:46:59] 10Operations, 10Gadgets, 10MediaWiki-Cache, 10Performance-Team: test.wp is using test2.wp's message cache - https://phabricator.wikimedia.org/T197450 (10aaron) Testing around, I see that https://test2.wikipedia.org/wiki/MediaWiki_talk:Msg-override doesn't use te {{msg-override}} message from test.wkipedia.... [20:53:20] !log Bringing down Cassandra for hardware testing, restbase2010 - T197477 [20:53:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:53:23] T197477: RESTBase storage capacity planning - https://phabricator.wikimedia.org/T197477 [21:00:04] bawolff and Reedy: #bothumor I � Unicode. All rise for Weekly Security deployment window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T2100). [21:13:54] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: WDQS diskspace is low - https://phabricator.wikimedia.org/T196485 (10RobH) [21:22:30] 10Operations, 10ops-eqiad, 10Cloud-Services, 10DC-Ops: labsdb1004 and labsdb1005 some hard disks not healthy - https://phabricator.wikimedia.org/T194012 (10Andrew) p:05Triage>03High [21:23:17] 10Operations, 10ops-eqiad, 10Cloud-Services, 10DC-Ops: labsdb1004 and labsdb1005 some hard disks not healthy - https://phabricator.wikimedia.org/T194012 (10chasemp) a:03Cmjohnson [21:27:32] (03CR) 10Krinkle: [C: 031] "Go ahead and deploy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440543 (https://phabricator.wikimedia.org/T197475) (owner: 10Anomie) [21:29:22] (03CR) 10EddieGP: [C: 031] Give a name to en-rtl wiki in Special:SiteMatrix in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443475 (https://phabricator.wikimedia.org/T195675) (owner: 10C. Scott Ananian) [21:34:04] (03CR) 10Krinkle: [C: 031] "LGTM, also verified locally on terbium." [puppet] - 10https://gerrit.wikimedia.org/r/441153 (owner: 10Tim Starling) [21:52:17] (03CR) 10Alex Monk: "And I think the way that works is when Puppet::Network::HTTP.initialize gets a pool it has a Puppet::Network::HTTP::Pool, which uses a Pup" [puppet] - 10https://gerrit.wikimedia.org/r/437057 (https://phabricator.wikimedia.org/T194962) (owner: 10Alex Monk) [23:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: That opportune time is upon us again. Time for a Evening SWAT (Max 6 patches) deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180702T2300). [23:00:04] RoanKattouw: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [23:00:17] I'll SWAT since I'm the only customer [23:00:42] (03PS8) 10Catrope: CORS whitelist chapter wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441096 (https://phabricator.wikimedia.org/T181165) (owner: 10Hagar Shilo) [23:00:48] (03CR) 10Catrope: [C: 032] CORS whitelist chapter wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441096 (https://phabricator.wikimedia.org/T181165) (owner: 10Hagar Shilo) [23:02:20] (03Merged) 10jenkins-bot: CORS whitelist chapter wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441096 (https://phabricator.wikimedia.org/T181165) (owner: 10Hagar Shilo) [23:02:35] (03CR) 10jenkins-bot: CORS whitelist chapter wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441096 (https://phabricator.wikimedia.org/T181165) (owner: 10Hagar Shilo) [23:09:06] !log catrope@deploy1001 Synchronized wmf-config/CommonSettings.php: Add chapter wikis to CORS domain list (T181165) (duration: 00m 52s) [23:09:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:09:09] T181165: In chapter wikis can't see notifications from other wikis on Special:Notifications due to lack of CORS whitelisting - https://phabricator.wikimedia.org/T181165 [23:33:04] !log catrope@deploy1001 Synchronized php-1.32.0-wmf.10/resources/src/: Watchlist performance fixes (T198359, T198399) (duration: 00m 51s) [23:33:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:33:08] T198359: Reduce number of times we apply highlights - https://phabricator.wikimedia.org/T198359 [23:33:09] T198399: Avoid unnecessary calls to updateIfHeightChanged on page load when highlighting is in query params - https://phabricator.wikimedia.org/T198399 [23:50:26] (03PS2) 10Tim Starling: Rewrite sql script to use the new mysql.php wrapper [puppet] - 10https://gerrit.wikimedia.org/r/441153 [23:58:34] (03PS3) 10Tim Starling: Rewrite sql script to use the new mysql.php wrapper [puppet] - 10https://gerrit.wikimedia.org/r/441153 [23:59:43] (03CR) 10Tim Starling: "> We should probably detect if $1 is --write or --group and handle it somehow, if only by giving a more helpful error message about the ch" [puppet] - 10https://gerrit.wikimedia.org/r/441153 (owner: 10Tim Starling)