[01:15:47] PROBLEM - puppet last run on ms-be1017 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [01:28:47] PROBLEM - Check systemd state on ms-be1035 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:02:58] PROBLEM - Check systemd state on ms-be1028 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [02:33:56] !log l10nupdate@deploy1001 scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s) [02:33:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:35:38] PROBLEM - Maps tiles generation on einsteinium is CRITICAL: CRITICAL: 90.07% of data under the critical threshold [5.0] https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [02:44:19] !log l10nupdate@deploy1001 ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s) [02:44:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:54:24] (03PS3) 10Marostegui: mariadb: Enable auto-rehash [puppet] - 10https://gerrit.wikimedia.org/r/444332 (https://phabricator.wikimedia.org/T199009) [04:55:47] (03CR) 10Marostegui: [C: 032] mariadb: Enable auto-rehash [puppet] - 10https://gerrit.wikimedia.org/r/444332 (https://phabricator.wikimedia.org/T199009) (owner: 10Marostegui) [04:56:37] (03PS1) 10Marostegui: db-eqiad.php: Depool db1074 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444485 (https://phabricator.wikimedia.org/T146591) [04:59:35] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1074 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444485 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [05:00:57] RECOVERY - Maps tiles generation on einsteinium is OK: OK: Less than 90.00% under the threshold [10.0] https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1 [05:01:03] (03PS3) 10Marostegui: mariadb/config.pp: Parse final space at prompt [puppet] - 10https://gerrit.wikimedia.org/r/444333 (https://phabricator.wikimedia.org/T199009) [05:01:36] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1074 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444485 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [05:02:10] (03CR) 10Marostegui: [C: 032] mariadb/config.pp: Parse final space at prompt [puppet] - 10https://gerrit.wikimedia.org/r/444333 (https://phabricator.wikimedia.org/T199009) (owner: 10Marostegui) [05:02:20] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1074 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444485 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [05:02:54] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s) [05:02:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:04:18] !log Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459 [05:04:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:04:21] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [05:06:44] 10Operations, 10DBA, 10Patch-For-Review: sql config differs between mwmaint1001 and deploy1001 - https://phabricator.wikimedia.org/T199009 (10Marostegui) 05Open>03Resolved a:03Marostegui This is now fixed after merging both patches. [05:11:57] !log Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379 [05:12:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:12:03] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [05:12:03] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [05:12:04] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:17:07] PROBLEM - Check systemd state on ms-be1028 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [05:22:57] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1074" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 [05:24:08] (03CR) 10jerkins-bot: [V: 04-1] Revert "db-eqiad.php: Depool db1074" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 (owner: 10Marostegui) [05:24:48] (03CR) 10Marostegui: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 (owner: 10Marostegui) [05:26:17] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1074" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 (owner: 10Marostegui) [05:27:28] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1074" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 (owner: 10Marostegui) [05:28:09] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1074" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444488 (owner: 10Marostegui) [05:29:05] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s) [05:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:31:09] !log Deploy schema change on db1090 - T146591 T197891 T196379 [05:31:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:31:14] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [05:31:15] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [05:31:15] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:31:30] !log  Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459 [05:31:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:31:33] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [05:41:25] !log Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521 [05:41:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:41:29] T187521: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521 [05:53:50] !log Deploy schema change on db1076 - T146591 T197891 T196379 [05:53:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:53:56] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [05:53:56] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [05:53:56] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:55:21] !log Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379 [05:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:57:50] * elukey waves to the alter table master marostegui [05:58:05] * marostegui hugs elukey [05:58:08] ahahhaha [06:01:03] !log Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379 [06:01:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:01:09] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:01:09] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:01:10] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:03:46] !log Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521 [06:03:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:03:51] T187521: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521 [06:12:05] !log Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379 [06:12:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:12:10] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:12:11] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:12:11] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:13:28] (03PS1) 10Ema: varnish: load separate VCL files on service startup [puppet] - 10https://gerrit.wikimedia.org/r/444489 (https://phabricator.wikimedia.org/T164609) [06:13:44] !log  Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459 [06:13:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:13:47] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:14:36] (03CR) 10Elukey: "> > Does it also need a cleanup in role::restbase::alerts ?" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [06:17:45] (03CR) 10Ema: [C: 032] varnish: load separate VCL files on service startup [puppet] - 10https://gerrit.wikimedia.org/r/444489 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [06:23:08] (03PS2) 10Elukey: hue: change smtp_host to localhost [puppet] - 10https://gerrit.wikimedia.org/r/441130 (https://phabricator.wikimedia.org/T196920) (owner: 10Herron) [06:24:18] (03CR) 10Elukey: [C: 032] hue: change smtp_host to localhost [puppet] - 10https://gerrit.wikimedia.org/r/441130 (https://phabricator.wikimedia.org/T196920) (owner: 10Herron) [06:25:00] !log restart hue on thorium to pick up new smtp changes - T196920 [06:25:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:25:04] T196920: Add email queueing/failover to services currently using mail_smarthost[0] - https://phabricator.wikimedia.org/T196920 [06:25:11] (03PS7) 10Giuseppe Lavagetto: [WIP] Add a WMF-specific tool for managing db config in MediaWiki [software/conftool] - 10https://gerrit.wikimedia.org/r/441396 (https://phabricator.wikimedia.org/T197126) [06:26:21] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add a WMF-specific tool for managing db config in MediaWiki [software/conftool] - 10https://gerrit.wikimedia.org/r/441396 (https://phabricator.wikimedia.org/T197126) (owner: 10Giuseppe Lavagetto) [06:27:42] (03CR) 10Elukey: [C: 032] oozie: change smtp_host to localhost [puppet] - 10https://gerrit.wikimedia.org/r/441132 (https://phabricator.wikimedia.org/T196920) (owner: 10Herron) [06:28:32] (03CR) 10Elukey: [C: 032] "Cannot merge now but let's coordinate when you do it so I'll restart oozie :)" [puppet] - 10https://gerrit.wikimedia.org/r/441132 (https://phabricator.wikimedia.org/T196920) (owner: 10Herron) [06:28:47] PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/bash_autologout.sh] [06:28:59] (03CR) 10Elukey: [C: 031] Enable base::service_auto_restart for Memcached Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444198 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [06:30:31] !log Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379 [06:30:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:30:36] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:30:37] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:30:37] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:34:35] (03PS11) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [06:35:31] (03CR) 10Ema: [C: 032] reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [06:41:19] !log  Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459 [06:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:41:34] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:49:34] (03PS1) 10Marostegui: db-eqiad.php: Depool db1089 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444490 (https://phabricator.wikimedia.org/T146591) [06:51:14] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1089 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444490 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [06:52:48] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1089 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444490 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [06:53:04] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1089 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444490 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [06:53:54] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s) [06:53:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:54:12] !log Deploy schema change on db1089 - T146591 T197891 T196379 [06:54:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:54:19] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:54:19] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:54:19] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:59:17] RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:01:05] (03CR) 10Chad: "We can remove the logic around smtp_encryption. Also Where's the diff actually setting up exim on local host?" [puppet] - 10https://gerrit.wikimedia.org/r/440970 (https://phabricator.wikimedia.org/T196920) (owner: 10Herron) [07:01:29] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1089" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444491 [07:04:13] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1089" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444491 (owner: 10Marostegui) [07:05:48] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1089" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444491 (owner: 10Marostegui) [07:06:56] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s) [07:06:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:07:50] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1089" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444491 (owner: 10Marostegui) [07:11:10] (03PS1) 10Marostegui: db-eqiad.php: Depool db1119 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444492 (https://phabricator.wikimedia.org/T146591) [07:16:04] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1119 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444492 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:17:43] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1119 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444492 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:17:56] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1119 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444492 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:18:35] !log update filter analytics-in4 on cr1/cr2 eqiad [07:18:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:45] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s) [07:18:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:48] !log Deploy schema change on db1119 - T146591 T197891 T196379 [07:18:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:53] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:18:54] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:18:54] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:21:03] 10Operations, 10Analytics, 10Analytics-Kanban, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) Fixed archiva and removed puppet in analytics-in4. The last step is to drop Ganglia and git-deploy terms from common-infrastructu... [07:21:07] RECOVERY - Check systemd state on ms-be1035 is OK: OK - running: The system is fully operational [07:22:27] RECOVERY - Check systemd state on ms-be1037 is OK: OK - running: The system is fully operational [07:25:40] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1119" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444493 [07:25:48] RECOVERY - Check systemd state on ms-be1028 is OK: OK - running: The system is fully operational [07:29:42] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1119" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444493 (owner: 10Marostegui) [07:30:48] (03PS2) 10TerraCodes: Finish $wmfRealm to $wmgRealm [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444425 (https://phabricator.wikimedia.org/T45956) [07:31:19] 10Operations, 10Analytics, 10Analytics-Kanban, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) a:03elukey [07:31:21] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1119" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444493 (owner: 10Marostegui) [07:31:35] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1119" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444493 (owner: 10Marostegui) [07:32:35] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s) [07:32:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:33:10] (03PS1) 10Marostegui: db-eqiad.php: Depool db1067 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444494 (https://phabricator.wikimedia.org/T146591) [07:34:58] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1067 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444494 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:36:38] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1067 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444494 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:37:43] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s) [07:37:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:21] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1067 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444494 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:40:36] !log reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825 [07:40:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:40] T127825: Re-add intel-microcode - https://phabricator.wikimedia.org/T127825 [07:40:54] !log Deploy schema change on db1067 T146591 T197891 T196379 [07:40:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:40:59] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:40:59] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:41:00] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:44:59] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1067" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444495 [07:47:20] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1067" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444495 (owner: 10Marostegui) [07:48:37] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1067" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444495 (owner: 10Marostegui) [07:49:01] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1067" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444495 (owner: 10Marostegui) [07:49:38] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s) [07:49:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:50:01] (03PS1) 10Marostegui: db-eqiad.php: Depool db1099:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444496 (https://phabricator.wikimedia.org/T146591) [07:50:25] (03CR) 10MarcoAurelio: [C: 031] Don't assign sysop-level privileges to bureaucrats explicitly [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440092 (https://phabricator.wikimedia.org/T197095) (owner: 10Urbanecm) [07:51:50] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1099:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444496 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:52:56] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1099:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444496 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:53:08] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1099:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444496 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:54:04] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s) [07:54:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:54:07] !log Deploy schema change on db1099:3311 T146591 T197891 T196379 [07:54:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:54:13] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:54:13] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:54:14] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:56:42] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1099:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444497 [07:59:05] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1099:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444497 (owner: 10Marostegui) [08:00:47] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1099:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444497 (owner: 10Marostegui) [08:00:59] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1099:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444497 (owner: 10Marostegui) [08:01:50] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s) [08:01:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:02:05] (03PS1) 10Marostegui: db-eqiad.php: Depool db1105:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444498 (https://phabricator.wikimedia.org/T146591) [08:04:44] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1105:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444498 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:06:18] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1105:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444498 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:07:29] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s) [08:07:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:08:26] !log Deploy schema change on db1105:3311 T146591 T197891 T196379 [08:08:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:08:32] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [08:08:32] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [08:08:32] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [08:10:11] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1105:3311 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444498 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:14:32] 10Operations, 10Graphoid, 10Services (watching): Graphoid returns a 400 on MW API time-out - https://phabricator.wikimedia.org/T134237 (10TheDragonFire) This is still occurring, eg. https://en.wikipedia.org/wiki/Template:Graph:PageViews randomly fails: Request: ``` :authority: en.wikipedia.org :method: GET... [08:15:27] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1105:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444499 [08:18:45] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1105:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444499 (owner: 10Marostegui) [08:20:19] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1105:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444499 (owner: 10Marostegui) [08:20:31] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1105:3311" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444499 (owner: 10Marostegui) [08:21:30] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s) [08:21:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:22:05] (03PS1) 10Marostegui: db-eqiad.php: Depool db1106 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444500 (https://phabricator.wikimedia.org/T146591) [08:23:33] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1106 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444500 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:25:07] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1106 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444500 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:25:19] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1106 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444500 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:26:18] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s) [08:26:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:26:28] !log Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379 [08:26:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:26:33] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [08:26:33] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [08:26:34] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [08:29:11] 10Operations, 10ops-eqiad, 10media-storage: Degraded RAID on ms-be1017 - https://phabricator.wikimedia.org/T199063 (10Volans) [08:31:07] !log Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521 [08:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:31:11] T187521: Optimize recentchanges and wbc_entity_usage table across wikis - https://phabricator.wikimedia.org/T187521 [08:32:00] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1106" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444501 [08:33:46] 10Operations, 10Goal: Perform a datacenter switchover - https://phabricator.wikimedia.org/T199073 (10Volans) [08:33:58] (03Abandoned) 10Alexandros Kosiaris: Adds git-lfs package to ores base.pp [puppet] - 10https://gerrit.wikimedia.org/r/432432 (owner: 10Halfak) [08:34:35] 10Operations, 10Goal: Perform a datacenter switchover - https://phabricator.wikimedia.org/T199073 (10Marostegui) [08:34:37] 10Operations, 10DBA, 10Epic: DB meta task for next DC failover issues - https://phabricator.wikimedia.org/T189107 (10Marostegui) [08:35:09] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1106" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444501 (owner: 10Marostegui) [08:35:26] !log updating password for labdb1004/5 for admin users [08:35:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:36:34] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1106" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444501 (owner: 10Marostegui) [08:37:38] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 49s) [08:37:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:38:08] (03PS1) 10Marostegui: db-eqiad.php: Depool db1083 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444502 (https://phabricator.wikimedia.org/T146591) [08:38:15] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1106" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444501 (owner: 10Marostegui) [08:51:08] PROBLEM - MariaDB Slave Lag: s2 on db1125 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 327.00 seconds [08:53:18] RECOVERY - MariaDB Slave Lag: s2 on db1125 is OK: OK slave_sql_lag Replication lag: 0.00 seconds [08:59:45] (03CR) 10Vgutierrez: [C: 032] vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [08:59:56] (03PS6) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [09:02:27] (03PS3) 10Muehlenhoff: deployment-prep: Remove mediawiki06 from dsh groups [puppet] - 10https://gerrit.wikimedia.org/r/443767 (https://phabricator.wikimedia.org/T192996) (owner: 10Krinkle) [09:04:27] (03CR) 10Muehlenhoff: [C: 032] deployment-prep: Remove mediawiki06 from dsh groups [puppet] - 10https://gerrit.wikimedia.org/r/443767 (https://phabricator.wikimedia.org/T192996) (owner: 10Krinkle) [09:06:05] 10Operations, 10Beta-Cluster-Infrastructure, 10Security-Team, 10Patch-For-Review: Delete deployment-mediawiki06 - https://phabricator.wikimedia.org/T192996 (10MoritzMuehlenhoff) p:05Triage>03Normal I've merge the patch to remove it from dsh, the instance can be removed any time now. [09:07:27] !log ppchelko@deploy1001 Started deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 [09:07:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:07:30] T190689: FY17/18 Q4 Program 7 Services Goal: Language variants support - https://phabricator.wikimedia.org/T190689 [09:08:23] 10Operations, 10Internet-Archive, 10Wikimedia-Mailing-lists: Consider allowing mailing lists to be indexed by archive.org - https://phabricator.wikimedia.org/T193573 (10MoritzMuehlenhoff) p:05Triage>03Normal [09:09:35] (03CR) 10Ema: [C: 031] Remove cp3048 from site.pp/DHCP config [puppet] - 10https://gerrit.wikimedia.org/r/444171 (https://phabricator.wikimedia.org/T190607) (owner: 10Muehlenhoff) [09:22:58] !log ppchelko@deploy1001 Finished deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 (duration: 15m 32s) [09:23:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:23:02] T190689: FY17/18 Q4 Program 7 Services Goal: Language variants support - https://phabricator.wikimedia.org/T190689 [09:30:44] (03PS2) 10Muehlenhoff: Enable base::service_auto_restart for Memcached Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444198 (https://phabricator.wikimedia.org/T135991) [09:33:16] (03CR) 10Muehlenhoff: [C: 032] Enable base::service_auto_restart for Memcached Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444198 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [09:35:57] 10Operations: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [09:44:17] 10Operations, 10monitoring, 10Patch-For-Review: Netbox: add Icinga check for PostgreSQL - https://phabricator.wikimedia.org/T185504 (10Volans) What is the current status on this? I still see the `UNKNOWN` on `netmon2001` Icinga checks and no check on `netmon1002` for the PostgreSQL process, but there might... [09:44:30] 10Operations: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [09:44:32] 10Operations, 10monitoring, 10Patch-For-Review: Netbox: add Icinga check for PostgreSQL - https://phabricator.wikimedia.org/T185504 (10Volans) [09:45:48] 10Operations, 10netops, 10Patch-For-Review: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144 (10Volans) This is now part of this quarter goals, moving it as child of T199083. [09:46:01] 10Operations: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [09:46:09] (03PS2) 10Muehlenhoff: Enable base::service_auto_restart for mysqld Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444210 (https://phabricator.wikimedia.org/T135991) [09:46:14] 10Operations, 10netops, 10Patch-For-Review: Evaluate NetBox as a Racktables replacement & IPAM - https://phabricator.wikimedia.org/T170144 (10Volans) [09:47:15] (03CR) 10Muehlenhoff: [C: 032] Enable base::service_auto_restart for mysqld Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444210 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [09:49:35] 10Operations: Decommission servermon - https://phabricator.wikimedia.org/T198939 (10MoritzMuehlenhoff) p:05Triage>03Normal [09:50:24] 10Operations, 10ops-esams: Relabel hooft to bast3002 - https://phabricator.wikimedia.org/T198790 (10MoritzMuehlenhoff) p:05Triage>03Normal [09:55:20] (03PS1) 10Ladsgroup: Write to change_tag_def and the new column in change_tag in Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444555 (https://phabricator.wikimedia.org/T194165) [09:57:32] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444556 (https://phabricator.wikimedia.org/T128546) [09:57:43] (03CR) 10jerkins-bot: [V: 04-1] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444556 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [09:58:34] (03Abandoned) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444556 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [09:59:43] 10Operations, 10ops-eqiad: Degraded RAID on labvirt1019 - https://phabricator.wikimedia.org/T198918 (10MoritzMuehlenhoff) 05Open>03declined Duplicate of T196507 [10:01:25] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444558 (https://phabricator.wikimedia.org/T128546) [10:02:47] (03CR) 10Jdrewniak: [C: 032] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444558 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:04:25] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444558 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:06:51] !log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:444558|Bumping portals to master (T128546)]] (duration: 00m 51s) [10:06:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:06:55] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [10:07:42] !log jdrewniak@deploy1001 Synchronized portals: Wikimedia Portals Update: [[gerrit:444558|Bumping portals to master (T128546)]] (duration: 00m 50s) [10:07:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:08:01] (03CR) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444558 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:08:24] (03PS2) 10Arturo Borrero Gonzalez: openstack: eqiad1: add dhcp agents configuration [puppet] - 10https://gerrit.wikimedia.org/r/444224 (https://phabricator.wikimedia.org/T196633) [10:09:48] (03PS2) 10Muehlenhoff: Remove cp3048 from site.pp/DHCP config [puppet] - 10https://gerrit.wikimedia.org/r/444171 (https://phabricator.wikimedia.org/T190607) [10:10:47] (03CR) 10Muehlenhoff: [C: 032] Remove cp3048 from site.pp/DHCP config [puppet] - 10https://gerrit.wikimedia.org/r/444171 (https://phabricator.wikimedia.org/T190607) (owner: 10Muehlenhoff) [10:12:38] !log powercycle ms-be1017 - can't login on ssh/console [10:12:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:13:42] 10Operations, 10ops-esams, 10Traffic, 10Patch-For-Review: cp3048 hardware issues - https://phabricator.wikimedia.org/T190607 (10MoritzMuehlenhoff) [10:14:27] PROBLEM - Host ms-be1017 is DOWN: PING CRITICAL - Packet loss = 100% [10:16:07] RECOVERY - MD RAID on ms-be1017 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [10:16:07] RECOVERY - Host ms-be1017 is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms [10:16:27] RECOVERY - swift-container-updater on ms-be1017 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-updater [10:16:47] RECOVERY - Check systemd state on ms-be1017 is OK: OK - running: The system is fully operational [10:16:48] RECOVERY - Disk space on ms-be1017 is OK: DISK OK [10:20:07] RECOVERY - puppet last run on ms-be1017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:22:55] (03PS1) 10Muehlenhoff: Remove cp3048 prod DNS entries [dns] - 10https://gerrit.wikimedia.org/r/444560 (https://phabricator.wikimedia.org/T190607) [10:24:30] 10Operations, 10ops-esams, 10Traffic, 10Patch-For-Review: cp3048 hardware issues - https://phabricator.wikimedia.org/T190607 (10MoritzMuehlenhoff) [10:25:20] (03CR) 10Arturo Borrero Gonzalez: [C: 032] openstack: eqiad1: add dhcp agents configuration [puppet] - 10https://gerrit.wikimedia.org/r/444224 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [10:25:33] (03PS3) 10Arturo Borrero Gonzalez: openstack: eqiad1: add dhcp agents configuration [puppet] - 10https://gerrit.wikimedia.org/r/444224 (https://phabricator.wikimedia.org/T196633) [10:31:18] !log upgrade hp raid firmware on ms-be1017 - T141756 [10:31:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:31:22] T141756: audit / test / upgrade hp smartarray P840 firmware - https://phabricator.wikimedia.org/T141756 [10:31:40] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1083 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444502 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:32:23] 10Operations, 10ops-codfw, 10media-storage: audit / test / upgrade hp smartarray P840 firmware - https://phabricator.wikimedia.org/T141756 (10fgiunchedi) [10:33:14] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1083 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444502 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:33:30] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1083 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444502 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:34:24] RECOVERY - Device not healthy -SMART- on ms-be1017 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=ms-be1017&var-datasource=eqiad%2520prometheus%252Fops [10:34:57] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 50s) [10:34:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:35:30] (03CR) 10Arturo Borrero Gonzalez: [C: 032] "Compiler happy:" [puppet] - 10https://gerrit.wikimedia.org/r/444224 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [10:35:53] !log Deploy schema change on db1083 T146591 T197891 T196379 [10:35:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:35:59] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [10:35:59] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [10:35:59] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [10:40:47] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1083" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444563 [10:42:23] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1083" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444563 (owner: 10Marostegui) [10:43:14] PROBLEM - High CPU load on API appserver on mw1227 is CRITICAL: CRITICAL - load average: 48.50, 32.41, 22.02 [10:44:02] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1083" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444563 (owner: 10Marostegui) [10:45:08] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 50s) [10:45:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:45:35] ^s3 seems to have a spike on api queries [10:45:58] (03PS1) 10Marostegui: db-eqiad.php: Depool db1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444564 (https://phabricator.wikimedia.org/T146591) [10:46:14] PROBLEM - High CPU load on API appserver on mw1280 is CRITICAL: CRITICAL - load average: 61.68, 46.34, 29.98 [10:46:32] latency increased 50% [10:46:54] PROBLEM - High CPU load on API appserver on mw1233 is CRITICAL: CRITICAL - load average: 51.22, 35.05, 22.28 [10:46:57] it is going down now [10:47:12] could that be the cause or the consecuence of the above app servers? [10:47:28] api calls cause both [10:47:43] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1083" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444563 (owner: 10Marostegui) [10:47:44] PROBLEM - High CPU load on API appserver on mw1226 is CRITICAL: CRITICAL - load average: 50.62, 35.63, 23.91 [10:48:33] PROBLEM - High CPU load on API appserver on mw1281 is CRITICAL: CRITICAL - load average: 60.78, 37.30, 22.11 [10:48:40] purge latency seems also going up, so it could be writes [10:49:39] it seems tewiki [10:51:14] there is a spike on the master traffic, but it seems kinda usual [10:51:44] PROBLEM - High CPU load on API appserver on mw1232 is CRITICAL: CRITICAL - load average: 57.06, 36.50, 23.24 [10:51:46] I don't think tewiki normally has 50% of s3 traffic [10:51:49] https://grafana.wikimedia.org/dashboard/db/mysql?panelId=7&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1075&var-port=9104&from=1531047097440&to=1531133497440 those tmp tables aren't usual I think [10:52:13] PROBLEM - High CPU load on API appserver on mw1234 is CRITICAL: CRITICAL - load average: 50.08, 36.24, 23.63 [10:52:31] and quite a few api appservers have increased cpu usage [10:53:29] I don't see much edit activity, though on rcs [10:54:29] it ended [10:55:14] RECOVERY - High CPU load on API appserver on mw1227 is OK: OK - load average: 14.77, 22.98, 23.47 [10:55:35] I saw some imports ongoing [10:55:55] who knows, wikis are too fragile [10:57:14] RECOVERY - High CPU load on API appserver on mw1281 is OK: OK - load average: 27.10, 31.89, 26.56 [11:00:05] jan_drewniak: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Wikimedia Portals Update deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1100). [11:00:05] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: How many deployers does it take to do European Mid-day SWAT(Max 6 patches) deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1100). [11:00:05] Urbanecm, Lith, and Amir1: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [11:00:28] ignore my patch [11:00:33] I have to run away [11:01:14] that kinda works anyhow, since you added patch #7 to a 6 patch max window [11:01:58] 10Operations: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [11:02:00] 10Operations: Netbox: setup backups - https://phabricator.wikimedia.org/T190184 (10Volans) [11:02:50] 10Operations: Netbox: setup backups - https://phabricator.wikimedia.org/T190184 (10Volans) As Netbox will be il full production as part of the quarterly goal, this needs to be done before that. [11:02:56] I can SWAT today [11:03:10] Urbanecm, Lith: around for SWAT? [11:03:28] 10Operations, 10Goal: Migrate the hardware inventory from Racktables to Netbox - https://phabricator.wikimedia.org/T199083 (10Volans) [11:06:13] RECOVERY - High CPU load on API appserver on mw1234 is OK: OK - load average: 12.25, 20.14, 23.90 [11:06:23] zeljkof, I'm here [11:06:55] I didn't know SWAT has moved to 11:00 UTC :D [11:07:01] Urbanecm: ok, starting with your patches, double-checking attendance today since swat is at new time :) [11:07:21] Lith: let me know if you are around, your patch will not be deployed if you are not here [11:07:35] ye I'm around [11:09:04] RECOVERY - High CPU load on API appserver on mw1232 is OK: OK - load average: 11.48, 17.77, 23.85 [11:09:20] Lith: your patch can not be deployed as-is [11:09:35] why? [11:09:36] zeljkof, this is IMHO eligible for full sync [11:09:38] there is a rule that a patch should be deployed with one command [11:10:06] Urbanecm: hm, not sure that there will be time for that, how long does it take these days? [11:10:25] hm, how would that work then? [11:10:41] Around 10 minutes zeljkof [11:10:41] b/c I don't really want to spam gerrit with a patch for each file [11:11:03] Lith: does not have to be patch per file, but patch per folder? [11:11:15] ah [11:11:27] that's how I would do it [11:11:47] this patch is also probably a bit to big for a swat, touches many things [11:12:12] before deploying it I would prefer it to have at least one +1 from anybody that has some experience [11:12:17] like other deployers [11:12:47] Maybe the easiest thing would be to deploy this in an extra window [11:12:51] Lith: your patch might be fine as-is, but not for swat, maybe for a separate deployment [11:12:58] Urbanecm: agreed [11:13:18] but please make sure at least a few people take a look at it and there is at least one +1 [11:13:46] Because full sync takes 10 minutes in usual occasions, but it can take up to 45 minutes, which is almost full SWAT. [11:15:22] ah [11:16:06] Urbanecm: reviewing 444378 [11:16:10] also can I put docroot/noc/db.php and w/robots.php into the same patch, or would they need to be two separate patches? [11:16:10] Ack [11:16:25] Lith, if you would ask for separate deployment window, it can stay as is [11:16:34] This is SWAT only rule [11:16:42] Lith: take a look at other patches in this swat, they mostly touch a few lines in a few files, that's the usual format, bigger cleanup should have it's own window [11:17:33] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [11:18:27] ah, I'll see if I can get a window [11:19:01] Lith: that might be the best option [11:20:56] (03PS1) 10Gergő Tisza: Enable TemplateStyles on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444567 (https://phabricator.wikimedia.org/T197603) [11:21:04] RECOVERY - High CPU load on API appserver on mw1280 is OK: OK - load average: 9.76, 12.44, 29.15 [11:21:22] Urbanecm: I'm still getting used to the new gerrit UI, 444378 is marked as WIP? [11:21:33] RECOVERY - High CPU load on API appserver on mw1226 is OK: OK - load average: 11.09, 13.34, 23.37 [11:21:35] and it did not get merged after a +2 [11:21:44] Ergh... [11:21:46] you're right [11:21:48] did you submit it as draft? [11:22:02] I didn't submit it as anything, I'm taking care about patch uploaded by somebody else :-) [11:22:23] Let's skip it, I cannot do anything with this as well... [11:22:37] I hate the WIP thing [11:22:48] zeljkof, ^ [11:22:59] Urbanecm: I can't see in the UI how to un-WIP it :) ok, skipping [11:23:19] Only owner can un-WIP a patch AFAICS [11:23:27] (03CR) 10Zfilipin: "Marked as WIP, refuses to merge after +2." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [11:23:54] (03PS2) 10Zfilipin: Create group eventparticipant on zhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442286 (https://phabricator.wikimedia.org/T198167) (owner: 10Urbanecm) [11:24:04] Urbanecm: reviewing 442286 [11:24:09] ack [11:24:18] (03CR) 10Urbanecm: [C: 031] Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [11:25:13] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442286 (https://phabricator.wikimedia.org/T198167) (owner: 10Urbanecm) [11:26:42] (03Merged) 10jenkins-bot: Create group eventparticipant on zhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442286 (https://phabricator.wikimedia.org/T198167) (owner: 10Urbanecm) [11:27:14] RECOVERY - High CPU load on API appserver on mw1233 is OK: OK - load average: 12.85, 12.68, 24.00 [11:27:55] (03CR) 10jenkins-bot: Create group eventparticipant on zhwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/442286 (https://phabricator.wikimedia.org/T198167) (owner: 10Urbanecm) [11:28:11] Urbanecm: 442286is at mwdebug [11:29:01] ack [11:30:07] Please deploy [11:30:25] Urbanecm: deploying [11:31:27] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:442286|Create group eventparticipant on zhwiki (T198167)]] (duration: 00m 51s) [11:31:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:31:31] T198167: Creating new usergroup "event participant" for zh.wikipedia - https://phabricator.wikimedia.org/T198167 [11:31:35] Urbanecm: deployed [11:31:40] ack [11:32:27] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438085 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:32:41] Urbanecm: 438085 can be deployed directly, right? [11:32:50] or should I push to mwdebug first? [11:33:07] please skip mwdebug zeljkof [11:33:42] !log installing glibc updates from stretch 9.4 point release [11:33:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:33:58] (03Merged) 10jenkins-bot: Upload wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438085 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:35:46] !log zfilipin@deploy1001 Synchronized static/images/mobile/copyright/wikivoyage-wordmark-bn.svg: SWAT: [[gerrit:438085|Upload wordmark for bnwikivoyage (T196680)]] (duration: 00m 50s) [11:35:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:35:50] T196680: Use the correct Bengali Wikivoyage wordmark on mobile site - https://phabricator.wikimedia.org/T196680 [11:36:04] Urbanecm: 438085 deployed [11:36:07] (03Abandoned) 10Aklapper: Phabricator: Block vandalism IP addresses [puppet] - 10https://gerrit.wikimedia.org/r/440510 (owner: 10Aklapper) [11:36:13] ack [11:37:03] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:37:59] (03CR) 10jenkins-bot: Upload wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438085 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:39:23] (03PS2) 10Zfilipin: Use new wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:40:49] 10Operations: Integrate stretch 9.4 point update - https://phabricator.wikimedia.org/T189435 (10MoritzMuehlenhoff) These are fully rolled out: systemd postgresql-9.6 dbus glibc [11:42:32] (03CR) 10Zfilipin: Use new wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:42:39] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:43:11] Urbanecm: sorry for the delay, did not notice 438086 needed a rebase :/ [11:43:24] Will do it [11:44:22] rebased and merging, but it took longer... [11:44:34] (03Merged) 10jenkins-bot: Use new wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:44:35] Ok then :) [11:44:44] Nothing happened, I was eating :D [11:46:06] Urbanecm: 438086 is at mwdebug1002 [11:46:17] working, please deploy zeljkof [11:46:23] Urbanecm: deploying [11:46:27] thx [11:47:54] (03CR) 10jenkins-bot: Use new wordmark for bnwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/438086 (https://phabricator.wikimedia.org/T196680) (owner: 10Urbanecm) [11:48:01] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:438086|Use new wordmark for bnwikivoyage (T196680)]] (duration: 00m 51s) [11:48:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:48:05] T196680: Use the correct Bengali Wikivoyage wordmark on mobile site - https://phabricator.wikimedia.org/T196680 [11:48:32] Urbanecm: 438086 deployed [11:48:35] ack [11:50:27] (03PS5) 10Zfilipin: Add importsources to ru.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444467 (https://phabricator.wikimedia.org/T199045) (owner: 10Bodhisattwa) [11:51:22] !log installing chromium 67.0.3396.87-1~deb9u1 security updates on proton* (tested compatability of new release in deployment-prep) [11:51:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:51:45] (03CR) 10Zfilipin: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444467 (https://phabricator.wikimedia.org/T199045) (owner: 10Bodhisattwa) [11:53:00] (03Merged) 10jenkins-bot: Add importsources to ru.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444467 (https://phabricator.wikimedia.org/T199045) (owner: 10Bodhisattwa) [11:53:12] (03CR) 10jenkins-bot: Add importsources to ru.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444467 (https://phabricator.wikimedia.org/T199045) (owner: 10Bodhisattwa) [11:54:03] Urbanecm: 444467 at mwdebug [11:54:48] Sorry, I'm not a local admin/steward, which is required. Please deploy. [11:54:59] (03PS1) 10Muehlenhoff: Add Cumin alias for proton [puppet] - 10https://gerrit.wikimedia.org/r/444568 [11:55:02] Urbanecm: ok, deploying [11:55:50] (03PS2) 10Arturo Borrero Gonzalez: openstack: bootstrap: neutron: refresh and add more hints [puppet] - 10https://gerrit.wikimedia.org/r/444222 (https://phabricator.wikimedia.org/T196633) [11:56:08] !log zfilipin@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:444467|Add importsources to ru.wikinews (T199045)]] (duration: 00m 50s) [11:56:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:56:12] T199045: Enable transwiki import to ru.wikinews - https://phabricator.wikimedia.org/T199045 [11:56:25] Urbanecm: 444467 deployed [11:56:30] Thanks [11:56:33] !log EU SWAT finished [11:56:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:56:49] Urbanecm: thanks for deploying with #releng! ;) [11:56:53] Yw [11:58:31] (03CR) 10Muehlenhoff: [C: 032] Add Cumin alias for proton [puppet] - 10https://gerrit.wikimedia.org/r/444568 (owner: 10Muehlenhoff) [12:41:35] (03PS1) 10Prtksxna: Remove obsolete $wgPopupsBetaFeature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444574 [12:48:33] (03CR) 10Phuedx: Remove obsolete $wgPopupsBetaFeature (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444574 (owner: 10Prtksxna) [12:56:10] (03CR) 10Phuedx: [C: 04-1] "Per the above." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444574 (owner: 10Prtksxna) [12:59:18] (03PS2) 10Prtksxna: Remove obsolete $wgPopupsBetaFeature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444574 [13:00:03] !log depool codfw for mathoid in preparation for kubernetes cluster update [13:00:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:00:29] (03PS1) 10DCausse: [cirrus] cleanup unused config vars 1/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444576 [13:00:31] (03PS1) 10DCausse: [cirrus] cleanup unused config vars 2/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444577 [13:06:36] (03PS2) 10Marostegui: db-eqiad.php: Depool db1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444564 (https://phabricator.wikimedia.org/T146591) [13:08:27] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444564 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:10:02] !log upgrade acrux to kubernetes 1.8.14 [13:10:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:10:31] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444564 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:10:44] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1114 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444564 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:11:38] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 50s) [13:11:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:42] !log Deploy schema change on db1114 T146591 T197891 T196379 [13:11:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:11:47] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [13:11:47] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [13:11:48] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [13:13:00] 10Operations, 10Cloud-Services: 10G ports seem not to work on new HP hardware - https://phabricator.wikimedia.org/T197169 (10chasemp) 05Open>03Resolved a:03chasemp >>! In T197169#4283053, @faidon wrote: > So for at least labvirt1019 it was indeed about PXE not working (the card worked under Linux) and th... [13:13:03] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1114" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444580 [13:14:29] (03CR) 10Filippo Giunchedi: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [13:15:20] (03CR) 10Filippo Giunchedi: [C: 04-1] "The linked PCC run also changes ferm rules for cassandra-cql removing a bunch of hosts, not sure why but doesn't seem correct?" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [13:16:53] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1114" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444580 (owner: 10Marostegui) [13:18:37] (03PS1) 10Arturo Borrero Gonzalez: cloudvps: reimage and rename labvirt1021 to cloudvirt1001 [puppet] - 10https://gerrit.wikimedia.org/r/444581 (https://phabricator.wikimedia.org/T199107) [13:18:41] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1114" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444580 (owner: 10Marostegui) [13:19:58] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 00m 50s) [13:20:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:20:04] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1114" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444580 (owner: 10Marostegui) [13:21:04] (03PS1) 10Marostegui: db-eqiad.php: Depool db1080 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444582 (https://phabricator.wikimedia.org/T146591) [13:21:50] (03CR) 10Alexandros Kosiaris: [C: 031] "For the disabling of those floppy devices, some investigation shows that we would have to pass" [puppet] - 10https://gerrit.wikimedia.org/r/444238 (owner: 10Muehlenhoff) [13:22:48] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1080 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444582 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:23:34] (03PS1) 10Arturo Borrero Gonzalez: cloudvps: rename labvirt1021.eqiad.wmnet to cloudvirt1001.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/444584 (https://phabricator.wikimedia.org/T199107) [13:24:31] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1080 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444582 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:24:44] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1080 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444582 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [13:25:38] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 00m 50s) [13:25:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:25:58] !log Deploy schema change on db1080 T146591 T197891 T196379 [13:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:26:04] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [13:26:04] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [13:26:04] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [13:26:51] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1080" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444585 [13:29:10] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1080" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444585 (owner: 10Marostegui) [13:29:12] !log upgrade acrab to kubernetes 1.8.14 [13:29:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:29:25] !log upgrade kubernetes200{1,2,3,4} to kubernetes 1.8.14 [13:29:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:30:11] (03PS2) 10DCausse: [cirrus] cleanup unused config vars 1/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444576 [13:30:13] (03PS2) 10DCausse: [cirrus] cleanup unused config vars 2/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444577 [13:30:51] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1080" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444585 (owner: 10Marostegui) [13:31:03] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1080" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444585 (owner: 10Marostegui) [13:32:03] !log Deploy schema change on s4 codfw master (db2051) this will generate lag on s4 codfw T146591 T197891 T196379 [13:32:07] (03PS1) 10Pmiazga: Hygiene: Remove unsued VectorExperimentalPrintStyles [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444586 [13:32:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:32:09] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [13:32:09] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [13:32:09] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [13:32:10] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 50s) [13:32:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:34:31] (03CR) 10Muehlenhoff: "Yeah, I think we can bypass qemu's default devices with such kernel black lists, it's a sensible workaround until qemu provides more fine-" [puppet] - 10https://gerrit.wikimedia.org/r/444238 (owner: 10Muehlenhoff) [13:36:14] PROBLEM - kubelet operational latencies on kubernetes2002 is CRITICAL: instance=kubernetes2002.codfw.wmnet operation_type={create_container,start_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [13:37:04] (03CR) 10Ottomata: [C: 031] profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [13:37:23] RECOVERY - kubelet operational latencies on kubernetes2002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [13:38:14] (03CR) 10Filippo Giunchedi: [C: 04-1] "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [13:42:13] PROBLEM - kubelet operational latencies on kubernetes2003 is CRITICAL: instance=kubernetes2003.codfw.wmnet operation_type={create_container,start_container,stop_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [13:43:32] (03PS2) 10Muehlenhoff: Blacklist floppy driver [puppet] - 10https://gerrit.wikimedia.org/r/444238 [13:45:30] (03PS6) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [13:49:07] (03CR) 10Muehlenhoff: [C: 032] Blacklist floppy driver [puppet] - 10https://gerrit.wikimedia.org/r/444238 (owner: 10Muehlenhoff) [13:53:08] (03CR) 10Elukey: [C: 032] profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [13:53:13] RECOVERY - kubelet operational latencies on kubernetes2003 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [13:53:16] (03PS7) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [13:53:46] I have been +2 sniped but can't really complain if the patch merged before mine is "Blacklist floppy driver" [13:53:54] :D [13:54:24] sorry :-) [13:56:52] (03CR) 10Mobrovac: "> The linked PCC run also changes ferm rules for cassandra-cql" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [13:57:36] !log repool codfw mathoid discovery [13:57:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:01:40] (03PS8) 10Giuseppe Lavagetto: Add a WMF-specific tool for managing db config in MediaWiki [software/conftool] - 10https://gerrit.wikimedia.org/r/441396 (https://phabricator.wikimedia.org/T197126) [14:02:47] (03CR) 10jerkins-bot: [V: 04-1] Add a WMF-specific tool for managing db config in MediaWiki [software/conftool] - 10https://gerrit.wikimedia.org/r/441396 (https://phabricator.wikimedia.org/T197126) (owner: 10Giuseppe Lavagetto) [14:03:04] <_joe_> why don't you like my code, jerkins [14:03:17] (03PS1) 10Pmiazga: Hygiene: remove unsued MFForceSecureLogin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444591 [14:04:24] (03CR) 10Filippo Giunchedi: [C: 031] "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [14:04:31] <_joe_> ah pretty obvious [14:08:24] (03PS1) 10Pmiazga: Hygiene: remove unused MinervaDownloadIcon [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444592 [14:08:33] PROBLEM - puppet last run on db1108 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:08:45] this is me --^ [14:08:50] (03PS1) 10Elukey: profile::mariadb::misc::el::sanitization: fix cron require [puppet] - 10https://gerrit.wikimedia.org/r/444593 (https://phabricator.wikimedia.org/T198766) [14:09:40] (03CR) 10Elukey: [C: 032] profile::mariadb::misc::el::sanitization: fix cron require [puppet] - 10https://gerrit.wikimedia.org/r/444593 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [14:09:51] (03PS12) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [14:09:53] (03PS12) 10Ema: cache_text: add misc directors and alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [14:09:55] (03PS7) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [14:09:57] (03PS5) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [14:14:53] 10Operations, 10ops-codfw, 10monitoring, 10Patch-For-Review: rack/setup/install graphite2003 - https://phabricator.wikimedia.org/T196483 (10fgiunchedi) [14:15:18] 10Operations, 10ops-eqiad, 10monitoring: rack/setup/install graphite1004 - https://phabricator.wikimedia.org/T196484 (10fgiunchedi) [14:16:27] !log rmmoding floppy kernel module from hosts which had it loaded prior to blacklisting [14:16:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:20:32] cmjohnson1: hi! ms-be1036 is up and running but I can't access its ipmi, can you take a look at what's up ? once that's done I'll repool it [14:22:59] (03PS1) 10Elukey: profile::analytics::refinery::repository: use root instead hdfs for log [puppet] - 10https://gerrit.wikimedia.org/r/444596 (https://phabricator.wikimedia.org/T198766) [14:25:17] !log depool eqiad mathoid discovery RR for kubernetes upgrade [14:25:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:27:07] (03PS2) 10Elukey: profile::analytics::refinery::repository: remove logging config [puppet] - 10https://gerrit.wikimedia.org/r/444596 (https://phabricator.wikimedia.org/T198766) [14:27:58] (03CR) 10Elukey: [C: 032] profile::analytics::refinery::repository: remove logging config [puppet] - 10https://gerrit.wikimedia.org/r/444596 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [14:32:32] (03PS1) 10ArielGlenn: quick script to show runtimes of dump jobs [dumps] - 10https://gerrit.wikimedia.org/r/444603 (https://phabricator.wikimedia.org/T199117) [14:32:56] (03CR) 10jerkins-bot: [V: 04-1] quick script to show runtimes of dump jobs [dumps] - 10https://gerrit.wikimedia.org/r/444603 (https://phabricator.wikimedia.org/T199117) (owner: 10ArielGlenn) [14:33:51] (03PS7) 10Andrew Bogott: deployment-prep logstash: replace deployment-tin reference [puppet] - 10https://gerrit.wikimedia.org/r/438001 (https://phabricator.wikimedia.org/T192071) (owner: 10Dzahn) [14:34:03] RECOVERY - puppet last run on db1108 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [14:34:55] (03CR) 10Andrew Bogott: [C: 032] deployment-prep logstash: replace deployment-tin reference [puppet] - 10https://gerrit.wikimedia.org/r/438001 (https://phabricator.wikimedia.org/T192071) (owner: 10Dzahn) [14:35:32] (03CR) 10Ottomata: [C: 031] profile::analytics::refinery::repository: remove logging config [puppet] - 10https://gerrit.wikimedia.org/r/444596 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [14:36:23] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation=compareAndSwap https://grafana.wikimedia.org/dashboard/db/kubernetes-api [14:37:23] RECOVERY - etcd request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [14:38:06] (03CR) 10Andrew Bogott: prometheus: tools: scrape paws metrics into prometheus (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/441514 (https://phabricator.wikimedia.org/T195030) (owner: 10Chico Venancio) [14:43:05] (03PS3) 10Elukey: Enable snappy compression for Varnishkafka eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) [14:43:37] (03CR) 10Ottomata: [C: 031] Enable snappy compression for Varnishkafka eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) (owner: 10Elukey) [14:49:04] (03PS2) 10ArielGlenn: quick script to show runtimes of dump jobs [dumps] - 10https://gerrit.wikimedia.org/r/444603 (https://phabricator.wikimedia.org/T199117) [14:51:26] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10Papaul) @MoritzMuehlenhoff can you paste the disk log error here. According to HP the ILO log is not showing any bad disk. Thanks. [14:51:43] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10Papaul) Also please see T193394 [14:52:05] (03CR) 10Elukey: [C: 032] Enable snappy compression for Varnishkafka eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) (owner: 10Elukey) [14:52:18] !log upgrade eqiad kubernetes cluster to 1.8.14 [14:52:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:53:01] 10Operations, 10Wikimedia-Apache-configuration, 10Patch-For-Review, 10User-Joe: Re-organize the apache configuration for MediaWiki in puppet - https://phabricator.wikimedia.org/T196968 (10Joe) [14:53:03] 10Operations, 10MediaWiki-Platform-Team, 10HHVM, 10TechCom-RFC (TechCom-Approved), 10User-ArielGlenn: Migrate to PHP 7 in WMF production - https://phabricator.wikimedia.org/T176370 (10Joe) [15:00:34] PROBLEM - kubelet operational latencies on kubernetes1002 is CRITICAL: instance=kubernetes1002.eqiad.wmnet operation_type={create_container,start_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [15:00:40] expected ^ [15:01:43] RECOVERY - kubelet operational latencies on kubernetes1002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [15:02:01] (03PS8) 10Muehlenhoff: webperf: Get graphite_host for coal::processor from Hiera [puppet] - 10https://gerrit.wikimedia.org/r/442900 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [15:03:06] (03CR) 10Muehlenhoff: [C: 032] webperf: Get graphite_host for coal::processor from Hiera [puppet] - 10https://gerrit.wikimedia.org/r/442900 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [15:04:23] PROBLEM - kubelet operational latencies on kubernetes1004 is CRITICAL: instance=kubernetes1004.eqiad.wmnet operation_type={create_container,start_container,stop_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [15:05:24] RECOVERY - kubelet operational latencies on kubernetes1004 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [15:06:43] 10Operations, 10ops-codfw, 10ops-eqiad, 10netops: Audit switch ports/descriptions/enable - https://phabricator.wikimedia.org/T189519 (10Papaul) [15:07:46] 10Operations, 10ops-codfw, 10ops-eqiad, 10netops: Audit switch ports/descriptions/enable - https://phabricator.wikimedia.org/T189519 (10Papaul) @ayounsi osm-web2001, db2021, db2022 and db2024 are not showing in icinga so i don't know what is the update on those servers [15:09:38] !log repool eqiad mathoid discovery RR [15:09:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:13:58] (03PS13) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [15:14:00] (03PS13) 10Ema: cache_text: add misc directors and alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [15:14:02] (03PS8) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [15:14:04] (03PS6) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [15:19:38] 10Operations, 10ops-codfw, 10ops-eqiad, 10netops: Audit switch ports/descriptions/enable - https://phabricator.wikimedia.org/T189519 (10Papaul) a:05Papaul>03Cmjohnson @Cmjohnson hey Chris assigning you the task so you can do your audit. Once done you can assign it to @ayounsi. Thanks. [15:23:03] RECOVERY - Host ms-be1036.mgmt is UP: PING OK - Packet loss = 0%, RTA = 5.95 ms [15:23:12] godog mgmt on ms-be1036 is up [15:23:12] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10Papaul) @joe is this task the same as T193394 ? [15:26:16] 10Operations, 10ops-ulsfo, 10Traffic, 10netops: troubleshoot cr3/cr4 link - https://phabricator.wikimedia.org/T196030 (10ayounsi) Did the loop test on cr3-ulsfo but still no stable link. Update from JTAC: > I checked in my lab, I am able to bring up this link, however I couldn’t find Fiberstore optics in... [15:28:53] RECOVERY - HP RAID on ms-be1028 is OK: OK: Slot 3: OK: 2I:4:1, 2I:4:2, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4 - Controller: OK - Battery/Capacitor: OK [15:31:04] (03PS3) 10Tacsipacsi: Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) [15:32:05] !log enabled snappy compression for varnishkafka eventlogging [15:32:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:33:15] 10Operations, 10ops-codfw, 10ops-eqiad, 10netops: Audit switch ports/descriptions/enable - https://phabricator.wikimedia.org/T189519 (10ayounsi) [15:33:28] 10Operations, 10ops-codfw, 10ops-eqiad, 10netops: Audit switch ports/descriptions/enable - https://phabricator.wikimedia.org/T189519 (10ayounsi) Thanks, switch port description updated. [15:35:11] cmjohnson1: sweet! thanks [16:02:29] 10Operations, 10ops-eqsin, 10Traffic: cp5006 unresponsive - https://phabricator.wikimedia.org/T187157 (10BBlack) a:03RobH [16:03:38] (03PS1) 10EBernhardson: [WIP] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [16:04:20] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [16:06:36] (03PS1) 10Rush: openstack: add neutron 172 IP designations for network data.yaml [puppet] - 10https://gerrit.wikimedia.org/r/444611 (https://phabricator.wikimedia.org/T184209) [16:09:21] !log Set disk 32:0 offline on db1069 for a replacement - T199056 [16:09:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:09:25] T199056: db1069 bad disk - https://phabricator.wikimedia.org/T199056 [16:10:19] 10Operations, 10ops-eqiad, 10DBA: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) @Cmjohnson you can now proceed, I have set the disk offline: ``` Adapter #0 Enclosure Device ID: 32 Slot Number: 0 Drive's position: DiskGroup: 0, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 0 W... [16:11:35] 10Operations, 10Puppet, 10DBA: Remove all usages of $::mw_primary on puppet - https://phabricator.wikimedia.org/T199124 (10jcrespo) [16:15:45] (03CR) 10Jdlrobson: [C: 031] Remove obsolete $wgPopupsBetaFeature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444574 (owner: 10Prtksxna) [16:33:13] PROBLEM - MegaRAID on db1069 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [16:33:14] ACKNOWLEDGEMENT - MegaRAID on db1069 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T199127 [16:33:30] 10Operations, 10ops-eqiad: Degraded RAID on db1069 - https://phabricator.wikimedia.org/T199127 (10ops-monitoring-bot) [16:33:32] 10Operations, 10ops-eqiad, 10DBA: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) disk swapped by chris: ``` root@db1069:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -a0 Rebuild Progress on Device at Enclosure 32, Slot 0 Completed 28% in 16 Minutes. ``` [16:33:51] 10Operations, 10ops-eqiad: Degraded RAID on db1069 - https://phabricator.wikimedia.org/T199127 (10Marostegui) [16:33:53] 10Operations, 10ops-eqiad, 10DBA: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) [16:34:06] 10Operations, 10ops-eqiad, 10DBA: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) disk swapped by chris: ``` root@db1069:~# megacli -PDRbld -ShowProg -PhysDrv [32:0] -a0 Rebuild Progress on Device at Enclosure 32, Slot 0 Completed 28% in 16 Minutes. ``` [16:34:41] (03PS2) 10Arturo Borrero Gonzalez: cloudvps: rename labvirt1021.eqiad.wmnet to cloudvirt1001.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/444584 (https://phabricator.wikimedia.org/T199107) [16:37:05] (03PS2) 10Arturo Borrero Gonzalez: cloudvps: reimage and rename labvirt1021 to cloudvirt1001 [puppet] - 10https://gerrit.wikimedia.org/r/444581 (https://phabricator.wikimedia.org/T199107) [16:44:45] (03PS3) 10Arturo Borrero Gonzalez: cloudvps: reimage and rename labvirt1021 to cloudvirt1021 [puppet] - 10https://gerrit.wikimedia.org/r/444581 (https://phabricator.wikimedia.org/T199107) [16:45:53] 10Operations, 10Cloud-VPS, 10procurement: rack/setup/install labvirt102[34] - https://phabricator.wikimedia.org/T199125 (10chasemp) wait, can we make these cloudvirt1023 and cloudvirt1024? I think @aborrero is getting into the naming adjustments now. [16:47:44] (03PS1) 10Alexandros Kosiaris: grafana-admin: Redirecto to grafana.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/444622 [16:48:04] (03PS1) 10Andrew Bogott: pdns recursor: update allow_from [puppet] - 10https://gerrit.wikimedia.org/r/444623 (https://phabricator.wikimedia.org/T199123) [16:48:30] (03CR) 10Rush: [C: 04-1] "code looks good to me (rob is a better approver) but commit message doesn't match" [dns] - 10https://gerrit.wikimedia.org/r/444584 (https://phabricator.wikimedia.org/T199107) (owner: 10Arturo Borrero Gonzalez) [16:49:05] (03CR) 10Rush: [C: 031] cloudvps: reimage and rename labvirt1021 to cloudvirt1021 [puppet] - 10https://gerrit.wikimedia.org/r/444581 (https://phabricator.wikimedia.org/T199107) (owner: 10Arturo Borrero Gonzalez) [16:51:28] 10Operations, 10Cloud-VPS, 10procurement: rack/setup/install labvirt102[34] - https://phabricator.wikimedia.org/T199125 (10RobH) >>! In T199125#4408761, @chasemp wrote: > wait, can we make these cloudvirt1023 and cloudvirt1024? I think @aborrero is getting into the naming adjustments now. This seems 100% a... [16:52:18] 10Operations, 10Cloud-VPS, 10procurement: rack/setup/install cloudvirt102[34] - https://phabricator.wikimedia.org/T199125 (10RobH) [16:52:31] 10Operations: Decommission servermon - https://phabricator.wikimedia.org/T198939 (10akosiaris) FWIW, servermon allows searching for specific facts (for specific hosts as well) under the `fact query` menu allowing to relatively easily slice information present in puppetdb (actually in mysql activerecord db, but t... [16:55:16] (03PS2) 10Alexandros Kosiaris: grafana-admin: Redirecto to grafana.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/444622 [16:56:18] (03PS3) 10Alexandros Kosiaris: grafana-admin: Redirect to to grafana.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/444622 [16:58:29] (03PS3) 10Arturo Borrero Gonzalez: cloudvps: rename labvirt1021.eqiad.wmnet to cloudvirt1021.eqiad.wmnet [dns] - 10https://gerrit.wikimedia.org/r/444584 (https://phabricator.wikimedia.org/T199107) [17:00:05] gehel: (Dis)respected human, time to deploy Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1700). Please do the needful. [17:01:12] (03PS2) 10Gehel: Enable kafka poller on test hosts [puppet] - 10https://gerrit.wikimedia.org/r/444265 (owner: 10Smalyshev) [17:01:14] (03PS1) 10Gehel: wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 [17:01:33] elukey: ^ if you have a minute to check those 2 patches [17:01:40] jouncebot: o/ [17:01:45] gehel: we need to deploy the new Updater code before enabling it (just in case :) [17:01:54] SMalyshev: yep [17:02:20] gehel: otherwise, this is what should happen: https://puppet-compiler.wmflabs.org/compiler02/11718/wdqs1009.eqiad.wmnet/ [17:03:05] SMalyshev: also, after discussion with elukey, we should probably use kafka main instead of jumbo (the second patch above) [17:03:21] if I remember correctly, that was the plan all along [17:03:36] (03PS1) 10Vgutierrez: Make pylint match flake8 line length limit [software/certcentral] - 10https://gerrit.wikimedia.org/r/444629 [17:04:03] gehel: ok, then I'd propose to do it in the order of that patch first, then I'll manually check it works (i.e. run compiler on my patch again with new config and run the config manually) [17:04:11] and then we can merge the other patch [17:04:20] because I never manually tested with main [17:04:29] it shouldn't matter, but.... [17:04:35] you mean first enabling on jumbo and then switch to main? Ok, will rebase the patch [17:04:57] gehel: no, the reverse [17:05:18] first merging the main patch, then see which config it produces, test this config manually and only then enable [17:05:31] (03PS1) 10Vgutierrez: [WIP] get rid of openssl CLI usage [software/certcentral] - 10https://gerrit.wikimedia.org/r/444631 [17:05:43] Oh, ok, then that's all good, the patches are already in that order [17:05:48] gehel: wdqs is only in eqiad? [17:05:50] ah, ok, great [17:05:52] (super ignoran about it) [17:06:15] elukey: no, there's eqiad and codfw parts for wdqs in general, but *test* hosts are only in eqiad [17:06:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP] get rid of openssl CLI usage [software/certcentral] - 10https://gerrit.wikimedia.org/r/444631 (owner: 10Vgutierrez) [17:06:30] elukey: we are enabling only on test now, if it works we'll enable on others too [17:06:33] RECOVERY - Device not healthy -SMART- on db1069 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1069&var-datasource=eqiad%2520prometheus%252Fops [17:06:38] elukey: silly me, I just did a replace without actually reading... [17:06:42] ah ok because https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444627/1/modules/profile/manifests/wdqs.pp mentions main-eqiad [17:06:48] okok :) [17:07:02] (03CR) 10Vgutierrez: [C: 04-1] "WIP, feedback welcome :)" [software/certcentral] - 10https://gerrit.wikimedia.org/r/444631 (owner: 10Vgutierrez) [17:07:28] elukey: would it be ok to use main-eqiad there for codfw hosts too or they need to use main-codfw or something? [17:07:32] so this should be kafka_config("main-${::site}")['brokers']['string'] ? Or do we have a function to get the closest cluster? [17:08:55] * gehel is already starting on deploying new updater, will see about enabling kafka once this is done [17:09:07] what you wrote seems correct to me gehel, but I have no idea atm what the wdqs codfw hosts are doing now.. [17:09:29] in theory, if we switch dc from eqiad to codfw then main-codfw will get the recent changes that you need [17:09:47] our test nodes are only on eqiad, and we're only enabling kafka for test nodes atm [17:10:10] elukey: they are basically the same as eqiad hosts but in codfw :) all hosts pull both sets of topics - codfw.* and eqiad.* ones [17:10:23] wdqs is active-active, so we need the recent changes on both sides [17:10:40] gehel: yeah but if codfw is the right cluster to use we need to fix the config [17:10:46] I mean the puppet config [17:10:50] yep [17:11:06] (03PS1) 10Thiemo Kreuz (WMDE): Do not leak local $wgWBShared… variables to th eglobal scope [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444632 [17:11:25] I am totally ok with any choice you pick :) [17:11:46] (as long as you are aware of the tradeoff etc..) [17:12:06] !log gehel@deploy1001 Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only) [17:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:12:21] o/ [17:12:25] should I read all backscroll? :) [17:12:38] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s) [17:12:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:13:15] ottomata: I just suggested to use main instead of jumbo for the wdqs use case [17:13:24] to bypass mirror maker [17:13:32] yes [17:13:34] indeed! [17:13:48] is main cluster now enabled with offsets and all that thing? [17:14:23] what's the diff between them, in general, any docs? [17:14:45] 10Operations: Decommission servermon - https://phabricator.wikimedia.org/T198939 (10Volans) >>! In T198939#4408774, @akosiaris wrote: > There is a query tab but I get > > ``` > What you were looking for has been disabled by the administrator. > ``` We decided to disable access to the query tab because it allo... [17:14:58] !log gehel@deploy1001 Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater [17:15:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:15:59] (03CR) 10Andrew Bogott: [C: 032] pdns recursor: update allow_from [puppet] - 10https://gerrit.wikimedia.org/r/444623 (https://phabricator.wikimedia.org/T199123) (owner: 10Andrew Bogott) [17:16:09] SMalyshev: yes it is, and it provides multi-dc, where jumbo does not. So as far as I understand, a much better option for us. [17:16:39] gehel: what yu mean by provides multi-dc? [17:17:18] SMalyshev: yup! [17:17:25] clusters are the same versions with same features [17:17:33] we currently have 3 kafka clusters [17:17:40] jumbo-eqiad, main-eqiad, main-codfw [17:17:41] jumbo is only available in eqiad, so if we loose eqiad, we need to switch manually our config. Main has 2 clusters, one in eqiad, one in codfw, with the same data (as far as I understand) [17:17:51] main-eqiad and main-codfw are mirrors of each other [17:17:57] you got it [17:18:04] gehel: aha, so then I assume we just need to use main-$site [17:18:12] yes, if you are configuring via puppet [17:18:24] you just use kafka_config('main') [17:18:32] gehel: ok, let's do it like that then [17:18:34] and you'll get the proper config hash for the current DC [17:18:56] ottomata: sounds great, thanks! [17:19:07] e.g. like this [17:19:07] https://github.com/wikimedia/puppet/blob/1a5f987438c7a65ebd2193748bc840405c98c5ae/modules/profile/manifests/changeprop.pp#L16-L17 [17:19:31] (03PS2) 10Gehel: wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 [17:19:33] (03PS3) 10Gehel: Enable kafka poller on test hosts [puppet] - 10https://gerrit.wikimedia.org/r/444265 (owner: 10Smalyshev) [17:19:41] FYI, these clustesr are defined in common.yaml hhiera [17:19:41] https://github.com/wikimedia/puppet/blob/1a5f987438c7a65ebd2193748bc840405c98c5ae/hieradata/common.yaml#L419 [17:20:11] ottomata: Oh, so even easier than what I was trying to do. Let me fix that again [17:20:57] (03PS3) 10Gehel: wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 [17:20:59] (03PS4) 10Gehel: Enable kafka poller on test hosts [puppet] - 10https://gerrit.wikimedia.org/r/444265 (owner: 10Smalyshev) [17:21:27] ottomata: does https://gerrit.wikimedia.org/r/c/operations/puppet/+/444627 finally looks good to you? [17:21:46] (03CR) 10Smalyshev: [C: 031] wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 (owner: 10Gehel) [17:22:00] ya htat should work! [17:22:13] ottomata: kool! thanks! [17:22:20] (03CR) 10Ottomata: [C: 031] wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 (owner: 10Gehel) [17:22:46] ah gehel sorry that is definitely better, didn't think about it :) [17:23:43] RECOVERY - MegaRAID on db1069 is OK: OK: optimal, 1 logical, 2 physical, WriteBack policy [17:23:45] elukey: no problem! We're all learning (at least I am) [17:24:03] (03PS2) 10EBernhardson: [WIP] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [17:24:17] !log gehel@deploy1001 Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (duration: 09m 18s) [17:24:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:24:26] SMalyshev: standup in a few minutes, let's just deploy the updater atm and enable kafka a bit later if that's OK with you [17:24:50] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [17:25:33] RECOVERY - Long running screen/tmux on furud is OK: OK: No SCREEN or tmux processes detected. [17:26:28] gehel: yes, don't enable it yet, I want to manually test [17:26:46] gehel: just merge 444627 [17:27:50] SMalyshev: this is not going to bring any change until we enable kafka, but shoudl allow you to run a puppet compiler to see the output [17:28:02] (03PS4) 10Gehel: wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 [17:28:33] * gehel is probably stating the obvious :) [17:29:06] (03CR) 10Gehel: [C: 032] wdqs: use kafka main cluster instead of jumbo [puppet] - 10https://gerrit.wikimedia.org/r/444627 (owner: 10Gehel) [17:31:19] (03PS3) 10EBernhardson: [WIP] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [17:31:40] (03PS2) 10Andrew Bogott: openstack: add neutron 172 IP designations for network data.yaml [puppet] - 10https://gerrit.wikimedia.org/r/444611 (https://phabricator.wikimedia.org/T184209) (owner: 10Rush) [17:32:57] !log refactoring analytics firewall policies [17:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:34:03] gehel: correct, that's what I want to do, run the compiler [17:34:11] SMalyshev: so you're all set! [17:34:53] (03PS5) 10Smalyshev: Enable kafka poller on test hosts [puppet] - 10https://gerrit.wikimedia.org/r/444265 [17:35:34] (03CR) 10Andrew Bogott: [C: 032] openstack: add neutron 172 IP designations for network data.yaml [puppet] - 10https://gerrit.wikimedia.org/r/444611 (https://phabricator.wikimedia.org/T184209) (owner: 10Rush) [17:43:59] (03CR) 10Alex Monk: [C: 032] Make pylint match flake8 line length limit [software/certcentral] - 10https://gerrit.wikimedia.org/r/444629 (owner: 10Vgutierrez) [17:44:31] (03Merged) 10jenkins-bot: Make pylint match flake8 line length limit [software/certcentral] - 10https://gerrit.wikimedia.org/r/444629 (owner: 10Vgutierrez) [17:45:00] (03CR) 10jenkins-bot: Make pylint match flake8 line length limit [software/certcentral] - 10https://gerrit.wikimedia.org/r/444629 (owner: 10Vgutierrez) [17:49:42] jouncebot: next [17:49:42] In 0 hour(s) and 10 minute(s): Morning SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1800) [18:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Dear deployers, time to do the Morning SWAT (Max 6 patches) deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1800). [18:00:04] bmansurov, RoanKattouw, Amir1, and Urbanecm: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [18:00:14] Here [18:00:37] Ignore my patch. Afk [18:00:50] I'm here, and I can do the deploy [18:01:34] Good! [18:02:23] I'll stand in instead of Amir1 then, if that's OK [18:02:38] (03CR) 10Catrope: [C: 032] Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [18:02:48] I have https://gerrit.wikimedia.org/r/c/mediawiki/core/+/444648 which is sort of urgent but SWAT was at capacity [18:03:12] tgr: that would be fantastic [18:03:56] (03Merged) 10jenkins-bot: Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [18:05:07] 10Operations, 10Operations-Software-Development: DNS repo: add CI checks for obvious configuration errors - https://phabricator.wikimedia.org/T182028 (10Volans) Current implementation takes into account only `A`, `AAAA` and `PTR` records, with their respective `$ORIGIN` and skips: - `svc.*` domains - some chec... [18:05:12] (03PS1) 10Volans: Add zone_validator script [dns] - 10https://gerrit.wikimedia.org/r/444649 (https://phabricator.wikimedia.org/T182028) [18:06:21] 10Operations, 10Research: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10Miriam) Thanks @Pirroh, @RobH could you please add @Pirroh's key to the system? Many thanks! [18:07:15] Urbanecm: Your patch is on mwdebug1002, please test [18:07:50] Ack [18:08:03] gehel: did you deploy the new updater? [18:08:05] (03CR) 10Volans: "For more context and details:" [dns] - 10https://gerrit.wikimedia.org/r/444649 (https://phabricator.wikimedia.org/T182028) (owner: 10Volans) [18:08:11] (03CR) 10jenkins-bot: Enable TemplateStyles on huwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444378 (https://phabricator.wikimedia.org/T198725) (owner: 10Tacsipacsi) [18:08:23] ah, wait, I see it now [18:08:34] SMalyshev: in interview right now, but yes, I did deploy [18:09:23] PROBLEM - Check systemd state on wdqs1009 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [18:10:33] RECOVERY - Check systemd state on wdqs1009 is OK: OK - running: The system is fully operational [18:11:28] RoanKattouw, please deploy worldwide :) [18:13:06] !log catrope@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on huwiktionary (T198725) (duration: 00m 52s) [18:13:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:13:11] T198725: Deploy TemplateStyles on huwiktionary - https://phabricator.wikimedia.org/T198725 [18:13:58] (03PS2) 10Catrope: Rollout Watchlist Structured Filters to most wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440641 (https://phabricator.wikimedia.org/T181193) (owner: 10Mooeypoo) [18:14:33] !log updating mr1-eqsin security policies - T199074 [18:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:38] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Urbanecm) [18:14:49] (03CR) 10Catrope: [C: 032] Rollout Watchlist Structured Filters to most wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440641 (https://phabricator.wikimedia.org/T181193) (owner: 10Mooeypoo) [18:15:37] Uhh oops [18:15:53] No wait that was correct [18:15:59] Urbanecm: Your patch is deployed now [18:16:08] (03Merged) 10jenkins-bot: Rollout Watchlist Structured Filters to most wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440641 (https://phabricator.wikimedia.org/T181193) (owner: 10Mooeypoo) [18:16:28] Thank you RoanKattouw! [18:17:19] 10Operations, 10Research, 10SRE-Access-Requests: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10RobH) Please note this task was NOT filed in the proper manner, so it has not been picked up by Operations until I noticed it via ping. All access requests shoul... [18:17:46] 10Operations, 10Research, 10SRE-Access-Requests: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10RobH) [18:18:08] (03PS3) 10C. Scott Ananian: Remove $wgUseTidy and $wgTidyConfig from configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443645 (https://phabricator.wikimedia.org/T175706) [18:18:11] (03PS1) 10C. Scott Ananian: Remove unnecessary code: $wgTidyConfig can never be null [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 [18:18:13] (03CR) 10jenkins-bot: Rollout Watchlist Structured Filters to most wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440641 (https://phabricator.wikimedia.org/T181193) (owner: 10Mooeypoo) [18:19:20] (03CR) 10C. Scott Ananian: "> You're not meant to have patches that touch both CS and IS any more" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443645 (https://phabricator.wikimedia.org/T175706) (owner: 10C. Scott Ananian) [18:20:22] (03PS2) 10C. Scott Ananian: Remove unnecessary code: $wgTidyConfig can never be null [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 [18:20:36] gehel: do you know who owns webproxy on production? [18:20:40] (03PS4) 10C. Scott Ananian: Remove $wgUseTidy and $wgTidyConfig from configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443645 (https://phabricator.wikimedia.org/T175706) [18:20:54] I think there's a problem there (not related to kafka), it eats 204 responses... [18:21:01] and converts them to 200... [18:23:06] (03CR) 10Jforrester: [C: 031] Remove $wgUseTidy and $wgTidyConfig from configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443645 (https://phabricator.wikimedia.org/T175706) (owner: 10C. Scott Ananian) [18:23:11] (03CR) 10Jforrester: [C: 031] Remove unnecessary code: $wgTidyConfig can never be null [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [18:23:15] !log catrope@deploy1001 Synchronized wmf-config/: Rollout Watchlist Structured Filters to most wikis (T181193) (duration: 00m 53s) [18:23:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:19] T181193: [EPIC] Graduate the New Filters UX on Watchlist out of beta on all wikis - https://phabricator.wikimedia.org/T181193 [18:23:57] 10Operations, 10Research, 10SRE-Access-Requests: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10RobH) @miriam & @pirroh: We cannot add your key until all of the steps/requirements have been met, and at this time they have not. Please note that I do not see... [18:25:23] 10Operations, 10Research, 10SRE-Access-Requests: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10RobH) [18:26:59] !log catrope@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Resyncing to shut up warnings (duration: 00m 48s) [18:27:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:30:10] 10Operations, 10Research, 10SRE-Access-Requests: Request access to data for citation usage research - https://phabricator.wikimedia.org/T198662 (10RStallman-legalteam) I can confirm that there is a valid NDA on file for Michele Catasta. This info is also on the sheet in the "research collaborators" tab. Thanks! [18:30:23] 10Operations, 10netops, 10Goal: Increase network capacity (2018-19 Q1 Goal) - https://phabricator.wikimedia.org/T199142 (10ayounsi) [18:30:45] tgr: Your patch is on mwdebug1002, please test [18:31:25] 10Operations, 10netops, 10Goal: Increase network capacity (2018-19 Q1 Goal) - https://phabricator.wikimedia.org/T199142 (10ayounsi) [18:31:27] 10Operations, 10ops-eqiad, 10Cloud-VPS, 10cloud-services-team: Rack/cable/configure asw2-b-eqiad switch stack - https://phabricator.wikimedia.org/T183585 (10ayounsi) [18:32:05] 10Operations, 10netops: Rack/setup cr2-eqdfw - https://phabricator.wikimedia.org/T196941 (10ayounsi) [18:32:30] 10Operations, 10netops: Rack/Setup new codfw QFX5100 10G switch - https://phabricator.wikimedia.org/T197147 (10ayounsi) [18:33:41] RoanKattouw: works [18:36:22] Syncing [18:37:05] !log catrope@deploy1001 Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Avoid losing cached ParserOutput in doEditContent (T198483) (duration: 00m 51s) [18:37:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:37:09] T198483: Save Timing increased 50% since 2018-06-28 20:53 - https://phabricator.wikimedia.org/T198483 [18:38:59] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Reedy) [18:39:49] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Reedy) It's a CPAN module https://metacpan.org/pod/Text::CSV_XS Or libtext-csv-xs-perl I guess [18:41:29] thanks! [18:42:34] (03PS4) 10EBernhardson: Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [18:42:36] (03PS16) 10EBernhardson: Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 [18:42:40] (03PS19) 10EBernhardson: convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 [18:42:42] (03PS23) 10EBernhardson: prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 [18:42:44] (03PS26) 10EBernhardson: Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 [18:42:46] (03PS54) 10EBernhardson: Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 [18:43:29] (03CR) 10jerkins-bot: [V: 04-1] Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 (owner: 10EBernhardson) [18:44:08] (03CR) 10jerkins-bot: [V: 04-1] convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 (owner: 10EBernhardson) [18:44:38] (03CR) 10jerkins-bot: [V: 04-1] prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 (owner: 10EBernhardson) [18:44:43] (03CR) 10jerkins-bot: [V: 04-1] Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 (owner: 10EBernhardson) [18:44:53] (03CR) 10Alex Monk: [WIP] get rid of openssl CLI usage (035 comments) [software/certcentral] - 10https://gerrit.wikimedia.org/r/444631 (owner: 10Vgutierrez) [18:45:14] (03CR) 10jerkins-bot: [V: 04-1] Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 (owner: 10EBernhardson) [18:45:36] (03PS2) 10Catrope: Enable ORES edit quality filters on bswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444015 (https://phabricator.wikimedia.org/T197010) [18:45:42] (03CR) 10Catrope: [C: 032] Enable ORES edit quality filters on bswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444015 (https://phabricator.wikimedia.org/T197010) (owner: 10Catrope) [18:46:56] (03Merged) 10jenkins-bot: Enable ORES edit quality filters on bswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444015 (https://phabricator.wikimedia.org/T197010) (owner: 10Catrope) [18:48:06] (03CR) 10jenkins-bot: Enable ORES edit quality filters on bswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444015 (https://phabricator.wikimedia.org/T197010) (owner: 10Catrope) [18:48:54] !log apply analytics-in6 firewall filter to cr1/2-eqiad with a default permit+log [18:48:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:20] !log catrope@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enable ORES edit quality filters on bswiki (T197010) (duration: 00m 50s) [18:52:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:52:24] T197010: Enable bswiki edit quality filters in RecentChanges - https://phabricator.wikimedia.org/T197010 [18:53:19] (03PS2) 10Catrope: Enable ORES edit quality filters on srwiki (damaging only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444018 (https://phabricator.wikimedia.org/T197012) [18:53:56] (03CR) 10Catrope: [C: 032] Enable ORES edit quality filters on srwiki (damaging only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444018 (https://phabricator.wikimedia.org/T197012) (owner: 10Catrope) [18:55:09] (03Merged) 10jenkins-bot: Enable ORES edit quality filters on srwiki (damaging only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444018 (https://phabricator.wikimedia.org/T197012) (owner: 10Catrope) [18:55:14] (03PS17) 10EBernhardson: Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 [18:55:44] (03CR) 10Alex Monk: [WIP] get rid of openssl CLI usage (031 comment) [software/certcentral] - 10https://gerrit.wikimedia.org/r/444631 (owner: 10Vgutierrez) [18:56:02] (03CR) 10jerkins-bot: [V: 04-1] Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 (owner: 10EBernhardson) [18:57:45] (03CR) 10jenkins-bot: Enable ORES edit quality filters on srwiki (damaging only) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444018 (https://phabricator.wikimedia.org/T197012) (owner: 10Catrope) [18:59:06] (03PS18) 10EBernhardson: Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 [19:00:04] Niharika and MaxSem: Time to snap out of that daydream and deploy GlobalPreferences deploy to Wikipedias. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T1900). [19:01:20] !log catrope@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enable ORES damaging filter on srwiki (T197012) (duration: 00m 50s) [19:01:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:01:24] T197012: Enable srwiki edit quality filters in RecentChanges - https://phabricator.wikimedia.org/T197012 [19:02:40] \o/ [19:03:14] (03PS1) 10MaxSem: Deploy GlobalPreferences to the rest of SUL wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444652 (https://phabricator.wikimedia.org/T189806) [19:04:26] Undefined variable: wmgWlFiltersDefault in /srv/mediawiki/wmf-config/CommonSettings.php on line 3728 [19:05:08] RoanKattouw: ^ [19:05:22] HOW [19:05:38] I already re-synced InitialiseSettings.php and logstash stopped showing those errors [19:05:44] Half an hour ago [19:05:52] It's on top of fatalmonitor right now. [19:05:58] what the... [19:06:36] I'm not seeing that (fatalmonitor on logstash) [19:06:53] I'm looking at the CLI script. [19:07:01] MaxSem: Where did you look? [19:07:13] fatalmonitor on logstash is happier than i've seen in awhile [19:07:32] Oh yeah the fatalmonitor script shows you the 1000 most recent errors no matter how old they are [19:07:51] heh, and because it's so clean, the script is showing old stuff :} [19:07:51] yes, it looks the cluster is generating so few errors that old ones stay on top [19:07:53] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10ezachte) Yes in my code it's CPAN as follows: use Text::CSV_XS; [19:08:05] Gotcha. [19:09:11] (03PS20) 10EBernhardson: convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 [19:09:13] (03PS24) 10EBernhardson: prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 [19:09:15] (03PS27) 10EBernhardson: Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 [19:09:17] (03PS55) 10EBernhardson: Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 [19:10:18] solid [19:10:21] haha [19:10:49] 10Operations, 10Beta-Cluster-Infrastructure, 10Security-Team, 10Patch-For-Review: Delete deployment-mediawiki06 - https://phabricator.wikimedia.org/T192996 (10Krenair) 05Open>03Resolved a:03Krinkle thanks [19:10:58] It's pretty funny that things are running so well, we all freaked out a bit :) [19:12:00] <3 [19:18:10] (03CR) 10MaxSem: [C: 032] Deploy GlobalPreferences to the rest of SUL wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444652 (https://phabricator.wikimedia.org/T189806) (owner: 10MaxSem) [19:18:19] (03PS1) 10Jforrester: Cleanup: Remove wgWikiEditorFeatures, dropped in master in Ia1eb91d2d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444655 [19:18:21] (03PS1) 10Jforrester: Cleanup: Remove Beta Cluster use of wikieditor-preview preference, no longer around [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444656 [19:18:23] (03PS1) 10Jforrester: Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444657 [19:18:25] (03PS1) 10Jforrester: Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion, dropped in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444658 [19:18:27] (03PS1) 10Jforrester: Cleanup: Stop setting wgTmhEnableMp3Uploads, dropped ages ago [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444659 [19:18:29] (03PS1) 10Jforrester: Cleanup: Stop setting wmgTmhEnableMp3Uploads, default true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444660 [19:18:31] (03PS1) 10Jforrester: Cleanup: No need for officewiki-specific upload for MP3s any more [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444661 [19:19:04] PROBLEM - Memory correctable errors -EDAC- on cp1053 is CRITICAL: 225 ge 4 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=cp1053&var-datasource=eqiad%2520prometheus%252Fops [19:19:50] (03Merged) 10jenkins-bot: Deploy GlobalPreferences to the rest of SUL wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444652 (https://phabricator.wikimedia.org/T189806) (owner: 10MaxSem) [19:20:53] (03CR) 10jenkins-bot: Deploy GlobalPreferences to the rest of SUL wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444652 (https://phabricator.wikimedia.org/T189806) (owner: 10MaxSem) [19:20:56] This is the first time I have seen [19:20:58] TOO MANY CONCURRENT CONNECTIONS [19:20:58] You ("92.40.248.204") have too many concurrent connections. [19:21:10] Connecting to phab [19:21:22] Niharika: pulled on mwdebug1002 [19:21:50] paladox: Started happening since last week. [19:22:13] Yeh, though I haven’t experienced that yet until now :) [19:22:45] 10Operations, 10Citoid, 10VisualEditor, 10Services (watching): Transition citoid to use Zotero's translation-server-v2 - https://phabricator.wikimedia.org/T197242 (10Jrbranaa) Hey @Mvolz, just wanted to check in on this task. It seems like we are waiting on Zotero team in order to move forward. With the A... [19:22:56] Someone has been on phab using the ip I have way to many times within the rate limit time [19:24:00] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) [19:27:04] !log maxsem@deploy1001 Synchronized wmf-config/InitialiseSettings.php: GlobalPrefs everywhere https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/444652/ (duration: 00m 51s) [19:27:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:27:28] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) p:05Triage>03High [19:28:06] 10Operations, 10Citoid, 10Code-Stewardship-Reviews, 10VisualEditor, and 2 others: zotero translation server: code stewardship request - https://phabricator.wikimedia.org/T187194 (10Jrbranaa) 05Open>03Resolved a:03Jrbranaa Closing this as resolved as Audiences->Contributors has been identified as Code... [19:29:22] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) [19:31:15] (03PS1) 10Jforrester: Cleanup: Stop trying to set wgLicenseURL, never read [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444663 (https://phabricator.wikimedia.org/T154069) [19:33:57] (03PS1) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [19:34:27] (03PS1) 10Jforrester: Fix case of wgLocalTZoffset to apply [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444665 [19:34:31] (03PS2) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [19:35:11] (03PS3) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [19:36:34] (03CR) 10Urbanecm: [C: 031] "LGTM, thank you!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [19:36:42] (03CR) 10Jforrester: "Bit worried about this one – AFAICS it's never worked and no-one's noticed. Maybe we should drop this?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444665 (owner: 10Jforrester) [19:39:12] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) [19:39:34] (03CR) 10Anomie: [C: 031] "Go ahead whenever you're ready." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [19:41:49] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Jonas) My IP is also blocked {F23514678} [19:44:07] (03CR) 10Jforrester: [C: 04-1] "As discussed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444665 (owner: 10Jforrester) [19:45:44] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10ezachte) @Reedy FYI I'm asking as I don't have server rights, so this is normal procedure. [19:45:44] (03CR) 10Anomie: "I note this will cause $wgUseTidy to be false rather than true on most wikis, as mentioned in code review on I8b430477. Have the few place" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/443645 (https://phabricator.wikimedia.org/T175706) (owner: 10C. Scott Ananian) [19:45:57] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10BBlack) This raises some questions that are probably unrelated to the problem at hand, but might affect things in... [19:46:26] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Reedy) >>! In T199131#4409454, @ezachte wrote: > @Reedy FYI I'm asking as I don't have server rights, so this is normal procedure. I know. I was just clarifying for @Ottomata :) [19:47:59] 10Operations: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Ottomata) Ah a deb package! thanks @reedy! [19:49:18] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) > Why is an internal service (wdqs) querying a public endpoint? It needs to load Wikidata data, and... [19:49:43] (03PS1) 10Ottomata: Install libtext-csv-xs-perl on stat servers [puppet] - 10https://gerrit.wikimedia.org/r/444668 (https://phabricator.wikimedia.org/T199131) [19:50:47] (03PS5) 10Chico Venancio: prometheus: tools: scrape paws metrics into prometheus [puppet] - 10https://gerrit.wikimedia.org/r/441514 (https://phabricator.wikimedia.org/T195030) [19:51:13] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) Yes, I verified, I get the same block with `curl --noproxy \*`. Just different IP in the error message. [19:51:29] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Gehel) >>! In T199146#4409455, @BBlack wrote: > This raises some questions that are probably unrelated to the pro... [19:53:21] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) For the record, `204 NO CONTENT` or 200 with RDF output is the right answer. For most items, it's 204... [19:57:00] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Gehel) It looks to me that the block is done by mediawiki itself (see P7355 for details): < x-cache: cp1066... [20:00:05] cscott, arlolra, subbu, bearND, halfak, and Amir1: Your horoscope predicts another unfortunate Services – Parsoid / Citoid / Mobileapps / ORES / … deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T2000). [20:00:22] Nothing for ORES today [20:04:14] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) Yeah looks like ipblocks table for wikidata has block on `2620:0:862:101:0:0:0:0/96` by user "Merlissi... [20:08:03] (03PS2) 10Ottomata: Install libtext-csv-xs-perl on stat servers [puppet] - 10https://gerrit.wikimedia.org/r/444668 (https://phabricator.wikimedia.org/T199131) [20:08:06] (03CR) 10Ottomata: [V: 032 C: 032] Install libtext-csv-xs-perl on stat servers [puppet] - 10https://gerrit.wikimedia.org/r/444668 (https://phabricator.wikimedia.org/T199131) (owner: 10Ottomata) [20:08:26] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Gehel) >>! In T199146#4409514, @Smalyshev wrote: > Yeah looks like ipblocks table for wikidata has block on `2620... [20:09:29] 10Operations, 10Release Pipeline, 10Release-Engineering-Team (Kanban), 10Services (watching): Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10thcipriani) p:05Triage>03Normal [20:09:42] no mobileapps deploy today [20:13:22] (03PS1) 10Ottomata: Install python(3)-tk so that Jupyter can render charts with matplotlib [puppet] - 10https://gerrit.wikimedia.org/r/444735 (https://phabricator.wikimedia.org/T190443) [20:14:13] 10Operations, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Ottomata) a:03Ottomata [20:14:26] (03CR) 10Ottomata: [C: 032] Install python(3)-tk so that Jupyter can render charts with matplotlib [puppet] - 10https://gerrit.wikimedia.org/r/444735 (https://phabricator.wikimedia.org/T190443) (owner: 10Ottomata) [20:14:27] 10Operations, 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Please install Text::CSV_XS at stat1005 - https://phabricator.wikimedia.org/T199131 (10Ottomata) Done! [20:17:14] PROBLEM - Device not healthy -SMART- on db1069 is CRITICAL: cluster=mysql device=megaraid,0 instance=db1069:9100 job=node site=eqiad https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1069&var-datasource=eqiad%2520prometheus%252Fops [20:19:12] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) This block seems to be driven by `$wgSoftBlockRanges` setting in CommonSettings.php, which includes `$... [20:28:46] (03PS5) 10EBernhardson: Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [20:29:26] (03CR) 10jerkins-bot: [V: 04-1] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [20:35:58] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Mahir256) Hi @Jonas: I blocked that particular range, among others allocated to Telefonica Germany, in an attempt... [20:38:14] 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: "Blocked" response when trying to access constraintsrdf action from production host - https://phabricator.wikimedia.org/T199146 (10Smalyshev) Looks like we don't need to change blocks - instead, 'constraintsrdf' should be marked as read action... [20:42:18] ACKNOWLEDGEMENT - Device not healthy -SMART- on labstore1006 is CRITICAL: cluster=misc device={cciss,14,cciss,15,cciss,16,cciss,17,cciss,18,cciss,19,cciss,20,cciss,21,cciss,22,cciss,23} instance=labstore1006:9100 job=node site=eqiad Bstorm NFS services may be intentionally not up yet. Following up before returning to service. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=labstore1006&var-datasource=eqiad%2 [20:42:18] ps [20:42:18] ACKNOWLEDGEMENT - NFS on labstore1006 is CRITICAL: connect to address 208.80.154.7 and port 2049: Connection refused Bstorm NFS services may be intentionally not up yet. Following up before returning to service. [20:48:05] (03PS4) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [20:49:13] (03PS5) 10Urbanecm: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [20:49:27] (03PS6) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [20:50:43] (03CR) 10Urbanecm: [C: 04-1] "No spaces are allowed in extra namespaces definitions. Please replace the space with an underscore." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [20:57:21] (03PS7) 10Zoranzoki21: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) [20:57:35] (03PS5) 10Jforrester: Stop loading the MwEmbedSupport extension, part I [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441519 [20:57:47] (03CR) 10EBernhardson: [C: 04-1] "Should be -2. This needs tls certs added to this patch, and private keys added to the private repo in modules/secret/secrets/ssl/" [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [20:59:37] (03CR) 10Urbanecm: [C: 031] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [21:00:04] bawolff and Reedy: That opportune time is upon us again. Time for a Weekly Security deployment window deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T2100). [21:06:04] (03PS10) 10Smalyshev: Generate daily diffs for categories RDF [puppet] - 10https://gerrit.wikimedia.org/r/378355 (https://phabricator.wikimedia.org/T198356) [21:07:59] (03PS6) 10EBernhardson: Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 [21:12:43] (03CR) 10EBernhardson: [C: 031] "puppet compiler looks okay. Deployed to beta cluster, after working out appropriate tls key/cert all looks happy there." [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [21:13:10] (03CR) 10EBernhardson: [C: 04-1] Switch elasticsearch to use tlsproxy module [puppet] - 10https://gerrit.wikimedia.org/r/444610 (owner: 10EBernhardson) [21:25:02] (03PS8) 10Aftab: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [21:41:33] PROBLEM - puppet last run on labtestcontrol2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:47:12] Does prod still have trusty hosts? [21:47:22] (03CR) 1020after4: [C: 031] "@krinkle: That patch makes the score 0 for whitelisted ips. I'm not sure why it wouldn't work." [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [21:48:09] !log createAndPromote.php --wiki=officewiki BWolff_\(WMF\) --sysop --force [21:48:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:50:04] Krenair: some db hosts maybe? [21:51:15] was just wondering because I noticed we still have a couple in deployment-prep [21:51:21] for zotero and urldownloader [21:51:43] RECOVERY - puppet last run on labtestcontrol2003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [21:52:36] (03PS9) 10Urbanecm: Create Publisher namespace in Bengali Wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [21:52:43] (03CR) 10Krinkle: "It does work (afaik), but I meant to ask: Do you prefer moving the list of whitelisted IPs from the Phab extension source to here (puppet)" [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [21:53:15] (03CR) 10Urbanecm: "@Aftab: This isn't relevant to creating Publisher namespace. I will create separate bug for it." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444664 (https://phabricator.wikimedia.org/T199028) (owner: 10Zoranzoki21) [21:56:01] (03PS1) 10Urbanecm: Replace spaces with underscores in bnwikisource ExtraNamespaces definition [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444749 (https://phabricator.wikimedia.org/T199161) [22:02:55] (03CR) 1020after4: [C: 031] "I don't have a strong preference. I did it that way at the time because it was expedient." [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [22:10:22] 10Operations, 10Services (done), 10User-mobrovac: Migrate SCA cluster to SCB (Jessie and Node 4.2) - https://phabricator.wikimedia.org/T96017 (10Krinkle) [22:11:46] moritzm: sorry for the genericness, but could you take a look at https://gerrit.wikimedia.org/r/#/q/hashtag:beta-picked+is:open and see if some can be merged, or should be closed/unpicked, or other kind of feedback? [22:19:24] (03PS2) 10Bstorm: labsdb1006: Reimage as stretch and make it osm::master [puppet] - 10https://gerrit.wikimedia.org/r/443799 (https://phabricator.wikimedia.org/T197246) (owner: 10Alexandros Kosiaris) [22:33:05] (03CR) 10Bstorm: [C: 032] labsdb1006: Reimage as stretch and make it osm::master [puppet] - 10https://gerrit.wikimedia.org/r/443799 (https://phabricator.wikimedia.org/T197246) (owner: 10Alexandros Kosiaris) [22:54:40] !log added disk from new shelf to /srv/dumps on labstore1006 [22:54:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:58:31] RoanKattouw: Want to supervise me SWATing again? [22:59:36] 10Operations, 10ops-eqiad, 10Cloud-VPS, 10Datasets-General-or-Unknown: rack upgraded storage capacity in labstore100[67].eqiad.wmnet - https://phabricator.wikimedia.org/T196651 (10Bstorm) New shelf is now live and part of the /srv/dumps filesystem on labstore1006. It isn't fully restored to service yet, b... [23:00:04] addshore, hashar, anomie, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Evening SWAT (Max 6 patches) . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20180709T2300). [23:00:04] bd808, miriam, RoanKattouw, and JamesF: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [23:00:17] I'll do the SWAT [23:00:19] Oh [23:00:21] James_F: Sure [23:00:22] ^^ [23:00:24] Kk. [23:00:26] I'll come over [23:02:20] * bd808 is here [23:02:36] Oh. Huh. [23:03:06] bd808: Ready to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/441568 ? [23:03:44] (03PS3) 10Jforrester: Stop collecting data for Schema:CitationUsage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441568 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [23:04:42] (03PS1) 10Ayounsi: Reserve cloud-instance-transport1-b-eqiad [dns] - 10https://gerrit.wikimedia.org/r/444759 (https://phabricator.wikimedia.org/T184209) [23:05:01] James_F: sure. I need to remember how to check it... ;) [23:05:10] Kk. [23:05:13] (03CR) 10Jforrester: [C: 032] "SWATage ahoy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441568 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [23:05:30] (03CR) 10Jforrester: [C: 032] "SWATage ahoy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441519 (owner: 10Jforrester) [23:05:50] (03CR) 10Jforrester: [C: 032] "SWATage ahoy." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [23:06:27] (03Merged) 10jenkins-bot: Stop collecting data for Schema:CitationUsage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441568 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [23:06:29] (03CR) 10Ayounsi: [C: 032] Reserve cloud-instance-transport1-b-eqiad [dns] - 10https://gerrit.wikimedia.org/r/444759 (https://phabricator.wikimedia.org/T184209) (owner: 10Ayounsi) [23:06:45] (03Merged) 10jenkins-bot: Stop loading the MwEmbedSupport extension, part I [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441519 (owner: 10Jforrester) [23:07:18] James_F: I figured it out! mw.config.get('wgWMECitationUsagePopulationSize', 0) tells me it is still active [23:07:46] Kk. [23:08:15] (03CR) 10jenkins-bot: Stop collecting data for Schema:CitationUsage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441568 (https://phabricator.wikimedia.org/T191086) (owner: 10Bmansurov) [23:08:33] once you stage the patch on mwdebug1001 (or 1002) I should be able to see that change [23:11:00] [Roan is testing on mwdebug1002 now.] [23:12:23] bd808: On mwdebug1002, please test. [23:12:38] * bd808 does the needful [23:13:21] James_F: I'm getting the expected "0" result. If nothing is going crazy in the error logs it should be good to go. [23:14:05] Okie dokie. [23:16:18] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT for bd808 (duration: 00m 51s) [23:16:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:16:32] bd808: Everywhere now. Hope it looks OK? [23:17:15] James_F: looks good for me on enwiki [23:17:28] Excellent. [23:17:46] (03PS3) 10Jforrester: Remove unnecessary code: $wgTidyConfig can never be null [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [23:18:14] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: Stop loading the MwEmbedSupport extension, part I (duration: 00m 50s) [23:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:19:30] (03CR) 10Smalyshev: [C: 031] [cirrus] cleanup unused config vars 1/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444576 (owner: 10DCausse) [23:19:47] !log jforrester@deploy1001 Synchronized php-1.32.0-wmf.10/extensions/ORES: ORES fix Ia1b09d7711 for RoanKattouw (duration: 00m 50s) [23:19:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:19:50] (03CR) 10Smalyshev: [C: 031] [cirrus] cleanup unused config vars 2/2 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444577 (owner: 10DCausse) [23:20:22] (03CR) 10Jforrester: [C: 032] "`" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [23:23:24] (03CR) 10Jforrester: [C: 032] Stop loading the MwEmbedSupport extension, part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 (owner: 10Jforrester) [23:23:43] (03Merged) 10jenkins-bot: Remove unnecessary code: $wgTidyConfig can never be null [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [23:26:05] (03PS5) 10Jforrester: Stop loading the MwEmbedSupport extension, part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 [23:26:17] (03CR) 10Jforrester: [C: 032] Stop loading the MwEmbedSupport extension, part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 (owner: 10Jforrester) [23:26:36] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT 444650 (duration: 00m 49s) [23:26:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:26:39] PROBLEM - Restbase edge eqsin on text-lb.eqsin.wikimedia.org is CRITICAL: /api/rest_v1/feed/onthisday/{type}/{mm}/{dd} (Retrieve all events for Jan 15) timed out before a response was received [23:26:56] (03CR) 10Jforrester: [C: 032] Stop loading the MwEmbedSupport extension, part III [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441520 (owner: 10Jforrester) [23:27:03] (03PS5) 10Jforrester: Stop loading the MwEmbedSupport extension, part III [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441520 [23:27:11] (03PS5) 10Jforrester: Stop loading the MwEmbedSupport extension, part IV [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441521 [23:27:13] (03CR) 10jerkins-bot: [V: 04-1] Stop loading the MwEmbedSupport extension, part III [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441520 (owner: 10Jforrester) [23:27:17] (03CR) 10Jforrester: [C: 032] Stop loading the MwEmbedSupport extension, part IV [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441521 (owner: 10Jforrester) [23:27:31] (03Merged) 10jenkins-bot: Stop loading the MwEmbedSupport extension, part II [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441518 (owner: 10Jforrester) [23:29:19] (03CR) 10Jforrester: "Deployed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444650 (owner: 10C. Scott Ananian) [23:29:44] James_F: oh praise the lord, are we finally killing that extension [23:29:49] RECOVERY - Restbase edge eqsin on text-lb.eqsin.wikimedia.org is OK: All endpoints are healthy [23:29:50] (03Merged) 10jenkins-bot: Stop loading the MwEmbedSupport extension, part III [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441520 (owner: 10Jforrester) [23:29:52] bawolff: You're welcome. :-) [23:29:53] (03Merged) 10jenkins-bot: Stop loading the MwEmbedSupport extension, part IV [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441521 (owner: 10Jforrester) [23:30:23] bawolff: (Sadly the code still exists for now, just it lives inside TimedMediaHandler.) [23:30:48] Well that's a start in any case [23:31:11] Yeah, and "soon" b.rion will get the replacement good enough to push out as a Beta Feature. [23:32:53] (03CR) 10Jforrester: [C: 032] Cleanup: Remove wgWikiEditorFeatures, dropped in master in Ia1eb91d2d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444655 (owner: 10Jforrester) [23:33:07] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT MwEmbedSupport extension, part II (duration: 00m 51s) [23:33:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:34:45] !log jforrester@deploy1001 Synchronized multiversion/submodules.json: SWAT MwEmbedSupport extension, part III (duration: 00m 50s) [23:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:34:52] (03PS19) 10EBernhardson: Prep work for multi-instance elasticsearch refactor [puppet] - 10https://gerrit.wikimedia.org/r/440498 [23:34:54] (03PS21) 10EBernhardson: convert role::logstash::elasticsearch to profiles [puppet] - 10https://gerrit.wikimedia.org/r/441894 [23:34:56] (03PS25) 10EBernhardson: prometheus/elasticsearch support multiple exporters per host [puppet] - 10https://gerrit.wikimedia.org/r/441321 [23:34:58] (03PS28) 10EBernhardson: Split instance define out of elasticsearch class [puppet] - 10https://gerrit.wikimedia.org/r/441338 [23:35:00] (03PS56) 10EBernhardson: Allow multiple elasticsearch instances per host [puppet] - 10https://gerrit.wikimedia.org/r/440049 [23:35:02] (03PS1) 10EBernhardson: Cleanup ensure => absent after refactoring [puppet] - 10https://gerrit.wikimedia.org/r/444765 [23:36:09] (03PS2) 10Jforrester: Cleanup: Remove wgWikiEditorFeatures, dropped in master in Ia1eb91d2d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444655 [23:36:13] !log jforrester@deploy1001 Synchronized wmf-config/extension-list: Stop loading the MwEmbedSupport extension, part IV (duration: 00m 50s) [23:36:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:36:16] (03CR) 10Jforrester: [C: 032] Cleanup: Remove wgWikiEditorFeatures, dropped in master in Ia1eb91d2d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444655 (owner: 10Jforrester) [23:36:27] (03PS2) 10Jforrester: Cleanup: Remove Beta Cluster use of wikieditor-preview preference, no longer around [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444656 [23:36:32] (03CR) 10Jforrester: [C: 032] Cleanup: Remove Beta Cluster use of wikieditor-preview preference, no longer around [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444656 (owner: 10Jforrester) [23:36:40] (03PS2) 10Jforrester: Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444657 [23:36:44] (03CR) 10Jforrester: [C: 032] Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444657 (owner: 10Jforrester) [23:36:52] (03PS2) 10Jforrester: Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion, dropped in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444658 [23:36:57] (03CR) 10Jforrester: [C: 032] Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion, dropped in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444658 (owner: 10Jforrester) [23:39:26] (03Merged) 10jenkins-bot: Cleanup: Remove wgWikiEditorFeatures, dropped in master in Ia1eb91d2d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444655 (owner: 10Jforrester) [23:39:31] (03Merged) 10jenkins-bot: Cleanup: Remove Beta Cluster use of wikieditor-preview preference, no longer around [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444656 (owner: 10Jforrester) [23:40:38] (03Merged) 10jenkins-bot: Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion to false [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444657 (owner: 10Jforrester) [23:40:49] (03Merged) 10jenkins-bot: Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion, dropped in master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444658 (owner: 10Jforrester) [23:42:02] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Remove wgWikiEditorFeatures I7580e94ecf7 (duration: 00m 50s) [23:42:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:43:15] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings-labs.php: SWAT Cleanup: Remove Beta Cluster use of wikieditor-preview preference Ib16e6e19b1 (duration: 00m 50s) [23:43:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:24] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion If2ee24ae (duration: 00m 50s) [23:47:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:47:56] (03PS2) 10Jforrester: Cleanup: Stop setting wgTmhEnableMp3Uploads, dropped ages ago [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444659 [23:48:02] (03CR) 10Jforrester: [C: 032] Cleanup: Stop setting wgTmhEnableMp3Uploads, dropped ages ago [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444659 (owner: 10Jforrester) [23:48:10] (03PS2) 10Jforrester: Cleanup: Stop setting wmgTmhEnableMp3Uploads, default true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444660 [23:48:14] (03CR) 10Jforrester: [C: 032] Cleanup: Stop setting wmgTmhEnableMp3Uploads, default true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444660 (owner: 10Jforrester) [23:48:20] (03PS2) 10Jforrester: Cleanup: No need for officewiki-specific upload for MP3s any more [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444661 [23:48:24] (03CR) 10Jforrester: [C: 032] Cleanup: No need for officewiki-specific upload for MP3s any more [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444661 (owner: 10Jforrester) [23:48:37] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion I3b8cccc00 (duration: 00m 50s) [23:48:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:49:16] (03Merged) 10jenkins-bot: Cleanup: Stop setting wgTmhEnableMp3Uploads, dropped ages ago [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444659 (owner: 10Jforrester) [23:49:34] (03Merged) 10jenkins-bot: Cleanup: Stop setting wmgTmhEnableMp3Uploads, default true everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444660 (owner: 10Jforrester) [23:49:49] (03Merged) 10jenkins-bot: Cleanup: No need for officewiki-specific upload for MP3s any more [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444661 (owner: 10Jforrester) [23:50:39] (03PS2) 10Mahir256: Replace spaces with underscores in bnwikisource ExtraNamespaces definition [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444749 (https://phabricator.wikimedia.org/T199161) (owner: 10Urbanecm) [23:51:36] (03PS2) 10Jforrester: Cleanup: Stop trying to set wgLicenseURL, never read [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444663 (https://phabricator.wikimedia.org/T154069) [23:51:41] (03CR) 10Jforrester: [C: 032] Cleanup: Stop trying to set wgLicenseURL, never read [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444663 (https://phabricator.wikimedia.org/T154069) (owner: 10Jforrester) [23:52:02] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4 (duration: 00m 50s) [23:52:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:52:58] (03Merged) 10jenkins-bot: Cleanup: Stop trying to set wgLicenseURL, never read [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444663 (https://phabricator.wikimedia.org/T154069) (owner: 10Jforrester) [23:53:26] !log jforrester@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813 (duration: 00m 50s) [23:53:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:55:11] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6 (duration: 00m 50s) [23:55:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:57:27] Any last shouts for SWAT? [23:58:03] !log jforrester@deploy1001 Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s) [23:58:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:58:06] T154069: Collection license URL mishmash in WMF prod - https://phabricator.wikimedia.org/T154069 [23:58:09] OK, that's SWAT done, 15 patches deployed. Thank you everyone.