[00:00:11] PROBLEM - Check systemd state on wdqs1009 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [00:21:11] RECOVERY - Check systemd state on wdqs1009 is OK: OK - running: The system is fully operational [00:33:13] (03PS4) 10Huji: Add several rights to eliminators in fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/430627 (https://phabricator.wikimedia.org/T176553) [00:56:18] (03CR) 10Tim Starling: "> @tim: regardless of x-forwarded-for headers, this patch has other important changes." [puppet] - 10https://gerrit.wikimedia.org/r/443665 (owner: 1020after4) [01:27:33] (03CR) 10Ori.livneh: phabricator: refactor preamble.php to separate unrelated functionality. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/443665 (owner: 1020after4) [01:54:14] PROBLEM - LVS HTTP IPv4 on thumbor.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 20 seconds [01:55:04] RECOVERY - LVS HTTP IPv4 on thumbor.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 281 bytes in 0.002 second response time [03:06:16] (03PS12) 10Krinkle: webperf: Add statsv, navtiming and coal to scap::sources [puppet] - 10https://gerrit.wikimedia.org/r/436601 (https://phabricator.wikimedia.org/T195314) [03:06:18] (03PS6) 10Krinkle: webperf: Get graphite_host for coal::processor from Hiera [puppet] - 10https://gerrit.wikimedia.org/r/442900 (https://phabricator.wikimedia.org/T195314) [03:06:20] (03PS8) 10Krinkle: webperf: Move site vars to profile class params (set from Hiera) [puppet] - 10https://gerrit.wikimedia.org/r/443739 (https://phabricator.wikimedia.org/T195314) [03:06:22] (03PS6) 10Krinkle: webperf: Rename webperf profiles for clarity [puppet] - 10https://gerrit.wikimedia.org/r/443752 (https://phabricator.wikimedia.org/T195314) [03:06:24] (03PS5) 10Krinkle: webperf: Rename role::xenon to profile::webperf::xenon [puppet] - 10https://gerrit.wikimedia.org/r/443757 (https://phabricator.wikimedia.org/T195312) [03:06:26] (03PS5) 10Krinkle: mediawiki: Change xenon interval for Beta Cluster from 10min to 30s [puppet] - 10https://gerrit.wikimedia.org/r/443762 [03:06:28] (03PS3) 10Krinkle: webperf: Enable xenondata_host on perfsite in Beta Cluster [puppet] - 10https://gerrit.wikimedia.org/r/443764 (https://phabricator.wikimedia.org/T195312) [03:06:40] (03CR) 10Krinkle: webperf: Rename webperf profiles for clarity (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/443752 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [03:06:52] (03CR) 10Krinkle: webperf: Rename role::xenon to profile::webperf::xenon (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/443757 (https://phabricator.wikimedia.org/T195312) (owner: 10Krinkle) [04:42:49] !log Deploy schema change on db1082 with replication T146591 T197891 T196379 [04:42:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:42:55] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [04:42:55] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [04:42:56] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [04:45:09] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444150 [04:48:39] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444150 (owner: 10Marostegui) [04:50:07] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444150 (owner: 10Marostegui) [04:50:54] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444150 (owner: 10Marostegui) [04:51:24] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s) [04:51:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:53:47] (03PS1) 10Marostegui: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444151 (https://phabricator.wikimedia.org/T146591) [04:55:22] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444151 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [04:56:48] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444151 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [04:58:17] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s) [04:58:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:58:57] !log Deploy schema change on db1110 T146591 T197891 T196379 [04:59:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:59:03] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [04:59:03] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [04:59:04] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [05:00:00] !log Deploy schema change on s5 primary master (db1070) T146591 T197891 T196379 [05:00:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:00:17] !log kartik@deploy1001 Started deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830) [05:00:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:00:21] T198830: Apertium translation fails with sections with references - https://phabricator.wikimedia.org/T198830 [05:01:17] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1110 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444151 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [05:01:48] !log Optimize dewiki.logging on db1110 - T197459 [05:01:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:01:52] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [05:03:43] !log kartik@deploy1001 Finished deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 (T198830) (duration: 03m 26s) [05:03:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:37:21] PROBLEM - Router interfaces on cr1-eqord is CRITICAL: CRITICAL: host 208.80.154.198, interfaces up: 37, down: 1, dormant: 0, excluded: 0, unused: 0 [05:58:32] RECOVERY - Router interfaces on cr1-eqord is OK: OK: host 208.80.154.198, interfaces up: 39, down: 0, dormant: 0, excluded: 0, unused: 0 [06:15:46] (03PS1) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [06:21:54] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444155 [06:22:16] !log Optimize dewiki.logging on db1070 (s5 primary master) - T197459 [06:22:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:22:20] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:25:45] !log Deploy schema change on s6 codfw master (db2039) with replication, this will generate lag on s6 codfw T146591 T197891 T196379 [06:25:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:25:53] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:25:53] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:25:53] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:28:32] PROBLEM - puppet last run on phab1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [06:31:25] !log Optimize frwiki.logging on db2039 (s6 codfw master), with replication, this will generate lag on s6 codfw - T197459 [06:31:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:31:28] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:32:21] PROBLEM - puppet last run on mw1307 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/php/7.0/fpm/php.ini] [06:34:04] !log Deploy schema change on dbstore1002:s6 T146591 T197891 T196379 [06:34:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:34:09] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [06:34:10] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [06:34:10] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [06:34:37] (03PS2) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [06:38:18] (03PS3) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [06:49:51] !log Optimize frwiki.logging on dbstore1002 - T197459 [06:49:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:49:55] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [06:50:44] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444155 (owner: 10Marostegui) [06:52:03] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444155 (owner: 10Marostegui) [06:52:13] (03CR) 10Ema: vcl: Bump AES128-SHA redirection to 100% (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [06:52:19] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1110" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444155 (owner: 10Marostegui) [06:53:21] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 52s) [06:53:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:57:51] RECOVERY - puppet last run on mw1307 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:59:11] RECOVERY - puppet last run on phab1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [07:03:36] (03PS1) 10Marostegui: db-eqiad.php: Depool db1096:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444158 (https://phabricator.wikimedia.org/T146591) [07:05:21] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1096:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444158 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:06:37] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1096:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444158 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:07:49] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 51s) [07:07:49] (03CR) 10Volans: "Probably worth adapting also modules/profile/manifests/openstack/main/cumin/master.pp" [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [07:07:51] !log Deploy schema change on db1096:3316 T146591 T197891 T196379 [07:07:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:07:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:07:57] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:07:57] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:07:58] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:08:01] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1096:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444158 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:11:48] !log Optimize frwiki.logging on db1096:3316 - T197459 [07:11:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:11:52] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [07:12:20] !log upgrading remaining video scalers to vp9-row-mt enabled ffmpeg build (T190333) [07:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:12:24] T190333: Backport libvpx 1.7.0, ffmpeg packages for VP9 -row-mt option - https://phabricator.wikimedia.org/T190333 [07:16:18] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1096:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444159 [07:18:08] (03PS8) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [07:18:10] (03PS8) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [07:18:12] (03PS9) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [07:18:14] (03PS9) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [07:18:18] (03PS4) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [07:18:20] (03PS2) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [07:19:31] 10Operations, 10TimedMediaHandler-Transcode, 10Patch-For-Review: Backport libvpx 1.7.0, ffmpeg packages for VP9 -row-mt option - https://phabricator.wikimedia.org/T190333 (10MoritzMuehlenhoff) 05Open>03Resolved The new ffmpeg packages are deployed on all video scalers (and the job runners now also servin... [07:19:32] 10Operations, 10TimedMediaHandler-Transcode, 10Patch-For-Review: Backport libvpx 1.7.0, ffmpeg packages for VP9 -row-mt option - https://phabricator.wikimedia.org/T190333 (10MoritzMuehlenhoff) 05Open>03Resolved The new ffmpeg packages are deployed on all video scalers (and the job runners now also servin... [07:20:32] RECOVERY - Check systemd state on notebook1003 is OK: OK - running: The system is fully operational [07:21:48] (03CR) 10Krinkle: "The following seems to have been merged instead:" [puppet] - 10https://gerrit.wikimedia.org/r/444124 (https://phabricator.wikimedia.org/T198612) (owner: 10Alex Monk) [07:22:43] !log re-enabled puppet on mw2246 [07:22:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:25:12] (03PS2) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [07:27:51] (03PS1) 10Jcrespo: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) [07:30:55] (03CR) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [07:36:34] (03PS2) 10Jcrespo: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) [07:38:13] (03PS4) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [07:38:32] (03CR) 10Marostegui: mariadb: Add new instance to tendril to store statistics (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [07:39:08] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1096:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444159 (owner: 10Marostegui) [07:40:08] 10Operations, 10Cloud-Services: Ops access request - https://phabricator.wikimedia.org/T198900 (10akosiaris) I am adding the cloud services team for the 2fa reset and removing the SRE-access-request tag per @Krenair 's comment above. [07:40:11] 10Operations, 10Cloud-Services: Ops access request - https://phabricator.wikimedia.org/T198900 (10akosiaris) I am adding the cloud services team for the 2fa reset and removing the SRE-access-request tag per @Krenair 's comment above. [07:40:39] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1096:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444159 (owner: 10Marostegui) [07:40:51] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1096:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444159 (owner: 10Marostegui) [07:41:03] (03CR) 10Arturo Borrero Gonzalez: [C: 031] "Great idea! thanks!!" [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [07:41:39] (03CR) 10Krinkle: mediawiki: add vhost define (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/439893 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:41:43] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 50s) [07:41:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:42:43] (03PS1) 10Marostegui: db-eqiad.php: Depool db1098:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444161 (https://phabricator.wikimedia.org/T146591) [07:44:40] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1098:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444161 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:45:13] !log Deploy schema change on db1098:3316 T146591 T197891 T196379 [07:45:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:45:19] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [07:45:19] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [07:45:19] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [07:45:52] (03PS5) 10Elukey: profile::mariadb::misc:el::sanitization: move whitelist to Refinery [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) [07:46:21] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1098:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444161 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:46:57] !log marostegui@deploy1001 sync-file aborted: Depool db1098:3315 for alter table (duration: 00m 01s) [07:46:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:47:31] !log Optimize frwiki.logging on db1098:3316 - T197459 [07:47:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:47:35] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [07:47:50] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 50s) [07:47:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:49:11] (03CR) 10Krinkle: mediawiki: add vhost define (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/439893 (https://phabricator.wikimedia.org/T196968) (owner: 10Giuseppe Lavagetto) [07:49:14] (03PS1) 10Alexandros Kosiaris: user piccardi ssh key update [puppet] - 10https://gerrit.wikimedia.org/r/444162 (https://phabricator.wikimedia.org/T151969) [07:49:46] (03CR) 10jerkins-bot: [V: 04-1] user piccardi ssh key update [puppet] - 10https://gerrit.wikimedia.org/r/444162 (https://phabricator.wikimedia.org/T151969) (owner: 10Alexandros Kosiaris) [07:50:53] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1098:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444161 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [07:52:56] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler02/11701/" [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [07:54:15] (03CR) 10Elukey: "Let me know what you guys think about it!" [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [07:54:54] 10Operations, 10SRE-Access-Requests: WMF-NDA-Request for User:Braveheart - https://phabricator.wikimedia.org/T198190 (10akosiaris) 05Open>03stalled p:05Triage>03Low I am setting this to Stalled and a low priority, pending @Braveheart 's response. [07:55:00] 10Operations, 10SRE-Access-Requests: WMF-NDA-Request for User:Braveheart - https://phabricator.wikimedia.org/T198190 (10akosiaris) 05Open>03stalled p:05Triage>03Low I am setting this to Stalled and a low priority, pending @Braveheart 's response. [07:58:13] (03PS2) 10Alexandros Kosiaris: user piccardi ssh key update [puppet] - 10https://gerrit.wikimedia.org/r/444162 (https://phabricator.wikimedia.org/T151969) [07:59:44] (03CR) 10Alexandros Kosiaris: [C: 032] user piccardi ssh key update [puppet] - 10https://gerrit.wikimedia.org/r/444162 (https://phabricator.wikimedia.org/T151969) (owner: 10Alexandros Kosiaris) [08:00:39] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1098:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444166 [08:01:08] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Tiziano Piccardi shell request + analytics-privatedata-users - https://phabricator.wikimedia.org/T151969 (10akosiaris) 05Open>03Resolved Key updated, should make it to the cluster in the next 30 mins. I am resolving this, feel free to... [08:01:14] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Tiziano Piccardi shell request + analytics-privatedata-users - https://phabricator.wikimedia.org/T151969 (10akosiaris) 05Open>03Resolved Key updated, should make it to the cluster in the next 30 mins. I am resolving this, feel free to... [08:01:41] (03PS13) 10Muehlenhoff: webperf: Add statsv, navtiming and coal to scap::sources [puppet] - 10https://gerrit.wikimedia.org/r/436601 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [08:02:18] (03CR) 10Muehlenhoff: [C: 032] webperf: Add statsv, navtiming and coal to scap::sources [puppet] - 10https://gerrit.wikimedia.org/r/436601 (https://phabricator.wikimedia.org/T195314) (owner: 10Krinkle) [08:06:52] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Tiziano Piccardi shell request + analytics-privatedata-users - https://phabricator.wikimedia.org/T151969 (10Miriam) Thanks @akosiaris !! [08:06:54] 10Operations, 10Research, 10SRE-Access-Requests, 10Patch-For-Review: Tiziano Piccardi shell request + analytics-privatedata-users - https://phabricator.wikimedia.org/T151969 (10Miriam) Thanks @akosiaris !! [08:09:20] 10Operations: Decommission servermon - https://phabricator.wikimedia.org/T198939 (10MoritzMuehlenhoff) [08:09:22] 10Operations: Decommission servermon - https://phabricator.wikimedia.org/T198939 (10MoritzMuehlenhoff) [08:09:55] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1098:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444166 (owner: 10Marostegui) [08:11:19] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1098:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444166 (owner: 10Marostegui) [08:11:32] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1098:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444166 (owner: 10Marostegui) [08:12:27] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1096:3318 after alter table (duration: 00m 50s) [08:12:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:12:39] (03CR) 10Jcrespo: mariadb: Add new instance to tendril to store statistics (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:13:46] (03PS3) 10Jcrespo: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) [08:15:04] !log Deploy schema change on db1113:3316 T146591 T197891 T196379 [08:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:15:09] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [08:15:10] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [08:15:10] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [08:15:38] (03CR) 10Marostegui: [C: 031] mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:16:20] (03CR) 10Marostegui: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:18:21] !log Optimize frwiki.logging on db1113:3316 - T197459 [08:18:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:18:24] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [08:19:01] (03CR) 10Marostegui: mariadb: Add new instance to tendril to store statistics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:26:00] (03CR) 10Jcrespo: mariadb: Add new instance to tendril to store statistics (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:27:13] (03CR) 10Marostegui: [C: 031] ">" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:30:39] (03PS1) 10Muehlenhoff: Remove cp3048 from Hiera/conftool [puppet] - 10https://gerrit.wikimedia.org/r/444169 [08:31:56] (03PS2) 10Ema: Remove cp3048 from Hiera/conftool [puppet] - 10https://gerrit.wikimedia.org/r/444169 (https://phabricator.wikimedia.org/T190607) (owner: 10Muehlenhoff) [08:32:19] (03CR) 10Ema: [C: 031] Remove cp3048 from Hiera/conftool [puppet] - 10https://gerrit.wikimedia.org/r/444169 (https://phabricator.wikimedia.org/T190607) (owner: 10Muehlenhoff) [08:35:57] (03CR) 10Muehlenhoff: [C: 032] Remove cp3048 from Hiera/conftool [puppet] - 10https://gerrit.wikimedia.org/r/444169 (https://phabricator.wikimedia.org/T190607) (owner: 10Muehlenhoff) [08:42:25] (03PS1) 10Muehlenhoff: Remove cp3048 from site.pp/DHCP config [puppet] - 10https://gerrit.wikimedia.org/r/444171 (https://phabricator.wikimedia.org/T190607) [08:45:29] (03PS4) 10Jcrespo: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) [08:45:31] (03CR) 10Jcrespo: "Fixing https://puppet-compiler.wmflabs.org/compiler02/11704/" [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [08:45:38] (03PS1) 10Marostegui: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444172 (https://phabricator.wikimedia.org/T146591) [08:46:01] (03PS9) 10Ema: reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) [08:46:38] (03CR) 10Ema: [C: 032] reload-vcl: do not include layer information in additional VCL labels [puppet] - 10https://gerrit.wikimedia.org/r/443904 (https://phabricator.wikimedia.org/T164609) (owner: 10Ema) [08:47:52] RECOVERY - IPsec on kafka-jumbo1004 is OK: Strongswan OK - 134 ESP OK [08:49:40] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10MoritzMuehlenhoff) [08:49:43] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10MoritzMuehlenhoff) [08:49:55] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10MoritzMuehlenhoff) >>! In T174431#4391005, @TerraCodes wrote: > Can "Deployment servers" be checked off since the two tasks next to it are resolved? Th... [08:49:59] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10MoritzMuehlenhoff) >>! In T174431#4391005, @TerraCodes wrote: > Can "Deployment servers" be checked off since the two tasks next to it are resolved? Th... [08:50:40] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444172 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:51:56] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444172 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:52:11] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444172 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [08:53:33] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 51s) [08:53:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:53:36] !log Optimize frwiki.logging on db11085 with replication (this will generate lag on s6 on labsdb hosts) - T197459 [08:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:53:39] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [09:03:52] (03CR) 10Jcrespo: "I don't think I can make this work https://puppet-compiler.wmflabs.org/compiler02/11705/db1115.eqiad.wmnet/change.db1115.eqiad.wmnet.err" [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [09:04:12] (03Abandoned) 10Jcrespo: mariadb: Add new instance to tendril to store statistics [puppet] - 10https://gerrit.wikimedia.org/r/444160 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [09:08:49] !log restart db1115 mariadb instance (this will cause temporary downtime if tendril and dbtree) [09:08:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:19:49] (03PS1) 10Jcrespo: mariadb: Increase buffer pool size of tendril [puppet] - 10https://gerrit.wikimedia.org/r/444177 (https://phabricator.wikimedia.org/T198937) [09:20:50] (03CR) 10Jcrespo: [C: 032] mariadb: Increase buffer pool size of tendril [puppet] - 10https://gerrit.wikimedia.org/r/444177 (https://phabricator.wikimedia.org/T198937) (owner: 10Jcrespo) [09:27:23] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10TerraCodes) >>! In T174431#4402506, @MoritzMuehlenhoff wrote: >>>! In T174431#4391005, @TerraCodes wrote: >> Can "Deployment servers" be checked off sin... [09:27:33] 10Operations, 10HHVM, 10Patch-For-Review, 10User-Elukey: Upgrade mw* servers to Debian Stretch (using HHVM) - https://phabricator.wikimedia.org/T174431 (10TerraCodes) >>! In T174431#4402506, @MoritzMuehlenhoff wrote: >>>! In T174431#4391005, @TerraCodes wrote: >> Can "Deployment servers" be checked off sin... [09:37:07] (03CR) 10TerraCodes: [C: 04-1] mw_maintenace: remove temp change for wikidata crons (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/441381 (https://phabricator.wikimedia.org/T192092) (owner: 10Dzahn) [09:45:47] (03CR) 10Hashar: [C: 031] "The image has been build properly yesterday :]" [puppet] - 10https://gerrit.wikimedia.org/r/438164 (owner: 10Muehlenhoff) [09:46:09] 10Operations, 10Release Pipeline, 10Release-Engineering-Team (Kanban), 10Services (watching): Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10mobrovac) [09:46:13] 10Operations, 10Release Pipeline, 10Release-Engineering-Team (Kanban), 10Services (watching): Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10mobrovac) [09:46:17] !log Deploy schema change on db1085 with replication, this will generate lag on s6 labsdb T146591 T197891 T196379 [09:46:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:46:23] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [09:46:23] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [09:46:23] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [09:48:57] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444181 [09:50:02] (03PS2) 10Giuseppe Lavagetto: mediawiki: use alternative module for the apache sites in the test env [puppet] - 10https://gerrit.wikimedia.org/r/443927 [09:50:04] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: use compile_redirects as a parser function [puppet] - 10https://gerrit.wikimedia.org/r/444182 [09:50:06] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: convert all of main.conf to individual sites [puppet] - 10https://gerrit.wikimedia.org/r/444183 [09:50:08] (03PS1) 10Giuseppe Lavagetto: mediawiki: start splitting up remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/444184 [09:50:10] (03PS1) 10Giuseppe Lavagetto: mediawiki: unify the small private wikis definitions [puppet] - 10https://gerrit.wikimedia.org/r/444185 [09:50:14] (03PS1) 10Giuseppe Lavagetto: mediawiki: move private wikis to a separate virtual host [puppet] - 10https://gerrit.wikimedia.org/r/444186 [09:50:18] (03PS1) 10Giuseppe Lavagetto: mediawiki: split all of remnant.conf into individual vhosts [puppet] - 10https://gerrit.wikimedia.org/r/444187 [09:50:25] 10Operations, 10MediaWiki-extensions-Translate, 10Language-2018-July-September, 10User-Nikerabbit, and 2 others: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293 (10Pginer-WMF) [09:50:28] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444181 (owner: 10Marostegui) [09:50:30] 10Operations, 10MediaWiki-extensions-Translate, 10Language-2018-July-September, 10User-Nikerabbit, and 2 others: 503 error attempting to open multiple projects (Wikipedia and meta wiki are loading very slowly) - https://phabricator.wikimedia.org/T195293 (10Pginer-WMF) [09:51:09] (03CR) 10Hashar: "Sorry for the delay.." [puppet] - 10https://gerrit.wikimedia.org/r/336840 (owner: 10Hashar) [09:54:14] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444181 (owner: 10Marostegui) [09:55:17] (03PS3) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [09:55:18] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 51s) [09:55:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:55:26] (03PS1) 10Marostegui: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444188 (https://phabricator.wikimedia.org/T146591) [09:56:52] (03PS2) 10Marostegui: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444188 (https://phabricator.wikimedia.org/T146591) [09:58:02] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444181 (owner: 10Marostegui) [09:58:30] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444188 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [09:59:54] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444188 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:00:14] (03PS9) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [10:00:16] (03PS10) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [10:00:18] (03PS10) 10Ema: cache_text: add misc cache::alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [10:00:20] (03PS5) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [10:00:22] (03PS3) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [10:00:47] 10Operations, 10Cassandra, 10Patch-For-Review, 10Services (doing), 10User-Eevans: Revisit default settings for c-foreach-restart - https://phabricator.wikimedia.org/T198787 (10MoritzMuehlenhoff) I had a look at the repo; the 1.0.2-1 package currently deployed to production has a date stamp from May 24 20... [10:00:50] 10Operations, 10Cassandra, 10Patch-For-Review, 10Services (doing), 10User-Eevans: Revisit default settings for c-foreach-restart - https://phabricator.wikimedia.org/T198787 (10MoritzMuehlenhoff) I had a look at the repo; the 1.0.2-1 package currently deployed to production has a date stamp from May 24 20... [10:00:53] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 51s) [10:00:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:00:56] !log Deploy schema change on db1093 T146591 T197891 T196379 [10:01:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:01:01] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [10:01:01] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [10:01:02] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [10:01:11] !log restbase depool restbase2001 to test the cassandra node driver v3.5.0 - T169009 [10:01:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:01:15] T169009: Cassandra Node.JS driver v3.2.2 issues - https://phabricator.wikimedia.org/T169009 [10:02:19] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444188 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:02:46] !log Optimize frwiki.logging on db1093 - T197459 [10:02:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:02:49] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [10:08:17] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444190 [10:12:46] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: use alternative module for the apache sites in the test env [puppet] - 10https://gerrit.wikimedia.org/r/443927 (owner: 10Giuseppe Lavagetto) [10:13:24] (03CR) 10Ema: [C: 031] "Two nits, lgtm otherwise." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [10:18:12] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki_test: use compile_redirects as a parser function [puppet] - 10https://gerrit.wikimedia.org/r/444182 (owner: 10Giuseppe Lavagetto) [10:22:32] PROBLEM - puppet last run on mwdebug1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:27:20] (03PS4) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [10:28:39] (03PS5) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) [10:28:55] (03CR) 10Vgutierrez: vcl: Bump AES128-SHA redirection to 100% (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [10:35:59] (03PS10) 10Ema: reload-vcl: label separate VCLs before compiling the main one [puppet] - 10https://gerrit.wikimedia.org/r/443905 (https://phabricator.wikimedia.org/T164609) [10:36:01] (03PS11) 10Ema: cache_text: add support for alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443906 (https://phabricator.wikimedia.org/T164609) [10:36:03] (03PS11) 10Ema: cache_text: add misc directors and alternate_domains [puppet] - 10https://gerrit.wikimedia.org/r/443907 (https://phabricator.wikimedia.org/T164609) [10:36:05] (03PS6) 10Ema: cache_text: load misc VCL as wikimedia_misc in VTC files [puppet] - 10https://gerrit.wikimedia.org/r/443930 (https://phabricator.wikimedia.org/T164609) [10:36:07] (03PS4) 10Ema: cache_text: add misc-specific VTC tests [puppet] - 10https://gerrit.wikimedia.org/r/443974 (https://phabricator.wikimedia.org/T164609) [10:37:25] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444190 (owner: 10Marostegui) [10:38:59] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444190 (owner: 10Marostegui) [10:39:15] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444190 (owner: 10Marostegui) [10:40:17] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 51s) [10:40:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:41:51] (03PS1) 10Marostegui: db-eqiad.php: Depool db1088 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444193 (https://phabricator.wikimedia.org/T146591) [10:45:12] (03CR) 10Marostegui: [C: 032] db-eqiad.php: Depool db1088 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444193 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:45:33] (03CR) 10Ema: [C: 031] vcl: Bump AES128-SHA redirection to 100% [puppet] - 10https://gerrit.wikimedia.org/r/444005 (https://phabricator.wikimedia.org/T192555) (owner: 10Vgutierrez) [10:46:41] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1088 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444193 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:48:35] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 50s) [10:48:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:48:44] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1088 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444193 (https://phabricator.wikimedia.org/T146591) (owner: 10Marostegui) [10:48:46] !log Deploy schema change on db1088 T146591 T197891 T196379 [10:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:48:52] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [10:48:52] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [10:48:52] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [10:49:33] !log Optimize frwiki.logging on db1088 - T197459 [10:49:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:49:36] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [10:55:05] (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1088" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444194 [10:58:43] (03CR) 10Marostegui: [C: 032] Revert "db-eqiad.php: Depool db1088" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444194 (owner: 10Marostegui) [11:00:09] (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1088" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444194 (owner: 10Marostegui) [11:00:25] (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1088" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444194 (owner: 10Marostegui) [11:01:27] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 50s) [11:01:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:02:21] !log Deploy schema change on db1061 (s6 primary master) T146591 T197891 T196379 [11:02:38] (03PS1) 10Giuseppe Lavagetto: compile_redirects: force utf-8 encoding when reading redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/444197 [11:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:02:51] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [11:02:54] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [11:02:57] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [11:03:51] (03CR) 10Giuseppe Lavagetto: [C: 032] compile_redirects: force utf-8 encoding when reading redirects.dat [puppet] - 10https://gerrit.wikimedia.org/r/444197 (owner: 10Giuseppe Lavagetto) [11:04:36] !log Optimize frwiki.logging on db1061 (s6 primary master) - T197459 [11:04:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:04:58] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [11:08:25] !log Optimize shwiki.logging on db2043 (codfw s3 master), this will generate lag on s3 codfw - T197459 [11:08:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:08:31] RECOVERY - puppet last run on mwdebug1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:09:49] !log Optimize shwiki.logging on s3 eqiad hosts (one by one) T197459 [11:09:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:10:12] PROBLEM - Restbase root url on restbase2001 is CRITICAL: connect to address 10.192.16.152 and port 7231: Connection refused [11:11:12] (03PS1) 10Muehlenhoff: Enable base::service_auto_restart for Memcached Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444198 (https://phabricator.wikimedia.org/T135991) [11:12:11] PROBLEM - Check systemd state on restbase2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [11:12:48] we're working on it ^^ [11:15:34] !log Optimize {itwiki,enwiktionary,nlwiki}.logging on db2035 (codfw s2 master), this will generate lag on s2 codfw - T197459 [11:15:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:15:37] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [11:18:21] scheduled downtime for rb2001 ^ [11:36:39] (03PS1) 10Arturo Borrero Gonzalez: openstack: eqiad1: enable neutron-l3-agent [puppet] - 10https://gerrit.wikimedia.org/r/444204 (https://phabricator.wikimedia.org/T196633) [11:50:57] (03CR) 10Arturo Borrero Gonzalez: [C: 032] "Compiler seems good:" [puppet] - 10https://gerrit.wikimedia.org/r/444204 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez) [11:51:20] !log Optimize {itwiki,enwiktionary,nlwiki}.logging on dbstore1002:s2 - T197459 [11:51:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:51:24] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [11:56:42] PROBLEM - Check systemd state on labnet1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [11:58:01] PROBLEM - puppet last run on labnet1004 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[neutron-linuxbridge-agent] [11:59:03] silencing ^^^ [12:03:02] RECOVERY - puppet last run on labnet1004 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [12:03:21] RECOVERY - Check systemd state on labnet1004 is OK: OK - running: The system is fully operational [12:18:02] !log Deploy schema change on s2 codfw master (db2035) with replication, this will generate lag on s2 codfw T146591 T197891 T196379 [12:18:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:18:08] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [12:18:08] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [12:18:09] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [12:23:39] (03CR) 10Filippo Giunchedi: [C: 031] Enable base::service_auto_restart for Memcached Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444198 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [12:27:22] (03PS1) 10Muehlenhoff: Enable base::service_auto_restart for mysqld Prometheus exporter [puppet] - 10https://gerrit.wikimedia.org/r/444210 (https://phabricator.wikimedia.org/T135991) [12:29:33] !log mobrovac@deploy1001 Started deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0 [12:29:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:30:15] !log Deploy schema change on dbstore1002:s2 and db1122 T146591 T197891 T196379 [12:30:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:30:20] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [12:30:20] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [12:30:21] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [12:30:41] RECOVERY - Check systemd state on restbase2001 is OK: OK - running: The system is fully operational [12:31:51] RECOVERY - Restbase root url on restbase2001 is OK: HTTP OK: HTTP/1.1 200 - 15984 bytes in 0.125 second response time [12:33:49] !log Deploy schema change on db1105:3312 - T146591 T197891 T196379 [12:33:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:47] !log Optimize {itwiki,enwiktionary,nlwiki}.logging on db1105 and db1122 - T197459 [12:38:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:50] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [12:41:23] 10Operations, 10procurement: leases expiring on labvirt1010 and 1011 - https://phabricator.wikimedia.org/T198762 (10faidon) The answer is that it's theoretically possible but we don't really know any details yet, as we haven't had a lease expire so far. We would have to figure both the process and the price, w... [12:41:25] 10Operations, 10procurement: leases expiring on labvirt1010 and 1011 - https://phabricator.wikimedia.org/T198762 (10faidon) The answer is that it's theoretically possible but we don't really know any details yet, as we haven't had a lease expire so far. We would have to figure both the process and the price, w... [12:41:48] (03PS2) 10Giuseppe Lavagetto: mediawiki_test: convert all of main.conf to individual sites [puppet] - 10https://gerrit.wikimedia.org/r/444183 [12:45:05] !log mobrovac@deploy1001 Finished deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0 (duration: 15m 32s) [12:45:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:48:59] 10Operations, 10Patch-For-Review: Alert when used_memory gets too high for redis queues - https://phabricator.wikimedia.org/T118331 (10Joe) 05Open>03declined [12:49:04] 10Operations, 10Patch-For-Review: Alert when used_memory gets too high for redis queues - https://phabricator.wikimedia.org/T118331 (10Joe) 05Open>03declined [12:49:24] 10Operations, 10Patch-For-Review: Alert when used_memory gets too high for redis queues - https://phabricator.wikimedia.org/T118331 (10Joe) Closing as declined as we've removed the redis-based jobqueue. [12:49:30] 10Operations, 10Patch-For-Review: Alert when used_memory gets too high for redis queues - https://phabricator.wikimedia.org/T118331 (10Joe) Closing as declined as we've removed the redis-based jobqueue. [12:51:37] (03CR) 10Muehlenhoff: "PCC: https://puppet-compiler.wmflabs.org/compiler02/11709/" [puppet] - 10https://gerrit.wikimedia.org/r/444210 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [12:55:17] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki_test: convert all of main.conf to individual sites [puppet] - 10https://gerrit.wikimedia.org/r/444183 (owner: 10Giuseppe Lavagetto) [12:56:15] (03PS1) 10Giuseppe Lavagetto: Revert "mediawiki_test: convert all of main.conf to individual sites" [puppet] - 10https://gerrit.wikimedia.org/r/444214 [12:56:44] <_joe_> uhm [12:57:34] !log Restarting Zuul [12:57:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:00:25] (03PS1) 10Giuseppe Lavagetto: mediawiki: followup for Idb594ddeb [puppet] - 10https://gerrit.wikimedia.org/r/444215 [13:00:47] !log installing mercurial security updates [13:00:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:01:16] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: followup for Idb594ddeb [puppet] - 10https://gerrit.wikimedia.org/r/444215 (owner: 10Giuseppe Lavagetto) [13:03:54] (03PS1) 10Filippo Giunchedi: graphite: take graphite200[12] out of service [puppet] - 10https://gerrit.wikimedia.org/r/444217 (https://phabricator.wikimedia.org/T196483) [13:04:16] (03CR) 10jerkins-bot: [V: 04-1] graphite: take graphite200[12] out of service [puppet] - 10https://gerrit.wikimedia.org/r/444217 (https://phabricator.wikimedia.org/T196483) (owner: 10Filippo Giunchedi) [13:05:13] (03PS2) 10Filippo Giunchedi: graphite: take graphite200[12] out of service [puppet] - 10https://gerrit.wikimedia.org/r/444217 (https://phabricator.wikimedia.org/T196483) [13:05:17] !log installing bouncycastle security updates [13:05:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:07:51] PROBLEM - puppet last run on mwdebug1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:09:39] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: use local defines for mediawiki::site [puppet] - 10https://gerrit.wikimedia.org/r/444218 [13:13:45] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki_test: use local defines for mediawiki::site [puppet] - 10https://gerrit.wikimedia.org/r/444218 (owner: 10Giuseppe Lavagetto) [13:15:18] (03PS1) 10Filippo Giunchedi: grafana: use host-overview in favour of server-board for featured dashboard [puppet] - 10https://gerrit.wikimedia.org/r/444219 (https://phabricator.wikimedia.org/T178690) [13:18:01] RECOVERY - puppet last run on mwdebug1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:20:45] (03PS1) 10Arturo Borrero Gonzalez: openstack: bootstrap: neutron: add more hints about l3agents [puppet] - 10https://gerrit.wikimedia.org/r/444222 (https://phabricator.wikimedia.org/T196633) [13:22:15] !log Deploy schema change on db1103:3312 - T146591 T197891 T196379 [13:22:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:21] T196379: Schema change: Add unique index on archive.ar_rev_id - https://phabricator.wikimedia.org/T196379 [13:22:21] T197891: Schema change to drop default from externallinks.el_index_60 - https://phabricator.wikimedia.org/T197891 [13:22:22] T146591: Add a primary key to l10n_cache - https://phabricator.wikimedia.org/T146591 [13:22:33] (03CR) 10Jcrespo: [C: 031] "I don't know the details of such resouce, but assuming you do, just merge- worst case scenario, prometheus exporter explodes- but that sho" [puppet] - 10https://gerrit.wikimedia.org/r/444210 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff) [13:22:40] !log Optimize {itwiki,enwiktionary,nlwiki}.logging on db1103:3312 - T197459 [13:22:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:22:44] T197459: Optimize logging table - https://phabricator.wikimedia.org/T197459 [13:26:54] (03PS1) 10Arturo Borrero Gonzalez: openstack: eqiad1: add dhcp agents configuration [puppet] - 10https://gerrit.wikimedia.org/r/444224 (https://phabricator.wikimedia.org/T196633) [13:29:48] (03PS1) 10Alexandros Kosiaris: proton: Add discovery hiera [puppet] - 10https://gerrit.wikimedia.org/r/444225 [13:31:11] (03PS1) 10Alexandros Kosiaris: proton: Add proton.discovery.wmnet RR [dns] - 10https://gerrit.wikimedia.org/r/444226 (https://phabricator.wikimedia.org/T186748) [13:31:24] (03CR) 10jerkins-bot: [V: 04-1] proton: Add proton.discovery.wmnet RR [dns] - 10https://gerrit.wikimedia.org/r/444226 (https://phabricator.wikimedia.org/T186748) (owner: 10Alexandros Kosiaris) [13:31:31] (03PS2) 10Alexandros Kosiaris: proton: Add discovery hiera [puppet] - 10https://gerrit.wikimedia.org/r/444225 (https://phabricator.wikimedia.org/T186748) [13:31:37] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "you also need to add the data to conftool" [puppet] - 10https://gerrit.wikimedia.org/r/444225 (https://phabricator.wikimedia.org/T186748) (owner: 10Alexandros Kosiaris) [13:32:42] (03CR) 10Giuseppe Lavagetto: "You also need to add the stub disc to config-geo-test" [dns] - 10https://gerrit.wikimedia.org/r/444226 (https://phabricator.wikimedia.org/T186748) (owner: 10Alexandros Kosiaris) [13:36:36] !log installing openldap security updates on trusty [13:36:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:57:28] (03PS13) 10Andrew Bogott: Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 [14:08:41] (03PS1) 10Muehlenhoff: Enable base::service_auto_restart for SSH [puppet] - 10https://gerrit.wikimedia.org/r/444230 (https://phabricator.wikimedia.org/T135991) [14:17:05] (03CR) 10Andrew Bogott: [C: 032] Add an explicit keystone_host var to hiera [puppet] - 10https://gerrit.wikimedia.org/r/444049 (owner: 10Andrew Bogott) [14:23:34] 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Better organization for ops grafana dashboards - https://phabricator.wikimedia.org/T178690 (10fgiunchedi) >>! In T178690#3890148, @faidon wrote: > @ori recently sent his thoughts about this to the ops list, and I found it a very eloquent de... [14:23:36] 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Better organization for ops grafana dashboards - https://phabricator.wikimedia.org/T178690 (10fgiunchedi) >>! In T178690#3890148, @faidon wrote: > @ori recently sent his thoughts about this to the ops list, and I found it a very eloquent de... [14:24:18] mhhh looks like wikibugs or some other thing is generating double notifications? [14:28:07] (03PS1) 10Elukey: Enable snappy compression for Varnishkafka eventlogging/statsv [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) [14:28:15] probably two processes running on labs again [14:28:43] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:29:50] (03PS1) 10Andrew Bogott: Define profile::openstack::main::keystone_host on VMs [puppet] - 10https://gerrit.wikimedia.org/r/444234 [14:31:06] (03CR) 10Andrew Bogott: [C: 032] Define profile::openstack::main::keystone_host on VMs [puppet] - 10https://gerrit.wikimedia.org/r/444234 (owner: 10Andrew Bogott) [14:35:05] (03PS2) 10Elukey: Enable snappy compression for Varnishkafka eventlogging [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) [14:36:36] (03PS1) 10Andrew Bogott: Nodepool: pass in keystone_host rather than nova_controller [puppet] - 10https://gerrit.wikimedia.org/r/444236 [14:37:48] <_joe_> godog: yes [14:37:58] <_joe_> I noticed that earlier, thought it was just a glitch [14:38:21] (03CR) 10Elukey: "Andrew let me know what you think about this change.. IIUC this will change the messages stored in the EL topic, since they'll be compress" [puppet] - 10https://gerrit.wikimedia.org/r/444232 (https://phabricator.wikimedia.org/T198070) (owner: 10Elukey) [14:39:25] (03CR) 10Mforns: [C: 031] "Thanks a lot Elukey for tackling this so fast :D" [puppet] - 10https://gerrit.wikimedia.org/r/444154 (https://phabricator.wikimedia.org/T198766) (owner: 10Elukey) [14:39:28] (03CR) 10Andrew Bogott: [C: 032] Nodepool: pass in keystone_host rather than nova_controller [puppet] - 10https://gerrit.wikimedia.org/r/444236 (owner: 10Andrew Bogott) [14:41:00] ugh, Reedy _joe_ know how to fix it? [14:42:02] (03PS1) 10Muehlenhoff: Blacklist floppy driver [puppet] - 10https://gerrit.wikimedia.org/r/444238 [14:42:57] _joe_ same thing on the analytics chan, I was wondering about that [14:43:09] Cc godog --^ [14:44:03] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:47:21] https://www.mediawiki.org/wiki/Wikibugs#Muting_wikibugs [14:47:31] check the jobs running, and kill one [14:47:38] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10MoritzMuehlenhoff) a:03Papaul [14:47:42] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10MoritzMuehlenhoff) a:03Papaul [14:48:12] * Reedy looks [14:48:56] looks right on job [14:48:57] s [14:49:08] thanks Reedy ! I was looking too but indeed seems right [14:49:52] possibly something orphaned somewhere [14:51:30] (03PS2) 10Giuseppe Lavagetto: mediawiki: start splitting up remnant.conf [puppet] - 10https://gerrit.wikimedia.org/r/444184 [14:51:32] (03PS2) 10Giuseppe Lavagetto: mediawiki: unify the small private wikis definitions [puppet] - 10https://gerrit.wikimedia.org/r/444185 [14:51:34] (03PS2) 10Giuseppe Lavagetto: mediawiki: move private wikis to a separate virtual host [puppet] - 10https://gerrit.wikimedia.org/r/444186 [14:51:36] (03PS2) 10Giuseppe Lavagetto: mediawiki: split all of remnant.conf into individual vhosts [puppet] - 10https://gerrit.wikimedia.org/r/444187 [14:51:38] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: split wikimania.conf [puppet] - 10https://gerrit.wikimedia.org/r/444240 [14:51:40] (03PS1) 10Giuseppe Lavagetto: mediawiki_test: complete the transition to one wiki per template. [puppet] - 10https://gerrit.wikimedia.org/r/444241 [14:52:16] * Reedy waits [14:52:16] 10Operations: TEST - https://phabricator.wikimedia.org/T198972 (10Reedy) [14:52:25] * Reedy waits some more [14:52:34] LGTM [14:52:54] 10Operations: TEST - https://phabricator.wikimedia.org/T198972 (10Reedy) 05Open>03Invalid [14:53:18] looks good, thanks! [14:54:17] (03CR) 10jerkins-bot: [V: 04-1] mediawiki_test: complete the transition to one wiki per template. [puppet] - 10https://gerrit.wikimedia.org/r/444241 (owner: 10Giuseppe Lavagetto) [14:55:05] (03CR) 10Filippo Giunchedi: [C: 031] Blacklist floppy driver [puppet] - 10https://gerrit.wikimedia.org/r/444238 (owner: 10Muehlenhoff) [14:57:08] (03PS1) 10Andrew Bogott: openstack cumin: use the new keystone_host var instead of nova_controller [puppet] - 10https://gerrit.wikimedia.org/r/444243 [14:59:24] (03CR) 10Andrew Bogott: [C: 032] openstack cumin: use the new keystone_host var instead of nova_controller [puppet] - 10https://gerrit.wikimedia.org/r/444243 (owner: 10Andrew Bogott) [15:10:27] (03CR) 10Andrew Bogott: [C: 031] Blacklist floppy driver [puppet] - 10https://gerrit.wikimedia.org/r/444238 (owner: 10Muehlenhoff) [15:17:10] 10Operations, 10monitoring, 10Patch-For-Review, 10User-fgiunchedi: Better organization for ops grafana dashboards - https://phabricator.wikimedia.org/T178690 (10Eevans) >>! In T178690#4403664, @fgiunchedi wrote: > [ ... ] > * restbase (and restbase staging) dashboards I believe can be deleted for the most... [15:35:58] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Jc86035) [15:47:23] (03PS1) 10Dbarratt: Enable anonymous cookie blocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444246 (https://phabricator.wikimedia.org/T192017) [15:47:45] (03PS5) 10Mobrovac: restbase: cleanup remaining detritus from storage transition [puppet] - 10https://gerrit.wikimedia.org/r/443114 (https://phabricator.wikimedia.org/T191659) (owner: 10Eevans) [15:54:37] (03PS1) 10Mobrovac: RESTBase: Disable cassandra-metrics-collector [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) [15:55:43] (03PS1) 10Eevans: Merge master into debian [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/444248 [15:55:45] (03PS1) 10Eevans: Updated for 1.0.3-1 package release. [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/444249 [15:56:09] (03CR) 10Eevans: [V: 032 C: 032] Merge master into debian [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/444248 (owner: 10Eevans) [15:56:34] (03CR) 10Eevans: [C: 032] Updated for 1.0.3-1 package release. [debs/cassandra-tools-wmf] - 10https://gerrit.wikimedia.org/r/444249 (owner: 10Eevans) [16:01:49] (03CR) 10Mobrovac: "PCC lookin' good: https://puppet-compiler.wmflabs.org/compiler02/11715/" [puppet] - 10https://gerrit.wikimedia.org/r/444247 (https://phabricator.wikimedia.org/T186567) (owner: 10Mobrovac) [16:04:37] 10Operations, 10Cassandra, 10Patch-For-Review, 10Services (doing), 10User-Eevans: Revisit default settings for c-foreach-restart - https://phabricator.wikimedia.org/T198787 (10Eevans) So (just to be clear), I use `gbp` on this repo, and the Debian packaging is in the `debian` branch, changes to `master`... [16:06:38] !log labcontrol1003:~# /sbin/reboot T198950 [16:06:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:21:48] 10Operations, 10Cassandra, 10Patch-For-Review, 10Services (doing), 10User-Eevans: Revisit default settings for c-foreach-restart - https://phabricator.wikimedia.org/T198787 (10Eevans) >>! In T198787#4403893, @Eevans wrote: > So (just to be clear), I use `gbp` on this repo, and the Debian packaging is in... [16:24:11] PROBLEM - kubelet operational latencies on kubernetes1001 is CRITICAL: instance=kubernetes1001.eqiad.wmnet operation_type={create_container,start_container} https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [16:25:12] RECOVERY - kubelet operational latencies on kubernetes1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1 [16:53:06] 10Operations, 10Analytics, 10Analytics-Kanban, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) >>! In T198623#4397016, @elukey wrote: > I am pretty sure that this is a pre-scap thing, we should drop it :) Great! > Other thi... [16:59:44] 10Operations, 10Analytics, 10Analytics-Kanban, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10elukey) @ayounsi are we sure that we can touch common-infrastructure4 without affecting anything else? Is there any trace of who made it... [17:13:08] (03PS1) 10Rush: openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) [17:13:42] (03CR) 10jerkins-bot: [V: 04-1] openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) (owner: 10Rush) [17:14:14] (03PS2) 10Rush: openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) [17:14:45] (03CR) 10jerkins-bot: [V: 04-1] openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) (owner: 10Rush) [17:18:51] (03PS3) 10Rush: openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) [17:20:19] 10Operations, 10Analytics, 10Analytics-Kanban, 10netops, 10Patch-For-Review: Review analytics-in4 rules on cr1/cr2 eqiad - https://phabricator.wikimedia.org/T198623 (10ayounsi) >>! In T198623#4403989, @elukey wrote: > @ayounsi are we sure that we can touch common-infrastructure4 without affecting anythi... [17:20:45] (03CR) 10Rush: "http://puppet-compiler.wmflabs.org/11716/" [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) (owner: 10Rush) [17:38:59] (03CR) 10Andrew Bogott: [C: 032] openstack: stop nova-api from managing iptables for metadata service [puppet] - 10https://gerrit.wikimedia.org/r/444254 (https://phabricator.wikimedia.org/T198950) (owner: 10Rush) [17:40:35] 10Operations, 10Cloud-Services, 10cloud-services-team (Kanban): Ops access request - https://phabricator.wikimedia.org/T198900 (10bd808) @dchen The easiest way to do the verification for [[https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication|a 2FA reset on Wikitech]] is for you... [18:08:08] !log reset 2FA for User:Howcheng [18:08:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:49:18] 10Operations, 10ops-codfw, 10Traffic, 10netops: switch port configuration for lvs200[7-10] - https://phabricator.wikimedia.org/T196946 (10ayounsi) NIC2/3/4 configured as of the current task description. [19:15:07] (03PS1) 10Smalyshev: Enable kafka poller on test hosts [puppet] - 10https://gerrit.wikimedia.org/r/444265 [20:03:35] 10Operations, 10Mail, 10Phabricator, 10Patch-For-Review, and 3 others: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916 (10greg) (thanks @herron ) [20:09:26] 10Operations, 10Release-Engineering-Team, 10Scap, 10Scoring-platform-team: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842 (10greg) I think to do much on this we'll need some performance numbers. Also, if this isn't 100% addressed by git-lfs reducing t... [20:54:12] 10Operations, 10MediaWiki-Parser, 10MediaWiki-Platform-Team, 10Parsing-Team, and 2 others: Servers using tidy-html5 are rendering pages differently, especially with - https://phabricator.wikimedia.org/T193414 (10Jdforrester-WMF) 05Open>03declined Now that we've stopped using Tidy in production, t... [22:32:59] (03CR) 10Bstorm: "Sorry, I saw the reply a little late. I'll merge and re-image on Monday so we don't go into the weekend with potential problems." [puppet] - 10https://gerrit.wikimedia.org/r/443799 (https://phabricator.wikimedia.org/T197246) (owner: 10Alexandros Kosiaris) [22:49:12] PROBLEM - Check systemd state on ms-be1037 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [23:08:57] (03CR) 10Jforrester: [C: 031] "Surely it's now time to do this?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440745 (owner: 10MacFan4000) [23:11:03] (03CR) 10Reedy: "T197669" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440745 (owner: 10MacFan4000) [23:19:31] (03PS1) 10Catrope: Enable PageTriage for Draft namespace on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444327 (https://phabricator.wikimedia.org/T198898) [23:34:10] (03CR) 10Jforrester: [C: 04-2] "Aha, cool. In that case, C-2 until T197669 is resolved." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440745 (owner: 10MacFan4000) [23:38:41] 10Operations, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 (10Krinkle) Are there known unresolved issues with the new host? It seems `deployment-tin` is still used as primary... [23:46:07] (03CR) 10Catrope: [C: 032] Enable PageTriage for Draft namespace on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444327 (https://phabricator.wikimedia.org/T198898) (owner: 10Catrope) [23:46:40] 10Operations, 10MediaWiki-Platform-Team, 10HHVM, 10TechCom-RFC (TechCom-Approved), 10User-ArielGlenn: Migrate to PHP 7 in WMF production - https://phabricator.wikimedia.org/T176370 (10Krinkle) 05Resolved>03Open This task still has lots of open sub tasks. While dumps are now running fine on PHP 7, the... [23:47:40] (03PS1) 10Amire80: Remove priyankaivy.blogspot.com from Planet [puppet] - 10https://gerrit.wikimedia.org/r/444328 [23:47:56] (03Merged) 10jenkins-bot: Enable PageTriage for Draft namespace on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444327 (https://phabricator.wikimedia.org/T198898) (owner: 10Catrope) [23:48:09] (03CR) 10jenkins-bot: Enable PageTriage for Draft namespace on beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/444327 (https://phabricator.wikimedia.org/T198898) (owner: 10Catrope) [23:54:02] 10Operations, 10Patch-For-Review: setup replacements for maintenance_server (terbium, wasat) on Stretch - https://phabricator.wikimedia.org/T192092 (10Krinkle)