[00:05:41] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM! But please re-test it against af-netbox (add/modify/delete instances) and potential error cases." [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[00:09:47] <wikibugs>	 (03PS5) 10Tim Eulitz: Set up exceptions for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494270 (https://phabricator.wikimedia.org/T217436)
[00:10:10] <wikibugs>	 (03PS2) 10Tim Eulitz: Add default user config for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436)
[00:14:05] <wikibugs>	 (03PS12) 10CRusnov: Add system timer for running ganeti->netbox sync. [puppet] - 10https://gerrit.wikimedia.org/r/493774 (https://phabricator.wikimedia.org/T215229)
[00:16:20] <icinga-wm>	 PROBLEM - citoid endpoints health on scb1001 is CRITICAL: /api (bad URL) timed out before a response was received
[00:17:28] <icinga-wm>	 RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy
[00:35:24] <wikibugs>	 (03PS1) 10Bstorm: toolforge: limit file size to 50 GB [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508)
[02:33:19] <wikibugs>	 (03PS1) 10EBernhardson: Allow apifeatureusage to work in beta cluster [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156)
[02:36:06] <wikibugs>	 (03CR) 10Alex Monk: Allow apifeatureusage to work in beta cluster (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156) (owner: 10EBernhardson)
[02:36:23] <wikibugs>	 (03CR) 10Alex Monk: [C: 04-1] "new host does not exist" [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156) (owner: 10EBernhardson)
[02:44:00] <wikibugs>	 (03PS2) 10EBernhardson: Allow apifeatureusage to work in beta cluster [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156)
[02:55:40] <wikibugs>	 10Operations, 10Parsoid-PHP: Install PHP7 on scandium - https://phabricator.wikimedia.org/T213493 (10ssastry)
[04:00:04] <jouncebot>	 kart_: I seem to be stuck in Groundhog week. Sigh. Time for (yet another)  deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T0400).
[04:11:07] <kart_>	 !log Started manual run of unpublished ContentTranslation draft purge script (T217818)
[04:11:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:11:10] <stashbot>	 T217818: Run unpublished draft purge script for CX (Week of 03/10) - https://phabricator.wikimedia.org/T217818
[05:04:34] <wikibugs>	 (03PS7) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[05:06:46] <wikibugs>	 (03PS8) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[06:11:37] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496098
[06:12:42] <wikibugs>	 (03PS1) 10BryanDavis: Add helpers for local testing [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496099
[06:12:44] <wikibugs>	 (03PS1) 10BryanDavis: Tweak build reporting output [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496100
[06:12:46] <wikibugs>	 (03PS1) 10BryanDavis: all: install locales used in Toolforge grid [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496101 (https://phabricator.wikimedia.org/T218072)
[06:12:48] <wikibugs>	 (03PS1) 10BryanDavis: php72: tideways & switch to component/php72 [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496102 (https://phabricator.wikimedia.org/T202825)
[06:12:50] <wikibugs>	 (03PS1) 10BryanDavis: php: install php5-xdebug [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496103 (https://phabricator.wikimedia.org/T202825)
[06:12:52] <wikibugs>	 (03PS1) 10BryanDavis: python35: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496104
[06:12:54] <wikibugs>	 (03PS1) 10BryanDavis: node10: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496105 (https://phabricator.wikimedia.org/T195103)
[06:12:56] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496098 (owner: 10Marostegui)
[06:13:55] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496098 (owner: 10Marostegui)
[06:15:21] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1096 (duration: 01m 07s)
[06:15:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:21:03] <marostegui>	 !log Testing snapshotting on db1117:3321 to > dbstore1001 - T210292
[06:21:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:21:06] <stashbot>	 T210292: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292
[06:22:33] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496098 (owner: 10Marostegui)
[06:24:27] <kart_>	 !log Finished manual run of unpublished ContentTranslation draft purge script (T217818)
[06:24:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:24:32] <stashbot>	 T217818: Run unpublished draft purge script for CX (Week of 03/10) - https://phabricator.wikimedia.org/T217818
[06:30:09] <icinga-wm>	 PROBLEM - puppet last run on ganeti1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled]
[06:30:35] <icinga-wm>	 PROBLEM - puppet last run on phab1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled]
[06:31:17] <icinga-wm>	 PROBLEM - puppet last run on acmechief1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ferm/conf.d/00_main]
[06:31:31] <icinga-wm>	 PROBLEM - puppet last run on labmon1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/profile.d/field.sh]
[06:40:44] <marostegui>	 !log Stop MySQL on db1096 for upgrade
[06:40:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:52:13] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Slowly repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496108
[06:53:50] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Slowly repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496108 (owner: 10Marostegui)
[06:54:47] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Slowly repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496108 (owner: 10Marostegui)
[06:56:00] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Slowly repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496108 (owner: 10Marostegui)
[06:56:00] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool db1096 (duration: 00m 55s)
[06:56:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:56:07] <icinga-wm>	 RECOVERY - puppet last run on ganeti1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:56:31] <icinga-wm>	 RECOVERY - puppet last run on phab1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:11] <icinga-wm>	 RECOVERY - puppet last run on acmechief1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:57:23] <icinga-wm>	 RECOVERY - puppet last run on labmon1002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[06:58:11] <wikibugs>	 (03PS2) 10Elukey: Set notifications disabled for analytics-tool1004 [puppet] - 10https://gerrit.wikimedia.org/r/495876
[06:58:37] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: More traffic to db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496109
[06:58:59] <marostegui>	 !log Upgrade MySQL and kernel on db2036
[06:59:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:59:38] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10ema) >>! In T213214#5018951, @The_RedBurn wrote: > I did noti...
[07:02:26] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Set notifications disabled for analytics-tool1004 [puppet] - 10https://gerrit.wikimedia.org/r/495876 (owner: 10Elukey)
[07:02:33] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10The_RedBurn) >>! In T213214#5020003, @ema wrote: >>>! In T213...
[07:03:14] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration, 10User-Ryasmeen: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10ema) 05Open→03Resolved a:03ema >>! I...
[07:06:16] <wikibugs>	 (03PS1) 10Elukey: role::aqs: use nodejs-10 for the aqs service [puppet] - 10https://gerrit.wikimedia.org/r/496110 (https://phabricator.wikimedia.org/T210706)
[07:09:48] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: More traffic to db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496109 (owner: 10Marostegui)
[07:10:45] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: More traffic to db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496109 (owner: 10Marostegui)
[07:11:10] <wikibugs>	 (03PS6) 10Vgutierrez: acme-chief: Store certificates in unique directories [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295)
[07:11:12] <wikibugs>	 (03PS8) 10Vgutierrez: acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295)
[07:11:14] <wikibugs>	 (03CR) 10Vgutierrez: "> Patch Set 7:" (038 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[07:12:00] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: More traffic to db1096 (duration: 00m 55s)
[07:12:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:12:42] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[07:13:56] <marostegui>	 !log Test snapshot dbstore1001:3311 to dbstore1001 - T210292
[07:13:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:13:59] <stashbot>	 T210292: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292
[07:15:39] <wikibugs>	 (03PS7) 10Vgutierrez: acme-chief: Store certificates in unique directories [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295)
[07:15:39] <wikibugs>	 (03PS9) 10Vgutierrez: acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295)
[07:17:07] <wikibugs>	 (03CR) 10Vgutierrez: acme-chief: Store certificates in unique directories (031 comment) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[07:17:11] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Fully repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496111
[07:18:35] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: More traffic to db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496109 (owner: 10Marostegui)
[07:23:39] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Fully repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496111 (owner: 10Marostegui)
[07:24:37] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Fully repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496111 (owner: 10Marostegui)
[07:25:48] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Fully repool db1096 (duration: 00m 58s)
[07:25:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:26:58] <moritzm>	 !log upgrading remaining app servers in eqiad to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates)
[07:26:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:27:28] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496112
[07:28:39] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496112 (owner: 10Marostegui)
[07:29:35] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496112 (owner: 10Marostegui)
[07:29:47] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Fully repool db1096 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496111 (owner: 10Marostegui)
[07:29:49] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1093 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496112 (owner: 10Marostegui)
[07:30:41] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 00m 54s)
[07:30:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:35:08] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), 10User-Elukey: Consider raising Memcached MWObject cache memory size limit - https://phabricator.wikimedia.org/T217731 (10elukey) a:03elukey
[07:41:30] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), 10User-Elukey: Consider raising Memcached MWObject cache memory size limit - https://phabricator.wikimedia.org/T217731 (10elukey) Very interesting results for mc1019 after a day of metrics.   * https://grafana.wikimedia.org/...
[07:48:25] <wikibugs>	 (03CR) 10WMDE-Fisch: [C: 03+1] Set up exceptions for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494270 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[07:48:34] <wikibugs>	 (03CR) 10WMDE-Fisch: Add default user config for rollback confirmation (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[07:54:29] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496114
[07:57:44] <wikibugs>	 (03CR) 10GTirloni: [C: 03+1] toolforge: limit file size to 50 GB [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508) (owner: 10Bstorm)
[08:07:27] <icinga-wm>	 ACKNOWLEDGEMENT - Check systemd state on phab1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. daniel_zahn https://phabricator.wikimedia.org/T215332
[08:08:37] <mutante>	 gtirloni: labmon1001 is low on disk on /srv (98%). should i make a ticket for it?
[08:09:00] <gtirloni>	 mutante: yes, please. i can take a look
[08:09:13] <mutante>	 ok, doing
[08:09:16] <gtirloni>	 thanks
[08:09:55] <moritzm>	 !log upgrading job runners in eqiad to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates)
[08:09:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:12:09] <mutante>	 gtirloni: https://phabricator.wikimedia.org/T218185
[08:12:47] <icinga-wm>	 ACKNOWLEDGEMENT - Disk space on labmon1001 is CRITICAL: DISK CRITICAL - free space: /srv 57435 MB (2% inode=93%): daniel_zahn https://phabricator.wikimedia.org/T218185
[08:12:55] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496114 (owner: 10Marostegui)
[08:13:52] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496114 (owner: 10Marostegui)
[08:14:05] <icinga-wm>	 ACKNOWLEDGEMENT - Memory correctable errors -EDAC- on wtp2013 is CRITICAL: 195 ge 4 daniel_zahn https://phabricator.wikimedia.org/T194174 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=wtp2013&var-datasource=codfw+prometheus/ops
[08:14:51] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1093" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496114 (owner: 10Marostegui)
[08:14:56] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1093 (duration: 00m 55s)
[08:14:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:19:07] <wikibugs>	 (03PS1) 10Dzahn: set phab1002 as a spare::system [puppet] - 10https://gerrit.wikimedia.org/r/496116 (https://phabricator.wikimedia.org/T215332)
[08:19:59] <icinga-wm>	 PROBLEM - puppet last run on mw1330 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 7 minutes ago with 14 failures. Failed resources (up to 3 shown): Package[php7.2-opcache],Package[php7.2-gd],Package[php7.2-gmp],Package[php7.2-intl]
[08:23:01] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1113:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496117
[08:23:36] <wikibugs>	 (03PS1) 10GTirloni: toolforge: Allow prometheus to create up to 150 Linux threads [puppet] - 10https://gerrit.wikimedia.org/r/496118 (https://phabricator.wikimedia.org/T218186)
[08:23:57] <wikibugs>	 (03PS1) 10Dzahn: replace phab1002 with phab1003 [puppet] - 10https://gerrit.wikimedia.org/r/496119 (https://phabricator.wikimedia.org/T215335)
[08:24:02] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1113:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496117 (owner: 10Marostegui)
[08:24:34] <wikibugs>	 (03CR) 10GTirloni: [C: 03+2] toolforge: Allow prometheus to create up to 150 Linux threads [puppet] - 10https://gerrit.wikimedia.org/r/496118 (https://phabricator.wikimedia.org/T218186) (owner: 10GTirloni)
[08:25:06] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1113:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496117 (owner: 10Marostegui)
[08:25:34] <wikibugs>	 (03PS1) 10Dzahn: update mariadb grants from phab1002 to phab1003 [puppet] - 10https://gerrit.wikimedia.org/r/496120
[08:26:01] <wikibugs>	 (03CR) 10Dzahn: [C: 04-2] "needs to wait until we have the new hardware and IP but should not be forgotten" [puppet] - 10https://gerrit.wikimedia.org/r/496120 (owner: 10Dzahn)
[08:26:17] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1113:3316 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496117 (owner: 10Marostegui)
[08:26:18] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 (duration: 00m 55s)
[08:26:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:27:14] <wikibugs>	 (03CR) 10Dzahn: [V: 03+1 C: 03+1] "@Krinkle ok to go ahead here? lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[08:28:38] <wikibugs>	 (03PS12) 10Mathew.onipe: elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921)
[08:28:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[08:29:10] <wikibugs>	 (03PS4) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899
[08:30:37] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899 (owner: 10Dzahn)
[08:30:55] <wikibugs>	 10Operations, 10ExternalGuidance, 10Traffic, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Deliver mobile-based version for automatic translations - https://phabricator.wikimedia.org/T212197 (10KartikMistry) Update: We are deploying ExternalGuidance to all Wikis (but mainly visible for...
[08:38:06] <wikibugs>	 (03PS13) 10Mathew.onipe: elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921)
[08:38:27] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: refactor elastic icinga checks (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[08:38:56] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1113:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496121
[08:39:01] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[08:40:24] <wikibugs>	 (03PS14) 10Mathew.onipe: elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921)
[08:42:01] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 04-1] "CommonSettings and InitializeSettings should be in separate patches, per SWAT rules: https://wikitech.wikimedia.org/wiki/SWAT_deploys#Guid" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495957 (https://phabricator.wikimedia.org/T218137) (owner: 10Mholloway)
[08:42:21] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+1] "Good Enough (tm) for now" [puppet] - 10https://gerrit.wikimedia.org/r/495836 (owner: 10Dzahn)
[08:45:38] <wikibugs>	 (03PS18) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[08:45:44] <icinga-wm>	 RECOVERY - puppet last run on mw1330 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[08:46:30] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[08:46:45] <wikibugs>	 (03PS2) 10Dzahn: swift: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495836
[08:48:45] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] swift: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495836 (owner: 10Dzahn)
[08:48:48] <wikibugs>	 10Operations, 10Traffic, 10Patch-For-Review: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893 (10ema) 05Open→03Resolved a:03ema Closing this as cp1077 is fine and back in service. For the general Varnish s...
[08:51:14] <ema>	 !log cp3030: wipe frontend cache to get rid of large objects T216006
[08:51:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:51:17] <stashbot>	 T216006: nuke_limit often reached on esams varnish frontends - https://phabricator.wikimedia.org/T216006
[08:52:04] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC is Ok https://puppet-compiler.wmflabs.org/compiler1002/15070/" [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[08:55:05] <wikibugs>	 10ops-pmtpa: disk wipe etc for brewster - https://phabricator.wikimedia.org/T83804 (10Dzahn)
[08:57:55] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1113:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496121 (owner: 10Marostegui)
[08:58:55] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1113:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496121 (owner: 10Marostegui)
[08:58:56] <marostegui>	 !log Upgrade mysql and kernel on db2050
[08:58:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:59:40] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1113:3316" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496121 (owner: 10Marostegui)
[08:59:59] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 (duration: 00m 55s)
[09:00:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:00:00] <icinga-wm>	 PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=list https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[09:05:40] <icinga-wm>	 RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[09:06:34] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10MoritzMuehlenhoff) Production has been switched to the new component, all working fine. The new approach is also fairly straightforw...
[09:06:51] <moritzm>	 !log installing PHP 7.0 security updates
[09:06:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:07:43] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Addshore) >>  In the case that lead to this ticket, it was a remote client at Orange issuing a very high rate of these uncachea...
[09:09:01] <wikibugs>	 (03PS1) 10Elukey: profile::kafka::broker: move java.security settings to its own class [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412)
[09:11:20] <wikibugs>	 (03PS2) 10Elukey: profile::kafka::broker: move java.security settings to its own class [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412)
[09:13:33] <wikibugs>	 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, 10serviceops, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10akosiaris) >>! In T212189#5017053, @Tarrow wrote: >>>! In T212189#5011311, @akosiaris wrote: >> I have to say I am wondering a bit about th...
[09:15:22] <wikibugs>	 (03CR) 10Elukey: "pcc looks good https://puppet-compiler.wmflabs.org/compiler1002/15071/" [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[09:19:23] <wikibugs>	 (03PS1) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128
[09:20:14] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128 (owner: 10Dzahn)
[09:21:39] <icinga-wm>	 PROBLEM - Disk space on prometheus2003 is CRITICAL: DISK CRITICAL - free space: /srv/prometheus/k8s 0 MB (0% inode=99%)
[09:22:02] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10Paladox) Phabricator uses thirdparty/php72 I think.
[09:22:19] <godog>	 prometheus2003 is me btw
[09:23:41] <icinga-wm>	 RECOVERY - Disk space on prometheus2003 is OK: DISK OK
[09:27:01] <icinga-wm>	 PROBLEM - Check systemd state on ms-be1035 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[09:28:26] <wikibugs>	 (03PS2) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128
[09:29:30] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128 (owner: 10Dzahn)
[09:30:43] <wikibugs>	 (03PS19) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[09:30:58] <wikibugs>	 (03PS3) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128
[09:31:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[09:32:03] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10MoritzMuehlenhoff) Actually, I had only been looking at servers with php7.2-fpm installed, the deployment, maintenance and snapshot...
[09:32:59] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10MoritzMuehlenhoff) >>! In T216712#5020202, @Paladox wrote: > Phabricator uses thirdparty/php72 I think.  This was already converted...
[09:34:10] <wikibugs>	 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, 10serviceops, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10WMDE-leszek) > Final question and just for verification, this ain't going to be exposed directly to the internet after all, right? Rather t...
[09:34:58] <wikibugs>	 (03CR) 10Tim Eulitz: Add default user config for rollback confirmation (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[09:35:18] <wikibugs>	 (03PS3) 10Tim Eulitz: Add default user config for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436)
[09:37:34] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Addshore) The only other thing I was going to add (forgot before i hit submit on the last post)   Within the cluster varnish ca...
[09:37:46] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496129
[09:38:17] <addshore>	 elukey: looks like the memcached hits dropped off a bit :)
[09:39:09] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496129 (owner: 10Marostegui)
[09:40:05] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496129 (owner: 10Marostegui)
[09:41:15] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 00m 55s)
[09:41:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:41:26] <marostegui>	 !log Deploy schema change on db1085 with replication, there will be lag on labsdb:s6
[09:41:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:42:13] <wikibugs>	 (03PS4) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128
[09:42:33] <elukey>	 addshore: seems a bit better :)
[09:44:33] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496129 (owner: 10Marostegui)
[09:45:22] <ema>	 !log cp1071: upgrade trafficserver to 8.0.3~rc0 for testing purposes
[09:45:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:50:42] <wikibugs>	 (03PS1) 10GTirloni: labmon: Delete archived Graphite directories [puppet] - 10https://gerrit.wikimedia.org/r/496130 (https://phabricator.wikimedia.org/T218185)
[09:52:29] <icinga-wm>	 RECOVERY - Check systemd state on ms-be1035 is OK: OK - running: The system is fully operational
[09:52:31] <icinga-wm>	 PROBLEM - puppet last run on labweb1001 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown): Package[python-ldap],Package[php7.0-common],Package[php7.0-opcache],Package[php7.0-bcmath]
[09:52:37] <mutante>	 !log ms-be1035 - sudo  systemctl reset-failed
[09:52:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:52:53] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+1] "I"m happy to +2 and merge this at any time. May I?" [dumps/dcat] - 10https://gerrit.wikimedia.org/r/484011 (owner: 10Hoo man)
[09:54:16] <moritzm>	  ^ labweb is me, will recover soon
[09:54:58] <wikibugs>	 (03PS6) 10Tim Eulitz: Set up exceptions for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494270 (https://phabricator.wikimedia.org/T217436)
[09:55:23] <wikibugs>	 (03PS4) 10Tim Eulitz: Add default user config for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436)
[09:57:47] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496132
[09:58:04] <wikibugs>	 (03PS1) 10GTirloni: diamond: Do not collect disk usage for devicemapper devices [puppet] - 10https://gerrit.wikimedia.org/r/496133 (https://phabricator.wikimedia.org/T218185)
[09:58:50] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496132 (owner: 10Marostegui)
[09:59:47] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496132 (owner: 10Marostegui)
[10:00:52] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1085 (duration: 00m 56s)
[10:00:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:01:11] <wikibugs>	 (03CR) 10Jcrespo: "@Volans we would thank you for your input here, specially at the higher level for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/4" [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[10:02:19] <wikibugs>	 (03CR) 10GTirloni: "I'm proposing this from a Toolforge k8s point of view. I really have no idea what is the impact on the overall infrastructure, or if this " [puppet] - 10https://gerrit.wikimedia.org/r/496133 (https://phabricator.wikimedia.org/T218185) (owner: 10GTirloni)
[10:03:59] <icinga-wm>	 PROBLEM - DPKG on mw1232 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[10:06:43] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1085" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496132 (owner: 10Marostegui)
[10:10:07] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "At this point Diamond is essentially only used on WMCS-related roles (sans some noise with servers being setup which don't have the Hiera " [puppet] - 10https://gerrit.wikimedia.org/r/496133 (https://phabricator.wikimedia.org/T218185) (owner: 10GTirloni)
[10:20:59] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128 (owner: 10Dzahn)
[10:21:01] <wikibugs>	 (03PS5) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 6 [puppet] - 10https://gerrit.wikimedia.org/r/496128
[10:23:25] <icinga-wm>	 RECOVERY - puppet last run on labweb1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[10:23:43] <wikibugs>	 10Operations, 10Dumps-Generation: Switch dumps to component/php7.2 - https://phabricator.wikimedia.org/T218193 (10MoritzMuehlenhoff)
[10:24:25] <wikibugs>	 10Operations, 10Dumps-Generation: Switch dumps to component/php7.2 - https://phabricator.wikimedia.org/T218193 (10MoritzMuehlenhoff)
[10:24:36] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10MoritzMuehlenhoff)
[10:28:35] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "With so many +1 I would say this is certainly ready to go." [dumps/dcat] - 10https://gerrit.wikimedia.org/r/484011 (owner: 10Hoo man)
[10:31:24] <wikibugs>	 10Operations, 10Dumps-Generation: Switch dumps to component/php7.2 - https://phabricator.wikimedia.org/T218193 (10ArielGlenn) p:05Triage→03Normal
[10:33:39] <wikibugs>	 (03PS2) 10Alaa Sarhan: Enable musical notation datatype in wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493011 (https://phabricator.wikimedia.org/T216730) (owner: 10Ladsgroup)
[10:41:06] <mutante>	 win 36
[10:45:40] <wikibugs>	 (03Abandoned) 10Corinna Hillebrand: Remove disable confirmation prompt configuration. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494706 (https://phabricator.wikimedia.org/T215019) (owner: 10Corinna Hillebrand)
[10:52:13] <wikibugs>	 (03PS1) 10Muehlenhoff: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991)
[10:53:11] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[10:54:29] <wikibugs>	 (03PS2) 10Muehlenhoff: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991)
[10:54:39] <wikibugs>	 (03PS1) 10Dzahn: lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143
[10:55:20] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[10:55:28] <marostegui>	 !log Upgrade db2057
[10:55:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:55:33] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143 (owner: 10Dzahn)
[11:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for European Mid-day SWAT(Max 6 patches). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1100).
[11:00:05] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[11:00:12] <wikibugs>	 (03PS3) 10Muehlenhoff: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991)
[11:01:15] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[11:01:48] <wikibugs>	 (03PS2) 10Dzahn: lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143
[11:02:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143 (owner: 10Dzahn)
[11:02:45] <zeljkof>	 no patches, no swat :P
[11:03:10] <Jayprakash12345>	 Reddy: Are you here?
[11:03:13] <wikibugs>	 (03PS4) 10Muehlenhoff: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991)
[11:03:37] <wikibugs>	 (03CR) 10Gehel: "This sounds like a great idea! Minor comment inline." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:03:46] <Jayprakash12345>	 Reedy: ping
[11:03:53] <wikibugs>	 (03PS15) 10Gehel: elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[11:04:10] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[11:05:17] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] elasticsearch: refactor elastic icinga checks [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[11:05:27] <wikibugs>	 (03PS1) 10Dzahn: lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144
[11:05:33] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: systemd: slice for all users: drop cleanup code [puppet] - 10https://gerrit.wikimedia.org/r/496145
[11:06:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144 (owner: 10Dzahn)
[11:06:37] * onimisionipe crosses fingers..
[11:08:15] <mutante>	 waits for integration.wikimedia.org to load 
[11:08:44] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:08:46] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: systemd: slice for all users: also add pinning for libpam-systemd [puppet] - 10https://gerrit.wikimedia.org/r/496147 (https://phabricator.wikimedia.org/T215154)
[11:09:35] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: systemd: slice for all users: drop cleanup code [puppet] - 10https://gerrit.wikimedia.org/r/496145
[11:10:12] <wikibugs>	 (03CR) 10Elukey: profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:11:10] <wikibugs>	 (03PS1) 10Vgutierrez: acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295)
[11:11:20] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] systemd: slice for all users: drop cleanup code [puppet] - 10https://gerrit.wikimedia.org/r/496145 (owner: 10Arturo Borrero Gonzalez)
[11:11:54] <wikibugs>	 (03PS2) 10Dzahn: lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144
[11:11:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[11:12:52] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144 (owner: 10Dzahn)
[11:12:57] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:13:13] <wikibugs>	 (03PS2) 10Vgutierrez: acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295)
[11:14:52] <wikibugs>	 (03CR) 10Vgutierrez: "pcc shows almost a NOOP (as expected) for existing acme-chief clients: https://puppet-compiler.wmflabs.org/compiler1002/15073/" [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[11:15:18] <wikibugs>	 (03PS3) 10Elukey: profile::kafka::broker: move java.security settings to its own class [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412)
[11:16:19] <wikibugs>	 (03PS3) 10Dzahn: lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144
[11:19:40] <wikibugs>	 (03CR) 10Vgutierrez: profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:20:02] <wikibugs>	 (03PS1) 10Dzahn: doc/dumps::web::htmldumps: remove diamond collector [puppet] - 10https://gerrit.wikimedia.org/r/496149
[11:20:29] <wikibugs>	 (03CR) 10Muehlenhoff: systemd: slice for all users: fix apt pinning (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154) (owner: 10Arturo Borrero Gonzalez)
[11:22:06] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144 (owner: 10Dzahn)
[11:22:58] <wikibugs>	 (03CR) 10Elukey: profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:25:08] <wikibugs>	 (03PS3) 10Dzahn: lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143
[11:25:12] <elukey>	 vgutierrez: if you feel strongly about adding versioning in --^ I'll do it, just need to know what you have in mind :)
[11:26:00] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143 (owner: 10Dzahn)
[11:26:16] <vgutierrez>	 the plans for jre 9 migration that you mention.. is the kafka migration plan? or WMF wise?
[11:26:16] <wikibugs>	 (03PS3) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:26:27] <wikibugs>	 (03CR) 10Dzahn: systemd: slice for all users: fix apt pinning (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154) (owner: 10Arturo Borrero Gonzalez)
[11:26:55] <vgutierrez>	 mainly cause you aim to provide java::security, so you should consider our whole infra and not only kafka
[11:27:03] * vgutierrez hides
[11:27:22] <wikibugs>	 (03PS4) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:27:25] <wikibugs>	 (03CR) 10Muehlenhoff: profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:28:25] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] profile::kafka::broker: move java.security settings to its own class [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:29:27] <wikibugs>	 (03CR) 10Vgutierrez: [C: 03+1] profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:29:43] <wikibugs>	 (03PS4) 10Dzahn: lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143
[11:32:11] <elukey>	 vgutierrez: yes yes you are completely right, I just don't want to complicate things beforehand since I am not aware of any migration to Java 9+
[11:32:27] <wikibugs>	 10Operations, 10Toolforge, 10Patch-For-Review, 10cloud-services-team (Kanban): Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff
[11:32:28] <elukey>	 if that happens we'll surely modify accordingly that class
[11:32:38] <vgutierrez>	 as moritzm mentioned.. buster will bring jdk11
[11:32:56] <vgutierrez>	 but yeah.. we can tackle that in the future
[11:33:13] <elukey>	 super
[11:33:32] <elukey>	 I have to thank you since all the work done for Kafka's jvms will be reused for hadoop
[11:34:10] <marostegui>	 !log Test snapshot db1117:3325 to dbstore1001 - T210292
[11:34:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:34:13] <stashbot>	 T210292: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292
[11:34:25] <vgutierrez>	 elukey: :D that's nice
[11:36:40] <moritzm>	 vgutierrez, elukey: Java 11 will have different default crypto settings/cipher support anyway, so it needs to be revisited anyway
[11:36:50] <vgutierrez>	 ack
[11:37:48] <elukey>	 moritzm: lovely :D
[11:38:11] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] lvs/varnish: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/496143 (owner: 10Dzahn)
[11:39:01] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144 (owner: 10Dzahn)
[11:39:10] <wikibugs>	 (03PS4) 10Dzahn: lvs/icinga: add eqiad monitoring for Kartotherian [puppet] - 10https://gerrit.wikimedia.org/r/496144
[11:41:51] <elukey>	 gehel: o/
[11:41:59] <elukey>	 ok for me to merge the java security change?
[11:43:17] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154) (owner: 10Arturo Borrero Gonzalez)
[11:43:27] <wikibugs>	 (03PS5) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:45:04] <gehel>	 elukey: checking
[11:45:28] <wikibugs>	 (03PS2) 10Dzahn: mediawiki: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495834
[11:46:34] <wikibugs>	 (03CR) 10Gehel: [C: 03+1] profile::kafka::broker: move java.security settings to its own class (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:46:40] <gehel>	 elukey: all good to me!
[11:47:04] * gehel is wondering if he is brave enough to try applying those settings to elasticsearch
[11:47:32] * gehel goes back to post lunch coffee
[11:47:41] <elukey>	 thanks!
[11:48:12] <elukey>	 TIL YAGNI
[11:48:13] <elukey>	 :D
[11:48:18] <elukey>	 (from the code review)
[11:50:44] <wikibugs>	 10Operations, 10hardware-requests, 10Patch-For-Review: request to assign wmf6937 (mw1298, former imagescaler) (now: wmf4727)  as phab1002 - https://phabricator.wikimedia.org/T195623 (10Dzahn)
[11:50:50] <wikibugs>	 10Operations, 10hardware-requests, 10Patch-For-Review: requesting WMF7426 as phabricator system in eqiad - https://phabricator.wikimedia.org/T215335 (10Dzahn) 05Open→03Stalled Thanks!  Setting to stalled to reflect we are waiting another week or so to decide if T215837 is still requested (per IRC with vo...
[11:53:13] <wikibugs>	 (03PS6) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[11:53:15] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] mediawiki: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495834 (owner: 10Dzahn)
[11:53:39] <mutante>	 arturo: go ahead and merge first :)
[11:54:04] <wikibugs>	 (03PS4) 10Elukey: profile::kafka::broker: move java.security settings to its own class [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412)
[11:54:09] <mutante>	 i feel like i keep sniping you, hehe
[11:54:13] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/15075/" [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[11:54:16] <arturo>	 :-) it's OK
[11:54:19] <icinga-wm>	 PROBLEM - Getent speed check on labstore1005 is CRITICAL: CRITICAL: getent group tools.admin timed out (1s)
[11:54:22] <arturo>	 thanks, in fact, for taking a look
[11:54:34] <arturo>	 ops, that paged
[11:54:38] <volans>	 yep
[11:54:46] <mutante>	 ack, it did
[11:55:17] <mutante>	 no notes URL yet.. i thought i already got all the lab* stuff, looks like not
[11:55:28] <icinga-wm>	 RECOVERY - Getent speed check on labstore1005 is OK: OK: getent group returns within a second
[11:57:38] <arturo>	 not sure what this check is doing
[11:58:27] <mutante>	 tests if LDAP server has issues by running getent?
[11:58:59] <arturo>	 I can't run that command locally in the server, it does return nothing
[11:59:25] <arturo>	 also, I don't think this server fetches users/groups from LDAP?
[11:59:31] <arturo>	 weird
[11:59:52] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154) (owner: 10Arturo Borrero Gonzalez)
[12:00:01] <wikibugs>	 (03PS7) 10Arturo Borrero Gonzalez: systemd: slice for all users: fix apt pinning [puppet] - 10https://gerrit.wikimedia.org/r/496146 (https://phabricator.wikimedia.org/T215154)
[12:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1200)
[12:00:34] <wikibugs>	 (03PS1) 10Jbond: Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151
[12:01:08] * apergos peers in
[12:01:29] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151 (owner: 10Jbond)
[12:01:39] <mutante>	 arturo: it runs /usr/local/bin/getent_check
[12:02:20] <mutante>	 modules/labstore/templates/getent_check.erb
[12:02:38] <mutante>	 # Try to fetch a known group via LDAP hee
[12:02:47] <arturo>	 I see
[12:02:51] <mutante>	 and then /usr/bin/useldap /usr/bin/getent group tools.admin
[12:03:37] <mutante>	 there is some trap/timeout code in there.. probably related 
[12:03:37] <wikibugs>	 (03PS2) 10Jbond: Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151
[12:06:08] <wikibugs>	 (03CR) 10Muehlenhoff: Add openjpeg2 libraries to config file (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496151 (owner: 10Jbond)
[12:09:01] <wikibugs>	 (03PS3) 10Jbond: Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151
[12:10:50] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: systemd: slice for all users: also add pinning for libpam-systemd [puppet] - 10https://gerrit.wikimedia.org/r/496147 (https://phabricator.wikimedia.org/T215154)
[12:12:22] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] systemd: slice for all users: also add pinning for libpam-systemd [puppet] - 10https://gerrit.wikimedia.org/r/496147 (https://phabricator.wikimedia.org/T215154) (owner: 10Arturo Borrero Gonzalez)
[12:13:48] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151 (owner: 10Jbond)
[12:14:28] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151 (owner: 10Jbond)
[12:14:38] <wikibugs>	 (03PS4) 10Jbond: Add openjpeg2 libraries to config file [puppet] - 10https://gerrit.wikimedia.org/r/496151
[12:28:33] <icinga-wm>	 RECOVERY - DPKG on mw1232 is OK: All packages OK
[12:34:02] <wikibugs>	 (03PS12) 10Elukey: hadoop: allow the configuration of ssl-(server|client).xml configs [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412)
[12:38:43] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] "user/group was removed in Ic6bda187ada3412264636a71af02602382e58047 as a consequence of moving to systemd. The starting commit though desc" [puppet] - 10https://gerrit.wikimedia.org/r/464601 (owner: 10Jcrespo)
[12:38:57] <wikibugs>	 (03PS13) 10Elukey: hadoop: allow the configuration of ssl-(server|client).xml configs [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412)
[12:42:08] <wikibugs>	 (03CR) 10WMDE-Fisch: [C: 03+1] Add default user config for rollback confirmation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[12:42:42] <wikibugs>	 (03PS14) 10Elukey: hadoop: allow the configuration of ssl-(server|client).xml configs [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412)
[12:47:07] <wikibugs>	 (03CR) 10Elukey: "PCC: https://puppet-compiler.wmflabs.org/compiler1002/15081/" [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[12:52:36] <addshore>	 o/ If I am doing some debugging on mwdebug1001, and touch a file and want php to actually look at the touched file what should I be doing?
[12:53:00] <wikibugs>	 (03CR) 10Jbond: Enable base::service_auto_restart for rsync on doc1001 (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[12:53:08] <wikibugs>	 (03PS5) 10Jbond: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[13:00:04] <jouncebot>	 zeljkof: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for MediaWiki train - European version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1300).
[13:02:41] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: openstack: virt: install libguestfs-tools package in component class [puppet] - 10https://gerrit.wikimedia.org/r/496156 (https://phabricator.wikimedia.org/T216497)
[13:03:42] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: openstack: virt: install libguestfs-tools package in component class [puppet] - 10https://gerrit.wikimedia.org/r/496156 (https://phabricator.wikimedia.org/T216497)
[13:03:58] <zeljkof>	 pushing wmf.21 to group 1
[13:05:50] <addshore>	 woo
[13:08:47] <wikibugs>	 (03PS1) 10Zfilipin: group1 wikis to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496159
[13:08:49] <wikibugs>	 (03CR) 10Zfilipin: [C: 03+2] group1 wikis to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496159 (owner: 10Zfilipin)
[13:09:59] <wikibugs>	 (03Merged) 10jenkins-bot: group1 wikis to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496159 (owner: 10Zfilipin)
[13:12:05] <logmsgbot>	 !log zfilipin@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.21
[13:12:34] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] "Waiting for you to lead deployment. We know our MySQL hosts are not affected by this, but we don't know if other services are, so blocking" [puppet] - 10https://gerrit.wikimedia.org/r/464601 (owner: 10Jcrespo)
[13:13:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:13:49] <logmsgbot>	 !log zfilipin@deploy1001 Synchronized php: group1 wikis to 1.33.0-wmf.21 (duration: 01m 43s)
[13:14:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:15:51] <wikibugs>	 (03CR) 10jenkins-bot: group1 wikis to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496159 (owner: 10Zfilipin)
[13:17:44] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Package cxserver chart 0.0.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/496160 (https://phabricator.wikimedia.org/T213195)
[13:17:53] <wikibugs>	 (03CR) 10Dzahn: [C: 03+1] Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[13:18:03] <icinga-wm>	 PROBLEM - Apache HTTP on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:18:03] <icinga-wm>	 PROBLEM - Nginx local proxy to apache on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:18:03] <icinga-wm>	 PROBLEM - HHVM rendering on mw1249 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:18:17] <wikibugs>	 (03PS3) 10Dzahn: mediawiki: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495834
[13:19:36] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Package cxserver chart 0.0.2 [deployment-charts] - 10https://gerrit.wikimedia.org/r/496160 (https://phabricator.wikimedia.org/T213195) (owner: 10Alexandros Kosiaris)
[13:20:15] <wikibugs>	 (03PS1) 10Mahveotm: Increased maxSerializedEntitySize from 2500 to 3000 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496161 (https://phabricator.wikimedia.org/T217739)
[13:23:25] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] ":)" [puppet] - 10https://gerrit.wikimedia.org/r/496127 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[13:24:01] <godog>	 !log take a snapshot of prometheus@k8s data on prometheus2004
[13:24:03] <icinga-wm>	 RECOVERY - Nginx local proxy to apache on mw1249 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 1.020 second response time
[13:24:03] <icinga-wm>	 RECOVERY - Apache HTTP on mw1249 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 1.027 second response time
[13:24:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:24:05] <icinga-wm>	 RECOVERY - HHVM rendering on mw1249 is OK: HTTP OK: HTTP/1.1 200 OK - 75185 bytes in 3.631 second response time
[13:27:48] <wikibugs>	 (03PS6) 10Muehlenhoff: Enable base::service_auto_restart for rsync on doc1001 [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991)
[13:27:50] <wikibugs>	 (03Abandoned) 10Arturo Borrero Gonzalez: openstack: virt: install libguestfs-tools package in component class [puppet] - 10https://gerrit.wikimedia.org/r/496156 (https://phabricator.wikimedia.org/T216497) (owner: 10Arturo Borrero Gonzalez)
[13:28:40] <akosiaris>	 !log upgrade kubestage1002 to kubernetes 1.11.8
[13:28:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:30:38] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] "Couple of nits, but +1 merge at will!" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[13:30:43] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+1] "Agreed with Moritz" [puppet] - 10https://gerrit.wikimedia.org/r/496133 (https://phabricator.wikimedia.org/T218185) (owner: 10GTirloni)
[13:30:45] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Enable base::service_auto_restart for rsync on doc1001 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/496142 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[13:34:58] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] "Yep, as long as webperf1001 and tungsten are not changed, anything on webperf1002 is fine as it won't be exposed yet." [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[13:35:41] <wikibugs>	 (03PS1) 10Mathew.onipe: use hostname instead of fqdn [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921)
[13:41:54] <wikibugs>	 (03CR) 10Elukey: "Thanks a lot!" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412) (owner: 10Elukey)
[13:42:23] <akosiaris>	 !log upgrade neon to kubernetes 1.11.8
[13:42:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:42:28] <akosiaris>	 !log upgrade kubestage to kubernetes 1.11.8
[13:42:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:42:33] <wikibugs>	 (03PS15) 10Elukey: hadoop: allow the configuration of ssl-(server|client).xml configs [puppet] - 10https://gerrit.wikimedia.org/r/493693 (https://phabricator.wikimedia.org/T217412)
[13:44:09] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC output is ok: https://puppet-compiler.wmflabs.org/compiler1002/15084/" [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:48:28] <wikibugs>	 (03PS3) 10Vgutierrez: acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295)
[13:49:31] <wikibugs>	 (03PS4) 10Vgutierrez: acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295)
[13:49:52] <wikibugs>	 (03PS3) 10Arturo Borrero Gonzalez: openstack: drop -t jessie-backports for openstack component installs in stretch [puppet] - 10https://gerrit.wikimedia.org/r/495893 (https://phabricator.wikimedia.org/T216497)
[13:50:48] <wikibugs>	 (03PS5) 10Vgutierrez: acme_chief: Update acme_chief::cert resource to fetch several cert versions [puppet] - 10https://gerrit.wikimedia.org/r/496148 (https://phabricator.wikimedia.org/T207295)
[13:53:14] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), 10User-Elukey: Consider raising Memcached MWObject cache memory size limit - https://phabricator.wikimedia.org/T217731 (10elukey) I was wrong, evictions started happening, even if on a lower pace. The extra 10G of space allo...
[13:53:57] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] openstack: drop -t jessie-backports for openstack component installs in stretch [puppet] - 10https://gerrit.wikimedia.org/r/495893 (https://phabricator.wikimedia.org/T216497) (owner: 10Arturo Borrero Gonzalez)
[14:00:16] <wikibugs>	 (03PS2) 10Mathew.onipe: use hostname instead of fqdn [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921)
[14:00:38] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] use hostname instead of fqdn [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[14:02:25] <wikibugs>	 (03PS3) 10Mathew.onipe: use hostname instead of fqdn [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921)
[14:03:14] <wikibugs>	 10Operations, 10netops: eqiad 10G ports needs - https://phabricator.wikimedia.org/T190364 (10jcrespo) I have better information of Databases backups/provisioning service:  in the end, only 2 hosts per datacenter T216137 T216138 over the next 3-5 years  A summary of the logical architechture can be seen at: htt...
[14:07:52] <wikibugs>	 10Operations, 10ops-eqiad, 10DC-Ops, 10cloud-services-team (Kanban): cloudvirt1015: apparent hardware errors in CPU/Memory - https://phabricator.wikimedia.org/T215012 (10aborrero) While we wait for the new CPU, this server is not serving any purpose, so I will be using it to test some puppet changes (inclu...
[14:11:33] <wikibugs>	 (03PS17) 10Mobrovac: Initial configuration for session storage service [puppet] - 10https://gerrit.wikimedia.org/r/487885 (https://phabricator.wikimedia.org/T215883) (owner: 10Eevans)
[14:12:12] <icinga-wm>	 PROBLEM - kubelet operational latencies on kubestage1002 is CRITICAL: instance=kubestage1002.eqiad.wmnet https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1
[14:12:24] <icinga-wm>	 PROBLEM - kubelet operational latencies on kubestage1001 is CRITICAL: instance=kubestage1001.eqiad.wmnet https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1
[14:13:34] <icinga-wm>	 RECOVERY - kubelet operational latencies on kubestage1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1
[14:13:36] <marostegui>	 !log Upgrade db2074 (sanitarium master)
[14:13:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:14:38] <icinga-wm>	 RECOVERY - kubelet operational latencies on kubestage1002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-kubelets?orgId=1
[14:16:57] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC output: https://puppet-compiler.wmflabs.org/compiler1002/15086/" [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[14:27:05] <ema>	 !log cp2002: depool varnish-fe in preparation of pointing it to ATS T213263
[14:27:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:27:08] <stashbot>	 T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263
[14:29:43] <wikibugs>	 10Operations, 10CX-cxserver, 10Citoid, 10Graphoid, and 10 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10Pchelolo)
[14:31:51] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] use hostname instead of fqdn [puppet] - 10https://gerrit.wikimedia.org/r/496164 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[14:33:08] <wikibugs>	 (03PS2) 10Ema: cp2002: use ATS backends instead of Varnish [puppet] - 10https://gerrit.wikimedia.org/r/494761 (https://phabricator.wikimedia.org/T213263)
[14:33:58] <ema>	 pointing cp2002 varnish-fe to ATS, no user traffic affected yet as the frontend is depooled 
[14:34:09] <wikibugs>	 10Operations, 10RESTBase, 10RESTBase-API, 10serviceops, and 2 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10Pchelolo)
[14:34:11] <wikibugs>	 (03CR) 10Ema: [C: 03+2] cp2002: use ATS backends instead of Varnish [puppet] - 10https://gerrit.wikimedia.org/r/494761 (https://phabricator.wikimedia.org/T213263) (owner: 10Ema)
[14:35:37] <wikibugs>	 (03PS3) 10Gehel: aptrepo: create elastic65 component [puppet] - 10https://gerrit.wikimedia.org/r/495914 (https://phabricator.wikimedia.org/T218113)
[14:35:53] <wikibugs>	 (03PS1) 10Eevans: [WIP] role::sessionstore [puppet] - 10https://gerrit.wikimedia.org/r/496173
[14:36:08] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Pchelolo) 05Open→03Resolved a:03Pchelolo I think we have a consensus to go spec-compliant here. I've...
[14:36:29] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] role::sessionstore [puppet] - 10https://gerrit.wikimedia.org/r/496173 (owner: 10Eevans)
[14:36:56] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] aptrepo: create elastic65 component [puppet] - 10https://gerrit.wikimedia.org/r/495914 (https://phabricator.wikimedia.org/T218113) (owner: 10Gehel)
[14:38:36] <wikibugs>	 (03PS2) 10Eevans: [WIP] role::sessionstore [puppet] - 10https://gerrit.wikimedia.org/r/496173
[14:39:13] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] role::sessionstore [puppet] - 10https://gerrit.wikimedia.org/r/496173 (owner: 10Eevans)
[14:41:27] <wikibugs>	 10Operations, 10Patch-For-Review: Logrotate fails for: "$FILE No such file or directory" - https://phabricator.wikimedia.org/T153940 (10jbond) I had a varient of this during clini  ` /etc/cron.daily/logrotate: error: error creating output file /var/log/squid3/access.log.1.gz: File exists run-parts: /etc/cron.d...
[14:44:57] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] "Doubtful that I have the cycles for this right now however. Maybe in a couple of weeks?" [puppet] - 10https://gerrit.wikimedia.org/r/464601 (owner: 10Jcrespo)
[14:50:07] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Patch-For-Review, 10Performance-Team (Radar), 10User-Elukey: Consider raising Memcached MWObject cache memory size limit - https://phabricator.wikimedia.org/T217731 (10aaron) Can the space be dedicated more so towards the larger slabs, were we have more problems AFAIK?
[14:57:12] <wikibugs>	 10Operations, 10RESTBase, 10RESTBase-API, 10serviceops, and 2 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10Pchelolo)
[14:57:47] <wikibugs>	 10Operations, 10Patch-For-Review: wmf-auto-restart fails on certain legacy services - https://phabricator.wikimedia.org/T212219 (10jbond) Was this resolved with moritz's patch.  if not i came across a similar issue and created the following patch  https://gerrit.wikimedia.org/r/c/operations/puppet/+/494210
[14:58:15] <wikibugs>	 (03PS3) 10Gehel: Allow apifeatureusage to work in beta cluster [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156) (owner: 10EBernhardson)
[14:58:22] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Pchelolo)
[14:58:42] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Pchelolo)
[14:59:13] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] Allow apifeatureusage to work in beta cluster [puppet] - 10https://gerrit.wikimedia.org/r/496093 (https://phabricator.wikimedia.org/T183156) (owner: 10EBernhardson)
[14:59:37] <gehel>	 ebernhardson: ^ merged, I'll let you test
[14:59:45] <wikibugs>	 10Operations, 10CX-cxserver, 10Citoid, 10Graphoid, and 10 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10Pchelolo)
[15:00:25] <wikibugs>	 10Operations, 10RESTBase, 10RESTBase-API, 10serviceops, and 2 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10Pchelolo)
[15:02:02] <wikibugs>	 10Operations, 10Patch-For-Review: wmf-auto-restart fails on certain legacy services - https://phabricator.wikimedia.org/T212219 (10MoritzMuehlenhoff) Ah, I completely forgot to merge it, it's https://gerrit.wikimedia.org/r/480520, will do that later on  (This task is about a wrong service name passed to wmf-au...
[15:02:54] <wikibugs>	 10Operations, 10Patch-For-Review: wmf-auto-restart fails on certain legacy services - https://phabricator.wikimedia.org/T212219 (10jbond) didn't notice it wasn't merged :)
[15:04:40] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "minor style issues" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[15:08:40] <addshore>	 elukey: FYI https://wikitech.wikimedia.org/wiki/WMDE/Wikidata/Caching#Memcached
[15:10:33] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] "LGTM, checksums checked." [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/491297 (https://phabricator.wikimedia.org/T199791) (owner: 10DCausse)
[15:11:53] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+1] diamond: Do not collect disk usage for devicemapper devices [puppet] - 10https://gerrit.wikimedia.org/r/496133 (https://phabricator.wikimedia.org/T218185) (owner: 10GTirloni)
[15:13:13] <wikibugs>	 (03PS8) 10Dzahn: xhgui: setup git cloning and apache site [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761)
[15:16:16] <wikibugs>	 (03PS1) 10Muehlenhoff: Add missing pull definition for elastic65 [puppet] - 10https://gerrit.wikimedia.org/r/496180 (https://phabricator.wikimedia.org/T218113)
[15:16:26] <moritzm>	 ^gehel: found it
[15:16:53] <wikibugs>	 (03CR) 10Gehel: [C: 03+1] "Thanks! I knew I missed something!" [puppet] - 10https://gerrit.wikimedia.org/r/496180 (https://phabricator.wikimedia.org/T218113) (owner: 10Muehlenhoff)
[15:17:18] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] Add missing pull definition for elastic65 [puppet] - 10https://gerrit.wikimedia.org/r/496180 (https://phabricator.wikimedia.org/T218113) (owner: 10Muehlenhoff)
[15:19:45] <mutante>	 onimisionipe: you know about the new elastic alerts triggering?
[15:25:35] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "yep, thanks. adding compiler output to show that is the case:  https://puppet-compiler.wmflabs.org/compiler1002/15088/" [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[15:25:54] <wikibugs>	 (03PS9) 10Dzahn: xhgui: setup git cloning and apache site [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761)
[15:26:07] <elukey>	 addshore: s/3M/1M :)
[15:26:25] <gehel>	 mutante: yep, new alert, needs to be tuned! At least we now it works!
[15:26:43] <mutante>	 gehel: ok, ack!
[15:26:52] <addshore>	 elukey: ack!
[15:26:53] <gehel>	 I'm acking them now
[15:26:54] <mutante>	 they will start popping up soon
[15:27:00] <mutante>	 cool, thanks
[15:27:59] <gehel>	 onimisionipe is in transit atm, I'll ping him when he is back
[15:29:46] <mutante>	 :)
[15:31:05] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Remove eventgate-analytics access to gerrit [puppet] - 10https://gerrit.wikimedia.org/r/496182
[15:31:07] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Allow cxserver in kubernetes to access apertium [puppet] - 10https://gerrit.wikimedia.org/r/496183 (https://phabricator.wikimedia.org/T213195)
[15:32:32] <wikibugs>	 (03CR) 10Dzahn: "follow-up error:  AH00526: Syntax error on line 21 of /etc/apache2/sites-enabled/50-xhgui-apache-site.conf:" [puppet] - 10https://gerrit.wikimedia.org/r/494425 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[15:36:21] <kostajh>	 hi, I've had two builds in a row fail due to  "Cannot start ChromeHeadless", any ideas? e.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/39395/console
[15:37:43] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Remove eventgate-analytics access to gerrit [puppet] - 10https://gerrit.wikimedia.org/r/496182 (owner: 10Alexandros Kosiaris)
[15:37:48] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Allow cxserver in kubernetes to access apertium [puppet] - 10https://gerrit.wikimedia.org/r/496183 (https://phabricator.wikimedia.org/T213195) (owner: 10Alexandros Kosiaris)
[15:39:18] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver install -n production -f cxserver-codfw-values.yaml stable/cxserver [namespace: cxserver, clusters: codfw]
[15:39:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:20] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster codfw completed
[15:39:20] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[15:39:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:30] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver install -n production -f cxserver-eqiad-values.yaml stable/cxserver [namespace: cxserver, clusters: eqiad]
[15:39:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:32] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster eqiad completed
[15:39:32] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[15:39:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:40:33] <akosiaris>	 !log do the first deploy of cxserver in eqiad/codfw T213195
[15:40:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:40:37] <stashbot>	 T213195: Migrate cxserver to kubernetes - https://phabricator.wikimedia.org/T213195
[15:41:04] <wikibugs>	 (03PS1) 10Gehel: logstash: tune shard size check to reflect the current known good sizes [puppet] - 10https://gerrit.wikimedia.org/r/496184
[15:41:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] logstash: tune shard size check to reflect the current known good sizes [puppet] - 10https://gerrit.wikimedia.org/r/496184 (owner: 10Gehel)
[15:42:56] <wikibugs>	 (03PS2) 10Gehel: logstash: tune shard size check to reflect the current known good sizes [puppet] - 10https://gerrit.wikimedia.org/r/496184
[15:43:51] <wikibugs>	 (03PS1) 10Dzahn: webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761)
[15:44:40] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[15:46:59] <wikibugs>	 10Operations, 10Reading-Infrastructure-Team-Backlog, 10Maps (Kartotherian), 10Wikimedia-Incident: Create test in spec.yaml for the kartotherian / geoshape service - https://phabricator.wikimedia.org/T217910 (10Jhernandez) p:05Triage→03Normal
[15:50:31] <wikibugs>	 10Operations, 10DNS, 10Traffic, 10Patch-For-Review: Internal DNS resolver responds with NXDOMAIN for localhost AAAA - https://phabricator.wikimedia.org/T125170 (10jbond) FYI this got fixed in pdns version 4.1.0 (its actully in 4.1.0-alpha1) via https://github.com/PowerDNS/pdns/pull/5223
[15:52:08] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw]
[15:52:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:10] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster codfw completed
[15:52:10] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[15:52:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:27] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml eqiad stable/cxserver [namespace: cxserver, clusters: eqiad]
[15:52:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:35] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad]
[15:52:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:36] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster eqiad completed
[15:52:36] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[15:52:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:51] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging]
[15:52:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:52] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[15:52:52] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[15:52:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:52:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:00:05] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Dear deployers, time to do the Morning SWAT (Max 6 patches) deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1600).
[16:00:05] <jouncebot>	 ottomata: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[16:00:18] <ottomata>	 here
[16:01:53] <wikibugs>	 (03PS2) 10Dzahn: webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761)
[16:02:37] <wikibugs>	 (03CR) 10D3r1ck01: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885) (owner: 10Ammarpad)
[16:02:45] <wikibugs>	 (03CR) 10D3r1ck01: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496161 (https://phabricator.wikimedia.org/T217739) (owner: 10Mahveotm)
[16:02:47] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[16:04:27] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging]
[16:04:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:28] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[16:04:28] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:04:29] <wikibugs>	 (03Abandoned) 10Eevans: [WIP] role::sessionstore [puppet] - 10https://gerrit.wikimedia.org/r/496173 (owner: 10Eevans)
[16:04:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:31] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw]
[16:04:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:32] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster codfw completed
[16:04:32] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:04:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:43] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad]
[16:04:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:44] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster eqiad completed
[16:04:44] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:04:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:04:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:06:00] <ottomata>	 who's swatting?
[16:07:03] <ottomata>	 MaxSem: ? or who should I ping? :)
[16:08:14] <wikibugs>	 (03PS3) 10Dzahn: webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761)
[16:09:07] <wikibugs>	 10Operations, 10RESTBase, 10RESTBase-API, 10serviceops, and 3 others: Make RESTBase spec standard compliant and switch to OpenAPI 3.0 - https://phabricator.wikimedia.org/T218218 (10mobrovac)
[16:09:30] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761) (owner: 10Dzahn)
[16:09:40] <wikibugs>	 (03PS4) 10Dzahn: webperf::profiling_tools: include ::passwords::ldap::production [puppet] - 10https://gerrit.wikimedia.org/r/496185 (https://phabricator.wikimedia.org/T180761)
[16:10:33] <ottomata>	 thcipriani: who should swat?
[16:11:22] <wikibugs>	 (03PS1) 10Eevans: [WIP] Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192
[16:12:16] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192 (owner: 10Eevans)
[16:13:00] <thcipriani>	 ottomata: hrm, I'm in a meeting with a bunch of other releng folks. I'm not sure who is available :(
[16:13:19] <ottomata>	 but the swat calender is sacred! :o :p
[16:13:29] <ottomata>	 addshore: ? :)
[16:13:40] <addshore>	 hmmmm?
[16:13:50] <addshore>	 oh, hi :P
[16:14:04] <addshore>	 yes, i can do your thing
[16:14:06] <addshore>	 jouncebot: now
[16:14:06] <jouncebot>	 For the next 0 hour(s) and 45 minute(s): Morning SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1600)
[16:14:14] <ottomata>	 hooray danke
[16:14:25] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Fix typo, bump cxserver chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/496195
[16:14:39] <addshore>	 ottomata: looks like you merged it already? :)
[16:14:44] <ottomata>	 addshore:  best place for me to smoke test would be test wiki, can test wiki be accessed directly via X-Wikimedia-Debug ?
[16:14:50] <ottomata>	 addshore:  yes merged so we can test in beta
[16:14:51] <addshore>	 yup
[16:14:54] <ottomata>	 k
[16:15:00] <addshore>	 let me log into the places
[16:15:16] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10jijiki)
[16:15:24] <jijiki>	 !log Depool thumbor1004 to investigate memory issues - T215411
[16:15:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:15:27] <stashbot>	 T215411: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411
[16:15:57] <addshore>	 oh wait, it is not cherry picked yet ottomata ?
[16:16:05] <addshore>	 ottomata: which branches do you want to target?
[16:16:15] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Set up exceptions for rollback confirmation (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/494270 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[16:16:15] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Add default user config for rollback confirmation (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495667 (https://phabricator.wikimedia.org/T217436) (owner: 10Tim Eulitz)
[16:16:19] <addshore>	 https://tools.wmflabs.org/versions/
[16:16:20] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging /home/akosiaris/deployment-charts/charts/cxserver/ [namespace: cxserver, clusters: staging]
[16:16:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:21] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[16:16:21] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:16:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:42] <ottomata>	 addshore:  .21 will be fine, .21 will go to group2 tomorrow, ya?
[16:16:48] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging /home/akosiaris/deployment-charts/charts/cxserver/ [namespace: cxserver, clusters: staging]
[16:16:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:49] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[16:16:49] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:16:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:16:53] <addshore>	 ottomata: yup
[16:16:57] <ottomata>	 ya that's good then
[16:17:04] <ottomata>	 we won't enable this on group2 wikis today
[16:17:10] <ottomata>	 so we can wait for normal train on that
[16:17:21] <addshore>	 ottomata: cherry pick is https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/496197/ 
[16:17:24] <addshore>	 just gotta wait for CI now
[16:18:20] <wikibugs>	 (03PS2) 10Gehel: elasticsearch: expose elastic65 APT component to hiera [puppet] - 10https://gerrit.wikimedia.org/r/495915 (https://phabricator.wikimedia.org/T194199)
[16:19:25] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging /home/akosiaris/deployment-charts/charts/cxserver/ [namespace: cxserver, clusters: staging]
[16:19:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:19:26] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[16:19:27] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:19:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:19:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:19:43] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10jijiki) Server has been depooled and downtimed on icinga for 48 hours, @Cmjohnson you can power it down any time, tx :)
[16:20:10] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: Fix typo, bump cxserver chart version [deployment-charts] - 10https://gerrit.wikimedia.org/r/496195
[16:20:54] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10jijiki) a:03RobH
[16:23:09] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+1] elasticsearch: deploy elasticsearch config for ES6 [puppet] - 10https://gerrit.wikimedia.org/r/495921 (https://phabricator.wikimedia.org/T218116) (owner: 10Gehel)
[16:23:20] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+1] elasticsearch: upgrade to elastic 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/495920 (https://phabricator.wikimedia.org/T218116) (owner: 10Gehel)
[16:23:40] <icinga-wm>	 PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[16:23:43] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+1] elasticsearch: expose elastic65 APT component to hiera [puppet] - 10https://gerrit.wikimedia.org/r/495915 (https://phabricator.wikimedia.org/T194199) (owner: 10Gehel)
[16:24:13] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199) (owner: 10Gehel)
[16:25:08] <robh>	 Who is running swat and please let me know when done?
[16:25:16] <robh>	 I have a mw system to memtest post swat.
[16:25:18] <robh>	 =]
[16:25:38] <James_F>	 robh: memtest whilst in the pool?
[16:25:45] <robh>	 no, it comes offline
[16:25:57] <robh>	 its why im waiting for swat to be done, should be offline for like 30m
[16:26:06] <icinga-wm>	 RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[16:26:16] <robh>	 firmware update, then memtest run, then its either known as failed or we return to service.
[16:26:18] <ottomata>	 robh:  addshore  is swatting one thing for me
[16:26:33] <James_F>	 Oh, right. addshore is doing the SWAT but no updates in 10 mins. ;-)
[16:26:38] <addshore>	 :)
[16:26:41] <addshore>	 waiting for CI
[16:26:59] <robh>	 cool, just ping me when all done and i can do my hardware testing without causing any issues =]
[16:27:03] <addshore>	 looking at CI ottomata it is red :/
[16:27:05] <James_F>	 Oh, right,  you're deploying https://gerrit.wikimedia.org/r/c/mediawiki/core/+/496197 not https://gerrit.wikimedia.org/r/c/mediawiki/core/+/495399
[16:27:09] <ottomata>	 !
[16:27:11] <robh>	 (this is NOT urgent)
[16:27:15] <robh>	 to be clear.
[16:27:16] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/496197/
[16:27:23] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+1] elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199) (owner: 10Gehel)
[16:27:27] <James_F>	 addshore: Yeah, CI is totally broken in some ways right now. :-(
[16:27:29] <robh>	 this is regular hardware failure events of a non critical nature.
[16:27:38] <ottomata>	 addshore:  where red?
[16:27:43] <addshore>	 ack, James_F I may just ignore the hhvm bit then, as it passes all the rest
[16:28:07] <James_F>	 ottomata:  https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-hhvm-docker/10418/consolehttps://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-hhvm-docker/14639/console
[16:28:14] <addshore>	 ottomata: head to https://integration.wikimedia.org/zuul/ and search for 496197 under the heading gate-and-submit-swat
[16:28:21] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: cloudvirt1015: reimage as Debian Stretch [puppet] - 10https://gerrit.wikimedia.org/r/496201 (https://phabricator.wikimedia.org/T215012)
[16:28:48] <ottomata>	 ChromeHeadlesss?
[16:29:19] <Pchelolo>	 seen that error in another changeset
[16:29:32] <addshore>	 I'm happy to ignore that one
[16:29:36] <Lucas_WMDE>	 see https://phabricator.wikimedia.org/T218209 for the ChromeHeadless thingies
[16:29:40] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] cloudvirt1015: reimage as Debian Stretch [puppet] - 10https://gerrit.wikimedia.org/r/496201 (https://phabricator.wikimedia.org/T215012) (owner: 10Arturo Borrero Gonzalez)
[16:29:41] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10RobH) next steps:  * verify there is no ongoing SWAT or config change windows being pushed to the mw cluster * update the firmware of the bios * clear all memory errors fr...
[16:29:56] <ottomata>	 ok let's ignore then
[16:30:12] <addshore>	 ottomata: gonna pull it onto the server now
[16:30:20] <wikibugs>	 (03PS3) 10Gehel: elasticsearch: expose elastic65 APT component to hiera [puppet] - 10https://gerrit.wikimedia.org/r/495915 (https://phabricator.wikimedia.org/T194199)
[16:30:23] <ottomata>	 k
[16:31:27] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] elasticsearch: expose elastic65 APT component to hiera [puppet] - 10https://gerrit.wikimedia.org/r/495915 (https://phabricator.wikimedia.org/T194199) (owner: 10Gehel)
[16:31:35] <addshore>	 ottomata: it is on mwdebug1002
[16:31:44] <addshore>	 ottomata: ping me when your down checking :)
[16:31:59] <ottomata>	 great looks good
[16:32:13] <ottomata>	 issued api request to that server, verified thta logs came through as expected
[16:32:20] <Pchelolo>	 ottomata: friendly reminder about the meeting after you done
[16:32:23] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging /home/akosiaris/deployment-charts/charts/cxserver/ [namespace: cxserver, clusters: staging]
[16:32:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:24] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver cluster staging completed
[16:32:24] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm cxserver finished
[16:32:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:33:02] <ottomata>	 OH Pchelolo  coming now
[16:33:04] <ottomata>	 thank you
[16:34:37] <addshore>	 ottomata: ack, i'll sync it then? :)
[16:35:01] <ottomata>	 yes please!
[16:35:37] <addshore>	 doing
[16:36:32] <logmsgbot>	 !log addshore@deploy1001 Synchronized php-1.33.0-wmf.21/includes/api/ApiMain.php: SWAT: T214080 T212529 ApiMain.php api/request logging event changes [[gerrit:496197]] (duration: 00m 57s)
[16:36:37] <addshore>	 ottomata: all done
[16:36:39] <addshore>	 !log SWAT done
[16:36:45] <ottomata>	 thank you so much!
[16:36:48] <addshore>	 robh: ^^
[16:36:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:36:51] <stashbot>	 T214080: Rewrite Avro schemas (ApiAction, CirrusSearchRequestSet) as JSONSchema and produce to EventGate - https://phabricator.wikimedia.org/T214080
[16:36:52] <stashbot>	 T212529: Standardize datetimes/timestamps in the Data Lake - https://phabricator.wikimedia.org/T212529
[16:36:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:38:09] <wikibugs>	 (03PS2) 10Gehel: elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199)
[16:38:24] <wikibugs>	 (03PS3) 10Gehel: elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199)
[16:39:27] <wikibugs>	 10Operations, 10ops-codfw, 10decommission, 10Patch-For-Review, 10User-jijiki: Reclaim rdb2001, rdb2002 - https://phabricator.wikimedia.org/T209425 (10Papaul) The ports are labeled on the switch but wrong labels   ` papaul@asw-a-codfw> show interfaces descriptions | match "ge-5/0/[0-1]"    ge-5/0/0...
[16:39:41] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199) (owner: 10Gehel)
[16:41:45] <wikibugs>	 (03PS1) 10Dmaza: Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622)
[16:41:46] <robh>	 addshore: thx =]
[16:41:59] <MatmaRex>	 addshore: still around? could you swat another change? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/496072
[16:42:06] <MatmaRex>	 i'll fill in the table in a moment
[16:42:11] <robh>	 !log mw2206.codfw.wmnet is being powered down for firmware update, relying on auto depool function from clean shutdown for mw api server via T215415
[16:42:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:42:14] <stashbot>	 T215415: mw2206.codfw.wmnet memory issues  - https://phabricator.wikimedia.org/T215415
[16:42:30] <addshore>	 MatmaRex: I think robh has the floor now
[16:42:40] <robh>	 oh, i can hold off
[16:42:46] <robh>	 its still in the swat hour and i havent started.
[16:42:56] <MatmaRex>	 i can schedule it for the next swat as well
[16:43:08] <addshore>	 MatmaRex: I dont really have the time right now to commit to another patch :/
[16:43:11] <robh>	 i was about to hit enter and send shutdown ;D
[16:43:17] <robh>	 ok, sounds like its going on next one
[16:43:17] <robh>	 heh
[16:43:26] <MatmaRex>	 yeah. okay. next swat
[16:43:32] <MatmaRex>	 :)
[16:43:57] <dmaza>	 is there a way to test this locally or somewhere else without doing a SWAT ? https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/496202
[16:44:59] <addshore>	 dmaza: I guess beta would be the place to test it first
[16:45:33] <wikibugs>	 10Operations, 10ops-codfw, 10decommission, 10Patch-For-Review, 10User-jijiki: Reclaim rdb2001, rdb2002 - https://phabricator.wikimedia.org/T209425 (10Papaul) ` papaul@asw-a-codfw> show interfaces descriptions | match "ge-5/0/[0-1]"     ge-5/0/0        down  down DISABLED ge-5/0/1        down  down DISABLED
[16:45:35] <wikibugs>	 (03PS2) 10Dmaza: Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622)
[16:46:18] <wikibugs>	 10Operations, 10ops-codfw, 10decommission, 10Patch-For-Review, 10User-jijiki: Reclaim rdb2001, rdb2002 - https://phabricator.wikimedia.org/T209425 (10Papaul)
[16:46:24] <dmaza>	 addshore: how do I get to do that?
[16:46:52] <addshore>	 dmaza: see CommonSettings-labs.php in that same repo
[16:47:50] <dmaza>	 addshore: thanks
[16:51:35] <wikibugs>	 10Operations, 10ops-codfw, 10Discovery-Search (Current work): elastic2038 CPU/memory errors - https://phabricator.wikimedia.org/T217398 (10Papaul) 05Open→03Resolved @Gehel I checked the system log again today no errors. Closing this task for now.
[16:51:54] <gehel>	 papaul: ^ thanks for keeping track!
[16:52:01] * gehel is keeping his fingers crossed anyway!
[16:52:28] <robh>	 ok, im intentionally not silencing mw2206
[16:52:33] <robh>	 it wont cause sms, just irc echo
[16:52:36] <robh>	 and i wanna see it come back
[16:53:08] <papaul>	 gehel: you welcome
[16:55:22] <icinga-wm>	 PROBLEM - Host mw2206 is DOWN: PING CRITICAL - Packet loss = 100%
[16:55:26] <wikibugs>	 10Operations, 10CX-cxserver, 10Citoid, 10Graphoid, and 11 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10mobrovac)
[16:56:09] <wikibugs>	 (03CR) 10Dmaza: [C: 04-1] Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622) (owner: 10Dmaza)
[16:56:14] <jijiki>	 ^ we know about mw2206 
[16:56:47] <robh>	 !log mw2206.codfw.wmnet is being powered down for firmware update, relying on auto depool function from clean shutdown for mw api server via T215415
[16:56:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:56:51] <stashbot>	 T215415: mw2206.codfw.wmnet memory issues  - https://phabricator.wikimedia.org/T215415
[16:57:56] <wikibugs>	 (03PS2) 10Eevans: Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192
[16:58:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192 (owner: 10Eevans)
[16:58:53] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[16:58:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:58:59] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed
[16:58:59] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[16:59:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:59:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:01:34] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-codfw-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw]
[17:01:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:01:37] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster codfw completed
[17:01:37] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[17:01:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:01:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:02:36] <icinga-wm>	 RECOVERY - Host mw2206 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[17:03:12] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10RobH) a:05RobH→03Papaul ` Record:      1 Date/Time:   01/15/2015 23:20:17 Source:      system Severity:    Ok Description: Log cleared. -------------------------------...
[17:03:54] <wikibugs>	 (03PS3) 10Eevans: Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192
[17:04:04] <icinga-wm>	 PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:04:27] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10RobH) Actually, this is out of warranty as of Jan. 19, 2018.  So we may just want to decommission this host, unless we want to try just swapping the memory with an on site...
[17:05:04] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[17:05:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:05:05] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster eqiad completed
[17:05:05] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[17:05:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:05:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:05:08] <icinga-wm>	 RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:06:58] <wikibugs>	 (03PS1) 10Mahveotm: Remove flaggedrevs on the Hugarian Wikipedia, test period completed [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496205 (https://phabricator.wikimedia.org/T210224)
[17:08:21] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+1] "Thanks for the cleanup." [puppet] - 10https://gerrit.wikimedia.org/r/496149 (owner: 10Dzahn)
[17:18:03] <ema>	 ok let's try and serve some cache_upload traffic through ATS. Issues, if any, will be confined to the varnish frontend on cp2002 
[17:18:56] <ema>	 !log cp2002: pool varnish-fe for user traffic, routed through ATS backends T213263
[17:18:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:18:59] <stashbot>	 T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263
[17:19:17] <wikibugs>	 (03CR) 10Mobrovac: [C: 03+1] Ad-hoc Cassandra clusters for deployment-prep [puppet] - 10https://gerrit.wikimedia.org/r/496192 (owner: 10Eevans)
[17:19:22] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=yes; selector: name=cp2002.codfw.wmnet,service=nginx
[17:19:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:19:23] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=yes; selector: name=cp2002.codfw.wmnet,service=varnish-fe
[17:19:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:19:34] <wikibugs>	 (03CR) 10Jforrester: "This is replacing Ic6e268a815, I take it?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622) (owner: 10Dmaza)
[17:26:17] <elukey>	 ema: wow!!
[17:34:34] <XioNoX>	 !log add missing sandbox1-b-eqiad interface to ospf(3) passive on cr1/2-eqiad
[17:34:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:42:17] <wikibugs>	 (03PS1) 10Mholloway: Add WikimediaEditorTasks to extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496210 (https://phabricator.wikimedia.org/T218137)
[17:42:20] <wikibugs>	 (03PS1) 10Mholloway: WikimediaEditorTasks: Add config to InitializeSettings.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496211 (https://phabricator.wikimedia.org/T218137)
[17:42:23] <wikibugs>	 (03PS1) 10Mholloway: WikimediaEditorTasks: Add Beta Cluster config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496212 (https://phabricator.wikimedia.org/T218137)
[17:42:26] <wikibugs>	 (03PS1) 10Mholloway: WikimediaEditorTasks: Load extension in CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496213 (https://phabricator.wikimedia.org/T218137)
[17:43:16] <XioNoX>	 !log s/29073/202425/ on AMS-IX
[17:43:16] <wikibugs>	 (03Abandoned) 10Mholloway: Deploy WikimediaEditorTasks to Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495957 (https://phabricator.wikimedia.org/T218137) (owner: 10Mholloway)
[17:43:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:53:18] <wikibugs>	 (03PS1) 10CRusnov: netbox: ganeti-sync config: Remove quotes [puppet] - 10https://gerrit.wikimedia.org/r/496217
[17:54:33] <wikibugs>	 10Operations, 10Analytics, 10EventBus, 10Core Platform Team (Modern Event Platform (TEC2)), and 3 others: Possibly expand Kafka main-{eqiad,codfw} clusters in Q4 2019. - https://phabricator.wikimedia.org/T217359 (10herron)
[17:54:48] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/496217 (owner: 10CRusnov)
[17:56:27] <wikibugs>	 (03CR) 10CRusnov: [C: 03+2] netbox: ganeti-sync config: Remove quotes [puppet] - 10https://gerrit.wikimedia.org/r/496217 (owner: 10CRusnov)
[17:56:55] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:56:59] <icinga-wm>	 PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:57:11] <wikibugs>	 (03CR) 10D3r1ck01: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496205 (https://phabricator.wikimedia.org/T210224) (owner: 10Mahveotm)
[17:57:23] <XioNoX>	 !log set interface description on fasw-c-codfw:ge-0/0/47
[17:57:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:58:03] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] Add helpers for local testing [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496099 (owner: 10BryanDavis)
[17:58:07] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:58:25] <wikibugs>	 (03Merged) 10jenkins-bot: Add helpers for local testing [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496099 (owner: 10BryanDavis)
[17:59:23] <icinga-wm>	 RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[17:59:37] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10Cmjohnson) I moved DIMM from B side to A side and cleared the log...let's give it a day or so and see if the error follows.
[18:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1800)
[18:04:21] <wikibugs>	 (03CR) 10Mholloway: [C: 03+1] Pass flag use_nodejs10 for maps services [puppet] - 10https://gerrit.wikimedia.org/r/495735 (https://phabricator.wikimedia.org/T215523) (owner: 10MSantos)
[18:05:20] <wikibugs>	 10Operations, 10ops-eqiad, 10DC-Ops, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvirt1015: apparent hardware errors in CPU/Memory - https://phabricator.wikimedia.org/T215012 (10Cmjohnson) @aborrero the CPU is here...let me know when it's safe for me to change.
[18:07:28] <bstorm_>	 !log downtime labstore1006 for troubleshooting T217473
[18:07:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:07:31] <stashbot>	 T217473: labstore1006 spontaneous reboot - https://phabricator.wikimedia.org/T217473
[18:09:01] <bstorm_>	 !log rebooting labstore1006 T217473
[18:09:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:11:30] <wikibugs>	 (03PS2) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:11:32] <wikibugs>	 (03PS1) 10Andrew Bogott: Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224)
[18:11:37] <wikibugs>	 (03CR) 10CRusnov: [V: 03+2 C: 03+2] "Merging." [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[18:11:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133) (owner: 10Andrew Bogott)
[18:11:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224) (owner: 10Andrew Bogott)
[18:11:59] <wikibugs>	 (03PS1) 10Jbond: Exclude /mnt/hdfs from lsof operations in wmf-auto-restarts [puppet] - 10https://gerrit.wikimedia.org/r/496223 (https://phabricator.wikimedia.org/T217646)
[18:13:00] <wikibugs>	 (03Abandoned) 10CRusnov: Update to upstream v2.5.7 tag. [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/493637 (owner: 10CRusnov)
[18:13:55] <wikibugs>	 (03PS2) 10Andrew Bogott: Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224)
[18:13:58] <wikibugs>	 (03PS3) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:14:15] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224) (owner: 10Andrew Bogott)
[18:14:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133) (owner: 10Andrew Bogott)
[18:16:11] <wikibugs>	 (03PS3) 10Andrew Bogott: Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224)
[18:16:13] <wikibugs>	 (03PS4) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:18:21] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] Allocate IPs for ldap replicas [dns] - 10https://gerrit.wikimedia.org/r/496222 (https://phabricator.wikimedia.org/T218224) (owner: 10Andrew Bogott)
[18:23:25] <wikibugs>	 (03PS8) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[18:23:43] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:25:36] <wikibugs>	 (03PS1) 10CRusnov: Update to upstream v2.5.8 tag. [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/496229
[18:26:00] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: dbproxy1012 power supply without power - https://phabricator.wikimedia.org/T217394 (10Cmjohnson) 05Open→03Resolved
[18:27:38] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10Papaul) IDRAC firmware complete please depool server for BIOS upgrade
[18:27:47] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops: mw1280 crashed - https://phabricator.wikimedia.org/T218006 (10Cmjohnson) @MoritzMuehlenhoff  can you please depool the server
[18:28:07] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10Papaul)  Power State ON System Model  PowerEdge R420 System Revision  I System Host Name  Operating System  Operating System Version  Service Tag  B14K842 Express Service...
[18:28:47] <ema>	 !log cp2002: depool varnish-fe after 1 hour ATS experiment T213263
[18:28:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:28:50] <stashbot>	 T213263: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263
[18:29:08] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=no; selector: name=cp2002.codfw.wmnet,service=nginx
[18:29:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:29:09] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=no; selector: name=cp2002.codfw.wmnet,service=varnish-fe
[18:29:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:29:25] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC is ok(changes are expected): https://puppet-compiler.wmflabs.org/compiler1002/15091/" [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:30:29] <robh>	 !log thumbor1004 memtest in progress via T215411
[18:30:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:30:34] <stashbot>	 T215411: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411
[18:31:25] <wikibugs>	 (03PS5) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:31:27] <wikibugs>	 (03PS1) 10Andrew Bogott: Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231
[18:32:55] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133) (owner: 10Andrew Bogott)
[18:32:57] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231 (owner: 10Andrew Bogott)
[18:34:01] <andrewbogott>	 today is not a good day for Andrew vs. the linter
[18:34:06] <wikibugs>	 (03PS2) 10Andrew Bogott: Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231
[18:34:08] <wikibugs>	 (03PS6) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:34:42] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231 (owner: 10Andrew Bogott)
[18:34:52] <wikibugs>	 10Operations, 10Traffic, 10Patch-For-Review: Partial cache_upload traffic switchover to ATS and switchback to Varnish - https://phabricator.wikimedia.org/T213263 (10ema) Preliminary testing went well: the `cp-ats` cluster in codfw served all frontend misses from cp2002 for about one hour between 17:18 and 18...
[18:34:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133) (owner: 10Andrew Bogott)
[18:35:59] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) So attempting to boot this system shows:  Error: Memory initialization warning detected.   I cannot get what dimm has error, since it is overwritten on serial...
[18:36:02] <wikibugs>	 10Operations, 10ops-eqiad, 10DC-Ops, 10Patch-For-Review, 10cloud-services-team (Kanban): cloudvirt1015: apparent hardware errors in CPU/Memory - https://phabricator.wikimedia.org/T215012 (10Andrew) >>! In T215012#5021924, @Cmjohnson wrote: > @aborrero the CPU is here...let me know when it's safe for me t...
[18:36:07] <andrewbogott>	 ffs
[18:36:56] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) ` Error: Memory initialization warning detected. Management Engine Mode                : Active ManagementeEngineeFirmwaretVersiongent: 0002.0001 Copyright (C)...
[18:38:16] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) Next steps:  * Chris attach crash cart ** output on crash cart won't be overwritten like serial console, so the POST error will denote which dimm is bad. ** if...
[18:38:17] <wikibugs>	 (03PS3) 10Andrew Bogott: Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231
[18:38:19] <wikibugs>	 (03PS7) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[18:38:26] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] elasticsearch: mjolnir bulk update lag (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:40:03] <wikibugs>	 (03PS1) 10Ottomata: Enable api-request logging into kafka for group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496235 (https://phabricator.wikimedia.org/T214080)
[18:41:01] <wikibugs>	 (03CR) 10Ppchelko: [C: 03+1] "I've already had https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/495983/ but ok :)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496235 (https://phabricator.wikimedia.org/T214080) (owner: 10Ottomata)
[18:41:16] <wikibugs>	 (03Abandoned) 10Ppchelko: Enable api-request logging into kafka for group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495983 (owner: 10Ppchelko)
[18:42:21] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Enable api-request logging into kafka for group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496235 (https://phabricator.wikimedia.org/T214080) (owner: 10Ottomata)
[18:43:54] <wikibugs>	 (03PS9) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[18:44:09] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging to eventgate-analytics for group1 wikis (duration: 00m 58s)
[18:44:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:44:39] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:44:41] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:47:48] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+1] "This is worth a try.  It may have unintended consequences, but even if it does they may be better than the unintended consequences we alre" [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508) (owner: 10Bstorm)
[18:48:37] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops: mw1280 crashed - https://phabricator.wikimedia.org/T218006 (10jijiki) @Cmjohnson Server has been depooled, ping us to pool it back, tx!
[18:48:48] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[18:49:38] <wikibugs>	 (03PS10) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[18:50:27] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] Move ldap replicas to public IPs [dns] - 10https://gerrit.wikimedia.org/r/496231 (owner: 10Andrew Bogott)
[18:51:35] <jijiki>	 !log Depool mw1280  and mw2206 to hardware issues - T215415 T218006
[18:51:39] <wikibugs>	 (03CR) 10CRusnov: "Test deploys to af-netbox01 correctly." [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/496229 (owner: 10CRusnov)
[18:51:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:51:40] <stashbot>	 T215415: mw2206.codfw.wmnet memory issues  - https://phabricator.wikimedia.org/T215415
[18:51:40] <stashbot>	 T218006: mw1280 crashed - https://phabricator.wikimedia.org/T218006
[18:52:52] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10jijiki) @Papaul Server has been depooled, thank you!
[18:53:32] <wikibugs>	 10Operations, 10Traffic, 10netops, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10jbond) Some comments for consideration  >We set up those (now global across our network!) service IPs behind LVS in all of our sites. is auth dns already behi...
[18:54:46] <wikibugs>	 (03CR) 10Volans: [C: 04-1] "We first need to update the netbox repo. If you look at https://gerrit.wikimedia.org/g/operations/software/netbox/+/refs/heads/master it's" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/496229 (owner: 10CRusnov)
[18:55:07] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops, 10User-jijiki: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10Cmjohnson) DIMM A1 is now showing bad so it looks a DIMM replacement is needed.
[18:55:54] <wikibugs>	 (03CR) 10CRusnov: "> Patch Set 1: Code-Review-1" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/496229 (owner: 10CRusnov)
[18:56:03] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops: mw1280 crashed - https://phabricator.wikimedia.org/T218006 (10Cmjohnson) Record:      42 Date/Time:   03/10/2019 07:43:40 Source:      system Severity:    Non-Critical Description: Correctable memory error rate exceeded for DIMM_B1. -----------------------------------...
[18:56:30] <wikibugs>	 (03CR) 10CRusnov: "Rebase done." [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/496229 (owner: 10CRusnov)
[18:57:39] <icinga-wm>	 PROBLEM - Host mw1280 is DOWN: PING CRITICAL - Packet loss = 100%
[18:58:04] <wikibugs>	 (03CR) 10jenkins-bot: Enable api-request logging into kafka for group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496235 (https://phabricator.wikimedia.org/T214080) (owner: 10Ottomata)
[18:58:08] <wikibugs>	 (03PS2) 10Herron: logstash: add udp json logback localhost compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/496022 (https://phabricator.wikimedia.org/T213899)
[18:59:11] <icinga-wm>	 PROBLEM - Host mw1280.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:59:34] <jijiki>	 ^ Server is being serviced, please ignore
[18:59:59] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops: mw1280 crashed - https://phabricator.wikimedia.org/T218006 (10Cmjohnson) Swapped DIMM B1 with A1 cleared idrac log.
[19:00:05] <jouncebot>	 Deploy window MediaWiki train - Americas version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T1900)
[19:00:49] <icinga-wm>	 RECOVERY - Host mw1280 is UP: PING OK - Packet loss = 0%, RTA = 36.16 ms
[19:01:06] <volans>	 jijiki: is it pooled=inactive? otherwise it will still be targeted by scap
[19:01:20] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] Exclude /mnt/hdfs from lsof operations in wmf-auto-restarts [puppet] - 10https://gerrit.wikimedia.org/r/496223 (https://phabricator.wikimedia.org/T217646) (owner: 10Jbond)
[19:02:11] <jijiki>	 volans: I was  just looking who is doing this weeks trains
[19:02:13] <jijiki>	 train 
[19:02:17] <jijiki>	 it is not inactive 
[19:02:24] <volans>	 ack :)
[19:02:38] <volans>	 I'll leave it to you then ;)
[19:04:27] <icinga-wm>	 RECOVERY - Host mw1280.mgmt is UP: PING OK - Packet loss = 0%, RTA = 38.19 ms
[19:05:53] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+1] "Somebody will be really mad about this, but for now it seems like our best bet to slow the bleeding in NFS server over-utilization" [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508) (owner: 10Bstorm)
[19:11:28] <wikibugs>	 (03PS1) 10Andrew Bogott: Initial site.pp entries for ldap-eqiad-replica01 and 02 [puppet] - 10https://gerrit.wikimedia.org/r/496241 (https://phabricator.wikimedia.org/T46722)
[19:11:31] <wikibugs>	 (03PS2) 10Bstorm: toolforge: limit file size to 50 GB [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508)
[19:11:39] <sjoerddebruin>	 connection issues?
[19:12:24] <sjoerddebruin>	 i see reports of 503's and wikidata search results seem to be dead
[19:12:36] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] Initial site.pp entries for ldap-eqiad-replica01 and 02 [puppet] - 10https://gerrit.wikimedia.org/r/496241 (https://phabricator.wikimedia.org/T46722) (owner: 10Andrew Bogott)
[19:12:37] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqsin on icinga2001 is CRITICAL: cluster=cache_text site=eqsin https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:12:45] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqiad on icinga2001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:13:03] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:13:09] <icinga-wm>	 PROBLEM - HHVM rendering on mw1282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:13] <icinga-wm>	 PROBLEM - HHVM rendering on mw1343 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:15] <icinga-wm>	 PROBLEM - Apache HTTP on mw1315 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:15] <icinga-wm>	 PROBLEM - Nginx local proxy to apache on mw1282 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:15] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1015 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[19:13:17] <icinga-wm>	 PROBLEM - Apache HTTP on mw1279 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:21] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1016 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[19:13:23] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - api_80: Servers mw1348.eqiad.wmnet, mw1281.eqiad.wmnet, mw1314.eqiad.wmnet, mw1233.eqiad.wmnet, mw1343.eqiad.wmnet, mw1347.eqiad.wmnet, mw1345.eqiad.wmnet, mw1223.eqiad.wmnet, mw1282.eqiad.wmnet, mw1221.eqiad.wmnet, mw1317.eqiad.wmnet, mw1224.eqiad.wmnet, mw1279.eqiad.wmnet, mw1312.eqiad.wmnet are marked down but pooled: api-https_443: Serve
[19:13:23] <icinga-wm>	 mnet, mw1281.eqiad.wmnet, mw1314.eqiad.wmnet, mw1233.eqiad.wmnet, mw1343.eqiad.wmnet, mw1347.eqiad.wmnet, mw1345.eqiad.wmnet, mw1223.eqiad.wmnet, mw1282.eqiad.wmnet, mw1221.eqiad.wmnet, mw1317.eqiad.wmnet, mw1279.eqiad.wmnet, mw1312.eqiad.wmnet are marked down but pooled
[19:13:23] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1002 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[19:13:23] <icinga-wm>	  target) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target with seed) timed out before a response was received
[19:13:25] <icinga-wm>	 PROBLEM - citoid endpoints health on scb2004 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received
[19:13:25] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at ulsfo on icinga2001 is CRITICAL: job=varnish-text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:13:25] <icinga-wm>	 PROBLEM - Apache HTTP on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:25] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is CRITICAL: cluster=cache_text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:13:29] <icinga-wm>	 PROBLEM - HHVM rendering on mw1312 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:29] <icinga-wm>	 PROBLEM - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Graphoid
[19:13:31] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2001 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.c
[19:13:31] <icinga-wm>	 on - normal source and target with seed) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) timed out before a response was received
[19:13:35] <icinga-wm>	 PROBLEM - Apache HTTP on mw1343 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:36] <wikibugs>	 (03PS3) 10Bstorm: toolforge: limit file size to 50 GB [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508)
[19:13:37] <icinga-wm>	 PROBLEM - Nginx local proxy to apache on mw1221 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:37] <icinga-wm>	 PROBLEM - Apache HTTP on mw1344 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:37] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1009 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[19:13:37] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2016 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[19:13:37] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2013 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[19:13:39] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2003 is CRITICAL: /{domain}/v1/page/random/title (retrieve a random article title) timed out before a response was received
[19:13:39] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2007 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[19:13:39] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1001 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[19:13:39] <icinga-wm>	 ut before a response was received
[19:13:39] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1010 is CRITICAL: /en.wikipedia.org/v1/page/title/{title}{/revision} (Get rev by title from storage) timed out before a response was received: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) i
[19:13:40] <icinga-wm>	 Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[19:13:43] <wikibugs>	 (03PS3) 10Herron: logstash: add udp json logback localhost compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/496022 (https://phabricator.wikimedia.org/T213899)
[19:13:45] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2004 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target) timed out before a response was received
[19:13:45] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1004 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[19:13:45] <icinga-wm>	  target with seed) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) timed out before a response was received
[19:13:45] <icinga-wm>	 PROBLEM - proton endpoints health on proton2001 is CRITICAL: /{domain}/v1/pdf/{title}/{format}/{type} (Print the Bar page from en.wp.org in A4 format using optimized for reading on mobile devices) timed out before a response was received
[19:13:45] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at esams on icinga2001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:13:45] <icinga-wm>	 PROBLEM - Apache HTTP on mw1317 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[19:13:49] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqsin on icinga2001 is CRITICAL: job=varnish-text site=eqsin https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:13:53] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2014 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[19:13:56] <chaomodus>	 a catastrophe?
[19:13:59] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is CRITICAL: cluster=cache_text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:13:59] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1014 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[19:13:59] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1017 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[19:14:03] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2005 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[19:14:03] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2002 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[19:14:05] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is CRITICAL: cluster=cache_text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:14:17] <icinga-wm>	 RECOVERY - HHVM rendering on mw1282 is OK: HTTP OK: HTTP/1.1 200 OK - 75186 bytes in 2.384 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:19] <icinga-wm>	 RECOVERY - HHVM rendering on mw1343 is OK: HTTP OK: HTTP/1.1 200 OK - 75185 bytes in 0.644 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:21] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2006 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[19:14:21] <icinga-wm>	  target with seed) timed out before a response was received
[19:14:21] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2003 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received
[19:14:21] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1003 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[19:14:23] <icinga-wm>	 RECOVERY - Apache HTTP on mw1315 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 2.603 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:24] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] Exclude /mnt/hdfs from lsof operations in wmf-auto-restarts [puppet] - 10https://gerrit.wikimedia.org/r/496223 (https://phabricator.wikimedia.org/T217646) (owner: 10Jbond)
[19:14:25] <icinga-wm>	 RECOVERY - Nginx local proxy to apache on mw1282 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 619 bytes in 4.276 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:25] <icinga-wm>	 PROBLEM - graphoid endpoints health on scb1002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200)
[19:14:27] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at codfw on icinga2001 is CRITICAL: job=varnish-text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:14:29] <icinga-wm>	 RECOVERY - Apache HTTP on mw1279 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 5.312 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:33] <wikibugs>	 (03PS2) 10Jbond: Exclude /mnt/hdfs from lsof operations in wmf-auto-restarts [puppet] - 10https://gerrit.wikimedia.org/r/496223 (https://phabricator.wikimedia.org/T217646)
[19:14:37] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1016 is OK: All endpoints are healthy
[19:14:37] <icinga-wm>	 RECOVERY - Apache HTTP on mw1312 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 5.412 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:37] <icinga-wm>	 RECOVERY - HHVM rendering on mw1312 is OK: HTTP OK: HTTP/1.1 200 OK - 75186 bytes in 2.040 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:47] <icinga-wm>	 RECOVERY - Nginx local proxy to apache on mw1221 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 619 bytes in 4.187 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:47] <icinga-wm>	 RECOVERY - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid
[19:14:47] <icinga-wm>	 RECOVERY - Apache HTTP on mw1343 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 7.816 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:49] <icinga-wm>	 RECOVERY - Apache HTTP on mw1344 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 5.390 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:51] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1009 is OK: All endpoints are healthy
[19:14:51] <icinga-wm>	 RECOVERY - Apache HTTP on mw1317 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.561 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[19:14:51] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2003 is OK: All endpoints are healthy
[19:14:53] <icinga-wm>	 PROBLEM - citoid endpoints health on scb2001 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received
[19:14:57] <sjoerddebruin>	 yikes
[19:15:07] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1017 is OK: All endpoints are healthy
[19:15:07] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1014 is OK: All endpoints are healthy
[19:15:10] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] toolforge: limit file size to 50 GB [puppet] - 10https://gerrit.wikimedia.org/r/496082 (https://phabricator.wikimedia.org/T122508) (owner: 10Bstorm)
[19:15:13] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2005 is OK: All endpoints are healthy
[19:15:13] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2002 is OK: All endpoints are healthy
[19:15:27] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1003 is OK: All endpoints are healthy
[19:15:29] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2006 is OK: All endpoints are healthy
[19:15:29] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2003 is OK: All endpoints are healthy
[19:15:37] <icinga-wm>	 RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy
[19:15:39] <wikibugs>	 (03PS3) 10Jbond: Exclude /mnt/hdfs from lsof operations in wmf-auto-restarts [puppet] - 10https://gerrit.wikimedia.org/r/496223 (https://phabricator.wikimedia.org/T217646)
[19:15:43] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1015 is OK: All endpoints are healthy
[19:15:47] <icinga-wm>	 RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy
[19:15:51] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy
[19:16:01] <icinga-wm>	 RECOVERY - citoid endpoints health on scb2001 is OK: All endpoints are healthy
[19:16:03] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1010 is OK: All endpoints are healthy
[19:16:03] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2016 is OK: All endpoints are healthy
[19:16:03] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2013 is OK: All endpoints are healthy
[19:16:05] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2007 is OK: All endpoints are healthy
[19:16:09] <icinga-wm>	 RECOVERY - proton endpoints health on proton2001 is OK: All endpoints are healthy
[19:16:11] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2014 is OK: All endpoints are healthy
[19:17:01] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1002 is OK: All endpoints are healthy
[19:17:07] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2001 is OK: All endpoints are healthy
[19:17:11] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1001 is OK: All endpoints are healthy
[19:17:17] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1004 is OK: All endpoints are healthy
[19:17:17] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2004 is OK: All endpoints are healthy
[19:19:17] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:19:53] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqsin on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:20:23] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:20:43] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:21:07] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqsin on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:21:15] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:21:15] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:21:21] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[19:21:55] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:22:15] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[19:24:23] <icinga-wm>	 PROBLEM - eventgate-analytics LVS eqiad on eventgate-analytics.svc.eqiad.wmnet is CRITICAL: /robots.txt (robots.txt check) timed out before a response was received https://wikitech.wikimedia.org/wiki/Event%2A%23EventGate_%28repository%29
[19:25:29] <icinga-wm>	 RECOVERY - eventgate-analytics LVS eqiad on eventgate-analytics.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Event%2A%23EventGate_%28repository%29
[19:26:04] <herron>	 !log performing rolling restart of eqiad logstash instances
[19:26:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:29:33] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw2206 is CRITICAL: connect to address 10.192.32.94 port 5666: Connection refused
[19:30:47] <wikibugs>	 (03PS1) 10Andrew Bogott: install_server: move seaborgium and serpens to Stretch [puppet] - 10https://gerrit.wikimedia.org/r/496244
[19:30:49] <wikibugs>	 (03PS1) 10Andrew Bogott: install_server settings for ldap-eqiad-replica0[12].wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/496245 (https://phabricator.wikimedia.org/T46722)
[19:32:57] <icinga-wm>	 PROBLEM - Host mw2206 is DOWN: PING CRITICAL - Packet loss = 100%
[19:33:06] <wikibugs>	 (03PS2) 10Andrew Bogott: install_server settings for ldap-eqiad-replica0[12].wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/496245 (https://phabricator.wikimedia.org/T46722)
[19:33:20] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] install_server: move seaborgium and serpens to Stretch [puppet] - 10https://gerrit.wikimedia.org/r/496244 (owner: 10Andrew Bogott)
[19:35:23] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] install_server settings for ldap-eqiad-replica0[12].wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/496245 (https://phabricator.wikimedia.org/T46722) (owner: 10Andrew Bogott)
[19:37:59] <wikibugs>	 (03PS11) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[19:38:58] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[19:39:57] <wikibugs>	 (03PS1) 10Gehel: deployment-elastic: upgrade to 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/496248 (https://phabricator.wikimedia.org/T218247)
[19:39:57] <wikibugs>	 (03PS1) 10Gehel: deployment-elastic: upgrade elastic configuration to 6 [puppet] - 10https://gerrit.wikimedia.org/r/496249 (https://phabricator.wikimedia.org/T218247)
[19:40:03] <gehel>	 onimisionipe: ^
[19:40:09] <icinga-wm>	 RECOVERY - Host mw2206 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms
[19:40:31] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw2206 is OK: OK - load average: 5.50, 1.67, 0.58
[19:41:29] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] deployment-elastic: upgrade to 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/496248 (https://phabricator.wikimedia.org/T218247) (owner: 10Gehel)
[19:41:44] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] deployment-elastic: upgrade to 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/496248 (https://phabricator.wikimedia.org/T218247) (owner: 10Gehel)
[19:41:52] <wikibugs>	 (03PS2) 10Gehel: deployment-elastic: upgrade to 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/496248 (https://phabricator.wikimedia.org/T218247)
[19:42:16] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] deployment-elastic: upgrade elastic configuration to 6 [puppet] - 10https://gerrit.wikimedia.org/r/496249 (https://phabricator.wikimedia.org/T218247) (owner: 10Gehel)
[19:42:22] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10Papaul) Service Tag  B14K842 Express Service Code  24012733970 BIOS Version  2.6.0 Firmware Version  2.61.60.60 IP Address(es)  10.193.2.106 iDRAC MAC Address
[19:42:29] <icinga-wm>	 PROBLEM - Check systemd state on ms-be2029 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[19:45:20] <wikibugs>	 10Operations, 10ops-codfw, 10serviceops, 10User-jijiki: mw2206.codfw.wmnet memory issues - https://phabricator.wikimedia.org/T215415 (10Papaul) System log clear  Server can be repool  Monitoring DIMM error
[19:46:07] <jijiki>	 !log Pooling mw2206 - T215415
[19:46:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:46:10] <stashbot>	 T215415: mw2206.codfw.wmnet memory issues  - https://phabricator.wikimedia.org/T215415
[19:46:35] <icinga-wm>	 PROBLEM - Check systemd state on logstash1007 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[19:46:40] <wikibugs>	 (03CR) 10Dmaza: [C: 04-1] "> Patch Set 2:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622) (owner: 10Dmaza)
[19:48:55] <icinga-wm>	 RECOVERY - Check systemd state on logstash1007 is OK: OK - running: The system is fully operational
[19:51:09] <wikibugs>	 (03PS2) 10Gehel: deployment-elastic: upgrade elastic configuration to 6 [puppet] - 10https://gerrit.wikimedia.org/r/496249 (https://phabricator.wikimedia.org/T218247)
[19:51:51] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] deployment-elastic: upgrade elastic configuration to 6 [puppet] - 10https://gerrit.wikimedia.org/r/496249 (https://phabricator.wikimedia.org/T218247) (owner: 10Gehel)
[19:58:01] <icinga-wm>	 RECOVERY - Check systemd state on ms-be2029 is OK: OK - running: The system is fully operational
[19:59:08] <wikibugs>	 10Operations, 10Continuous-Integration-Infrastructure, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10hashar)  > As far as i can tell this only needs to go to contint2001.wikimedia.org and contint1001.wikimedia.org  Those run the CI st...
[20:00:04] <jouncebot>	 cscott, arlolra, subbu, bearND, halfak, and Amir1: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Services – Parsoid / Citoid / Mobileapps / ORES / … deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T2000).
[20:03:19] <herron>	 !log increased index.mapping.total_fields.limit to 1350 on index logstash-2019.03.13
[20:03:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:07:35] <icinga-wm>	 PROBLEM - NFS on labstore1006 is CRITICAL: connect to address 208.80.154.7 and port 2049: Connection refused https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Labstore
[20:11:25] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[20:12:25] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is CRITICAL: cluster=cache_text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:12:37] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[20:12:49] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqsin on icinga2001 is CRITICAL: cluster=cache_text site=eqsin https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:12:55] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqiad on icinga2001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:13:15] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:13:21] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2002 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target with seed) timed out before a response was received
[20:13:21] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2005 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[20:13:22] <niedzielski>	 i'm seeing sporadic 503s when editing. this is the request that fails: https://www.mediawiki.org/w/api.php?action=visualeditor&format=json&paction=wikitext&page=Reading%2FWeb%2FChores&uselang=en
[20:13:23] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1007 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:13:29] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2015 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:29] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1013 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:29] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2012 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:13:29] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2010 is CRITICAL: /en.wikipedia.org/v1/page/title/{title}{/revision} (Get rev by title from storage) timed out before a response was received
[20:13:31] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1008 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:33] <icinga-wm>	 PROBLEM - graphoid endpoints health on scb1002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200)
[20:13:33] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1012 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:33] <icinga-wm>	 PROBLEM - graphoid endpoints health on scb1003 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200)
[20:13:33] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2009 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:13:35] <icinga-wm>	 PROBLEM - Restbase LVS codfw on restbase.svc.codfw.wmnet is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received https://wikitech.wikimedia.org/wiki/RESTBase
[20:13:35] <icinga-wm>	 PROBLEM - Nginx local proxy to apache on mw1287 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[20:13:37] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2005 is CRITICAL: /{domain}/v1/page/metadata/{title}{/revision}{/tid} (retrieve extended metadata for Video article on English Wikipedia) timed out before a response was received: /{domain}/v1/page/mobile-html/{title}{/revision}{/tid} (Get page content HTML for test page) timed out before a response was received: /{domain}/v1/media/image/featured/{year}/{month}/{day} (retrieve featured 
[20:13:37] <icinga-wm>	 ril 29, 2016) timed out before a response was received
[20:13:39] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at ulsfo on icinga2001 is CRITICAL: job=varnish-text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:13:39] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2006 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[20:13:39] <icinga-wm>	  target) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target with seed) timed out before a response was received
[20:13:39] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2003 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[20:13:39] <icinga-wm>	  target with seed) timed out before a response was received
[20:13:40] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1003 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[20:13:40] <icinga-wm>	  target with seed) timed out before a response was received
[20:13:41] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb1001 is CRITICAL: /{domain}/v1/page/metadata/{title}{/revision}{/tid} (retrieve extended metadata for Video article on English Wikipedia) timed out before a response was received: /{domain}/v1/page/mobile-html/{title}{/revision}{/tid} (Get page content HTML for test page) timed out before a response was received: /{domain}/v1/media/image/featured/{year}/{month}/{day} (retrieve featured 
[20:13:41] <icinga-wm>	 ril 29, 2016) timed out before a response was received: /{domain}/v1/page/random/title (retrieve a random article title) timed out before a response was received
[20:13:42] <icinga-wm>	 PROBLEM - Apache HTTP on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[20:13:43] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1016 is CRITICAL: PYBAL CRITICAL - CRITICAL - api_80: Servers mw1229.eqiad.wmnet, mw1227.eqiad.wmnet, mw1340.eqiad.wmnet, mw1347.eqiad.wmnet, mw1345.eqiad.wmnet, mw1234.eqiad.wmnet, mw1342.eqiad.wmnet, mw1289.eqiad.wmnet are marked down but pooled: api-https_443: Servers mw1284.eqiad.wmnet, mw1346.eqiad.wmnet, mw1281.eqiad.wmnet, mw1229.eqiad.wmnet, mw1233.eqiad.wmnet, mw1286.eqiad.wmne
[20:13:43] <icinga-wm>	 mnet, mw1234.eqiad.wmnet, mw1235.eqiad.wmnet, mw1342.eqiad.wmnet, mw1289.eqiad.wmnet, mw1315.eqiad.wmnet, mw1285.eqiad.wmnet are marked down but pooled
[20:13:53] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1015 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:53] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1016 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:55] <icinga-wm>	 PROBLEM - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/Graphoid
[20:13:57] <icinga-wm>	 PROBLEM - HHVM rendering on mw1342 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Application_servers
[20:13:57] <icinga-wm>	 PROBLEM - citoid endpoints health on scb2006 is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received
[20:13:57] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2011 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:57] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at esams on icinga2001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:13:57] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase-dev1006 is CRITICAL: /en.wikipedia.org/v1/page/title/{title}{/revision} (Get rev by title from storage) timed out before a response was received: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:57] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase-dev1004 is CRITICAL: /en.wikipedia.org/v1/page/title/{title}{/revision} (Get rev by title from storage) timed out before a response was received: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:13:59] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1011 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:14:03] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1002 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[20:14:03] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2001 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[20:14:07] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1009 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:14:09] <icinga-wm>	 PROBLEM - cxserver endpoints health on scb1003 is CRITICAL: /v2/page/{sourcelanguage}/{targetlanguage}/{title}{/revision} (Fetch enwiki Oxygen page) timed out before a response was received
[20:14:09] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2016 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:14:09] <niedzielski>	 i'm seeing sporadic 503s when editing. this is the request that fails: https://www.mediawiki.org/w/api.php?action=visualeditor&format=json&paction=wikitext&page=Reading%2FWeb%2FChores&uselang=en
[20:14:11] <icinga-wm>	 PROBLEM - Restbase LVS eqiad on restbase.svc.eqiad.wmnet is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200) https://wikitech.wikimedia.org/wiki/RESTBase
[20:14:11] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is CRITICAL: cluster=cache_text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:14:13] <icinga-wm>	 PROBLEM - mobileapps endpoints health on scb2002 is CRITICAL: /{domain}/v1/page/metadata/{title}{/revision}{/tid} (retrieve extended metadata for Video article on English Wikipedia) timed out before a response was received: /{domain}/v1/page/media/{title}{/revision}{/tid} (Get media in test page) timed out before a response was received: /{domain}/v1/page/summary/{title}{/revision}{/tid} (Get summary for test page) timed out befo
[20:14:13] <icinga-wm>	  received: /{domain}/v1/page/mobile-html/{title}{/revision}{/tid} (Get page content HTML for test page) timed out before a response was received
[20:14:15] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1004 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - bad article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - 
[20:14:15] <icinga-wm>	  target) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - normal source and target with seed) timed out before a response was received
[20:14:15] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb1001 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received: /{domain}/v1/article/creation/translation/{source}{/seed} (article.creation.translation - bad seed) timed out before a response was received
[20:14:19] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is CRITICAL: cluster=cache_text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:14:23] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2014 is CRITICAL: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) is CRITICAL: Test Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:14:23] <icinga-wm>	 PROBLEM - recommendation_api endpoints health on scb2004 is CRITICAL: /{domain}/v1/article/creation/morelike/{seed} (article.creation.morelike - good article title) timed out before a response was received
[20:14:25] <icinga-wm>	 PROBLEM - graphoid endpoints health on scb2002 is CRITICAL: /{domain}/v1/{format}/{title}/{revid}/{id} (retrieve PNG from mediawiki.org) is CRITICAL: Test retrieve PNG from mediawiki.org returned the unexpected status 400 (expecting: 200)
[20:14:27] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - CRITICAL - api_80: Servers mw1288.eqiad.wmnet, mw1231.eqiad.wmnet are marked down but pooled: api-https_443: Servers mw1232.eqiad.wmnet, mw1287.eqiad.wmnet, mw1226.eqiad.wmnet, mw1222.eqiad.wmnet, mw1345.eqiad.wmnet, mw1341.eqiad.wmnet are marked down but pooled
[20:14:43] <logmsgbot>	 !log bsitzmann@deploy1001 Finished deploy [mobileapps/deploy@5f8e4e6]: Update mobileapps to 5865552 (7074964 d6dc3cd fbc6262) (duration: 01m 49s)
[20:14:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:14:47] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase2018 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) timed out before a response was received
[20:14:47] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1017 is CRITICAL: /en.wikipedia.org/v1/page/title/{title}{/revision} (Get rev by title from storage) timed out before a response was received: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received: /en.wikipedia.org/v1/page/graph/png/{title}/{revision}/{graph_id} (Get a graph from Graphoid) i
[20:14:47] <icinga-wm>	 Get a graph from Graphoid returned the unexpected status 400 (expecting: 200)
[20:14:47] <icinga-wm>	 PROBLEM - restbase endpoints health on restbase1014 is CRITICAL: /en.wikipedia.org/v1/feed/featured/{yyyy}/{mm}/{dd} (Retrieve aggregated feed content for April 29, 2016) timed out before a response was received
[20:15:00] <icinga-wm>	 RECOVERY - HHVM rendering on mw1342 is OK: HTTP OK: HTTP/1.1 200 OK - 75162 bytes in 2.351 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[20:15:01] <bearND>	 ^^ failed
[20:15:02] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1015 is OK: All endpoints are healthy
[20:15:02] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1016 is OK: All endpoints are healthy
[20:15:02] <icinga-wm>	 RECOVERY - citoid endpoints health on scb2006 is OK: All endpoints are healthy
[20:15:08] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase-dev1006 is OK: All endpoints are healthy
[20:15:08] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1011 is OK: All endpoints are healthy
[20:15:08] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase-dev1004 is OK: All endpoints are healthy
[20:15:10] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqsin on icinga2001 is CRITICAL: job=varnish-text site=eqsin https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:15:12] <icinga-wm>	 PROBLEM - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is CRITICAL: /v2/page/{sourcelanguage}/{targetlanguage}/{title}{/revision} (Fetch enwiki Oxygen page) timed out before a response was received https://wikitech.wikimedia.org/wiki/CX
[20:15:13] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: correct APT repository to only use OSS packages [puppet] - 10https://gerrit.wikimedia.org/r/496255 (https://phabricator.wikimedia.org/T218113)
[20:15:16] <icinga-wm>	 RECOVERY - cxserver endpoints health on scb1003 is OK: All endpoints are healthy
[20:15:24] <icinga-wm>	 RECOVERY - Restbase LVS eqiad on restbase.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase
[20:15:38] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1006 is CRITICAL: CRITICAL: Hosts in IPVS but unknown to PyBal: set([mw1316.eqiad.wmnet, mw1231.eqiad.wmnet, mw1283.eqiad.wmnet, mw1235.eqiad.wmnet, mw1221.eqiad.wmnet, mw1342.eqiad.wmnet, mw1343.eqiad.wmnet, mw1315.eqiad.wmnet, mw1345.eqiad.wmnet, mw1230.eqiad.wmnet])
[20:16:04] <bearND>	 MCS deploy on canary scb2001.codfw.wmnet failed, probably due to RB issues
[20:16:06] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2011 is OK: All endpoints are healthy
[20:16:06] <icinga-wm>	 RECOVERY - Cxserver LVS eqiad on cxserver.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/CX
[20:16:08] <icinga-wm>	 RECOVERY - Graphoid LVS eqiad on graphoid.svc.eqiad.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Graphoid
[20:16:14] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2001 is OK: All endpoints are healthy
[20:16:14] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1009 is OK: All endpoints are healthy
[20:16:14] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2016 is OK: All endpoints are healthy
[20:16:16] <mdholloway>	 bearND: didn't see you start a deploy.  looks like now may not be the best time, considering all the node services alerts...
[20:16:20] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2002 is OK: All endpoints are healthy
[20:16:22] <mdholloway>	 what failed?
[20:16:22] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1001 is OK: All endpoints are healthy
[20:16:22] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1004 is OK: All endpoints are healthy
[20:16:30] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2004 is OK: All endpoints are healthy
[20:16:34] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2014 is OK: All endpoints are healthy
[20:16:36] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1017 is OK: All endpoints are healthy
[20:16:36] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2018 is OK: All endpoints are healthy
[20:16:37] <bearND>	 mdholloway: The bot was down, and joined after I started the deploy
[20:16:38] <icinga-wm>	 RECOVERY - graphoid endpoints health on scb2002 is OK: All endpoints are healthy
[20:16:38] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1014 is OK: All endpoints are healthy
[20:16:44] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2010 is OK: All endpoints are healthy
[20:16:44] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2012 is OK: All endpoints are healthy
[20:16:48] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2006 is OK: All endpoints are healthy
[20:16:54] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1016 is OK: PYBAL OK - All pools are healthy
[20:17:10] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1002 is OK: All endpoints are healthy
[20:17:11] <mdholloway>	 bearND: ah, i see
[20:17:38] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy
[20:18:10] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] elasticsearch: correct APT repository to only use OSS packages [puppet] - 10https://gerrit.wikimedia.org/r/496255 (https://phabricator.wikimedia.org/T218113) (owner: 10Gehel)
[20:18:22] <icinga-wm>	 RECOVERY - Nginx local proxy to apache on mw1287 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.205 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[20:18:22] <bearND>	 mdholloway: The deploy on the canary server failed. I'm gonna try again in a few minutes. Looks like RB is stabilizing...
[20:18:28] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1012 is OK: All endpoints are healthy
[20:18:30] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb1001 is OK: All endpoints are healthy
[20:18:37] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] elasticsearch: correct APT repository to only use OSS packages [puppet] - 10https://gerrit.wikimedia.org/r/496255 (https://phabricator.wikimedia.org/T218113) (owner: 10Gehel)
[20:19:00] <wikibugs>	 10Operations, 10Readers-Web-Backlog, 10VisualEditor: [Bug] Sporadic 503 errors when editing - https://phabricator.wikimedia.org/T218252 (10Niedzielski)
[20:20:14] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqsin on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:20:24] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+1] WikimediaEditorTasks: Load extension in CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496213 (https://phabricator.wikimedia.org/T218137) (owner: 10Mholloway)
[20:20:35] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+1] WikimediaEditorTasks: Add Beta Cluster config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496212 (https://phabricator.wikimedia.org/T218137) (owner: 10Mholloway)
[20:20:40] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1006 is OK: OK: no difference between hosts in IPVS/PyBal
[20:21:52] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:21:54] <logmsgbot>	 !log bsitzmann@deploy1001 Started deploy [mobileapps/deploy@5f8e4e6]: Update mobileapps to 5865552 (7074964 d6dc3cd fbc6262)
[20:21:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:21:56] <wikibugs>	 (03PS1) 10Ottomata: Revert "Enable api-request logging into kafka for group1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496256
[20:22:00] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2015 is OK: All endpoints are healthy
[20:22:00] <icinga-wm>	 RECOVERY - mobileapps endpoints health on scb2005 is OK: All endpoints are healthy
[20:22:13] <wikibugs>	 10Operations, 10Readers-Web-Backlog, 10VisualEditor: [Bug] Sporadic 503 errors when editing - https://phabricator.wikimedia.org/T218252 (10Operator873) Problem confirmed on Simple English Wikipedia. HTTP 503 errors and "Server response empty" errors.
[20:22:14] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:22:20] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqsin on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:22:26] <wikibugs>	 (03PS3) 10Dmaza: Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622)
[20:22:28] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[20:22:52] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:23:21] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Revert "Enable api-request logging into kafka for group1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496256 (owner: 10Ottomata)
[20:23:30] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:23:38] <icinga-wm>	 RECOVERY - Apache HTTP on mw1342 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 0.103 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[20:23:40] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:24:09] <wikibugs>	 (03CR) 10Gergő Tisza: "The choice of cluster/DB should be reviewed by DBAs but adding them on this patch without context is probably not that helpful. It would b" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496211 (https://phabricator.wikimedia.org/T218137) (owner: 10Mholloway)
[20:24:23] <wikibugs>	 10Operations, 10Readers-Web-Backlog, 10VisualEditor: [Bug] Sporadic 503 errors when editing - https://phabricator.wikimedia.org/T218252 (10Kb03) Confirmed on Enwiki, errored out while tagging a page with Twinkle
[20:24:55] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Disabling api-request logging to eventgate-analytics for group1 wikis to investigate possible outage (duration: 00m 56s)
[20:24:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:25:29] <logmsgbot>	 !log bsitzmann@deploy1001 Finished deploy [mobileapps/deploy@5f8e4e6]: Update mobileapps to 5865552 (7074964 d6dc3cd fbc6262) (duration: 03m 35s)
[20:25:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:25:32] <icinga-wm>	 RECOVERY - Restbase LVS codfw on restbase.svc.codfw.wmnet is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/RESTBase
[20:25:34] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2005 is OK: All endpoints are healthy
[20:26:49] <wikibugs>	 (03CR) 10jenkins-bot: Revert "Enable api-request logging into kafka for group1" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496256 (owner: 10Ottomata)
[20:27:18] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase2009 is OK: All endpoints are healthy
[20:27:18] <icinga-wm>	 RECOVERY - graphoid endpoints health on scb1002 is OK: All endpoints are healthy
[20:28:33] <logmsgbot>	 !log arlolra@deploy1001 Started deploy [parsoid/deploy@e2e44bc]: Updating Parsoid to ea80d1b
[20:28:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:30:50] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1008 is OK: All endpoints are healthy
[20:30:50] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1013 is OK: All endpoints are healthy
[20:31:44] <wikibugs>	 (03PS3) 10Jbond: Add component/ci wikimedia repository to CI hosts [puppet] - 10https://gerrit.wikimedia.org/r/495681 (https://phabricator.wikimedia.org/T212774)
[20:32:36] <icinga-wm>	 PROBLEM - Host mr1-esams.oob is DOWN: CRITICAL - Time to live exceeded (164.138.24.90)
[20:32:40] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2002 is OK: All endpoints are healthy
[20:34:10] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[20:35:01] <volans>	 XioNoX: unrelated but worth mentioning: mr1-esams.oob ^^^
[20:35:11] <logmsgbot>	 !log arlolra@deploy1001 Finished deploy [parsoid/deploy@e2e44bc]: Updating Parsoid to ea80d1b (duration: 06m 38s)
[20:35:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:35:22] <wikibugs>	 (03PS1) 10BryanDavis: toolforge: install python-mwclient with grid exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/496258 (https://phabricator.wikimedia.org/T218242)
[20:35:44] <icinga-wm>	 PROBLEM - IPv4 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 56 probes of 434 (alerts on 35) - https://atlas.ripe.net/measurements/1791210/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts
[20:36:08] <icinga-wm>	 RECOVERY - graphoid endpoints health on scb1003 is OK: All endpoints are healthy
[20:36:10] <icinga-wm>	 RECOVERY - restbase endpoints health on restbase1007 is OK: All endpoints are healthy
[20:37:41] <wikibugs>	 (03PS1) 10Dmaza: Enforce 8 char password length requirements for non-privileged users on BETA [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496260
[20:37:52] <icinga-wm>	 RECOVERY - Host mr1-esams.oob is UP: PING OK - Packet loss = 0%, RTA = 143.80 ms
[20:37:58] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb1003 is OK: All endpoints are healthy
[20:37:58] <icinga-wm>	 RECOVERY - recommendation_api endpoints health on scb2003 is OK: All endpoints are healthy
[20:38:14] <wikibugs>	 10Operations, 10Wikimedia-Incident: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10jijiki) p:05Triage→03High
[20:38:52] <wikibugs>	 10Operations, 10Wikimedia-Incident: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10jijiki)
[20:39:00] <wikibugs>	 (03CR) 10Dmaza: [C: 04-1] Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622) (owner: 10Dmaza)
[20:39:01] <XioNoX>	 volans: thx
[20:39:08] <XioNoX>	 (it's back)
[20:40:54] <icinga-wm>	 RECOVERY - IPv4 ping to codfw on ripe-atlas-codfw is OK: OK - failed 2 probes of 434 (alerts on 35) - https://atlas.ripe.net/measurements/1791210/#!map https://wikitech.wikimedia.org/wiki/Network_monitoring%23RIPE_alerts
[20:44:08] <mobrovac>	 what happened here?
[20:44:18] <mobrovac>	 XioNoX: edge server brain freeze?
[20:44:32] <mobrovac>	 hm no, lvs
[20:45:38] <volans>	 mobrovac: no the current theory is eventgate
[20:45:41] <wikibugs>	 (03CR) 10Jforrester: "Per Stas." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[20:46:31] <mobrovac>	 volans: rb and scb services would not be affected then
[20:46:35] <volans>	 mobrovac: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/496256
[20:46:45] <volans>	 we're sstill trying to understand the chain of events
[20:47:10] <volans>	 and not yet 100% sure it was that tbh, it's the current theory though
[20:48:07] <cdanis>	 mobrovac: the current theory is some eventgate blip caused all the PHP workers on the api_appservers to get stuck
[20:48:20] <cdanis>	 and thus also the impacts on restbase and scb
[20:48:30] <mobrovac>	 makes sense cdanis, thnx
[20:50:53] <wikibugs>	 (03PS2) 10Smalyshev: Enable WikibaseCirrusSearch on wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[20:51:33] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] Tweak build reporting output [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496100 (owner: 10BryanDavis)
[20:51:39] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] all: install locales used in Toolforge grid [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496101 (https://phabricator.wikimedia.org/T218072) (owner: 10BryanDavis)
[20:51:45] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] php72: tideways & switch to component/php72 [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496102 (https://phabricator.wikimedia.org/T202825) (owner: 10BryanDavis)
[20:51:49] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] php: install php5-xdebug [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496103 (https://phabricator.wikimedia.org/T202825) (owner: 10BryanDavis)
[20:51:53] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] python35: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496104 (owner: 10BryanDavis)
[20:51:55] <wikibugs>	 (03Merged) 10jenkins-bot: Tweak build reporting output [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496100 (owner: 10BryanDavis)
[20:51:58] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable WikibaseCirrusSearch on wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[20:52:00] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+2] node10: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496105 (https://phabricator.wikimedia.org/T195103) (owner: 10BryanDavis)
[20:52:02] <wikibugs>	 (03Merged) 10jenkins-bot: all: install locales used in Toolforge grid [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496101 (https://phabricator.wikimedia.org/T218072) (owner: 10BryanDavis)
[20:52:08] <wikibugs>	 (03Merged) 10jenkins-bot: php72: tideways & switch to component/php72 [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496102 (https://phabricator.wikimedia.org/T202825) (owner: 10BryanDavis)
[20:52:16] <wikibugs>	 (03Merged) 10jenkins-bot: php: install php5-xdebug [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496103 (https://phabricator.wikimedia.org/T202825) (owner: 10BryanDavis)
[20:52:19] <wikibugs>	 (03Merged) 10jenkins-bot: python35: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496104 (owner: 10BryanDavis)
[20:52:23] <wikibugs>	 (03Merged) 10jenkins-bot: node10: new images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/496105 (https://phabricator.wikimedia.org/T195103) (owner: 10BryanDavis)
[20:58:01] <wikibugs>	 10Operations, 10VisualEditor, 10Readers-Web-Backlog (Tracking): [Bug] Sporadic 503 errors when editing - https://phabricator.wikimedia.org/T218252 (10Jdlrobson)
[20:58:22] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad,codfw]
[20:58:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:58:23] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics install -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[20:58:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:58:34] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[20:58:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:58:39] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed
[20:58:39] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[20:58:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:58:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:02:17] <wikibugs>	 (03PS12) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[21:03:32] <wikibugs>	 (03PS1) 10BryanDavis: Add python 3.5 and nodejs 10 types [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/496265
[21:03:44] <icinga-wm>	 PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:04:32] <icinga-wm>	 PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:05:04] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] toolforge: install python-mwclient with grid exec_environ [puppet] - 10https://gerrit.wikimedia.org/r/496258 (https://phabricator.wikimedia.org/T218242) (owner: 10BryanDavis)
[21:05:38] <wikibugs>	 (03PS1) 10Ottomata: Use httpGet  for liveness_probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/496271
[21:05:52] <wikibugs>	 10Operations, 10Wikimedia-Incident: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10Pchelolo)
[21:06:03] <wikibugs>	 10Operations, 10Wikimedia-Incident: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10Pchelolo)
[21:06:59] <wikibugs>	 10Operations, 10serviceops, 10Wikimedia-Incident: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10jijiki)
[21:07:02] <wikibugs>	 (03PS2) 10Ottomata: Use httpGet  for liveness_probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/496271
[21:07:52] <icinga-wm>	 PROBLEM - Request latencies on acrux is CRITICAL: instance=10.192.0.93:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:08:00] <icinga-wm>	 PROBLEM - Request latencies on acrab is CRITICAL: instance=10.192.16.26:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:08:06] <icinga-wm>	 RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:08:28] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:08:44] <wikibugs>	 (03PS3) 10Ottomata: Use httpGet  for liveness_probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/496271
[21:09:59] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-codfw-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw]
[21:10:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:10:02] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster codfw completed
[21:10:02] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[21:10:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:10:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:10:12] <wikibugs>	 10Operations, 10serviceops, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), 10Core Platform Team Kanban (Doing), and 2 others: Influx of service errors Mar 13: 19:12-19:14 UTC and 20:12-20:15 - https://phabricator.wikimedia.org/T218255 (10mobrovac)
[21:10:52] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:11:11] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[21:11:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:11:12] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster eqiad completed
[21:11:12] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[21:11:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:11:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:18:00] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:18:38] <icinga-wm>	 RECOVERY - Request latencies on acrux is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:18:44] <icinga-wm>	 RECOVERY - Request latencies on acrab is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:19:12] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:19:14] <icinga-wm>	 RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:20:50] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for sukhe - https://phabricator.wikimedia.org/T217438 (10Slaporte) >>! In T217438#5003384, @Nuria wrote: > Can @slaporte confirm NDA was signed?  Yes, he has an NDA with the Foundation.
[21:21:38] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to stat1007 for sukhe - https://phabricator.wikimedia.org/T217438 (10Nuria) Ok, approved on my end  please see June 30th end date.
[21:24:40] <icinga-wm>	 PROBLEM - Request latencies on acrux is CRITICAL: instance=10.192.0.93:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:25:16] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:26:30] <icinga-wm>	 PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:28:54] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:29:32] <icinga-wm>	 RECOVERY - Request latencies on acrux is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:30:10] <icinga-wm>	 RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:37:20] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:38:32] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:41:03] <wikibugs>	 10Operations, 10Analytics, 10EventBus, 10Prod-Kubernetes, 10serviceops: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata)
[21:41:24] <wikibugs>	 10Operations, 10Analytics, 10EventBus, 10Prod-Kubernetes, 10serviceops: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268 (10Ottomata) @akosiaris let's try to figure this out tomorrow. :)
[21:44:34] <icinga-wm>	 PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:45:44] <icinga-wm>	 RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[21:56:22] <wikibugs>	 (03PS2) 10Dmaza: Enforce 8 char password length requirements for non-privileged users on BETA [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496260
[21:57:00] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10MarcoAurelio) @Legoktm @herron Could we have an update on this one, please? Thanks.
[21:57:39] <wikibugs>	 (03PS1) 10Ottomata: Disable api-request eventgate logging for group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496329 (https://phabricator.wikimedia.org/T218268)
[21:58:45] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Disable api-request eventgate logging for group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496329 (https://phabricator.wikimedia.org/T218268) (owner: 10Ottomata)
[22:00:08] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Disabling api-request logging to eventgate-analytics for group0 wikis until we solve T218268 (duration: 00m 56s)
[22:00:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:00:13] <stashbot>	 T218268: eventgate-analytics k8s pods occasionally can't produce to kafka - https://phabricator.wikimedia.org/T218268
[22:00:23] <wikibugs>	 (03PS3) 10Smalyshev: Enable WikibaseCirrusSearch on wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[22:01:37] <wikibugs>	 (03PS4) 10Smalyshev: Enable WikibaseCirrusSearch on wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490648 (https://phabricator.wikimedia.org/T215684) (owner: 10Jforrester)
[22:04:03] <wikibugs>	 (03CR) 10jenkins-bot: Disable api-request eventgate logging for group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496329 (https://phabricator.wikimedia.org/T218268) (owner: 10Ottomata)
[22:06:02] <wikibugs>	 (03PS4) 10Dmaza: Enforce 8 char password length requirements for non-privileged users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622)
[22:13:19] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10Reedy) Presumably it’s the usual no activity == no update...
[22:15:00] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10Tgr) There has been a lot of activity on Discourse, OTOH. @Qgil might be able to say more on that.
[22:18:39] <wikibugs>	 (03CR) 10Dmaza: "There will be a "similar" version deployed on beta to run a quick test." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496202 (https://phabricator.wikimedia.org/T211622) (owner: 10Dmaza)
[22:21:59] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10MarcoAurelio) @Reedy As far as I could see, T52864#3241375 is resolved, and T52864#3940724 is as well. GNU seems happy with Postorius and HyperKitty...
[22:30:24] <wikibugs>	 10Operations, 10Traffic, 10netops, 10Patch-For-Review, 10Performance-Team (Radar): Anycast (Auth)DNS - https://phabricator.wikimedia.org/T98006 (10ayounsi) >>! In T98006#5022090, @jbond wrote: Thanks for your comments! > is auth dns already behind LVS.  AuthDNS is not behind LVS and currently have static...
[22:38:26] <icinga-wm>	 PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[22:40:50] <icinga-wm>	 RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[22:49:54] <wikibugs>	 (03PS1) 10Andrew Bogott: openldap: role/profile refactor for 'labs' and 'labtest' roles [puppet] - 10https://gerrit.wikimedia.org/r/496335 (https://phabricator.wikimedia.org/T46722)
[22:50:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] openldap: role/profile refactor for 'labs' and 'labtest' roles [puppet] - 10https://gerrit.wikimedia.org/r/496335 (https://phabricator.wikimedia.org/T46722) (owner: 10Andrew Bogott)
[22:55:06] <wikibugs>	 (03PS1) 10Andrew Bogott: ldap: moved some passwords to hiera to echo changes in prod [labs/private] - 10https://gerrit.wikimedia.org/r/496337
[22:57:53] <wikibugs>	 (03PS2) 10Andrew Bogott: openldap: role/profile refactor for 'labs' and 'labtest' roles [puppet] - 10https://gerrit.wikimedia.org/r/496335 (https://phabricator.wikimedia.org/T46722)
[22:57:59] <wikibugs>	 (03CR) 10Andrew Bogott: [V: 03+2 C: 03+2] ldap: moved some passwords to hiera to echo changes in prod [labs/private] - 10https://gerrit.wikimedia.org/r/496337 (owner: 10Andrew Bogott)
[23:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: My dear minions, it's time we take the moon! Just kidding. Time for Evening SWAT (Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190313T2300).
[23:00:05] <jouncebot>	 MatmaRex, Smalyshev, Jdlrobson, and RoanKattouw: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[23:00:13] <SMalyshev>	 here
[23:00:17] <RoanKattouw>	 I''ll SWAT
[23:01:25] <MatmaRex>	 hi
[23:02:51] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10Reedy) >>! In T52864#5022898, @MarcoAurelio wrote: > @Reedy As far as I could see, T52864#3241375 is resolved, and T52864#3940724 is as well. GNU se...
[23:04:53] <RoanKattouw>	 SMalyshev: Does your config change depend on your wmf.21 change or are they independent?
[23:05:46] <SMalyshev>	 RoanKattouw: the code change should go first
[23:05:58] <SMalyshev>	 otherwise config will enable code that is broken
[23:06:16] <SMalyshev>	 (which is not a huge deal since its on test/beta but still)
[23:06:16] <RoanKattouw>	 OK so then everyone's waiting for Jenkins except for jdlrobson 
[23:06:28] <jdlrobson>	 hello
[23:06:56] <RoanKattouw>	 You ready for your SWAT?
[23:07:07] <RoanKattouw>	 Oh wait that's ALSO a wmf.21 change
[23:07:19] <RoanKattouw>	 I guess everyone's gonna wait for the Jenkins queue for a bit then
[23:11:08] <jdlrobson>	 i am yes
[23:11:14] <jdlrobson>	 when you and Jenkins are :)
[23:30:02] <logmsgbot>	 !log catrope@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/GrowthExperiments/: Instrumentation fixes (T217802) (duration: 00m 57s)
[23:30:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:30:05] <stashbot>	 T217802: Help panel: impressions with VE are incorrectly marked as reading editor_interface - https://phabricator.wikimedia.org/T217802
[23:32:57] <SMalyshev>	 mmmkay jenkins is done
[23:33:14] <RoanKattouw>	 Yup, pulling onto mwdebug now
[23:33:28] <RoanKattouw>	 SMalyshev, jdlrobson, MatmaRex: Your patches are now on mwdebug1002, please test
[23:34:01] * MatmaRex wakes up
[23:34:05] <MatmaRex>	 looking
[23:34:26] <SMalyshev>	 ok checking.... wikidata seems to be very slow
[23:35:30] <MatmaRex>	 yeah mwdebug1002 is being extremely slow
[23:35:39] <SMalyshev>	 specifically via mwdebug yes
[23:35:48] <jdlrobson>	 on it
[23:35:52] <SMalyshev>	 tried without it and everything is fine
[23:36:20] <jdlrobson>	 ditto
[23:36:25] <MatmaRex>	 this has been the case for the last few months of SWAT though, so i'm not too worried. it seems to be at the normal operational level, heh
[23:37:14] <RoanKattouw>	 Yeah in my experience mwdebug is slow especially right after new code's been deployed to it
[23:37:24] <MatmaRex>	 we've waited 30 minutes for jenkins, we can wait 2 minutes for a page to load…
[23:37:49] <SMalyshev>	 RoanKattouw: looks ok to me
[23:38:07] <MatmaRex>	 my test page finallyl loaded and looks okay
[23:38:53] <wikibugs>	 (03PS3) 10Catrope: Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068 (owner: 10Smalyshev)
[23:39:06] <wikibugs>	 (03CR) 10Catrope: [C: 03+2] Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068 (owner: 10Smalyshev)
[23:39:20] <RoanKattouw>	 Syncing SMalyshev's WikibaseCirrusSearch change now
[23:39:29] <logmsgbot>	 !log catrope@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/WikibaseCirrusSearch/: Fix hook return values (duration: 00m 58s)
[23:39:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:40:19] <wikibugs>	 (03Merged) 10jenkins-bot: Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068 (owner: 10Smalyshev)
[23:40:39] <RoanKattouw>	 Now syncing MatmaRex's MobileFrontend change
[23:41:27] <logmsgbot>	 !log catrope@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/MobileFrontend/: Fix animation when visual section editing enabled on mobile only (T218167) (duration: 00m 58s)
[23:41:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:41:30] <stashbot>	 T218167:  The page jitters slightly when opening a section with section editing enabled on mobile VE - https://phabricator.wikimedia.org/T218167
[23:43:12] <RoanKattouw>	 SMalyshev: Your config change is live on mwdebug1002, please test (if possible?)
[23:43:58] <SMalyshev>	 RoanKattouw: great, works just as expected
[23:45:00] <SMalyshev>	 RoanKattouw: you can deploy
[23:45:04] <RoanKattouw>	 Syncing now
[23:45:21] <RoanKattouw>	 jdlrobson: Did you manage to test yours on mwdebug?
[23:45:31] <jdlrobson>	 RoanKattouw: yup all good
[23:45:54] <MatmaRex>	 (thanks)
[23:45:58] <logmsgbot>	 !log catrope@deploy1001 Synchronized wmf-config/WikibaseSearchSettings.php: Fix builder class definition for WBCS (duration: 00m 56s)
[23:45:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:47:03] <wikibugs>	 (03CR) 10jenkins-bot: Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068 (owner: 10Smalyshev)
[23:47:14] <RoanKattouw>	 OK syncing yours now
[23:48:03] <logmsgbot>	 !log catrope@deploy1001 Synchronized php-1.33.0-wmf.21/skins/MinervaNeue/: Remove unnecessary parameter from getHistoryPageAction (duration: 00m 56s)
[23:48:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:48:46] <SMalyshev>	 RoanKattouw: thanks!
[23:49:27] <RoanKattouw>	 And that's a wrap!