[00:00:05] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor Q:How do functions break up? A:They stop calling each other. Rise for Evening SWAT (Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T0000).
[00:00:05] <jouncebot>	 ebernhardson, bmansurov, and Jdlrobson: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[00:00:12] <ebernhardson>	 \o
[00:00:17] <ebernhardson>	 i can ship things
[00:00:37] <bmansurov>	 here
[00:01:00] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+2] Labs: Enable reader demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493103 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[00:02:02] <wikibugs>	 (03Merged) 10jenkins-bot: Labs: Enable reader demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493103 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[00:02:45] <wikibugs>	 (03CR) 10BBlack: [C: 03+2] Remove dead ulsfo cp servers [dns] - 10https://gerrit.wikimedia.org/r/493094 (https://phabricator.wikimedia.org/T167377) (owner: 10BBlack)
[00:03:14] <ebernhardson>	 bmansurov: labs only, so shipping now
[00:03:20] <bmansurov>	 ebernhardson: ok thanks
[00:03:28] <wikibugs>	 (03PS2) 10EBernhardson: [cirrus] autocomplete: enable subphrase suggester builds on officewiki (1/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492031 (https://phabricator.wikimedia.org/T150153)
[00:03:36] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492031 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:03:57] <logmsgbot>	 !log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: noop sync for labs files gerrit:493103 (duration: 00m 54s)
[00:03:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:04:35] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] autocomplete: enable subphrase suggester builds on officewiki (1/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492031 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:04:44] <wikibugs>	 (03PS4) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[00:04:53] <wikibugs>	 (03CR) 10jenkins-bot: Labs: Enable reader demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493103 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[00:04:55] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] autocomplete: enable subphrase suggester builds on officewiki (1/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492031 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:05:20] <jdlrobson>	 ebernhardson: i am ready when you are :)
[00:06:48] <ebernhardson>	 jdlrobson: alright, i kicked jenkins
[00:07:23] <logmsgbot>	 !log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [cirrus] autocomplete: enable subphrase suggester builds on officewiki (1/2) (duration: 00m 54s)
[00:07:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:08:49] <bmansurov>	 Thanks again, ebernhardson 
[00:09:14] <ebernhardson>	 bmansurov: no problem
[00:09:59] <wikibugs>	 (03PS2) 10EBernhardson: [cirrus] autocomplete: enable subphrase matching for officewiki (2/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492032 (https://phabricator.wikimedia.org/T150153)
[00:10:12] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492032 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:11:22] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] autocomplete: enable subphrase matching for officewiki (2/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492032 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:14:01] <logmsgbot>	 !log ebernhardson@deploy1001 sync-file aborted: (no justification provided) (duration: 00m 01s)
[00:14:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:15:06] <logmsgbot>	 !log ebernhardson@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [cirrus] autocomplete: enable subphrase matching for officewiki (2/2) (duration: 00m 54s)
[00:15:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:15:54] <bd808>	 !log Manually changed logging level and restarted Horizon on labweb100[12]
[00:15:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:16:51] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] autocomplete: enable subphrase matching for officewiki (2/2) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492032 (https://phabricator.wikimedia.org/T150153) (owner: 10EBernhardson)
[00:22:07] <ebernhardson>	 jdlrobson: you're up on mwdebug1001
[00:22:08] <icinga-wm>	 PROBLEM - Check systemd state on wdqs1006 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[00:23:42] <bd808>	 !log Disabled puppet on labweb100[12]
[00:23:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:25:18] <jdlrobson>	 on it
[00:27:40] <wikibugs>	 (03PS14) 10Paladox: Introduce gr-wikimedia-prettify-ci-comments [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/489483 (https://phabricator.wikimedia.org/T215658)
[00:28:10] <jdlrobson>	 ebernhardson: youcan sync
[00:29:43] <ebernhardson>	 syncing
[00:30:01] <bd808>	 !log Re-enabled puppet on labweb100[12]
[00:30:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:30:24] <logmsgbot>	 !log ebernhardson@deploy1001 Synchronized php-1.33.0-wmf.19/skins/MinervaNeue/resources/skins.minerva.scripts/errorLogging.js: MinervaNeue: Allow us to distinguish errors for logged in users (duration: 00m 53s)
[00:30:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:32:58] <jdlrobson>	 thanks ebernhardson 
[00:34:38] <ebernhardson>	 with that, SWAT should be complete
[00:34:47] <ebernhardson>	 oops, actually i have one more of my own :)
[00:34:54] <wikibugs>	 (03PS3) 10EBernhardson: [cirrus] decrease regex timeouts by 25% and drop timeout hack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492989 (https://phabricator.wikimedia.org/T216860) (owner: 10DCausse)
[00:34:59] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+2] [cirrus] decrease regex timeouts by 25% and drop timeout hack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492989 (https://phabricator.wikimedia.org/T216860) (owner: 10DCausse)
[00:35:58] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] decrease regex timeouts by 25% and drop timeout hack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492989 (https://phabricator.wikimedia.org/T216860) (owner: 10DCausse)
[00:38:21] <logmsgbot>	 !log ebernhardson@deploy1001 Synchronized wmf-config/CirrusSearch-common.php: [cirrus] decrease regex timeouts by 25% and drop timeout hack (duration: 00m 53s)
[00:38:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:39:17] <ebernhardson>	 ok, now SWAT complete
[00:39:52] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] decrease regex timeouts by 25% and drop timeout hack [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492989 (https://phabricator.wikimedia.org/T216860) (owner: 10DCausse)
[00:41:02] <icinga-wm>	 PROBLEM - puppet last run on cp3039 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[00:41:59] <wikibugs>	 10Operations, 10Discovery-Search, 10Wikimedia-Logstash: upgrade logstash and the logstash elasticsearch cluster to 5.6.14 - https://phabricator.wikimedia.org/T216052 (10EBernhardson) I tested logstash 5.6.14 with elastic 6.5.4 along with api-feature-usage in vagrant today, everything looks to continue workin...
[00:45:49] <wikibugs>	 (03PS1) 10CRusnov: Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138
[00:51:27] <wikibugs>	 (03PS5) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[00:54:11] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138 (owner: 10CRusnov)
[01:11:45] <icinga-wm>	 RECOVERY - puppet last run on cp3039 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[01:34:09] <wikibugs>	 (03PS2) 10CRusnov: Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138 (https://phabricator.wikimedia.org/T217072)
[01:38:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138 (https://phabricator.wikimedia.org/T217072) (owner: 10CRusnov)
[01:59:24] <wikibugs>	 (03PS3) 10CRusnov: Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138 (https://phabricator.wikimedia.org/T217072)
[01:59:30] <wikibugs>	 10Operations, 10netops: ulsfo <-> router1.corp BGP sessions down - https://phabricator.wikimedia.org/T217207 (10eliza) hard boot router1 and now it's up.  Find attached the SysLog  {F28292104}
[02:03:42] <wikibugs>	 10Operations, 10netops: ulsfo <-> router1.corp BGP sessions down - https://phabricator.wikimedia.org/T217207 (10ayounsi) 05Open→03Resolved
[02:03:44] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Netbox module for Spicerack [software/spicerack] - 10https://gerrit.wikimedia.org/r/493138 (https://phabricator.wikimedia.org/T217072) (owner: 10CRusnov)
[02:11:07] <icinga-wm>	 PROBLEM - puppet last run on ganeti1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[02:26:34] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10matmarex) May be related to using IPv6? https://www.mediawiki...
[02:42:27] <icinga-wm>	 RECOVERY - puppet last run on ganeti1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[03:00:04] <jouncebot>	 kart_: Time to snap out of that daydream and deploy . Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T0300).
[03:09:29] <kart_>	 !log Manual run of unpublished ContentTranslation draft purge script (T216983)
[03:09:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:09:33] <stashbot>	 T216983: Run unpublished draft purge script for CX (Week of 24/02) - https://phabricator.wikimedia.org/T216983
[03:11:52] <wikibugs>	 (03PS2) 10Mathew.onipe: nginx: add lua support regardless of version [puppet/nginx] - 10https://gerrit.wikimedia.org/r/492711 (https://phabricator.wikimedia.org/T216681)
[03:16:33] <wikibugs>	 (03PS6) 10Mathew.onipe: tlsproxy: add prometheus option [puppet] - 10https://gerrit.wikimedia.org/r/491972 (https://phabricator.wikimedia.org/T216681)
[03:17:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] tlsproxy: add prometheus option [puppet] - 10https://gerrit.wikimedia.org/r/491972 (https://phabricator.wikimedia.org/T216681) (owner: 10Mathew.onipe)
[03:58:22] <wikibugs>	 10Operations, 10ExternalGuidance, 10Traffic, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review: Deliver mobile-based version for automatic translations - https://phabricator.wikimedia.org/T212197 (10santhosh) >  @santhosh is there any scenario in a future architecture of the extension e whe...
[04:28:18] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.18; 2019-02-19), 10Patch-For-Review, and 3 others: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10aaron) The callback/get-set-cycle delay metric at...
[04:58:25] <icinga-wm>	 RECOVERY - Check systemd state on wdqs1006 is OK: OK - running: The system is fully operational
[04:58:47] <SMalyshev>	 !log repooled wdqs1006
[04:58:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:05:03] <kart_>	 !log Finished manual run of unpublished ContentTranslation draft purge script (T216983)
[05:05:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:05:06] <stashbot>	 T216983: Run unpublished draft purge script for CX (Week of 24/02) - https://phabricator.wikimedia.org/T216983
[06:07:46] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1076 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493151
[06:09:22] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1076 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493151 (owner: 10Marostegui)
[06:10:55] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1076 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493151 (owner: 10Marostegui)
[06:12:23] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1076 (duration: 01m 08s)
[06:12:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:13:12] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1076 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493151 (owner: 10Marostegui)
[06:14:41] <wikibugs>	 (03CR) 10Marostegui: "> BTW, jenkins will not be happy with this commit, because" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[06:26:49] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1076" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493152
[06:28:28] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1076" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493152 (owner: 10Marostegui)
[06:29:33] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1076" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493152 (owner: 10Marostegui)
[06:30:39] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1076 (duration: 00m 55s)
[06:30:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:35:42] <wikibugs>	 10Operations, 10Analytics, 10Analytics-Kanban, 10Product-Analytics, 10Patch-For-Review: dbstore1002 Mysql errors - https://phabricator.wikimedia.org/T213670 (10Marostegui) MySQL just crashed again: ` Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysql...
[06:36:22] <elukey>	 lol
[06:36:32] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1076" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493152 (owner: 10Marostegui)
[06:36:34] <elukey>	 execution day is close
[06:38:12] <marostegui>	 Yeah, last breathes from dbstore1002...
[06:51:22] <elukey>	 poor dbstore1002
[06:51:28] <elukey>	 after all these years of service
[06:51:29] <elukey>	 :D
[06:53:30] <marostegui>	 dbstore1002 must be jealous of dbstore1001, which is still enjoying a second chance after the big storage crash it had a year ago
[07:00:12] <wikibugs>	 (03PS1) 10Mathew.onipe: refactor lua support option [puppet] - 10https://gerrit.wikimedia.org/r/493154 (https://phabricator.wikimedia.org/T216681)
[07:01:30] <marostegui>	 !log Deploy schema change on s5 codfw master (db2052), this will generate lag on codfw - T86342
[07:01:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:01:34] <stashbot>	 T86342: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342
[07:03:39] <wikibugs>	 (03PS1) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[07:04:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[07:09:49] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on thumbor1003 - https://phabricator.wikimedia.org/T217121 (10jijiki) 05Open→03Invalid Server was being reimaged
[07:10:39] <wikibugs>	 (03PS2) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[07:11:51] <wikibugs>	 (03PS1) 10Elukey: Remove any occurrence of inherit [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/493156
[07:12:32] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[07:15:26] <wikibugs>	 (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1002/14887/" [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/493156 (owner: 10Elukey)
[07:19:54] <wikibugs>	 (03PS3) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[07:20:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[07:25:21] <kart_>	 dah
[08:02:49] <marostegui>	 !log Global rename of HeavyTony → QTHCCAN by alanajjar - T217222
[08:02:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:02:53] <stashbot>	 T217222: Global rename of HeavyTony → QTHCCAN: supervision needed - https://phabricator.wikimedia.org/T217222
[08:03:47] <wikibugs>	 (03PS1) 10Elukey: hadoop analytics: move Yarn rmstore from zookeeper to HDFS [puppet] - 10https://gerrit.wikimedia.org/r/493158 (https://phabricator.wikimedia.org/T216952)
[08:13:56] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM, please attach a PCC run as well" [puppet] - 10https://gerrit.wikimedia.org/r/493098 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[08:14:13] <wikibugs>	 (03PS1) 10Elukey: yarn-site.xml: automatically prepend hdfs:// details to HDFSURIs [puppet/cdh] - 10https://gerrit.wikimedia.org/r/493159 (https://phabricator.wikimedia.org/T216952)
[08:15:34] <wikibugs>	 (03CR) 10Filippo Giunchedi: "LGTM, this will also change (and thus bounce) rsyslog on all kafka_shipper hosts, although that should be fine even if puppet hasn't ran o" [puppet] - 10https://gerrit.wikimedia.org/r/493102 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[08:15:36] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] yarn-site.xml: automatically prepend hdfs:// details to HDFSURIs [puppet/cdh] - 10https://gerrit.wikimedia.org/r/493159 (https://phabricator.wikimedia.org/T216952) (owner: 10Elukey)
[08:16:51] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493163
[08:19:01] <wikibugs>	 (03PS2) 10Elukey: hadoop test analytics: simplify Yarn rmstore URI [puppet] - 10https://gerrit.wikimedia.org/r/493158 (https://phabricator.wikimedia.org/T216952)
[08:21:41] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/14890/" [puppet] - 10https://gerrit.wikimedia.org/r/493158 (https://phabricator.wikimedia.org/T216952) (owner: 10Elukey)
[08:23:54] <wikibugs>	 (03PS1) 10Elukey: hadoop analytics: move Yarn rmstore from zookeeper to HDFS [puppet] - 10https://gerrit.wikimedia.org/r/493164 (https://phabricator.wikimedia.org/T216952)
[08:25:23] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] Improve logging of errors, remove spurious print statements [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492647 (owner: 10Giuseppe Lavagetto)
[08:25:39] <wikibugs>	 (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1002/14891/an-master1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/493164 (https://phabricator.wikimedia.org/T216952) (owner: 10Elukey)
[08:25:53] <wikibugs>	 (03CR) 10jenkins-bot: Improve logging of errors, remove spurious print statements [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492647 (owner: 10Giuseppe Lavagetto)
[08:26:15] <wikibugs>	 (03CR) 10jenkins-bot: Do not try to pull/push if no registry is defined. [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492648 (owner: 10Giuseppe Lavagetto)
[08:26:42] <marostegui>	 !log Retroactive log, T216444 Global rename of Дагиров Умар → Takhirgeran Umar was done by alanajjar
[08:26:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:26:46] <stashbot>	 T216444: Global rename of Дагиров Умар → Takhirgeran Umar: supervision needed - https://phabricator.wikimedia.org/T216444
[08:27:11] <wikibugs>	 (03CR) 10Vgutierrez: [C: 04-1] "check comments inline" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[08:36:36] <wikibugs>	 (03PS4) 10Filippo Giunchedi: rsyslog: always use mmjsonparse when shipping to kafka [puppet] - 10https://gerrit.wikimedia.org/r/492632 (https://phabricator.wikimedia.org/T213189)
[08:36:43] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+2] rsyslog: always use mmjsonparse when shipping to kafka [puppet] - 10https://gerrit.wikimedia.org/r/492632 (https://phabricator.wikimedia.org/T213189) (owner: 10Filippo Giunchedi)
[08:37:38] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493163 (owner: 10Marostegui)
[08:38:42] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493163 (owner: 10Marostegui)
[08:40:02] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1082 for mysql upgrade (duration: 00m 54s)
[08:40:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:40:21] <marostegui>	 !log Deploy schema change on db1082 - will generate lag on labsdb:s5 - T86342
[08:40:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:40:24] <stashbot>	 T86342: Dropping page.page_no_title_convert on wmf databases - https://phabricator.wikimedia.org/T86342
[08:41:09] <godog>	 !log enable mmjsonparse by default on kafka outputs - T213189
[08:41:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:41:12] <stashbot>	 T213189: Enable rsyslog mmjsonparse module fleetwide - https://phabricator.wikimedia.org/T213189
[08:41:27] <wikibugs>	 (03PS4) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[08:42:41] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[08:44:10] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Remove any occurrence of inherit [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/493156 (owner: 10Elukey)
[08:44:42] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493163 (owner: 10Marostegui)
[08:45:08] <wikibugs>	 10Operations, 10Acme-chief, 10Traffic: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez)
[08:45:21] <wikibugs>	 10Operations, 10Acme-chief, 10Traffic: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez) p:05Triage→03High
[08:47:34] <wikibugs>	 (03PS1) 10Elukey: Update the zookeeper module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/493166
[08:47:54] <wikibugs>	 (03PS1) 10Vgutierrez: acme_chief: Avoid showing diffs of private keys [puppet] - 10https://gerrit.wikimedia.org/r/493167 (https://phabricator.wikimedia.org/T217231)
[08:49:32] <wikibugs>	 (03PS5) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[08:50:10] <wikibugs>	 (03CR) 10Vgutierrez: [C: 03+2] acme_chief: Avoid showing diffs of private keys [puppet] - 10https://gerrit.wikimedia.org/r/493167 (https://phabricator.wikimedia.org/T217231) (owner: 10Vgutierrez)
[08:50:19] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[08:51:38] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1001/14895/" [puppet] - 10https://gerrit.wikimedia.org/r/493166 (owner: 10Elukey)
[08:51:45] <wikibugs>	 (03PS2) 10Elukey: Update the zookeeper module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/493166
[08:51:47] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] Update the zookeeper module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/493166 (owner: 10Elukey)
[08:53:00] <wikibugs>	 10Operations, 10Acme-chief, 10Traffic, 10Patch-For-Review: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez) 05Open→03Resolved Thanks @Volans for reporting the issue <3
[08:53:11] <wikibugs>	 10Operations: cloudvirt1015: update raid config and move to 10Gb - https://phabricator.wikimedia.org/T217140 (10Peachey88)
[09:02:07] <wikibugs>	 10Operations, 10Acme-chief, 10Traffic: puppet leaks sensitive cryptographic acme-chief material - https://phabricator.wikimedia.org/T217231 (10Vgutierrez)
[09:05:56] <marostegui>	 !log Stop MySQL on db1082 for mysql upgrade
[09:05:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:12:40] <wikibugs>	 (03PS1) 10Elukey: hadoop analytics: set analytics1071 rack config to A5 [puppet] - 10https://gerrit.wikimedia.org/r/493169 (https://phabricator.wikimedia.org/T212348)
[09:13:45] <wikibugs>	 (03PS1) 10Volans: icinga: add htpasswd users and salt [labs/private] - 10https://gerrit.wikimedia.org/r/493170
[09:13:55] <wikibugs>	 (03PS1) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[09:14:54] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[09:15:43] <wikibugs>	 (03PS2) 10Gilles: Oversample navtiming on ruwiki and eswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493055 (https://phabricator.wikimedia.org/T187299)
[09:16:02] <wikibugs>	 (03CR) 10Gilles: Oversample navtiming on ruwiki and eswiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493055 (https://phabricator.wikimedia.org/T187299) (owner: 10Gilles)
[09:16:30] <wikibugs>	 (03PS2) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[09:17:28] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Slowly repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493172
[09:18:45] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Slowly repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493172 (owner: 10Marostegui)
[09:19:14] <wikibugs>	 (03CR) 10Volans: "The related private patch is Ie988fe71bd446e60219051b9f582677eb9d8e8c6" [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[09:19:51] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Slowly repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493172 (owner: 10Marostegui)
[09:21:03] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool db1082 after mysql upgrade (duration: 00m 54s)
[09:21:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:25:55] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Repool db1082 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493173
[09:26:58] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Slowly repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493172 (owner: 10Marostegui)
[09:27:55] <wikibugs>	 (03PS3) 10Volans: elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel)
[09:28:03] <wikibugs>	 (03PS4) 10Volans: elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel)
[09:29:00] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Performance-Team, 10Patch-For-Review, 10User-Elukey: Consider removing the last traces of nutcracker in Mediawiki configs - https://phabricator.wikimedia.org/T214275 (10elukey) nutcracker seems also used by thumbor when contacting its own memcached cluster, but due to T...
[09:32:55] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Repool db1082 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493173 (owner: 10Marostegui)
[09:33:57] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Repool db1082 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493173 (owner: 10Marostegui)
[09:35:07] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool in API db1082 after mysql upgrade (duration: 00m 53s)
[09:35:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:38:21] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Repool db1082 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493173 (owner: 10Marostegui)
[09:39:58] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Fully repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493175
[09:40:46] <wikibugs>	 (03CR) 10Jcrespo: "> So what do you want to do? Merge 449185 first?" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[09:40:54] <wikibugs>	 (03CR) 10Volans: "Patch looks good, if not too much trouble I'd prefer it to be split in two, so that the additional feature of the administrative module is" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341 (owner: 10Gehel)
[09:41:07] <wikibugs>	 (03CR) 10Volans: [C: 03+2] elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel)
[09:41:09] <wikibugs>	 (03CR) 10Jcrespo: "s/am/aim/" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[09:41:58] <wikibugs>	 (03CR) 10Marostegui: [C: 03+1] "+1 as I said earlier, let's deploy and keep polishing it while seeing how it goes!" [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[09:47:21] <wikibugs>	 (03Merged) 10jenkins-bot: elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel)
[09:47:39] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Fully repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493175 (owner: 10Marostegui)
[09:48:46] <wikibugs>	 (03CR) 10jenkins-bot: elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel)
[09:49:18] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Fully repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493175 (owner: 10Marostegui)
[09:49:40] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Fully repool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493175 (owner: 10Marostegui)
[09:50:21] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493177
[09:50:43] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Fully repool db1082 (duration: 00m 54s)
[09:50:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:52:23] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493177 (owner: 10Marostegui)
[09:54:08] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493177 (owner: 10Marostegui)
[09:54:17] <wikibugs>	 (03PS3) 10Giuseppe Lavagetto: Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793
[09:55:23] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1077 for MySQL upgrade (duration: 00m 53s)
[09:55:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:55:39] <marostegui>	 !log Stop MySQL on db1077 for mysql upgrade - this will generate lag on labsdb:s3
[09:55:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:01:24] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [dns] - 10https://gerrit.wikimedia.org/r/493095 (owner: 10BBlack)
[10:01:49] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493177 (owner: 10Marostegui)
[10:05:49] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "Couple of minor nitpicks inline, looks good otherwise. Feel free to merge without further review if updated." (033 comments) [dns] - 10https://gerrit.wikimedia.org/r/493050 (owner: 10BBlack)
[10:21:24] <wikibugs>	 (03PS9) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921)
[10:21:29] <wikibugs>	 (03PS6) 10Mathew.onipe: relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921)
[10:21:39] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10Gilles) Did something happen on 2019-02-21? I was looking at the ulsfo RUM perf metrics for the change in that DC. No change is apparent after the 2019-02-19 config change...
[10:21:41] <wikibugs>	 (03PS4) 10DCausse: Plugins for elasticsearch 6.5.4 [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/491297 (https://phabricator.wikimedia.org/T199791)
[10:22:57] <wikibugs>	 (03CR) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[10:27:40] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC is noop: https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/14896/console" [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[10:29:33] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Slowly repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493182
[10:30:10] <wikibugs>	 (03CR) 10Mathew.onipe: "No unexpected change from PCC: https://puppet-compiler.wmflabs.org/compiler1002/14898/" [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[10:32:32] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Slowly repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493182 (owner: 10Marostegui)
[10:33:07] <wikibugs>	 (03CR) 10MarcoAurelio: Restore bureaucrat rights on hi.wiktionary to default (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492447 (https://phabricator.wikimedia.org/T214765) (owner: 10Sau226)
[10:33:38] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Slowly repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493182 (owner: 10Marostegui)
[10:35:01] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1077 with low weight after MySQL upgrade (duration: 00m 53s)
[10:35:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:35:53] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Slowly repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493182 (owner: 10Marostegui)
[10:39:52] <wikibugs>	 (03PS19) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[10:40:05] <wikibugs>	 (03CR) 10DCausse: cirrus: fallback option to use localssl via acme subject (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[10:41:21] <godog>	 jynus marostegui db1115 has a bunch of unknowns in icinga for backups, known?
[10:41:47] <jynus>	 yes, as in handling it
[10:42:07] <jynus>	 not know why, but I guess some kind of mysql issue
[10:42:17] <wikibugs>	 (03PS20) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[10:42:21] <jynus>	 mysql connection
[10:43:01] <godog>	 yeah that seems like if nrpe is doing the connection, thanks
[10:43:47] <jynus>	 my first gut feeling tells me that maybe backuping the monitoring server may make the monitoring itslef fail?
[10:44:30] <marostegui>	 you mean backuping db1115's content?
[10:44:31] <wikibugs>	 (03CR) 10DCausse: cirrus: fallback option to use localssl via acme subject (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[10:44:49] <jynus>	 last tendril backup started at 2019-02-27 01:55:08
[10:45:24] <jynus>	 that is almost 9 h ago
[10:45:33] <marostegui>	 did it finish?
[10:45:39] <jynus>	 yes
[10:45:52] <jynus>	 however, the alerts are from 16h
[10:46:01] <jynus>	 so maybe some deploy or schema change
[10:46:36] <marostegui>	 Not sure I am following, why would a deploy or a schema change affect db1115?
[10:46:46] <jynus>	 don't worry, I will take care of this, don't waste your time on this too
[10:47:11] <marostegui>	 hehe, thanks, let me know if you need help 
[10:47:15] <jynus>	 will update if I need help, don't want to interrupt your work for something that probably is trivial
[10:47:24] <marostegui>	 :)
[10:48:16] <jynus>	 We could not connect to the backup metadata database
[10:49:14] <wikibugs>	 (03PS21) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[10:50:18] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+2] "No log entries afaics for log4j since stopping gerrit log4j logging" [puppet] - 10https://gerrit.wikimedia.org/r/492640 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi)
[10:50:22] <wikibugs>	 (03CR) 10Jbond: "thanks Ricardo, have addresses all comments except the minimum password length will ask moritz for a suggestion" (0313 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[10:50:32] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Give more traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493187
[10:52:18] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Give more traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493187 (owner: 10Marostegui)
[10:53:17] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Give more traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493187 (owner: 10Marostegui)
[10:53:33] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[10:54:59] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Give more traffic to db1077 after MySQL upgrade (duration: 00m 54s)
[10:55:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:55:03] <wikibugs>	 (03PS22) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[10:55:34] <wikibugs>	 (03CR) 10Filippo Giunchedi: [C: 03+2] "PCC https://puppet-compiler.wmflabs.org/compiler1002/14899/" [puppet] - 10https://gerrit.wikimedia.org/r/492640 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi)
[10:55:43] <wikibugs>	 (03PS3) 10Filippo Giunchedi: logstash: remove log4j input [puppet] - 10https://gerrit.wikimedia.org/r/492640 (https://phabricator.wikimedia.org/T213899)
[10:57:33] <wikibugs>	 (03PS23) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[10:58:50] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Give more traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493187 (owner: 10Marostegui)
[10:59:02] <wikibugs>	 (03PS10) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921)
[10:59:05] <wikibugs>	 (03PS7) 10Mathew.onipe: relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921)
[10:59:07] <godog>	 akosiaris: gah, I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/492640 however there was a calico change in there too, I haven't yet puppet-merge though, anything I'd need to do?
[10:59:18] <wikibugs>	 (03CR) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[11:02:15] <wikibugs>	 (03PS24) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[11:03:42] <godog>	 akosiaris: AFAICS the change is safe to merge as nothing will get changed by puppet, merging
[11:04:58] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: prometheus-statsd-exporter: Run as prometheus-statsd-exporter [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/493196
[11:06:11] <akosiaris>	 godog: no that's ok. That's an informational file. One I was anyway planning to update today
[11:06:15] <wikibugs>	 (03CR) 10Jbond: Add password reset function to ipmi module (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:06:17] <akosiaris>	 feel free to merge
[11:06:51] <godog>	 akosiaris: kk, thanks!
[11:07:21] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:13:47] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1016 is CRITICAL: CRITICAL: Services in IPVS but unknown to PyBal: set([10.2.2.36:4560])
[11:15:19] <wikibugs>	 (03PS6) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[11:15:30] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC is noop!! https://puppet-compiler.wmflabs.org/compiler1002/14900/" [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[11:15:48] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1006 is CRITICAL: CRITICAL: Services in IPVS but unknown to PyBal: set([10.2.2.36:4560])
[11:16:10] <wikibugs>	 (03CR) 10Mathew.onipe: "PCC changes are expected! https://puppet-compiler.wmflabs.org/compiler1002/14901/" [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[11:16:53] <volans>	 godog: ^^^
[11:17:01] <volans>	 seems related to your last change
[11:17:36] <godog>	 indeed, thanks volans 
[11:17:54] <godog>	 !log roll-restart pybal after removing logstash log4j service
[11:17:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:17:57] <wikibugs>	 (03CR) 10Jbond: "> Patch Set 18: Code-Review-1" (0315 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:18:32] <godog>	 should recover shortly
[11:19:00] <godog>	 or break in hideous ways, we'll find out!
[11:19:25] <volans>	 lol
[11:19:31] <volans>	 hoping for the former :)
[11:19:43] <volans>	 it's a limitation on our puppettization?
[11:19:45] <godog>	 me too, also because now I've jinxed it
[11:19:54] <volans>	 of not properly cleaning up a service
[11:19:57] <wikibugs>	 (03CR) 10Santhosh: Enable edittag for ExternalGuidance in CX and VE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[11:20:17] <godog>	 no I think we're not restarting/reloading pybal on config changes on purpose
[11:20:20] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: deployment_server: Remove deprecated deployment-charts clone [puppet] - 10https://gerrit.wikimedia.org/r/493111
[11:20:22] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: Run helm repo update more often [puppet] - 10https://gerrit.wikimedia.org/r/493112
[11:20:24] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: Pull every minute from git the charts [puppet] - 10https://gerrit.wikimedia.org/r/493113
[11:20:26] <wikibugs>	 (03CR) 10Jbond: Add password reset function to ipmi module (0313 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:20:44] <wikibugs>	 (03PS25) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[11:21:36] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Run helm repo update more often [puppet] - 10https://gerrit.wikimedia.org/r/493112 (owner: 10Alexandros Kosiaris)
[11:21:38] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: More traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493202
[11:21:42] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] deployment_server: Remove deprecated deployment-charts clone [puppet] - 10https://gerrit.wikimedia.org/r/493111 (owner: 10Alexandros Kosiaris)
[11:22:47] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: More traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493202 (owner: 10Marostegui)
[11:22:51] <godog>	 volans: maybe I'm wrong though and there is manual ipvs cleanup involved
[11:23:03] <wikibugs>	 (03PS26) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[11:24:20] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: More traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493202 (owner: 10Marostegui)
[11:24:40] <wikibugs>	 10Operations: Integrate Stretch 9.8 point update - https://phabricator.wikimedia.org/T216384 (10jbond)
[11:25:10] <wikibugs>	 (03CR) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[11:25:21] <wikibugs>	 (03PS7) 10KartikMistry: Enable edittag for ExternalGuidance in CX and VE [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493155 (https://phabricator.wikimedia.org/T216123)
[11:25:25] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Give more traffic to db1077 after MySQL upgrade (duration: 00m 54s)
[11:25:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:28:36] <godog>	 !log cleanup log4j from lvs eqiad / ipvsadm -D -t logstash.svc.eqiad.wmnet:4560
[11:28:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:28:56] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1016 is OK: OK: no difference between hosts in IPVS/PyBal
[11:30:31] <wikibugs>	 10Operations: Integrate Stretch 9.8 point update - https://phabricator.wikimedia.org/T216384 (10jbond)
[11:30:58] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1006 is OK: OK: no difference between hosts in IPVS/PyBal
[11:32:44] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: More traffic to db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493202 (owner: 10Marostegui)
[11:34:15] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Fully repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493203
[11:35:03] <wikibugs>	 (03PS1) 10Volans: Drop support for Python 3.4 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204
[11:35:05] <wikibugs>	 (03PS1) 10Volans: tests: refactor tox environments [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205
[11:35:07] <wikibugs>	 (03PS1) 10Volans: Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206
[11:36:27] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Fully repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493203 (owner: 10Marostegui)
[11:37:13] <wikibugs>	 (03CR) 10Muehlenhoff: Add password reset function to ipmi module (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:37:16] <wikibugs>	 (03PS3) 10Jcrespo: mariadb: Remove tls configuration from the client [puppet] - 10https://gerrit.wikimedia.org/r/492309
[11:37:18] <wikibugs>	 (03PS1) 10Jcrespo: mariadb-backups: Add safeguard against having null on a finished backup [puppet] - 10https://gerrit.wikimedia.org/r/493208
[11:37:32] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Fully repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493203 (owner: 10Marostegui)
[11:38:43] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Fully repool db1077 after MySQL upgrade (duration: 00m 53s)
[11:38:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:39:13] <wikibugs>	 (03CR) 10Marostegui: [C: 03+1] "> I am not going to merge this on a friday, but do you want me to do" [puppet] - 10https://gerrit.wikimedia.org/r/492309 (owner: 10Jcrespo)
[11:41:13] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs - https://phabricator.wikimedia.org/T213899 (10fgiunchedi)
[11:43:48] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Fully repool db1077 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493203 (owner: 10Marostegui)
[11:44:36] <wikibugs>	 (03CR) 10Marostegui: [C: 03+1] "I have tested the query on db1115 just to be safe" [puppet] - 10https://gerrit.wikimedia.org/r/493208 (owner: 10Jcrespo)
[11:45:27] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Pull every minute from git the charts [puppet] - 10https://gerrit.wikimedia.org/r/493113 (owner: 10Alexandros Kosiaris)
[11:46:48] <wikibugs>	 (03PS13) 10Jcrespo: mariadb: Add the option of postprocessing backups [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292)
[11:47:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb: Add the option of postprocessing backups [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[11:47:16] <wikibugs>	 (03PS2) 10Jcrespo: mariadb-backups: Add safeguard against having null on a finished backup [puppet] - 10https://gerrit.wikimedia.org/r/493208
[11:47:41] <wikibugs>	 (03CR) 10Jcrespo: [V: 03+2 C: 03+2] mariadb-backups: Add safeguard against having null on a finished backup [puppet] - 10https://gerrit.wikimedia.org/r/493208 (owner: 10Jcrespo)
[11:49:36] <jynus>	 so metadata on backups failed (but backups were successful), you will see a bunch of backup errors here
[11:49:52] <jynus>	 (don't be alarmed)
[11:50:24] <wikibugs>	 (03PS27) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[11:50:35] <wikibugs>	 (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492447 (https://phabricator.wikimedia.org/T214765) (owner: 10Sau226)
[11:51:14] <icinga-wm>	 PROBLEM - Backup of s3 in eqiad on db1115 is CRITICAL: Backup for s3 at eqiad (10 KB) is less than 2019-02-26 18:28:39: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of m1 in codfw on db1115 is CRITICAL: Backup for m1 at codfw (10 KB) is less than 2019-02-27 03:19:19: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of m2 in codfw on db1115 is CRITICAL: Backup for m2 at codfw (10 KB) is less than 2019-02-27 03:52:35: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of m2 in eqiad on db1115 is CRITICAL: Backup for m2 at eqiad (10 KB) is less than 2019-02-27 01:41:56: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of s2 in eqiad on db1115 is CRITICAL: Backup for s2 at eqiad (10 KB) is less than 2019-02-27 01:57:30: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of m3 in eqiad on db1115 is CRITICAL: Backup for m3 at eqiad (10 KB) is less than 2019-02-27 00:28:57: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of s1 in eqiad on db1115 is CRITICAL: Backup for s1 at eqiad (10 KB) is less than 2019-02-26 20:49:24: 0 bytes
[11:52:00] <icinga-wm>	 PROBLEM - Backup of m1 in eqiad on db1115 is CRITICAL: Backup for m1 at eqiad (10 KB) is less than 2019-02-26 22:40:06: 0 bytes
[11:52:01] <icinga-wm>	 PROBLEM - Backup of m3 in codfw on db1115 is CRITICAL: Backup for m3 at codfw (10 KB) is less than 2019-02-27 09:40:29: 0 bytes
[11:52:01] <icinga-wm>	 PROBLEM - Backup of s8 in eqiad on db1115 is CRITICAL: Backup for s8 at eqiad (10 KB) is less than 2019-02-26 21:02:12: 0 bytes
[11:52:02] <icinga-wm>	 PROBLEM - Backup of x1 in codfw on db1115 is CRITICAL: Backup for x1 at codfw (10 KB) is less than 2019-02-26 17:00:02: 0 bytes
[11:52:02] <icinga-wm>	 PROBLEM - Backup of m5 in eqiad on db1115 is CRITICAL: Backup for m5 at eqiad (10 KB) is less than 2019-02-27 03:19:49: 0 bytes
[11:52:03] <icinga-wm>	 PROBLEM - Backup of s7 in eqiad on db1115 is CRITICAL: Backup for s7 at eqiad (10 KB) is less than 2019-02-26 18:42:50: 0 bytes
[11:52:03] <icinga-wm>	 PROBLEM - Backup of m5 in codfw on db1115 is CRITICAL: Backup for m5 at codfw (10 KB) is less than 2019-02-27 09:23:41: 0 bytes
[11:52:04] <icinga-wm>	 PROBLEM - Backup of s4 in eqiad on db1115 is CRITICAL: Backup for s4 at eqiad (10 KB) is less than 2019-02-26 23:07:33: 0 bytes
[11:52:04] <icinga-wm>	 PROBLEM - Backup of s3 in codfw on db1115 is CRITICAL: Backup for s3 at codfw (10 KB) is less than 2019-02-26 17:00:02: 0 bytes
[11:52:05] <icinga-wm>	 PROBLEM - Backup of s5 in eqiad on db1115 is CRITICAL: Backup for s5 at eqiad (10 KB) is less than 2019-02-26 17:00:01: 0 bytes
[11:52:05] <icinga-wm>	 PROBLEM - Backup of s6 in eqiad on db1115 is CRITICAL: Backup for s6 at eqiad (10 KB) is less than 2019-02-26 17:00:01: 0 bytes
[11:52:06] <icinga-wm>	 PROBLEM - Backup of s7 in codfw on db1115 is CRITICAL: Backup for s7 at codfw (10 KB) is less than 2019-02-26 19:06:45: 0 bytes
[11:52:06] <icinga-wm>	 PROBLEM - Backup of s6 in codfw on db1115 is CRITICAL: Backup for s6 at codfw (10 KB) is less than 2019-02-26 20:39:00: 0 bytes
[11:52:07] <icinga-wm>	 PROBLEM - Backup of s5 in codfw on db1115 is CRITICAL: Backup for s5 at codfw (10 KB) is less than 2019-02-27 00:36:12: 0 bytes
[11:52:07] <icinga-wm>	 PROBLEM - Backup of s1 in codfw on db1115 is CRITICAL: Backup for s1 at codfw (10 KB) is less than 2019-02-27 03:20:30: 0 bytes
[11:52:08] <icinga-wm>	 PROBLEM - Backup of x1 in eqiad on db1115 is CRITICAL: Backup for x1 at eqiad (10 KB) is less than 2019-02-27 01:08:03: 0 bytes
[11:52:44] <jynus>	 I will disable alerting for some hours
[11:52:48] <jynus>	 on backups
[11:52:56] <marostegui>	 ^ expected as: jynus 12:49> so metadata on backups failed (but backups were successful), you will see a bunch of backup errors here jynus 12:49> (don't be alarmed)
[11:53:01] <marostegui>	 :)
[11:54:19] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10BBlack) Circa 2019-02-21, eqsin was depooled to install a new router, and most of the users normally mapped to eqsin had fallen back to ulsfo temporarily, which would dist...
[11:54:42] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[11:57:09] <wikibugs>	 (03PS28) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[12:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: I, the Bot under the Fountain, allow thee, The Deployer, to do European Mid-day SWAT(Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1200).
[12:00:04] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[12:00:14] <zeljkof>	 no patches no swat
[12:00:17] <zeljkof>	 problem solved
[12:00:57] <wikibugs>	 (03CR) 10Jbond: "all comments addressed ready for review" (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[12:04:02] <jbond42>	 !log rolling updates of apache on mw jobrunners
[12:04:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:08:10] <jbond42>	 !log correction: rolling updates of apache on mw api servers *not* jobrunners
[12:08:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:23:50] <wikibugs>	 (03PS1) 10Jcrespo: mariadb-backups: Fix parameter order for error message on check [puppet] - 10https://gerrit.wikimedia.org/r/493215
[12:24:22] <wikibugs>	 (03CR) 10Alexandros Kosiaris: Improve logging of errors, remove spurious print statements (031 comment) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492647 (owner: 10Giuseppe Lavagetto)
[12:28:26] <icinga-wm>	 PROBLEM - HHVM rendering on mw2203 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:29:24] <icinga-wm>	 RECOVERY - HHVM rendering on mw2203 is OK: HTTP OK: HTTP/1.1 200 OK - 74554 bytes in 0.234 second response time
[12:29:35] <wikibugs>	 (03PS2) 10Jcrespo: mariadb-backups: Fix parameter order for error message on check [puppet] - 10https://gerrit.wikimedia.org/r/493215
[12:30:45] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] mariadb-backups: Fix parameter order for error message on check [puppet] - 10https://gerrit.wikimedia.org/r/493215 (owner: 10Jcrespo)
[12:32:02] <wikibugs>	 (03CR) 10Jcrespo: [V: 03+2 C: 03+2] mariadb: Add the option of postprocessing backups [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[12:32:54] <wikibugs>	 (03PS5) 10Jcrespo: WMFMariaDB refactoring and adding tests [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/449185
[12:33:32] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WMFMariaDB refactoring and adding tests [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/449185 (owner: 10Jcrespo)
[12:33:34] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] cli.defaults was altered by read_config() [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487848 (owner: 10Hashar)
[12:34:08] <wikibugs>	 (03CR) 10jenkins-bot: cli.defaults was altered by read_config() [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487848 (owner: 10Hashar)
[12:40:40] <icinga-wm>	 PROBLEM - HHVM rendering on mw2141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:41:01] <wikibugs>	 (03PS4) 10Jcrespo: mariadb: Remove tls configuration from the client [puppet] - 10https://gerrit.wikimedia.org/r/492309
[12:41:03] <wikibugs>	 (03PS1) 10Jcrespo: mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292)
[12:41:44] <icinga-wm>	 RECOVERY - HHVM rendering on mw2141 is OK: HTTP OK: HTTP/1.1 200 OK - 74554 bytes in 0.268 second response time
[12:42:07] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[12:48:34] <wikibugs>	 (03CR) 10Jcrespo: "Volans, the -1 here is a CI mistake, you can check by running it on a python3-compatible pycodestyle. I believe it is running python2 or a" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[12:58:59] <wikibugs>	 (03CR) 10Volans: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[13:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1300)
[13:01:16] <wikibugs>	 (03PS1) 10Bmansurov: Labs: actually enable the demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493221 (https://phabricator.wikimedia.org/T217171)
[13:04:19] <wikibugs>	 (03CR) 10Jcrespo: "> > Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[13:13:27] <wikibugs>	 (03CR) 10DCausse: cirrus: fallback option to use localssl via acme subject (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:20:35] <wikibugs>	 (03CR) 10Volans: "Looks good! Just a couple of very minor details." (034 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[13:20:54] <wikibugs>	 (03PS1) 10Muehlenhoff: Update to 1.0.2r [debs/openssl] - 10https://gerrit.wikimedia.org/r/493224
[13:21:29] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Update to 1.0.2r [debs/openssl] - 10https://gerrit.wikimedia.org/r/493224 (owner: 10Muehlenhoff)
[13:24:26] <icinga-wm>	 PROBLEM - HHVM rendering on mw1225 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[13:25:32] <icinga-wm>	 RECOVERY - HHVM rendering on mw1225 is OK: HTTP OK: HTTP/1.1 200 OK - 74564 bytes in 0.355 second response time
[13:31:51] <wikibugs>	 (03PS11) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921)
[13:31:53] <wikibugs>	 (03PS8) 10Mathew.onipe: relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921)
[13:34:56] <wikibugs>	 (03CR) 10BBlack: k8s pod zone_validator cleanup (033 comments) [dns] - 10https://gerrit.wikimedia.org/r/493050 (owner: 10BBlack)
[13:35:35] <wikibugs>	 (03PS4) 10BBlack: k8s pod zone_validator cleanup [dns] - 10https://gerrit.wikimedia.org/r/493050
[13:35:37] <wikibugs>	 (03PS2) 10BBlack: [WIP] Mark Non-WMF IPs for zone_validator [dns] - 10https://gerrit.wikimedia.org/r/493101
[13:36:01] <wikibugs>	 (03CR) 10BBlack: [C: 03+2] Remove redundant ganeti comments check [dns] - 10https://gerrit.wikimedia.org/r/493095 (owner: 10BBlack)
[13:36:48] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM!" [dns] - 10https://gerrit.wikimedia.org/r/493050 (owner: 10BBlack)
[13:37:24] <wikibugs>	 (03CR) 10BBlack: [C: 03+2] k8s pod zone_validator cleanup [dns] - 10https://gerrit.wikimedia.org/r/493050 (owner: 10BBlack)
[13:37:34] <volans>	 bblack: if you're merging all thise meta-stuff on the dns repo, have a look at https://gerrit.wikimedia.org/r/c/operations/dns/+/492990 too ;)
[13:37:38] <icinga-wm>	 PROBLEM - HHVM rendering on mw1225 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.074 second response time
[13:37:46] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:38:39] <wikibugs>	 (03CR) 10Mathew.onipe: cirrus: fallback option to use localssl via acme subject (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:38:52] <icinga-wm>	 RECOVERY - HHVM rendering on mw1225 is OK: HTTP OK: HTTP/1.1 200 OK - 74564 bytes in 0.316 second response time
[13:38:56] <wikibugs>	 (03CR) 10Mathew.onipe: "still noop! https://puppet-compiler.wmflabs.org/compiler1002/14902/" [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:39:42] <wikibugs>	 (03CR) 10Mathew.onipe: "changes are expected: https://puppet-compiler.wmflabs.org/compiler1002/14903/" [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:40:17] <wikibugs>	 (03PS3) 10BBlack: tox: add Python 3.7 [dns] - 10https://gerrit.wikimedia.org/r/492990 (owner: 10Volans)
[13:40:55] <bblack>	 TIL: "trascurabile"
[13:41:18] <wikibugs>	 (03CR) 10BBlack: tox: add Python 3.7 (031 comment) [dns] - 10https://gerrit.wikimedia.org/r/492990 (owner: 10Volans)
[13:41:18] <volans>	 lol, from where?
[13:41:19] <volans>	 :D
[13:41:39] <wikibugs>	 (03CR) 10BBlack: [C: 03+2] tox: add Python 3.7 [dns] - 10https://gerrit.wikimedia.org/r/492990 (owner: 10Volans)
[13:41:41] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[13:41:51] <volans>	 ah my comment... ofc
[13:42:10] * volans doesn't know how to english ;)
[13:43:48] <Zppix>	 jouncebot:  now
[13:43:49] <jouncebot>	 For the next 0 hour(s) and 16 minute(s): Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1300)
[13:45:42] <wikibugs>	 (03PS2) 10Volans: icinga: Make use of the strong TLS ciphersuites configuration [puppet] - 10https://gerrit.wikimedia.org/r/492656 (https://phabricator.wikimedia.org/T217002) (owner: 10Vgutierrez)
[13:47:37] <wikibugs>	 (03CR) 10Volans: [C: 03+2] icinga: Make use of the strong TLS ciphersuites configuration [puppet] - 10https://gerrit.wikimedia.org/r/492656 (https://phabricator.wikimedia.org/T217002) (owner: 10Vgutierrez)
[13:47:40] <bblack>	 volans: feature request: flag to turn off ansi :)
[13:48:03] <volans>	 bblack: sure :) where does it break?
[13:48:15] <volans>	 in CI I thin is useful, isn't it?
[13:50:02] <bblack>	 yeah it is
[13:50:26] <volans>	 ah we run it in prod when doing authdns update I guess
[13:50:28] <bblack>	 my use-case is running it with "-w >foo" and trying to parse the output with other CLI tooling to look for easy cases to fix
[13:50:29] <volans>	 and there is better to avoid
[13:50:38] <bblack>	 the file is full of ansi escapes
[13:50:42] <volans>	 ahhhh
[13:50:44] <volans>	 got it
[13:51:00] <bblack>	 although stackexchange offered this helpful bit of perl
[13:51:12] <bblack>	 utils/zone_validator.py -z /tmp/z -w | perl -pe 's/\e\[?.*?[\@-~]//g' >foo
[13:51:17] <bblack>	 :)
[13:51:29] <volans>	 ahahah
[13:53:15] <wikibugs>	 (03PS14) 10Eevans: Initial configuration for session storage service [puppet] - 10https://gerrit.wikimedia.org/r/487885 (https://phabricator.wikimedia.org/T215883)
[13:54:06] <vgutierrez>	 thanks for merging that volans
[13:54:36] <volans>	 vgutierrez: thank you for the patch, although some external things are failing, we're debugging with cdanis ;)
[13:57:45] <vgutierrez>	 really? we had stuff not being able to talk TLSv1.2?
[13:57:47] <vgutierrez>	 :_(
[13:58:27] <Reedy>	 we? as in internally?
[13:58:35] <Reedy>	 Or wikipedia viewers?
[13:58:38] <bblack>	 we should probably push pretty hard on that, we're well past the point in history for non-human clients to talk tlsv1.2 :)
[13:58:56] <bblack>	 Reedy: this is about tools that hit sre meta-services like icinga
[13:58:57] <vgutierrez>	 Reedy: stuff related to our icinga
[13:59:23] <Zppix>	 Isn't anything before TLS1.2 depreciated, and not recommended for use?
[13:59:42] <Reedy>	 Deprecated doesn't mean it's not still in wide use
[13:59:50] <vgutierrez>	 hmm right now TLSv1.0 is considered insecure by PCI-DSS
[13:59:57] <vgutierrez>	 but we still have our fair amount of traffic using it
[14:00:04] <jouncebot>	 hashar: My dear minions, it's time we take the moon! Just kidding. Time for MediaWiki train - European version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1400).
[14:00:19] <moritzm>	 !log uploaded openssl 1.0.2r to jessie-wikimedia
[14:00:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:01:02] <volans>	 vgutierrez: yeah it's pretty depressing, but the good part is that we're replacing that check, so no worries
[14:01:07] <marostegui>	 !log Change indexes on abuse_filter_log on db1089 - T187295
[14:01:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:01:12] <stashbot>	 T187295: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295
[14:03:00] <bblack>	 even for the primary production service hostnames (e.g. wikipedia), TLS<1.2 usage is down to ~1%
[14:03:32] <bblack>	 we'll probably in the coming quarters start pushing harder on standards again (trying to kill tlsv1.[01], CBC ciphers, etc)
[14:13:06] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 04-2] "I 've left comments on the task already, but TL;DR is that kubernetes does garbage collection on its own anyway." [puppet] - 10https://gerrit.wikimedia.org/r/493085 (https://phabricator.wikimedia.org/T217159) (owner: 10GTirloni)
[14:14:19] <wikibugs>	 (03CR) 10Filippo Giunchedi: "> Patch Set 2:" [puppet] - 10https://gerrit.wikimedia.org/r/493102 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[14:14:49] <wikibugs>	 (03PS1) 10MSantos: Set right owner to he admin grants sql [puppet] - 10https://gerrit.wikimedia.org/r/493231
[14:15:56] <wikibugs>	 (03CR) 10CDanis: [C: 03+1] icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[14:16:53] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] Set right owner to he admin grants sql [puppet] - 10https://gerrit.wikimedia.org/r/493231 (owner: 10MSantos)
[14:19:34] <wikibugs>	 (03PS1) 10Filippo Giunchedi: deployment_server: ship logs through logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/493232 (https://phabricator.wikimedia.org/T86969)
[14:21:19] <wikibugs>	 (03PS2) 10MSantos: Set right owner for admin grants sql [puppet] - 10https://gerrit.wikimedia.org/r/493231
[14:22:13] <wikibugs>	 (03CR) 10Filippo Giunchedi: "This is effectively a noop now but will DTRT once scap logs to syslog" [puppet] - 10https://gerrit.wikimedia.org/r/493232 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi)
[14:25:59] <marostegui>	 !log Deploy schema change on abuse_filter_log on s7 codfw - lag will be generated on codfw - T187295
[14:26:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:26:03] <stashbot>	 T187295: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295
[14:26:10] <wikibugs>	 (03PS29) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[14:26:18] <wikibugs>	 (03Abandoned) 10GTirloni: docker: Run docker-gc daily [puppet] - 10https://gerrit.wikimedia.org/r/493085 (https://phabricator.wikimedia.org/T217159) (owner: 10GTirloni)
[14:27:51] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] hadoop analytics: set analytics1071 rack config to A5 [puppet] - 10https://gerrit.wikimedia.org/r/493169 (https://phabricator.wikimedia.org/T212348) (owner: 10Elukey)
[14:31:09] <wikibugs>	 (03PS31) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[14:32:13] <wikibugs>	 (03CR) 10Jbond: "all comments addressed, ready for review" (034 comments) [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[14:33:04] <wikibugs>	 (03CR) 10Vgutierrez: [C: 03+1] cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[14:33:14] <wikibugs>	 (03PS2) 10Jcrespo: mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292)
[14:34:14] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM! Thanks a lot for the patch." [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[14:34:16] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[14:34:24] <icinga-wm>	 PROBLEM - Check systemd state on cloudvirt1018 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[14:34:27] <icinga-wm>	 PROBLEM - nova-compute proc maximum on cloudvirt1018 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[14:34:52] <vgutierrez>	 ^^ that's expected?
[14:35:03] <icinga-wm>	 PROBLEM - nova-compute proc minimum on cloudvirt1018 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[14:35:19] <icinga-wm>	 PROBLEM - ensure kvm processes are running on cloudvirt1018 is CRITICAL: PROCS CRITICAL: 0 processes with regex args qemu-system-x86_64
[14:35:30] <Zppix>	 I dont think so (but i dont know)
[14:36:19] <andrewbogott>	 vgutierrez: I don't know, either it crashed or Chris is working on it
[14:36:25] <andrewbogott>	 I marked it in icinga for now
[14:36:35] <vgutierrez>	 thx for taking care of it <3
[14:37:46] <wikibugs>	 (03PS3) 10Jcrespo: mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292)
[14:38:17] <wikibugs>	 (03PS32) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026
[14:41:32] <wikibugs>	 (03PS5) 10Jcrespo: mariadb: Remove tls configuration from the client [puppet] - 10https://gerrit.wikimedia.org/r/492309
[14:42:22] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] mariadb: Remove tls configuration from the client [puppet] - 10https://gerrit.wikimedia.org/r/492309 (owner: 10Jcrespo)
[14:42:34] <wikibugs>	 (03PS1) 10Jbond: Add python3.7 support to test framework [software/spicerack] - 10https://gerrit.wikimedia.org/r/493233
[14:42:50] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[14:43:16] <wikibugs>	 (03CR) 10Volans: "We already have Ie6457c33b98c5fc0ee38aaca76a5b441496c687a pending :)" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493233 (owner: 10Jbond)
[14:44:57] <wikibugs>	 (03CR) 10Marostegui: "Is this the same change I already reviewed in a different patchset?" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[14:46:08] <wikibugs>	 (03CR) 10Jbond: "> Patch Set 1:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493233 (owner: 10Jbond)
[14:46:32] <wikibugs>	 (03Abandoned) 10Jbond: Add python3.7 support to test framework [software/spicerack] - 10https://gerrit.wikimedia.org/r/493233 (owner: 10Jbond)
[14:47:07] <wikibugs>	 (03PS1) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[14:47:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[14:48:20] <wikibugs>	 (03PS1) 10Bmansurov: Enable reader demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493235 (https://phabricator.wikimedia.org/T217080)
[14:48:40] <wikibugs>	 (03Merged) 10jenkins-bot: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[14:49:10] <wikibugs>	 (03PS1) 10Bmansurov: Disable reader demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493236 (https://phabricator.wikimedia.org/T217080)
[14:49:46] <wikibugs>	 (03CR) 10jenkins-bot: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond)
[14:53:19] <wikibugs>	 (03CR) 10CDanis: [C: 03+1] icinga: add htpasswd users and salt [labs/private] - 10https://gerrit.wikimedia.org/r/493170 (owner: 10Volans)
[14:54:20] <wikibugs>	 (03CR) 10Volans: [V: 03+2 C: 03+2] icinga: add htpasswd users and salt [labs/private] - 10https://gerrit.wikimedia.org/r/493170 (owner: 10Volans)
[14:56:13] <wikibugs>	 (03PS2) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[14:56:47] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[14:57:38] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "not overly failure with tox but LGTM" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204 (owner: 10Volans)
[14:57:57] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "not overly failure with tox but LGTM" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205 (owner: 10Volans)
[14:58:32] <wikibugs>	 (03CR) 10Jbond: "do you want to add the following to setup.py?" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:00:40] <wikibugs>	 (03PS3) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[15:01:15] <wikibugs>	 (03PS2) 10Volans: Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206
[15:01:31] <wikibugs>	 (03CR) 10Jcrespo: "> Is this the same change I already reviewed in a different patchset?" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[15:01:41] <wikibugs>	 (03CR) 10Volans: "> Patch Set 1:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:04:08] <wikibugs>	 (03PS2) 10Volans: Drop support for Python 3.4 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204
[15:04:56] <wikibugs>	 (03CR) 10Volans: "Compiler results seems sane, see here:" [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[15:05:48] <wikibugs>	 (03CR) 10Volans: [C: 03+2] Drop support for Python 3.4 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204 (owner: 10Volans)
[15:06:30] <wikibugs>	 (03PS5) 10Jbond: ulogd puppet profile. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[15:07:58] <wikibugs>	 (03PS5) 10Alexandros Kosiaris: [WIP] Set up eventgate-analytics.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/491861 (https://phabricator.wikimedia.org/T211247) (owner: 10Ottomata)
[15:08:19] <wikibugs>	 (03PS3) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[15:09:08] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:11:46] <wikibugs>	 (03Merged) 10jenkins-bot: Drop support for Python 3.4 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204 (owner: 10Volans)
[15:12:07] <wikibugs>	 (03PS2) 10Volans: tests: refactor tox environments [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205
[15:12:41] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Close the grwp-wici mailing list - https://phabricator.wikimedia.org/T217247 (10Richard_Nevell_WMUK)
[15:12:47] <wikibugs>	 (03CR) 10jenkins-bot: Drop support for Python 3.4 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493204 (owner: 10Volans)
[15:13:25] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM" [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:15:07] <wikibugs>	 (03CR) 10Herron: [C: 03+1] deployment_server: ship logs through logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/493232 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi)
[15:18:54] <wikibugs>	 (03PS5) 10Filippo Giunchedi: [WIP] move varnish logging to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/492676 (https://phabricator.wikimedia.org/T213899)
[15:18:56] <wikibugs>	 (03PS1) 10Filippo Giunchedi: WIP move webrequest-5xx to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/493243 (https://phabricator.wikimedia.org/T213899)
[15:23:14] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898 (10herron)
[15:24:39] <wikibugs>	 (03CR) 10Volans: [C: 03+2] tests: refactor tox environments [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205 (owner: 10Volans)
[15:25:57] <wikibugs>	 (03PS4) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[15:26:43] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:26:51] <wikibugs>	 (03PS1) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:27:29] <wikibugs>	 (03Merged) 10jenkins-bot: tests: refactor tox environments [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205 (owner: 10Volans)
[15:28:16] <wikibugs>	 (03CR) 10jenkins-bot: tests: refactor tox environments [software/spicerack] - 10https://gerrit.wikimedia.org/r/493205 (owner: 10Volans)
[15:28:31] <wikibugs>	 (03PS3) 10Volans: Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206
[15:31:21] <wikibugs>	 10Operations, 10Traffic, 10HTTPS: Make sure that services available for NDA-only users are using strong TLS ciphersuites - https://phabricator.wikimedia.org/T217002 (10Vgutierrez)
[15:32:03] <wikibugs>	 10Operations, 10Prod-Kubernetes, 10serviceops, 10Kubernetes, and 2 others: Make swift containers for docker registry cross replicated. - https://phabricator.wikimedia.org/T214289 (10fgiunchedi) >>! In T214289#4984959, @fsero wrote: > It seems there are some issues on the swift side regarding container-real...
[15:33:38] <wikibugs>	 (03CR) 10Marostegui: [C: 03+1] "+1 as I already reviewed the other patchset :)" [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[15:34:03] <wikibugs>	 (03PS5) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[15:34:19] <wikibugs>	 (03PS4) 10Jcrespo: mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292)
[15:34:21] <jbond42>	 !log rolling openssl security updates to jessie canary servers
[15:34:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:34:56] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:34:58] <wikibugs>	 (03CR) 10Volans: [C: 03+2] Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:35:11] <wikibugs>	 (03PS2) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:36:28] <wikibugs>	 (03PS3) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:37:01] <wikibugs>	 (03CR) 10Marostegui: [C: 03+1] mariadb: Remove tls configuration from the client (part 2) [puppet] - 10https://gerrit.wikimedia.org/r/493241 (owner: 10Jcrespo)
[15:37:13] <wikibugs>	 (03PS4) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:38:05] <wikibugs>	 (03PS5) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:38:22] <wikibugs>	 (03PS6) 10Krinkle: webperf: Rename /xenon to /arclamp on performance.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916)
[15:38:45] <wikibugs>	 (03PS6) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[15:39:03] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] "Two nits! the rest looks good to me :)" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[15:39:40] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:39:47] <wikibugs>	 (03Merged) 10jenkins-bot: Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:40:30] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] ulogd puppet profile. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[15:41:21] <wikibugs>	 (03PS4) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[15:41:43] <wikibugs>	 (03CR) 10Volans: "address comments, thanks for the review!" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[15:41:54] <wikibugs>	 (03CR) 10jenkins-bot: Add support for Python 3.7 [software/spicerack] - 10https://gerrit.wikimedia.org/r/493206 (owner: 10Volans)
[15:41:59] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] mariadb-backups: Update dump_section.py and recover_section.py [puppet] - 10https://gerrit.wikimedia.org/r/493218 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[15:42:21] <wikibugs>	 (03PS5) 10Muehlenhoff: Enable base::service_auto_restart for atftpd [puppet] - 10https://gerrit.wikimedia.org/r/424584 (https://phabricator.wikimedia.org/T135991)
[15:42:41] <wikibugs>	 10Operations, 10MediaWiki-Cache, 10Performance-Team, 10Patch-For-Review, 10User-Elukey: Consider removing the last traces of nutcracker in Mediawiki configs - https://phabricator.wikimedia.org/T214275 (10fgiunchedi) >>! In T214275#4987513, @elukey wrote: > nutcracker seems also used by thumbor when conta...
[15:42:58] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for atftpd [puppet] - 10https://gerrit.wikimedia.org/r/424584 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[15:43:20] <wikibugs>	 (03PS2) 10Jcrespo: mariadb: Remove tls configuration from the client (part 2) [puppet] - 10https://gerrit.wikimedia.org/r/493241
[15:44:04] <wikibugs>	 (03CR) 10Volans: "compiler seems still happy:" [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[15:45:22] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] ulogd puppet profile. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011) (owner: 10Jbond)
[15:45:54] <wikibugs>	 (03PS6) 10Jbond: ulogd puppet profile. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[15:46:04] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+2] mariadb: Remove tls configuration from the client (part 2) [puppet] - 10https://gerrit.wikimedia.org/r/493241 (owner: 10Jcrespo)
[15:49:29] <wikibugs>	 (03PS7) 10Jbond: ulogd puppet profile. [puppet] - 10https://gerrit.wikimedia.org/r/490836 (https://phabricator.wikimedia.org/T116011)
[15:50:14] <wikibugs>	 (03PS4) 10Giuseppe Lavagetto: Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793
[15:51:36] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793 (owner: 10Giuseppe Lavagetto)
[15:52:22] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] doc: minor tweaks [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/485626 (owner: 10Hashar)
[15:53:21] <wikibugs>	 (03CR) 10jenkins-bot: doc: minor tweaks [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/485626 (owner: 10Hashar)
[15:53:47] <wikibugs>	 (03CR) 10jenkins-bot: Warn about lack of changelog or Dockerfile.template [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/485880 (owner: 10Hashar)
[15:56:32] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] Never try apt_pkg when parsing control [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487849 (https://phabricator.wikimedia.org/T216836) (owner: 10Hashar)
[15:56:47] <wikibugs>	 (03PS12) 10Volans: cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[15:57:11] <wikibugs>	 (03CR) 10jenkins-bot: Never try apt_pkg when parsing control [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487849 (https://phabricator.wikimedia.org/T216836) (owner: 10Hashar)
[15:57:49] <wikibugs>	 (03PS6) 10Jbond: Enable base::service_auto_restart for atftpd [puppet] - 10https://gerrit.wikimedia.org/r/424584 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[15:58:32] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Enable base::service_auto_restart for atftpd [puppet] - 10https://gerrit.wikimedia.org/r/424584 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[15:58:34] <wikibugs>	 (03CR) 10Volans: [C: 03+2] cirrus: fallback option to use localssl via acme subject [puppet] - 10https://gerrit.wikimedia.org/r/493048 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[15:59:01] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493083 (https://phabricator.wikimedia.org/T217068) (owner: 10Bstorm)
[15:59:14] <wikibugs>	 (03PS2) 10Bstorm: dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493083 (https://phabricator.wikimedia.org/T217068)
[15:59:36] <wikibugs>	 (03PS7) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[15:59:40] <wikibugs>	 (03PS7) 10Jbond: Enable base::service_auto_restart for atftpd [puppet] - 10https://gerrit.wikimedia.org/r/424584 (https://phabricator.wikimedia.org/T135991) (owner: 10Muehlenhoff)
[16:00:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[16:01:04] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar)
[16:01:09] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar)
[16:06:41] <wikibugs>	 (03PS9) 10Volans: relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[16:07:36] <wikibugs>	 (03CR) 10Volans: [C: 03+2] relforge: switch relforge to use localssl [puppet] - 10https://gerrit.wikimedia.org/r/493063 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[16:11:59] <wikibugs>	 (03PS8) 10DCausse: [WIP] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[16:16:20] <wikibugs>	 (03PS3) 10Bstorm: dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493083 (https://phabricator.wikimedia.org/T217068)
[16:17:04] <wikibugs>	 (03CR) 10Cwhite: [C: 03+1] nagios_common::commands: Remove support for trusty [puppet] - 10https://gerrit.wikimedia.org/r/491460 (owner: 10Muehlenhoff)
[16:17:22] <icinga-wm>	 PROBLEM - puppet last run on relforge1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[acme-setup-acme-relforge1001_eqiad_wmnet]
[16:17:40] <onimisionipe>	 ^ that's me
[16:18:44] <wikibugs>	 (03PS1) 10BBlack: fix wmnet typo for wmf7370 [dns] - 10https://gerrit.wikimedia.org/r/493264
[16:18:50] <chaomodus>	 Yes we do
[16:19:03] <wikibugs>	 (03PS4) 10Muehlenhoff: nagios_common::commands: Remove support for trusty [puppet] - 10https://gerrit.wikimedia.org/r/491460
[16:19:22] <wikibugs>	 (03CR) 10BBlack: [C: 03+2] fix wmnet typo for wmf7370 [dns] - 10https://gerrit.wikimedia.org/r/493264 (owner: 10BBlack)
[16:20:53] <wikibugs>	 (03PS1) 10Bstorm: Revert "dumpsdistribution: Make pages go off for disk space" [puppet] - 10https://gerrit.wikimedia.org/r/493266
[16:20:59] <volans>	 !log force-rebooting icinga1001 (to test some puppet changes) - T214760
[16:21:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:21:07] <stashbot>	 T214760: icinga1001 crashed - https://phabricator.wikimedia.org/T214760
[16:21:13] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] nagios_common::commands: Remove support for trusty [puppet] - 10https://gerrit.wikimedia.org/r/491460 (owner: 10Muehlenhoff)
[16:21:32] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] Revert "dumpsdistribution: Make pages go off for disk space" [puppet] - 10https://gerrit.wikimedia.org/r/493266 (owner: 10Bstorm)
[16:21:43] <wikibugs>	 (03PS2) 10Bstorm: Revert "dumpsdistribution: Make pages go off for disk space" [puppet] - 10https://gerrit.wikimedia.org/r/493266
[16:24:07] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson) We have an issue with these servers. I completely forgot about the lack of power availability in rack A2.   We will need to move these to a different row or wait...
[16:24:44] <wikibugs>	 (03PS9) 10Giuseppe Lavagetto: scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar)
[16:24:46] <wikibugs>	 (03PS5) 10Giuseppe Lavagetto: Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793
[16:26:16] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793 (owner: 10Giuseppe Lavagetto)
[16:26:24] <volans>	 !log temporarily disabled puppet on icinga[12]001 to deploy g/493171
[16:26:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:26:38] <wikibugs>	 (03PS5) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[16:27:21] <chaomodus>	 echan
[16:30:19] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] "Affects webperfx001 (webperf site) and x002 (webperf arclamp)." [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916) (owner: 10Krinkle)
[16:30:46] <wikibugs>	 (03CR) 10Krinkle: [C: 03+1] "http://performance-beta.wmflabs.org/xenon/  redirects to http://performance-beta.wmflabs.org/arclamp/." [puppet] - 10https://gerrit.wikimedia.org/r/493247 (https://phabricator.wikimedia.org/T176916) (owner: 10Krinkle)
[16:31:29] <wikibugs>	 (03PS1) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[16:31:47] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708) (owner: 10Cwhite)
[16:32:07] <wikibugs>	 (03PS1) 10Volans: Revert "relforge: switch relforge to use localssl" [puppet] - 10https://gerrit.wikimedia.org/r/493270
[16:32:32] <wikibugs>	 (03PS2) 10Volans: Revert "relforge: switch relforge to use localssl" [puppet] - 10https://gerrit.wikimedia.org/r/493270
[16:32:55] <wikibugs>	 (03PS1) 10Bstorm: dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493271 (https://phabricator.wikimedia.org/T217068)
[16:33:02] <wikibugs>	 (03PS2) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[16:33:17] <wikibugs>	 (03CR) 10Volans: [C: 03+2] Revert "relforge: switch relforge to use localssl" [puppet] - 10https://gerrit.wikimedia.org/r/493270 (owner: 10Volans)
[16:33:23] <wikibugs>	 (03CR) 10Mathew.onipe: [C: 03+1] Revert "relforge: switch relforge to use localssl" [puppet] - 10https://gerrit.wikimedia.org/r/493270 (owner: 10Volans)
[16:33:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708) (owner: 10Cwhite)
[16:33:33] <wikibugs>	 (03PS3) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[16:33:50] <wikibugs>	 (03PS4) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[16:33:52] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708) (owner: 10Cwhite)
[16:34:14] <wikibugs>	 (03PS2) 10Bstorm: dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493271 (https://phabricator.wikimedia.org/T217068)
[16:35:14] <wikibugs>	 (03PS6) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[16:35:24] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] dumpsdistribution: Make pages go off for disk space [puppet] - 10https://gerrit.wikimedia.org/r/493271 (https://phabricator.wikimedia.org/T217068) (owner: 10Bstorm)
[16:35:41] * volans takes the number for the puppet-merge queue
[16:37:41] <wikibugs>	 (03PS7) 10Volans: icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171
[16:38:16] <icinga-wm>	 RECOVERY - puppet last run on relforge1001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[16:38:48] <wikibugs>	 (03CR) 10Volans: [C: 03+2] icinga: allow meta-monitoring bot user [puppet] - 10https://gerrit.wikimedia.org/r/493171 (owner: 10Volans)
[16:39:11] <wikibugs>	 (03CR) 10Giuseppe Lavagetto: [C: 03+2] scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar)
[16:39:15] <wikibugs>	 (03PS6) 10Alexandros Kosiaris: Set up eventgate-analytics.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/491861 (https://phabricator.wikimedia.org/T211247) (owner: 10Ottomata)
[16:39:45] <wikibugs>	 (03PS7) 10Alexandros Kosiaris: Set up eventgate-analytics.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/491861 (https://phabricator.wikimedia.org/T211247) (owner: 10Ottomata)
[16:39:57] <wikibugs>	 (03CR) 10jenkins-bot: scan and process templates in parallel [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/484578 (owner: 10Hashar)
[16:43:37] <wikibugs>	 (03CR) 10CRusnov: "> Patch Set 3:" (033 comments) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[16:44:07] <wikibugs>	 (03CR) 10Effie Mouzeli: [C: 03+2] Set up eventgate-analytics.discovery.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/491861 (https://phabricator.wikimedia.org/T211247) (owner: 10Ottomata)
[16:49:34] <jijiki>	 !log Deploy LVS for eventgate-analytics - T211247 
[16:49:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:49:37] <stashbot>	 T211247: Modern Event Platform: Stream Intake Service: Implementation: Deployment Pipeline - https://phabricator.wikimedia.org/T211247
[16:51:28] <wikibugs>	 (03PS1) 10RobH: adding Toni Sevener to wmf ldap group [puppet] - 10https://gerrit.wikimedia.org/r/493276 (https://phabricator.wikimedia.org/T216759)
[16:52:40] <wikibugs>	 (03CR) 10RobH: [C: 03+2] adding Toni Sevener to wmf ldap group [puppet] - 10https://gerrit.wikimedia.org/r/493276 (https://phabricator.wikimedia.org/T216759) (owner: 10RobH)
[16:52:49] <wikibugs>	 (03PS2) 10RobH: adding Toni Sevener to wmf ldap group [puppet] - 10https://gerrit.wikimedia.org/r/493276 (https://phabricator.wikimedia.org/T216759)
[16:53:08] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs1006 is CRITICAL: CRITICAL: 45 connections established with conf1004.eqiad.wmnet:4001 (min=46)
[16:53:14] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs2003 is CRITICAL: CRITICAL: 36 connections established with conf2001.codfw.wmnet:2379 (min=37)
[16:53:33] <akosiaris>	 expected ^
[16:53:50] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs2006 is CRITICAL: CRITICAL: 36 connections established with conf2001.codfw.wmnet:2379 (min=37)
[16:53:58] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1016 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.2.42:31192])
[16:55:36] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1006 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.2.42:31192])
[16:55:52] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs2006 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.1.42:31192])
[16:57:16] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs2003 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.1.42:31192])
[16:57:16] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs1016 is CRITICAL: CRITICAL: 45 connections established with conf1004.eqiad.wmnet:4001 (min=46)
[16:58:09] <hashar>	 deploying a trivial hotfix for Score https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Score/+/493267/
[16:58:22] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs1006 is OK: OK: 46 connections established with conf1004.eqiad.wmnet:4001 (min=46)
[16:58:36] <logmsgbot>	 !log hashar@deploy1001 Synchronized php-1.33.0-wmf.19/extensions/Score: Revert "beautify lilypond error message output" - T217241 (duration: 00m 56s)
[16:58:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:58:40] <stashbot>	 T217241: PHP Notice: Use of undefined constant PREG_UNMATCHED_AS_NULL - https://phabricator.wikimedia.org/T217241
[17:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: That opportune time is upon us again. Time for a Morning SWAT (Max 6 patches) deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1700).
[17:00:04] <jouncebot>	 bmansurov, stephanebisson, bmansurov, and kostajh: A patch you scheduled for Morning SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[17:00:06] <bmansurov>	 here
[17:00:14] <stephanebisson>	 hey
[17:00:50] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1006 is OK: OK: no difference between hosts in IPVS/PyBal
[17:02:45] <stephanebisson>	 Anyone available to run the SWAT? I would but I'm in a meeting so I'd rather not.
[17:04:17] <kostajh>	 here
[17:05:08] <wikibugs>	 (03PS6) 10Giuseppe Lavagetto: Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793
[17:05:27] <RoanKattouw>	 !log Running foreachwikiindblist dblists/echo.dblist extensions/Echo/maintenance/removeOrphanedEvents.php  on mwmaint1002
[17:05:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:05:35] <bmansurov>	 anyone can deploy?
[17:06:22] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793 (owner: 10Giuseppe Lavagetto)
[17:06:27] <cmjohnson1>	 !log powering off wtp1029 to move to different rack A5  T212348
[17:06:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:06:30] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[17:08:13] <bmansurov>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, zeljkof: can someone deploy patches?
[17:08:52] <Niharika>	 bmansurov: I'm in a meeting right but can do in 10-ish minutes if nobody else gets to it first. 
[17:08:59] <bmansurov>	 Niharika: ok thanks
[17:09:58] <icinga-wm>	 PROBLEM - Host wtp1028 is DOWN: PING CRITICAL - Packet loss = 100%
[17:11:19] <wikibugs>	 (03PS7) 10Giuseppe Lavagetto: Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793
[17:11:30] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs2006 is OK: OK: no difference between hosts in IPVS/PyBal
[17:12:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add an update action [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793 (owner: 10Giuseppe Lavagetto)
[17:12:56] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs2003 is OK: OK: no difference between hosts in IPVS/PyBal
[17:14:00] <icinga-wm>	 RECOVERY - Host wtp1028 is UP: PING OK - Packet loss = 0%, RTA = 36.39 ms
[17:14:10] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs2003 is OK: OK: 37 connections established with conf2001.codfw.wmnet:2379 (min=37)
[17:14:30] <cmjohnson1>	 !log powering off wtp1029 to move to different rack A5  T212348
[17:14:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:14:33] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[17:14:44] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs2006 is OK: OK: 37 connections established with conf2001.codfw.wmnet:2379 (min=37)
[17:14:52] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1016 is OK: OK: no difference between hosts in IPVS/PyBal
[17:17:10] <icinga-wm>	 PROBLEM - Host wtp1029 is DOWN: PING CRITICAL - Packet loss = 100%
[17:18:25] <icinga-wm>	 PROBLEM - Host wtp1029.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[17:19:19] <elukey>	 cmjohnson1: o/ are you moving the host today or tomorrow? I might have the wrong date :(
[17:19:27] <cmjohnson1>	 !log powering off wtp1030 to move to different rack A5  T212348
[17:19:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:19:47] <elukey>	 (if it is today I'll shutdown analytics1071 asap)
[17:19:50] <cmjohnson1>	 @elukey...tomorrow is the plan but the wtp servers didn't impact anyting
[17:19:57] <elukey>	 ah okok!
[17:20:01] <cmjohnson1>	 but if you want to go ahead and shut it down I will move it now since I'm doing them
[17:20:05] <cmjohnson1>	 elukey ^
[17:20:07] <cmjohnson1>	 i'm good
[17:20:20] <elukey>	 cmjohnson1: I'd need 10 minutes though, good for you?
[17:20:28] <cmjohnson1>	 yes, ping me when ready
[17:20:29] <icinga-wm>	 RECOVERY - Host wtp1029 is UP: PING OK - Packet loss = 0%, RTA = 36.83 ms
[17:20:34] <elukey>	 ack
[17:21:29] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs1016 is OK: OK: 46 connections established with conf1004.eqiad.wmnet:4001 (min=46)
[17:21:57] <icinga-wm>	 PROBLEM - Host wtp1030 is DOWN: PING CRITICAL - Packet loss = 100%
[17:22:05] <Niharika>	 bmansurov: I can SWAT now. 
[17:22:17] <bmansurov>	 Niharika: great! thanks
[17:22:29] <wikibugs>	 (03CR) 10Niharika29: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492345 (https://phabricator.wikimedia.org/T213969) (owner: 10Bmansurov)
[17:22:31] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Enable eventgate-analytics.discovery.wmnet [dns] - 10https://gerrit.wikimedia.org/r/493279 (https://phabricator.wikimedia.org/T211247)
[17:22:54] <elukey>	 !log drain + shutdown of analytics1071 to allow its move to A5 - T212348
[17:22:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:22:58] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[17:23:18] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Enable eventgate-analytics.discovery.wmnet [dns] - 10https://gerrit.wikimedia.org/r/493279 (https://phabricator.wikimedia.org/T211247) (owner: 10Alexandros Kosiaris)
[17:23:19] <icinga-wm>	 PROBLEM - Host wtp1030.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[17:23:31] <icinga-wm>	 RECOVERY - Host wtp1029.mgmt is UP: PING OK - Packet loss = 0%, RTA = 37.04 ms
[17:25:42] <wikibugs>	 (03PS2) 10Niharika29: Stop collecting data for CitaitonUsage and CitationUsagePageLoad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492345 (https://phabricator.wikimedia.org/T213969) (owner: 10Bmansurov)
[17:26:17] <icinga-wm>	 RECOVERY - Host wtp1030.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.73 ms
[17:26:43] <icinga-wm>	 RECOVERY - Host wtp1030 is UP: PING OK - Packet loss = 0%, RTA = 36.17 ms
[17:27:35] <wikibugs>	 (03CR) 10Niharika29: [C: 03+2] Stop collecting data for CitaitonUsage and CitationUsagePageLoad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492345 (https://phabricator.wikimedia.org/T213969) (owner: 10Bmansurov)
[17:28:35] <wikibugs>	 (03Merged) 10jenkins-bot: Stop collecting data for CitaitonUsage and CitationUsagePageLoad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492345 (https://phabricator.wikimedia.org/T213969) (owner: 10Bmansurov)
[17:31:05] <Niharika>	 bmansurov: Can you test it?
[17:31:09] <Niharika>	 It's on mwdebug1002. 
[17:31:27] <bmansurov>	 Niharika: on it
[17:31:47] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson)
[17:32:11] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10Cmjohnson)
[17:32:41] <bmansurov>	 Niharika: it's working. Please deploy everywhere.
[17:33:36] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) >>! In T212348#4988345, @Cmjohnson wrote: > We have an issue with these servers. I completely forgot about the lack of power availability in rack A2.   We will need t...
[17:34:38] <logmsgbot>	 !log niharika29@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Stop collecting data for CitaitonUsage and CitationUsagePageLoad T213969 (duration: 00m 55s)
[17:34:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:34:41] <stashbot>	 T213969: Citation Usage: run third round of data collection - https://phabricator.wikimedia.org/T213969
[17:34:41] <wikibugs>	 (03PS2) 10Niharika29: Welcome survey: add a control group to viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491975 (https://phabricator.wikimedia.org/T216669) (owner: 10Sbisson)
[17:34:52] <wikibugs>	 (03CR) 10Niharika29: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491975 (https://phabricator.wikimedia.org/T216669) (owner: 10Sbisson)
[17:34:59] <Niharika>	 bmansurov: Done!
[17:35:11] <bmansurov>	 Niharika: thanks! Are you able to deploy my other patch too?
[17:35:46] <Niharika>	 bmansurov: In a moment. Doing stephane's first. 
[17:35:51] <wikibugs>	 (03Merged) 10jenkins-bot: Welcome survey: add a control group to viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491975 (https://phabricator.wikimedia.org/T216669) (owner: 10Sbisson)
[17:36:00] <bmansurov>	 Niharika: ok sure
[17:36:23] <stephanebisson>	 Niharika: I can't really test this on a debug server. You can sync right away.
[17:36:27] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: x1 on db2033 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 301.23 seconds
[17:36:30] <Niharika>	 stephanebisson: Cool. 
[17:37:06] <marostegui>	 ^ that lag is from RoanKattouw script purging stuff from x1
[17:37:14] <RoanKattouw>	 Yikes
[17:37:19] <wikibugs>	 (03CR) 10Niharika29: [C: 03+2] Labs: actually enable the demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493221 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[17:37:22] <RoanKattouw>	 I have waitForSlaves in my script :|
[17:37:35] <marostegui>	 RoanKattouw: don't worry, it is codfw
[17:37:42] <RoanKattouw>	 Oooh right
[17:37:47] <marostegui>	 :)
[17:37:49] <logmsgbot>	 !log niharika29@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Welcome survey: add a control group to viwiki T216669 (duration: 00m 54s)
[17:37:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:37:52] <stashbot>	 T216669: Welcome survey: change experiment to include control group - https://phabricator.wikimedia.org/T216669
[17:37:59] <wikibugs>	 (03CR) 10jenkins-bot: Stop collecting data for CitaitonUsage and CitationUsagePageLoad [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492345 (https://phabricator.wikimedia.org/T213969) (owner: 10Bmansurov)
[17:38:01] <wikibugs>	 (03CR) 10jenkins-bot: Welcome survey: add a control group to viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491975 (https://phabricator.wikimedia.org/T216669) (owner: 10Sbisson)
[17:38:02] <Niharika>	 stephanebisson: Done.
[17:38:18] <wikibugs>	 (03Merged) 10jenkins-bot: Labs: actually enable the demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493221 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[17:38:31] <wikibugs>	 (03CR) 10jenkins-bot: Labs: actually enable the demographics survey [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493221 (https://phabricator.wikimedia.org/T217171) (owner: 10Bmansurov)
[17:38:41] <Niharika>	 bmansurov: I've merged your other patch, I think it gets auto-deployed since it's beta. I can deploy anyway if you want it out right away. 
[17:38:54] <bmansurov>	 Niharika: thanks! I think I can wait.
[17:38:54] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA, 10Patch-For-Review: db1114 crashed (HW memory issues) - https://phabricator.wikimedia.org/T214720 (10Cmjohnson) a new motherboard arrives tomorrow 28/2/2019 to be replaced.
[17:40:40] <stephanebisson>	 Niharika: thanks!
[17:43:37] <icinga-wm>	 PROBLEM - Host analytics1071.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[17:43:54] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA, 10Patch-For-Review: db1114 crashed (HW memory issues) - https://phabricator.wikimedia.org/T214720 (10Marostegui) @Cmjohnson remember this host has MySQL down already, so you can just power it off yourself whenever you are ready for the mainboard replacement.  Thanks
[17:48:43] <icinga-wm>	 RECOVERY - Host analytics1071.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.72 ms
[17:48:57] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) Setup new ports for the mw systems in a5:    ` [edit interfaces interface-range disabled] -    member ge-5/0/1; -    member ge-5/0/0; -    member ge-5/0/2; -    membe...
[17:49:58] <wikibugs>	 (03PS2) 10Elukey: hadoop analytics: set analytics1071 rack config to A5 [puppet] - 10https://gerrit.wikimedia.org/r/493169 (https://phabricator.wikimedia.org/T212348)
[17:51:45] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] hadoop analytics: set analytics1071 rack config to A5 [puppet] - 10https://gerrit.wikimedia.org/r/493169 (https://phabricator.wikimedia.org/T212348) (owner: 10Elukey)
[17:53:31] <Niharika>	 kostajh: Your patch is on mwdebug1002. 
[17:53:46] <kostajh>	 Niharika: thanks, testing now
[17:55:35] <kostajh>	 Niharika: looks good, please deploy
[17:55:46] <wikibugs>	 10Operations, 10ops-eqiad, 10netops, 10Patch-For-Review: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348 (10RobH) ports in a7 for ms-be and ms-fe setup, using ports that were for old mw systems on old switch stack and will be unused on new stack:  robh@asw2-a-eqiad# show | compar...
[17:56:29] <elukey>	 !log roll restart hadoop hdfs namenodes on an-master100[1,2] to pick up the new rack config of analytics1071
[17:56:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:57:18] <logmsgbot>	 !log niharika29@deploy1001 Synchronized php-1.33.0-wmf.19/extensions/Flow/: Make VisualEditor unwrap <section> tags T217206 (duration: 01m 00s)
[17:57:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:57:22] <stashbot>	 T217206: [wmf.19-regression] VE Flow pages misplaces toolbar - https://phabricator.wikimedia.org/T217206
[17:57:23] <Niharika>	 kostajh: Done. 
[17:57:35] <Niharika>	 And that wraps up today's morning swat. 
[18:13:01] <robh>	 Ok, since swat is done
[18:13:06] <robh>	 we are going to relocate 6 mw systems in eqiad
[18:13:19] <robh>	 please no mw config changes for now!  (will udpate when done)
[18:14:17] <icinga-wm>	 RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 124, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[18:14:45] <Krenair>	 moritzm, hi
[18:15:09] <cmjohnson1>	 !log powering off mw1261 to move to different rack A5  T212348
[18:15:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:15:13] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[18:15:36] <Krenair>	 moritzm, I was wondering about the details of your PHP package work on deployment-mediawiki-09.deployment-prep.eqiad.wmflabs. That host appears to have a problem: https://phabricator.wikimedia.org/T217261
[18:16:48] <Krenair>	 (looks idle, have posted what I've found so far on the task)
[18:18:52] <icinga-wm>	 PROBLEM - Host mw1261 is DOWN: PING CRITICAL - Packet loss = 100%
[18:19:00] <icinga-wm>	 PROBLEM - Host mw1261.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:20:15] <wikibugs>	 (03CR) 10Smalyshev: "> Patch Set 2: Code-Review-2" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489598 (https://phabricator.wikimedia.org/T215684) (owner: 10Smalyshev)
[18:20:57] <wikibugs>	 (03CR) 10CRusnov: "This change is ready for review." [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[18:21:23] <cmjohnson1>	 !log powering off mw1262 to move to different rack A5  T212348
[18:21:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:21:26] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[18:22:22] <icinga-wm>	 RECOVERY - Host mw1261 is UP: PING OK - Packet loss = 0%, RTA = 36.16 ms
[18:24:08] <icinga-wm>	 RECOVERY - Host mw1261.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.99 ms
[18:25:28] <icinga-wm>	 PROBLEM - Host mw1262 is DOWN: PING CRITICAL - Packet loss = 100%
[18:25:33] <logmsgbot>	 !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@465673b]: Redeploy GUI for T217161
[18:25:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:25:37] <stashbot>	 T217161: Wikidata Query Service short url function broken - https://phabricator.wikimedia.org/T217161
[18:27:02] <icinga-wm>	 PROBLEM - Host mw1262.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:28:16] <cmjohnson1>	 !log powering off mw126[3-6] one at a time  to move to different rack A5  T212348
[18:28:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:28:19] <stashbot>	 T212348: Move servers off asw2-a5-eqiad - https://phabricator.wikimedia.org/T212348
[18:29:00] <icinga-wm>	 RECOVERY - Host mw1262 is UP: PING WARNING - Packet loss = 44%, RTA = 162.50 ms
[18:30:00] <robh>	 since these mw system alerts arent generating pages
[18:30:03] <robh>	 i didnt put them into maint mode
[18:30:07] <robh>	 i rather see them recover ;]
[18:30:50] <icinga-wm>	 PROBLEM - Host mw1263 is DOWN: PING CRITICAL - Packet loss = 100%
[18:32:08] <icinga-wm>	 RECOVERY - Host mw1262.mgmt is UP: PING OK - Packet loss = 0%, RTA = 37.46 ms
[18:32:36] <icinga-wm>	 PROBLEM - Host mw1263.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:34:04] <wikibugs>	 10Operations, 10Proton, 10Core Platform Team Backlog (Watching / External), 10Reading-Infrastructure-Team-Backlog (Kanban), 10Services (watching): Proton fails with Chromium 72.0.3626.96 - https://phabricator.wikimedia.org/T216493 (10Tgr) We should probably ask upstream if they have a policy about this k...
[18:35:37] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Close the grwp-wici mailing list - https://phabricator.wikimedia.org/T217247 (10Aklapper) (Off-topic and not my business, but I'd be curious which other communication methods are preferred and why, if you're okay with adding another sentence.)
[18:36:25] <logmsgbot>	 !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@465673b]: Redeploy GUI for T217161 (duration: 10m 51s)
[18:36:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:36:28] <stashbot>	 T217161: Wikidata Query Service short url function broken - https://phabricator.wikimedia.org/T217161
[18:36:40] <icinga-wm>	 RECOVERY - Host mw1263 is UP: PING WARNING - Packet loss = 44%, RTA = 138.84 ms
[18:37:46] <icinga-wm>	 RECOVERY - Host mw1263.mgmt is UP: PING OK - Packet loss = 0%, RTA = 37.16 ms
[18:39:22] <icinga-wm>	 PROBLEM - Host mw1264 is DOWN: PING CRITICAL - Packet loss = 100%
[18:39:42] <icinga-wm>	 PROBLEM - Host mw1264.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:43:23] <wikibugs>	 10Operations, 10ops-eqiad: mw1264 DIMM error - https://phabricator.wikimedia.org/T217274 (10Cmjohnson)
[18:43:50] <wikibugs>	 10Operations, 10ops-eqiad: mw1264 DIMM error - https://phabricator.wikimedia.org/T217274 (10Cmjohnson) While powering the server off to move to rack A5 I swapped the B1 DIMM with A1 to see if error follows.
[18:43:56] <wikibugs>	 (03PS1) 10Kosta Harlan: GrowthExperiments: Start help panel experiment on viwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493287 (https://phabricator.wikimedia.org/T215666)
[18:44:45] <wikibugs>	 10Operations, 10serviceops, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Test different growth factors for memcached (prep step for upgrade to newer versions) - https://phabricator.wikimedia.org/T217020 (10elukey) I had a very interesting chat with upstream on IRC (thanks dormando!) and it w...
[18:45:13] <icinga-wm>	 RECOVERY - Host mw1264 is UP: PING OK - Packet loss = 0%, RTA = 37.82 ms
[18:45:21] <icinga-wm>	 RECOVERY - Host mw1264.mgmt is UP: PING OK - Packet loss = 0%, RTA = 36.81 ms
[18:45:36] <wikibugs>	 (03Abandoned) 10Herron: logstash: add kafka logging role to logstash101[012] [puppet] - 10https://gerrit.wikimedia.org/r/493102 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[18:47:37] <icinga-wm>	 PROBLEM - Host mw1265 is DOWN: PING CRITICAL - Packet loss = 100%
[18:47:57] <icinga-wm>	 PROBLEM - Host mw1265.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[18:51:51] <icinga-wm>	 RECOVERY - Host mw1265 is UP: PING OK - Packet loss = 0%, RTA = 36.14 ms
[18:52:23] <wikibugs>	 (03PS1) 10Alex Monk: network: Fix list of bastions for labs realm [puppet] - 10https://gerrit.wikimedia.org/r/493289
[18:53:07] <icinga-wm>	 RECOVERY - Host mw1265.mgmt is UP: PING OK - Packet loss = 0%, RTA = 37.04 ms
[18:54:39] <icinga-wm>	 PROBLEM - Host mw1266 is DOWN: PING CRITICAL - Packet loss = 100%
[18:54:56] <wikibugs>	 (03PS1) 10Herron: kafka-logging: replace logstash1004 with logstash1010 [puppet] - 10https://gerrit.wikimedia.org/r/493290 (https://phabricator.wikimedia.org/T213898)
[18:56:23] <SMalyshev>	 !log depooled wdqs1005 to let it catch up
[18:56:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:56:35] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] "Looks right -- thank you!" [puppet] - 10https://gerrit.wikimedia.org/r/493289 (owner: 10Alex Monk)
[18:56:59] <wikibugs>	 (03CR) 10Alex Monk: "To reproduce the issue Adam was running into, go onto secondary.bastion.wmflabs.org (bastion-eqiad1-02) and try `ssh deployment-mediawiki-" [puppet] - 10https://gerrit.wikimedia.org/r/493289 (owner: 10Alex Monk)
[18:57:07] <wikibugs>	 (03CR) 10Alex Monk: "thanks" [puppet] - 10https://gerrit.wikimedia.org/r/493289 (owner: 10Alex Monk)
[18:58:31] <icinga-wm>	 RECOVERY - Host mw1266 is UP: PING OK - Packet loss = 0%, RTA = 36.15 ms
[19:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T1900)
[19:01:55] <wikibugs>	 (03PS2) 10Herron: kafka-logging: replace logstash1004 with logstash1010 [puppet] - 10https://gerrit.wikimedia.org/r/493290 (https://phabricator.wikimedia.org/T213898)
[19:09:21] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10ayounsi)
[19:10:13] <wikibugs>	 10Operations, 10serviceops, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Test different growth factors for memcached (prep step for upgrade to newer versions) - https://phabricator.wikimedia.org/T217020 (10Joe) FWIW I would like to decrease the max key size instead than increasing it.
[19:13:32] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10Cmjohnson) @elukey I moved the host to A6 and updated netbox. Arzhel updated network switch cfg. DNS will need to be updated and then ready f...
[19:13:46] <wikibugs>	 10Operations, 10Beta-Cluster-Infrastructure, 10DNS, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) This is still caus...
[19:14:04] <wikibugs>	 (03CR) 10Herron: [C: 03+1] "PCC looks good https://puppet-compiler.wmflabs.org/compiler1002/14911/" [puppet] - 10https://gerrit.wikimedia.org/r/493290 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[19:16:19] <wikibugs>	 (03PS1) 10CRusnov: Add LimitCORE support for uwsgi units. [puppet] - 10https://gerrit.wikimedia.org/r/493294
[19:17:48] <wikibugs>	 (03PS1) 10Elukey: Add A/PTR records for labsdb1012 [dns] - 10https://gerrit.wikimedia.org/r/493295 (https://phabricator.wikimedia.org/T215231)
[19:20:14] <wikibugs>	 (03CR) 10Ayounsi: [C: 03+1] Add A/PTR records for labsdb1012 [dns] - 10https://gerrit.wikimedia.org/r/493295 (https://phabricator.wikimedia.org/T215231) (owner: 10Elukey)
[19:20:42] <wikibugs>	 10Operations, 10ops-eqiad, 10monitoring, 10Patch-For-Review: icinga1001 crashed - https://phabricator.wikimedia.org/T214760 (10Volans) @Cmjohnson my tests on icinga1001 are completed, so feel free to shutdown at will when parts are available.
[19:23:19] <wikibugs>	 10Operations, 10Beta-Cluster-Infrastructure, 10DNS, 10Traffic, and 4 others: Ferm's upstream Net::DNS Perl library questionable handling of NOERROR responses without records causing puppet errors when we try to @resolve AAAA in labs - https://phabricator.wikimedia.org/T153468 (10Krenair) Added a new deploy...
[19:25:11] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Add A/PTR records for labsdb1012 [dns] - 10https://gerrit.wikimedia.org/r/493295 (https://phabricator.wikimedia.org/T215231) (owner: 10Elukey)
[19:26:51] <herron>	 !log replacing kafka on logstash1004 with logstash1010 T213898
[19:26:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:26:54] <stashbot>	 T213898: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898
[19:27:19] <wikibugs>	 (03PS1) 10Volans: Initial tox setup [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/493297
[19:27:21] <wikibugs>	 (03PS1) 10Volans: icinga: add check_icinga script [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/493298
[19:32:48] <SMalyshev>	 !log repooled wdqs1005, caught up
[19:32:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:33:34] <wikibugs>	 (03PS3) 10Herron: kafka-logging: replace logstash1004 with logstash1010 [puppet] - 10https://gerrit.wikimedia.org/r/493290 (https://phabricator.wikimedia.org/T213898)
[19:34:54] <wikibugs>	 (03PS1) 10Elukey: Add labsdb1012 basic puppet settings [puppet] - 10https://gerrit.wikimedia.org/r/493299 (https://phabricator.wikimedia.org/T215231)
[19:35:04] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics, 10Patch-For-Review, 10User-Elukey: rack/setup/install labsdb1012.eqiad.wmnet - https://phabricator.wikimedia.org/T215231 (10elukey)
[19:35:32] <wikibugs>	 (03CR) 10CRusnov: [C: 03+1] "Looks good to me." [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/493297 (owner: 10Volans)
[19:37:00] <wikibugs>	 (03PS2) 10Elukey: Add labsdb1012 basic puppet settings [puppet] - 10https://gerrit.wikimedia.org/r/493299 (https://phabricator.wikimedia.org/T215231)
[19:38:19] <hashar>	 jouncebot: next
[19:38:19] <jouncebot>	 In 0 hour(s) and 21 minute(s): MediaWiki train - Americas version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T2000)
[19:38:21] <hashar>	 yeah so
[19:38:35] <hashar>	 I will take care of promoting 1.33.0-wmf.19 to group1 in 20 minutes ;D
[19:40:36] <wikibugs>	 10Operations, 10ops-codfw: Degraded RAID on thumbor2003 - https://phabricator.wikimedia.org/T217118 (10jijiki) 05Open→03Invalid Server was being reimaged
[19:42:40] <wikibugs>	 (03PS6) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[19:43:36] <bstorm_>	 !log downtimed labsdb1004 to stop mysql for transferring data for T193264
[19:43:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:43:40] <stashbot>	 T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 - https://phabricator.wikimedia.org/T193264
[19:44:39] <wikibugs>	 10Operations, 10monitoring, 10User-Joe, 10User-jijiki, 10Wikimedia-Incident: Monitor redis memory/disk usage - https://phabricator.wikimedia.org/T110169 (10jijiki)
[19:46:26] <wikibugs>	 (03CR) 10Herron: [C: 03+2] kafka-logging: replace logstash1004 with logstash1010 [puppet] - 10https://gerrit.wikimedia.org/r/493290 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[19:48:08] <wikibugs>	 10Operations, 10Analytics, 10Analytics-Kanban, 10hardware-requests: GPU upgrade for stat1005 - https://phabricator.wikimedia.org/T216226 (10EBernhardson) I have submitted a support request to an "AMD Expert" and will see what they have to say about GPU's that are physically compatible with our servers.
[19:49:31] <bstorm_>	 !log stopped slave on labsbd1004 for T193264
[19:49:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:49:34] <stashbot>	 T193264: Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020 - https://phabricator.wikimedia.org/T193264
[19:53:05] <icinga-wm>	 PROBLEM - Check systemd state on logstash1010 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[19:55:20] <herron>	 ^ that’s me and is expected
[19:55:23] <herron>	 sorry for the noise
[19:59:49] <wikibugs>	 (03PS1) 10Catrope: Enable GrowthExperiments's email features in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493301
[20:00:04] <jouncebot>	 hashar: That opportune time is upon us again. Time for a MediaWiki train - Americas version deploy. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T2000).
[20:00:57] <wikibugs>	 (03CR) 10Catrope: [C: 03+2] Enable GrowthExperiments's email features in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493301 (owner: 10Catrope)
[20:01:06] <wikibugs>	 (03PS1) 10Hashar: group1 wikis to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493302
[20:01:07] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] group1 wikis to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493302 (owner: 10Hashar)
[20:01:20] <wikibugs>	 (03CR) 10CRusnov: [C: 03+1] "Overall, looks good! I have minor nitpicks with the way it's layed out but they are stylistic choices not breaking." (032 comments) [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/493298 (owner: 10Volans)
[20:01:50] <wikibugs>	 (03PS1) 10Herron: logstash1010: add ipv6 mapped address [puppet] - 10https://gerrit.wikimedia.org/r/493303 (https://phabricator.wikimedia.org/T213898)
[20:02:02] <wikibugs>	 (03Merged) 10jenkins-bot: Enable GrowthExperiments's email features in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493301 (owner: 10Catrope)
[20:02:19] <wikibugs>	 (03Merged) 10jenkins-bot: group1 wikis to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493302 (owner: 10Hashar)
[20:03:46] <wikibugs>	 (03CR) 10Herron: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/14912/logstash1010.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/493303 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[20:04:03] <logmsgbot>	 !log hashar@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.33.0-wmf.19
[20:04:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:04:57] <logmsgbot>	 !log hashar@deploy1001 Synchronized php: group1 wikis to 1.33.0-wmf.19 (duration: 00m 53s)
[20:05:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:05:35] <wikibugs>	 (03CR) 10jenkins-bot: Enable GrowthExperiments's email features in beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493301 (owner: 10Catrope)
[20:05:37] <wikibugs>	 (03CR) 10jenkins-bot: group1 wikis to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493302 (owner: 10Hashar)
[20:07:56] <wikibugs>	 (03PS1) 10Herron: logstash100[12]: add ipv6 mapped address [puppet] - 10https://gerrit.wikimedia.org/r/493306 (https://phabricator.wikimedia.org/T213898)
[20:08:39] <icinga-wm>	 PROBLEM - Apache HTTP on mw1328 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[20:09:13] <wikibugs>	 (03PS2) 10Herron: logstash101[12]: add ipv6 mapped address [puppet] - 10https://gerrit.wikimedia.org/r/493306 (https://phabricator.wikimedia.org/T213898)
[20:09:43] <icinga-wm>	 RECOVERY - Apache HTTP on mw1328 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 0.110 second response time
[20:10:35] <wikibugs>	 (03CR) 10Herron: [C: 03+2] "pcc looks good https://puppet-compiler.wmflabs.org/compiler1002/14913/" [puppet] - 10https://gerrit.wikimedia.org/r/493306 (https://phabricator.wikimedia.org/T213898) (owner: 10Herron)
[20:11:39] <hashar>	  Wikimedia\Assert\PostconditionException from line 140 of /srv/mediawiki/php-1.33.0-wmf.19/vendor/wikimedia/assert/src/Assert.php: Postcondition failed: Namespace for entity type property must be defined!
[20:11:42] <hashar>	 time for a rollback :/
[20:14:55] <logmsgbot>	 !log hashar@deploy1001 rebuilt and synchronized wikiversions files: (no justification provided)
[20:14:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:15:05] <wikibugs>	 (03PS1) 10Hashar: Revert "group1 wikis to 1.33.0-wmf.19" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493309 (https://phabricator.wikimedia.org/T206673)
[20:15:19] <hashar>	 :(
[20:15:39] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Already deployed via a local revert and deploy" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493309 (https://phabricator.wikimedia.org/T206673) (owner: 10Hashar)
[20:15:50] <hashar>	 so now is the tedious part
[20:15:53] <hashar>	 fill those tasks :(
[20:16:11] <wikibugs>	 (03PS1) 10Paladox: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/493311
[20:16:47] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "group1 wikis to 1.33.0-wmf.19" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493309 (https://phabricator.wikimedia.org/T206673) (owner: 10Hashar)
[20:17:33] <wikibugs>	 (03CR) 10jenkins-bot: Revert "group1 wikis to 1.33.0-wmf.19" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493309 (https://phabricator.wikimedia.org/T206673) (owner: 10Hashar)
[20:24:51] <wikibugs>	 (03PS1) 10Herron: network::constants: replace logstash1004 with logstash1010 [puppet] - 10https://gerrit.wikimedia.org/r/493315
[20:32:19] <wikibugs>	 (03PS1) 10Thcipriani: Scap: upgrade to 3.9.1-1 [puppet] - 10https://gerrit.wikimedia.org/r/493317 (https://phabricator.wikimedia.org/T217287)
[20:34:03] <wikibugs>	 (03CR) 10Herron: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/14917/" [puppet] - 10https://gerrit.wikimedia.org/r/493315 (owner: 10Herron)
[20:38:49] <icinga-wm>	 PROBLEM - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/k8s/nodes/ready - 185 bytes in 0.226 second response time
[20:42:50] <icinga-wm>	 RECOVERY - Check systemd state on logstash1010 is OK: OK - running: The system is fully operational
[20:44:37] <wikibugs>	 (03PS1) 10Hashar: group1 (except commonswiki) to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493319 (https://phabricator.wikimedia.org/T217285)
[20:46:19] <hashar>	 ^^^ not rushing that yet
[20:46:26] <hashar>	 pending reviews with other releng people ;)
[20:48:23] <wikibugs>	 (03Abandoned) 10Hashar: group1 (except commonswiki) to 1.33.0-wmf.19 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493319 (https://phabricator.wikimedia.org/T217285) (owner: 10Hashar)
[20:48:43] <wikibugs>	 (03PS1) 10Smalyshev: Add config for switching Wikibase search to WikibaseCirrusSearch codebase [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493320 (https://phabricator.wikimedia.org/T215684)
[20:49:15] <icinga-wm>	 RECOVERY - toolschecker: All k8s worker nodes are healthy on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.247 second response time
[20:49:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add config for switching Wikibase search to WikibaseCirrusSearch codebase [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493320 (https://phabricator.wikimedia.org/T215684) (owner: 10Smalyshev)
[20:50:23] <hashar>	 !log 1.33.0-wmf.19 not rolled to group1. Pending T217285 (Wikibase raising exception on commonswiki). To be figured out during European day time.
[20:50:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:50:26] <stashbot>	 T217285: commonswiki / wikibase: Postcondition failed: Namespace for entity type property must be defined! - https://phabricator.wikimedia.org/T217285
[20:51:24] <wikibugs>	 (03PS2) 10Smalyshev: Add config for switching Wikibase search to WikibaseCirrusSearch codebase [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493320 (https://phabricator.wikimedia.org/T215684)
[20:53:56] <XioNoX>	 !log delete local pref for peering sessions in codfw/eqdfw - T204281
[20:53:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:54:00] <stashbot>	 T204281: Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281
[20:54:02] <wikibugs>	 (03PS1) 10Smalyshev: Enable WikibaseCirrusSearch on Beta Cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493321 (https://phabricator.wikimedia.org/T215684)
[20:55:37] <wikibugs>	 (03PS3) 10Smalyshev: Enable WikibaseCirrusSearch on testwikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/489598 (https://phabricator.wikimedia.org/T215684)
[21:00:04] <jouncebot>	 cscott, arlolra, subbu, bearND, halfak, and Amir1: Your horoscope predicts another unfortunate Services – Parsoid / Citoid / Mobileapps / ORES / … deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190227T2100).
[21:02:33] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) cr1/2-codfw + cr2-eqdfw `lang=diff [edit policy-options policy-statement BGP_community_actions] -    term peer-private-peer { -        from community PEER_PRIVATE...
[21:04:58] <wikibugs>	 (03PS1) 10Anomie: Logstash: Use log context for the api-feature-usage channel [puppet] - 10https://gerrit.wikimedia.org/r/493323 (https://phabricator.wikimedia.org/T217162)
[21:06:25] <wikibugs>	 (03CR) 10Anomie: [C: 04-1] "Do not merge/deploy until core change Ie891257140ea19369e10b2e91463a1fb4aa5d233 is deployed to all wikis." [puppet] - 10https://gerrit.wikimedia.org/r/493323 (https://phabricator.wikimedia.org/T217162) (owner: 10Anomie)
[21:18:41] <icinga-wm>	 RECOVERY - nova-compute proc minimum on cloudvirt1018 is OK: PROCS OK: 1 process with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[21:19:55] <icinga-wm>	 PROBLEM - nova-compute proc minimum on cloudvirt1018 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/pytho[n] /usr/bin/nova-compute
[21:24:19] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Because of the way codfw/eqdfw peers with eqiad and eqord and the fact we have similar peers in several IXPs Removing the preferred local-pref in Dallas caused th...
[21:26:35] <XioNoX>	 !log delete local pref for peering sessions in eqord - T204281
[21:26:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:26:39] <stashbot>	 T204281: Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281
[21:28:48] <wikibugs>	 (03PS3) 10Gehel: elasticsearch: use the admin Reason to get current hostname [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341
[21:28:50] <wikibugs>	 (03PS1) 10Gehel: Expose hostname from Reason. [software/spicerack] - 10https://gerrit.wikimedia.org/r/493325
[21:28:56] <icinga-wm>	 PROBLEM - Check systemd state on wdqs1005 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[21:32:08] <wikibugs>	 (03CR) 10Gehel: "> Patch Set 2:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341 (owner: 10Gehel)
[21:33:46] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Close the grwp-wici mailing list - https://phabricator.wikimedia.org/T217247 (10RobH) Can this list be wholly removed?  I checked, and while it has an archive, the archive is public.  So we can delete the list and leave the archive in place if needed.  Please advise if...
[21:37:10] <wikibugs>	 (03CR) 10Gehel: "> Patch Set 2:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492375 (owner: 10Gehel)
[21:37:48] <wikibugs>	 (03CR) 10Volans: [C: 03+2] "LGTM, thanks for splitting the commits." [software/spicerack] - 10https://gerrit.wikimedia.org/r/493325 (owner: 10Gehel)
[21:42:50] <wikibugs>	 (03CR) 10Gehel: "> P.S. it seems that there is a branch not covered (326->328)" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492375 (owner: 10Gehel)
[21:43:16] <wikibugs>	 (03Merged) 10jenkins-bot: Expose hostname from Reason. [software/spicerack] - 10https://gerrit.wikimedia.org/r/493325 (owner: 10Gehel)
[21:44:05] <wikibugs>	 (03CR) 10jenkins-bot: Expose hostname from Reason. [software/spicerack] - 10https://gerrit.wikimedia.org/r/493325 (owner: 10Gehel)
[21:44:27] <wikibugs>	 (03CR) 10Volans: [C: 03+2] "LGTM" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341 (owner: 10Gehel)
[21:46:59] <wikibugs>	 10Operations, 10Office-IT, 10Wikimedia-Mailing-lists, 10CommRel-Specialists-Support (Jan-Mar-2019): Mailing list migration for Arbitration Committee to Google Group - https://phabricator.wikimedia.org/T215940 (10RobH) a:03eross Since this isn't currently pending any #operations work, I'm removing that fl...
[21:48:27] <wikibugs>	 (03Merged) 10jenkins-bot: elasticsearch: use the admin Reason to get current hostname [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341 (owner: 10Gehel)
[21:49:16] <wikibugs>	 (03CR) 10jenkins-bot: elasticsearch: use the admin Reason to get current hostname [software/spicerack] - 10https://gerrit.wikimedia.org/r/492341 (owner: 10Gehel)
[21:55:46] <wikibugs>	 (03PS5) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[21:55:53] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: request of a new mailing list WIKI-BNCF - https://phabricator.wikimedia.org/T214059 (10RobH) 05Open→03Resolved
[21:56:47] <wikibugs>	 (03PS6) 10Cwhite: prometheus: clean up node exporter transition code [puppet] - 10https://gerrit.wikimedia.org/r/493269 (https://phabricator.wikimedia.org/T213708)
[21:57:21] <XioNoX>	 !log delete local pref for peering sessions in eqiad - T204281
[21:57:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:57:29] <stashbot>	 T204281: Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281
[21:58:00] <wikibugs>	 (03Abandoned) 10Cwhite: prometheus: post-upgrade node-exporter cleanup [puppet] - 10https://gerrit.wikimedia.org/r/489325 (https://phabricator.wikimedia.org/T213708) (owner: 10Cwhite)
[22:10:19] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Same config diff for eqord/eqiad  eqord: Transit: https://librenms.wikimedia.org/graphs/id=16837,16841/type=multiport_bits_separate Peering: https://librenms.wiki...
[22:12:14] <icinga-wm>	 PROBLEM - Check systemd state on logstash1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[22:12:21] <icinga-wm>	 PROBLEM - Kafka Broker Server on logstash1004 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties
[22:12:35] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on logstash1010 is CRITICAL: 1.16e+07 ge 5e+05 https://grafana.wikimedia.org/dashboard/db/kafka?panelId=16&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=logging-eqiad&var-kafka_broker=logstash1010
[22:12:59] <chaomodus>	 a page
[22:13:33] <godog>	 indeed, I believe there was some work on those brokers today
[22:14:07] <herron>	 that’s me, downtime expired
[22:14:08] <herron>	 acking
[22:14:24] <volans>	 ah ok, it was shutdown at 19:31 UTC apparently :)
[22:14:27] <herron>	 it’s actually not running kafka anymore
[22:15:07] <godog>	 ok!
[22:15:09] <godog>	 going back afk
[22:15:46] <volans>	 ack
[22:15:46] <icinga-wm>	 ACKNOWLEDGEMENT - Kafka Broker Server on logstash1004 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties Herron downtime expired -- host being decommed
[22:15:46] <herron>	 thanks, and sorry :/
[22:16:13] * apergos peeks in, takes note, disappears again
[22:17:58] <cdanis>	 glad it's easy :)\
[22:26:14] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on logstash1010 is OK: (C)5e+05 ge (W)1e+05 ge 4.346e+04 https://grafana.wikimedia.org/dashboard/db/kafka?panelId=16&fullscreen&orgId=1&var-datasource=eqiad+prometheus/ops&var-kafka_cluster=logging-eqiad&var-kafka_broker=logstash1010
[22:26:26] <wikibugs>	 (03PS1) 10Milimetric: Find db host and port using refinery [puppet] - 10https://gerrit.wikimedia.org/r/493331 (https://phabricator.wikimedia.org/T215290)
[22:27:15] <wikibugs>	 (03CR) 10Milimetric: [C: 04-1] "This should be merged after https://gerrit.wikimedia.org/r/#/c/analytics/refinery/+/492209 is successfully merged" [puppet] - 10https://gerrit.wikimedia.org/r/493331 (https://phabricator.wikimedia.org/T215290) (owner: 10Milimetric)
[22:28:34] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+1] "Needs to wait to merge until 1.33.0-wmf.20 has deployed to all wikis (in theory after 2019-03-07)." [puppet] - 10https://gerrit.wikimedia.org/r/493323 (https://phabricator.wikimedia.org/T217162) (owner: 10Anomie)
[22:46:41] <wikibugs>	 10Operations, 10netops, 10Performance-Team (Radar): Stop prioritizing peering over transit - https://phabricator.wikimedia.org/T204281 (10ayounsi) Shift from peering to transit are: ~200Mbps in eqdfw ~300Mbps in eqord ~300Mbps in eqiad
[22:47:55] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898 (10herron) kafka service from logstash1004 has been migrated to logstash1010, and logsta...
[22:49:44] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898 (10herron)
[22:49:59] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Replace and expand Elasticsearch storage in eqiad and upgrade the cluster from Debian jessie to stretch - https://phabricator.wikimedia.org/T213898 (10herron)
[23:02:51] <wikibugs>	 (03CR) 10Hashar: "The table at https://doc.wikimedia.org/docker-pkg/#api  looks better :)" [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/485626 (owner: 10Hashar)
[23:16:13] <wikibugs>	 (03CR) 10EBernhardson: [WIP] Add support for elasticsearch 6 (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[23:18:50] <wikibugs>	 (03CR) 10EBernhardson: [C: 03+1] [cirrus] Cleanup transitional states [mediawiki-config] - 10https://gerrit.wikimedia.org/r/476273 (https://phabricator.wikimedia.org/T210381) (owner: 10DCausse)
[23:21:20] <wikibugs>	 (03CR) 10EBernhardson: "Looks reasonable. Of course we can't merge until cutting the final release version of the plugins." (031 comment) [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/491297 (https://phabricator.wikimedia.org/T199791) (owner: 10DCausse)
[23:23:46] <Zppix>	 Jouncebot: next
[23:23:46] <jouncebot>	 In 0 hour(s) and 36 minute(s): Evening SWAT (Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190228T0000)
[23:32:42] <wikibugs>	 (03PS1) 10Ladsgroup: Add federation configs for commonswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/493339 (https://phabricator.wikimedia.org/T217285)
[23:36:12] <wikibugs>	 10Operations, 10Availability (MediaWiki-MultiDC), 10Core Platform Team Backlog (Designing), 10Performance-Team (Radar), 10Services (designing): Consider REST with SSL (HyperSwitch/Cassandra) for session storage - https://phabricator.wikimedia.org/T134811 (10mobrovac) 05Open→03Resolved a:03mobrovac...
[23:43:57] <Amir1>	 marktraceur matthiasmullie Does this make sense to you? https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/493339
[23:44:10] <Amir1>	 If it does, I deploy it so we stop getting fatals
[23:49:41] <icinga-wm>	 ACKNOWLEDGEMENT - HP RAID on db2033 is CRITICAL: CRITICAL: Slot 0: Failed: 1I:1:12 - OK: 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11 - Controller: OK - Cache: Temporarily Disabled - Battery/Capacitor: Recharging nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T217301
[23:49:48] <wikibugs>	 10Operations, 10ops-codfw: Degraded RAID on db2033 - https://phabricator.wikimedia.org/T217301 (10ops-monitoring-bot)
[23:59:37] <jdlrobson>	 Can I use X-Wikimedia-Debug (or any other header)  to specify I want content from mw1332 backend?