[00:00:28] <wikibugs>	 (03CR) 10Bartosz Dziewoński: [C: 03+1] "Thanks for looking into this!" [puppet] - 10https://gerrit.wikimedia.org/r/517564 (https://phabricator.wikimedia.org/T225882) (owner: 10Nuria)
[00:00:59] <wikibugs>	 (03CR) 10Bartosz Dziewoński: [C: 03+1] [WIP] Enable downloads of matomo.js (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/517564 (https://phabricator.wikimedia.org/T225882) (owner: 10Nuria)
[00:04:52] <wikibugs>	 (03PS15) 10CRusnov: profile::netbox: Reorganize for splitting front and back-end. [puppet] - 10https://gerrit.wikimedia.org/r/514395 (https://phabricator.wikimedia.org/T223291)
[00:05:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::netbox: Reorganize for splitting front and back-end. [puppet] - 10https://gerrit.wikimedia.org/r/514395 (https://phabricator.wikimedia.org/T223291) (owner: 10CRusnov)
[00:09:41] <icinga-wm>	 PROBLEM - Check systemd state on ms-be2027 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[00:13:26] <wikibugs>	 (03PS1) 10Catrope: Enable $wgEchoPollForUpdates on cawiki in beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517572
[00:14:51] <wikibugs>	 (03CR) 10Catrope: [C: 03+2] Enable $wgEchoPollForUpdates on cawiki in beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517572 (owner: 10Catrope)
[00:15:38] <RoanKattouw>	 !log Running populateRevisionSha1.php on testwiki for T219816
[00:15:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:15:44] <stashbot>	 T219816: Fatal IncompleteRevisionException when restoring an article (undelete page) - https://phabricator.wikimedia.org/T219816
[00:16:18] <wikibugs>	 (03Merged) 10jenkins-bot: Enable $wgEchoPollForUpdates on cawiki in beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517572 (owner: 10Catrope)
[00:16:33] <wikibugs>	 (03CR) 10jenkins-bot: Enable $wgEchoPollForUpdates on cawiki in beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517572 (owner: 10Catrope)
[00:19:29] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration, 10User-Ryasmeen: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10matmarex) The problem continues to happen...
[00:22:52] <RoanKattouw>	 !log Running populateRevisionSha1.php on dewikivoyage for T219816
[00:22:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:22:57] <stashbot>	 T219816: Fatal IncompleteRevisionException when restoring an article (undelete page) - https://phabricator.wikimedia.org/T219816
[00:25:01] <wikibugs>	 (03PS1) 10Thcipriani: blubberoid: Add policy file [deployment-charts] - 10https://gerrit.wikimedia.org/r/517573 (https://phabricator.wikimedia.org/T215319)
[00:25:58] <wikibugs>	 (03PS16) 10CRusnov: profile::netbox: Reorganize for splitting front and back-end. [puppet] - 10https://gerrit.wikimedia.org/r/514395 (https://phabricator.wikimedia.org/T223291)
[00:26:52] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] profile::netbox: Reorganize for splitting front and back-end. [puppet] - 10https://gerrit.wikimedia.org/r/514395 (https://phabricator.wikimedia.org/T223291) (owner: 10CRusnov)
[00:38:10] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Mailing list admin pass reset for winedale-l (for migration off lists.wikimedia.org) - https://phabricator.wikimedia.org/T224612 (10Quiddity) 05Open→03Resolved a:03Quiddity Confirmed resolved.
[00:40:05] <icinga-wm>	 RECOVERY - Check systemd state on ms-be2027 is OK: OK - running: The system is fully operational
[00:48:18] <wikibugs>	 (03PS17) 10CRusnov: profile::netbox: Reorganize for splitting front and back-end. [puppet] - 10https://gerrit.wikimedia.org/r/514395 (https://phabricator.wikimedia.org/T223291)
[01:20:29] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Wikimedia-au-members and wikimedia-au-announce password reset - https://phabricator.wikimedia.org/T225712 (10StevenCrossin) 05Declined→03Open Sorry, this actually is still an issue. Can this please be actioned. Thanks,
[01:23:12] <wikibugs>	 10Operations, 10Wikimedia Australia, 10Wikimedia-Mailing-lists: Wikimedia-au-members and wikimedia-au-announce password reset - https://phabricator.wikimedia.org/T225712 (10Samwilson)
[01:35:45] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s3 on db2098 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 863.43 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave
[02:26:39] <icinga-wm>	 PROBLEM - puppet last run on bast3002 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[02:31:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 17548672 and 0 seconds
[02:33:25] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 134576 and 12 seconds
[02:37:01] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps1002 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB template1 (host:localhost) 220672968 and 7 seconds
[02:38:27] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps1002 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB template1 (host:localhost) 17072 and 19 seconds
[02:42:34] <wikibugs>	 (03CR) 10Ema: Disable TCP selective acknowledgements (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/517461 (owner: 10Muehlenhoff)
[02:46:43] <icinga-wm>	 RECOVERY - EDAC syslog messages on thumbor1004 is OK: (C)4 ge (W)2 ge 1 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=thumbor1004&var-datasource=eqiad+prometheus/ops
[02:57:35] <icinga-wm>	 RECOVERY - Memory correctable errors -EDAC- on thumbor1004 is OK: (C)4 ge (W)2 ge 1 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=thumbor1004&var-datasource=eqiad+prometheus/ops
[02:59:15] <icinga-wm>	 RECOVERY - puppet last run on bast3002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[03:08:23] <icinga-wm>	 RECOVERY - MariaDB Slave Lag: s3 on db2098 is OK: OK slave_sql_lag Replication lag: 0.30 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_slave
[03:30:33] <wikibugs>	 (03PS5) 10KartikMistry: Remove ExternalGuidanceEnableContextDetection [mediawiki-config] - 10https://gerrit.wikimedia.org/r/504261 (https://phabricator.wikimedia.org/T219819)
[03:42:46] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver upgrade -f cxserver-staging-values.yaml staging stable/cxserver [namespace: cxserver, clusters: staging]
[03:42:47] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver cluster staging completed
[03:42:47] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver finished
[03:42:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:42:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:42:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:45:08] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver upgrade -f cxserver-codfw-values.yaml production stable/cxserver [namespace: cxserver, clusters: codfw]
[03:45:09] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver cluster codfw completed
[03:45:09] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver finished
[03:45:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:45:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:45:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:46:12] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver upgrade -f cxserver-eqiad-values.yaml production stable/cxserver [namespace: cxserver, clusters: eqiad]
[03:46:13] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver cluster eqiad completed
[03:46:13] <logmsgbot>	 !log kartik@deploy1001 scap-helm cxserver finished
[03:46:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:46:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[03:46:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:24:25] <wikibugs>	 10Operations, 10ops-codfw, 10DBA: db2097 (codfw s1&s6 source backups) mariadb@s6 *process* (10.1.39) crashed on 2019-06-08 - https://phabricator.wikimedia.org/T225378 (10Marostegui) Great! So what do you have in mind?
[04:24:35] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool pc1008" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517585
[04:26:02] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool pc1008" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517585 (owner: 10Marostegui)
[04:26:52] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1008" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517585 (owner: 10Marostegui)
[04:27:07] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool pc1008" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517585 (owner: 10Marostegui)
[04:28:14] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Close the engineering mailing list - https://phabricator.wikimedia.org/T222308 (10ArielGlenn) >>! In T222308#5264037, @Jdforrester-WMF wrote: > Ping @ArielGlenn who is on SRE clinic duty this week(?)  Nah, that was last week. I'll pass this on to the person for this wee...
[04:28:27] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool pc1008 after optimizing its tables T210725 (duration: 00m 47s)
[04:28:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:28:35] <stashbot>	 T210725: Replace parsercache keys to something more meaningful on db-XXXX.php - https://phabricator.wikimedia.org/T210725
[04:33:15] <wikibugs>	 (03PS1) 10Marostegui: db2051: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/517586 (https://phabricator.wikimedia.org/T221533)
[04:34:08] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db2051: Enable notifications [puppet] - 10https://gerrit.wikimedia.org/r/517586 (https://phabricator.wikimedia.org/T221533) (owner: 10Marostegui)
[04:41:52] <wikibugs>	 10Operations, 10ops-eqiad: Storage problems with new host db1133 - https://phabricator.wikimedia.org/T222731 (10Marostegui) p:05Triage→03High
[04:44:06] <marostegui>	 !log Deploy schema change on db1073 (labtestwiki and labswiki) - T225643
[04:44:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:44:13] <stashbot>	 T225643: Schema change to oathauth_users - https://phabricator.wikimedia.org/T225643
[05:05:30] <icinga-wm>	 RECOVERY - cassandra service on maps1001 is OK: OK - cassandra is active https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[05:09:18] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ayounsi) @Cmjohnson I'm pretty sure they will have to move :( Creating a new vlan and all the supporting config (DHCP, routing, IP allocation, etc) is a non trivial task for a vlan...
[05:12:46] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) So, to be clear from my side:  Out of those 4 hosts, which 2 are on C row and 2 are on D row, we need 2 of them (I don't mind which ones) to be in the same VLAN as dbpro...
[05:16:56] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ayounsi) Ok good so that's 1018/1019). In which vlan the other 2 (dbproxy1020/1021) need to go then?
[05:18:14] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) dbproxy1020/1021 can go the same vlans as dbproxy1001-1008 as those will be replacing some of those
[05:22:20] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ayounsi) dbproxy1001-1008 are in the private vlans across row A-B-C, none in D. Is row D private fine for dbproxy1020/1021 or should they be in private-A/B/C ?
[05:23:54] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) >>! In T225704#5264272, @ayounsi wrote: > dbproxy1001-1008 are in the private vlans across row A-B-C, none in D. Is row D private fine for dbproxy1020/1021 or should the...
[05:27:55] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10ayounsi) No differences other than the physical row they're in. They will be able to reach the same resources.  @Cmjohnson so dbproxy1020/1021 will go in private1-d-eqiad.
[05:28:37] <wikibugs>	 10Operations, 10ops-eqiad, 10DBA: eqiad: rack/setup/install (4) dbproxy systems. - https://phabricator.wikimedia.org/T225704 (10Marostegui) >>! In T225704#5264276, @ayounsi wrote: > No differences other than the physical row they're in. They will be able to reach the same resources.  Great! Thanks :-)
[05:47:46] <wikibugs>	 (03PS2) 10Marostegui: db-eqiad.php: Set s4 in read only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517362 (https://phabricator.wikimedia.org/T224852)
[05:47:54] <wikibugs>	 (03PS2) 10Marostegui: db-eqiad.php: Promote db1081 to s4 master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517363 (https://phabricator.wikimedia.org/T224852)
[05:48:06] <wikibugs>	 (03PS2) 10Marostegui: mariadb: Promote db1081 to s4 master [puppet] - 10https://gerrit.wikimedia.org/r/517361 (https://phabricator.wikimedia.org/T224852)
[05:48:22] <wikibugs>	 (03PS1) 10Marostegui: install_server: Allow reimage db1112 [puppet] - 10https://gerrit.wikimedia.org/r/517589 (https://phabricator.wikimedia.org/T225981)
[05:54:05] <marostegui>	 !log Stop slave and mysql on db1112 to copy its content to dbstore1001:/srv/tmp/db1112 - T225981
[05:54:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[05:54:11] <stashbot>	 T225981: Replace db1077 with db1112 - https://phabricator.wikimedia.org/T225981
[05:55:39] <wikibugs>	 10Operations, 10ops-eqiad, 10Cloud-Services, 10cloud-services-team (Kanban): rack/setup/install (3) new osd ceph nodes - https://phabricator.wikimedia.org/T224188 (10ayounsi) There is no security or technical limitation on having 1 interface in each vlan as long as they don't go across rows. Hosts in one v...
[05:56:10] <icinga-wm>	 RECOVERY - cassandra CQL 10.64.0.79:9042 on maps1001 is OK: TCP OK - 0.000 second response time on 10.64.0.79 port 9042 https://phabricator.wikimedia.org/T93886
[06:05:57] <wikibugs>	 (03PS1) 10Elukey: role::analytics_test_cluster::coordinator: remove hive monitors [puppet] - 10https://gerrit.wikimedia.org/r/517590
[06:06:38] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] role::analytics_test_cluster::coordinator: remove hive monitors [puppet] - 10https://gerrit.wikimedia.org/r/517590 (owner: 10Elukey)
[06:13:47] <wikibugs>	 (03PS2) 10Marostegui: wmnet: Update s4-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/517360 (https://phabricator.wikimedia.org/T224852)
[06:19:20] <marostegui>	 !log Stop slave and mysql on db1112 to copy its content to dbprov1001:/srv/backups/tmp/db1112 - T225981
[06:19:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:19:25] <stashbot>	 T225981: Replace db1077 with db1112 - https://phabricator.wikimedia.org/T225981
[06:31:02] <icinga-wm>	 PROBLEM - puppet last run on an-worker1084 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/maven/ivysettings.xml]
[06:48:58] <wikibugs>	 (03PS8) 10Elukey: mcrouter: allow async foreign set/delete WAN cache operations [puppet] - 10https://gerrit.wikimedia.org/r/492948 (owner: 10Aaron Schulz)
[06:56:49] <onimisionipe>	 !log pooling maps1001 - reimage is complete - T224395
[06:56:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:56:55] <stashbot>	 T224395: Maps[12]004 /srv disk space is critical - https://phabricator.wikimedia.org/T224395
[06:58:06] <icinga-wm>	 RECOVERY - puppet last run on an-worker1084 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[07:00:51] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad,db-codfw.php: Remove db2039 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517591 (https://phabricator.wikimedia.org/T221533)
[07:01:44] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad,db-codfw.php: Remove db2039 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517591 (https://phabricator.wikimedia.org/T221533) (owner: 10Marostegui)
[07:02:34] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad,db-codfw.php: Remove db2039 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517591 (https://phabricator.wikimedia.org/T221533) (owner: 10Marostegui)
[07:02:51] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad,db-codfw.php: Remove db2039 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517591 (https://phabricator.wikimedia.org/T221533) (owner: 10Marostegui)
[07:03:41] <wikibugs>	 (03PS3) 10Marostegui: db-eqiad.php: Set s4 in read only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517362 (https://phabricator.wikimedia.org/T224852)
[07:03:50] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Remove db2039 from config T221533 (duration: 00m 51s)
[07:03:51] <wikibugs>	 (03PS3) 10Marostegui: db-eqiad.php: Promote db1081 to s4 master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517363 (https://phabricator.wikimedia.org/T224852)
[07:03:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:03:56] <stashbot>	 T221533: Decommission old coredb machines (<=db2042) - https://phabricator.wikimedia.org/T221533
[07:04:44] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-codfw.php: Remove db2039 from config T221533 (duration: 00m 46s)
[07:04:48] <icinga-wm>	 PROBLEM - PHP opcache health on mwdebug1002 is CRITICAL: CRITICAL: opcache free space is below 50 MB https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_opcache_health
[07:04:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:06:30] <moritzm>	 !log disabling TCP selective acknowledgements on a number of internal test hosts
[07:06:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:18:06] <wikibugs>	 (03PS5) 10Ema: varnishlog: implement varnishfetcherr [puppet] - 10https://gerrit.wikimedia.org/r/517375 (https://phabricator.wikimedia.org/T224994)
[07:18:57] <wikibugs>	 (03CR) 10Ema: [C: 03+2] varnishlog: implement varnishfetcherr [puppet] - 10https://gerrit.wikimedia.org/r/517375 (https://phabricator.wikimedia.org/T224994) (owner: 10Ema)
[07:21:20] <elukey>	 !log upload matomo_3.9.1-3 to stretch-wikimedia and upgrade matomo1001
[07:21:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:26:28] <wikibugs>	 (03PS2) 10Elukey: matomo: make matomo.js/.php public [puppet] - 10https://gerrit.wikimedia.org/r/517564 (https://phabricator.wikimedia.org/T225882) (owner: 10Nuria)
[07:27:45] <wikibugs>	 (03PS3) 10Elukey: matomo: make matomo.js/.php public [puppet] - 10https://gerrit.wikimedia.org/r/517564 (https://phabricator.wikimedia.org/T225882) (owner: 10Nuria)
[07:28:35] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] matomo: make matomo.js/.php public [puppet] - 10https://gerrit.wikimedia.org/r/517564 (https://phabricator.wikimedia.org/T225882) (owner: 10Nuria)
[07:31:01] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+1] "Happy to merge this in my evening/your morning, lmk when" [puppet] - 10https://gerrit.wikimedia.org/r/517124 (owner: 1020after4)
[07:32:16] <elukey>	 !log reboot analytics-tool100* and an-tool100* for kernel upgrades
[07:32:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:36:39] <elukey>	 !log reboot archiva1001 for kernel upgrades
[07:36:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:39:26] <elukey>	 !log reboot matomo1001 for kernel upgrades
[07:39:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:45:06] <elukey>	 !log roll restart of cassandra on aqs* to pick up new openjdk upgrades
[07:45:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:51:18] <wikibugs>	 (03CR) 10Muehlenhoff: Disable TCP selective acknowledgements (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/517461 (owner: 10Muehlenhoff)
[07:53:29] <wikibugs>	 (03CR) 10Ema: [C: 03+1] Disable TCP selective acknowledgements [puppet] - 10https://gerrit.wikimedia.org/r/517461 (owner: 10Muehlenhoff)
[08:04:14] <icinga-wm>	 PROBLEM - Check systemd state on analytics-tool1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[08:06:11] <elukey>	 checking --^
[08:09:03] <jijiki>	 someone with superpowers, I believe I am on duty this week 
[08:11:44] <wikibugs>	 (03PS2) 10Marostegui: install_server: Allow reimage db1112 [puppet] - 10https://gerrit.wikimedia.org/r/517589 (https://phabricator.wikimedia.org/T225981)
[08:12:28] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] install_server: Allow reimage db1112 [puppet] - 10https://gerrit.wikimedia.org/r/517589 (https://phabricator.wikimedia.org/T225981) (owner: 10Marostegui)
[08:16:04] <marostegui>	 !log Remove db2039 from tendril and zarcillo - T225988
[08:16:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:16:10] <stashbot>	 T225988: decommission db2039 - https://phabricator.wikimedia.org/T225988
[08:16:40] <wikibugs>	 (03PS3) 10Muehlenhoff: Disable TCP selective acknowledgements [puppet] - 10https://gerrit.wikimedia.org/r/517461
[08:18:13] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] Disable TCP selective acknowledgements [puppet] - 10https://gerrit.wikimedia.org/r/517461 (owner: 10Muehlenhoff)
[08:19:21] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Disable TCP selective acknowledgements [puppet] - 10https://gerrit.wikimedia.org/r/517461 (owner: 10Muehlenhoff)
[08:20:38] <wikibugs>	 (03PS1) 10Marostegui: mariadb: Set db2039 to spare [puppet] - 10https://gerrit.wikimedia.org/r/517601 (https://phabricator.wikimedia.org/T225988)
[08:22:09] <wikibugs>	 (03PS2) 10Marostegui: mariadb: Set db2039 to spare [puppet] - 10https://gerrit.wikimedia.org/r/517601 (https://phabricator.wikimedia.org/T225988)
[08:23:15] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] mariadb: Set db2039 to spare [puppet] - 10https://gerrit.wikimedia.org/r/517601 (https://phabricator.wikimedia.org/T225988) (owner: 10Marostegui)
[08:23:53] <wikibugs>	 10Operations, 10Operations-Software-Development: cumin could use randomization/splay options - https://phabricator.wikimedia.org/T164587 (10ema) Honorable mention in the ugly-but-useful category, here's a command to get the list of cp-text nodes in decreasing DC order (cp5, cp4, cp3, cp2, cp1):   ` paste -d '\...
[08:23:56] <marostegui>	 !log Stop MySQL on db2039 - T225988
[08:24:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:24:01] <stashbot>	 T225988: decommission db2039 - https://phabricator.wikimedia.org/T225988
[08:24:44] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] termbox: Add kubernetes stanzas [puppet] - 10https://gerrit.wikimedia.org/r/515177 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[08:24:52] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: termbox: Add kubernetes stanzas [puppet] - 10https://gerrit.wikimedia.org/r/515177 (https://phabricator.wikimedia.org/T220402)
[08:25:45] <wikibugs>	 10Operations, 10ops-codfw, 10decommission: decommission db2039 - https://phabricator.wikimedia.org/T225988 (10Marostegui) a:05Marostegui→03RobH
[08:26:16] <wikibugs>	 10Operations, 10ops-codfw, 10DC-Ops, 10decommission: decommission db2039 - https://phabricator.wikimedia.org/T225988 (10Marostegui) This host is ready for #dc-ops to take over and decommission
[08:29:04] <wikibugs>	 (03PS3) 10Marostegui: mariadb: Promote db1081 to s4 master [puppet] - 10https://gerrit.wikimedia.org/r/517361 (https://phabricator.wikimedia.org/T224852)
[08:29:38] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at ulsfo on icinga1001 is CRITICAL: job=varnish-text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:30:10] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:30:10] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at codfw on icinga1001 is CRITICAL: cluster=cache_text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:30:12] <vgutierrez>	 ema: ^^^
[08:30:20] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga1001 is CRITICAL: cluster=cache_text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:30:23] <ema>	 looking
[08:30:42] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga1001 is CRITICAL: cluster=cache_text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:31:38] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5
[08:31:38] <elukey>	 there seems to be a lot of failed fetch for the eqiad text varnish backends afaics
[08:31:40] <icinga-wm>	 PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5
[08:31:46] <elukey>	 https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=now-1h&to=now&var-datasource=eqiad%20prometheus%2Fops&var-cache_type=text&var-server=All&var-layer=backend
[08:31:56] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqiad on icinga1001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:33:06] <elukey>	 ema: the failed fetches looks better now, maybe we're going to recover soon?
[08:33:17] <elukey>	 not sure if you did any magic :)
[08:33:28] <ema>	 elukey: no magic other than staring at 5xx.json
[08:33:34] <ema>	 looks over now yeah
[08:34:00] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:34:02] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at ulsfo on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:34:18] <wikibugs>	 (03PS1) 10Vgutierrez: acme_chief: Enforce staging time validation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/517605 (https://phabricator.wikimedia.org/T225945)
[08:34:26] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:34:48] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:35:16] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at codfw on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:35:16] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:35:33] <moritzm>	 there were alerts for etcd latency for acrux/acrab in Icinga which co-incide with the last puppet run, maybe the TPC sack change reset some long-standing TCP connections used by etcd?
[08:35:50] <akosiaris>	 no no that was me
[08:35:54] <moritzm>	 ok
[08:36:05] <akosiaris>	 well.. me. The termbox kubernetes tokens that is
[08:36:18] <ema>	 all 503 errors above were /w/api.php
[08:37:50] <ema>	 tried to figure it out with kibana, then gave up and went back to the shell on weblog1001
[08:37:56] <ema>	 grep 2019-06-18T08:2  /srv/log/webrequest/5xx.json | jq '.uri_path' | python ~ema/bar_chart.py
[08:37:59] <ema>	 much easier :)
[08:39:54] <ema>	 saved the logs under ~ema/2019-06-18-api.php-errors.log for further investigation
[08:40:12] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5
[08:40:12] <icinga-wm>	 RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5
[08:52:04] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox upgrade --install -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad,codfw]
[08:52:06] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox cluster eqiad completed
[08:52:08] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox cluster codfw completed
[08:52:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:08] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox finished
[08:52:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:35] <akosiaris>	 !log deploy termbox T220402
[08:52:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:52:39] <stashbot>	 T220402: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402
[08:54:11] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox upgrade --install -f termbox-values.yaml production stable/termbox [namespace: termbox, clusters: eqiad,codfw]
[08:54:12] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox cluster eqiad completed
[08:54:13] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox cluster codfw completed
[08:54:14] <logmsgbot>	 !log akosiaris@deploy1001 scap-helm termbox finished
[08:54:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:54:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:54:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:54:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:08:07] <elukey>	 !log reboot analytics-tool1004 a second time to pick up the new kernel upgrades
[09:08:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:08:28] <icinga-wm>	 RECOVERY - Check systemd state on analytics-tool1004 is OK: OK - running: The system is fully operational
[09:08:40] <wikibugs>	 10Operations, 10ops-codfw, 10decommission, 10observability: Decom graphite2001 - https://phabricator.wikimedia.org/T200209 (10MoritzMuehlenhoff) >>! In T200209#4888109, @fgiunchedi wrote: > I'm taking graphite2001 now to do some tests for prometheus v2 upgrade in {T187987}  @fgiunchedi Can you confirm this...
[09:19:48] <logmsgbot>	 !log jmm@cumin2001 START - Cookbook sre.hosts.downtime
[09:19:49] <logmsgbot>	 !log jmm@cumin2001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[09:19:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:19:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:25:01] <wikibugs>	 10Operations, 10ops-eqiad, 10Cassandra, 10Core Platform Team (Security, stability, performance and scalability (TEC1)), and 3 others: Fix restbase1017's physical rack - https://phabricator.wikimedia.org/T222960 (10jijiki) p:05Triage→03Normal
[09:34:56] <wikibugs>	 10Operations, 10WMDE-QWERTY-Team, 10serviceops, 10wikidiff2, and 3 others: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed - https://phabricator.wikimedia.org/T223391 (10Joe) p:05Triage→03High a:03jijiki
[09:36:33] <jijiki>	 !log restarting php-fpm in mwdebug1002
[09:36:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:36:47] <wikibugs>	 10Operations, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10MoritzMuehlenhoff)
[09:38:36] <icinga-wm>	 RECOVERY - PHP opcache health on mwdebug1002 is OK: OK: opcache is healthy https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_opcache_health
[09:39:51] <wikibugs>	 (03PS3) 10Ema: cache: stop passing gethdr_extrachance to varnish [puppet] - 10https://gerrit.wikimedia.org/r/517359 (https://phabricator.wikimedia.org/T224694)
[09:40:06] <wikibugs>	 10Operations, 10WMDE-QWERTY-Team, 10serviceops, 10wikidiff2, and 4 others: Deploy Wikidiff2 version 1.8.2 with the timeout issue fixed - https://phabricator.wikimedia.org/T223391 (10jijiki)
[09:40:48] <wikibugs>	 10Operations, 10Performance-Team, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10Gilles)
[09:41:07] <wikibugs>	 (03CR) 10Ema: [C: 03+2] "pcc looks good https://puppet-compiler.wmflabs.org/compiler1001/16973/" [puppet] - 10https://gerrit.wikimedia.org/r/517359 (https://phabricator.wikimedia.org/T224694) (owner: 10Ema)
[09:42:54] <jbond42>	 !log I will start a rolling reboot of all bastion servers at 10:00UTC
[09:42:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:44:36] <wikibugs>	 (03PS1) 10Fsero: new version [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517610
[09:44:38] <wikibugs>	 (03PS1) 10Fsero: better rules [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517611
[09:45:30] <wikibugs>	 (03CR) 10Fsero: [V: 03+2 C: 03+2] new version [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517610 (owner: 10Fsero)
[09:46:03] <wikibugs>	 (03Abandoned) 10Fsero: new version [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517610 (owner: 10Fsero)
[09:46:20] <wikibugs>	 (03PS2) 10Fsero: better rules [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517611
[09:46:40] <wikibugs>	 (03CR) 10Fsero: [V: 03+2 C: 03+2] better rules [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517611 (owner: 10Fsero)
[09:47:25] <wikibugs>	 10Operations, 10Performance-Team, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10Gilles) loadEventEnd seems to have regressed around the time the change was deployed. In the week-over-week you can see that the curves seem to d...
[09:47:39] <wikibugs>	 (03CR) 10Santhosh: [C: 03+1] Remove ExternalGuidanceEnableContextDetection [mediawiki-config] - 10https://gerrit.wikimedia.org/r/504261 (https://phabricator.wikimedia.org/T219819) (owner: 10KartikMistry)
[09:53:10] <ema>	 !log upgrade varnish packages to 5.1.3-1wm10 on all A:cp (no restarts yet)
[09:53:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:56:30] <wikibugs>	 10Operations, 10DBA: db2084 temporary correctable hardware errors - https://phabricator.wikimedia.org/T225884 (10jijiki) p:05Triage→03Normal @Marostegui are we good to mark this as resolved?
[09:57:07] <wikibugs>	 10Operations, 10DBA: db2084 temporary correctable hardware errors - https://phabricator.wikimedia.org/T225884 (10Marostegui) Not yet, I haven't seen more errors but I want to wait until icinga alert clears up, let's give it another 24h
[09:58:26] <wikibugs>	 10Operations, 10Performance-Team, 10Traffic, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10ema) p:05Triage→03Normal
[10:01:35] <wikibugs>	 10Operations, 10Performance-Team, 10Traffic, 10Performance: Study performance impact of disabling TCP selective acknowledgments - https://phabricator.wikimedia.org/T225998 (10ema) >>! In T225998#5264757, @Gilles wrote: > As long as a patched kernel is on the short horizon, I think we can wait for that befo...
[10:01:39] <moritzm>	 !log upgrading acmechief* to latest Buster
[10:01:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:03:34] <Reedy>	 jouncebot: next
[10:03:35] <jouncebot>	 In 0 hour(s) and 56 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1100)
[10:08:41] <jbond42>	 !log reboot bast5001
[10:08:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:10:22] <icinga-wm>	 PROBLEM - DPKG on iron is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[10:13:34] <jbond42>	 ^me
[10:14:17] <wikibugs>	 10Operations: Disable and re-enable 2FA for user GoranSMilovanovic on Wikitech/Horizon - https://phabricator.wikimedia.org/T226000 (10GoranSMilovanovic)
[10:17:32] <icinga-wm>	 PROBLEM - Prometheus bast5001/ops restarted: beware possible monitoring artifacts on bast5001 is CRITICAL: instance=127.0.0.1:9900 job=prometheus https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqsin+prometheus/ops
[10:17:55] <wikibugs>	 (03PS2) 10Vgutierrez: acme_chief: Enforce staging time validation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/517605 (https://phabricator.wikimedia.org/T225945)
[10:18:08] <wikibugs>	 (03CR) 10Fsero: [V: 03+2 C: 03+2] "works nicely, thanks for the polishing" [debs/helm-diff] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/512323 (owner: 10Hashar)
[10:19:09] <jbond42>	 !log reboot bast4001
[10:19:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:20:24] <icinga-wm>	 PROBLEM - Host bast4002 is DOWN: PING CRITICAL - Packet loss = 100%
[10:21:06] <elukey>	 reboot 4001 or 4002 ? :D
[10:21:25] <jbond42>	 yes sorry 4002 is me, i fogot to downtime and got the wrong number
[10:21:26] <icinga-wm>	 RECOVERY - Host bast4002 is UP: PING OK - Packet loss = 0%, RTA = 74.30 ms
[10:21:38] <ema>	 wow that was a fast reboot
[10:21:42] <icinga-wm>	 PROBLEM - Prometheus prometheus1003/global restarted: beware possible monitoring artifacts on prometheus1003 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=eqsin https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[10:21:56] <icinga-wm>	 PROBLEM - Prometheus prometheus2003/global restarted: beware possible monitoring artifacts on prometheus2003 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=eqsin https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[10:21:58] <icinga-wm>	 PROBLEM - Prometheus prometheus1004/global restarted: beware possible monitoring artifacts on prometheus1004 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=eqsin https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[10:22:44] <icinga-wm>	 PROBLEM - Prometheus prometheus2004/global restarted: beware possible monitoring artifacts on prometheus2004 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=eqsin https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[10:24:33] <jbond42>	 !log reboot iron.wikimedia.org
[10:24:34] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:24:34] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:24:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:24:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:24:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:27:04] <icinga-wm>	 PROBLEM - Prometheus prometheus2004/global restarted: beware possible monitoring artifacts on prometheus2004 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=ulsfo https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[10:27:30] <icinga-wm>	 PROBLEM - Prometheus prometheus1003/global restarted: beware possible monitoring artifacts on prometheus1003 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=ulsfo https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[10:28:19] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:28:20] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:28:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:28:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:28:27] <jbond42>	 !log reboot bast3002
[10:28:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:28:38] <icinga-wm>	 PROBLEM - Varnish traffic drop between 30min ago and now at ulsfo on icinga1001 is CRITICAL: 55.42 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1
[10:30:04] <icinga-wm>	 RECOVERY - Varnish traffic drop between 30min ago and now at ulsfo on icinga1001 is OK: (C)60 le (W)70 le 74.22 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1
[10:30:24] <wikibugs>	 (03PS2) 10Matthias Mullie: Add 'sms' and 'smn' langcodes to commons for use in captions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516760 (https://phabricator.wikimedia.org/T222309) (owner: 10Cparle)
[10:31:07] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:31:07] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:31:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:31:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:31:54] <icinga-wm>	 PROBLEM - Prometheus bast4002/ops restarted: beware possible monitoring artifacts on bast4002 is CRITICAL: instance=127.0.0.1:9900 job=prometheus https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=ulsfo+prometheus/ops
[10:36:12] <icinga-wm>	 PROBLEM - Prometheus prometheus1003/global restarted: beware possible monitoring artifacts on prometheus1003 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=esams https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[10:36:28] <icinga-wm>	 PROBLEM - Prometheus prometheus2003/global restarted: beware possible monitoring artifacts on prometheus2003 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=esams https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[10:36:28] <icinga-wm>	 PROBLEM - Prometheus prometheus1004/global restarted: beware possible monitoring artifacts on prometheus1004 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=esams https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[10:37:14] <icinga-wm>	 PROBLEM - Prometheus prometheus2004/global restarted: beware possible monitoring artifacts on prometheus2004 is CRITICAL: instance=127.0.0.1:9900 job=prometheus site=esams https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[10:38:36] <icinga-wm>	 PROBLEM - Varnish traffic drop between 30min ago and now at esams on icinga1001 is CRITICAL: 48.36 le 60 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1
[10:40:04] <icinga-wm>	 RECOVERY - Varnish traffic drop between 30min ago and now at esams on icinga1001 is OK: (C)60 le (W)70 le 91.09 https://grafana.wikimedia.org/dashboard/db/varnish-http-requests?panelId=6&fullscreen&orgId=1
[10:40:49] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:40:49] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:40:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:40:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:42:14] <icinga-wm>	 PROBLEM - Prometheus bast3002/ops restarted: beware possible monitoring artifacts on bast3002 is CRITICAL: instance=127.0.0.1:9900 job=prometheus https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=esams+prometheus/ops
[10:45:10] <icinga-wm>	 RECOVERY - Prometheus bast5001/ops restarted: beware possible monitoring artifacts on bast5001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqsin+prometheus/ops
[10:46:05] <jbond42>	 !log reboot bast2002
[10:46:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:46:15] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:46:16] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:46:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:46:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:50:54] <icinga-wm>	 PROBLEM - DPKG on iron is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[10:52:37] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:52:38] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:52:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:52:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:53:42] <moritzm>	 !log rebooting pybal-test2001 for some tests with the new 4.9 kernel for jessie
[10:53:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:53:53] <logmsgbot>	 !log jmm@cumin2001 START - Cookbook sre.hosts.downtime
[10:53:54] <logmsgbot>	 !log jmm@cumin2001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:53:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:54:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:55:08] <icinga-wm>	 RECOVERY - Prometheus bast4002/ops restarted: beware possible monitoring artifacts on bast4002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=ulsfo+prometheus/ops
[10:55:32] <wikibugs>	 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to ops group in admin for jeh - https://phabricator.wikimedia.org/T224627 (10jcrespo) 05Resolved→03Open Contact info seems to be wrong,  `Address ***@wikimedia.org does not exist`
[10:56:42] <Urbanecm>	 jouncebot, next
[10:56:42] <jouncebot>	 In 0 hour(s) and 3 minute(s): European Mid-day SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1100)
[10:57:17] <jbond42>	 !log reboot bast1002
[10:57:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:58:35] <logmsgbot>	 !log jbond@cumin1001 START - Cookbook sre.hosts.downtime
[10:58:35] <logmsgbot>	 !log jbond@cumin1001 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0)
[10:58:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:58:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:58:55] <wikibugs>	 (03PS6) 10DCausse: [cirrus] Load cirrus using wfLoadExtension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513068 (https://phabricator.wikimedia.org/T87892)
[10:58:57] <wikibugs>	 (03PS5) 10DCausse: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [1/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513605 (https://phabricator.wikimedia.org/T87892)
[10:58:59] <wikibugs>	 (03PS5) 10DCausse: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [2/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892)
[10:59:01] <wikibugs>	 (03PS6) 10DCausse: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513607 (https://phabricator.wikimedia.org/T87892)
[10:59:03] <wikibugs>	 (03PS3) 10DCausse: [cirrus] remove unused wgCirrusSearchRequestEventSampling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513982
[10:59:06] <wikibugs>	 (03PS4) 10DCausse: [cirrus] Enable UTR30 as a lookup method for ns prefixes on group0 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/512195
[11:00:04] <jouncebot>	 Amir1, Lucas_WMDE, awight, and Urbanecm: I, the Bot under the Fountain, allow thee, The Deployer, to do European Mid-day SWAT(Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1100).
[11:00:04] <jouncebot>	 matthiasmullie and dcausse: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[11:00:19] <icinga-wm>	 ACKNOWLEDGEMENT - MD RAID on bast1002 is CRITICAL: connect to address 208.80.154.86 port 5666: No route to host nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T226007 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering
[11:00:19] <dcausse>	 o/
[11:00:23] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on bast1002 - https://phabricator.wikimedia.org/T226007 (10ops-monitoring-bot)
[11:00:24] <Urbanecm>	 dcausse, I guess you want to SWAT your own patches?
[11:00:42] <dcausse>	 sure
[11:01:21] <awight>	 Urbanecm: I'm happy to do the caption langcode patch, unless you're already on it?
[11:01:32] <Urbanecm>	 awight, no, I'm not :)
[11:01:48] <awight>	 :) okay, I'll deploy and then will ping dcausse 
[11:02:30] <Urbanecm>	 hi matthiasmullie, your patch is about to be deployed by awight. Please be around to test if it works correctly.
[11:03:00] * awight waits courtesy minutes
[11:03:29] <matthiasmullie>	 aight cool
[11:03:38] <Urbanecm>	 matthiasmullie, in case you haven't yet, please do install https://wikitech.wikimedia.org/wiki/X-Wikimedia-Debug#Browser_extensions, you might need them to test the patch at the debug servers.
[11:03:42] <wikibugs>	 (03PS3) 10Alexandros Kosiaris: Introduce termbox LVS configuration [puppet] - 10https://gerrit.wikimedia.org/r/515074 (https://phabricator.wikimedia.org/T220402)
[11:03:52] <wikibugs>	 (03CR) 10Awight: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516760 (https://phabricator.wikimedia.org/T222309) (owner: 10Cparle)
[11:03:58] <awight>	 matthiasmullie: Thanks, starting now.
[11:04:49] <icinga-wm>	 RECOVERY - Prometheus prometheus2004/global restarted: beware possible monitoring artifacts on prometheus2004 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[11:04:56] <wikibugs>	 (03Merged) 10jenkins-bot: Add 'sms' and 'smn' langcodes to commons for use in captions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516760 (https://phabricator.wikimedia.org/T222309) (owner: 10Cparle)
[11:05:15] <icinga-wm>	 RECOVERY - Prometheus prometheus1003/global restarted: beware possible monitoring artifacts on prometheus1003 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[11:05:29] <icinga-wm>	 RECOVERY - Prometheus prometheus2003/global restarted: beware possible monitoring artifacts on prometheus2003 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=codfw+prometheus/global
[11:05:29] <icinga-wm>	 RECOVERY - Prometheus bast3002/ops restarted: beware possible monitoring artifacts on bast3002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=esams+prometheus/ops
[11:05:31] <icinga-wm>	 RECOVERY - Prometheus prometheus1004/global restarted: beware possible monitoring artifacts on prometheus1004 is OK: All metrics within thresholds. https://grafana.wikimedia.org/d/000000271/prometheus-stats?var-datasource=eqiad+prometheus/global
[11:05:50] <wikibugs>	 (03CR) 10Hashar: Polish up Debian packaging (031 comment) [debs/helm-diff] (debian/stretch-wikimedia) - 10https://gerrit.wikimedia.org/r/512323 (owner: 10Hashar)
[11:05:55] <awight>	 matthiasmullie: Please test on mwdebug1002
[11:06:11] <matthiasmullie>	 cormacparle_: can you test?
[11:06:37] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/517475 (owner: 10Mholloway)
[11:06:39] <wikibugs>	 (03CR) 10jenkins-bot: Add 'sms' and 'smn' langcodes to commons for use in captions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516760 (https://phabricator.wikimedia.org/T222309) (owner: 10Cparle)
[11:07:11] * awight wonders why I don't see cormacparle_ in channel people
[11:07:16] <wikibugs>	 (03PS3) 10Alexandros Kosiaris: Assign termbox.svc.{eqiad,codfw}.wmnet LVS IPs [dns] - 10https://gerrit.wikimedia.org/r/515067 (https://phabricator.wikimedia.org/T220402)
[11:07:23] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Assign termbox.svc.{eqiad,codfw}.wmnet LVS IPs [dns] - 10https://gerrit.wikimedia.org/r/515067 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[11:07:32] <Urbanecm>	 awight, probably because he isn't in this channel
[11:08:09] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1002/16974/ says PCC is happy" [puppet] - 10https://gerrit.wikimedia.org/r/515074 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[11:08:37] <matthiasmullie>	 haha
[11:09:10] <wikibugs>	 (03PS1) 10Elukey: Add hive/yarn/oozie/hive users to the puppet catalog [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257)
[11:09:16] <matthiasmullie>	 guess my irc client caches room members...
[11:09:30] <apergos>	 doesn't hang out here, try another one
[11:09:33] <apergos>	 releng for example
[11:09:43] <matthiasmullie>	 already asked him elsewhere
[11:09:51] <Urbanecm>	 apergos, I can't even whois them
[11:09:56] <Urbanecm>	 matthiasmullie, thanks
[11:10:14] <Urbanecm>	 according to my testing, the caption language selector behaves the same way both with and without the patch
[11:10:55] <awight>	 Urbanecm: thank you.  So language code "smn" works without the patch?
[11:11:05] <Urbanecm>	 tried smn
[11:11:14] <matthiasmullie>	 seems to work for me (without: 'sms - Skolt Sami' doesn't show up, but it does with the patch)
[11:11:20] <matthiasmullie>	 so I think it works fine
[11:11:28] <awight>	 okay this is about ULS, https://phabricator.wikimedia.org/T222309
[11:11:37] <awight>	 matthiasmullie: good enough, thank you for testing!
[11:11:47] <matthiasmullie>	 also ok from cormac
[11:11:52] <matthiasmullie>	 go ahead!
[11:11:54] <matthiasmullie>	 thanks!
[11:12:02] <Urbanecm>	 okay, then I'm probably misunderstanding something
[11:12:54] <logmsgbot>	 !log awight@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:516760|Add 'sms' and 'smn' langcodes to commons for use in captions (T222309)]] (duration: 00m 48s)
[11:13:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:13:02] <stashbot>	 T222309: Language code "sms" not recognized in Commons - https://phabricator.wikimedia.org/T222309
[11:13:11] <awight>	 dcausse: Done--all yours now.
[11:13:20] <dcausse>	 awight: thanks!
[11:14:30] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Fix typo in termbox conftool definition [puppet] - 10https://gerrit.wikimedia.org/r/517624
[11:14:42] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Fix typo in termbox conftool definition [puppet] - 10https://gerrit.wikimedia.org/r/517624 (owner: 10Alexandros Kosiaris)
[11:14:45] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1015 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.2.46:3030]) https://wikitech.wikimedia.org/wiki/PyBal
[11:14:56] <wikibugs>	 (03CR) 10DCausse: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513068 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:15:07] <akosiaris>	 !log deploy lvs termbox configuration changes
[11:15:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:15:47] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] Load cirrus using wfLoadExtension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513068 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:15:51] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs1016 is CRITICAL: CRITICAL: 66 connections established with conf1004.eqiad.wmnet:4001 (min=67) https://wikitech.wikimedia.org/wiki/PyBal
[11:16:09] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs1016 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.2.46:3030]) https://wikitech.wikimedia.org/wiki/PyBal
[11:16:19] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] Load cirrus using wfLoadExtension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513068 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:21:56] <logmsgbot>	 !log akosiaris@puppetmaster1001 conftool action : set/pooled=inactive; selector: name=elastic1029.eqiad.wmnet
[11:21:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:22:16] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs2003 is CRITICAL: CRITICAL: 41 connections established with conf2001.codfw.wmnet:2379 (min=42) https://wikitech.wikimedia.org/wiki/PyBal
[11:22:35] <akosiaris>	 !log set elastic1029 as inactive in all conftool data. Command was sudo confctl select "name=elastic1029.eqiad.wmnet" set/pooled=inactive T214283
[11:22:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:22:40] <stashbot>	 T214283: Memory correctable errors -EDAC- elastic1029 - https://phabricator.wikimedia.org/T214283
[11:22:52] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs2003 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.1.46:3030]) https://wikitech.wikimedia.org/wiki/PyBal
[11:24:16] <icinga-wm>	 PROBLEM - PyBal connections to etcd on lvs2006 is CRITICAL: CRITICAL: 41 connections established with conf2001.codfw.wmnet:2379 (min=42) https://wikitech.wikimedia.org/wiki/PyBal
[11:24:37] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/CommonSettings.php: [cirrus] Load cirrus using wfLoadExtension 1/2 (duration: 00m 47s)
[11:24:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:24:52] <icinga-wm>	 PROBLEM - PyBal IPVS diff check on lvs2006 is CRITICAL: CRITICAL: Services known to PyBal but not to IPVS: set([10.2.1.46:3030]) https://wikitech.wikimedia.org/wiki/PyBal
[11:25:52] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/extension-list: [cirrus] Load cirrus using wfLoadExtension 2/2 (duration: 00m 46s)
[11:25:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:26:16] <logmsgbot>	 !log akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: service=termbox
[11:26:17] <wikibugs>	 (03CR) 10DCausse: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513605 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:26:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:26:27] <akosiaris>	 !log pool all hosts for termbox
[11:26:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:27:14] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [1/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513605 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:27:29] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [1/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513605 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:29:28] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs2006 is OK: OK: 42 connections established with conf2001.codfw.wmnet:2379 (min=42) https://wikitech.wikimedia.org/wiki/PyBal
[11:29:40] <wikibugs>	 (03PS1) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[11:29:46] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/CirrusSearch-common.php: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [1/3] (duration: 00m 46s)
[11:29:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:29:55] <wikibugs>	 (03PS8) 10Elukey: Allow Hadoop-related profiles to deploy Kerberos keytabs [puppet] - 10https://gerrit.wikimedia.org/r/515010 (https://phabricator.wikimedia.org/T212257)
[11:30:04] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs2006 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal
[11:30:06] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1015 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal
[11:30:42] <wikibugs>	 (03CR) 10DCausse: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:32:52] <wikibugs>	 (03CR) 10Muehlenhoff: "I think we can do this even simpler, I've just tested this in a WMCS VM; if the kernel does not yet provide tcp_min_snd_mss, then Puppet w" [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[11:33:33] <wikibugs>	 (03PS6) 10DCausse: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [2/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892)
[11:33:56] <wikibugs>	 (03CR) 10DCausse: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:34:00] <icinga-wm>	 PROBLEM - PHP opcache health on mwdebug1001 is CRITICAL: CRITICAL: opcache free space is below 50 MB https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_opcache_health
[11:34:40] <jijiki>	 !log restarting php-fpm on mwdebug1001
[11:34:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:34:51] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [2/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:35:16] <icinga-wm>	 RECOVERY - PHP opcache health on mwdebug1001 is OK: OK: opcache is healthy https://wikitech.wikimedia.org/wiki/Application_servers/Runbook%23PHP7_opcache_health
[11:35:22] <akosiaris>	 jijiki: most mws are in the warning state (<100MB). Probably worth doing for all of them in a slow rolling reboot
[11:35:27] <akosiaris>	 s/reboot/restart/
[11:36:20] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [2/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513606 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:38:58] <akosiaris>	 !log restart pybal on lvs2003
[11:39:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:39:33] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [2/3] (duration: 00m 47s)
[11:39:36] <akosiaris>	 !log restart pybal on lvs1016
[11:39:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:39:40] <Urbanecm>	 dcausse, if you end before end of SWAT, lmk, I have a few patches to deploy
[11:39:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:39:58] <dcausse>	 Urbanecm: sure, a last one and I'm done
[11:40:04] <Urbanecm>	 ok, that's nice :)
[11:40:24] <wikibugs>	 (03PS7) 10DCausse: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513607 (https://phabricator.wikimedia.org/T87892)
[11:41:51] <wikibugs>	 (03CR) 10DCausse: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513607 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:41:56] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs1016 is OK: OK: 67 connections established with conf1004.eqiad.wmnet:4001 (min=67) https://wikitech.wikimedia.org/wiki/PyBal
[11:42:16] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs1016 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal
[11:42:42] <wikibugs>	 (03Merged) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513607 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:42:56] <wikibugs>	 (03CR) 10jenkins-bot: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] [mediawiki-config] - 10https://gerrit.wikimedia.org/r/513607 (https://phabricator.wikimedia.org/T87892) (owner: 10DCausse)
[11:43:22] <icinga-wm>	 RECOVERY - PyBal connections to etcd on lvs2003 is OK: OK: 42 connections established with conf2001.codfw.wmnet:2379 (min=42) https://wikitech.wikimedia.org/wiki/PyBal
[11:45:21] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: termbox: Use HTTP for WMF service checks [puppet] - 10https://gerrit.wikimedia.org/r/517628 (https://phabricator.wikimedia.org/T220402)
[11:46:05] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/CommonSettings.php: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] (1/3) (duration: 00m 47s)
[11:46:06] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] termbox: Use HTTP for WMF service checks [puppet] - 10https://gerrit.wikimedia.org/r/517628 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[11:46:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:47:27] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/CirrusSearch-production.php: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] (2/3) (duration: 00m 47s)
[11:47:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:49:23] <logmsgbot>	 !log akosiaris@puppetmaster1001 conftool action : set/weight=10; selector: service=termbox
[11:49:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:49:43] <logmsgbot>	 !log dcausse@deploy1001 Synchronized wmf-config/CirrusSearch-common.php: [cirrus] drop most wmgCirrusSearch* ephemeral config vars [3/3] (3/3) (duration: 00m 46s)
[11:49:46] <akosiaris>	 !log set all termbox backends with weight 10 (from 0) for consistency's sake
[11:49:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:49:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:49:59] <dcausse>	 Urbanecm: I'm done
[11:50:31] <wikibugs>	 (03CR) 10Muehlenhoff: Add hive/yarn/oozie/hive users to the puppet catalog (033 comments) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[11:50:43] <dcausse>	 sorry it took longer than I expected I had to ship the last one in 3 steps...
[11:51:11] <wikibugs>	 (03PS4) 10Urbanecm: Set two new namespace aliases for es.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516195 (https://phabricator.wikimedia.org/T216143) (owner: 10MarcoAurelio)
[11:51:36] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516195 (https://phabricator.wikimedia.org/T216143) (owner: 10MarcoAurelio)
[11:52:39] <wikibugs>	 (03Merged) 10jenkins-bot: Set two new namespace aliases for es.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516195 (https://phabricator.wikimedia.org/T216143) (owner: 10MarcoAurelio)
[11:52:54] <wikibugs>	 (03CR) 10jenkins-bot: Set two new namespace aliases for es.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/516195 (https://phabricator.wikimedia.org/T216143) (owner: 10MarcoAurelio)
[11:53:03] <Urbanecm>	 dcausse, no problem, have just a few (nothing urgent), and the remaining time should be enough :)
[11:54:03] <wikibugs>	 10Operations, 10DBA: db2084 temporary correctable hardware errors - https://phabricator.wikimedia.org/T225884 (10Marostegui) a:03Marostegui
[11:54:25] <logmsgbot>	 !log urbanecm@deploy1001 Synchronized wmf-config/InitialiseSettings.php: [[:gerrit:516195|Set two new namespace aliases for es.wikibooks]] (T216143) (duration: 00m 47s)
[11:54:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:54:30] <stashbot>	 T216143: Set new namespace aliases for es.wikibooks - https://phabricator.wikimedia.org/T216143
[11:55:39] <Urbanecm>	 !log running namespaceDupes.php for eswikibooks (T216143)
[11:55:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:58:20] <wikibugs>	 (03PS1) 10Urbanecm: Allow sysops to manage flaggedrevs group membership only if the group exists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517629 (https://phabricator.wikimedia.org/T225797)
[11:58:28] <wikibugs>	 10Operations, 10puppet-compiler: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section - https://phabricator.wikimedia.org/T224977 (10Vgutierrez) After checking https://puppet-compiler.wmflabs.org/compiler1001/16855/ change error/warning logs for hosts marked as "fail to c...
[11:58:37] <wikibugs>	 (03CR) 10Urbanecm: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517629 (https://phabricator.wikimedia.org/T225797) (owner: 10Urbanecm)
[11:59:29] <wikibugs>	 (03Merged) 10jenkins-bot: Allow sysops to manage flaggedrevs group membership only if the group exists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517629 (https://phabricator.wikimedia.org/T225797) (owner: 10Urbanecm)
[11:59:44] <wikibugs>	 (03CR) 10jenkins-bot: Allow sysops to manage flaggedrevs group membership only if the group exists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517629 (https://phabricator.wikimedia.org/T225797) (owner: 10Urbanecm)
[12:00:50] <Urbanecm>	 !log EU SWAT is going a few minutes beyond its slot
[12:00:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:00:57] <icinga-wm>	 RECOVERY - PyBal IPVS diff check on lvs2003 is OK: OK: no difference between hosts in IPVS/PyBal https://wikitech.wikimedia.org/wiki/PyBal
[12:03:05] <awight>	 hehe we can just copy and paste that note for these catch-up-after-SRE-summit SWATs
[12:03:09] <logmsgbot>	 !log urbanecm@deploy1001 Synchronized wmf-config/flaggedrevs.php: [[:gerrit:517629|Allow sysops to manage flaggedrevs group membership only if the group exists]] (T225797) (duration: 00m 47s)
[12:03:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:03:14] <stashbot>	 T225797: Remove the "autoreview" and "reviewer"  user groups from ru.wikisource - https://phabricator.wikimedia.org/T225797
[12:03:29] <Urbanecm>	 !log EU SWAT done
[12:03:35] <icinga-wm>	 PROBLEM - puppet last run on icinga1001 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[12:03:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:07:22] <wikibugs>	 10Operations, 10Gerrit, 10cloud-services-team, 10serviceops: Change /r/p/ to /r/ on all hosts (where https://gerrit.wikimedia.org/r/p/ exists) - https://phabricator.wikimedia.org/T222093 (10Paladox) p:05High→03Normal
[12:12:25] <akosiaris>	 !log slowly rolling restart php7 on mw1299-mw1338 to avoid opcache exhaustion
[12:12:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:13:39] <Urbanecm>	 !log Assigned an email address to Eritha@enwiki per user request (T223960)
[12:13:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:13:44] <stashbot>	 T223960: Add email address to specific user account on en.wikipedia - https://phabricator.wikimedia.org/T223960
[12:14:01] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to ops group in admin for jeh - https://phabricator.wikimedia.org/T224627 (10Reedy)
[12:14:19] <icinga-wm>	 RECOVERY - puppet last run on icinga1001 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[12:23:04] <XioNoX>	 !log activate bgp to telia on cr1-codfw - T222967
[12:23:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:23:11] <stashbot>	 T222967: Interface errors on cr1-codfw: xe-5/3/1 - https://phabricator.wikimedia.org/T222967
[12:25:50] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to ops group in admin for jeh - https://phabricator.wikimedia.org/T224627 (10JHedden) I fixed the incorrect email alias, thanks for letting me know
[12:27:34] <ema>	 !log cp5007 (varnish-be text): reboot for kernel and varnish upgrade T224694
[12:27:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:27:49] <stashbot>	 T224694: cp3041 - Varnish frontend child restarted icinga alert - https://phabricator.wikimedia.org/T224694
[12:29:12] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[12:29:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:29:44] <wikibugs>	 10Operations, 10ops-codfw: Interface errors on cr1-codfw: xe-5/3/1 - https://phabricator.wikimedia.org/T222967 (10ayounsi) 05Open→03Resolved > The issue should be resolved as per our second line , and any further optimization on the circuit will be done through maintenance window and of course you will be...
[12:34:29] <wikibugs>	 (03CR) 10Ayounsi: [C: 03+2] Network monitoring, make core router down page [puppet] - 10https://gerrit.wikimedia.org/r/514332 (https://phabricator.wikimedia.org/T224535) (owner: 10Ayounsi)
[12:34:46] <wikibugs>	 (03PS3) 10Ayounsi: Network monitoring, make core router down page [puppet] - 10https://gerrit.wikimedia.org/r/514332 (https://phabricator.wikimedia.org/T224535)
[12:37:49] <XioNoX>	 !log merge puppet change to make all router down alerts paging - T224535
[12:37:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:37:54] <stashbot>	 T224535: Investigate cr2-eqord's disconnection from the rest of the network - https://phabricator.wikimedia.org/T224535
[12:38:07] <icinga-wm>	 RECOVERY - proton endpoints health on proton1001 is OK: All endpoints are healthy https://wikitech.wikimedia.org/wiki/Services/Monitoring/proton
[12:38:20] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[12:38:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:42:30] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "Good job! See comments inline." (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/516992 (https://phabricator.wikimedia.org/T224073) (owner: 10Mathew.onipe)
[12:45:03] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to ops group in admin for jeh - https://phabricator.wikimedia.org/T224627 (10jcrespo) 05Open→03Resolved This is resolved, then?
[12:48:01] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "a few minor comments inline." (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/511819 (https://phabricator.wikimedia.org/T224072) (owner: 10Mathew.onipe)
[12:48:42] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] add WDQS reboot cookbook [cookbooks] - 10https://gerrit.wikimedia.org/r/512915 (https://phabricator.wikimedia.org/T224385) (owner: 10Mathew.onipe)
[12:49:09] <ema>	 !log cp3034 (ats-be upload) cp2002 (varnish-be upload): reboot for kernel and varnish upgrade T224694
[12:49:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:49:14] <stashbot>	 T224694: cp3041 - Varnish frontend child restarted icinga alert - https://phabricator.wikimedia.org/T224694
[12:49:40] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[12:49:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:52:05] <wikibugs>	 10Operations, 10SRE-Access-Requests: Requesting access to ops group in admin for jeh - https://phabricator.wikimedia.org/T224627 (10JHedden) Yes, this is resolved. The invalid `jeh` address defined in the exim alias was fixed.
[12:53:13] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[12:53:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:55:57] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[12:56:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:59:45] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[12:59:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:07:59] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: gmail users being suspended from mediawiki-l due to excessive bounces - https://phabricator.wikimedia.org/T225553 (10Nikerabbit) I got such an email again. Is this happening weekly now?
[13:10:04] <ema>	 !log cache nodes: begin rolling reboots for kernel and varnish upgrades T224694
[13:10:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:10:09] <stashbot>	 T224694: cp3041 - Varnish frontend child restarted icinga alert - https://phabricator.wikimedia.org/T224694
[13:10:45] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[13:10:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:12:58] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[13:13:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:13:36] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: gmail users being suspended from mediawiki-l due to excessive bounces - https://phabricator.wikimedia.org/T225553 (10Paladox) I got this too on yahoo.
[13:17:06] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[13:17:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:19:39] <wikibugs>	 (03CR) 10Vgutierrez: LVS for cloudelastic (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/512925 (https://phabricator.wikimedia.org/T224324) (owner: 10EBernhardson)
[13:22:49] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[13:22:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:25:50] <wikibugs>	 (03CR) 10Elukey: Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[13:30:41] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on helium - https://phabricator.wikimedia.org/T224794 (10jijiki) p:05Triage→03Normal That looks about right  ` Physical Disk: 7 Enclosure Device ID: 15 Slot Number: 7 Drive's position: DiskGroup: 0, Span: 0, Arm: 7 Enclosure position: N/A Device Id: 8 WWN: 5000C5006...
[13:31:35] <XioNoX>	 !log push new syslog target to mr* - T224128
[13:31:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:31:40] <stashbot>	 T224128: Migrate network device syslogs to Kafka logging pipeline - https://phabricator.wikimedia.org/T224128
[13:33:17] <wikibugs>	 (03CR) 10Ottomata: Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[13:36:08] <wikibugs>	 (03PS1) 10Ottomata: Produce page-properties-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517643 (https://phabricator.wikimedia.org/T211248)
[13:37:07] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[13:37:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:40:02] <wikibugs>	 (03CR) 10Elukey: Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[13:40:35] <wikibugs>	 (03PS2) 10Elukey: Add hive/yarn/oozie/hive users to the puppet catalog [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257)
[13:42:08] <XioNoX>	 !log push new syslog target to msw* - T224128
[13:42:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:42:13] <stashbot>	 T224128: Migrate network device syslogs to Kafka logging pipeline - https://phabricator.wikimedia.org/T224128
[13:42:50] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[13:42:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:44:00] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[13:44:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:45:13] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Produce page-properties-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517643 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[13:45:49] <wikibugs>	 (03PS2) 10Alexandros Kosiaris: sessionstore: Enable LVS paging [puppet] - 10https://gerrit.wikimedia.org/r/514028 (https://phabricator.wikimedia.org/T220401)
[13:46:09] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] sessionstore: Enable LVS paging [puppet] - 10https://gerrit.wikimedia.org/r/514028 (https://phabricator.wikimedia.org/T220401) (owner: 10Alexandros Kosiaris)
[13:46:17] <wikibugs>	 (03CR) 10jenkins-bot: Produce page-properties-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517643 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[13:46:47] <ottomata>	 !log deploying mediawiki-config to produce page-properties-change events to eventgate-main
[13:46:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:46:55] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: LGBT mailing list moderator password reset - https://phabricator.wikimedia.org/T225787 (10jijiki) p:05Triage→03Normal @Ladsgroup Should we mark this as resolved?
[13:47:01] <wikibugs>	 (03PS3) 10Alexandros Kosiaris: Enable discovery for termbox [dns] - 10https://gerrit.wikimedia.org/r/515069 (https://phabricator.wikimedia.org/T220402)
[13:47:20] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Enable discovery for termbox [dns] - 10https://gerrit.wikimedia.org/r/515069 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[13:48:14] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Produce page-properties-change to eventgate-main - T211248 (duration: 00m 48s)
[13:48:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:48:19] <stashbot>	 T211248: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248
[13:48:36] <wikibugs>	 10Operations, 10ops-eqiad: kafka1023 correctable memory errors - https://phabricator.wikimedia.org/T194249 (10Marostegui) 05Open→03Resolved This recovered itself - no more issues since 13th May
[13:48:53] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics: Degraded RAID on analytics1032 - https://phabricator.wikimedia.org/T225864 (10jijiki)
[13:49:02] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics: Degraded RAID on analytics1032 - https://phabricator.wikimedia.org/T225864 (10jijiki) ` DISK GROUP: 2 Number of Spans: 1 SPAN: 0 Span Reference: 0x02 Number of PDs: 1 Number of VDs: 1 Number of dedicated Hotspares: 0 Virtual Drive Information: Virtual Drive: 2 (Target...
[13:49:17] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[13:49:20] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics: Degraded RAID on analytics1032 - https://phabricator.wikimedia.org/T225864 (10jijiki) p:05Triage→03High
[13:49:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:50:46] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics: Degraded RAID on analytics1032 - https://phabricator.wikimedia.org/T225864 (10elukey) Lovely. This is the testing cluster, all OOW nodes, so there is not much that we can do.. I'll add puppet code to force hdfs/yarn to not use this disk, then we should be ok!
[13:51:36] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: LGBT mailing list moderator password reset - https://phabricator.wikimedia.org/T225787 (10Ladsgroup) >>! In T225787#5265538, @jijiki wrote: > @Ladsgroup Should we mark this as resolved?  I can't reach her to ask for confirmation if it's received. I will give it another...
[13:51:41] <wikibugs>	 10Operations, 10ops-eqiad, 10Analytics: Degraded RAID on analytics1032 - https://phabricator.wikimedia.org/T225864 (10jijiki) p:05High→03Normal
[13:52:15] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Enable discovery for termbox [dns] - 10https://gerrit.wikimedia.org/r/515069 (https://phabricator.wikimedia.org/T220402) (owner: 10Alexandros Kosiaris)
[13:53:03] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: LGBT mailing list moderator password reset - https://phabricator.wikimedia.org/T225787 (10jijiki) @Ladsgroup I think this can be added by the list admin themselves, ping me if this is not possible or if there is something else we can do
[13:55:05] <wikibugs>	 10Operations, 10puppet-compiler: puppet-catalog-compiler: compilation result randomly places servers in the 'failed' section - https://phabricator.wikimedia.org/T224977 (10jijiki) p:05Triage→03Normal
[13:56:01] <Amir1>	 !log ladsgroup@mwmaint1002:~$ mwscript sql.php --wiki=testwikidatawiki /srv/mediawiki/php-1.34.0-wmf.8/extensions/Wikibase/repo/sql/AddNormalizedTermsTablesDDL.sql (T225039)
[13:56:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:56:06] <stashbot>	 T225039: Add normalized tables to production - https://phabricator.wikimedia.org/T225039
[13:56:21] <Amir1>	 !log ladsgroup@mwmaint1002:~$ mwscript sql.php --wiki=wikidatawiki /srv/mediawiki/php-1.34.0-wmf.8/extensions/Wikibase/repo/sql/AddNormalizedTermsTablesDDL.sql (T225039)
[13:56:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:56:43] <wikibugs>	 10Operations, 10cloud-services-team: Disable and re-enable 2FA for user GoranSMilovanovic on Wikitech/Horizon - https://phabricator.wikimedia.org/T226000 (10jijiki) p:05Triage→03Normal
[13:56:49] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Fix termbox discovery RRs [dns] - 10https://gerrit.wikimedia.org/r/517647
[13:57:23] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Fix termbox discovery RRs [dns] - 10https://gerrit.wikimedia.org/r/517647 (owner: 10Alexandros Kosiaris)
[13:57:30] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+1] "This can move forward now \o/" [puppet] - 10https://gerrit.wikimedia.org/r/514411 (https://phabricator.wikimedia.org/T225038) (owner: 10Alaa Sarhan)
[13:58:22] <wikibugs>	 (03PS1) 10Ottomata: Produce page-links-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248)
[13:59:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Produce page-links-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[13:59:27] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] "One nit q but +1" (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[14:00:19] <wikibugs>	 (03CR) 10Ottomata: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:01:00] <wikibugs>	 (03PS1) 10Ema: varnishfetcherr: do send logs, handle interesting tags [puppet] - 10https://gerrit.wikimedia.org/r/517649 (https://phabricator.wikimedia.org/T224994)
[14:02:34] <wikibugs>	 (03PS1) 10Alexandros Kosiaris: Fix termbox discovery LVS reference [puppet] - 10https://gerrit.wikimedia.org/r/517650
[14:03:07] <wikibugs>	 (03CR) 10Elukey: Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[14:03:32] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[14:03:43] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Produce page-links-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:04:00] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[14:04:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:04:05] <ottomata>	 !log  deploying mediawiki-config to Produce page-linkT211248s-change stream to eventgate-main - T211248
[14:04:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:04:10] <stashbot>	 T211248: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248
[14:04:41] <wikibugs>	 (03Merged) 10jenkins-bot: Produce page-links-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:04:45] <icinga-wm>	 PROBLEM - Apache HTTP on mw1290 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1309 bytes in 0.003 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[14:05:45] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Produce page-links-change to eventgate-main - T211248 (duration: 00m 48s)
[14:05:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:06:06] <wikibugs>	 (03CR) 10Vgutierrez: varnishfetcherr: do send logs, handle interesting tags (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/517649 (https://phabricator.wikimedia.org/T224994) (owner: 10Ema)
[14:06:13] <icinga-wm>	 RECOVERY - Apache HTTP on mw1290 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 616 bytes in 0.044 second response time https://wikitech.wikimedia.org/wiki/Application_servers
[14:06:29] <wikibugs>	 (03CR) 10jenkins-bot: Produce page-links-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517648 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:06:37] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+2] Fix termbox discovery LVS reference [puppet] - 10https://gerrit.wikimedia.org/r/517650 (owner: 10Alexandros Kosiaris)
[14:07:25] <wikibugs>	 (03PS1) 10Andrew Bogott: no-op patch for catalog testing [puppet] - 10https://gerrit.wikimedia.org/r/517651
[14:09:15] <wikibugs>	 (03CR) 10Muehlenhoff: Add hive/yarn/oozie/hive users to the puppet catalog (031 comment) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[14:09:17] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[14:09:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:09:25] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 04-2] no-op patch for catalog testing [puppet] - 10https://gerrit.wikimedia.org/r/517651 (owner: 10Andrew Bogott)
[14:09:47] <wikibugs>	 (03PS1) 10Ottomata: Produce page-restrictions-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517652 (https://phabricator.wikimedia.org/T211248)
[14:09:57] <wikibugs>	 (03CR) 10Elukey: [V: 03+2 C: 03+2] "I'll go for doing exactly what the deb does as first pass, to avoid any weirdness :)" [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517623 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[14:10:30] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[14:10:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:10:54] <wikibugs>	 (03PS2) 10Ema: varnishfetcherr: do send logs, handle interesting tags [puppet] - 10https://gerrit.wikimedia.org/r/517649 (https://phabricator.wikimedia.org/T224994)
[14:11:11] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Produce page-restrictions-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517652 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:11:30] <wikibugs>	 (03CR) 10jenkins-bot: Produce page-restrictions-change stream to eventgate-main [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517652 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:12:30] <wikibugs>	 (03PS1) 10Elukey: Update CDH module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517654
[14:12:50] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Produce page-restrictions-change to eventgate-main - T211248 (duration: 00m 47s)
[14:12:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:12:56] <stashbot>	 T211248: Modern Event Platform: Stream Intake Service: Migrate Mediawiki Eventbus events to eventgate-main - https://phabricator.wikimedia.org/T211248
[14:15:09] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on bast1002 - https://phabricator.wikimedia.org/T226007 (10jbond) This is a false positive cause by a reboot of bast1002
[14:15:31] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[14:15:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:15:59] <wikibugs>	 (03PS2) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[14:16:01] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Update CDH module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517654 (owner: 10Elukey)
[14:16:27] <logmsgbot>	 !log akosiaris@puppetmaster1001 conftool action : set/pooled=true; selector: dnsdisc=sessionstore
[14:16:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:16:33] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on bast1002 - https://phabricator.wikimedia.org/T226007 (10jijiki) 05Open→03Invalid ` 10:57 jbond42: reboot bast1002 `  Scheduled rolling bastion reboots
[14:16:36] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[14:16:36] <logmsgbot>	 !log akosiaris@puppetmaster1001 conftool action : set/pooled=true; selector: dnsdisc=termbox
[14:16:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:19:16] <wikibugs>	 (03CR) 10Ema: [C: 03+2] varnishfetcherr: do send logs, handle interesting tags [puppet] - 10https://gerrit.wikimedia.org/r/517649 (https://phabricator.wikimedia.org/T224994) (owner: 10Ema)
[14:19:26] <wikibugs>	 (03PS3) 10Ema: varnishfetcherr: do send logs, handle interesting tags [puppet] - 10https://gerrit.wikimedia.org/r/517649 (https://phabricator.wikimedia.org/T224994)
[14:21:52] <wikibugs>	 (03PS1) 10Vgutierrez: acme_chief: Make non_canonical_certificates reachable to other puppet roles [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548)
[14:28:43] <wikibugs>	 (03PS1) 10Jbond: VPS resolv.conf: fiox spec tests [puppet] - 10https://gerrit.wikimedia.org/r/517663 (https://phabricator.wikimedia.org/T224828)
[14:28:55] <wikibugs>	 (03PS1) 10Ottomata: Move newly migrated eventgate-main tables to new Refine job [puppet] - 10https://gerrit.wikimedia.org/r/517664 (https://phabricator.wikimedia.org/T211248)
[14:29:35] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] VPS resolv.conf: fiox spec tests [puppet] - 10https://gerrit.wikimedia.org/r/517663 (https://phabricator.wikimedia.org/T224828) (owner: 10Jbond)
[14:30:30] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[14:30:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:32:23] <wikibugs>	 (03PS3) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[14:32:30] <wikibugs>	 (03PS1) 10Elukey: hive: correct user comment (s/daemon/User) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517665
[14:32:48] <wikibugs>	 10Operations, 10Release Pipeline, 10Release-Engineering-Team, 10serviceops, and 2 others: TEC3:O3:O3.1:Q4 Goal - Move cpjobqueue, Wikidata Termbox SSR (new service), Kask (session storage service) and ORES (partially) through the production CD Pipeline - https://phabricator.wikimedia.org/T220398 (10akosiari...
[14:32:57] <wikibugs>	 10Operations, 10Release Pipeline, 10Release-Engineering-Team, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10akosiaris) 05Open→03Resolved a:03akosiaris ` curl -s -I -X GET 'http://termbox.discovery.wmnet:3030/termbox?editLink=%2Fe...
[14:33:00] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] hive: correct user comment (s/daemon/User) [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517665 (owner: 10Elukey)
[14:33:38] <wikibugs>	 10Operations, 10Traffic, 10observability, 10Patch-For-Review: varnish: implement FetchError logging - https://phabricator.wikimedia.org/T224994 (10ema) 05Open→03Resolved a:03ema Done: https://logstash.wikimedia.org/goto/86af68c6761c458d70d693ac32ed1931
[14:34:02] <wikibugs>	 (03PS2) 10Ottomata: Move newly migrated eventgate-main tables to new Refine job [puppet] - 10https://gerrit.wikimedia.org/r/517664 (https://phabricator.wikimedia.org/T211248)
[14:35:20] <wikibugs>	 (03PS1) 10Elukey: Update the cdh module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517666
[14:35:31] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[14:35:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:35:47] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Update the cdh module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517666 (owner: 10Elukey)
[14:36:12] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] "https://puppet-compiler.wmflabs.org/compiler1001/16985/an-coord1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/517664 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:36:26] <wikibugs>	 (03PS3) 10Ottomata: Move newly migrated eventgate-main tables to new Refine job [puppet] - 10https://gerrit.wikimedia.org/r/517664 (https://phabricator.wikimedia.org/T211248)
[14:36:28] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] Move newly migrated eventgate-main tables to new Refine job [puppet] - 10https://gerrit.wikimedia.org/r/517664 (https://phabricator.wikimedia.org/T211248) (owner: 10Ottomata)
[14:37:53] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[14:37:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:40:28] <wikibugs>	 10Operations, 10decommission, 10media-storage, 10User-fgiunchedi: Decom ms-be101[345] - https://phabricator.wikimedia.org/T220590 (10MoritzMuehlenhoff) Can we please move forward with the decom steps for at least 1013? This host is down due to hardware trouble for nearly two months( T220907) and always sho...
[14:41:36] <wikibugs>	 (03PS2) 10Vgutierrez: acme_chief: Make non_canonical_certificates reachable to other puppet roles [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548)
[14:43:12] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[14:43:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:44:35] <wikibugs>	 (03PS1) 10Elukey: hadoop: add the mapred user to the puppet catalog [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517667
[14:45:16] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] hadoop: add the mapred user to the puppet catalog [puppet/cdh] - 10https://gerrit.wikimedia.org/r/517667 (owner: 10Elukey)
[14:46:34] <wikibugs>	 (03PS1) 10Elukey: Update cdh module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517668
[14:47:33] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Update cdh module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517668 (owner: 10Elukey)
[14:47:42] <wikibugs>	 (03PS2) 10Elukey: Update cdh module to its latest version [puppet] - 10https://gerrit.wikimedia.org/r/517668
[14:57:53] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[14:57:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:03:13] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[15:03:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:05:44] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Looks good and tested successfully on a labs instance. Two comments inline." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[15:06:52] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[15:06:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:08:22] <wikibugs>	 (03PS1) 10Ladsgroup: Set EntityUsageTable addUsage batch size to 150 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517669 (https://phabricator.wikimedia.org/T225500)
[15:10:27] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[15:10:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:10:38] <wikibugs>	 (03PS1) 10ArielGlenn: refactor wikidata entity dumps into wikibase + wikidata specific bits [puppet] - 10https://gerrit.wikimedia.org/r/517670 (https://phabricator.wikimedia.org/T221917)
[15:11:01] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] refactor wikidata entity dumps into wikibase + wikidata specific bits [puppet] - 10https://gerrit.wikimedia.org/r/517670 (https://phabricator.wikimedia.org/T221917) (owner: 10ArielGlenn)
[15:13:45] <wikibugs>	 (03PS4) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[15:13:56] <wikibugs>	 (03PS3) 10Vgutierrez: acme_chief: Introduce the concept of shared certificates [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548)
[15:15:24] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme_chief: Introduce the concept of shared certificates [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548) (owner: 10Vgutierrez)
[15:15:38] <vgutierrez>	 of course, I already missed those V:-1
[15:15:47] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[15:15:49] <wikibugs>	 (03CR) 10Jbond: cap mss: to 538 (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[15:16:44] <wikibugs>	 (03PS4) 10Vgutierrez: acme_chief: Introduce the concept of shared certificates [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548)
[15:18:41] <wikibugs>	 (03PS5) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[15:21:24] <wikibugs>	 (03PS5) 10Vgutierrez: acme_chief: Introduce the concept of shared certificates [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548)
[15:22:05] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626 (owner: 10Jbond)
[15:25:04] <wikibugs>	 (03PS1) 10Alaa Sarhan: Switch property terms migration to WRITE_BOTH on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517673 (https://phabricator.wikimedia.org/T225051)
[15:25:26] <wikibugs>	 (03CR) 10Vgutierrez: "PCC shows the expected changes in the profile parameters and that the generated /etc/acme-chief/config.yaml is the same for production && " [puppet] - 10https://gerrit.wikimedia.org/r/517660 (https://phabricator.wikimedia.org/T133548) (owner: 10Vgutierrez)
[15:26:07] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Switch property terms migration to WRITE_BOTH on test wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517673 (https://phabricator.wikimedia.org/T225051) (owner: 10Alaa Sarhan)
[15:26:29] <wikibugs>	 (03PS1) 10Alaa Sarhan: Switch property terms migration to WRITE_BOTH on wikidata production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517674 (https://phabricator.wikimedia.org/T225051)
[15:26:53] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[15:26:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:26:59] <icinga-wm>	 PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[15:27:37] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Switch property terms migration to WRITE_BOTH on wikidata production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517674 (https://phabricator.wikimedia.org/T225051) (owner: 10Alaa Sarhan)
[15:27:39] <icinga-wm>	 PROBLEM - Router interfaces on cr2-esams is CRITICAL: CRITICAL: host 91.198.174.244, interfaces up: 71, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[15:29:05] <ema>	 XioNoX: ^
[15:30:14] <wikibugs>	 (03PS4) 10Paladox: Update plugins to 2.15.13 [software/gerrit] (wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/507521
[15:30:27] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[15:30:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:31:34] <wikibugs>	 (03PS1) 10Ottomata: Make eventlogging_eventbus Refine job use more explicit whitelist regex [puppet] - 10https://gerrit.wikimedia.org/r/517675
[15:32:17] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Make eventlogging_eventbus Refine job use more explicit whitelist regex [puppet] - 10https://gerrit.wikimedia.org/r/517675 (owner: 10Ottomata)
[15:33:24] <wikibugs>	 (03PS9) 10Elukey: Allow Hadoop-related profiles to deploy Kerberos keytabs [puppet] - 10https://gerrit.wikimedia.org/r/515010 (https://phabricator.wikimedia.org/T212257)
[15:34:09] <wikibugs>	 (03PS2) 10Ottomata: Make eventlogging_eventbus Refine job use more explicit whitelist regex [puppet] - 10https://gerrit.wikimedia.org/r/517675
[15:35:22] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[15:35:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:23] <XioNoX>	 ema: hello!
[15:39:55] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[15:39:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:40:40] <ema>	 XioNoX: hi! sorry to bother you, google calendar being down I wasn't sure how top proceed
[15:40:45] <ema>	 s/top/to/
[15:42:15] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Make eventlogging_eventbus Refine job use more explicit whitelist regex [puppet] - 10https://gerrit.wikimedia.org/r/517675 (owner: 10Ottomata)
[15:42:17] <XioNoX>	 ema, https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down :)
[15:42:55] <XioNoX>	 no emergency though
[15:43:21] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 04-1] "Nice!!! Some minor nitpicks from me, overall the premise and implementation seem fine." (036 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/517573 (https://phabricator.wikimedia.org/T215319) (owner: 10Thcipriani)
[15:44:07] <wikibugs>	 (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/compiler1002/16990/" [puppet] - 10https://gerrit.wikimedia.org/r/515010 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[15:44:23] <ema>	 XioNoX: yes, I was following that very document and got stuck at the maintenance calendar phase :P
[15:44:40] <wikibugs>	 (03CR) 10Alexandros Kosiaris: [C: 03+1] better rules [debs/helmfile] (debian/buster-wikimedia) - 10https://gerrit.wikimedia.org/r/517611 (owner: 10Fsero)
[15:45:32] <XioNoX>	 haha great! thanks for letting me know!
[15:48:01] <wikibugs>	 (03PS6) 10Jbond: cap mss: to 538 [puppet] - 10https://gerrit.wikimedia.org/r/517626
[15:48:37] <icinga-wm>	 RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 231, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[15:49:21] <icinga-wm>	 RECOVERY - Router interfaces on cr2-esams is OK: OK: host 91.198.174.244, interfaces up: 73, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[15:49:44] <XioNoX>	 yay
[15:54:17] <icinga-wm>	 PROBLEM - puppet last run on mw1341 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[15:54:21] <icinga-wm>	 PROBLEM - puppet last run on db1134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[15:55:03] <wikibugs>	 (03PS1) 10Ottomata: eventgate 0.0.6 - set max_body_size in values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/517682 (https://phabricator.wikimedia.org/T226026)
[15:55:23] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[15:55:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:55:45] <icinga-wm>	 PROBLEM - puppet last run on graphite2003 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[15:55:47] <icinga-wm>	 PROBLEM - puppet last run on ms-be2026 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. Could be an interrupted request or a dependency cycle.
[15:56:26] <jbond42>	 ^^looking^^ not sure what theses faliure are but manule runs are working fine
[15:56:34] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2] eventgate 0.0.6 - set max_body_size in values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/517682 (https://phabricator.wikimedia.org/T226026) (owner: 10Ottomata)
[15:56:40] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] eventgate 0.0.6 - set max_body_size in values.yaml [deployment-charts] - 10https://gerrit.wikimedia.org/r/517682 (https://phabricator.wikimedia.org/T226026) (owner: 10Ottomata)
[15:58:39] <logmsgbot>	 !log bmansurov@deploy1001 Started deploy [recommendation-api/deploy@09404fb]: Update the recommendation API service
[15:58:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:59:13] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+1] cloudstore: move secondary monitoring stuff into profile and fix it [puppet] - 10https://gerrit.wikimedia.org/r/517470 (https://phabricator.wikimedia.org/T225265) (owner: 10Bstorm)
[15:59:43] <icinga-wm>	 RECOVERY - puppet last run on mw1341 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[15:59:47] <icinga-wm>	 RECOVERY - puppet last run on db1134 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[15:59:56] <logmsgbot>	 !log ema@cumin1001 START - Cookbook sre.hosts.upgrade-and-reboot
[16:00:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:00:04] <jouncebot>	 godog and _joe_: (Dis)respected human, time to deploy Puppet SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1600). Please do the needful.
[16:00:04] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[16:01:13] <icinga-wm>	 RECOVERY - puppet last run on graphite2003 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures
[16:01:13] <icinga-wm>	 RECOVERY - puppet last run on ms-be2026 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[16:01:49] <logmsgbot>	 !log bmansurov@deploy1001 Finished deploy [recommendation-api/deploy@09404fb]: Update the recommendation API service (duration: 03m 09s)
[16:01:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:01:55] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[16:01:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:06:24] <logmsgbot>	 !log ema@cumin1001 END (PASS) - Cookbook sre.hosts.upgrade-and-reboot (exit_code=0)
[16:06:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:07:38] <ema>	 !log cache nodes: stop rolling reboots for today, 17/80 done T224694
[16:07:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:07:43] <stashbot>	 T224694: cp3041 - Varnish frontend child restarted icinga alert - https://phabricator.wikimedia.org/T224694
[16:09:04] <wikibugs>	 (03CR) 10Krinkle: "See above questions in my last comment." [puppet] - 10https://gerrit.wikimedia.org/r/511078 (https://phabricator.wikimedia.org/T113114) (owner: 10Ladsgroup)
[16:13:30] <wikibugs>	 (03PS1) 10Ottomata: eventgate 0.0.7 - set max_body_size in proper location in app.conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/517687 (https://phabricator.wikimedia.org/T226026)
[16:14:09] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] eventgate 0.0.7 - set max_body_size in proper location in app.conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/517687 (https://phabricator.wikimedia.org/T226026) (owner: 10Ottomata)
[16:14:11] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] eventgate 0.0.7 - set max_body_size in proper location in app.conf [deployment-charts] - 10https://gerrit.wikimedia.org/r/517687 (https://phabricator.wikimedia.org/T226026) (owner: 10Ottomata)
[16:14:41] <icinga-wm>	 PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[16:15:25] <icinga-wm>	 PROBLEM - Router interfaces on cr2-esams is CRITICAL: CRITICAL: host 91.198.174.244, interfaces up: 71, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[16:15:39] <icinga-wm>	 PROBLEM - Esams HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=esams&var-cache_type=All&var-status_type=5
[16:16:53] <icinga-wm>	 PROBLEM - Text HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5
[16:18:59] <bblack>	 with the gcal outage I'm not sure if that's part of something planned
[16:19:17] <bblack>	 we do have two links out to esams, so should be a small blip to lose one, in theory.
[16:19:46] <jynus>	 bblack: on my cache I don't see any ongoing maintenance
[16:20:25] <jynus>	 but of course I cannot assure the accuracy of that, with cthe latest status and with reality
[16:21:04] <bblack>	 in any case, yes, other stats seem to say it was a very short spike of 503 (which we have to expect with a network link failover like this)
[16:21:11] <icinga-wm>	 RECOVERY - Text HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5
[16:21:15] <bblack>	 looks like there was one earlier in the day too
[16:21:25] <icinga-wm>	 RECOVERY - Esams HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=esams&var-cache_type=All&var-status_type=5
[16:21:29] <bblack>	 near ~08:30 UTC
[16:22:02] <bblack>	 ah that wasn't esams-specific and was something else
[16:23:08] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: staging]
[16:23:09] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main cluster staging completed
[16:23:10] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main finished
[16:23:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:23:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:23:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:25:47] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/codfw-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: codfw]
[16:25:49] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main cluster codfw completed
[16:25:49] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main finished
[16:25:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:25:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:25:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:26:57] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main upgrade main -f /srv/scap-helm/eventgate/main/eqiad-values.yaml --reset-values stable/eventgate [namespace: eventgate-main, clusters: eqiad]
[16:26:58] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main cluster eqiad completed
[16:26:58] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-main finished
[16:27:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:27:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:27:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:40] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade analytics -f /srv/scap-helm/eventgate/analytics/staging-values.yaml --reset-values stable/eventgate [namespace: eventgate-analytics, clusters: staging]
[16:32:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:45] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed
[16:32:45] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[16:32:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:32:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:34:57] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade analytics -f /srv/scap-helm/eventgate/analytics/codfw-values.yaml --reset-values stable/eventgate [namespace: eventgate-analytics, clusters: codfw]
[16:35:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:35:03] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster codfw completed
[16:35:03] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[16:35:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:35:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:38:42] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade analytics -f /srv/scap-helm/eventgate/analytics/eqiad-values.yaml --reset-values stable/eventgate [namespace: eventgate-analytics, clusters: eqiad]
[16:38:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:38:48] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster eqiad completed
[16:38:48] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[16:38:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:38:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:43:23] <wikibugs>	 (03CR) 10Jeena Huneidi: "This change is ready for review." [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935) (owner: 10Jeena Huneidi)
[16:43:48] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] Allow Hadoop-related profiles to deploy Kerberos keytabs [puppet] - 10https://gerrit.wikimedia.org/r/515010 (https://phabricator.wikimedia.org/T212257) (owner: 10Elukey)
[16:55:57] <wikibugs>	 (03CR) 10Fsero: "some nits" (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935) (owner: 10Jeena Huneidi)
[17:00:04] <jouncebot>	 cscott, arlolra, subbu, and halfak: Your horoscope predicts another unfortunate Services – Graphoid / Parsoid / Citoid / ORES deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1700).
[17:00:28] <subbu>	 no parsoid deploy today
[17:07:23] <wikibugs>	 (03PS1) 10ArielGlenn: testing paging settings for labstore1006 [puppet] - 10https://gerrit.wikimedia.org/r/517693
[17:10:48] <wikibugs>	 (03PS1) 10Jbond: ipmi - pxe: Ensure ipmi is not overriding the boot order [puppet] - 10https://gerrit.wikimedia.org/r/517694
[17:11:13] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] ipmi - pxe: Ensure ipmi is not overriding the boot order [puppet] - 10https://gerrit.wikimedia.org/r/517694 (owner: 10Jbond)
[17:13:02] <wikibugs>	 (03PS2) 10Jeena Huneidi: Add restbase chart (port from local-charts) [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935)
[17:14:43] <wikibugs>	 (03CR) 10Jeena Huneidi: "Thanks!" (032 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935) (owner: 10Jeena Huneidi)
[17:15:45] <wikibugs>	 (03PS2) 10Jbond: ipmi - pxe: Ensure ipmi is not overriding the boot order [puppet] - 10https://gerrit.wikimedia.org/r/517694
[17:27:24] <wikibugs>	 (03PS1) 10Jbond: sysctl: fix new fact [puppet] - 10https://gerrit.wikimedia.org/r/517699
[17:29:49] <logmsgbot>	 !log mbsantos@deploy1001 Started deploy [mobileapps/deploy@dea8e94]: Update mobileapps to c6804c5
[17:29:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:30:28] <wikibugs>	 (03PS2) 10Jbond: sysctl: fix new fact [puppet] - 10https://gerrit.wikimedia.org/r/517699
[17:31:38] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] sysctl: fix new fact [puppet] - 10https://gerrit.wikimedia.org/r/517699 (owner: 10Jbond)
[17:33:39] <wikibugs>	 (03PS2) 10ArielGlenn: testing paging settings for labstore1006 [puppet] - 10https://gerrit.wikimedia.org/r/517693
[17:33:55] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 46.67% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[17:34:30] <logmsgbot>	 !log mbsantos@deploy1001 Finished deploy [mobileapps/deploy@dea8e94]: Update mobileapps to c6804c5 (duration: 04m 41s)
[17:34:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:34:53] <mateusbs17>	 ^ bearND and mdholloway
[17:38:01] <jynus>	 !log testing switchover automation on es2001/es2002 T224852
[17:38:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:38:06] <stashbot>	 T224852: Failover s4 primary master: db1068 to db1081 - https://phabricator.wikimedia.org/T224852
[17:38:39] <mdholloway>	 mateusbs17: awesome! everything looks stable.
[17:40:08] <wikibugs>	 (03CR) 10Nuria: "This looks good but it will worth checking how will puppet react to this as it needs to reclone all repos. Any advice?" [puppet] - 10https://gerrit.wikimedia.org/r/517085 (https://phabricator.wikimedia.org/T221064) (owner: 10Fdans)
[17:40:47] <wikibugs>	 (03CR) 10Jbond: "It was mentioned on the off site that sometime boxes get stuck with PXE boot settings this tries to resolve that issue.  also adds some fa" [puppet] - 10https://gerrit.wikimedia.org/r/517694 (owner: 10Jbond)
[17:51:40] <wikibugs>	 (03PS1) 10Bstorm: dumps distribution: move the alert settings to the right place [puppet] - 10https://gerrit.wikimedia.org/r/517705 (https://phabricator.wikimedia.org/T217474)
[17:52:35] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1
[17:54:06] <wikibugs>	 10Operations, 10DBA, 10Patch-For-Review: Failover s4 primary master: db1068 to db1081 - https://phabricator.wikimedia.org/T224852 (10jcrespo) Testing went as expected:  ` root@cumin1001:~/wmfmariadbpy/wmfmariadbpy$ ./switchover.py --skip-slave-move es2001 es2002 Starting preflight checks... [ERROR]: Initial...
[17:57:33] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] mariadb: Promote db1081 to s4 master [puppet] - 10https://gerrit.wikimedia.org/r/517361 (https://phabricator.wikimedia.org/T224852) (owner: 10Marostegui)
[17:58:44] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] db-eqiad.php: Set s4 in read only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517362 (https://phabricator.wikimedia.org/T224852) (owner: 10Marostegui)
[18:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1800)
[18:00:23] <wikibugs>	 (03CR) 10Bstorm: "This looks right https://puppet-compiler.wmflabs.org/compiler1001/16995/" [puppet] - 10https://gerrit.wikimedia.org/r/517705 (https://phabricator.wikimedia.org/T217474) (owner: 10Bstorm)
[18:01:03] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+1] "Sure looks good from here." [puppet] - 10https://gerrit.wikimedia.org/r/517705 (https://phabricator.wikimedia.org/T217474) (owner: 10Bstorm)
[18:02:31] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] dumps distribution: move the alert settings to the right place [puppet] - 10https://gerrit.wikimedia.org/r/517705 (https://phabricator.wikimedia.org/T217474) (owner: 10Bstorm)
[18:03:50] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] "This is ok to me as is, just some wondering- should we reduce db1081 load in advance? I think the last connection issue we had is to pool " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517363 (https://phabricator.wikimedia.org/T224852) (owner: 10Marostegui)
[18:04:28] <wikibugs>	 (03CR) 10Jcrespo: [C: 03+1] wmnet: Update s4-master CNAME [dns] - 10https://gerrit.wikimedia.org/r/517360 (https://phabricator.wikimedia.org/T224852) (owner: 10Marostegui)
[18:05:21] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+1] "In light of the merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/517705/ this should be ok." [puppet] - 10https://gerrit.wikimedia.org/r/517117 (https://phabricator.wikimedia.org/T217474) (owner: 10Bstorm)
[18:10:01] <wikibugs>	 (03CR) 1020after4: [C: 03+1] "@ArielGlenn: please feel free if it's not too late. Or I could put it up for puppet swat on thursday it hasn't merged by then." [puppet] - 10https://gerrit.wikimedia.org/r/517124 (owner: 1020after4)
[18:14:43] <wikibugs>	 10Operations, 10DBA, 10Patch-For-Review: Failover s4 primary master: db1068 to db1081 - https://phabricator.wikimedia.org/T224852 (10jcrespo) Reviewed all patches, only commented on https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/517363 (but +1ed too).  Running a last compare.py just to be sup...
[18:18:09] <wikibugs>	 (03PS3) 10ArielGlenn: phabricator: change owner / mode on redirect_config.json [puppet] - 10https://gerrit.wikimedia.org/r/517124 (owner: 1020after4)
[18:20:07] <wikibugs>	 (03CR) 10ArielGlenn: [C: 03+2] phabricator: change owner / mode on redirect_config.json [puppet] - 10https://gerrit.wikimedia.org/r/517124 (owner: 1020after4)
[18:20:41] <jynus>	 !log running data compare on s4 (commons) databases T224852
[18:20:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:20:46] <stashbot>	 T224852: Failover s4 primary master: db1068 to db1081 - https://phabricator.wikimedia.org/T224852
[18:22:01] <wikibugs>	 10Operations, 10DBA, 10Patch-For-Review: Failover s4 primary master: db1068 to db1081 - https://phabricator.wikimedia.org/T224852 (10Marostegui) Thanks for all the checks! I will depool db1081 early in the morning, good idea :)
[18:51:35] <wikibugs>	 (03PS13) 10Effie Mouzeli: WIP: mediawiki: check and restart php7 if needed [puppet] - 10https://gerrit.wikimedia.org/r/514673 (https://phabricator.wikimedia.org/T224491)
[18:52:59] <logmsgbot>	 !log ebernhardson@deploy1001 Started deploy [wikimedia/discovery/analytics@deb30dc]: Ship search analytics jobs updated to source from eventgate
[18:53:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:53:12] <ottomata>	 ebernhardson:  <3
[18:53:16] <logmsgbot>	 !log ebernhardson@deploy1001 Finished deploy [wikimedia/discovery/analytics@deb30dc]: Ship search analytics jobs updated to source from eventgate (duration: 00m 17s)
[18:53:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:57:17] <jijiki>	 !log depool thumbor1001
[18:57:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:00:04] <jouncebot>	 twentyafterfour: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for MediaWiki train - American version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T1900).
[19:04:26] <ebernhardson>	 !log deployed discovery.query_clicks_{hourly,daily} fill jobs updated to use eventgate to oozie
[19:04:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:15:32] <twentyafterfour>	 !log branching 1.34.0-wmf.10
[19:15:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:29:50] <wikibugs>	 (03CR) 10EBernhardson: LVS for cloudelastic (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/512925 (https://phabricator.wikimedia.org/T224324) (owner: 10EBernhardson)
[19:45:50] <wikibugs>	 (03PS1) 1020after4: testwikis wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517728
[19:45:52] <wikibugs>	 (03CR) 1020after4: [C: 03+2] testwikis wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517728 (owner: 1020after4)
[19:46:56] <wikibugs>	 (03Merged) 10jenkins-bot: testwikis wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517728 (owner: 1020after4)
[19:47:04] <logmsgbot>	 !log twentyafterfour@deploy1001 Started scap: testwikis wikis to 1.34.0-wmf.10  refs T220735
[19:47:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:47:09] <stashbot>	 T220735: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735
[19:47:11] <wikibugs>	 (03CR) 10jenkins-bot: testwikis wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517728 (owner: 1020after4)
[20:00:04] <jouncebot>	 twentyafterfour: #bothumor I � Unicode. All rise for MediaWiki train - American version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T2000).
[20:00:41] <hashar>	 fun
[20:00:55] <hashar>	 James_F: php70-phan-docker job refers to releng/composer-php72 container :D
[20:01:17] <James_F>	 hashar: Oops. I didn't rename the job?
[20:01:45] <James_F>	 Oh, no, that.
[20:01:52] <James_F>	 Yeah, it uses new composer but still old phan.
[20:02:09] <James_F>	 It's an interstitial state. :-)
[20:02:17] <James_F>	 I'm going to fix that later today now that Wikibase is finally fixed.
[20:07:25] <hashar>	 yeah i noticed that
[20:07:33] <hashar>	 and I think Wikibase was the last repo still relying on phan 0.x
[20:07:37] <hashar>	 which unlock a few things
[20:09:44] <James_F>	 Yup.
[20:09:49] <James_F>	 (Here we go.)
[20:19:42] <logmsgbot>	 !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@0a1c946]: deploy new GUI for T226017
[20:19:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:19:48] <stashbot>	 T226017: Barchart no longer works in Wikidata Query Service - https://phabricator.wikimedia.org/T226017
[20:21:39] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1276 is CRITICAL: CRITICAL - load average: 80.92, 37.51, 23.12
[20:21:51] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1285 is CRITICAL: CRITICAL - load average: 69.74, 39.39, 24.02
[20:22:07] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1280 is CRITICAL: CRITICAL - load average: 72.37, 32.65, 20.46
[20:22:11] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1289 is CRITICAL: CRITICAL - load average: 79.05, 35.80, 21.49
[20:22:27] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1290 is CRITICAL: CRITICAL - load average: 65.04, 33.72, 20.55
[20:22:47] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1283 is CRITICAL: CRITICAL - load average: 61.09, 33.36, 21.03
[20:23:53] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1290 is OK: OK - load average: 26.47, 28.81, 19.96
[20:24:14] <logmsgbot>	 !log twentyafterfour@deploy1001 Finished scap: testwikis wikis to 1.34.0-wmf.10  refs T220735 (duration: 37m 10s)
[20:24:15] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1283 is OK: OK - load average: 22.30, 27.65, 20.15
[20:24:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:24:19] <stashbot>	 T220735: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735
[20:24:46] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1285 is OK: OK - load average: 17.53, 28.16, 22.28
[20:25:01] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1280 is OK: OK - load average: 18.23, 27.61, 20.87
[20:25:03] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1289 is OK: OK - load average: 18.34, 28.18, 21.10
[20:26:01] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1276 is OK: OK - load average: 15.52, 27.81, 23.18
[20:27:22] <wikibugs>	 (03PS1) 1020after4: group0 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517741
[20:27:24] <wikibugs>	 (03CR) 1020after4: [C: 03+2] group0 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517741 (owner: 1020after4)
[20:28:53] <wikibugs>	 (03Merged) 10jenkins-bot: group0 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517741 (owner: 1020after4)
[20:29:33] <wikibugs>	 (03CR) 10jenkins-bot: group0 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517741 (owner: 1020after4)
[20:31:56] <logmsgbot>	 !log twentyafterfour@deploy1001 rebuilt and synchronized wikiversions files: group0 wikis to 1.34.0-wmf.10  refs T220735
[20:32:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:32:02] <stashbot>	 T220735: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735
[20:32:41] <icinga-wm>	 PROBLEM - Memory correctable errors -EDAC- on mw1254 is CRITICAL: 12 ge 4 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw1254&var-datasource=eqiad+prometheus/ops
[20:33:27] <icinga-wm>	 PROBLEM - EDAC syslog messages on mw1254 is CRITICAL: 11.5 ge 4 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw1254&var-datasource=eqiad+prometheus/ops
[20:36:13] <twentyafterfour>	 uhm, what is EDAC ?
[20:36:43] <twentyafterfour>	 "Evidence-Based Design Accreditation and Certification" ?
[20:37:16] <chaomodus>	 something dell specific
[20:37:46] <chaomodus>	 I think?
[20:37:52] <twentyafterfour>	 "misc errors" indeed has spiked, from 0  to a whopping ~0.00003
[20:38:34] <twentyafterfour>	 which looks bad on a graph with range 0, 0.00004
[20:40:22] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: gmail users being suspended from mediawiki-l due to excessive bounces - https://phabricator.wikimedia.org/T225553 (10Aklapper) No idea until someone shares a complete email which includes all mail headers (but not in public please)... :)
[20:41:27] <chaomodus>	 oic it stands for "Error Detection and Correction"
[20:49:45] <logmsgbot>	 !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@0a1c946]: deploy new GUI for T226017 (duration: 30m 03s)
[20:49:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:49:51] <stashbot>	 T226017: Barchart no longer works in Wikidata Query Service - https://phabricator.wikimedia.org/T226017
[20:53:02] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops: Heating alerts / memory errors on mw1254 - https://phabricator.wikimedia.org/T204491 (10jijiki) 05Resolved→03Open p:05Triage→03Normal
[20:54:04] <icinga-wm>	 ACKNOWLEDGEMENT - EDAC syslog messages on mw1254 is CRITICAL: 11.5 ge 4 Effie Mouzeli It is being investigated - T204491 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw1254&var-datasource=eqiad+prometheus/ops
[20:54:04] <icinga-wm>	 ACKNOWLEDGEMENT - Memory correctable errors -EDAC- on mw1254 is CRITICAL: 12 ge 4 Effie Mouzeli It is being investigated - T204491 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=mw1254&var-datasource=eqiad+prometheus/ops
[20:54:28] <wikibugs>	 (03CR) 10Mobrovac: "I haven't read or looked at the code, because I'm currently at the team offsite this week, but if you haven't done so yet, please look at " [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935) (owner: 10Jeena Huneidi)
[21:06:43] <wikibugs>	 (03CR) 10Alex Monk: [C: 03+1] "this is looking OK but I don't have time for a full review right now" [software/acme-chief] - 10https://gerrit.wikimedia.org/r/517605 (https://phabricator.wikimedia.org/T225945) (owner: 10Vgutierrez)
[21:31:04] <twentyafterfour>	 PHP Fatal Error from line 121 of /srv/mediawiki/w/fatal-error.php: Call to undefined function thisFunctionIntentionallyDoesNotExist()
[21:31:10] <twentyafterfour>	 heh ok
[21:33:42] <twentyafterfour>	 !log Promoting Group 1 wikis to MediaWiki 1.34.0-wmf.10 ahead of schedule because tomorrow is a WMF holiday.
[21:33:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:34:01] <wikibugs>	 (03PS1) 1020after4: group1 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517753
[21:34:03] <wikibugs>	 (03CR) 1020after4: [C: 03+2] group1 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517753 (owner: 1020after4)
[21:35:03] <wikibugs>	 (03Merged) 10jenkins-bot: group1 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517753 (owner: 1020after4)
[21:36:22] <wikibugs>	 (03CR) 10jenkins-bot: group1 wikis to 1.34.0-wmf.10  refs T220735 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/517753 (owner: 1020after4)
[21:46:47] <Reedy>	 I'm slightly surprised the logs aren't spammier
[21:47:05] <logmsgbot>	 !log twentyafterfour@deploy1001 rebuilt and synchronized wikiversions files: group1 wikis to 1.34.0-wmf.10  refs T220735
[21:47:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:47:17] <stashbot>	 T220735: 1.34.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T220735
[21:47:40] <twentyafterfour>	 Reedy: me too ;)
[21:48:00] <logmsgbot>	 !log twentyafterfour@deploy1001 Synchronized php: group1 wikis to 1.34.0-wmf.10  refs T220735 (duration: 00m 54s)
[21:48:01] <Reedy>	 Nothing surprising at all atm
[21:48:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[21:48:17] <twentyafterfour>	 Reedy: check out my new and improved dashboard: https://logstash.wikimedia.org/app/kibana#/dashboard/77cc3e90-aa27-11e7-9109-51bd3197f7a9?_g=()
[21:49:06] * Reedy waits for java
[21:51:27] <wikibugs>	 (03PS1) 10Effie Mouzeli: haproxy: haproxy.cfg.erb [puppet] - 10https://gerrit.wikimedia.org/r/517755 (https://phabricator.wikimedia.org/T225284)
[21:55:14] <wikibugs>	 (03CR) 10Smalyshev: refactor wikidata entity dumps into wikibase + wikidata specific bits (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/517670 (https://phabricator.wikimedia.org/T221917) (owner: 10ArielGlenn)
[22:09:31] <wikibugs>	 (03CR) 10Smalyshev: refactor wikidata entity dumps into wikibase + wikidata specific bits (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/517670 (https://phabricator.wikimedia.org/T221917) (owner: 10ArielGlenn)
[22:10:48] <wikibugs>	 (03CR) 10Smalyshev: "The idea looks workable, nitpicks inside." [puppet] - 10https://gerrit.wikimedia.org/r/517670 (https://phabricator.wikimedia.org/T221917) (owner: 10ArielGlenn)
[22:20:25] <logmsgbot>	 !log krinkle@deploy1001 Synchronized php-1.34.0-wmf.8/includes/htmlform/fields/HTMLSelectAndOtherField.php: 90b513d96e36 / T222170 (duration: 00m 57s)
[22:20:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:20:31] <stashbot>	 T222170: InvalidArgumentException from SpecialBlock.php: "$comment can not be null" - https://phabricator.wikimedia.org/T222170
[22:24:37] <wikibugs>	 10Operations, 10CX-cxserver, 10Citoid, 10Graphoid, and 10 others: Make services swagger specs standard compliant - https://phabricator.wikimedia.org/T218217 (10MSantos)
[22:33:34] <jijiki>	 !log pool thumbor1001
[22:33:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:39:23] <wikibugs>	 (03CR) 10Jeena Huneidi: "> Patch Set 2:" [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935) (owner: 10Jeena Huneidi)
[23:00:04] <jouncebot>	 MaxSem, RoanKattouw, and Niharika: Time to snap out of that daydream and deploy Evening SWAT (Max 6 patches). Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190618T2300).
[23:00:04] <jouncebot>	 No GERRIT patches in the queue for this window AFAICS.
[23:18:33] <wikibugs>	 10Operations, 10Annual-Report: Redirects for 2018 Annual Report - https://phabricator.wikimedia.org/T226066 (10LTraer)
[23:30:15] <wikibugs>	 (03PS3) 10Jeena Huneidi: Add restbase chart (port from local-charts) [deployment-charts] - 10https://gerrit.wikimedia.org/r/517557 (https://phabricator.wikimedia.org/T224935)
[23:40:55] <icinga-wm>	 RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 231, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[23:42:27] <icinga-wm>	 RECOVERY - Router interfaces on cr2-esams is OK: OK: host 91.198.174.244, interfaces up: 73, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down