[00:03:31] <wikibugs>	 (03PS1) 10CRusnov: Add dummy secret keys for Ganeti RAPI [labs/private] - 10https://gerrit.wikimedia.org/r/495817
[00:14:42] <wikibugs>	 (03PS15) 10CRusnov: Add configuration for the ganeti->netbox sync. [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229)
[00:17:09] <wikibugs>	 (03CR) 10CRusnov: [C: 03+1] "With type fix, compile looks good:" [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[00:35:23] <wikibugs>	 (03CR) 10Ayounsi: [C: 03+1] Add dummy secret keys for Ganeti RAPI [labs/private] - 10https://gerrit.wikimedia.org/r/495817 (owner: 10CRusnov)
[01:07:06] <wikibugs>	 (03CR) 10CRusnov: [V: 03+2 C: 03+2] Add dummy secret keys for Ganeti RAPI [labs/private] - 10https://gerrit.wikimedia.org/r/495817 (owner: 10CRusnov)
[01:17:17] <wikibugs>	 (03PS3) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[01:17:53] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[01:20:02] <wikibugs>	 (03PS4) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[01:25:22] <wikibugs>	 (03PS5) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[01:25:52] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[01:28:45] <wikibugs>	 (03PS6) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[01:43:04] <wikibugs>	 (03PS7) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[02:28:27] <wikibugs>	 (03PS8) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[02:30:16] <wikibugs>	 (03CR) 10CRusnov: "Puppet compile looks good. https://puppet-compiler.wmflabs.org/compiler1002/15061/" [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[03:44:35] <wikibugs>	 (03PS3) 10Mathew.onipe: nginx: add lua support regardless of version [puppet/nginx] - 10https://gerrit.wikimedia.org/r/492711 (https://phabricator.wikimedia.org/T216681)
[04:00:04] <jouncebot>	 kart_: Dear deployers, time to do the  deploy. Dont look at me like that. You signed up for it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T0400).
[04:03:03] <kart_>	 Yes
[04:04:25] <kart_>	 !log Started manual run of unpublished ContentTranslation draft purge script (T217818)
[04:04:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[04:04:28] <stashbot>	 T217818: Run unpublished draft purge script for CX (Week of 03/10) - https://phabricator.wikimedia.org/T217818
[04:30:48] <wikibugs>	 (03CR) 10BryanDavis: puppet_alert: Email projectadmins instead of members (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495757 (https://phabricator.wikimedia.org/T218009) (owner: 10Alex Monk)
[04:47:36] <icinga-wm>	 PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=list https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[04:48:46] <icinga-wm>	 RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[06:00:28] <icinga-wm>	 PROBLEM - HHVM rendering on mw2212 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[06:01:30] <icinga-wm>	 RECOVERY - HHVM rendering on mw2212 is OK: HTTP OK: HTTP/1.1 200 OK - 76563 bytes in 0.254 second response time
[06:14:14] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1113 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495828
[06:15:24] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1113 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495828 (owner: 10Marostegui)
[06:16:23] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1113 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495828 (owner: 10Marostegui)
[06:17:35] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1113 for schema change and upgrade (duration: 00m 50s)
[06:17:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:18:03] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1113 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495828 (owner: 10Marostegui)
[06:24:24] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10The_RedBurn) >>! In T213214#4870224, @CDanis wrote: > If anyo...
[06:40:30] <kart_>	 !log Finished manual run of unpublished ContentTranslation draft purge script (T217818)
[06:40:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:40:33] <stashbot>	 T217818: Run unpublished draft purge script for CX (Week of 03/10) - https://phabricator.wikimedia.org/T217818
[06:40:44] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10santhosh) After reading the openapi spec and examples, I think the best approach to address optional params...
[06:42:14] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Joe) Well instead of copy/pasta, there is that thing called YAML anchors/references that deal with data rep...
[06:43:06] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Joe) >>! In T217881#5016597, @santhosh wrote: > After reading the openapi spec and examples, I think the be...
[06:50:17] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10santhosh) > I would advise against making a breaking change to our API so that a YAML file is easier to wri...
[07:09:33] <marostegui>	 !log Upgrade mysql and kernel on db1113
[07:09:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:10:15] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1113" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495831
[07:15:32] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1113" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495831 (owner: 10Marostegui)
[07:16:32] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1113" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495831 (owner: 10Marostegui)
[07:17:36] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1113 after schema change and upgrade (duration: 00m 49s)
[07:17:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:18:34] <marostegui>	 !log Deploy schema change on db2052 (s5 codfw master), this will generate lag on codfw T71127 T51199
[07:18:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[07:18:38] <stashbot>	 T71127: Discrepancies with logging table on different wikis - https://phabricator.wikimedia.org/T71127
[07:18:39] <stashbot>	 T51199: Add index log_type_action - https://phabricator.wikimedia.org/T51199
[07:25:52] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1113" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495831 (owner: 10Marostegui)
[07:42:56] <wikibugs>	 (03PS2) 10Dzahn: puppetmaster/puppetboard: set notes URLs [puppet] - 10https://gerrit.wikimedia.org/r/494743
[07:48:25] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "we can always add better ones if we have any, for now it's just about having any URL in there" [puppet] - 10https://gerrit.wikimedia.org/r/494743 (owner: 10Dzahn)
[07:50:15] <wikibugs>	 (03PS14) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[07:51:15] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[07:51:25] <mutante>	 'This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here.'  hah :p
[07:59:39] <wikibugs>	 10Operations, 10Continuous-Integration-Infrastructure, 10Traffic, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10ema) 05Resolved→03Open >>! In T128188#4841316, @hashar wrote: > I am pretty sure @ema finished up the integration of varnishtest with CI / r...
[08:00:15] <wikibugs>	 (03CR) 10Ema: "> @Ema is there a way to run the vtc tests in CI?" [puppet] - 10https://gerrit.wikimedia.org/r/495643 (https://phabricator.wikimedia.org/T216339) (owner: 10Gilles)
[08:01:50] <wikibugs>	 (03CR) 10Gilles: "Alright, then. I did my best adapting the vtc I wrote and tested locally to what should work for the prod tests." [puppet] - 10https://gerrit.wikimedia.org/r/495643 (https://phabricator.wikimedia.org/T216339) (owner: 10Gilles)
[08:05:58] <wikibugs>	 (03PS3) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/495008
[08:06:25] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: add notes URLs to various monitoring checks, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/495008 (owner: 10Dzahn)
[08:12:53] <wikibugs>	 (03PS1) 10Marostegui: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495833
[08:13:46] <wikibugs>	 (03PS4) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/495008
[08:15:38] <ema>	 !log cp1099: ferm.service failed to resolve prometheus1003.eqiad.wmnet. ferm restarted T202966
[08:15:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:15:41] <stashbot>	 T202966: Make cp1099 the new pinkunicorn - https://phabricator.wikimedia.org/T202966
[08:16:26] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] icinga: add notes URLs to various monitoring checks, part 4 [puppet] - 10https://gerrit.wikimedia.org/r/495008 (owner: 10Dzahn)
[08:17:05] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Worth a try!" [puppet] - 10https://gerrit.wikimedia.org/r/495731 (https://phabricator.wikimedia.org/T217731) (owner: 10Elukey)
[08:17:59] <wikibugs>	 (03CR) 10Dzahn: "please review this hopefully uncontroversial and small change" [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873) (owner: 10Dzahn)
[08:18:01] <wikibugs>	 (03CR) 10Muehlenhoff: [V: 03+2 C: 03+2] Add option to debdeploy server to also show automated restarts [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495707 (owner: 10Muehlenhoff)
[08:19:40] <wikibugs>	 (03CR) 10Vgutierrez: [C: 04-1] "please check the inline comments" (033 comments) [puppet/nginx] - 10https://gerrit.wikimedia.org/r/492711 (https://phabricator.wikimedia.org/T216681) (owner: 10Mathew.onipe)
[08:25:50] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Looks good" [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495705 (owner: 10Jbond)
[08:26:36] <wikibugs>	 (03PS1) 10Dzahn: mediawiki: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495834
[08:30:40] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] Add an update action (037 comments) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/487793 (owner: 10Giuseppe Lavagetto)
[08:31:43] <wikibugs>	 10Operations, 10Cloud-Services, 10Cloud-VPS: investigate slapd memory leak - https://phabricator.wikimedia.org/T130593 (10MoritzMuehlenhoff) @GTirloni upgraded OpenLDAP on serpens to 2.4.47, but that doesn't change the memory leak.
[08:32:58] <wikibugs>	 (03PS1) 10Dzahn: swift: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495836
[08:34:01] <wikibugs>	 (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495833 (owner: 10Marostegui)
[08:34:19] <wikibugs>	 (03CR) 10Muehlenhoff: "The memory leak pattern on serpens is still pretty much the same." [puppet] - 10https://gerrit.wikimedia.org/r/495490 (https://phabricator.wikimedia.org/T217280) (owner: 10GTirloni)
[08:34:26] <jynus>	 !log deploy core replica events to db1118
[08:34:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:35:18] <wikibugs>	 (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495833 (owner: 10Marostegui)
[08:36:43] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1082 for schema change (duration: 00m 48s)
[08:36:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:37:41] <icinga-wm>	 PROBLEM - HHVM rendering on mw1224 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:38:43] <icinga-wm>	 RECOVERY - HHVM rendering on mw1224 is OK: HTTP OK: HTTP/1.1 200 OK - 76579 bytes in 0.602 second response time
[08:41:35] <wikibugs>	 10Operations, 10Traffic: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893 (10ema) p:05Triage→03Normal At the time of the issue, cp1077 [[https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1...
[08:42:15] <wikibugs>	 (03PS1) 10Dzahn: DNS: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495838
[08:44:33] <wikibugs>	 (03CR) 10jenkins-bot: db-eqiad.php: Depool db1082 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495833 (owner: 10Marostegui)
[08:44:35] <wikibugs>	 (03PS2) 10Dzahn: DNS: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495838
[08:46:17] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
[08:46:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:46:33] <ema>	 !log cp1077 repooled T217893
[08:46:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:46:36] <stashbot>	 T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893
[08:47:41] <moritzm>	 !log upgrading app servers in codfw to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates)
[08:47:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:48:07] <ema>	 !log restart varnish-be on cp1077 T217893
[08:48:09] <moritzm>	 !log upgrading app servers in codfw to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates) (T216712)
[08:48:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:48:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:48:12] <stashbot>	 T216712: Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712
[08:48:20] <wikibugs>	 (03PS1) 10Dzahn: mail: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495840
[08:48:49] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is CRITICAL: cluster=cache_text site=codfw https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:49:29] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is CRITICAL: cluster=cache_text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:49:41] <ema>	 very strange, cp1077 still has troubles. Depooling
[08:49:45] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:49:51] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=no; selector: name=cp1077.eqiad.wmnet
[08:49:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:50:20] <ema>	 !log cp1077 depooled again T217893
[08:50:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:50:29] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at eqiad on icinga2001 is CRITICAL: job=varnish-text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:50:33] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at ulsfo on icinga2001 is CRITICAL: job=varnish-text site=ulsfo https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:51:48] <wikibugs>	 (03PS1) 10KartikMistry: Add 'campaign' prefix for EG tag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495842 (https://phabricator.wikimedia.org/T216123)
[08:53:49] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:53:51] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at codfw on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:54:33] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:54:37] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:55:15] <wikibugs>	 (03PS3) 10Elukey: Raise memcached dedicated memory on mc1019 [puppet] - 10https://gerrit.wikimedia.org/r/495731 (https://phabricator.wikimedia.org/T217731)
[08:55:34] <wikibugs>	 (03CR) 10Vgutierrez: [C: 03+1] "authdns/dnsrecursor link looks good" [puppet] - 10https://gerrit.wikimedia.org/r/495838 (owner: 10Dzahn)
[08:55:37] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at ulsfo on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:55:47] <wikibugs>	 (03PS1) 10Dzahn: debmonitor: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495843
[08:55:54] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] Raise memcached dedicated memory on mc1019 [puppet] - 10https://gerrit.wikimedia.org/r/495731 (https://phabricator.wikimedia.org/T217731) (owner: 10Elukey)
[08:55:59] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at esams on icinga2001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[08:56:47] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is CRITICAL: cluster={cache_text,cache_upload} site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[08:57:23] <elukey>	 !log restart memcached on mc1019 to apply new settings - T217731
[08:57:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:57:26] <stashbot>	 T217731: Consider raising Memcached MWObject cache memory size limit - https://phabricator.wikimedia.org/T217731
[09:00:49] <icinga-wm>	 PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 229, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[09:01:23] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[09:02:13] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[09:02:29] <icinga-wm>	 PROBLEM - HTTP availability for Varnish at esams on icinga2001 is CRITICAL: job=varnish-text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[09:03:19] <icinga-wm>	 PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is CRITICAL: cluster=cache_text site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[09:03:37] <elukey>	 ema: --^
[09:04:10] <wikibugs>	 10Operations, 10Traffic: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893 (10ema) Re-pooling the service caused the issue to show up again. For some reason, the cron jobs restarting `varnish.service` do not seem t...
[09:04:16] <ema>	 elukey: looking, thanks
[09:05:02] <wikibugs>	 (03PS1) 10Dzahn: aqs/archiva/druid/turnilo/eventstreams/superset: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495845
[09:06:08] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] "thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/495845 (owner: 10Dzahn)
[09:06:27] <icinga-wm>	 RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 231, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[09:06:43] <icinga-wm>	 RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1
[09:07:01] <icinga-wm>	 RECOVERY - HTTP availability for Varnish at esams on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1
[09:07:35] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "thanks for review! the one i used for the wmcs related check is per another review on a second change for wmcs related ones" [puppet] - 10https://gerrit.wikimedia.org/r/495838 (owner: 10Dzahn)
[09:07:50] <wikibugs>	 (03PS3) 10Dzahn: DNS: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495838
[09:08:08] <ema>	 so the first spike was due to the repooling of cp1077, I'm not sure about the second one affecting esams 
[09:08:22] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Gehel) Somewhat related to this issue: T207837. This ticket is about sharing the common workload of updating, including the fet...
[09:09:03] <ema>	 maybe the cr2-eqiad alerts have something to do with it?
[09:09:33] <ema>	 looks like it's over now
[09:10:53] <wikibugs>	 (03PS2) 10Dzahn: aqs/archiva/druid/turnilo/eventstreams/superset: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495845
[09:11:45] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "thanks for the quick review" [puppet] - 10https://gerrit.wikimedia.org/r/495845 (owner: 10Dzahn)
[09:12:46] <ema>	 yeah so the second issue affected both text_esams and upload_esams at the exact same time, indicating that it's likely a network issue
[09:12:47] <elukey>	 ema: yeah I thought the same
[09:13:03] <ema>	 see:
[09:13:04] <ema>	 https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=1552380254224&to=1552381672101&var-datasource=esams%20prometheus%2Fops&var-cache_type=text&var-server=All&var-layer=backend
[09:13:12] <ema>	 and 
[09:13:13] <ema>	 https://grafana.wikimedia.org/d/000000352/varnish-failed-fetches?orgId=1&from=1552380254224&to=1552381672101&var-datasource=esams%20prometheus%2Fops&var-cache_type=upload&var-server=All&var-layer=backend
[09:15:07] <elukey>	 ema: I see from the logs on cr2-eqiad that the port going down is the one connected to cr2-esams
[09:15:12] <elukey>	 so it makes sense
[09:15:31] <elukey>	 (via Level3)
[09:17:08] <ema>	 XioNoX: see above, it seems we've had esams<->eqiad troubles from 08:54 till 09:05
[09:17:10] <mutante>	 gehel: one more notes URL needed for Elasticsearch. the one for "tlsproxy" / "Elasticsearch HTTPS for .."  https://wikitech.wikimedia.org/wiki/Search  or a more specific anchor?
[09:17:48] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Gehel) Given the discussion above, I'm not sure I understand why a `If-Modified-Since:` would not work. What am I missing?
[09:18:21] <ema>	 elukey: thanks!
[09:18:35] <elukey>	 ema: <3
[09:18:51] <gehel>	 mutante: looking
[09:19:42] <elukey>	 XioNoX: there seems to be UPS maintenance in Equinix afaics..
[09:20:20] <gehel>	 mutante: do you have a CR already?
[09:20:41] <mutante>	 gehel: https://gerrit.wikimedia.org/r/c/operations/puppet/+/495846
[09:20:48] * mutante looks at gerrit bot 
[09:22:15] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Gehel) In the end, this looks like a more generic issue of processing events with some level of transactionality. It looks like...
[09:22:17] <wikibugs>	 (03PS1) 10Dzahn: elasticsearch: add Icinga notes_urls part 2 [puppet] - 10https://gerrit.wikimedia.org/r/495846
[09:22:19] <wikibugs>	 (03PS1) 10Marostegui: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495847
[09:22:31] <wikibugs_>	 (03CR) 10Muehlenhoff: "That wikitech page is not a run book? We should only set those notes to checks which actually have one (e.g. so that it's obvious which st" [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[09:22:34] <gehel>	 mutante: I don't have anything better. The tlsproxy check is more about nginx than elasticsearch, so I'm wondering if we have an nginx runbook somewhere alreaedy
[09:23:52] <wikibugs_>	 (03CR) 10Marostegui: [C: 03+2] Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495847 (owner: 10Marostegui)
[09:24:36] <wikibugs_>	 (03CR) 10Dzahn: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[09:24:50] <wikibugs_>	 (03Merged) 10jenkins-bot: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495847 (owner: 10Marostegui)
[09:25:29] <wikibugs_>	 (03CR) 10Gehel: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/495846 (owner: 10Dzahn)
[09:25:36] <mutante>	 gehel: unfortunately not, we dont even have the page "Nginx" :p
[09:25:38] <mutante>	 thanks
[09:25:58] <logmsgbot>	 !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1082 (duration: 00m 48s)
[09:25:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:25:59] <gehel>	 mutante: yeah, I did not find anything better, so let's call it good enough for now
[09:26:17] <mutante>	 ok, sounds good
[09:26:26] <icinga-wm>	 PROBLEM - HHVM rendering on mw2145 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[09:27:20] <icinga-wm>	 RECOVERY - HHVM rendering on mw2145 is OK: HTTP OK: HTTP/1.1 200 OK - 76501 bytes in 0.137 second response time
[09:28:51] <wikibugs>	 (03CR) 10Muehlenhoff: "But there are better ways to enforce that parameter for new Icinga checks, e.g. via a CI check. I think it would be better if that link wo" [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[09:29:28] <wikibugs>	 (03CR) 10jenkins-bot: Revert "db-eqiad.php: Depool db1082" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495847 (owner: 10Marostegui)
[09:30:04] <wikibugs>	 (03CR) 10Dzahn: "> Patch Set 1:" [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[09:31:58] <wikibugs>	 (03CR) 10Santhosh: [C: 03+1] Add 'campaign' prefix for EG tag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495842 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[09:36:48] <wikibugs>	 (03PS1) 10Vgutierrez: acme-chief: Update certs-sync to mirror the new directory tree [puppet] - 10https://gerrit.wikimedia.org/r/495854 (https://phabricator.wikimedia.org/T207295)
[09:37:23] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme-chief: Update certs-sync to mirror the new directory tree [puppet] - 10https://gerrit.wikimedia.org/r/495854 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[09:37:29] <vgutierrez>	 lovely
[09:38:33] <wikibugs>	 (03PS2) 10Vgutierrez: acme-chief: Update certs-sync to mirror the new directory tree [puppet] - 10https://gerrit.wikimedia.org/r/495854 (https://phabricator.wikimedia.org/T207295)
[09:39:19] <jynus>	 !log stop db1114 and restart it empty
[09:39:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:52:25] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] elasticsearch: add Icinga notes_urls part 2 [puppet] - 10https://gerrit.wikimedia.org/r/495846 (owner: 10Dzahn)
[09:52:36] <wikibugs>	 (03PS2) 10Dzahn: elasticsearch: add Icinga notes_urls part 2 [puppet] - 10https://gerrit.wikimedia.org/r/495846
[09:57:28] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10daniel) >>! In T217881#5016597, @santhosh wrote: > ` > /required?optional1=value1&optional2=value2 > `  Doe...
[09:59:51] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "hey Paladox, i see several rebases on this but the comments still need replies and addressing" [puppet] - 10https://gerrit.wikimedia.org/r/463519 (https://phabricator.wikimedia.org/T200739) (owner: 10Paladox)
[10:01:30] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM as this is the best we have. I took the liberty to expand a bit the documentation page at:" [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[10:05:55] <wikibugs>	 (03CR) 10GTirloni: [C: 03+1] xmldumps: set notes URL and contact group to wmcs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873) (owner: 10Dzahn)
[10:06:30] <wikibugs>	 (03CR) 10GTirloni: "Agreed. I'll be reverting the change today. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/495490 (https://phabricator.wikimedia.org/T217280) (owner: 10GTirloni)
[10:06:34] <wikibugs>	 (03Abandoned) 10GTirloni: openldap: Use newer slapd from stretch-backports [puppet] - 10https://gerrit.wikimedia.org/r/495490 (https://phabricator.wikimedia.org/T217280) (owner: 10GTirloni)
[10:13:09] <icinga-wm>	 PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 4 minutes ago with 11 failures. Failed resources (up to 3 shown): Package[php7.2-opcache],Package[php7.2-bcmath],Package[php7.2-bz2],Package[php7.2-curl]
[10:14:34] <wikibugs>	 (03CR) 10Gehel: [C: 04-1] "see comments inline" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[10:14:55] <gtirloni>	 !log upgrading seaborgium to slapd 2.4.47
[10:14:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:17:40] <moritzm>	 !log upgrading API servers/job runners servers in codfw to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates) (T216712)
[10:17:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:17:47] <stashbot>	 T216712: Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712
[10:17:57] <wikibugs>	 (03PS1) 10GTirloni: openldap: Use newer slapd from stretch-backports [puppet] - 10https://gerrit.wikimedia.org/r/495861 (https://phabricator.wikimedia.org/T217280)
[10:18:58] <gtirloni>	 !log updated slapd to version 2.4.47 on seaborgium (T217280)
[10:19:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:19:02] <stashbot>	 T217280: LDAP server running out of memory frequently and disrupting Cloud VPS clients - https://phabricator.wikimedia.org/T217280
[10:19:03] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Nikerabbit) https://cxserver.wikimedia.org/v2?doc#!/Tools/get_v1_list_tool_from_to has two optional paramet...
[10:20:12] <wikibugs>	 (03PS2) 10GTirloni: openldap: Use newer slapd from stretch-backports [puppet] - 10https://gerrit.wikimedia.org/r/495861 (https://phabricator.wikimedia.org/T217280)
[10:25:25] <wikibugs>	 (03PS1) 10Ema: varnish-backend-restart: log execution to syslog [puppet] - 10https://gerrit.wikimedia.org/r/495862 (https://phabricator.wikimedia.org/T217893)
[10:27:27] <wikibugs>	 (03CR) 10GTirloni: [C: 04-1] "This should not be merged yet because it can impact the other LDAP cluster we have." [puppet] - 10https://gerrit.wikimedia.org/r/495861 (https://phabricator.wikimedia.org/T217280) (owner: 10GTirloni)
[10:29:15] <wikibugs>	 (03CR) 10Vgutierrez: [C: 03+1] "this makes a lot of sense :)" [puppet] - 10https://gerrit.wikimedia.org/r/495862 (https://phabricator.wikimedia.org/T217893) (owner: 10Ema)
[10:29:31] <gtirloni>	 !log re-enabled puppet on serpens and seaborgium
[10:29:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:32:09] <zeljkof>	 cutting the branch https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment/Train_deploys#Create_the_new_branch_in_Gerrit
[10:32:38] <marostegui>	 !log Deploy schema change on db1082, lag will happen on s5 on labs
[10:32:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:37:28] <icinga-wm>	 PROBLEM - Disk space on labmon1001 is CRITICAL: DISK CRITICAL - free space: /srv 81976 MB (3% inode=93%)
[10:38:30] <icinga-wm>	 RECOVERY - puppet last run on mw2185 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[10:38:30] <icinga-wm>	 PROBLEM - puppet last run on mw2220 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[php7.2-opcache],Package[php7.2-gmp],Package[php7.2-intl],Package[php7.2-mysql]
[10:43:42] <icinga-wm>	 PROBLEM - puppet last run on mw2249 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 2 minutes ago with 4 failures. Failed resources (up to 3 shown): Package[php7.2-opcache],Package[php7.2-gmp],Package[php7.2-intl],Package[php7.2-mysql]
[10:43:48] <wikibugs>	 (03PS1) 10Dzahn: wmcs/simplelamp: remove mysql puppetization [puppet] - 10https://gerrit.wikimedia.org/r/495864 (https://phabricator.wikimedia.org/T162070)
[10:50:23] <wikibugs>	 (03CR) 10Alex Monk: puppet_alert: Email projectadmins instead of members (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495757 (https://phabricator.wikimedia.org/T218009) (owner: 10Alex Monk)
[10:52:22] <wikibugs>	 (03PS3) 10Alex Monk: puppet_alert: Email projectadmins instead of members [puppet] - 10https://gerrit.wikimedia.org/r/495757 (https://phabricator.wikimedia.org/T218009)
[10:54:15] <wikibugs>	 (03PS4) 10Dzahn: xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873)
[10:54:49] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873) (owner: 10Dzahn)
[10:57:02] <wikibugs>	 (03PS13) 10Paladox: Gerrit: Add flogger javaopts [puppet] - 10https://gerrit.wikimedia.org/r/463519 (https://phabricator.wikimedia.org/T200739)
[10:57:21] <wikibugs>	 (03CR) 10Paladox: "@Dzahn I’ve removed the request tracing now." [puppet] - 10https://gerrit.wikimedia.org/r/463519 (https://phabricator.wikimedia.org/T200739) (owner: 10Paladox)
[10:58:19] <wikibugs>	 (03CR) 10Dzahn: "lolwut, now " WARNING // comment found (slash_comments)" because of the http://" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873) (owner: 10Dzahn)
[10:59:45] <wikibugs>	 (03PS5) 10Dzahn: xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873)
[11:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: #bothumor My software never has bugs. It just develops random features. Rise for European Mid-day SWAT(Max 6 patches). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1100).
[11:00:04] <jouncebot>	 kart_: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[11:00:46] <kart_>	 Only 1 patch? Nice!
[11:00:59] <kart_>	 zeljkof: around? I'll go ahead with my own patch.
[11:01:37] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] "also makes sure wmcs team gets notifications for this and not just "admins" (SRE)" [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873) (owner: 10Dzahn)
[11:01:57] <wikibugs>	 (03PS2) 10Jbond: Use new excluded_mounts config option [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495705
[11:02:23] <wikibugs>	 (03CR) 10Jbond: [V: 03+2] Use new excluded_mounts config option [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495705 (owner: 10Jbond)
[11:02:25] <wikibugs>	 (03CR) 10Jbond: [V: 03+2 C: 03+2] Use new excluded_mounts config option [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495705 (owner: 10Jbond)
[11:02:31] <wikibugs>	 (03PS5) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[11:02:43] <zeljkof>	 kart_: I'm around if you need me :)
[11:02:58] <zeljkof>	 kart_: let me know when you're done, I have some train things to prepare
[11:03:08] <kart_>	 cool.
[11:03:13] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[11:03:37] <wikibugs>	 (03CR) 10KartikMistry: [C: 03+2] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495842 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[11:03:51] <icinga-wm>	 RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[11:04:22] <wikibugs>	 (03PS6) 10Dzahn: xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873)
[11:04:38] <wikibugs>	 (03Merged) 10jenkins-bot: Add 'campaign' prefix for EG tag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495842 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[11:11:04] <wikibugs>	 (03CR) 10jenkins-bot: Add 'campaign' prefix for EG tag [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495842 (https://phabricator.wikimedia.org/T216123) (owner: 10KartikMistry)
[11:11:30] <moritzm>	 !log upgrading API servers/job runners servers in eqiad to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates) (T216712)
[11:11:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:11:33] <stashbot>	 T216712: Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712
[11:11:43] <icinga-wm>	 PROBLEM - puppet last run on mw2279 is CRITICAL: CRITICAL: Puppet has 22 failures. Last run 6 minutes ago with 22 failures. Failed resources (up to 3 shown): Package[php7.2-common],Package[php7.2-opcache],Package[php7.2-bcmath],Package[php7.2-bz2]
[11:11:46] <kart_>	 Deploying gerrit:495842
[11:11:59] <logmsgbot>	 !log kartik@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:495842]] Add campaign prefix for EG tag (T216123) (duration: 00m 49s)
[11:12:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:12:02] <stashbot>	 T216123: Tag is not added when new page is created through External Guidance - https://phabricator.wikimedia.org/T216123
[11:12:12] <kart_>	 zeljkof: I'm done :)
[11:12:26] <zeljkof>	 kart_: great! I'll take over with train preparations
[11:12:57] <zeljkof>	 !log EU SWAT finished
[11:12:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:13:28] <wikibugs>	 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, 10serviceops, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10Tarrow) >>! In T212189#5011311, @akosiaris wrote: > I have to say I am wondering a bit about the latency as the low end seems to be quite h...
[11:14:17] <icinga-wm>	 RECOVERY - puppet last run on mw2249 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[11:18:46] <wikibugs>	 (03PS3) 10Arturo Borrero Gonzalez: aptrepo: rename thirdparty component in stretch-wikimedia to be more concrete [puppet] - 10https://gerrit.wikimedia.org/r/495727 (https://phabricator.wikimedia.org/T215605)
[11:19:51] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: "(I synced with Moritz via IRC)" [puppet] - 10https://gerrit.wikimedia.org/r/495727 (https://phabricator.wikimedia.org/T215605) (owner: 10Arturo Borrero Gonzalez)
[11:19:53] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] aptrepo: rename thirdparty component in stretch-wikimedia to be more concrete [puppet] - 10https://gerrit.wikimedia.org/r/495727 (https://phabricator.wikimedia.org/T215605) (owner: 10Arturo Borrero Gonzalez)
[11:21:09] <wikibugs>	 (03PS6) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[11:21:18] <wikibugs>	 (03PS15) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[11:21:33] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[11:21:54] <wikibugs>	 (03PS11) 10Jcrespo: mariadb: Refactor dump_section.py and rename to match functionality [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/493664 (https://phabricator.wikimedia.org/T206203)
[11:22:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[11:22:19] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb: Refactor dump_section.py and rename to match functionality [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/493664 (https://phabricator.wikimedia.org/T206203) (owner: 10Jcrespo)
[11:24:32] <wikibugs>	 (03PS2) 10Jbond: Add component/ci wikimedia repository to CI hosts [puppet] - 10https://gerrit.wikimedia.org/r/495681 (https://phabricator.wikimedia.org/T212774)
[11:24:34] <wikibugs>	 (03CR) 10Jbond: Add component/ci wikimedia repository to CI hosts (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495681 (https://phabricator.wikimedia.org/T212774) (owner: 10Jbond)
[11:25:28] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: rename thirdparty component in stretch-wikimedia to be more concrete [puppet] - 10https://gerrit.wikimedia.org/r/495727 (https://phabricator.wikimedia.org/T215605) (owner: 10Arturo Borrero Gonzalez)
[11:25:35] <wikibugs>	 (03CR) 10Jcrespo: [V: 03+2 C: 03+2] mariadb: Refactor dump_section.py and rename to match functionality [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/493664 (https://phabricator.wikimedia.org/T206203) (owner: 10Jcrespo)
[11:27:24] <wikibugs>	 (03PS1) 10Zfilipin: Group0 to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495868
[11:32:58] <wikibugs>	 10Operations, 10Wikidata, 10Wikidata-Termbox-Hike, 10serviceops, and 4 others: New Service Request: Wikidata Termbox SSR - https://phabricator.wikimedia.org/T212189 (10WMDE-leszek) >>! In T212189#5011311, @akosiaris wrote:  > I have to say I am wondering a bit about the latency as the low end seems to be q...
[11:33:52] <wikibugs>	 10Operations, 10Traffic, 10Patch-For-Review: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893 (10ema) >>! In T217893#5016736, @ema wrote: > Re-pooling the service caused the issue to show up again. For some reas...
[11:35:45] <arturo>	 !log delete wrong stretch-wikimedia `thirdparty` component in install1002
[11:35:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:36:33] <ema>	 !log cp1077: repool varnish-be after service restart T217893
[11:36:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:36:36] <stashbot>	 T217893: Traffic (text) instability due to misbehaving cache server (cp1077), causing a 1.5-2% requests failing - https://phabricator.wikimedia.org/T217893
[11:36:50] <logmsgbot>	 !log ema@puppetmaster1001 conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
[11:36:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:37:35] <icinga-wm>	 RECOVERY - puppet last run on mw2279 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[11:39:50] <elukey>	 !log raise mysql's max_user_connection to 1000 for the Analytics user on labsdb1012
[11:39:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:41:43] <logmsgbot>	 !log zfilipin@deploy1001 Pruned MediaWiki: 1.33.0-wmf.16 (duration: 12m 41s)
[11:41:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:42:11] <arturo>	 !log T215605 include python-oath .deb in stretch-wikimedia thirdparty/oath
[11:42:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:42:14] <stashbot>	 T215605: cloudvps: missing packages in stretch for cloudcontrol servers - https://phabricator.wikimedia.org/T215605
[11:44:52] <wikibugs>	 (03PS16) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[11:45:06] <logmsgbot>	 !log zfilipin@deploy1001 Pruned MediaWiki: 1.33.0-wmf.18 [keeping static files] (duration: 01m 35s)
[11:45:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:45:48] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[11:47:11] <logmsgbot>	 !log zfilipin@deploy1001 Pruned MediaWiki: 1.33.0-wmf.17 [keeping static files] (duration: 01m 40s)
[11:47:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:48:53] <logmsgbot>	 !log zfilipin@deploy1001 Started scap: testwiki to php-1.33.0-wmf.21 and rebuild l10n cache
[11:48:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:53:34] <wikibugs>	 10Operations, 10Continuous-Integration-Infrastructure, 10Packaging, 10Patch-For-Review: Upgrade jenkins-debian-glue to v0.20.0 - https://phabricator.wikimedia.org/T212774 (10jbond) @hashar i have taken another look at the patch i created yesterday and  i now think it is incorrect.  As far as i can tell `pr...
[11:55:00] <wikibugs>	 (03CR) 10Jbond: "compile results indicate these change dose nothing" [puppet] - 10https://gerrit.wikimedia.org/r/495681 (https://phabricator.wikimedia.org/T212774) (owner: 10Jbond)
[11:55:39] <icinga-wm>	 PROBLEM - puppet last run on mw1288 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 3 minutes ago with 20 failures. Failed resources (up to 3 shown): Exec[set debconf flag seen for wireshark-common/install-setuid],Package[php7.2-common],Package[php7.2-opcache],Package[php7.2-bcmath]
[11:57:39] <icinga-wm>	 PROBLEM - Check systemd state on analytics-tool1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[11:57:49] <icinga-wm>	 PROBLEM - superset on analytics-tool1004 is CRITICAL: connect to address 10.64.36.116 and port 9080: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset
[11:59:16] <mutante>	 !log analytics-tool1004 - start superset service
[11:59:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:59:43] <elukey>	 mutante: I am working on it :)
[11:59:48] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: aptrepo: introduce stretch-wikimedia thirdparty/mwclient component [puppet] - 10https://gerrit.wikimedia.org/r/495873 (https://phabricator.wikimedia.org/T215605)
[11:59:50] <elukey>	 it is a testing host
[12:00:01] <icinga-wm>	 RECOVERY - Check systemd state on analytics-tool1004 is OK: OK - running: The system is fully operational
[12:00:02] <yannf>	 https://upload.wikimedia.org/wikipedia/commons/thumb/7/7c/Louise_Elisabeth_Vig%C3%A9e-Lebrun_-_Marie-Antoinette_de_Lorraine-Habsbourg%2C_reine_de_France_et_ses_enfants_-_Google_Art_Project.jpg/5000px-Louise_Elisabeth_Vig%C3%A9e-Lebrun_-_Marie-Antoinette_de_Lorraine-Habsbourg%2C_reine_de_France_et_ses_enfants_-_Google_Art_Project.jpg
[12:00:04] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1200)
[12:00:09] <yannf>	 this repeatedly fails
[12:00:11] <icinga-wm>	 RECOVERY - superset on analytics-tool1004 is OK: TCP OK - 0.036 second response time on 10.64.36.116 port 9080 https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset
[12:00:21] <mutante>	 elukey: if it's a test host per definition, should we disable notifications via hiera?
[12:01:17] <elukey>	 mutante: it is testing for the moment, I was planning to swap it with analytics-tool1003 as part of its upgrade to buster, but I am finding some issues while doing the upgrade
[12:01:21] <elukey>	 I was about to do it
[12:02:53] <mutante>	 elukey: okey dokey, i'll leave a note to ingore it
[12:05:01] <icinga-wm>	 ACKNOWLEDGEMENT - superset on analytics-tool1004 is CRITICAL: connect to address 10.64.36.116 and port 9080: Connection refused daniel_zahn its being worked on - The acknowledgement expires at: 2019-03-13 16:04:29. https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset
[12:09:08] <moritzm>	 !log upgrading mw1238-mw1258 to component/php72 / PHP 7.2.16 (combined with glibc/OpenSSL updates)
[12:09:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:16:15] <icinga-wm>	 PROBLEM - puppet last run on mw1271 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libc6-dbg]
[12:17:50] <moritzm>	 ^ that's me
[12:18:07] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1227 is CRITICAL: CRITICAL - load average: 50.81, 23.59, 16.14
[12:18:15] <wikibugs>	 10Operations, 10decommission, 10hardware-requests: decommission wmf6937 as phab1002, reimage as mw1298 - https://phabricator.wikimedia.org/T215332 (10Dzahn)
[12:18:22] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install phab1002(WMF4727) - https://phabricator.wikimedia.org/T196019 (10Dzahn)
[12:18:31] <icinga-wm>	 PROBLEM - High CPU load on API appserver on mw1233 is CRITICAL: CRITICAL - load average: 66.68, 31.15, 17.94
[12:18:50] <jijiki>	 moritzm: are you the CPU load as well?
[12:19:01] <jijiki>	 or should we look into it ?
[12:19:30] <wikibugs>	 10Operations, 10hardware-requests, 10Patch-For-Review: request to assign wmf6937 (mw1298, former imagescaler) (now: wmf4727)  as phab1002 - https://phabricator.wikimedia.org/T195623 (10Dzahn)
[12:19:38] <wikibugs>	 10Operations, 10ops-eqiad, 10Patch-For-Review: setup/install phab1002(WMF4727) - https://phabricator.wikimedia.org/T196019 (10Dzahn) 05Open→03Resolved re-closing.  replaced by new request in T215335. will be reverted in T215332
[12:22:09] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1233 is OK: OK - load average: 14.39, 24.07, 18.16
[12:22:11] <moritzm>	 jijiki: those two I haven't upgraded, seem unrelated
[12:22:39] <moritzm>	 (I mean HHVM wasn't upgraded/changed)
[12:22:54] <jijiki>	 ok 
[12:22:57] <icinga-wm>	 RECOVERY - High CPU load on API appserver on mw1227 is OK: OK - load average: 12.51, 22.86, 18.81
[12:23:18] <logmsgbot>	 !log zfilipin@deploy1001 Finished scap: testwiki to php-1.33.0-wmf.21 and rebuild l10n cache (duration: 34m 25s)
[12:23:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:23:42] <jynus>	 !log testing snapshotting on db1117:3325 -> dbstore1001 T210292
[12:23:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:23:45] <stashbot>	 T210292: Implement a proof of concept of a snapshot cycle automation for a mediawiki section database - https://phabricator.wikimedia.org/T210292
[12:24:50] <wikibugs>	 (03PS1) 10Elukey: Set notifications disabled for analytics-tool1004 [puppet] - 10https://gerrit.wikimedia.org/r/495876
[12:24:56] <elukey>	 mutante: --^ :)
[12:26:47] <icinga-wm>	 RECOVERY - puppet last run on mw1288 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures
[12:40:59] <arturo>	 !log T215605 include python-mwclient .deb in openstack-mitaka-jessie/jessie-wikimedia in install1002
[12:41:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[12:41:02] <stashbot>	 T215605: cloudvps: missing packages in stretch for cloudcontrol servers - https://phabricator.wikimedia.org/T215605
[12:41:53] <wikibugs>	 (03Abandoned) 10Arturo Borrero Gonzalez: aptrepo: introduce stretch-wikimedia thirdparty/mwclient component [puppet] - 10https://gerrit.wikimedia.org/r/495873 (https://phabricator.wikimedia.org/T215605) (owner: 10Arturo Borrero Gonzalez)
[12:45:30] <wikibugs>	 (03PS2) 10WMDE-leszek: Added wmgWikibaseEntitySources setting for defining Wikibase "entity sources" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490104 (https://phabricator.wikimedia.org/T214557)
[12:45:40] <wikibugs>	 (03PS2) 10WMDE-leszek: Added wmgWikibaseRepoLocalEntitySourceName to define the "local" source of Wikibase Repo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490633 (https://phabricator.wikimedia.org/T214557)
[12:47:23] <icinga-wm>	 RECOVERY - puppet last run on mw1271 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures
[12:48:59] <wikibugs>	 (03PS4) 10WMDE-leszek: DNM Define Wikibase "entity sources" on beta commons [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490108 (https://phabricator.wikimedia.org/T214557)
[12:50:30] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Investigate more efficient memcached solution for CacheAwarePropertyInfoStore - https://phabricator.wikimedia.org/T97368 (10Addshore) p:05Normal→03High
[12:51:04] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368 (10Addshore)
[12:57:52] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] debmonitor: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[12:58:25] <wikibugs>	 (03PS2) 10Muehlenhoff: Update debian/changelog for new deb build/release [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495708
[12:59:10] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Update debian/changelog for new deb build/release [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495708 (owner: 10Muehlenhoff)
[13:00:04] <jouncebot>	 zeljkof: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for MediaWiki train - European version. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1300).
[13:01:05] <zeljkof>	 long time to train
[13:05:02] <TheDragonFire>	 test wiki page load is slow
[13:10:02] <Zppix>	 zeljkof: test.wikipedia is running awfully slow today i've recieved 2 reports and experienced myself abnormal slowless is there something going on with that wiki that is planned?
[13:11:26] <zeljkof>	 TheDragonFire, Zppix: I've just updated testwiki to wmf.21, as far as I remember, it's slow for a while after the upgrade
[13:11:46] <Zppix>	 zeljkof:  ah ok that makes sense thanks
[13:17:54] <wikibugs>	 10Operations: Audit our puppet tree for uses of jessie-backports - https://phabricator.wikimedia.org/T216711 (10aborrero)
[13:20:28] <wikibugs>	 (03CR) 10Zfilipin: [C: 03+2] "Train" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495868 (owner: 10Zfilipin)
[13:21:36] <wikibugs>	 (03Merged) 10jenkins-bot: Group0 to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495868 (owner: 10Zfilipin)
[13:24:20] <logmsgbot>	 !log zfilipin@deploy1001 rebuilt and synchronized wikiversions files: group0 to 1.33.0-wmf.21
[13:24:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:25:55] <wikibugs>	 (03CR) 10jenkins-bot: Group0 to 1.33.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495868 (owner: 10Zfilipin)
[13:27:16] <marostegui>	 !log Deploy schema change on s6 codfw, lag will be generated on s6 codfw
[13:27:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:33:14] <wikibugs>	 (03CR) 10DCausse: Add support for elasticsearch 6 (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[13:33:24] <wikibugs>	 (03PS19) 10DCausse: Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196)
[13:33:25] <icinga-wm>	 PROBLEM - Disk space on labmon1001 is CRITICAL: DISK CRITICAL - free space: /srv 81924 MB (3% inode=93%)
[13:34:11] <wikibugs>	 (03CR) 10Herron: [C: 03+1] "Looks good!" [puppet] - 10https://gerrit.wikimedia.org/r/495840 (owner: 10Dzahn)
[13:34:47] <zeljkof>	 1.33.0-wmf.21 at group 0 T206675
[13:34:48] <stashbot>	 T206675: 1.33.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T206675
[13:37:07] <wikibugs>	 10Operations, 10monitoring, 10Patch-For-Review: Serve >= 50% of production Prometheus systems with Prometheus v2 - https://phabricator.wikimedia.org/T187987 (10fgiunchedi) Status update:  The migrations have finished for all instances on prometheus2003, however the same symptom remains. To explain in detail...
[13:41:32] <wikibugs>	 10Operations, 10RESTBase-API, 10TechCom, 10serviceops, and 2 others: Decide whether to keep violating OpenAPI/Swagger specification in our REST services - https://phabricator.wikimedia.org/T217881 (10Pchelolo) > Well instead of copy/pasta, there is that thing called YAML anchors/references that deal with d...
[13:43:30] <marostegui>	 !log Upgrade MySQL and kernel on db2094 (inactive sanitarium)
[13:43:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:45:48] <wikibugs>	 (03CR) 10CDanis: [C: 03+1] check_icinga: fix retry logic (031 comment) [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/495447 (https://phabricator.wikimedia.org/T217599) (owner: 10Volans)
[13:47:07] <wikibugs>	 (03PS2) 10Volans: check_icinga: fix retry logic [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/495447 (https://phabricator.wikimedia.org/T217599)
[13:47:14] <wikibugs>	 (03CR) 10Volans: check_icinga: fix retry logic (031 comment) [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/495447 (https://phabricator.wikimedia.org/T217599) (owner: 10Volans)
[13:57:18] <wikibugs>	 (03PS2) 10Ottomata: Add pycountry to analytics cluster [puppet] - 10https://gerrit.wikimedia.org/r/484994 (https://phabricator.wikimedia.org/T209857) (owner: 10Gilles)
[13:57:44] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] Add pycountry to analytics cluster [puppet] - 10https://gerrit.wikimedia.org/r/484994 (https://phabricator.wikimedia.org/T209857) (owner: 10Gilles)
[14:01:05] <icinga-wm>	 PROBLEM - superset on analytics-tool1004 is CRITICAL: connect to address 10.64.36.116 and port 9080: Connection refused https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset
[14:01:17] <icinga-wm>	 PROBLEM - Check systemd state on analytics-tool1004 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[14:03:09] <wikibugs>	 (03CR) 10Volans: [C: 03+2] check_icinga: fix retry logic [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/495447 (https://phabricator.wikimedia.org/T217599) (owner: 10Volans)
[14:03:38] <wikibugs>	 (03Merged) 10jenkins-bot: check_icinga: fix retry logic [software/external-monitoring] - 10https://gerrit.wikimedia.org/r/495447 (https://phabricator.wikimedia.org/T217599) (owner: 10Volans)
[14:06:55] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10ema) p:05Triage→03Normal
[14:14:23] <wikibugs>	 (03PS1) 10Jbond: Ensure deb package creates config directory [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495891
[14:14:45] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Good catch" [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495891 (owner: 10Jbond)
[14:16:11] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] Ensure deb package creates config directory [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495891 (owner: 10Jbond)
[14:22:48] <wikibugs>	 10Operations, 10Performance-Team, 10TechCom-RFC, 10Traffic, and 3 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10kchapman)
[14:23:50] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: openstack: drop -t jessie-backports for openstack component installs in stretch [puppet] - 10https://gerrit.wikimedia.org/r/495893 (https://phabricator.wikimedia.org/T216497)
[14:23:52] <wikibugs>	 10Operations, 10Core Platform Team Backlog, 10Performance-Team, 10TechCom-RFC, and 4 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10kchapman)
[14:24:19] <wikibugs>	 10Operations, 10Performance-Team, 10TechCom-RFC, 10Traffic, and 3 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10kchapman)
[14:24:35] <wikibugs>	 10Operations, 10Performance-Team, 10TechCom-RFC, 10Traffic, and 3 others: Serve Main Page of WMF wikis from a consistent URL - https://phabricator.wikimedia.org/T120085 (10kchapman)
[14:26:06] <mutante>	 !log phab1002 - reboot
[14:26:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:29:17] <wikibugs>	 (03PS2) 10Dzahn: mail: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495840
[14:33:44] <wikibugs>	 (03CR) 10Dzahn: "> FWIW I added some troubleshooting notes under https://wikitech.wikimedia.org/wiki/Mail#Troubleshooting.  We could link directly to that " [puppet] - 10https://gerrit.wikimedia.org/r/495840 (owner: 10Dzahn)
[14:34:16] <wikibugs>	 (03PS3) 10Dzahn: mail: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495840
[14:34:28] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] mail: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495840 (owner: 10Dzahn)
[14:34:32] <ottomata>	 anomie:  o/  if you find time could you give this one more look and a possible +1 https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/495399/ ?
[14:35:00] <ottomata>	 oh Pchelolo  I think we already missed teh train for ^
[14:36:38] <anomie>	 ottomata: I should get to it later today
[14:37:32] <ottomata>	 k thanks anomie 
[14:37:32] <wikibugs>	 (03CR) 10Dzahn: "ah, forgot it in the second link, i will add that in my next change" [puppet] - 10https://gerrit.wikimedia.org/r/495840 (owner: 10Dzahn)
[14:37:53] <wikibugs>	 10Operations, 10Traffic: Content purges are unreliable - https://phabricator.wikimedia.org/T133821 (10mobrovac)
[14:39:50] <wikibugs>	 (03PS1) 10Muehlenhoff: Properly bump version [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495894
[14:40:37] <wikibugs>	 (03CR) 10Jbond: [C: 03+1] "lgtm" [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495894 (owner: 10Muehlenhoff)
[14:40:47] <wikibugs>	 (03CR) 10Dzahn: [C: 04-1] "technical -1 because of the hash tag. should not be merged before 2.16 upgrade but on "the day of"" [puppet] - 10https://gerrit.wikimedia.org/r/463519 (https://phabricator.wikimedia.org/T200739) (owner: 10Paladox)
[14:43:05] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+2] Properly bump version [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495894 (owner: 10Muehlenhoff)
[14:45:21] <icinga-wm>	 RECOVERY - Check systemd state on analytics-tool1004 is OK: OK - running: The system is fully operational
[14:50:05] <icinga-wm>	 RECOVERY - Device not healthy -SMART- on phab1002 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=phab1002&var-datasource=eqiad+prometheus/ops
[14:51:35] <mutante>	 self-healing HD, nice
[14:53:52] <wikibugs>	 (03PS2) 10Bstorm: osmdb: Switch the replica to the VM that needs to become the master [puppet] - 10https://gerrit.wikimedia.org/r/495290 (https://phabricator.wikimedia.org/T193264)
[15:00:34] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade -n staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[15:00:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:02:11] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[15:02:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:02:36] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[15:02:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:02:38] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics cluster staging completed
[15:02:38] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics finished
[15:02:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:02:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:04:34] <wikibugs>	 (03PS1) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899
[15:05:28] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899 (owner: 10Dzahn)
[15:07:05] <icinga-wm>	 RECOVERY - superset on analytics-tool1004 is OK: TCP OK - 0.036 second response time on 10.64.36.116 port 9080 https://wikitech.wikimedia.org/wiki/Analytics/Systems/Superset
[15:08:35] <wikibugs>	 (03PS2) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899
[15:08:52] <wikibugs>	 (03PS20) 10Gehel: Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:09:27] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899 (owner: 10Dzahn)
[15:09:41] <wikibugs>	 (03CR) 10Volans: "The change looks reasonable." (037 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[15:11:23] <wikibugs>	 (03PS3) 10Dzahn: icinga: add notes URLs to various monitoring checks, part 5 [puppet] - 10https://gerrit.wikimedia.org/r/495899
[15:11:59] <wikibugs>	 (03CR) 10DCausse: [C: 03+1] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:12:43] <wikibugs>	 (03CR) 10Gehel: "PCC looks happy (NOOP): https://puppet-compiler.wmflabs.org/compiler1002/15065/" [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:13:12] <wikibugs>	 (03PS2) 10Dzahn: debmonitor: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495843
[15:13:38] <wikibugs>	 (03PS1) 10Ottomata: eventgate-analytics - add schema_precache_uris configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/495904 (https://phabricator.wikimedia.org/T217661)
[15:14:15] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] debmonitor: add notes URLs for Icinga [puppet] - 10https://gerrit.wikimedia.org/r/495843 (owner: 10Dzahn)
[15:15:23] <wikibugs>	 (03PS2) 10Ottomata: eventgate-analytics - add schema_precache_uris configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/495904 (https://phabricator.wikimedia.org/T217661)
[15:15:29] <wikibugs>	 (03PS21) 10Gehel: Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:15:38] <wikibugs>	 (03PS7) 10Dzahn: xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873)
[15:16:34] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] Add support for elasticsearch 6 [puppet] - 10https://gerrit.wikimedia.org/r/493234 (https://phabricator.wikimedia.org/T217196) (owner: 10DCausse)
[15:16:44] <wikibugs>	 (03CR) 10Ottomata: [V: 03+2 C: 03+2] eventgate-analytics - add schema_precache_uris configuration [deployment-charts] - 10https://gerrit.wikimedia.org/r/495904 (https://phabricator.wikimedia.org/T217661) (owner: 10Ottomata)
[15:17:00] <wikibugs>	 (03PS1) 10Ppchelko: Enable api-request logging into kafka for group0. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495906
[15:17:23] <wikibugs>	 (03PS8) 10Dzahn: xmldumps: set notes URL and contact group to wmcs [puppet] - 10https://gerrit.wikimedia.org/r/494733 (https://phabricator.wikimedia.org/T197873)
[15:18:07] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] "Yeehaw!  Let's get the latest version of eventgate-analytics stuff out and then try it!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495906 (owner: 10Ppchelko)
[15:18:49] <wikibugs>	 (03PS5) 10Ema: varnish: text/misc frontend size-based cutoffs [puppet] - 10https://gerrit.wikimedia.org/r/494937 (https://phabricator.wikimedia.org/T216006)
[15:19:40] <wikibugs>	 (03CR) 10Ema: [C: 03+2] varnish: text/misc frontend size-based cutoffs [puppet] - 10https://gerrit.wikimedia.org/r/494937 (https://phabricator.wikimedia.org/T216006) (owner: 10Ema)
[15:22:49] <wikibugs>	 10Operations, 10netops: Bird multihop BFD - https://phabricator.wikimedia.org/T209989 (10ayounsi) Another suggestion from the [[ https://bird.network.cz/pipermail/bird-users/2019-March/013171.html | Bird mailing list ]]. Junos extensive output mentions `Multi-hop min-recv-TTL 254`. I'd guess this is set as the...
[15:23:24] <logmsgbot>	 !log ayounsi@puppetmaster1001 conftool action : set/pooled=no; selector: name=dns2001.wikimedia.org,service=pdns_^Ccursor
[15:23:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:26:40] <logmsgbot>	 !log ayounsi@puppetmaster1001 conftool action : set/pooled=no; selector: name=dns2001.wikimedia.org,service=pdns_recursor
[15:26:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:28:21] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade staging -f eventgate-analytics-staging-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: staging]
[15:28:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:28:23] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster staging completed
[15:28:23] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[15:28:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:28:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:33:01] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade production stable/eventgate-analytics -f eventgate-analytics-eqiad-values.yaml [namespace: eventgate-analytics, clusters: eqiad]
[15:33:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:33:48] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:33:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:34:26] <wikibugs>	 (03PS2) 10Ema: varnish-backend-restart: log execution to syslog [puppet] - 10https://gerrit.wikimedia.org/r/495862 (https://phabricator.wikimedia.org/T217893)
[15:35:58] <wikibugs>	 (03CR) 10Ema: [C: 03+2] varnish-backend-restart: log execution to syslog [puppet] - 10https://gerrit.wikimedia.org/r/495862 (https://phabricator.wikimedia.org/T217893) (owner: 10Ema)
[15:36:03] <wikibugs>	 (03CR) 10Herron: elasticsearch: mjolnir bulk update lag (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[15:36:32] <wikibugs>	 (03PS1) 10Gehel: aptrepo: create elastic65 component [puppet] - 10https://gerrit.wikimedia.org/r/495914 (https://phabricator.wikimedia.org/T218113)
[15:37:22] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:37:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:38:29] <wikibugs>	 (03PS2) 10Gehel: aptrepo: create elastic65 component [puppet] - 10https://gerrit.wikimedia.org/r/495914 (https://phabricator.wikimedia.org/T218113)
[15:38:55] <logmsgbot>	 !log ayounsi@puppetmaster1001 conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org,service=pdns_recursor
[15:38:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:48] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:39:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:39:51] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: expose elastic65 APT component to hiera [puppet] - 10https://gerrit.wikimedia.org/r/495915 (https://phabricator.wikimedia.org/T194199)
[15:40:07] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+1] Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/495424 (owner: 10Paladox)
[15:41:29] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] aptrepo: create elastic65 component [puppet] - 10https://gerrit.wikimedia.org/r/495914 (https://phabricator.wikimedia.org/T218113) (owner: 10Gehel)
[15:41:35] <wikibugs>	 (03CR) 10Volans: Add ganeti->netbox sync script (031 comment) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[15:41:54] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:41:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:42:26] <wikibugs>	 (03PS1) 10Dbarratt: Enable Partial Blocks on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495917 (https://phabricator.wikimedia.org/T217278)
[15:42:39] <logmsgbot>	 !log ppchelko@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:42:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:42:54] <wikibugs>	 10Operations, 10netops: Bird multihop BFD - https://phabricator.wikimedia.org/T209989 (10ayounsi) Victory, the BFD session went up as soon as I changed the TTL to 255.
[15:43:06] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368 (10elukey) Another big SET bursts to mc1022 caused timeouts for the MW ap...
[15:43:07] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:43:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:43:15] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-eqiad-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: eqiad]
[15:43:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:43:18] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster eqiad completed
[15:43:18] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[15:43:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:43:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:43:25] <wikibugs>	 (03PS4) 10Vgutierrez: acme-chief: Store certificates in unique directories [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295)
[15:43:27] <wikibugs>	 (03PS6) 10Vgutierrez: acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295)
[15:43:50] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[15:44:07] <wikibugs>	 (03CR) 10Vgutierrez: "Thanks for taking the time and review the change <3" (037 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[15:44:54] <wikibugs>	 (03PS1) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[15:45:22] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics upgrade production -f eventgate-analytics-codfw-values.yaml stable/eventgate-analytics [namespace: eventgate-analytics, clusters: codfw]
[15:45:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:45:24] <wikibugs>	 (03PS16) 10CRusnov: Add configuration for the ganeti->netbox sync. [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229)
[15:45:25] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics cluster codfw completed
[15:45:25] <logmsgbot>	 !log otto@deploy1001 scap-helm eventgate-analytics finished
[15:45:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:45:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:45:43] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme-chief: Store certificates in unique directories [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[15:45:45] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[15:46:14] <vgutierrez>	 :_(
[15:47:00] <wikibugs>	 (03CR) 10CRusnov: [C: 03+2] Add configuration for the ganeti->netbox sync. [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[15:47:11] <wikibugs>	 (03CR) 10EBernhardson: elasticsearch: mjolnir bulk update lag (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[15:47:49] <wikibugs>	 (03PS5) 10Vgutierrez: acme-chief: Store certificates in unique directories [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295)
[15:47:51] <wikibugs>	 (03PS7) 10Vgutierrez: acme-chief-api: Add support for puppet HTTP API search operation [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295)
[15:49:12] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Implement per property caching for CacheAwarePropertyInfoStore - https://phabricator.wikimedia.org/T218115 (10Addshore)
[15:49:21] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Implement per property caching for CacheAwarePropertyInfoStore - https://phabricator.wikimedia.org/T218115 (10Addshore)
[15:49:31] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Implement per property caching for CacheAwarePropertyInfoStore - https://phabricator.wikimedia.org/T218115 (10Addshore)
[15:49:48] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368 (10Addshore) a:05Addshore→03None
[15:49:59] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Implement per property caching for CacheAwarePropertyInfoStore - https://phabricator.wikimedia.org/T218115 (10Addshore) p:05Triage→03High
[15:50:59] <wikibugs>	 (03PS2) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[15:52:15] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: allow a version of configuration independant of the package [puppet] - 10https://gerrit.wikimedia.org/r/495919 (https://phabricator.wikimedia.org/T194199)
[15:52:17] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: upgrade to elastic 6.5.4 [puppet] - 10https://gerrit.wikimedia.org/r/495920 (https://phabricator.wikimedia.org/T218116)
[15:52:20] <wikibugs>	 (03PS1) 10Gehel: elasticsearch: deploy elasticsearch config for ES6 [puppet] - 10https://gerrit.wikimedia.org/r/495921 (https://phabricator.wikimedia.org/T218116)
[15:54:34] <wikibugs>	 (03CR) 10Dmaza: [C: 03+1] Enable Partial Blocks on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495917 (https://phabricator.wikimedia.org/T217278) (owner: 10Dbarratt)
[15:56:10] <wikibugs>	 (03CR) 10Herron: elasticsearch: mjolnir bulk update lag (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[15:58:00] <wikibugs>	 10Operations, 10DC-Ops: Request spare systems to test ipmi password reset cookbook - https://phabricator.wikimedia.org/T218117 (10jbond)
[15:58:36] <wikibugs>	 10Operations, 10Puppet, 10Packaging, 10Patch-For-Review: Prepare puppet for Debian  buster - https://phabricator.wikimedia.org/T213546 (10jcrespo) Oh, I think I used the wrong task for my changes. Sorry about that, I will rename the task to prevent further confusion.
[15:58:45] <wikibugs>	 (03PS3) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[15:59:08] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885) (owner: 10Ammarpad)
[15:59:17] <wikibugs>	 10Operations, 10Puppet, 10Packaging, 10Patch-For-Review: Prepare puppet infrastructure for Debian  buster - https://phabricator.wikimedia.org/T213546 (10jcrespo)
[16:00:04] <jouncebot>	 godog and _joe_: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Puppet SWAT(Max 6 patches) . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1600).
[16:00:04] <jouncebot>	 niedzielski: A patch you scheduled for Puppet SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[16:00:23] <niedzielski>	 o/ I am here!
[16:00:45] <godog>	 niedzielski: I'll take a look
[16:00:59] <niedzielski>	 thanks godog!
[16:01:11] <godog>	 niedzielski: is this https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/495905 ?
[16:01:42] <niedzielski>	 sorry, i will update the table. that patch was replaced by https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/MobileFrontend/+/495907/
[16:01:44] <niedzielski>	 (i had the wrong target branch)
[16:02:29] <godog>	 niedzielski: ah ok, thanks! also I'm asking because this is puppet swat, i.e. puppet.git swat, not mediawiki
[16:02:46] <wikibugs>	 (03PS1) 10Jbond: Fix shebang regression in client/debdeploy-restarts [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495923
[16:03:24] <niedzielski>	 oof, today is not my morning. sorry about that godog . i'll be back later today :]
[16:03:26] <wikibugs>	 10Operations, 10Puppet, 10monitoring: prometheus-node-exporter makes puppet fail because requiring a version that no longer exists on buster - https://phabricator.wikimedia.org/T218118 (10jcrespo) p:05Triage→03Low
[16:03:56] <godog>	 niedzielski: hehe ok no worries at all, good luck
[16:04:07] <wikibugs>	 (03CR) 10Volans: acme-chief: Store certificates in unique directories (033 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494956 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[16:05:04] <wikibugs>	 (03PS4) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[16:05:20] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885) (owner: 10Ammarpad)
[16:06:32] <wikibugs>	 (03PS7) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494)
[16:07:04] <wikibugs>	 (03CR) 10Mathew.onipe: elasticsearch: mjolnir bulk update lag (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/495693 (https://phabricator.wikimedia.org/T214494) (owner: 10Mathew.onipe)
[16:07:10] <wikibugs>	 (03PS5) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[16:08:11] <wikibugs>	 (03CR) 10Muehlenhoff: [C: 03+1] "Looks good, see comment inline." (031 comment) [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495923 (owner: 10Jbond)
[16:11:11] <wikibugs>	 (03CR) 10Jbond: [C: 03+2] Fix shebang regression in client/debdeploy-restarts [debs/debdeploy] - 10https://gerrit.wikimedia.org/r/495923 (owner: 10Jbond)
[16:11:36] <icinga-wm>	 PROBLEM - puppet last run on netmon2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:11:49] <chaomodus>	 oh you
[16:11:55] <chaomodus>	 that's probably my fault
[16:12:38] <icinga-wm>	 PROBLEM - puppet last run on netmon1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues
[16:14:18] <moritzm>	 chaomodus: the new hiera setting isn't set and doesn't have a default value
[16:14:43] <chaomodus>	 moritzm: indeed, I'm looking
[16:14:44] <moritzm>	 fails to find profile::netbox::tokens::read_write
[16:15:13] <wikibugs>	 (03CR) 10Volans: [C: 04-1] "I didn't see the private repo commit yet, that must be sent before this can be merged." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[16:16:02] <chaomodus>	 ah
[16:16:08] <chaomodus>	 my fault
[16:16:23] <chaomodus>	 private value fail :)
[16:17:43] <wikibugs>	 (03CR) 10Mathew.onipe: "one comment" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495921 (https://phabricator.wikimedia.org/T218116) (owner: 10Gehel)
[16:20:40] <chaomodus>	 okay fixed
[16:21:36] <icinga-wm>	 RECOVERY - puppet last run on netmon2001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[16:22:14] <wikibugs>	 (03CR) 10CRusnov: "> Patch Set 8: Code-Review-1" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[16:24:48] <icinga-wm>	 PROBLEM - Check systemd state on phab1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed.
[16:26:08] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Read from key-per-property cache - https://phabricator.wikimedia.org/T218124 (10alaa_wmde)
[16:28:02] <addshore>	 elukey: it looks like there was quite an increase in memcached traffic in the last few hours, https://grafana.wikimedia.org/d/000000574/t204083-investigation?orgId=1&from=now-24h&to=now
[16:29:42] <moritzm>	 !log upgraded buster installation image to daily build from 12th of March (T213527)
[16:29:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:29:47] <stashbot>	 T213527: Prepare our base system layer for Debian buster - https://phabricator.wikimedia.org/T213527
[16:30:14] <elukey>	 addshore: yep, but it seems well distributed among all the shards 
[16:30:24] <elukey>	 not sure if it happens from time to time or not
[16:30:31] <elukey>	 what I am worried about are spikes for single hosts
[16:30:45] <elukey>	 correlating with timeouts for mcrouter/mediawiki :)
[16:30:48] <addshore>	 so, from the initial ticket though, the pipes on those machines are 1Gbps? and the shard with that key on is pretty close to that>/
[16:30:49] <addshore>	 ?
[16:31:09] <addshore>	 it actually looks like it saturated it @ https://grafana.wikimedia.org/d/000000574/t204083-investigation?orgId=1&from=1552402857623&to=1552406155715
[16:31:43] <addshore>	 jouncebot: now
[16:31:43] <jouncebot>	 For the next 0 hour(s) and 28 minute(s): Puppet SWAT(Max 6 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1600)
[16:31:54] <elukey>	 addshore: the problem seems to be located in mc1022, where the wikidata key is stored (slab 140)
[16:32:01] <elukey>	 the baseline for that is really big
[16:32:13] <elukey>	 so anything that adds on top brings the host close to the saturation
[16:32:23] <elukey>	 (and yep we have 1G interfaces)
[16:32:47] <addshore>	 Yeh, so we picked up the 2 tickets we need to solve the problem, should be done in a week or 2
[16:32:59] <addshore>	 but i might increase the bandaid that we stuck on this issue from last year now
[16:33:40] <elukey>	 still don't have the whole context on that, need to read the task :(
[16:35:45] <addshore>	 jouncebot: next
[16:35:46] <jouncebot>	 In 0 hour(s) and 24 minute(s): Services – Graphoid / Parsoid / Citoid / ORES (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1700)
[16:36:20] <addshore>	 elukey: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Wikibase/+/495930/ basically, there is an on server cache for that cache key too, I'll double it again for now
[16:36:42] <addshore>	 If noone is doing puppet swat now then I'll do it right now
[16:37:39] <godog>	 no go ahead addshore, puppet swat isn't on
[16:37:43] <godog>	 today that is
[16:37:50] <addshore>	 ack! okay, will do it now before the next slot
[16:38:04] <icinga-wm>	 RECOVERY - puppet last run on netmon1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures
[16:39:32] <elukey>	 addshore: there are two issues that I am seeing though (i'll comment in the task) - 1) the volume of sent traffic generated by GETs 2) the bursts of SETs when the key is evicted or expires, that probably comes from multiple appservers at once
[16:39:48] <elukey>	 the 2) generates the timeouts for mw/mcrouter
[16:39:53] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Smalyshev) >  I'm not sure I understand why a If-Modified-Since: would not work.   How would you see it working? In Varnish, it...
[16:42:17] <addshore>	 elukey: aaah, so the sets are causing the mcrouter timeout
[16:42:32] <logmsgbot>	 !log addshore@deploy1001 Synchronized php-1.33.0-wmf.20/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: T97368 Double on server cache for PropertyInfoStore (duration: 00m 57s)
[16:42:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:42:36] <stashbot>	 T97368: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368
[16:43:27] <wikibugs>	 (03CR) 10Gehel: "very minor comments inline, otherwise LGTM." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/494499 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe)
[16:43:32] <elukey>	 addshore: yes this is what I am seeing from the metrics, we have been battling with a similar issue in T203786
[16:43:33] <stashbot>	 T203786: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786
[16:43:40] <wikibugs>	 10Operations, 10Cloud-VPS, 10Toolforge, 10LDAP, 10Patch-For-Review: LDAP server running out of memory frequently and disrupting Cloud VPS clients - https://phabricator.wikimedia.org/T217280 (10bd808)
[16:43:48] <logmsgbot>	 !log addshore@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: T97368 Double on server cache for PropertyInfoStore (duration: 00m 55s)
[16:43:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:44:34] <addshore>	 elukey: okay, well, the things i just synced should decrease the # of reads from the key
[16:44:55] <wikibugs>	 (03CR) 10Gehel: elasticsearch: deploy elasticsearch config for ES6 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/495921 (https://phabricator.wikimedia.org/T218116) (owner: 10Gehel)
[16:45:00] <elukey>	 addshore: is it leveraging APC rather than memcached?
[16:45:03] <addshore>	 they should add a bit more randomness regarding how many servers realize the key is not there and thus try to set it too
[16:45:05] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Smalyshev) > it at least make sense to raise the issue so that the WDQS use case is addressed if it can be addressed  For that,...
[16:45:24] <addshore>	 elukey: yes, so there is 20 seconds of apc cache now, before hitting memcached, the previous band aid was 20 seconds
[16:45:28] <addshore>	 *was 10 seconds
[16:45:34] <elukey>	 ack understood
[16:45:58] <elukey>	 addshore: are you guys using WANObjectCache? There are a lot of things in there to avoid SET/CAS thundering herd
[16:46:08] <addshore>	 *looks*
[16:46:22] <addshore>	 elukey: no, just bag o stuff currently
[16:46:25] <elukey>	 https://doc.wikimedia.org/mediawiki-core/master/php/classWANObjectCache.html
[16:46:29] <elukey>	 ahhhh
[16:46:42] <elukey>	 so one good thing to do could be to see if --^ fits
[16:47:06] <elukey>	 we moved the translate extension to WANObjectCache (was using directly bag of stuff)
[16:47:15] <elukey>	 for the issue that I linked above
[16:47:31] <addshore>	 I imagine we could probably use that without to much issue, could you write a comment in the ticket regarding maybe using that and write what you said about the SET bursts?
[16:48:05] <elukey>	 of course, will do in a bit
[16:48:29] <wikibugs>	 10Operations, 10DC-Ops, 10hardware-requests: Request spare systems to test ipmi password reset cookbook - https://phabricator.wikimedia.org/T218117 (10RobH)
[16:48:36] <addshore>	 elukey: thanks!
[16:49:03] <wikibugs>	 (03PS1) 10Ema: varnish-backend-restart: do not spam cron [puppet] - 10https://gerrit.wikimedia.org/r/495936 (https://phabricator.wikimedia.org/T217893)
[16:50:24] <wikibugs>	 (03PS4) 10Paladox: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/495424
[16:52:27] <wikibugs>	 (03CR) 10Paladox: [V: 03+2 C: 03+2] Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/495424 (owner: 10Paladox)
[16:53:57] <addshore>	 elukey: interestingly doubling the ttl for the apc cache didn't seem to decrease traffic from that host all that much
[16:53:59] <davidwbarratt>	 would someone (with access) mind giving a +2 to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/495917 ?
[16:54:17] <wikibugs>	 (03CR) 10Addshore: [C: 03+2] Enable Partial Blocks on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495917 (https://phabricator.wikimedia.org/T217278) (owner: 10Dbarratt)
[16:54:22] <addshore>	 davidwbarratt: I can :)
[16:54:24] <davidwbarratt>	 addshore thanks!
[16:54:55] <elukey>	 addshore: :(
[16:54:59] <wikibugs>	 (03PS9) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[16:55:19] <wikibugs>	 (03Merged) 10jenkins-bot: Enable Partial Blocks on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495917 (https://phabricator.wikimedia.org/T217278) (owner: 10Dbarratt)
[16:55:59] <elukey>	 addshore: I've ran memkeys (like top for memcached) and sort for bandwidth usage, wikibase_shared/1_33_0-wmf_20-wikidatawiki-hhvm:CacheAwarePropertyInfoStore seems to be the first
[16:56:15] <elukey>	 but in theory with 20s of APC cache it should decrease right?
[16:56:22] <elukey>	 or is it a different key?
[16:56:28] <addshore>	 elukey: yes, unless something isn't working somewhere
[16:57:54] <logmsgbot>	 !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: beta only (duration: 00m 53s)
[16:57:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[16:58:03] <addshore>	 bah, wait, I think the apc cache might actually be needed somewhere else too
[16:59:06] <addshore>	 indeed, there is 1 other place to touch
[16:59:36] <wikibugs>	 (03CR) 10jenkins-bot: Enable Partial Blocks on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495917 (https://phabricator.wikimedia.org/T217278) (owner: 10Dbarratt)
[17:00:04] <jouncebot>	 cscott, arlolra, subbu, halfak, and Amir1: Time to snap out of that daydream and deploy Services – Graphoid / Parsoid / Citoid / ORES. Get on with it. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1700).
[17:01:59] <elukey>	 need to go afk for 10/15 mins, will be back to check
[17:02:09] <addshore>	 [=
[17:02:30] <jbond42>	 !log rolling update of debdeploy
[17:02:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:06:57] <addshore>	 jouncebot: next
[17:06:57] <jouncebot>	 In 0 hour(s) and 53 minute(s): Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1800)
[17:07:03] <wikibugs>	 (03PS1) 10Gehel: wdqs: add custom domains to the prometeus whitelist [puppet] - 10https://gerrit.wikimedia.org/r/495944
[17:07:28] <wikibugs>	 (03PS2) 10Gehel: wdqs: add custom domains to the prometeus whitelist [puppet] - 10https://gerrit.wikimedia.org/r/495944 (https://phabricator.wikimedia.org/T217945)
[17:09:20] <wikibugs>	 (03CR) 10Smalyshev: [C: 03+1] wdqs: add custom domains to the prometeus whitelist [puppet] - 10https://gerrit.wikimedia.org/r/495944 (https://phabricator.wikimedia.org/T217945) (owner: 10Gehel)
[17:09:32] <wikibugs>	 (03CR) 10Gehel: [C: 03+2] wdqs: add custom domains to the prometeus whitelist [puppet] - 10https://gerrit.wikimedia.org/r/495944 (https://phabricator.wikimedia.org/T217945) (owner: 10Gehel)
[17:10:13] <logmsgbot>	 !log addshore@deploy1001 Synchronized php-1.33.0-wmf.20/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: T97368 Increase APC cache for PropertyInfoLookup from 15 to 20s (duration: 00m 57s)
[17:10:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:10:16] <stashbot>	 T97368: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368
[17:11:21] <logmsgbot>	 !log addshore@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/Wikibase/repo/includes/Store/Sql/SqlStore.php: T97368 Increase APC cache for PropertyInfoLookup from 15 to 20s (duration: 00m 55s)
[17:11:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[17:13:01] <wikibugs>	 (03PS17) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[17:15:15] <wikibugs>	 (03CR) 10Volans: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[17:16:23] <wikibugs>	 (03PS10) 10CRusnov: Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717
[17:17:06] <addshore>	 elukey: well, i raised one apc ttl from 10 to 20s, and the other from 15 to 20s, raising too much more and people will start to notice on wiki
[17:17:54] <elukey>	 addshore: sure :)
[17:17:55] <wikibugs>	 (03CR) 10CRusnov: [C: 03+2] Add certs for RAPI and adjust config to use them [puppet] - 10https://gerrit.wikimedia.org/r/495717 (owner: 10CRusnov)
[17:18:03] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10BBlack) I think it would be better, from my perspective, to really understand the use-cases better (which I don't).  Why do the...
[17:18:05] <elukey>	 addshore: is there a possible long term fix for this ?
[17:18:33] <addshore>	 elukey: yes, we are going to stop using that key so much, and split the data up into 1 kery per property in memcache also
[17:18:48] <addshore>	 those are the tickets the team just picked up today
[17:19:47] <elukey>	 ahhh nice
[17:20:01] <addshore>	 the full cache eky will still be used, bu no where near as much
[17:20:24] <elukey>	 are the task linked in the one in which we are discussing?
[17:20:32] <addshore>	 elukey: yup,
[17:20:35] <elukey>	 super
[17:20:38] <elukey>	 going to add my comments
[17:20:56] <addshore>	 elukey: these spikes are not nice at all, in the past 3 hours that host shot up to 1Gbps twice
[17:21:29] <wikibugs>	 10Operations, 10monitoring: LibreNMS upgrade to 1.49 - https://phabricator.wikimedia.org/T207706 (10ayounsi)
[17:22:28] <addshore>	 is the spike definitely the wikibase key?
[17:24:09] <wikibugs>	 (03PS18) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[17:24:53] <wikibugs>	 10Operations, 10Traffic, 10Wikidata, 10Wikidata-Query-Service: Reduce / remove the aggessive cache busting behaviour of wdqs-updater - https://phabricator.wikimedia.org/T217897 (10Smalyshev) > Why do these remote clients need "realtime" (no staleness) fetches of Q items?   Because that's what Query Service...
[17:24:54] <elukey>	 addshore: I don't think it is only that, since most of the shards are affected. Probably it is an increase in traffic for mw, that translates to that 
[17:25:21] <wikibugs>	 (03CR) 10Ema: [C: 03+2] varnish-backend-restart: do not spam cron [puppet] - 10https://gerrit.wikimedia.org/r/495936 (https://phabricator.wikimedia.org/T217893) (owner: 10Ema)
[17:25:28] <wikibugs>	 (03PS2) 10Ema: varnish-backend-restart: do not spam cron [puppet] - 10https://gerrit.wikimedia.org/r/495936 (https://phabricator.wikimedia.org/T217893)
[17:26:04] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 5 others: Read from key-per-property cache - https://phabricator.wikimedia.org/T218124 (10Addshore) p:05Triage→03High
[17:29:55] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368 (10Addshore) >>! In T97368#5017978, @elukey wrote: > Another big SET burs...
[17:33:28] <elukey>	 addshore: can you tell me a bit more about what happened during the past deployment?
[17:33:31] <elukey>	 https://grafana.wikimedia.org/d/000000316/memcache?panelId=45&fullscreen&orgId=1&from=now-7d&to=now
[17:34:02] <addshore>	 elukey: I saw that was I deployed, but I'm pretty certain that it is not related to the deployment
[17:34:13] <addshore>	 I think it is just another spike, similar to the one a few hours ago
[17:34:28] <elukey>	 nono wait this is a week ago
[17:34:40] <elukey>	 when the traffic started to rise for mc1022
[17:34:52] <addshore>	 aah, gotcha *looks*
[17:34:52] <elukey>	 I can see a traffic shift from mc1021 to mc1022
[17:34:58] <addshore>	 yes, so that happens every week
[17:35:04] <elukey>	 did you guys rename the keys?
[17:35:05] <addshore>	 as the mw version is in the cache key
[17:35:11] <elukey>	 ahhhhhh
[17:35:21] <elukey>	 this is why it moved
[17:35:27] <addshore>	 so it rotates hosts constantly
[17:35:37] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10CDanis) Thank you @The_RedBurn, that was helpful.  I loaded u...
[17:36:01] <elukey>	 lovely
[17:36:06] <elukey>	 so it was there before
[17:36:13] <elukey>	 but on a different shard
[17:36:38] <addshore>	 elukey: this is the funny one, as you can see it bouncing https://grafana.wikimedia.org/d/000000574/t204083-investigation?orgId=1&from=now-6M&to=now
[17:37:44] <addshore>	 elukey: https://phabricator.wikimedia.org/T97368#4640195 shows the drop in traffic when we first added the APC caching too
[17:38:21] <wikibugs>	 10Operations, 10Cloud-VPS, 10Toolforge, 10LDAP, 10Patch-For-Review: LDAP server running out of memory frequently and disrupting Cloud VPS clients - https://phabricator.wikimedia.org/T217280 (10Andrew)
[17:38:23] <addshore>	 back at the start of october, but it looks like the traffic has slowly increased since then anyway
[17:38:58] <andrewbogott>	 moritzm (or anyone else interested in ldap): I'd appreciate your thoughts on T218133.  Assuming the idea isn't controversial, I think I mostly know how to do it but I'll need some advice and reviews and such.
[17:38:58] <stashbot>	 T218133: Put our ldap servers behind LVS - https://phabricator.wikimedia.org/T218133
[17:40:36] <wikibugs>	 (03PS6) 10Ammarpad: Create new protection levels for dewiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495918 (https://phabricator.wikimedia.org/T216885)
[17:42:21] <wikibugs>	 10Operations, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 8 others: Fix inefficient CacheAwarePropertyInfoStore memcached access pattern - https://phabricator.wikimedia.org/T97368 (10elukey) To summarize what I am currently seeing:  * after the last mw...
[17:43:05] <elukey>	 addshore: yeah that bouncing is not great since it creates variability in the shards
[17:43:18] <addshore>	 elukey: yes, it's all super silly
[17:43:19] <addshore>	 :P
[17:43:27] <elukey>	 ahahah okok
[17:44:10] <addshore>	 but, if your not worried about anything bad happening in the next 2 weeks, then I am not either and we can just wait for the patches to get worked on and merged :)
[17:44:21] <addshore>	 but if you are more worried, let me know!
[17:49:30] <elukey>	 yeah let's see how it goes, I'd ask if possible to prioritize the work (but not rush!)
[17:49:51] <elukey>	 is there any sort of big band aid that we can add in case of fire
[17:49:51] <elukey>	 ?
[17:50:00] <elukey>	 say in 2h everything burns :D
[17:50:01] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Patch-For-Review, 10User-herron: Migrate at least 3 existing Logstash inputs and associated producers to the new Kafka-logging pipeline, and remove the associated non-Kafka Logstash inputs - https://phabricator.wikimedia.org/T213899 (10herron)
[17:50:17] <elukey>	 or in two days
[17:50:59] <elukey>	 addshore: --^
[17:51:45] <addshore>	 elukey: yes, the first band aid i would consider would be bumping that apc cache from 20 to 60, should be a fairly big band aid
[17:52:31] <elukey>	 all right, let's hope we don't have to do it :)
[17:52:34] <addshore>	 the only other bandaid really (a little more like a bandage) would be to duplicate the cache key, and have 2 of them, so the traffic would be split between 2 hosts
[17:53:01] <elukey>	 :D
[17:53:13] <elukey>	 all right, thanks a lot for the moment :)
[17:54:16] <elukey>	 (going afk!)
[18:00:05] <jouncebot>	 Deploy window Pre MediaWiki train sanity break (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1800)
[18:06:26] <wikibugs>	 10Operations, 10ops-eqiad, 10monitoring, 10Patch-For-Review: icinga1001 crashed - https://phabricator.wikimedia.org/T214760 (10RobH) a:05Cmjohnson→03Volans @volans: Can you return this server to service so we can see if the problem has been resolved?
[18:07:59] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10CDanis) Talked to @ema and he thinks this is related to T2153...
[18:09:35] <wikibugs>	 10Operations, 10ops-eqiad, 10monitoring, 10Patch-For-Review: icinga1001 crashed - https://phabricator.wikimedia.org/T214760 (10Volans) @RobH Icinga runs on both hosts and generates the same load, being active or passive changes very little. I've been monitoring this host with our beta icinga meta-monitorin...
[18:22:14] <icinga-wm>	 PROBLEM - Memory correctable errors -EDAC- on wtp2013 is CRITICAL: 195 ge 4 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=wtp2013&var-datasource=codfw+prometheus/ops
[18:26:06] <wikibugs>	 10Operations, 10ops-eqiad, 10decommission, 10netops: Decommission asw-b-eqiad - https://phabricator.wikimedia.org/T208788 (10ayounsi)
[18:27:48] <wikibugs>	 (03PS1) 10Mholloway: Deploy WikimediaEditorTasks to Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495957 (https://phabricator.wikimedia.org/T218137)
[18:28:21] <wikibugs>	 10Operations, 10ops-eqiad, 10decommission, 10Patch-For-Review: Decommission asw2-a5-eqiad - https://phabricator.wikimedia.org/T217383 (10ayounsi)
[18:30:09] <wikibugs>	 10Operations, 10ops-eqiad, 10decommission, 10netops, 10Patch-For-Review: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10ayounsi)
[18:32:47] <wikibugs>	 (03PS2) 10Petar.petkovic: Add publish restrictions config for enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495677 (https://phabricator.wikimedia.org/T217237)
[18:34:53] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on sodium - https://phabricator.wikimedia.org/T202705 (10RobH) Ok, the steps previously taken on this, and the next steps to take, are as follows:  Past steps: * sodium had a failed disk 6 months ago, and Chris opened request WO10392987. * Dell sent a replacement disk,...
[18:37:41] <wikibugs>	 10Operations, 10ops-eqiad: Degraded RAID on sodium - https://phabricator.wikimedia.org/T212010 (10RobH) Ok, the steps previously taken on this, and the next steps to take, are as follows:  Past steps: * sodium had a failed disk 6 months ago on T202705, and Chris opened request WO10392987. * Dell sent a replace...
[18:38:00] <wikibugs>	 (03PS1) 10Paladox: Gerrit 2.15.11 release [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/495960 (https://phabricator.wikimedia.org/T214359)
[18:38:13] <wikibugs>	 (03PS2) 10Paladox: Gerrit 2.15.11 release [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/495960 (https://phabricator.wikimedia.org/T214359)
[18:38:17] <wikibugs>	 (03CR) 10Volans: "I've just skimmed the tests and as far as the code itself goes I'm missing surely some context, but in general looks in line with the rest" (036 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[18:40:13] <wikibugs>	 10Operations, 10ops-eqiad, 10DC-Ops: icinga1001 mysterious reboots - https://phabricator.wikimedia.org/T210108 (10RobH)
[18:40:22] <wikibugs>	 10Operations, 10ops-eqiad, 10monitoring, 10Patch-For-Review: icinga1001 crashed - https://phabricator.wikimedia.org/T214760 (10RobH) 05Open→03Stalled p:05High→03Low
[18:43:13] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+1] "perfect, I'll get this setup on a test instance, thanks!" [software/gerrit] (deploy/wmf/stable-2.15) - 10https://gerrit.wikimedia.org/r/495960 (https://phabricator.wikimedia.org/T214359) (owner: 10Paladox)
[18:43:56] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) >>! In T215411#4946625, @jijiki wrote: > ` > [Tue Feb 12 06:13:31 2019] mce: [Hardware Error]: Machine check events logged > [Tue Feb 12 06:13:31 2019] EDAC sbridge MC1: HANDLI...
[18:45:50] <wikibugs>	 (03PS1) 10Herron: logstash: migrate mediawiki syslogs to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/495962 (https://phabricator.wikimedia.org/T213899)
[18:50:54] <wikibugs>	 (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler1002/15068/mw1239.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/495962 (https://phabricator.wikimedia.org/T213899) (owner: 10Herron)
[18:55:18] <wikibugs>	 (03PS19) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[18:58:29] <wikibugs>	 (03PS20) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[18:59:30] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: aptrepo: openstack-mitaka-jessie: don't import systemd [puppet] - 10https://gerrit.wikimedia.org/r/495969 (https://phabricator.wikimedia.org/T216497)
[19:00:04] <jouncebot>	 Deploy window MediaWiki train - Americas version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T1900)
[19:00:04] <jouncebot>	 niedzielski: A patch you scheduled for MediaWiki train - Americas version is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[19:00:22] <niedzielski>	 o/ I am here!
[19:00:43] <niedzielski>	 mother of bugs and breaker of prod, etc, etc
[19:01:38] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 03+2] aptrepo: openstack-mitaka-jessie: don't import systemd [puppet] - 10https://gerrit.wikimedia.org/r/495969 (https://phabricator.wikimedia.org/T216497) (owner: 10Arturo Borrero Gonzalez)
[19:07:09] <robh>	 !log rebooting thumbor1004 for memory troubleshooting via T215411
[19:07:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:07:14] <stashbot>	 T215411: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411
[19:07:18] <icinga-wm>	 PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[19:08:01] <Pchelolo>	 niedzielski: hey! me and ottomata wanna deploy a config change as well, ping us when you done please
[19:09:42] <icinga-wm>	 RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[19:09:46] <arturo>	 !log T216497 manually delete systemd 230-7~bpo8+2 from jessie-wikimedia/openstack-mitaka-jessie
[19:09:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:09:49] <stashbot>	 T216497: CloudVPS: workaround archival of jessie-backports repo - https://phabricator.wikimedia.org/T216497
[19:10:10] <icinga-wm>	 PROBLEM - Host thumbor1004 is DOWN: PING CRITICAL - Packet loss = 100%
[19:10:15] <niedzielski>	 Pchelolo: dang, i keep screwing up today. this isn't the swat window. yeesh. i will reschedule my patch for evening swat. all you
[19:11:18] <Pchelolo>	 niedzielski: it's ok to deploy now according to releng, if you still want to
[19:12:09] <niedzielski>	 Pchelolo: i don't have swat permissions
[19:12:25] <Pchelolo>	 oh.. ok, ottomata let's do it!
[19:13:08] <ottomata>	 k let's!
[19:13:17] <wikibugs>	 (03PS2) 10Ottomata: Enable api-request logging into kafka for group0. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495906 (owner: 10Ppchelko)
[19:14:13] <arturo>	 !log T216497 manually delete libpam-systemd and libsystemd0 230-7~bpo8+2 from jessie-wikimedia/openstack-mitaka-jessie
[19:14:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:14:30] <wikibugs>	 (03CR) 10Ottomata: [C: 03+2] Enable api-request logging into kafka for group0. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495906 (owner: 10Ppchelko)
[19:16:47] <logmsgbot>	 !log otto@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Enabling api-request logging to eventgate-analytics for group0 wikis (duration: 01m 01s)
[19:16:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:17:11] <Pchelolo>	 ottomata: I see events!!
[19:17:15] <wikibugs>	 (03PS1) 10Herron: rsyslog: add netdev_kafka_relay compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/495980 (https://phabricator.wikimedia.org/T213899)
[19:17:59] <ottomata>	 YEAHHH me too!
[19:18:44] <wikibugs>	 (03CR) 10jenkins-bot: Enable api-request logging into kafka for group0. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495906 (owner: 10Ppchelko)
[19:19:41] <ottomata>	 very cool!
[19:20:19] <ottomata>	 https://grafana.wikimedia.org/d/POYzU8rmz/eventgate-analytics?refresh=1m&orgId=1
[19:20:20] <ottomata>	 :)
[19:20:27] <Pchelolo>	 ottomata: 10 events/s!
[19:22:21] <Pchelolo>	 ottomata: there's some errors in the logstash, looking
[19:23:31] <wikibugs>	 (03CR) 10Herron: "https://puppet-compiler.wmflabs.org/compiler1002/15069/lithium.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/495980 (https://phabricator.wikimedia.org/T213899) (owner: 10Herron)
[19:25:14] <wikibugs>	 10Operations, 10Traffic, 10VisualEditor, 10Wikimedia-Apache-configuration: Visual Editor gets stuck opening article (net::ERR_SPDY_PROTOCOL_ERROR 200/Loading failed for the <script> with source ...) - https://phabricator.wikimedia.org/T213214 (10The_RedBurn) That's great to hear!  I did notice that shift +...
[19:27:33] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) a:05RobH→03Cmjohnson Ok, updating this after IRC discussion for clarity.  This system has had repeated memory errors reported above, and the SEL previously showed errors on...
[19:29:32] <icinga-wm>	 PROBLEM - citoid endpoints health on scb1003 is CRITICAL: /api (bad URL) timed out before a response was received
[19:29:32] <icinga-wm>	 PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (bad URL) timed out before a response was received
[19:30:30] <wikibugs>	 (03PS2) 10Herron: rsyslog: add netdev_kafka_relay compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/495980 (https://phabricator.wikimedia.org/T213899)
[19:30:51] <herron>	 I’ll try bouncing citoid
[19:31:48] <herron>	 !log restarted citoid on scb1003
[19:31:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:31:56] <icinga-wm>	 RECOVERY - citoid endpoints health on scb1003 is OK: All endpoints are healthy
[19:32:57] <herron>	 scb1003 citoid[39417]: FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
[19:33:07] <herron>	 looks to be why it alerted
[19:34:16] <icinga-wm>	 RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy
[19:34:42] <icinga-wm>	 RECOVERY - Host thumbor1004 is UP: PING OK - Packet loss = 0%, RTA = 36.22 ms
[19:37:32] <wikibugs>	 (03PS2) 10Arturo Borrero Gonzalez: openstack: drop -t jessie-backports for openstack component installs in stretch [puppet] - 10https://gerrit.wikimedia.org/r/495893 (https://phabricator.wikimedia.org/T216497)
[19:38:40] <wikibugs>	 10Operations, 10ops-eqiad, 10Thumbor, 10serviceops: thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10RobH) Ok, system is powered back on for now.  The next steps are as follows:  * @cmjohnson to sync with @jijiki tomorrow before powering down system. ** this typically only involves...
[19:41:16] <icinga-wm>	 PROBLEM - Request latencies on chlorine is CRITICAL: instance=10.64.0.45:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[19:43:50] <icinga-wm>	 PROBLEM - Request latencies on acrux is CRITICAL: instance=10.192.0.93:6443 verb=CONNECT https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[19:49:26] <wikibugs>	 (03PS1) 10Ppchelko: Enable api-request logging into kafka for group1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/495983
[19:49:48] <icinga-wm>	 RECOVERY - Request latencies on acrux is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[19:50:52] <icinga-wm>	 RECOVERY - Request latencies on chlorine is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api
[20:16:52] <dmaza>	 I'm making some changes to https://gerrit.wikimedia.org/r/plugins/gitiles/operations/mediawiki-config/+/master/wmf-config/CommonSettings.php and changing/adding some things dynamically 
[20:16:52] <dmaza>	 Is there a way to debug my changes other than running the portion I'm modifying as a separate script?
[20:17:55] <hauskatze>	 x-wikimedia-debug?
[20:18:28] <dmaza>	 hauskatze is taht the browser extension?
[20:18:43] <dmaza>	 I'm trying to debug/test before I submit a patch
[20:19:10] <hauskatze>	 dmaza: yes & ah, no patch to test; then I'm not sure.
[20:19:25] <hauskatze>	 I was going to say mw-vagrant but perhaps not
[20:20:59] <dmaza>	 let me look.. maybe
[20:34:02] <wikibugs>	 (03PS3) 10Cwhite: expose status file mod time [debs/prometheus-icinga-exporter] - 10https://gerrit.wikimedia.org/r/490639
[20:35:47] <wikibugs>	 (03CR) 10Cwhite: [C: 03+2] expose status file mod time [debs/prometheus-icinga-exporter] - 10https://gerrit.wikimedia.org/r/490639 (owner: 10Cwhite)
[20:46:48] <wikibugs>	 (03PS1) 10Tpt: Install xauth on Toolforge Stretch instances [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699)
[20:49:28] <shdubsh>	 !log manually upgrade prometheus-icinga-exporter to 0.5 on standby icinga
[20:49:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:49:51] <wikibugs>	 (03PS1) 10Andrew Bogott: Service name and IPs for ldap-behind-lvs [dns] - 10https://gerrit.wikimedia.org/r/496007 (https://phabricator.wikimedia.org/T218133)
[20:50:01] <wikibugs>	 (03PS1) 10Tpt: Use tesseract-ocr-all instead of a list of most Tesseract packages [puppet] - 10https://gerrit.wikimedia.org/r/496008 (https://phabricator.wikimedia.org/T218151)
[20:54:01] <wikibugs>	 (03CR) 10Tpt: "It's a blocker for Wsexport migration to Stretch. It would be great for this change to be deployed before migration deadline." [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699) (owner: 10Tpt)
[20:55:30] <wikibugs>	 (03PS2) 10Tpt: Install xauth on Toolforge Stretch instances [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699)
[21:18:42] <wikibugs>	 (03PS1) 10Herron: logstash: add tcp json_lines localhost compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/496021 (https://phabricator.wikimedia.org/T213899)
[21:18:45] <wikibugs>	 (03PS1) 10Herron: logstash: add udp json logback localhost compatability endpoint [puppet] - 10https://gerrit.wikimedia.org/r/496022 (https://phabricator.wikimedia.org/T213899)
[21:39:23] <wikibugs>	 (03CR) 10Alex Monk: "I think the context is the search operation would presumably allow the use of File's ensure => directory and recurse => true, thereby simp" [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[21:58:31] <wikibugs>	 (03CR) 10Alex Monk: acme-chief-api: Add support for puppet HTTP API search operation (033 comments) [software/acme-chief] - 10https://gerrit.wikimedia.org/r/494957 (https://phabricator.wikimedia.org/T207295) (owner: 10Vgutierrez)
[22:13:32] <wikibugs>	 (03PS1) 10MarcoAurelio: WIP: note table exclusion [puppet] - 10https://gerrit.wikimedia.org/r/496063
[22:15:56] <wikibugs>	 (03PS2) 10MarcoAurelio: maintain-views: Note explicit exclusion of `oathauth_users` from replicas [puppet] - 10https://gerrit.wikimedia.org/r/496063 (https://phabricator.wikimedia.org/T218165)
[22:16:09] <Platonides>	 18
[22:18:13] <wikibugs>	 (03PS1) 10Andrew Bogott: Add an lvs service in front of our two ldap servers [puppet] - 10https://gerrit.wikimedia.org/r/496065 (https://phabricator.wikimedia.org/T218133)
[22:22:39] <wikibugs>	 (03CR) 10Base: [C: 03+1] maintain-views: Note explicit exclusion of `oathauth_users` from replicas [puppet] - 10https://gerrit.wikimedia.org/r/496063 (https://phabricator.wikimedia.org/T218165) (owner: 10MarcoAurelio)
[22:23:19] <wikibugs>	 (03CR) 10BryanDavis: [C: 03+1] Install xauth on Toolforge Stretch instances [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699) (owner: 10Tpt)
[22:30:50] <wikibugs>	 (03CR) 10Bstorm: "The following NEW packages will be installed:" [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699) (owner: 10Tpt)
[22:31:00] <wikibugs>	 (03CR) 10Bstorm: [C: 03+2] Install xauth on Toolforge Stretch instances [puppet] - 10https://gerrit.wikimedia.org/r/496004 (https://phabricator.wikimedia.org/T215699) (owner: 10Tpt)
[22:40:01] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 04-2] "Doing cross-dc balancing isn't going to work; we'll need to build out some more ldap servers." [puppet] - 10https://gerrit.wikimedia.org/r/496065 (https://phabricator.wikimedia.org/T218133) (owner: 10Andrew Bogott)
[22:40:35] <wikibugs>	 (03PS1) 10Smalyshev: Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068
[22:41:34] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068 (owner: 10Smalyshev)
[22:46:33] <wikibugs>	 (03PS2) 10Smalyshev: Fix builder class definition for WBCS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/496068
[22:47:23] <SMalyshev>	 oh come on hhvm segfaults again??
[23:00:04] <jouncebot>	 addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: I, the Bot under the Fountain, allow thee, The Deployer, to do Evening SWAT (Max 6 patches) deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190312T2300).
[23:00:04] <jouncebot>	 niedzielski: A patch you scheduled for Evening SWAT (Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[23:00:23] <niedzielski>	 I am here!
[23:03:17] <wikibugs>	 10Operations, 10ops-eqiad, 10serviceops, 10HHVM: mw1272 crashed: Bad page map in process hhvm - https://phabricator.wikimedia.org/T211668 (10RobH) 05Open→03Resolved
[23:05:02] <thcipriani>	 I can SWAT
[23:05:18] <niedzielski>	 thanks thcipriani 
[23:26:13] <thcipriani>	 niedzielski: your change is live on mwdebug1002 check please
[23:26:26] <niedzielski>	 on my way
[23:28:52] <wikibugs>	 (03CR) 10Volans: [C: 04-1] "Almost there, just some small bug to fix, see inline." (034 comments) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:30:03] <niedzielski>	 ok i see responses coming back from mwdebug1002.eqiad.wmnet. the settings page is not broken
[23:30:12] <niedzielski>	 thcipriani: looks good to me!
[23:30:20] <thcipriani>	 niedzielski: okie doke, going live
[23:30:46] <niedzielski>	 👍
[23:30:59] <wikibugs>	 (03PS17) 10Jcrespo: mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292)
[23:32:04] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] mariadb-backup: Deploy new snapshot cycle to cumin and provisioning hosts [puppet] - 10https://gerrit.wikimedia.org/r/494899 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo)
[23:32:55] <wikibugs>	 10Puppet, 10Beta-Cluster-Infrastructure: puppetmaster config in deployment-prep may be inadvertently breaking store,logstash reports? - https://phabricator.wikimedia.org/T218175 (10Krenair)
[23:33:02] <logmsgbot>	 !log thcipriani@deploy1001 Synchronized php-1.33.0-wmf.21/extensions/MobileFrontend/includes/specials/SpecialMobileOptions.php: SWAT: [[gerrit:495907|Fix: undefined locals in SpecialMobileOptions.setJsConfigVars()]] T218098 (duration: 00m 57s)
[23:33:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[23:33:06] <stashbot>	 T218098: Call to a member function addJsConfigVars() on a non-object (null) - https://phabricator.wikimedia.org/T218098
[23:33:12] <thcipriani>	 ^ niedzielski should be live everywhere now
[23:33:18] <thcipriani>	 thanks for unblocking train :)
[23:33:49] <niedzielski>	 sweet. thanks thcipriani ! things still look good to me on mw1325.eqiad.wmnet
[23:34:04] <thcipriani>	 great, glad to hear it :)
[23:34:59] <wikibugs>	 (03PS21) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[23:35:24] <wikibugs>	 (03CR) 10CRusnov: Add ganeti->netbox sync script (039 comments) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:36:45] <wikibugs>	 (03PS22) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[23:41:18] <wikibugs>	 (03CR) 10Volans: "I missed one thing..." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:42:42] <wikibugs>	 (03CR) 10CRusnov: "> Patch Set 16:" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/493348 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:44:21] <wikibugs>	 (03CR) 10Volans: Add ganeti->netbox sync script (031 comment) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:50:52] <wikibugs>	 (03PS23) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)
[23:51:47] <wikibugs>	 (03CR) 10CRusnov: "I have verified that the script works as is." (031 comment) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov)
[23:53:37] <wikibugs>	 (03PS24) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229)