[00:01:21] (03PS3) 10CRusnov: Add ganeti->netbox sync script [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) [00:07:10] (03PS1) 10CRusnov: Update to upstream v2.5.7 tag. [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492577 [00:33:39] 10Operations, 10Operations-Software-Development: Netbox: cable termination names report - https://phabricator.wikimedia.org/T216469 (10crusnov) Just a note the models are a bit confused here. The termination is identified in the cable, i'm not sure that there are ports without cables or if that is important, b... [01:14:18] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 30.77% of data above the critical threshold [140.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:21:40] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [01:31:06] PROBLEM - Disk space on labstore1004 is CRITICAL: DISK CRITICAL - free space: /srv/tools 473469 MB (5% inode=79%) [05:26:07] 10Operations, 10Research-Programs, 10SRE-Access-Requests, 10Patch-For-Review: access to analytics-privatedata-users for @toddleroux, @Afandian, & @RyanSteinberg - https://phabricator.wikimedia.org/T209298 (10RyanSteinberg) 05Resolved→03Open I'm reopening this so someone can take a look at @toddleroux's... [05:38:32] (03CR) 10KartikMistry: WIP: Cron to run script to purge old CX drafts (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/486454 (https://phabricator.wikimedia.org/T189091) (owner: 10KartikMistry) [05:38:50] (03PS6) 10KartikMistry: WIP: Cron to run script to purge old CX drafts [puppet] - 10https://gerrit.wikimedia.org/r/486454 (https://phabricator.wikimedia.org/T189091) [05:39:26] (03CR) 10jerkins-bot: [V: 04-1] WIP: Cron to run script to purge old CX drafts [puppet] - 10https://gerrit.wikimedia.org/r/486454 (https://phabricator.wikimedia.org/T189091) (owner: 10KartikMistry) [05:58:06] (03PS1) 10Marostegui: db-eqiad: Depool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492596 [05:59:21] (03PS2) 10Marostegui: db-eqiad: Depool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492596 [06:00:29] (03CR) 10Marostegui: [C: 03+2] db-eqiad: Depool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492596 (owner: 10Marostegui) [06:01:35] (03Merged) 10jenkins-bot: db-eqiad: Depool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492596 (owner: 10Marostegui) [06:02:48] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1104 for MySQL upgrade (duration: 00m 50s) [06:02:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:02:51] !log Stop MySQL on db1104 for mysql upgrade [06:02:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:07:33] (03CR) 10jenkins-bot: db-eqiad: Depool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492596 (owner: 10Marostegui) [06:08:27] (03PS1) 10Marostegui: db-eqiad.php: Slowly repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492597 [06:11:03] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Slowly repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492597 (owner: 10Marostegui) [06:11:59] (03Merged) 10jenkins-bot: db-eqiad.php: Slowly repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492597 (owner: 10Marostegui) [06:13:00] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool db1104 after MySQL upgrade (duration: 00m 45s) [06:13:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:18:56] (03CR) 10jenkins-bot: db-eqiad.php: Slowly repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492597 (owner: 10Marostegui) [06:20:08] (03PS1) 10Marostegui: db-eqiad.php: Repool db1104 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492598 [06:25:36] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Repool db1104 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492598 (owner: 10Marostegui) [06:27:05] (03Merged) 10jenkins-bot: db-eqiad.php: Repool db1104 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492598 (owner: 10Marostegui) [06:28:25] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Repool db1104 in API after MySQL upgrade (duration: 00m 45s) [06:28:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:28:56] PROBLEM - netbox HTTPS on netmon1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 547 bytes in 0.167 second response time [06:30:09] (03CR) 10jenkins-bot: db-eqiad.php: Repool db1104 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492598 (owner: 10Marostegui) [06:30:36] PROBLEM - Check systemd state on netmon1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:35:50] (03PS1) 10Marostegui: parsercachepurging.pp: Increase TTL from 24 days to 30 [puppet] - 10https://gerrit.wikimedia.org/r/492599 (https://phabricator.wikimedia.org/T210992) [06:37:10] (03PS1) 10Marostegui: InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) [06:37:42] RECOVERY - netbox HTTPS on netmon1002 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.639 second response time [06:38:06] RECOVERY - Check systemd state on netmon1002 is OK: OK - running: The system is fully operational [07:05:39] (03PS1) 10Marostegui: db-eqiad.php: Fully repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492601 [07:07:04] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Fully repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492601 (owner: 10Marostegui) [07:08:04] (03Merged) 10jenkins-bot: db-eqiad.php: Fully repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492601 (owner: 10Marostegui) [07:09:10] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Fully repool db1104 after MySQL upgrade (duration: 00m 45s) [07:09:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:15:42] (03CR) 10jenkins-bot: db-eqiad.php: Fully repool db1104 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492601 (owner: 10Marostegui) [07:15:47] 10Operations, 10Analytics: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) p:05Triage→03Normal [07:16:02] 10Operations, 10Analytics: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) [07:16:15] 10Operations, 10Analytics, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) [07:19:24] (03CR) 10Marostegui: mariadb: Add the option of postprocessing backups (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [07:30:07] 10Operations, 10Analytics, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) List of parent znodes in main-eqiad: ` [zk: localhost:2181(CONNECTED) 0] ls / [registry, brokers, zookeeper, yarn-leader-election, hadoop-ha, r... [07:30:46] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [07:31:44] 10Operations, 10Analytics, 10User-Elukey: Review znodes on Zookeeper cluster to possibly remove not-used data - https://phabricator.wikimedia.org/T216979 (10elukey) * Main codfw is less crowded and probably doesn't need a clean up: ` [zk: localhost:2181(CONNECTED) 0] ls / [burrow, kafka, zookeeper] [zk: lo... [07:35:24] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5 [07:35:40] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga2001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [07:35:42] PROBLEM - Text HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5 [07:37:20] seems to be a spike for graphite-labs [07:37:43] should recover soon [07:37:49] (I mean the alert) [07:40:16] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=eqiad&var-cache_type=All&var-status_type=5 [07:40:34] RECOVERY - Text HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=text&var-status_type=5 [07:46:52] (03PS2) 10Muehlenhoff: Explicitly install ruby-safe-yaml to fix Puppet Icinga check [puppet] - 10https://gerrit.wikimedia.org/r/492346 (https://phabricator.wikimedia.org/T213546) [07:48:07] (03CR) 10Muehlenhoff: [C: 03+2] Explicitly install ruby-safe-yaml to fix Puppet Icinga check [puppet] - 10https://gerrit.wikimedia.org/r/492346 (https://phabricator.wikimedia.org/T213546) (owner: 10Muehlenhoff) [08:03:25] 10Operations, 10ops-eqiad, 10serviceops, 10HHVM: mw1272 crashed: Bad page map in process hhvm - https://phabricator.wikimedia.org/T211668 (10MoritzMuehlenhoff) a:03Cmjohnson [08:04:21] 10Operations, 10ops-eqiad, 10serviceops, 10HHVM: mw1272 crashed: Bad page map in process hhvm - https://phabricator.wikimedia.org/T211668 (10MoritzMuehlenhoff) This server is still under warranty for another 6-7 weeks. [08:04:44] ACKNOWLEDGEMENT - mediawiki-installation DSH group on mw1272 is CRITICAL: Host mw1272 is not in mediawiki-installation dsh group Muehlenhoff T211668 [08:24:49] 10Operations, 10Patch-For-Review: Reallocate former image scalers - https://phabricator.wikimedia.org/T192457 (10MoritzMuehlenhoff) @Dzahn https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/485968/ removed the spare role from mw2151,but the host is still installed with role(spare) and puppet is failing:... [08:43:02] (03CR) 10Vgutierrez: [C: 03+2] gerrit: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492283 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [08:43:10] (03PS2) 10Vgutierrez: gerrit: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492283 (https://phabricator.wikimedia.org/T207389) [08:44:10] 10Operations, 10monitoring: google safe browsing icinga checks sporadic UNKNOWN due to 403 - https://phabricator.wikimedia.org/T216985 (10fgiunchedi) [08:48:24] 10Operations, 10ops-eqiad: Degraded RAID on ms-be1020 - https://phabricator.wikimedia.org/T214778 (10fgiunchedi) @Cmjohnson sounds good, let me know when you are ready to go and I'll poweroff the host. [08:49:09] <_joe_> !log generating mcrouter certificate for mw2151 T192457 [08:49:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:49:12] T192457: Reallocate former image scalers - https://phabricator.wikimedia.org/T192457 [08:52:03] !log Deploy schema change on s2 on codfw master - lag will happen on s2 codfw - T187295 [08:52:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:52:07] T187295: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 [08:56:07] 10Operations, 10Patch-For-Review: Reallocate former image scalers - https://phabricator.wikimedia.org/T192457 (10Joe) >>! In T192457#4979764, @MoritzMuehlenhoff wrote: > @Dzahn https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/485968/ removed the spare role from mw2151,but the host is still installed with... [08:58:07] (03CR) 10Jcrespo: [C: 03+1] parsercachepurging.pp: Increase TTL from 24 days to 30 [puppet] - 10https://gerrit.wikimedia.org/r/492599 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [08:58:30] (03CR) 10Jcrespo: [C: 03+1] InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [09:00:29] (03CR) 10Jcrespo: ">" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:01:02] (03CR) 10Filippo Giunchedi: [C: 04-1] "See inline" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/492408 (https://phabricator.wikimedia.org/T213708) (owner: 10Cwhite) [09:01:52] (03CR) 10Jcrespo: "> >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:02:15] (03CR) 10Marostegui: "> >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:02:40] (03PS2) 10Marostegui: parsercachepurging.pp: Increase TTL from 24 days to 30 [puppet] - 10https://gerrit.wikimedia.org/r/492599 (https://phabricator.wikimedia.org/T210992) [09:02:52] (03PS2) 10Marostegui: InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) [09:03:19] (03CR) 10Jcrespo: "> > >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:04:35] (03CR) 10Marostegui: [C: 03+2] parsercachepurging.pp: Increase TTL from 24 days to 30 [puppet] - 10https://gerrit.wikimedia.org/r/492599 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [09:04:54] (03CR) 10Marostegui: [C: 03+2] InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [09:05:52] (03Merged) 10jenkins-bot: InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [09:07:26] !log marostegui@deploy1001 Synchronized wmf-config/InitialiseSettings.php: Increase ParserCache TTL from 24 days to 30 - T210992 (duration: 00m 46s) [09:07:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:07:29] T210992: Increase parsercache keys TTL from 22 days back to 30 days - https://phabricator.wikimedia.org/T210992 [09:09:45] (03CR) 10Marostegui: "> > > >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:10:11] (03CR) 10jenkins-bot: InitialiseSettings.php: Restore TTL back to 30 days [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492600 (https://phabricator.wikimedia.org/T210992) (owner: 10Marostegui) [09:11:09] (03PS1) 10Muehlenhoff: Record extended MOU date for flemmerich [puppet] - 10https://gerrit.wikimedia.org/r/492609 [09:13:52] (03CR) 10Muehlenhoff: [C: 03+2] Record extended MOU date for flemmerich [puppet] - 10https://gerrit.wikimedia.org/r/492609 (owner: 10Muehlenhoff) [09:15:46] PROBLEM - Check systemd state on mw2151 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [09:16:03] 10Operations, 10DBA, 10Patch-For-Review, 10Performance-Team (Radar): Increase parsercache keys TTL from 22 days back to 30 days - https://phabricator.wikimedia.org/T210992 (10Marostegui) I have increased the TTL back to 30 days. Going to monitor the graphs for a few days before closing this. [09:17:29] (03PS1) 10Muehlenhoff: Partly remove access for awight [puppet] - 10https://gerrit.wikimedia.org/r/492611 [09:18:21] (03CR) 10Jcrespo: "> > > > >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:19:12] PROBLEM - nutcracker port on mw2151 is CRITICAL: connect to address 127.0.0.1 and port 11212: Connection refused [09:19:28] (03CR) 10Muehlenhoff: [C: 03+2] Partly remove access for awight [puppet] - 10https://gerrit.wikimedia.org/r/492611 (owner: 10Muehlenhoff) [09:19:52] PROBLEM - puppet last run on mw2151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[scap] [09:20:56] PROBLEM - nutcracker process on mw2151 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 112 (nutcracker), command name nutcracker [09:22:48] 10Operations, 10Thumbor, 10serviceops: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10Gilles) [09:23:03] (03CR) 10Marostegui: [C: 03+1] "> > > > > >" (031 comment) [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [09:25:06] (03CR) 10Vgutierrez: [C: 03+2] gerrit: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492284 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [09:25:13] (03PS3) 10Vgutierrez: gerrit: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492284 (https://phabricator.wikimedia.org/T207389) [09:34:58] PROBLEM - PHP7 rendering on mw2151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 404 Not Found - header X-Powered-By: PHP/7. not found on http://10.192.32.39:9005/w/health-check.php - 380 bytes in 0.001 second response time [09:35:10] <_joe_> uhm interesting [09:35:16] (03CR) 10Vgutierrez: [C: 03+2] icinga: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492285 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [09:35:29] (03PS2) 10Vgutierrez: icinga: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492285 (https://phabricator.wikimedia.org/T207389) [09:36:30] <_joe_> moritzm: ^^ installation of a new server failed, because of T216712 [09:36:31] T216712: Switch PHP 7.2 packages to an internal component - https://phabricator.wikimedia.org/T216712 [09:37:23] (03PS2) 10Muehlenhoff: postgresql: Remove support for trusty [puppet] - 10https://gerrit.wikimedia.org/r/489631 [09:40:26] PROBLEM - mediawiki-installation DSH group on mw2151 is CRITICAL: Host mw2151 is not in mediawiki-installation dsh group [09:41:32] (03CR) 10Vgutierrez: [C: 03+2] icinga: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492286 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [09:41:39] (03PS2) 10Vgutierrez: icinga: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492286 (https://phabricator.wikimedia.org/T207389) [09:42:17] (03CR) 10Muehlenhoff: [C: 03+2] postgresql: Remove support for trusty [puppet] - 10https://gerrit.wikimedia.org/r/489631 (owner: 10Muehlenhoff) [09:43:36] (03PS3) 10Vgutierrez: icinga: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492286 (https://phabricator.wikimedia.org/T207389) [09:44:18] 10Operations, 10Discovery-Search, 10Wikimedia-Logstash: Upgrade logstash plugins to 5.6.14 - https://phabricator.wikimedia.org/T216993 (10Mathew.onipe) [09:44:36] 10Operations, 10Discovery-Search, 10Wikimedia-Logstash: Upgrade logstash plugins to 5.6.14 - https://phabricator.wikimedia.org/T216993 (10Mathew.onipe) p:05Triage→03High [09:45:11] <_joe_> I'll ack mw2151 alerts once I'm done with it, please be patient :) [09:45:50] RECOVERY - PHP7 rendering on mw2151 is OK: HTTP OK: HTTP/1.1 200 OK - 322 bytes in 0.006 second response time [09:47:13] yeah, should be failing because of the pcre package conflict [09:50:40] (03CR) 10Filippo Giunchedi: "LGTM overall" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/492390 (owner: 10Herron) [09:54:18] (03CR) 10Filippo Giunchedi: [C: 04-1] "AFAICT this class is used by shinken too, which is still jessie (shinken-02.eqiad.wmflabs, +Giovanni)" [puppet] - 10https://gerrit.wikimedia.org/r/491460 (owner: 10Muehlenhoff) [09:54:49] !log Deploy schema change on db1074, this will generate lag on labsdb:s2 - T187295 [09:54:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:54:52] T187295: Apply AbuseFilter patch-fix-index - https://phabricator.wikimedia.org/T187295 [09:55:13] (03CR) 10Filippo Giunchedi: [C: 03+1] "LGTM (+Daniel)" [puppet] - 10https://gerrit.wikimedia.org/r/491780 (owner: 10CDanis) [10:01:30] (03CR) 10Vgutierrez: [C: 04-1] "please check the comments inline" (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/491972 (https://phabricator.wikimedia.org/T216681) (owner: 10Mathew.onipe) [10:01:36] (03CR) 10Marostegui: mariadb: Add the option of postprocessing backups [software/wmfmariadbpy] - 10https://gerrit.wikimedia.org/r/491818 (https://phabricator.wikimedia.org/T210292) (owner: 10Jcrespo) [10:03:50] (03PS1) 10Filippo Giunchedi: [WIP] Use mmjsonparse by default [puppet] - 10https://gerrit.wikimedia.org/r/492632 (https://phabricator.wikimedia.org/T213189) [10:07:50] (03PS1) 10Filippo Giunchedi: gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) [10:08:38] (03CR) 10jerkins-bot: [V: 04-1] gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi) [10:10:49] (03PS2) 10Filippo Giunchedi: gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) [10:11:47] (03CR) 10jerkins-bot: [V: 04-1] gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi) [10:13:47] (03PS1) 10Jbond: pin pdns-recursor to openstack-mitaka-jessie on jessie instead of jessie-backports [puppet] - 10https://gerrit.wikimedia.org/r/492635 [10:14:20] (03CR) 10jerkins-bot: [V: 04-1] pin pdns-recursor to openstack-mitaka-jessie on jessie instead of jessie-backports [puppet] - 10https://gerrit.wikimedia.org/r/492635 (owner: 10Jbond) [10:16:19] !log Depooling thumbor1001 to reimage - T214597 [10:16:22] (03CR) 10Filippo Giunchedi: "jenkins fails because of wmf-style: 10:11:44 modules/profile/manifests/gerrit/server.pp:12 wmf-style: Parameter 'log_host' of class 'profi" [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi) [10:16:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:16:23] T214597: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 [10:19:17] (03PS2) 10Jbond: update pdns-recursor pin on jessie [puppet] - 10https://gerrit.wikimedia.org/r/492635 [10:20:07] (03CR) 10DCausse: cloudelastic: Add cloudelastic configs (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/487129 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe) [10:20:56] (03CR) 10Vgutierrez: [C: 03+2] librenms: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492288 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:21:04] (03PS2) 10Vgutierrez: librenms: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492288 (https://phabricator.wikimedia.org/T207389) [10:23:27] (03PS3) 10Filippo Giunchedi: gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) [10:24:01] (03CR) 10jerkins-bot: [V: 04-1] gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi) [10:26:51] (03CR) 10Vgutierrez: [C: 03+2] librenms: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492289 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:26:58] (03PS2) 10Vgutierrez: librenms: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492289 (https://phabricator.wikimedia.org/T207389) [10:27:24] (03PS4) 10Filippo Giunchedi: gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) [10:28:40] RECOVERY - Disk space on labstore1004 is OK: DISK OK [10:30:04] jan_drewniak: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikimedia Portals Update . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190225T1030). [10:31:46] !log labstore1004 restarted nfsd and killed stuck rpc.mountd.real processed (T216988) [10:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:32:17] T216988: labstore1004 - DISK CRITICAL - free space: /srv/tools 115904 MB (1% inode=79%): - https://phabricator.wikimedia.org/T216988 [10:32:18] PROBLEM - toolschecker: toolsdb on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/toolsdb - 356 bytes in 60.076 second response time [10:32:23] (03PS1) 10Jdrewniak: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492636 (https://phabricator.wikimedia.org/T128546) [10:32:34] RECOVERY - toolschecker: toolsdb on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.103 second response time [10:32:59] 10Operations: Collate jessie-wikimedia/backports into jessie-wikimedia/main - https://phabricator.wikimedia.org/T167292 (10MoritzMuehlenhoff) 05Open→03Declined a:05MoritzMuehlenhoff→03None This doesn't seem useful at this point, it will simply vanish along with jessie installations being phased out. The... [10:33:03] 10Operations, 10Patch-For-Review: Restructure our internal repositories further - https://phabricator.wikimedia.org/T158583 (10MoritzMuehlenhoff) [10:33:49] (03CR) 10Jdrewniak: [C: 03+2] Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492636 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:34:14] (03CR) 10Muehlenhoff: [C: 03+1] update pdns-recursor pin on jessie [puppet] - 10https://gerrit.wikimedia.org/r/492635 (owner: 10Jbond) [10:34:47] (03Merged) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492636 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:35:38] 10Operations, 10Traffic, 10HTTPS: Make sure that services available for NDA-only users are using strong TLS ciphersuites - https://phabricator.wikimedia.org/T217002 (10Vgutierrez) [10:35:50] 10Operations, 10Traffic, 10HTTPS: Make sure that services available for NDA-only users are using strong TLS ciphersuites - https://phabricator.wikimedia.org/T217002 (10Vgutierrez) p:05Triage→03Normal [10:37:48] (03CR) 10Vgutierrez: [C: 03+2] lists: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492292 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:37:57] (03PS3) 10Vgutierrez: lists: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492292 (https://phabricator.wikimedia.org/T207389) [10:39:14] !log jdrewniak@deploy1001 Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: [[gerrit:492636| Bumping portals to master (T128546, T202497)]] (duration: 00m 46s) [10:39:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:39:19] T202497: Add fundraising appeal on Wikipedia portal page - https://phabricator.wikimedia.org/T202497 [10:39:20] T128546: [Recurring Task] Update Wikipedia and sister projects portals statistics - https://phabricator.wikimedia.org/T128546 [10:40:01] !log jdrewniak@deploy1001 Synchronized portals: Wikimedia Portals Update: [[gerrit:492636| Bumping portals to master (T128546, T202497)]] (duration: 00m 46s) [10:40:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:41:15] (03CR) 10jenkins-bot: Bumping portals to master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492636 (https://phabricator.wikimedia.org/T128546) (owner: 10Jdrewniak) [10:43:26] (03CR) 10Vgutierrez: [C: 03+2] lists: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492293 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:43:37] (03PS3) 10Vgutierrez: lists: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492293 (https://phabricator.wikimedia.org/T207389) [10:49:15] (03CR) 10Vgutierrez: [C: 03+2] mirrors: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492295 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:49:23] (03PS3) 10Vgutierrez: mirrors: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492295 (https://phabricator.wikimedia.org/T207389) [10:52:40] (03PS1) 10Filippo Giunchedi: logstash: remove log4j input [puppet] - 10https://gerrit.wikimedia.org/r/492640 (https://phabricator.wikimedia.org/T213899) [10:53:32] (03CR) 10Vgutierrez: [C: 03+2] mirrors: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492296 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [10:53:40] (03PS3) 10Vgutierrez: mirrors: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492296 (https://phabricator.wikimedia.org/T207389) [10:54:16] godog: hi, you probaly want to wait for that to be removed gerrit side first. [10:54:22] *probably [10:54:55] I have this patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/490797/ to remove it [10:55:35] paladox: oh nice! I didn't know about that patch so I published https://gerrit.wikimedia.org/r/c/operations/puppet/+/492633 today [10:55:47] which is the same afaics [10:55:53] Ah ok [10:55:55] Heh [10:56:15] (03PS1) 10Marostegui: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492641 [10:57:05] (03PS1) 10Effie Mouzeli: Upgrade all Thumbor servers to stretch [puppet] - 10https://gerrit.wikimedia.org/r/492642 (https://phabricator.wikimedia.org/T214597) [11:07:33] (03PS6) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [11:09:35] (03CR) 10Effie Mouzeli: [C: 03+2] Upgrade all Thumbor servers to stretch [puppet] - 10https://gerrit.wikimedia.org/r/492642 (https://phabricator.wikimedia.org/T214597) (owner: 10Effie Mouzeli) [11:10:15] (03PS2) 10Effie Mouzeli: Upgrade all Thumbor servers to stretch [puppet] - 10https://gerrit.wikimedia.org/r/492642 (https://phabricator.wikimedia.org/T214597) [11:10:17] (03PS7) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [11:11:49] (03PS8) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [11:13:10] (03PS9) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [11:13:38] 10Operations, 10monitoring: google safe browsing icinga checks sporadic UNKNOWN due to 403 - https://phabricator.wikimedia.org/T216985 (10Volans) There could be some throttling ongoing. Also from a very quick look at [1] we might be using an older API version/url... [1] https://developers.google.com/safe-bro... [11:19:02] 10Operations, 10Thumbor, 10serviceops, 10Patch-For-Review, 10User-jijiki: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jiji on cumin1001.eqiad.wmnet for hosts: ` thumbor1001.eqiad.wmnet ` The log can be found in... [11:19:38] !log Reimageing thumbor1001 - T214597 [11:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:19:43] T214597: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 [11:20:50] (03CR) 10Volans: [C: 04-1] "The operations/software/netbox repository master has not been updated, although the tags are there, it has diverged from upstream because " [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492577 (owner: 10CRusnov) [11:21:27] (03PS1) 10Giuseppe Lavagetto: Improve logging of errors, remove spurious print statements [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492647 [11:21:29] (03PS1) 10Giuseppe Lavagetto: Do not try to pull/push if no registry is defined. [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492648 [11:21:38] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [11:26:10] (03CR) 10Mathew.onipe: cloudelastic: Add cloudelastic configs (035 comments) [puppet] - 10https://gerrit.wikimedia.org/r/487129 (https://phabricator.wikimedia.org/T214921) (owner: 10Mathew.onipe) [11:27:09] (03PS11) 10Mathew.onipe: cloudelastic: Add cloudelastic configs [puppet] - 10https://gerrit.wikimedia.org/r/487129 (https://phabricator.wikimedia.org/T214921) [11:28:30] (03CR) 10Volans: "> Patch Set 2:" [software/spicerack] - 10https://gerrit.wikimedia.org/r/492375 (owner: 10Gehel) [11:35:27] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "I would +2 this, but don't have the right to do so in this repository." [dumps/dcat] - 10https://gerrit.wikimedia.org/r/484011 (owner: 10Hoo man) [11:39:11] (03PS10) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [11:42:46] (03CR) 10Vgutierrez: [C: 03+2] exim: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492310 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [11:43:04] (03PS3) 10Vgutierrez: exim: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492310 (https://phabricator.wikimedia.org/T207389) [11:45:04] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [11:46:46] (03CR) 10Vgutierrez: [C: 03+2] mail: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492311 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [11:46:53] (03PS3) 10Vgutierrez: mail: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492311 (https://phabricator.wikimedia.org/T207389) [11:49:37] !log rolling out intel-microcode 3.20180807a.2 on all jessie/stretch servers, tests on a number of previously unsupported servers with Westmere CPU were successful and I've verified that all other microcode files are identical compared to the current 3.20180807a.1 microcode [11:49:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:50:31] ACKNOWLEDGEMENT - Check systemd state on mw2151 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. Giuseppe Lavagetto Server just reimaging, depends on a few tickets to be fixed. [11:50:31] ACKNOWLEDGEMENT - mediawiki-installation DSH group on mw2151 is CRITICAL: Host mw2151 is not in mediawiki-installation dsh group Giuseppe Lavagetto Server just reimaging, depends on a few tickets to be fixed. [11:50:31] ACKNOWLEDGEMENT - nutcracker port on mw2151 is CRITICAL: connect to address 127.0.0.1 and port 11212: Connection refused Giuseppe Lavagetto Server just reimaging, depends on a few tickets to be fixed. [11:50:31] ACKNOWLEDGEMENT - nutcracker process on mw2151 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 112 (nutcracker), command name nutcracker Giuseppe Lavagetto Server just reimaging, depends on a few tickets to be fixed. [11:50:31] ACKNOWLEDGEMENT - puppet last run on mw2151 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 25 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[scap],Package[libpcre3-dbg] Giuseppe Lavagetto Server just reimaging, depends on a few tickets to be fixed. [11:51:15] (03CR) 10Vgutierrez: [C: 03+2] netbox: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492317 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [11:51:28] (03PS2) 10Vgutierrez: netbox: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492317 (https://phabricator.wikimedia.org/T207389) [11:55:32] (03CR) 10Vgutierrez: [C: 03+2] netbox: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492318 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [11:55:42] (03PS2) 10Vgutierrez: netbox: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492318 (https://phabricator.wikimedia.org/T207389) [12:00:04] addshore, hashar, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, thcipriani, Niharika, and zeljkof: Your horoscope predicts another unfortunate European Mid-day SWAT(Max 6 patches) deploy. May Zuul be (nice) with you. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190225T1200). [12:00:04] Zoranzoki21: A patch you scheduled for European Mid-day SWAT(Max 6 patches) is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker. [12:00:21] I can SWAT today [12:00:27] Zoranzoki21: around for swat? [12:01:02] (03PS2) 10Zfilipin: Added and subdomains of mehrnews.com to wgCopyUploadDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492448 (https://phabricator.wikimedia.org/T213961) (owner: 10Zoranzoki21) [12:01:38] (03CR) 10Zfilipin: "Scheduled for EU SWAT but the developer was not around so it was not deployed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492448 (https://phabricator.wikimedia.org/T213961) (owner: 10Zoranzoki21) [12:01:55] (03CR) 10Vgutierrez: [C: 03+2] tendril: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492326 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [12:02:07] (03PS2) 10Vgutierrez: tendril: Switch from certcentral to acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/492326 (https://phabricator.wikimedia.org/T207389) [12:06:13] (03CR) 10Vgutierrez: [C: 03+2] tendril: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492327 (https://phabricator.wikimedia.org/T207389) (owner: 10Vgutierrez) [12:06:21] (03PS2) 10Vgutierrez: tendril: Get rid of certcentral certificates [puppet] - 10https://gerrit.wikimedia.org/r/492327 (https://phabricator.wikimedia.org/T207389) [12:07:55] (03PS11) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [12:12:12] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [12:13:46] (03PS1) 10Vgutierrez: icinga: Make use of the strong TLS ciphersuites configuration [puppet] - 10https://gerrit.wikimedia.org/r/492656 (https://phabricator.wikimedia.org/T217002) [12:16:13] (03CR) 10Vgutierrez: [C: 03+1] "pcc looks happy: https://puppet-compiler.wmflabs.org/compiler1002/14817/" [puppet] - 10https://gerrit.wikimedia.org/r/492656 (https://phabricator.wikimedia.org/T217002) (owner: 10Vgutierrez) [12:21:26] !log gilles@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [12:21:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:22:41] !log gilles@deploy1001 Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 01m 15s) [12:22:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:27:36] !log gilles@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [12:27:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:27:41] !log gilles@deploy1001 Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 05s) [12:27:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:28:41] (03PS12) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [12:34:35] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [12:36:26] 10Operations, 10serviceops, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Test different growth factors for memcached (prep step for upgrade to newer versions - https://phabricator.wikimedia.org/T217020 (10elukey) p:05Triage→03Normal [12:36:31] 10Operations, 10serviceops, 10Performance-Team (Radar), 10User-Elukey, 10User-jijiki: Test different growth factors for memcached (prep step for upgrade to newer versions) - https://phabricator.wikimedia.org/T217020 (10elukey) [12:38:27] !log gilles@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [12:38:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:38:34] !log gilles@deploy1001 Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 07s) [12:38:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:39:55] PROBLEM - puppet last run on mc2028 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [12:40:39] PROBLEM - puppet last run on ms-be2027 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [12:40:39] PROBLEM - puppet last run on db2052 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [12:40:47] PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[intel-microcode] [12:50:05] RECOVERY - puppet last run on mc2028 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:50:47] RECOVERY - puppet last run on ms-be2027 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:50:49] RECOVERY - puppet last run on db2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:50:51] RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:05:37] 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Doing), 10Services (doing): Increase the CPU count for proton[12]00[12] - https://phabricator.wikimedia.org/T197862 (10Jhernandez) [13:08:05] PROBLEM - puppet last run on cloudvirt1014 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [13:08:43] PROBLEM - puppet last run on ms-fe1006 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [13:09:45] PROBLEM - puppet last run on ms-be1032 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [13:10:41] PROBLEM - puppet last run on druid1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[initramfs-tools] [13:13:17] RECOVERY - puppet last run on cloudvirt1014 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:13:57] RECOVERY - puppet last run on ms-fe1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:14:55] RECOVERY - puppet last run on ms-be1032 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:14:57] <_joe_> moritzm: is that expected? [13:15:12] <_joe_> ^^ puppet failing trying to install initramfs-tools? [13:17:46] (03CR) 10DCausse: [C: 03+1] elasticsearch: upgrade rows one after the other [software/spicerack] - 10https://gerrit.wikimedia.org/r/492308 (owner: 10Gehel) [13:20:45] yeah, those are all puppet spam caused by the intel-microcode update (which rebuilds the initrd) [13:23:07] 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog, 10Core Platform Team Kanban (Doing), 10Services (doing): Requests to MW 404 when on HTTPS - https://phabricator.wikimedia.org/T202982 (10Jhernandez) [13:26:19] RECOVERY - puppet last run on druid1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [13:32:54] !log upgrade etherpad-lite to 1.7.5 [13:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:37:51] PROBLEM - puppet last run on etherpad1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[tzdata] [13:39:50] 10Operations, 10MediaWiki-Containers, 10Release-Engineering-Team, 10Core Platform Team Kanban (Done with CPT), and 4 others: FY2017/18 Program 6 - Outcome 2 - Objective 3: Integrated, container-based development environment - https://phabricator.wikimedia.org/T170456 (10CCicalese_WMF) [13:44:45] (03PS1) 10Giuseppe Lavagetto: Add golang image [docker-images/production-images] - 10https://gerrit.wikimedia.org/r/492667 [13:45:40] (03PS3) 10WMDE-Fisch: Show referencePreviews on group0 wikis as beta feature [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491959 (https://phabricator.wikimedia.org/T214905) [13:49:42] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] "Good to go when the dependency is merged." (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/491959 (https://phabricator.wikimedia.org/T214905) (owner: 10WMDE-Fisch) [13:53:10] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492641 (owner: 10Marostegui) [13:54:00] 10Operations, 10Proton, 10Reading-Infrastructure-Team-Backlog: proton experienced a period of high CPU usage, busy queue, lockups - https://phabricator.wikimedia.org/T214975 (10Jhernandez) [13:54:12] (03Merged) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492641 (owner: 10Marostegui) [13:55:34] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Depool db1085 for MySQL upgrade and schema change (duration: 00m 46s) [13:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:55:43] (03CR) 10jenkins-bot: db-eqiad.php: Depool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492641 (owner: 10Marostegui) [13:56:55] (03PS1) 10Volans: doc: fix reStructuredText formatting [software/spicerack] - 10https://gerrit.wikimedia.org/r/492671 [14:03:34] (03CR) 10Volans: [C: 03+2] doc: fix reStructuredText formatting [software/spicerack] - 10https://gerrit.wikimedia.org/r/492671 (owner: 10Volans) [14:03:59] RECOVERY - puppet last run on etherpad1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:04:00] !log Stop MySQL on db1085 for mysql upgrade [14:04:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:44] 10Operations, 10Thumbor, 10serviceops, 10Patch-For-Review, 10User-jijiki: Thumbor upgrade to stretch plan - https://phabricator.wikimedia.org/T214597 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['thumbor1001.eqiad.wmnet'] ` Of which those **FAILED**: ` ['thumbor1001.eqiad.wmnet'] ` [14:07:36] (03PS1) 10Marostegui: db-eqiad.php: Slowly repool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492674 [14:07:58] (03Merged) 10jenkins-bot: doc: fix reStructuredText formatting [software/spicerack] - 10https://gerrit.wikimedia.org/r/492671 (owner: 10Volans) [14:09:23] (03CR) 10jenkins-bot: doc: fix reStructuredText formatting [software/spicerack] - 10https://gerrit.wikimedia.org/r/492671 (owner: 10Volans) [14:13:08] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Slowly repool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492674 (owner: 10Marostegui) [14:14:41] (03Merged) 10jenkins-bot: db-eqiad.php: Slowly repool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492674 (owner: 10Marostegui) [14:15:54] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool db1085 after MySQL upgrade (duration: 00m 45s) [14:15:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:17:53] (03PS1) 10Filippo Giunchedi: [WIP] move varnish logging to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/492676 (https://phabricator.wikimedia.org/T213899) [14:18:34] (03CR) 10jenkins-bot: db-eqiad.php: Slowly repool db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492674 (owner: 10Marostegui) [14:21:52] (03PS2) 10Filippo Giunchedi: [WIP] move varnish logging to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/492676 (https://phabricator.wikimedia.org/T213899) [14:27:56] (03PS3) 10Volans: remote: suppress Cumin's output [software/spicerack] - 10https://gerrit.wikimedia.org/r/481858 (https://phabricator.wikimedia.org/T212783) [14:27:58] (03PS1) 10Marostegui: db-eqiad.php: Pool db1085 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492678 [14:28:20] (03PS4) 10Volans: remote: suppress Cumin's output [software/spicerack] - 10https://gerrit.wikimedia.org/r/481858 (https://phabricator.wikimedia.org/T212783) [14:29:17] (03CR) 10Volans: "This has been already used as a hotfix on cumin2001 during last Friday Elasticsearch upgrade cookbook run." [software/spicerack] - 10https://gerrit.wikimedia.org/r/481858 (https://phabricator.wikimedia.org/T212783) (owner: 10Volans) [14:29:50] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Pool db1085 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492678 (owner: 10Marostegui) [14:30:56] (03Merged) 10jenkins-bot: db-eqiad.php: Pool db1085 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492678 (owner: 10Marostegui) [14:30:58] jouncebot: next [14:30:58] In 3 hour(s) and 29 minute(s): Wikidata Query Service weekly deploy (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190225T1800) [14:31:09] (03CR) 10jenkins-bot: db-eqiad.php: Pool db1085 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492678 (owner: 10Marostegui) [14:32:57] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Slowly repool into API db1085 after MySQL upgrade (duration: 00m 45s) [14:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:38:23] 10Operations: Integrate Stretch 9.8 point update - https://phabricator.wikimedia.org/T216384 (10MoritzMuehlenhoff) [14:38:58] (03CR) 10Fsero: "i've built it over the builder-envoy cloud VM. Could anyone vote for or against that so i can move forward?" [debs/envoyproxy] (wikimedia-stretch) - 10https://gerrit.wikimedia.org/r/491951 (https://phabricator.wikimedia.org/T215810) (owner: 10Fsero) [14:39:59] <_joe_> fsero: I'll get to review it soon, promised [14:40:16] thx _joe_ [14:43:32] (03PS1) 10Marostegui: db-eqiad.php: Increase weight for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492680 [14:44:53] (03PS13) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [14:46:05] (03PS3) 10Filippo Giunchedi: [WIP] move varnish logging to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/492676 (https://phabricator.wikimedia.org/T213899) [14:46:58] !log jiji@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [14:46:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:17] !log jiji@deploy1001 deploy aborted: (no justification provided) (duration: 00m 19s) [14:47:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:48] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Increase weight for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492680 (owner: 10Marostegui) [14:48:58] (03Merged) 10jenkins-bot: db-eqiad.php: Increase weight for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492680 (owner: 10Marostegui) [14:49:03] !log jiji@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [14:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:49:07] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [14:49:14] (03CR) 10jenkins-bot: db-eqiad.php: Increase weight for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492680 (owner: 10Marostegui) [14:49:18] !log jiji@deploy1001 Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 15s) [14:49:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:50:05] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Increase traffic for db1085 after MySQL upgrade (duration: 00m 45s) [14:50:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:50:33] (03PS14) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [14:53:50] (03PS1) 10Marostegui: db-eqiad.php: Increase traffic for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492683 [14:57:21] (03PS15) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [14:57:58] (03PS1) 10Andrew Bogott: apache: update ssl config [wikitech-static] - 10https://gerrit.wikimedia.org/r/492684 [14:58:00] (03PS1) 10Andrew Bogott: mediawiki config: catch up with upstream mw changes [wikitech-static] - 10https://gerrit.wikimedia.org/r/492685 [14:59:06] (03PS2) 10Andrew Bogott: apache: update ssl config [wikitech-static] - 10https://gerrit.wikimedia.org/r/492684 [14:59:08] (03PS2) 10Andrew Bogott: mediawiki config: catch up with upstream mw changes [wikitech-static] - 10https://gerrit.wikimedia.org/r/492685 [15:01:49] (03CR) 10jerkins-bot: [V: 04-1] Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 (owner: 10Jbond) [15:01:51] (03CR) 10Andrew Bogott: "Daniel, I'm adding you on this patch just so you know that this repo exists :) It's slightly better than having the wikitech-static host " [wikitech-static] - 10https://gerrit.wikimedia.org/r/492684 (owner: 10Andrew Bogott) [15:02:05] (03CR) 10Ottomata: Add eventbus analytics logging alongside with kafka logging. (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/490668 (https://phabricator.wikimedia.org/T216163) (owner: 10Ppchelko) [15:02:36] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Increase traffic for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492683 (owner: 10Marostegui) [15:03:05] (03CR) 10Giuseppe Lavagetto: [C: 03+1] "Some nitpick but overall a very good work." (036 comments) [debs/envoyproxy] (wikimedia-stretch) - 10https://gerrit.wikimedia.org/r/491951 (https://phabricator.wikimedia.org/T215810) (owner: 10Fsero) [15:03:58] (03Merged) 10jenkins-bot: db-eqiad.php: Increase traffic for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492683 (owner: 10Marostegui) [15:04:13] (03CR) 10jenkins-bot: db-eqiad.php: Increase traffic for db1085 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492683 (owner: 10Marostegui) [15:04:51] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Increase traffic for db1085 after MySQL upgrade (duration: 00m 45s) [15:04:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:06:52] (03PS1) 10Vgutierrez: netbox: Make use of the strong TLS ciphersuites configuration [puppet] - 10https://gerrit.wikimedia.org/r/492686 (https://phabricator.wikimedia.org/T217002) [15:06:54] (03PS1) 10Vgutierrez: librenms: Make use of the strong TLS ciphersuites configuration [puppet] - 10https://gerrit.wikimedia.org/r/492687 (https://phabricator.wikimedia.org/T217002) [15:08:40] 10Operations, 10Traffic, 10HTTPS, 10Patch-For-Review: Make sure that services available for NDA-only users are using strong TLS ciphersuites - https://phabricator.wikimedia.org/T217002 (10Vgutierrez) [15:09:45] (03CR) 10Volans: [C: 03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/492686 (https://phabricator.wikimedia.org/T217002) (owner: 10Vgutierrez) [15:10:25] (03CR) 10Alexandros Kosiaris: [C: 03+2] Add citoid, cxserver kubernetes tokens [puppet] - 10https://gerrit.wikimedia.org/r/492273 (https://phabricator.wikimedia.org/T213194) (owner: 10Alexandros Kosiaris) [15:10:32] (03PS2) 10Alexandros Kosiaris: Add citoid, cxserver kubernetes tokens [puppet] - 10https://gerrit.wikimedia.org/r/492273 (https://phabricator.wikimedia.org/T213194) [15:10:35] (03CR) 10Alexandros Kosiaris: [V: 03+2 C: 03+2] Add citoid, cxserver kubernetes tokens [puppet] - 10https://gerrit.wikimedia.org/r/492273 (https://phabricator.wikimedia.org/T213194) (owner: 10Alexandros Kosiaris) [15:14:27] (03PS16) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [15:14:54] (03CR) 10Volans: "[meta review, CR is still in WIP]" (034 comments) [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/492007 (https://phabricator.wikimedia.org/T215229) (owner: 10CRusnov) [15:15:29] (03PS1) 10Marostegui: db-eqiad.php: Increase db1085 API traffic [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492688 [15:16:32] (03CR) 10Marostegui: [C: 03+2] db-eqiad.php: Increase db1085 API traffic [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492688 (owner: 10Marostegui) [15:17:29] (03Merged) 10jenkins-bot: db-eqiad.php: Increase db1085 API traffic [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492688 (owner: 10Marostegui) [15:17:42] (03CR) 10jenkins-bot: db-eqiad.php: Increase db1085 API traffic [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492688 (owner: 10Marostegui) [15:18:13] (03PS17) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [15:18:27] !log marostegui@deploy1001 Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1085 after MySQL upgrade (duration: 00m 45s) [15:18:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:20:52] !log shutting down certcentral VMs for decommission - T207389 [15:20:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:20:55] T207389: Rename the Certcentral project to Acme-chief - https://phabricator.wikimedia.org/T207389 [15:20:57] (03CR) 10Hashar: "+1 on getting rid of print() statement, but maybe we can keep some kind of logging." (033 comments) [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492647 (owner: 10Giuseppe Lavagetto) [15:22:00] so long certcentral [15:22:40] yeah... renaming and up&running project is great fun [15:22:50] :-( [15:23:04] s/and/an/ [15:23:32] involved commits so far: https://gerrit.wikimedia.org/r/q/topic:%22T207389%22+(status:open%20OR%20status:merged) [15:24:43] :(( [15:26:44] (03PS1) 10Marostegui: db-eqiad.php: Fully repool db1085 in API [mediawiki-config] - 10https://gerrit.wikimedia.org/r/492691 [15:27:05] (03CR) 10Hashar: [C: 03+2] Do not try to pull/push if no registry is defined. [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/492648 (owner: 10Giuseppe Lavagetto) [15:27:13] (03PS5) 10Filippo Giunchedi: gerrit: stop sending logs directly to logstash [puppet] - 10https://gerrit.wikimedia.org/r/492633 (https://phabricator.wikimedia.org/T213899) [15:27:15] (03PS2) 10Filippo Giunchedi: logstash: remove log4j input [puppet] - 10https://gerrit.wikimedia.org/r/492640 (https://phabricator.wikimedia.org/T213899) [15:27:17] (03PS4) 10Filippo Giunchedi: [WIP] move varnish logging to logging pipeline [puppet] - 10https://gerrit.wikimedia.org/r/492676 (https://phabricator.wikimedia.org/T213899) [15:27:19] (03PS1) 10Filippo Giunchedi: logstash: disable syslog-tls input, unused [puppet] - 10https://gerrit.wikimedia.org/r/492692 (https://phabricator.wikimedia.org/T213899) [15:27:51] !log jiji@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [15:27:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:27:54] !log jiji@deploy1001 deploy aborted: (no justification provided) (duration: 00m 04s) [15:27:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:01] !log jiji@deploy1001 Started deploy [3d2png/deploy@ca39432]: (no justification provided) [15:28:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:08] !log jiji@deploy1001 Finished deploy [3d2png/deploy@ca39432]: (no justification provided) (duration: 00m 07s) [15:28:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:29] jouncebot: now [15:28:29] No deployments scheduled for the next 2 hour(s) and 31 minute(s) [15:29:06] (03PS18) 10Jbond: Add password reset function to ipmi module [software/spicerack] - 10https://gerrit.wikimedia.org/r/492026 [15:30:30] !log akosiaris@deploy1001 scap-helm citoid install -n staging -f citoid-staging-values.yaml stable/citoid [namespace: citoid, clusters: staging] [15:30:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:30:32] !log akosiaris@deploy1001 scap-helm citoid cluster staging completed [15:30:32] !log akosiaris@deploy1001 scap-helm citoid finished [15:30:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:30:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:30:53] <_joe_> akosiaris: oh we're deploying citoid to production? [15:31:25] (03CR) 10Herron: [C: 03+1] logstash: disable syslog-tls input, unused [puppet] - 10https://gerrit.wikimedia.org/r/492692 (https://phabricator.wikimedia.org/T213899) (owner: 10Filippo Giunchedi) [15:33:40] !log vgutierrez@cumin1001 START - Cookbook sre.hosts.decommission [15:33:41]