[00:14:25] 10serviceops, 10Operations: reinstall xhgui* with buster - https://phabricator.wikimedia.org/T259206 (10Dzahn) [01:03:20] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2009.codfw.wmnet'] ` and were **ALL** successful. [01:16:31] 10serviceops, 10Operations: reinstall xhgui* with buster - https://phabricator.wikimedia.org/T259206 (10Dzahn) p:05Triage→03High [01:16:41] 10serviceops, 10Operations: reinstall xhgui* with buster - https://phabricator.wikimedia.org/T259206 (10Dzahn) a:03Dzahn [01:18:14] 10serviceops, 10Operations: reinstall xhgui* with buster - https://phabricator.wikimedia.org/T259206 (10Dzahn) [02:12:45] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2010.codfw.wmnet'] ` and were **ALL** successful. [02:19:31] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10Dzahn) wtp2002 through wtp2010 done and repooled. [02:28:54] 10serviceops, 10Graphoid, 10Operations, 10Chinese-Sites, 10Platform Engineering (Icebox): Undeploy graphoid for phase 2 wiki's - https://phabricator.wikimedia.org/T258463 (10Jseddon) 05Open→03Resolved [02:28:58] 10serviceops, 10Graphoid, 10Operations, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Platform Engineering (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Jseddon) [02:29:48] 10serviceops, 10Graphoid, 10Operations, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Platform Engineering (Icebox): Undeploy graphoid for phase 3 wiki's - https://phabricator.wikimedia.org/T259207 (10Jseddon) [02:29:58] 10serviceops, 10Graphoid, 10Operations, 10Platform Engineering (Icebox): Undeploy graphoid for phase 3 wiki's - https://phabricator.wikimedia.org/T259207 (10Jseddon) [07:30:46] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: chartmuseum GET /api/production/charts failing after some time - https://phabricator.wikimedia.org/T259221 (10JMeybohm) [07:31:00] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: chartmuseum GET /api/production/charts failing after some time - https://phabricator.wikimedia.org/T259221 (10JMeybohm) p:05Triage→03Medium [08:37:35] <_joe_> akosiaris, jayme so at jayme's suggestion I'm adding a rake task to validate the generated envoy configuration in deployment-charts [08:37:47] <_joe_> sadly, this needs a valid cert keypair, and a valid ca cert [08:37:58] <_joe_> where valid == will parse and is not expired [08:38:25] <_joe_> I can re-generate all of them on the fly via rake, or we add a few certs as fixtures and make them expire in 2037 or something [10:55:41] 4037 :P [11:01:04] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10akosiaris) 05Open→03Resolved a:03akosia... [11:50:26] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10akosiaris) An update: The upgrade on the new node using a test database has progressed ok. A couple of issues met: The script ./DBUpdate... [11:59:16] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10akosiaris) DNS and edge cache changes have been merged, this is ready to be tested by agents. I 'll ping on the OTRS wiki noticeboard aski... [13:10:40] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: chartmuseum GET /api/production/charts failing after some time - https://phabricator.wikimedia.org/T259221 (10JMeybohm) 05Open→03Resolved https://github.com/chartmuseum/storage/pull/47 Fixed with chartmuseum_0.12.0-3 [13:10:42] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move helm chart repository out of git - https://phabricator.wikimedia.org/T253843 (10JMeybohm) [13:25:34] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10jcrespo) It takes very little to load another snapshot if you think you need it. [13:48:45] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10hashar) >>! In T257879#6346810, @Tgr wrote: > Per the [[https://www.mediawiki.org/wiki/Support_policy_for_PHP|po... [14:21:21] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) [14:21:42] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) >>! In T257879#6348477, @hashar wrote: > Hope that clarify? It does, thanks! [15:13:24] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10hashar) >>! In T257879#6348477, @hashar wrote: > ... I think we had asked for a rebuild of php7.2 for Buster al... [15:56:26] hi! about half of codfw parsoid and all of parse2* reinstalled yesterday. doing the rest today [15:58:56] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2011.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [15:59:33] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2012.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [17:59:46] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Move mobileapps to use TLS only - https://phabricator.wikimedia.org/T255876 (10MSantos) [18:11:17] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Akuckartz) Maybe a compromise is possible: # No official support for 7.2 # Continue supporting 7.2 as long as i... [18:19:08] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2011.codfw.wmnet'] ` and were **ALL** successful. [19:25:01] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2013.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:04:24] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2012.codfw.wmnet'] ` Of which those **FAILED**: ` ['wtp2012.codfw.wmnet'] ` [20:46:08] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2014.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [21:21:42] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Krinkle) @Akuckartz As I understand it, yes, that is and has been the proposal from the start of this task. [21:42:36] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2013.codfw.wmnet'] ` and were **ALL** successful. [21:45:26] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2015.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [22:55:32] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2014.codfw.wmnet'] ` and were **ALL** successful. [23:32:22] 10serviceops, 10Performance-Team: Package XHGui as .deb - https://phabricator.wikimedia.org/T254310 (10Dzahn) [23:33:34] 10serviceops, 10Operations: reinstall xhgui* with buster - https://phabricator.wikimedia.org/T259206 (10Dzahn) xhgui2001 has been reinstalled with buster. xhgui is now installed by puppet. xhgui1001 is as before for right now. [23:45:24] 10serviceops, 10observability, 10Developer Productivity, 10Patch-For-Review: Logstash entries from php7-fatal-error.php use level "ERR" instead of "ERROR" - https://phabricator.wikimedia.org/T248181 (10Krinkle) 05Open→03Resolved a:05Krinkle→03herron >>! **Task description**: > From 10serviceops, 10observability, 10Developer Productivity: Logstash entries from php7-fatal-error.php use level "ERR" instead of "ERROR" - https://phabricator.wikimedia.org/T248181 (10Krinkle) [23:55:02] hey folks [23:55:24] does the ci k8s cluster have persistent volumes or dynamic provisioning set up? [23:59:21] we're trying to deploy kask for testing during gate-and-submit and it's bailing on `pod has unbound immediate PersistentVolumeClaims (repeated 2 times)` [23:59:25] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2015.codfw.wmnet'] ` and were **ALL** successful.