[00:33:17] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp1046.eqiad.wmnet'] ` and were **ALL** successful. [01:13:15] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp1047.eqiad.wmnet'] ` and were **ALL** successful. [01:17:57] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp1048.eqiad.wmnet'] ` and were **ALL** successful. [01:25:59] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10Dzahn) >>! In T258775#6341636, @JMeybohm wrote: > All hosts but `wtp104[6-8].eqiad.wmnet` completed. wtp1046, wtp1047, wtp1048 completed and repooled on wtp1... [01:38:15] 10serviceops, 10Operations, 10Performance-Team, 10Sustainability (Incident Followup): Test gutter pool failover in production and memcached 1.5.x - https://phabricator.wikimedia.org/T240684 (10aaron) 05Open→03Resolved Yeah, technically, all sorts of anomalies are possible, so callers should always (a)... [01:38:19] 10serviceops, 10Operations, 10Patch-For-Review: Upgrade and improve our application object caching service (memcached) - https://phabricator.wikimedia.org/T244852 (10aaron) [02:09:51] 10serviceops, 10MediaWiki-Cache, 10Performance-Team, 10Patch-For-Review, 10Sustainability (Incident Followup): Let WANObjectCache store "sister keys" on the same backend as the main value key - https://phabricator.wikimedia.org/T252564 (10Krinkle) p:05Medium→03High [02:18:44] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Krinkle) [02:20:58] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Krinkle) 05Stalled→03Open a:03dpifke Unstalled per T235456. Should be safe to go back up to 90 days now. [02:21:20] 10serviceops, 10Arc-Lamp, 10Performance-Team: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10Krinkle) [08:02:41] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10JMeybohm) >>! In T258775#6343259, @Dzahn wrote: >>>! In T258775#6341636, @JMeybohm wrote: >> All hosts but `wtp104[6-8].eqiad.wmnet` completed. > > wtp1046, w... [08:31:06] jayme: FYI I had to run apt-get clean on 3 wtp nodes yesterday because they were at 100% and failing scap [08:31:25] volans: yeah, read that. Thanks a lot! [08:33:02] mutant.e has completed the eqiad ones "yesterday" but wtp2001 is still pretty full. I think I'll reimage that one shortly as well to be sure [08:33:52] ack, thanks! [08:39:05] <_joe_> jayme: yes, or we start preparing the parse2* machines [08:39:19] <_joe_> spoiler alert: that needs a mediawiki code deploy too [08:41:30] _joe_: I'll do that one host to get it out of the way I would say. We'll then need a reimage of parse* anyways in first place [08:41:52] <_joe_> ack [08:42:27] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by jayme on cumin1001.eqiad.wmnet for hosts: ` wtp2001.codfw.wmnet ` The log can be found in `/var/log... [08:50:54] 10serviceops, 10Prod-Kubernetes, 10Proton, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "Failed to fork" errors on kubernetes100[1,3,4] - https://phabricator.wikimedia.org/T257679 (10akosiaris) 05Open→03Resolved Indeed. And https://w.wiki/YAr validates it across all the kuberne... [08:51:29] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10thiemowmde) I'm certainly ignoring many of the very relevant arguments listed above. However, as far as I'm conc... [10:39:31] <_joe_> hnowlan: oh so puppet did work on the first run on builder-03? [10:39:37] * _joe_ is impressed [10:39:51] It didn't actually, I need to post a patch :) [10:39:54] <_joe_> I did lose it in the backscroll [10:40:09] <_joe_> what didn't work? [10:40:22] I had to mkdir -p /var/cache/pbuilder/aptcache/buster-amd64 [10:41:45] <_joe_> oh right, I had the same problem but that's tied to the basic builder profile [10:42:08] <_joe_> I did punt on that issue, I maybe didn't open a task [10:42:26] also docker needed to be restarted to properly set up the dirs in /var/lib/docker which seems particularly odd [10:43:22] <_joe_> yeah that sounds new to me too [10:44:20] like seems to me that if docker is running at all those would be set up (and it was running) [10:44:34] apart from those, relatively smooth process A+ would package again [10:44:47] <_joe_> rzl: ^^ [10:44:49] especially reflecting on how it felt trying to run bazel etc locally [10:45:06] <_joe_> we're cheating though [10:45:10] <_joe_> we're not running the tests [10:47:19] even doing the build is hassle enough [10:48:01] <_joe_> this means, though, that if we have to modify ourselves envoy we will need to create a better build env [11:25:52] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2001.codfw.wmnet'] ` and were **ALL** successful. [13:08:13] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) As it is now, 1.35 supports both 7.2 and 7.3. So the goal is to get out of the situation where code using M... [13:12:26] yeah I do think we'll want to write some Envoy protocol filters eventually [13:15:43] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10RazeSoldier) >>! In T257879#6344561, @Tgr wrote: > As it is now, 1.35 supports both 7.2 and 7.3. Can't it be an... [13:27:14] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) >>! In T257879#6344581, @RazeSoldier wrote: > Can't it be announced that 1.35 supports 7.4? It actually do... [13:33:07] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10JMeybohm) @Dzahn I did `wtp2001.codfw.wmnet` as that was pretty full as well. [13:44:58] 10serviceops, 10Platform Team Initiatives (MW REST API in PHP): Move CORE REST API to be served from the MW API Cluster - https://phabricator.wikimedia.org/T246002 (10eprodromou) [13:50:27] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10thiemowmde) > […] code using MB/PCRE methods need to account for two different behaviors, and might end up worki... [14:03:55] hnowlan: \o/ thanks for doing that [14:04:12] and thanks _joe_ for getting everything scripted and puppetized [14:05:47] <_joe_> rzl: https://cdn.theatlantic.com/thumbor/KeZ-JHlvtCd_HVhdinCyUaY1sy8=/28x15:2022x1137/720x405/media/img/mt/2018/04/RTRS0M7/original.jpg [14:06:58] _joe_: https://www.youtube.com/watch?v=IjmtVKOAHPM [14:07:19] <_joe_> aahahah [14:07:59] <_joe_> rzl: the image I posted is what I used when we finished switching to HHVM. 5 minutes later the apis were crashlooping [14:08:16] <_joe_> so it always seem appropriate whenever we declare victory [14:09:33] "Hopefully (famous last words) this image will not be very long-lived and we can just get rid of it before we get out of sync." [14:09:47] hnowlan: btw on this image's birthday every year I am going to mail you this sentence in increasingly elaborate commemorative form [14:10:06] :D :D [14:10:07] calligraphy, and then needlepoint, and then woodcut, and then eventually maybe stone tablets [14:13:44] one year your doorbell will ring and Donald Knuth will be standing there [14:13:50] "Are you Hugh Nowlan?" he'll ask [14:14:25] "Hopefully (famous last words)," he'll say, glancing up from his notes to look over his glasses at you, "this image will not be very long-lived and we can just get rid of it before we get out of sync." [14:14:35] then he'll turn around and disappear into the night, ignoring all your questions [14:19:44] actually laughed mid-meeting, superb <3 [14:39:45] <_joe_> ahahahahaha [14:41:19] <_joe_> rzl: we can just send Tim Starling. https://bash.toolforge.org/quip/AU7VTzhg6snAnmqnK_pc [15:32:45] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2002.codfw.wmnet ` The log can be found in `/var/log... [15:33:36] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2003.codfw.wmnet ` The log can be found in `/var/log... [15:56:48] 10serviceops, 10Operations, 10Traffic, 10observability, 10Patch-For-Review: monitoring for mismatched LVS realserver addresses/configurations - https://phabricator.wikimedia.org/T258648 (10CDanis) 05Open→03Declined The structure of the data makes this teeth-pullingly impossibly difficult to do well,... [15:56:52] 10serviceops, 10Operations, 10Traffic, 10observability: Alert "LVS api codfw port 80/tcp - MediaWiki API cluster- api.svc.eqiad.wmnet IPv4" is flapping - https://phabricator.wikimedia.org/T258614 (10CDanis) [16:01:47] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2001.codfw.wmnet', 'parse2002.codfw.wmnet', 'par... [16:02:55] 10serviceops, 10Operations, 10Patch-For-Review: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10Dzahn) >>! In T258775#6344708, @JMeybohm wrote: > @Dzahn I did `wtp2001.codfw.wmnet` as that was pretty full as well. Thank you. Taking over with wtp2002,2003... [16:14:42] 10serviceops, 10Operations, 10Traffic, 10observability, 10Patch-For-Review: monitoring for mismatched LVS realserver addresses/configurations - https://phabricator.wikimedia.org/T258648 (10Joe) Truth is that every host that has multiple pools that use the same backend IP can happily live with only one po... [16:18:41] <_joe_> jayme: I did some work towards having a rake task verifying the envoy config, but I still have some work to do [16:19:22] <_joe_> it will happen tomorrow though [16:26:23] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2002.codfw.wmnet', 'parse2003.codfw.wmnet', 'parse2001.codfw.wmnet'] ` and were **ALL** successful. [16:28:15] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2004.codfw.wmnet', 'parse2005.codfw.wmnet', 'parse2006.codfw.wmnet'] `... [16:36:40] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:14] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:18] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Dzahn) [16:42:43] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:54] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:42:58] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10Dzahn) [16:44:27] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) [16:51:32] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2006.codfw.wmnet', 'parse2005.codfw.wmnet', 'parse2004.codfw.wmnet'] ` and were **ALL** successful. [16:52:10] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10hashar) The error message refers to `push-no... [16:56:08] 10serviceops, 10Continuous-Integration-Infrastructure, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service: puppet errors on contint servers related to helmfiles for push-notifications - https://phabricator.wikimedia.org/T259152 (10Dzahn) >>! In T259152#6345663, @hashar wrote... [17:01:37] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2007.codfw.wmnet', 'parse2008.codfw.wmnet', 'parse2009.codfw.wmnet'] `... [17:18:11] 10serviceops, 10Operations, 10Phabricator, 10Traffic, and 2 others: Phabricator downtime due to aphlict and websockets (aphlict current disabled) - https://phabricator.wikimedia.org/T238593 (10mmodell) So I think that this is all that remains: * [] cache layer proxy wss://phabricator.wikimedia.org to aphl... [17:28:21] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2008.codfw.wmnet', 'parse2007.codfw.wmnet', 'parse2009.codfw.wmnet'] ` and were **ALL** successful. [17:45:46] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2002.codfw.wmnet'] ` and were **ALL** successful. [17:46:19] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2010.codfw.wmnet', 'parse2011.codfw.wmnet', 'parse2012.codfw.wmnet'] `... [17:49:10] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2003.codfw.wmnet'] ` and were **ALL** successful. [17:51:26] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2004.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [17:56:36] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2005.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [17:56:40] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2005.codfw.wmnet'] ` Of which those **FAILED**: ` ['wtp2005.codfw.wmnet'] ` [17:57:27] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10Dzahn) Note wtp2005 is missing because of T257903. [17:58:05] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2006.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [18:09:47] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2011.codfw.wmnet', 'parse2012.codfw.wmnet', 'parse2010.codfw.wmnet'] ` and were **ALL** successful. [18:12:44] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10hashar) Responding for CI / release engineering. **TLDR: support 7.2 and drop support for it later on as it bec... [18:27:51] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2013.codfw.wmnet', 'parse2014.codfw.wmnet', 'parse2015.codfw.wmnet'] `... [18:33:01] 10serviceops, 10Arc-Lamp, 10Performance-Team, 10Patch-For-Review: Resolve arclamp disk exhaustion problem (Oct 2019) - https://phabricator.wikimedia.org/T235455 (10dpifke) 05Open→03Resolved [18:53:26] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2013.codfw.wmnet', 'parse2014.codfw.wmnet'] ` Of which those **FAILED**: ` ['parse2015.codfw.wmnet'] ` [19:02:02] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2016.codfw.wmnet', 'parse2017.codfw.wmnet', 'parse2018.codfw.wmnet'] `... [19:25:42] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2018.codfw.wmnet', 'parse2016.codfw.wmnet', 'parse2017.codfw.wmnet'] ` and were **ALL** successful. [19:26:35] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` ['parse2019.codfw.wmnet', 'parse2020.codfw.wmnet'] ` The log can be found in... [19:52:10] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['parse2019.codfw.wmnet', 'parse2020.codfw.wmnet'] ` and were **ALL** successful. [20:05:52] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2004.codfw.wmnet'] ` and were **ALL** successful. [20:10:20] 10serviceops, 10Prod-Kubernetes, 10Release-Engineering-Team, 10Kubernetes: CI pipeline/job to build and release helm chart artifacts - https://phabricator.wikimedia.org/T257333 (10jeena) I talked with @dduvall on how we could add this to CI. I initially thought we could do this pre-merge in CI, but another... [20:16:19] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10Dzahn) all parse2* hosts done: ` [cumin1001:~] $ sudo cumin parse2* 'df -h | grep srv | cut -d " " -f12' .. 20 hosts will be targeted: 1%... [20:19:49] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2007.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:32:37] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2006.codfw.wmnet'] ` and were **ALL** successful. [20:34:44] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2008.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [20:46:25] FYI it seems that helm-chartctl-package-all.service is flapping on chartmuseum2001 [20:47:13] logging some [20:47:13] ERROR[helm-chartctl] Error getting data from chartmuseum: 500 Server Error: Internal Server Error for url: http:/ [21:02:54] <_joe_> jayme: ^^ [21:16:54] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 3 others: Drop PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Tgr) >>! In T257879#6346169, @hashar wrote: > Once 7.2 is end of life and if it becomes too complicated to keep... [22:33:55] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2007.codfw.wmnet'] ` and were **ALL** successful. [22:48:36] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2009.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020... [23:22:29] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['wtp2008.codfw.wmnet'] ` and were **ALL** successful. [23:41:52] 10serviceops, 10Operations: All wtp and parse servers have a bad partition scheme. - https://phabricator.wikimedia.org/T258775 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` wtp2010.codfw.wmnet ` The log can be found in `/var/log/wmf-auto-reimage/2020...