[07:00:52] mutante: cool! Glad it worked :-) [07:53:43] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move cxserver to use TLS only - https://phabricator.wikimedia.org/T255879 (10JMeybohm) [07:53:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move termbox to use TLS only - https://phabricator.wikimedia.org/T254581 (10JMeybohm) [10:02:20] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move eventgate-main to use TLS only - https://phabricator.wikimedia.org/T255873 (10JMeybohm) [10:02:32] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move eventgate-analytics to use TLS only - https://phabricator.wikimedia.org/T255870 (10JMeybohm) [10:04:57] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10JMeybohm) [11:14:01] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10Joe) [11:17:05] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki) 05Open→03Resolved [11:18:14] <_joe_> akosiaris, jayme: have you ever looked at https://shipper-k8s.readthedocs.io/en/latest/ ? [11:18:41] <_joe_> it seems an helmfile alternative, by booking.com, pretty interesting at first sight [11:18:41] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, and 2 others: Deploy push-notifications service to Kubernetes - https://phabricator.wikimedia.org/T256973 (10jijiki) @Mholloway I am marking this as resolved unless we believe there is a reason not to. [13:06:14] _joe_: niah, first time I see it [13:32:37] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Papaul) a:03Papaul [13:33:51] 10serviceops, 10Push-Notification-Service, 10Product-Infrastructure-Team-Backlog (Kanban), 10User-jijiki: Set up secrets for Token clean-up - https://phabricator.wikimedia.org/T262957 (10jijiki) @Msantos I will get to it next week or the week after, unless you have a deadline to catch, thank you! [13:35:56] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Papaul) ` [edit interfaces interface-range vlan-private1-b-codfw] - member ge-3/0/2; [edit interfaces interface-range disabled] member ge-4/0/39 {... [13:50:13] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Papaul) [x] - disable switch port / set to asset tag if host isn't being unracked / remove from switch if being unracked. [] - system disks removed (by on... [14:14:25] 10serviceops, 10Citoid, 10Operations, 10Prod-Kubernetes, and 2 others: Move citoid to use TLS only - https://phabricator.wikimedia.org/T255868 (10JMeybohm) a:03JMeybohm [14:14:29] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move proton to use TLS only - https://phabricator.wikimedia.org/T255877 (10JMeybohm) [14:15:19] 10serviceops, 10MediaWiki-General, 10Operations, 10Patch-For-Review, 10Service-Architecture: Create a service-to-service proxy for handling HTTP calls from services to other entities - https://phabricator.wikimedia.org/T244843 (10JMeybohm) [14:36:09] 10serviceops, 10MW-on-K8s, 10Operations, 10Platform Engineering, 10TechCom-RFC: Decide on logging in k8s for ShellBox - https://phabricator.wikimedia.org/T263545 (10Pchelolo) [14:36:15] 10serviceops, 10Operations, 10Prod-Kubernetes, 10Kubernetes: Move mathoid to use TLS only - https://phabricator.wikimedia.org/T255875 (10JMeybohm) a:03JMeybohm [15:30:42] <_joe_> TIL https://k9scli.io/ [15:30:52] <_joe_> can we install it in production? akosiaris? [15:31:05] <_joe_> and well, I should also ping our official go packager, jayme [15:32:09] _joe_: lol [15:32:30] <_joe_> it's basically midnight commander for k8s [15:32:47] yup, I saw that [15:33:30] uhf [15:33:42] or...whoof? [15:33:54] <_joe_> akosiaris: we just made jayme google "midnight commander" [15:34:09] ahahaha [15:34:11] <_joe_> or at least, I'd hope younger people would not know what I'm talking about [15:34:13] dude...i'm not 20 [15:34:35] (and yeah...I said dude to make that point stand out :P) [15:34:45] <_joe_> jayme: I hoped the use of midnight commander faded circa 2001 [15:35:52] It probably did but still, history stays and you hear storys from the elders ... [15:37:11] 10serviceops, 10MW-on-K8s, 10Operations, 10Platform Engineering, 10TechCom-RFC: Decide on logging in k8s for ShellBox - https://phabricator.wikimedia.org/T263545 (10TK-999) This also relies on FPM's [[ https://www.php.net/manual/en/install.fpm.configuration.php#catch-workers-output | `catch_workers_outpu... [15:46:08] <_joe_> omg https://github.com/docker-library/php/blob/d6838d8e45f6356d867dbc8ff8b3513492cba500/7.3/buster/fpm/Dockerfile [15:53:09] needs more \ [15:55:36] I have discovered a truly remarkable proof that PHP is awesome which this series of shell commands with \-continuations is too small to contain [15:55:49] <_joe_> oh do tell [15:56:26] I would but my backslash key broke from abuse [15:57:17] I wonder what the rate is [15:58:02] 67.82% [15:58:15] * apergos peeks in, raises an eyebrow at bblack and heads right back out [15:58:23] and that's with blanklines [15:58:25] interesting [15:58:35] it might've made more sense to ship a proper shellscript for most of it [15:59:02] probablya hella lot more readable [15:59:50] I love how every RUN starts with set -eux; \ [16:00:15] ok, I think I wasted enough time on it [16:01:28] <_joe_> most of that thing could've used a COPY of a couple ini files [16:04:38] 10serviceops, 10Operations, 10ops-eqiad: mw1360's NIC is faulty - https://phabricator.wikimedia.org/T262151 (10RobH) Ok, running the troubleshooting steps as follows: Copy over the SEL, erase it so it won't throw errors in testing. At this time, there are no errors for the NIC on the SEL, but full testing... [16:31:01] 10serviceops, 10Operations, 10ops-eqiad: mw1360's NIC is faulty - https://phabricator.wikimedia.org/T262151 (10RobH) quick tests complete no errors, full testing continuing, eta on screen 3 hours. [16:48:22] _joe_: when are we installing k9s? I miss `mc` [17:01:46] * apergos gives cd anis the side-eye [17:09:07] apergos: did you see https://github.com/opencredo/kubefuse [17:09:54] no I had not [17:11:15] seems pretty great on first glance, if scary :-D :-D [17:13:19] 10serviceops, 10DC-Ops, 10Operations, 10ops-codfw: decom mw2256 (was: mw2256 - CPU/board hardware issue) - https://phabricator.wikimedia.org/T263065 (10Papaul) 05Open→03Resolved Complete [17:58:01] * volans uses mc at least once a month [18:09:07] I don't think I've launched mc in at least a decade, but I have nostalgia for it somehow [18:09:19] writing FUSE filesystems, though, I have done very recently [18:09:25] lol [18:09:49] I use it for very specific things but would miss it a lot if I couldn't use it [18:12:41] as far as I'm concerned mc is on the same list as emacs and joe: tools I never had an affinity for and probably never will :) [18:13:04] (joe the editor, not the person) :) [18:17:08] <_joe_> bblack: you can say it, you already offended emacs [19:18:32] :-D [19:47:06] 10serviceops, 10Operations, 10Product-Infrastructure-Team-Backlog, 10Push-Notification-Service, 10observability: illegal_argument_exception - https://phabricator.wikimedia.org/T262429 (10Mholloway) Is this resolved? It looks like the last occurrences of this error happened on 9/9, and logs seem to be co... [20:12:43] 10serviceops, 10MW-on-K8s, 10Operations, 10TechCom-RFC: Decide on logging in k8s for ShellBox - https://phabricator.wikimedia.org/T263545 (10holger.knust) [22:34:40] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) [22:35:34] 10serviceops, 10Operations, 10Patch-For-Review: Decommission mw2135-mw2147, mw2187-mw2214 (all PowerEdge R420) - https://phabricator.wikimedia.org/T260654 (10Papaul) [22:35:41] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) 05Open→03Resolved This is complete