[06:25:36] 10serviceops, 10Operations: Deploy wikidiff2 v1.9.0 - https://phabricator.wikimedia.org/T234175 (10jijiki) [08:08:40] 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10Joe) >>! In T233828#5531056, @Krinkle wrote: > I think this should be fixed at the source in... [08:33:02] 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10Joe) >>! In T233828#5532958, @Joe wrote: >> The json fields here were modelled after MediaWi... [13:27:56] 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10Krinkle) p:05Triage→03High >>! In T233828#5532983, @Joe wrote: >[…] After looking furthe... [13:41:31] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10Mholloway) [13:43:11] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10Mholloway) I've confirmed with @JoeWalsh that the apps aren't using the /page/media... [14:57:09] 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10herron) Copying `excepetion.message` to `message` looks to have made an improvement here. I... [15:55:19] 10serviceops, 10Operations, 10Core Platform Team (Needs Cleaning - Services Operations): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Mholloway) [15:59:27] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10mobrovac) [16:03:04] 10serviceops, 10Operations, 10Core Platform Team (Needs Cleaning - Services Operations): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Mholloway) [16:18:06] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10Mholloway) Ah, I see now from https://github.com/wikimedia/mediawiki-services-chang... [16:20:03] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 4 others: Revisit the logging work done on Q1 2017-2018 for the standard pod setup - https://phabricator.wikimedia.org/T207200 (10akosiaris) [16:27:26] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 4 others: Revisit the logging work done on Q1 2017-2018 for the standard pod setup - https://phabricator.wikimedia.org/T207200 (10akosiaris) Logs are now making it to logstash so I am gonna boldly resolve this. That being... [16:43:05] 10serviceops, 10Operations, 10Release Pipeline, 10Release-Engineering-Team-TODO, and 4 others: Revisit the logging work done on Q1 2017-2018 for the standard pod setup - https://phabricator.wikimedia.org/T207200 (10akosiaris) 05Open→03Resolved [16:45:04] 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10Cmjohnson) a:03Jclark-ctr John, please reseat the green mgmt cable [17:05:21] 10serviceops, 10Operations, 10Core Platform Team (Needs Cleaning - Services Operations): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Jdforrester-WMF) [18:09:25] i am here now. i just need to also do 2 separate trainings and stuff [18:44:32] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Andrew) Is T210008.wikistats.eqiad.wmflabs associated with this bug? It has had broken puppet for many weeks -- perhaps it can be deleted? [18:45:33] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) @Andrew Yea, it is. I will look into it later today. [18:47:52] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) This ticket is superseded by T224247. krypton has meanwhile been replaced by miscweb1001/2001 (stretch). [18:48:53] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [18:48:56] 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn) [18:49:17] 10serviceops, 10Operations: upgrade and rename krypton & create its codfw equivalent - https://phabricator.wikimedia.org/T224247 (10Dzahn) [18:49:20] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) [18:59:49] 10serviceops, 10Operations: upgrade krypton (webserver_misc_apps) to stretch - https://phabricator.wikimedia.org/T210008 (10Dzahn) >>! In T210008#5535469, @Andrew wrote: > Is T210008.wikistats.eqiad.wmflabs associated with this bug? It has had broken puppet for many weeks -- perhaps it can be deleted? Per T2... [20:53:08] 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson @Cmjohnson Reseated green mgmt cable [21:01:36] 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10Dzahn) 05Open→03Resolved Thanks @Jclark-ctr @Cmjohnson It seems to be working fine right now. If it comes back we will just reopen this ticket. Calling it resolved tentatively.