[05:18:37] <wikibugs>	 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10jijiki)
[06:31:23] <wikibugs>	 10serviceops, 10Operations: Update component/php72 to 7.2.22 - https://phabricator.wikimedia.org/T230024 (10jijiki) @Dzahn Is it ok if you upgrade on phab*?
[06:34:41] <wikibugs>	 10serviceops, 10Operations, 10HHVM, 10Performance-Team (Radar): Remove HHVM from production - https://phabricator.wikimedia.org/T229792 (10jijiki)
[16:43:58] <wikibugs>	 10serviceops: Free up space on mwdebug* - https://phabricator.wikimedia.org/T234063 (10jijiki)
[17:30:06] <wikibugs>	 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10Dzahn) Just tried to ssh to it now and it works for me. I get to login.
[17:51:44] <wikibugs>	 10serviceops, 10Operations, 10ops-eqiad: mw1286.mgmt is down - https://phabricator.wikimedia.org/T234009 (10Dzahn) p:05Triage→03Normal @Cmjohnson This seems to be flapping. Sometimes it works and sometimes it doesn't. Could you check for loose cable and/or switch port, maybe just reconnecting it will do it.
[18:23:13] <wikibugs>	 10serviceops: Free up space on mwdebug* - https://phabricator.wikimedia.org/T234063 (10Dzahn) 05Open→03Resolved
[20:52:06] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10Puppet: Puppet systemd::mask is an anti pattern that has unwanted side effect - https://phabricator.wikimedia.org/T233839 (10Dzahn)
[21:05:12] <wikibugs>	 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability: Errors managed by wmf-errors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10herron) These logs appear to be nesting the message field inside the exception field, and the message field at the r...
[21:17:45] <wikibugs>	 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by wmf-errors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10herron) A few ideas to address this:  Parse the nested exception field into the root of the lo...
[21:21:18] <wikibugs>	 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10Krinkle)
[21:24:41] <wikibugs>	 10serviceops, 10Operations, 10Wikimedia-Logstash, 10observability, 10Patch-For-Review: Errors managed by php-wmerrors (like OOMs) lack normalized_message on logstash - https://phabricator.wikimedia.org/T233828 (10Krinkle) I think this should be fixed at the source in [puppet: php7-fatal-error.php](https:...
[22:10:15] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn)
[22:13:04] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) >>! In T210411#5496180, @Vgutierrez wrote: > Please note that the docker-registry certificate is missing the public hostname: `docker-registry.wikimedia.org`  Per I...
[22:13:28] <wikibugs>	 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10Dzahn) https://performance.wikimedia.org switch to https://performance.discovery.wmnet as backend.
[22:40:35] <wikibugs>	 10serviceops, 10Operations: Update component/php72 to 7.2.22 - https://phabricator.wikimedia.org/T230024 (10Dzahn) @jijiki Any examples how it was done on the other servers?  Did you keep the locally modified php.ini and fpm/php.ini files? I let the package overwrite but then let puppet revert that.