[02:57:05] 10serviceops, 10Platform Engineering, 10Release-Engineering-Team-TODO, 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Ladsgroup) [07:50:53] 10serviceops, 10Operations: Ugrade the MediaWiki appservers to debian buster, icu63 - https://phabricator.wikimedia.org/T264991 (10Joe) [08:17:31] 10serviceops, 10Operations: Ugrade the MediaWiki appservers to debian buster, icu63 - https://phabricator.wikimedia.org/T264991 (10MoritzMuehlenhoff) I've created a standalone backport of icu63 in the component/icu63. Rebuilding PHP 7.2 with it is a little tricky, since PHP build-depends on libxml2 (for php7.2... [13:29:43] I'll be <5m late, sorry [17:17:49] 10serviceops, 10observability, 10User-jijiki: Handle unknown stats in rsyslog_exporter - https://phabricator.wikimedia.org/T210137 (10colewhite) p:05Triage→03Low a:03colewhite [17:18:48] 10serviceops, 10observability, 10User-jijiki: Handle unknown stats in rsyslog_exporter - https://phabricator.wikimedia.org/T210137 (10colewhite) Found a [[ https://github.com/aleroyer/rsyslog_exporter | new upstream ]] and have deployed it to netbox-dev2001 and centrallog1001 to run for a few days. If all c... [19:24:01] 10serviceops, 10Operations, 10Thumbor, 10Patch-For-Review, 10User-jijiki: Investigate systemd hardening to replace Firejail for Thumbor - https://phabricator.wikimedia.org/T212941 (10Ladsgroup) [21:28:39] 10serviceops, 10Machine Learning Platform, 10ORES, 10Okapi, and 3 others: ORES redis: max number of clients reached... - https://phabricator.wikimedia.org/T263910 (10calbon) {F32378655} Suspiciously similar just happened to ores on eqiad. I reset the uwsgi service on all ores100x boxes and will monitor.