[07:12:07] 10serviceops, 10Wikimedia-Rdbms, 10Performance-Team (Radar): mw1317 issue: "DatabaseMysqli.php: Class undefined: stdClass" - https://phabricator.wikimedia.org/T232233 (10Joe) 05Open→03Resolved a:03Joe I'm not sure about php7 - it was fixed when I got around fixing this machine. HHVM just needed a rest... [07:13:13] 10serviceops, 10Prod-Kubernetes, 10User-fsero: recreate codfw cluster state from code stored in deployment-charts with helmfile [MIGHT CAUSE DOWNTIME] - https://phabricator.wikimedia.org/T228837 (10Joe) 05Open→03Resolved [07:13:15] 10serviceops, 10Prod-Kubernetes, 10User-fsero: Kubernetes clusters roadmap - https://phabricator.wikimedia.org/T212123 (10Joe) [07:16:51] 10serviceops, 10Scap, 10PHP 7.2 support, 10Patch-For-Review, and 3 others: Enhance MediaWiki deployments for support of php7.x - https://phabricator.wikimedia.org/T224857 (10Joe) 05Open→03Resolved This is now 99% done. We just need a confctl release to be able to make scap pull work as intended. Resolv... [10:14:34] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Applayer services without TLS - https://phabricator.wikimedia.org/T210411 (10ema) [10:17:17] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: `tureis.codfw.wmnet` - tureis.codfw.wmnet - Removed from Puppet mast... [10:18:57] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2001 for hosts: `roentgenium.eqiad.wmnet` - roentgenium.eqiad.wmnet - Removed from P... [10:33:32] 10serviceops, 10Operations, 10Traffic, 10Patch-For-Review: Migrate Failoid hosts to Stretch/Buster - https://phabricator.wikimedia.org/T224559 (10MoritzMuehlenhoff) 05Open→03Resolved New instances (failoid1001 and failoid2001) have been set up with Buster and are in use. The old instances (roentgenium... [11:00:20] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10mobrovac) [12:39:56] 10serviceops, 10Mobile-Content-Service, 10Page Content Service, 10Patch-For-Review, 10Product-Infrastructure-Team-Backlog (Kanban): "worker died, restarting" mobileapps issue - https://phabricator.wikimedia.org/T229286 (10Mholloway) Stopping /page/definition pregeneration appears to have helped, but we'r... [14:07:56] 10serviceops, 10Operations, 10Thumbor, 10ops-eqiad, 10User-jijiki: (OoW) thumbor1004 memory errors - https://phabricator.wikimedia.org/T215411 (10jijiki) [14:53:33] 10serviceops, 10Thumbor, 10Performance-Team (Radar), 10User-jijiki: Terminate Thumbor with SSL - https://phabricator.wikimedia.org/T180696 (10jijiki) a:03jijiki [15:36:34] effie: i am back now [15:36:49] just a bit sick still [15:37:08] though much better than yesterday [16:40:26] 10serviceops, 10Operations, 10Patch-For-Review, 10Performance-Team (Radar), 10User-jijiki: Ramp up percentage of users on php7.2 to 100% on both API and appserver clusters - https://phabricator.wikimedia.org/T219150 (10Jdforrester-WMF) [17:02:48] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10BBlack) 05Resolved→03Open Re-open as this isn't really complete yet, the battery came in and replacement is proceeding. Since @jijiki did this before and claims it's just a depool command, w... [17:19:46] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10Papaul) Smart storage replacement complete. Embedded HPE Smart Storage Battery 875241-B21 878643-001 6WQXL0BB2BQ4H8 01 0.60 OK [17:40:17] 10serviceops, 10MediaWiki-General, 10Operations, 10CPT Initiatives (PHP7 (TEC4)), and 2 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10WDoranWMF) [17:53:59] 10serviceops, 10Operations, 10Release-Engineering-Team-TODO, 10Patch-For-Review, 10Wikimedia-Incident: docker-registry: some layers has been corrupted due to deleting other swift containers - https://phabricator.wikimedia.org/T228196 (10thcipriani) [17:59:37] 10serviceops, 10Operations, 10ops-codfw: (OoW) restbase2009 lockup - https://phabricator.wikimedia.org/T227408 (10Papaul) 05Open→03Resolved This can be resolved [18:39:22] 10serviceops, 10Operations, 10User-fgiunchedi, 10Wikimedia-production-error: PHP Fatal error: The UdpSocket to 127.0.0.1:10514 has been closed (from Monolog/SyslogUdp on mwdebug1002) - https://phabricator.wikimedia.org/T214734 (10Krinkle) [21:11:34] 10serviceops, 10ORES, 10Operations, 10Scoring-platform-team (Current): celery-ores-worker service failed on ores100[2,4,5] without any apparent reason or significant log - https://phabricator.wikimedia.org/T230917 (10Halfak) @Dzahn, what would it take to implement a check like the one I described above? I... [23:18:55] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) p:05Low→03Normal [23:19:05] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Dzahn) 05Stalled→03Open [23:20:09] 10serviceops, 10Gerrit, 10Operations, 10Release-Engineering-Team-TODO, 10Release-Engineering-Team (Development services): Gerrit Hardware Upgrade - https://phabricator.wikimedia.org/T222391 (10Paladox)