[07:43:53] 10serviceops, 10Graphoid, 10Operations, 10MW-1.35-notes (1.35.0-wmf.34; 2020-05-26), 10Platform Engineering (Icebox): Undeploy graphoid - https://phabricator.wikimedia.org/T242855 (10Gilles) [10:14:21] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10akosiaris) Layout of the process for quick reference on Monday: * Disable puppet on mendelevium. Command for that `sudo disable-puppet "P... [11:15:11] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10akosiaris) >>! In T187984#6435616, @jcrespo wrote: > @akosiaris This is my proposal, db-wise: > > * Disable https://ticket-test.wikimedia... [12:30:35] did kubernetes have issues overnight? I'm seeing 21 evicted pods in codfw's api-gateway mostly with age 6h55m [12:37:46] hnowlan: nothing I know of, will take a look [12:38:40] thanks. doesn't look like it caused issues longterm but might be a sign of something [12:55:23] hnowlan: looks like we had disk pressure on node(s) [12:56:11] so nothing really irregular [12:56:22] jsonpath magic for reference: kubectl get pods --all-namespaces -o jsonpath='{range .items[?(@.status.reason=="Evicted")]}{.metadata.name}{"\t"}{.status.startTime}{"\t"}{.status.message}{"\n"}{end}' [13:11:35] akosiaris: ever thought of ingesting kubernetes events to elasticsearch? AFAICT we're only keeping them for an hour in etcd which is not ideal... [13:12:07] but keeping them longer in etcd is not ideal as well ofc :-) [13:59:55] jayme: ah, cool [14:00:52] 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Store Kubernetes events for more than one hour - https://phabricator.wikimedia.org/T262675 (10JMeybohm) p:05Triage→03Medium [15:15:03] 10serviceops, 10OTRS, 10Operations, 10Patch-For-Review, 10User-notice: Update OTRS to the latest stable version (6.0.x) - https://phabricator.wikimedia.org/T187984 (10jcrespo) > * At some point before the maintenance, clone only otrs database into db1077 again and make it replicate from m2 primary (db110... [17:17:07] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) [17:34:08] 10serviceops, 10Operations, 10Patch-For-Review: move 20 new codfw parsoid servers (parse2*) into production - https://phabricator.wikimedia.org/T247441 (10Dzahn) p:05Medium→03High [17:34:56] 10serviceops, 10Operations, 10ops-codfw: decommission mw2135-mw2147, mw2187-mw2214 - physical / datacenter part - https://phabricator.wikimedia.org/T261524 (10Papaul) [17:38:06] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10Golbez81) Might be too late, but can I highly urge the project to step back the PHP requirem... [18:19:53] 10serviceops, 10Operations, 10Wikimedia-production-error: PHP7 corruption reports in 2020 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) [19:16:37] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10MarkAHershberger) RHEL 8 has PHP 7.3.20 available. To use it: ` yum module install php:7.3/* ` [19:23:53] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10Reedy) >>! In T257879#6454753, @Golbez81 wrote: > Might be too late, but can I highly urge t... [19:39:28] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10MarkAHershberger) Reedy, it is especially hard to find the official RH versions without a su... [19:52:04] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10MoritzMuehlenhoff) >>! In T257879#6455213, @Reedy wrote: >>>! In T257879#6454753, @Golbez81... [19:53:39] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10Reedy) Yay, paywalls! :) So if 7.3.20 is officially supported by RH... The question lies wi... [21:20:53] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10Golbez81) I have module php:7.3 installed on CentOS Linux release 8.2.2004 and it's currentl... [21:35:05] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10Golbez81) Ok I guess CentOS 8 is just lagging a little behind as usual. My RHEL8 Azure mach... [21:38:48] 10serviceops, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10Release-Engineering-Team, and 5 others: Drop PHP 7.2 support in MediaWiki 1.35; require PHP 7.3.19 - https://phabricator.wikimedia.org/T257879 (10MarkAHershberger) FWIW, the maintainer of the RH package is the same person who maintains th...