[03:57:16] <jinxer-wm>	 FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike
[04:12:16] <jinxer-wm>	 RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1006.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike
[06:15:57] <jinxer-wm>	 FIRING: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw2002.frack.codfw.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn
[06:16:57] <jinxer-wm>	 FIRING: HostMemoryUtilizationCritical: Host high memory utilization (instance fransw2002.frack.codfw.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationCritical
[06:31:57] <jinxer-wm>	 RESOLVED: HostMemoryUtilizationCritical: Host high memory utilization (instance fransw2002.frack.codfw.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationCritical
[06:35:57] <jinxer-wm>	 RESOLVED: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw2002.frack.codfw.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn
[07:59:03] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_46) - 10https://gerrit.wikimedia.org/r/1285056 (owner: 10L10n-bot)
[07:59:39] <jinxer-wm>	 FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [12.4]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[08:04:39] <jinxer-wm>	 RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.2]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[08:11:22] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1283357 (owner: 10L10n-bot)
[08:25:10] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1285599 (owner: 10L10n-bot)
[08:43:39] <jinxer-wm>	 FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.6]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[08:51:08] <wikibugs>	 (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_45) - 10https://gerrit.wikimedia.org/r/1284256 (owner: 10L10n-bot)
[08:51:22] <wikibugs>	 (03CR) 10EMcFarland: "@murbanec@wikimedia.org Thanks again for abandoning a change that was discarded for a different approach. 🙏" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1272666 (https://phabricator.wikimedia.org/T422082) (owner: 10EMcFarland)
[08:53:46] <wikibugs>	 (03PS1) 10Eileen: Composer update - fairly tame [wikimedia/fundraising/cv] - 10https://gerrit.wikimedia.org/r/1285734
[08:54:19] <wikibugs>	 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Gerrit, 06Release-Engineering-Team: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11906694 (10Eileenmcnaughton) It seems...
[09:19:18] <wikibugs>	 (03CR) 10Raimond Spekking: [C:03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1285599 (owner: 10L10n-bot)
[09:43:39] <jinxer-wm>	 RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.0]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[11:22:39] <jinxer-wm>	 FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.0]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[11:23:16] <jinxer-wm>	 FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike
[11:27:39] <jinxer-wm>	 RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [10.3]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[11:54:39] <jinxer-wm>	 FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.5]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[11:58:16] <jinxer-wm>	 RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100)   - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike
[11:59:39] <jinxer-wm>	 RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.4]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages
[12:52:07] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] build: Updating dependencies [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1283222 (owner: 10Libraryupgrader)
[14:28:01] <ejegg|offsite>	 morning fr-tech
[14:28:53] <ejegg>	 my work machine is not booting so I'll be a bit unproductive this morning till I figure it out
[14:54:49] <jgleeson|skrun>	 Morning fr-tech
[14:58:28] <ejegg>	 ok, back in business
[15:03:36] <jgleeson>	 wb ejegg 
[15:26:11] <wikibugs>	 10fundraising-tech-ops: Reallocate disk space on frmon1002 and frmon2002 - https://phabricator.wikimedia.org/T424393#11908465 (10Jgreen) frmon1002 is done, /srv/ is 1.2T and 58% full
[15:29:11] <jinxer-wm>	 FIRING: NodeDown: Node frmon2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmon2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[15:34:11] <jinxer-wm>	 FIRING: NodeDown: Node frmon2002 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmon2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[15:37:46] <wikibugs>	 10fundraising-tech-ops: Reallocate disk space on frmon1002 and frmon2002 - https://phabricator.wikimedia.org/T424393#11908597 (10Jgreen) frmon2002 is done, /srv is 1.7T, 42% full
[15:38:12] <wikibugs>	 10fundraising-tech-ops: Reallocate disk space on frmon1002 and frmon2002 - https://phabricator.wikimedia.org/T424393#11908600 (10Jgreen) 05Open→03Resolved p:05Triage→03Medium a:03Jgreen
[15:39:11] <jinxer-wm>	 RESOLVED: NodeDown: Node frmon2002 has been down for long. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmon2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[15:44:11] <jinxer-wm>	 RESOLVED: NodeDown: Node frmon2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frmon2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[16:08:39] <jinxer-wm>	 FIRING: SystemdService: Systemd service coworker.service stopped on civi1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DSystemdService
[16:50:15] <wikibugs>	 10fundraising-tech-ops, 10observability: Deprecate Fundraising nsca icinga alert collection - https://phabricator.wikimedia.org/T425424#11909128 (10tappof) I believe you can go ahead with the plan you shared. I checked the configuration, and the other components in place are the NSCA daemon itself (which is in...
[17:07:57] <jinxer-wm>	 FIRING: MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004  - https://frmon.wikimedia.org/d/000000273/mariadb-system?orgId=1&var-server=frdb2003.frack.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning
[17:10:30] <wikibugs>	 10fundraising-tech-ops, 10observability, 13Patch-For-Review: Deprecate Fundraising nsca icinga alert collection - https://phabricator.wikimedia.org/T425424#11909250 (10Jgreen) >>! In T425424#11909128, @tappof wrote: > I believe you can go ahead with the plan you shared.  Ok great, I submitted the commit, if...
[17:12:57] <jinxer-wm>	 RESOLVED: [2x] MariadbReplicationRunning: MariaDB replica thread not running on : frdb2003.frack.codfw.wmnet:9004   - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationRunning
[18:01:57] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Don't merge identical email addresses with different location types [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282929 (https://phabricator.wikimedia.org/T425195) (owner: 10Lars SG)
[18:07:14] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface, and 2 others: Update fundraising code for PHP 8.4 - https://phabricator.wikimedia.org/T425578#11909564 (10Ejegg) Hi @Jdforrester-WMF I think the plan for the fundraising cluster is to go to trix...
[18:46:00] <wikibugs>	 10fundraising-tech-ops: Fundraising access request for passimacopoulos - https://phabricator.wikimedia.org/T425669#11909738 (10Dwisehaupt)
[18:48:20] <wikibugs>	 10fundraising-tech-ops: Fundraising access request for passimacopoulos - https://phabricator.wikimedia.org/T425669#11909759 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt Access was set up last week at the offsite. Following up with the paper trail.  ` Date: Thu, 7 May 2026 05:25:16 From: Lisa Seitz G...
[18:56:36] <wikibugs>	 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11909793 (10Jgreen)
[19:02:11] <wikibugs>	 10fundraising-tech-ops: Fundraising systems Debian Trixie support - https://phabricator.wikimedia.org/T401847#11909799 (10Ejegg)
[19:02:14] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface, and 2 others: Update fundraising code for PHP 8.4 - https://phabricator.wikimedia.org/T425578#11909800 (10Ejegg)
[19:16:39] <jinxer-wm>	 FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [16.2]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures
[19:16:39] <jinxer-wm>	 FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [16.2]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures
[19:16:39] <jinxer-wm>	 FIRING: HighPaymentSchemeFailures: Average  payment method scheme failures are high [16.2]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures
[19:21:39] <jinxer-wm>	 RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [5.1]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures
[19:21:39] <jinxer-wm>	 RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [5.1]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures
[19:21:39] <jinxer-wm>	 RESOLVED: HighPaymentSchemeFailures: Average  payment method scheme failures are high [5.1]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures
[19:30:51] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Composer update - fairly tame [wikimedia/fundraising/cv] - 10https://gerrit.wikimedia.org/r/1285734 (owner: 10Eileen)
[19:33:39] <jinxer-wm>	 FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [43.1]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore
[19:37:39] <wikibugs>	 10fundraising-tech-ops: Fundraising access request for Liv Burke - https://phabricator.wikimedia.org/T425990 (10IAckerman-WMF) 03NEW p:05Triage→03Medium
[19:38:39] <jinxer-wm>	 RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [42.1]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore
[19:38:42] <wikibugs>	 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add contribution search Searchkit/Formbuilder for DR - https://phabricator.wikimedia.org/T404987#11909939 (10Lars) Pending DR review: https://civicrm.wikimedia.org/civicrm/dr/donorfind...
[19:43:56] <wikibugs>	 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Get Paypal Fundraise Up Audit reports coming in by SFTP - https://phabricator.wikimedia.org/T424083#11909961 (10Cstone) trying to get my account login for this (having 2fa issues)
[19:46:02] <wikibugs>	 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Create a payment attempts table for fraud scoring service - https://phabricator.wikimedia.org/T417217#11909966 (10Cstone) At the offsite we also talk...
[19:46:25] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Clean up emails where contact has two of the same type and address after backfilling location types - https://phabricator.wikimedia.org/T425190#11909968 (10XenoRyet) p:05Triage→03Medium
[19:48:23] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Duplicate check - ty email - https://phabricator.wikimedia.org/T424788#11909971 (10XenoRyet) p:05Triage→03Medium
[19:48:45] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Civi using incorrect donor name - https://phabricator.wikimedia.org/T422957#11909973 (10XenoRyet) p:05Triage→03Medium
[19:49:31] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Duplicate check - ty email - https://phabricator.wikimedia.org/T424788#11909976 (10Ejegg) Let's revisit the idea of sending emails via a queue rather than a batch lookup: {T172300}
[19:55:39] <jinxer-wm>	 FIRING: HighPaymentsMedianRiskScore: Median payments risk_score is running high [94.2]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore
[20:00:39] <jinxer-wm>	 RESOLVED: HighPaymentsMedianRiskScore: Median payments risk_score is running high [90.5]   - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentsMedianRiskScore
[20:20:28] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Show donor's address in TY email when donors is in NY State - https://phabricator.wikimedia.org/T425992 (10Lars) 03NEW
[20:22:18] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Clean up communications opt out fields - https://phabricator.wikimedia.org/T425993 (10Lars) 03NEW
[20:27:11] <jinxer-wm>	 FIRING: NodeDown: Node pay-lb2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=pay-lb2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[20:27:16] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Implement new audience segments and statuses - https://phabricator.wikimedia.org/T425994 (10Lars) 03NEW
[20:28:47] <jinxer-wm>	 FIRING: HAProxyExporterDown: HAProxy on pay-lb2002.frack.codfw.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown
[20:31:41] <wikibugs>	 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Add contribution search Searchkit/Formbuilder for DR - https://phabricator.wikimedia.org/T404987#11910089 (10Lars) To do: Add as a managed entity for local testing use.
[20:32:11] <jinxer-wm>	 RESOLVED: NodeDown: Node pay-lb2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=pay-lb2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[20:33:47] <jinxer-wm>	 RESOLVED: HAProxyExporterDown: HAProxy on pay-lb2002.frack.codfw.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown
[20:34:07] <wikibugs>	 (03CR) 10Lars SG: [C:03+2] Update text of 'sent to damaged' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284662 (owner: 10Ejegg)
[20:34:41] <jinxer-wm>	 FIRING: [2x] NodeDown: Node pay-lb2001 is down. - TODO  - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[20:36:47] <jinxer-wm>	 FIRING: HAProxyExporterDown: HAProxy on pay-lb2001.frack.codfw.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown
[20:39:41] <jinxer-wm>	 RESOLVED: [2x] NodeDown: Node pay-lb2001 is down. - TODO  - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[20:41:47] <jinxer-wm>	 RESOLVED: [2x] HAProxyExporterDown: HAProxy on pay-lb2001.frack.codfw.wmnet is down   - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown
[20:59:25] <wikibugs>	 (03Merged) 10jenkins-bot: Update text of 'sent to damaged' [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1284662 (owner: 10Ejegg)
[21:10:53] <wikibugs>	 06Fundraising-Backlog, 06Reader Experience Team: Add Donate link in hamburger menu on mobile for logged in users - https://phabricator.wikimedia.org/T416101#11910227 (10Jdlrobson-WMF)
[21:32:41] <jinxer-wm>	 FIRING: [2x] NodeDown: Node payments2004 is down. - TODO  - https://alerts.wikimedia.org/?q=alertname%3DNodeDown
[21:33:01] <jinxer-wm>	 FIRING: [2x] BirdServiceDown: bird service down on pay-lb2001.frack.codfw.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DBirdServiceDown
[21:35:13] <dwisehaupt>	 ^^^ that is all me. the blanket maint week ignore had expired.
[23:33:39] <wikibugs>	 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Gerrit, 06Release-Engineering-Team: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11910511 (10thcipriani) >>! In T423961...