[00:13:19] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Handle Settlement from Audit process - https://phabricator.wikimedia.org/T399782 (10Eileenmcnaughton) 03NEW [00:21:52] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Handle Settlement from Audit process - https://phabricator.wikimedia.org/T399782#11011520 (10Eileenmcnaughton) [01:22:18] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07fr-current-sprint, 07payments-orchestration: Start updating the fee information into Civi once we get it - https://phabricator.wikimedia.org/T396071#11011661 (10Eileenmcnaughton) a:03Eileenmcnaughton Per outl... [02:22:27] (03PS1) 10Eileen: Standardise / fix getFields function [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170222 (https://phabricator.wikimedia.org/T396071) [03:34:30] (03CR) 10Eileen: [C:03+2] Implement Gravy payment method deletion webhook handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167625 (https://phabricator.wikimedia.org/T398723) (owner: 10Jgleeson) [03:35:01] (03Merged) 10jenkins-bot: Implement Gravy payment method deletion webhook handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167625 (https://phabricator.wikimedia.org/T398723) (owner: 10Jgleeson) [03:49:04] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1169876 (owner: 10L10n-bot) [04:07:11] RECOVERY - check_mailq on frmx1001 is OK: OK: postfix mailq (4194) is below threshold (5000/7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [05:58:02] 06Fundraising-Backlog, 07fr-acoustic: Acoustic fields wishlist - discussion - https://phabricator.wikimedia.org/T394089#11011835 (10MSuijkerbuijk_WMF) @AKanji-WMF In view of the no-email window needed for this, could we look at 2-3 days in July or August with Noah? (I can't find his handle in phab) Thanks [06:21:45] (03PS1) 10Eileen: First cut at WMFAudit::settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) [06:36:33] (03CR) 10CI reject: [V:04-1] First cut at WMFAudit::settle api [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170226 (https://phabricator.wikimedia.org/T396071) (owner: 10Eileen) [12:24:41] (03PS1) 10Jgleeson: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1170319 [12:25:30] (03CR) 10Jgleeson: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1170319 (owner: 10Jgleeson) [12:25:58] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1170319 (owner: 10Jgleeson) [12:28:59] !log SmashPig upgraded from f373fe4d to de30a87f [12:29:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:47:20] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Civi recurring donations display question - https://phabricator.wikimedia.org/T393227#11013704 (10krobinson) Thanks for confirming that, @Eileenmcnaughton . Can I confirm what will happen when we import over all recurrings to... [15:08:14] (03CR) 10Thiemo Kreuz (WMDE): [C:03+1] build: Updating mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1168019 (owner: 10Libraryupgrader) [15:11:13] (03PS1) 10Reedy: CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 [15:14:36] (03CR) 10CI reject: [V:04-1] CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 (owner: 10Reedy) [15:33:38] (03PS2) 10Reedy: CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 [15:33:38] (03PS1) 10Reedy: composer.json: Update mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170370 [15:36:57] (03CR) 10CI reject: [V:04-1] composer.json: Update mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170370 (owner: 10Reedy) [15:37:06] (03CR) 10CI reject: [V:04-1] CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 (owner: 10Reedy) [15:40:47] (03Abandoned) 10Reedy: composer.json: Update mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170370 (owner: 10Reedy) [15:41:00] (03PS3) 10Reedy: CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 [15:41:50] (03CR) 10Reedy: [C:03+2] build: Updating mediawiki/mediawiki-phan-config to 0.16.0 [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1168019 (owner: 10Libraryupgrader) [15:42:21] (03PS4) 10Reedy: CentralNoticeCampaignLogPager: Reduce code duplication [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170368 [15:46:28] 06Fundraising-Backlog, 10fundraising-tech-ops: Email send delays for yahoo and aol domains - https://phabricator.wikimedia.org/T399670#11014026 (10greg) Not sure what that middle plateau is about, but we're back down to a reasonable queue it seems. {F65061242} [15:49:12] 06Fundraising-Backlog: [Discussion] Recurring subscr_id handling: Mixed approaches across processors - Feature or Bug? - https://phabricator.wikimedia.org/T399868 (10jgleeson) 03NEW [16:02:28] (03PS1) 10Reedy: Campaigns: Remove out of date comment from campaignLogs() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170377 [16:02:29] (03PS1) 10Reedy: api: Update ApiBase constant usages [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170378 [16:05:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 91.10, 80.62, 43.82 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:08:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 97.91, 86.71, 51.90 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [16:10:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 79.55, 87.45, 57.17 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:10:19] (03PS1) 10Reedy: Use === comparison for _action [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170379 [16:13:13] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 31.38, 55.12, 48.21 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [16:15:01] I wonder why that is ^ I don't see anything obvious looking at the Security grafana dashboard [16:15:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 66.34, 70.48, 58.55 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:17:31] avishua: can you see if there's anything obvious in the logs or top for frban2002 that would be causing it to have such high load? [16:17:46] 06Fundraising-Backlog, 06FR-donorrelations: Consider alternate approaches to ingesting BT's + delayed payment methods to Civi - https://phabricator.wikimedia.org/T399871 (10MBeat33) 03NEW [16:18:11] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 6.19, 27.38, 38.50 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [16:19:05] greg-g looking now [16:20:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 2.77, 30.41, 44.83 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:20:37] im seeing high cpu load from kafkatee: [16:20:37] ``` [16:20:37] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND [16:20:37] 1386 kafkatee 20 0 56.5g 47.2g 6060 S 211.9 75.7 16w+0d kafkatee [16:20:37] ``` [16:25:17] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 1.86, 12.49, 33.05 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [16:26:22] (03PS1) 10Reedy: CentralNoticeCampaignLogPager: Fix comment [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170385 [16:36:39] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Two recurring donations missing from CiviCRM. - https://phabricator.wikimedia.org/T399471#11014276 (10MBeat33) There's nothing relevant in Civi for Recurring GRAVY% + $7 around the date, so it doesn't look like a dedupe error. [16:43:13] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 2.81, 3.07, 10.00 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [16:50:17] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 2.22, 4.00, 9.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [17:16:35] 06Fundraising-Backlog, 07fr-acoustic: Acoustic fields wishlist - discussion - https://phabricator.wikimedia.org/T394089#11014497 (10AKanji-WMF) Adding in @nisrael [17:46:31] (03PS1) 10Reedy: README: Remove $wgCentralDBname [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170394 (https://phabricator.wikimedia.org/T389348) [17:56:51] (03PS1) 10Reedy: Campaign: Add and use ability to read from primary in getCampaignSettings [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170395 (https://phabricator.wikimedia.org/T389348) [17:59:53] (03CR) 10CI reject: [V:04-1] Campaign: Add and use ability to read from primary in getCampaignSettings [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170395 (https://phabricator.wikimedia.org/T389348) (owner: 10Reedy) [18:03:21] (03PS2) 10Reedy: Campaign: Add and use ability to read from primary in getCampaignSettings [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170395 (https://phabricator.wikimedia.org/T389348) [18:07:53] (03PS6) 10Reedy: Rename CentralNotice to SpecialCentralNotice for clarity [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1011339 (owner: 10Jforrester) [18:10:44] (03CR) 10CI reject: [V:04-1] Rename CentralNotice to SpecialCentralNotice for clarity [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1011339 (owner: 10Jforrester) [18:10:48] (03PS2) 10Reedy: Move short-list helper into a dedicated class [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/833775 (https://phabricator.wikimedia.org/T318222) (owner: 10Awight) [18:23:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Card Payment AVS check triggering false positives. - https://phabricator.wikimedia.org/T396355#11014804 (10EMartin) I was able to get someone with an AMEX card to test another scenario which no includes 23 testing out ok: https://docs.google.com... [18:36:13] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (2229367) = 96.1% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [18:36:32] (03PS3) 10Umherirrender: build: Updating mediawiki/mediawiki-phan-config to 0.16.0 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1168358 (owner: 10Libraryupgrader) [18:41:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (2229367) = 96.1% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [18:46:03] (03PS1) 10Reedy: CentralNotice: Remove return from outputHeader() [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170406 [18:46:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (2229367) = 97.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [18:47:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (1197902) = 96.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [18:51:27] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (2229367) = 98.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [18:52:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (1197902) = 96.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [18:56:49] heh, speaking of weird alerts avishua, but at least this one is fransw :) ^ [18:57:03] oh geez [18:57:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (1197902) = 96.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [18:57:32] thats usually the one we see with issues [18:58:10] im gonna unmute this channel [19:01:47] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 99% used. Largest process: trino-server-co (2229367) = 98.7% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [19:06:13] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (2229367) = 95.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [19:10:18] 10BDC-Implementation: Bring Wikipedia traffic data into the data lakehouse - https://phabricator.wikimedia.org/T399893 (10IAckerman-WMF) 03NEW [19:11:13] RECOVERY - check_memory on fransw2003 is OK: OK Memory 80% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [19:12:13] RECOVERY - check_memory on fransw2002 is OK: OK Memory 66% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [20:17:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Card Payment AVS check triggering false positives. - https://phabricator.wikimedia.org/T396355#11015170 (10EMartin) @jgleeson @RKumar_WMF I was able to find some AMEX testers and was able to test the AVS results as above. I've captured the results... [21:29:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Gravy Card Payment AVS check triggering false positives. - https://phabricator.wikimedia.org/T396355#11015335 (10jgleeson) Thanks @EMartin, that's great! [21:48:17] (03PS2) 10Reedy: CentralNoticeCampaignLogPager: Fix comment [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170385 [21:48:21] (03CR) 10Umherirrender: [C:03+2] CentralNoticeCampaignLogPager: Fix comment [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170385 (owner: 10Reedy) [22:04:10] (03Merged) 10jenkins-bot: CentralNoticeCampaignLogPager: Fix comment [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1170385 (owner: 10Reedy) [22:16:41] 06Fundraising-Backlog: [Discussion] Recurring subscr_id handling: Mixed approaches across processors - Feature or Bug? - https://phabricator.wikimedia.org/T399868#11015398 (10jgleeson) After some digging, I found this in DonationQueueConsumer: `php if ($message->getSubscriptionID()) { $gateway_subscr_id =... [22:21:40] 06Fundraising-Backlog, 07Epic: Epic: Migrate Master Suppression List to Civi - https://phabricator.wikimedia.org/T399907 (10AKanji-WMF) 03NEW [22:38:55] (03PS1) 10Jgleeson: WIP: Use recurring_payment_token for Gravy PayPal subscription trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170438 (https://phabricator.wikimedia.org/T399868) [22:39:45] (03PS2) 10Jgleeson: WIP: Use recurring_payment_token for Gravy PayPal subscription trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170438 (https://phabricator.wikimedia.org/T399868) [22:40:17] (03PS3) 10Jgleeson: WIP: Use recurring_payment_token for Gravy PayPal subscription trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170438 (https://phabricator.wikimedia.org/T399868) [22:41:08] fr-tech that will temporarily fix our gravy paypal deletion issues ^ [22:41:16] lemme take that out of WIP [22:41:50] (03PS4) 10Jgleeson: Temp Fix: Use recurring_payment_token for Gravy PayPal subscription trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170438 (https://phabricator.wikimedia.org/T399868)