[00:00:17] so, if not thankyou emails what is queuing up in the mailqueue? [00:02:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7512 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [00:02:54] (I can't ACK the alert in icinga, going to ask prod sre for perms for that or whatever) [00:07:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7513 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [00:12:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7513 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [00:12:21] ACKNOWLEDGEMENT - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7513 (threshold c = 7500) BryanDavis greg-g I cant ACK an icinga alert that is firing that I want to silence. Can someone do that for me? its the frmx1001 check_mailq service. Default of 24 hour is fine. - The acknowledgement expires at: 2025-07-17 00:11:34. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [00:34:22] he jeff, thanks for joining, I'll send you the email i was just about to send to fr-tech [00:34:24] hey* [00:34:44] Jeff_Green: just emailed my summary [01:15:26] 06Fundraising-Backlog, 10fundraising-tech-ops: Email send delays for yahoo and aol domains - https://phabricator.wikimedia.org/T399670 (10greg) 03NEW p:05Triage→03High [04:36:52] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10MediaWiki-extensions-CentralNotice, 10Temporary accounts, 06Trust and Safety Product Team: [Temporary Accounts] Update CentralNotice extension to support Temporary Accounts - https://phabricator.wikimedia.org/T374437#11007565 (10Niharika) [04:47:10] RECOVERY - check_mailq on frmx1001 is OK: OK: postfix mailq (3666) is below threshold (5000/7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [06:44:11] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1169876 (owner: 10L10n-bot) [06:49:20] (03PS1) 10Eileen: Minor logging output improvements [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170019 [06:49:20] (03PS1) 10Eileen: Loosen Adyen audit regex [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170020 (https://phabricator.wikimedia.org/T399034) [07:23:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11007946 (10Eileenmcnaughton) I dug into the audit files and transaction LSDHB27JGXR2MRF3 appears in the audit files and payment listener... [07:23:52] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 13Patch-For-Review: Adyen daily audit file check - https://phabricator.wikimedia.org/T399034#11007948 (10Eileenmcnaughton) [07:53:10] 06Fundraising-Backlog, 10Quibble, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Remove reliance on EXECUTOR_NUMBER environment variable in CI - https://phabricator.wikimedia.org/T399283#11008023 (10hashar) 05Open→03Resolved a:03hashar [10:01:06] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 10FR-email-preferences: Comms preference centre - scope adding to Acoustic email footers - https://phabricator.wikimedia.org/T392742#11008455 (10ppenloglou) Talked with fr-online regarding the pref center link, everyone's h... [13:15:36] (03CR) 10Jgleeson: [C:03+2] "LGTM! I think we started receiving these in relation to the 'one report to rule them all' reconciliation report project." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170020 (https://phabricator.wikimedia.org/T399034) (owner: 10Eileen) [14:56:11] 06Fundraising-Backlog, 10fundraising-tech-ops: Email send delays for yahoo and aol domains - https://phabricator.wikimedia.org/T399670#11009666 (10greg) Now it's just sitting at ~3k in the mailqueue for some reason: {F64811429} [17:31:30] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Add browserInfo parameters to Gravy for Adyen 3DS flow compatibility - https://phabricator.wikimedia.org/T398845#11010321 (10jgleeson) [19:07:16] (03CR) 10Jgleeson: [C:03+2] Minor logging output improvements [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170019 (owner: 10Eileen) [19:24:22] (03Merged) 10jenkins-bot: Minor logging output improvements [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170019 (owner: 10Eileen) [19:24:23] (03Merged) 10jenkins-bot: Loosen Adyen audit regex [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1170020 (https://phabricator.wikimedia.org/T399034) (owner: 10Eileen) [19:52:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7624 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [19:57:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7732 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:02:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7856 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:07:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 7969 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:12:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8099 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:17:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8182 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:22:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8346 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:27:10] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8461 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:27:16] annnd there it goes again. [20:32:12] PROBLEM - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8550 (threshold c = 7500) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:33:34] ACKNOWLEDGEMENT - check_mailq on frmx1001 is CRITICAL: CRITICAL: postfix mailq is 8550 (threshold c = 7500) BryanDavis greg-g bd808: can you do the alert ACK/silence again, please? this time until monday the 21st at 22:00 UTC (yeah, sorry, Jeff is out, situation is fine, really the alert threshold should be adjusted, but thats a discussion with him when hes back) -  check_mailq on frmx1001 - The acknowledgement expires at: 2025-07-21 2 [20:33:34] tps://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx1001&service=check_mailq [20:58:26] (03PS6) 10Jgleeson: Implement Gravy payment method deletion webhook handling [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1167625 (https://phabricator.wikimedia.org/T398723) [21:00:06] eileen: I've pushed up all the changes we discussed on our recorded tech-talk review in the above PS ^^^. Thanks for reviewing that with me! [21:00:29] thanks jgleeson I'll take a look [21:11:33] ty! [21:24:14] 06Fundraising-Backlog: Benevity import not converting foreign currency matches to USD - https://phabricator.wikimedia.org/T399772 (10MDemosWMF) 03NEW [21:55:43] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog: Add partial_name_match AVS response code to Gravy fraud filters - https://phabricator.wikimedia.org/T399775 (10jgleeson) 03NEW [22:01:31] (03PS1) 10Jgleeson: Add support for partial_name_match AVS code. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) [22:02:00] (03CR) 10CI reject: [V:04-1] Add support for partial_name_match AVS code. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) (owner: 10Jgleeson) [22:03:34] (03PS2) 10Jgleeson: Add support for partial_name_match AVS code. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) [22:08:51] (03PS3) 10Jgleeson: Add support for Gravy 'partial_name_match' AVS result. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1170214 (https://phabricator.wikimedia.org/T399775) [22:09:34] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1170215 [22:09:54] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1170215 (owner: 10Eileen) [22:10:58] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1170215 (owner: 10Eileen) [22:27:57] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Map Settlement data to Civi field and table locations - https://phabricator.wikimedia.org/T397662#11011326 (10Eileenmcnaughton) [22:29:48] 03Fundraising Sprint: Naan-Sequitur, 06Fundraising-Backlog, 07fr-current-sprint: Map Settlement data to Civi field and table locations - https://phabricator.wikimedia.org/T397662#11011334 (10Eileenmcnaughton) I just renamed this because I think the task is really about how we handle settlement data (final fe... [22:59:59] !log civicrm upgraded from 71b58ed5 to bf098cc5 [23:00:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log