[00:26:13] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1575) = 93.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [00:31:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1575) = 94.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [00:36:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1575) = 94.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [00:41:15] RECOVERY - check_memory on fransw2003 is OK: OK Memory 63% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [01:23:52] (03PS8) 10Wfan: Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) [01:26:11] (03CR) 10Wfan: [C:03+2] Avoid warning when no checksum given [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211255 (owner: 10Ejegg) [01:34:38] (03Merged) 10jenkins-bot: Avoid warning when no checksum given [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211255 (owner: 10Ejegg) [01:50:27] thanks wfan! [01:50:49] I am refactoring the popup component with slot [01:51:05] great! [01:51:30] Trying to fixing the unit test now, thanks for this hint~ could be nice to save the trouble for translation, thanks! [01:52:42] and even more important, it lets us be flexible with the content, without having to mix different things in the popup component! [01:56:13] dwisehaupt: if you're still around, I have a puppet patch to add the path for the silverpop stats export to prometheus in my homedir on frpm: 0001-Add-settings-for-silverpop-export-stats.patch [01:57:13] I went with the stat names you suggested in https://phabricator.wikimedia.org/T410071#11372549 [01:57:46] well, the simpler version, with just phase [01:59:45] i'll take a look at that. [01:59:54] thanks! [02:02:42] huh, about once an hour we get a failure to load the donations page because of a 'Temporary failure in name resolution' looking up the redis server [02:02:45] ok. merged and puppet run forced on civi1002 [02:02:49] seems to be all payments1005 [02:02:55] thanks dwisehaupt ! [02:02:58] odd, is that back again? [02:03:08] ah, have you seen it before? [02:03:29] we have. but it's not anything we can spot or figure out why it's happening yet. [02:03:29] Yeah, I just saw a couple in the last 2 hours so I checked the log from a couple days ago and saw 27 [02:03:49] we could make the code a bit more persistent [02:04:11] i mean, we retry API calls to payment providers, we might as well retry calls to our own infrastructure [02:04:12] and i think it's only been on that host, not the others iirc [02:05:55] (03PS1) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1211297 [02:05:58] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1211297 (owner: 10Ejegg) [02:06:28] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/1211297 (owner: 10Ejegg) [02:07:02] i'll add it to the list of things to investigate again. see if we can dig in some more. [02:07:11] but i need to go eat dinner with the family. [02:07:23] bon apetit [02:07:36] ok [02:07:54] agreed [02:09:24] hmm, so we could install the TimedMediaHandler extension on donorwiki and get a videojs player with a decent interface [02:09:37] or just get the plain Video.js from https://videojs.org [02:11:30] oh man, 'Support level: Pending The Reader Growth Team is the maintainer of last resort for the time being.' (on the TimedMediaHandler extension's doc page) [02:11:42] well, I guess I'll look into video.js :) [02:12:23] (03PS9) 10Wfan: Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) [02:12:38] Guess it should use that ForeignResourceDir thing again [02:14:50] !log fundraising python tools upgraded from 7efd562a to 8e900e85 [02:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:26:18] (03PS10) 10Wfan: Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) [02:30:09] (03PS1) 10Eileen: Declare some more api fields for visibility [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211298 [02:37:06] (03PS10) 10Wfan: Donor Portal: Add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) [02:40:39] (03PS11) 10Wfan: Donor Portal: Add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) [02:41:20] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Duplicate donations made via Apple Pay and Google Pay from the WP app - https://phabricator.wikimedia.org/T407383#11408135 (10Cstone) I think I was able to replicate this today, but the logs on our ends look good. I'm asking if the... [02:50:52] (03PS1) 10Eileen: Address deprecation warning [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211300 [02:50:53] (03PS1) 10Eileen: Updated version of contact layout editor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211301 [02:52:46] (03PS11) 10Ejegg: Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) (owner: 10Wfan) [02:53:15] (03CR) 10Ejegg: "PS11: whitespace normalization, one qqq text change." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) (owner: 10Wfan) [02:53:27] (03CR) 10Ejegg: [C:03+2] Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) (owner: 10Wfan) [02:54:57] thx, ejegg~ will do that wikiminute later, and do we still want that coming soon? [02:55:34] (03Merged) 10jenkins-bot: Donor Portal: Add popup component [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208015 (https://phabricator.wikimedia.org/T396271) (owner: 10Wfan) [02:55:37] wfan: I guess it depends if they decide they want to go ahead with the flow to actually create a new recurring donation [02:56:13] I have a couple wording changes for qqq.json in the 'problem login popup' too - do you mind if I just make those in the patch? [03:09:03] (03PS12) 10Ejegg: Donor Portal: Add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) (owner: 10Wfan) [03:11:28] (03CR) 10Ejegg: [C:03+2] Donor Portal: Add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) (owner: 10Wfan) [03:15:38] (03PS1) 10Ejegg: Remove mustache-related DonorPortal code [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211305 [03:16:30] (03CR) 10Ejegg: [C:03+2] Declare some more api fields for visibility [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211298 (owner: 10Eileen) [03:17:28] (03PS1) 10Ejegg: Add VideoJS player [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211306 [03:18:59] so we could use ^^^ to get some controls for the player without having to add the TimedMediaHandler extension to payments + donorwiki, though it'll be about 700k more JS to handle a click that most folks probably won't make [03:19:41] the TimedMediaHandler extension gets that down to closer to 300k by loading only the 'core' bits of videojs and I think implementing their own skin [03:19:55] oops, gotta get to bed [03:20:05] g'night! [03:32:40] (03Merged) 10jenkins-bot: Donor Portal: Add problem login popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203193 (https://phabricator.wikimedia.org/T407453) (owner: 10Wfan) [03:32:49] (03Merged) 10jenkins-bot: Declare some more api fields for visibility [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211298 (owner: 10Eileen) [03:59:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review Monolog file output config - https://phabricator.wikimedia.org/T411067 (10Eileenmcnaughton) 03NEW [04:25:04] (03CR) 10Wfan: [C:03+2] Remove mustache-related DonorPortal code [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211305 (owner: 10Ejegg) [04:25:24] cool will check tmr, good night~ [05:06:22] (03PS1) 10Eileen: Debug output to generate a command to try for replication [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211332 (https://phabricator.wikimedia.org/T383194) [05:36:54] (03Merged) 10jenkins-bot: Remove mustache-related DonorPortal code [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211305 (owner: 10Ejegg) [06:12:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1555) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:14:29] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 05MW-1.46-notes (1.46.0-wmf.3; 2025-11-19): Validation for AmEx BINS in India - https://phabricator.wikimedia.org/T409211#11408209 (10RKumar_WMF) There’s no widespread “Discover credit card” issued in India like Visa or Mastercard... [06:17:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1555) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:22:11] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1555) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:27:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1555) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:30:50] (03PS1) 10Eileen: Fix merge fatal error [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211344 (https://phabricator.wikimedia.org/T383194) [06:32:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1555) = 95.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:33:06] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Review Monolog file output config - https://phabricator.wikimedia.org/T411067#11408226 (10Eileenmcnaughton) https://civicrm.wikimedia.org/civicrm/search#/display/Monolog%20configuration/Monologs is the config [06:37:13] PROBLEM - check_memory on fransw2002 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1555) = 94.0% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:42:13] RECOVERY - check_memory on fransw2002 is OK: OK Memory 58% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2002&service=check_memory [06:48:32] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1211395 (owner: 10L10n-bot) [09:43:53] (03PS1) 10Eileen: Fix setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211612 [09:53:59] (03CR) 10Eileen: [C:03+2] "Self-merging due to others not being around..." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211612 (owner: 10Eileen) [10:07:18] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1211619 [10:07:28] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1211619 (owner: 10Eileen) [10:08:58] !log civicrm upgraded from e7df633d to 96715bf8 [10:09:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:17:41] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1211395 (owner: 10L10n-bot) [12:19:17] (03CR) 10Damilare Adedoyin: [C:03+2] Fix setting [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211612 (owner: 10Eileen) [13:12:19] seems my local civi proxy error is ssl related [13:22:26] and some weird behaviour with the volume mounting, the config.php and proxy.php isn't getting mounted [13:29:44] cleared all the volumes and running setup on all services to see if the fresh install would get them mounted [14:22:05] Ok looks like running the full setup fixed it, thanks all for your help [14:57:13] sorry damilare I should have sent an email last night [14:57:36] I also found a problem with a very recent (not yet deployed) patch to do more validation on the chechsum parameter [14:57:55] so I submitted a couple fixes for that last night and wfan merged them [14:57:56] np ejegg I saw that too, thanks for finding that [14:58:05] fixing* [15:08:38] (03PS3) 10Thiemo Kreuz (WMDE): Fix special pages not calling the parent constructor [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1206832 [15:08:43] (03CR) 10Ejegg: [C:03+2] Fix special pages not calling the parent constructor [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1206832 (owner: 10Thiemo Kreuz (WMDE)) [15:11:37] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1211701 [15:11:39] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1211701 (owner: 10Ejegg) [15:17:01] (03Merged) 10jenkins-bot: Fix special pages not calling the parent constructor [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1206832 (owner: 10Thiemo Kreuz (WMDE)) [15:17:02] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/1211701 (owner: 10Ejegg) [16:11:56] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Don't show banners on action=submit - https://phabricator.wikimedia.org/T411109 (10Pcoombe) 03NEW [16:16:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22220 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:16:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22224 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:21:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22520 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:21:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22524 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:26:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22820 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:26:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22824 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:26:17] (03PS1) 10Ejegg: Skip showing banners on action=submit [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1211716 (https://phabricator.wikimedia.org/T411109) [16:28:04] the db slow replica alert went off yesterday around this time as well, cc Jef, dw :( [16:31:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23120 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:31:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23124 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:32:39] (03CR) 10Damilare Adedoyin: [C:03+2] Skip showing banners on action=submit [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1211716 (https://phabricator.wikimedia.org/T411109) (owner: 10Ejegg) [16:33:11] Thanks for the quick fix ejegg, lgtm! [16:34:51] awesome damilare [16:36:11] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23421 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:36:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23424 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:36:42] (03Merged) 10jenkins-bot: Skip showing banners on action=submit [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1211716 (https://phabricator.wikimedia.org/T411109) (owner: 10Ejegg) [16:40:26] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 13Patch-For-Review: Don't show banners on action=submit - https://phabricator.wikimedia.org/T411109#11410137 (10Ejegg) [16:40:47] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 13Patch-For-Review: Don't show banners on action=submit - https://phabricator.wikimedia.org/T411109#11410139 (10Ejegg) a:03Ejegg [16:41:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23720 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:41:12] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23724 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:46:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24020 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:46:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24024 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:46:19] !log donorwiki upgraded from 2a73a08d to bb179e9c [16:46:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:47:32] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: US addresses not displaying correctly on Summary page after updating - https://phabricator.wikimedia.org/T410765#11410174 (10Lars) @SHust I'm not sure I understand the bug here, as I've been unable... [16:51:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24320 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:51:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24324 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [16:56:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24620 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [16:56:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24624 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:01:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24920 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:01:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24924 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:04:19] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 07fr-current-sprint, 05MW-1.46-notes (1.46.0-wmf.5; 2025-12-02): Prevent [ change email ] from breaking onto multiple lines in email preferences centre - https://phabricator.wikimedia.org/T409576#11410257 (10Ejegg) this is deployed [17:06:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22542 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:06:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23067 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:10:30] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Active recurring next donation date in the past - https://phabricator.wikimedia.org/T410592#11410288 (10Cstone) We would also see this when manually doing test donations on recurring charges, they can also get stuck in the 23 day difference check [17:15:12] (03Abandoned) 10Ejegg: PoC: use slots for problems logging in popup [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211260 (owner: 10Ejegg) [17:15:27] !log payments-wiki upgraded from f254a5ab to bb179e9c [17:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:16:09] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 1117912 Threads: 4 Questions: 198100574 Slow queries: 600 Opens: 25454 Open tables: 1472 Queries per second avg: 177.205 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10920 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:16:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 1118073 Threads: 4 Questions: 197755958 Slow queries: 601 Opens: 24940 Open tables: 1396 Queries per second avg: 176.872 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12406 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:31:16] (03CR) 10Wfan: [C:03+2] Show error on duplicate banner name when cloning [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1205227 (https://phabricator.wikimedia.org/T286673) (owner: 10Ejegg) [17:36:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5644 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:36:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7315 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:37:24] (03Merged) 10jenkins-bot: Show error on duplicate banner name when cloning [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1205227 (https://phabricator.wikimedia.org/T286673) (owner: 10Ejegg) [17:41:08] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 1119412 Threads: 4 Questions: 202172611 Slow queries: 604 Opens: 28399 Open tables: 1477 Queries per second avg: 180.606 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:41:12] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5914 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:44:09] ^^^ known, it's just about caught up. [17:51:14] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 1120174 Threads: 4 Questions: 202209694 Slow queries: 605 Opens: 28311 Open tables: 1399 Queries per second avg: 180.516 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:06:14] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: US addresses not displaying correctly on Summary page after updating - https://phabricator.wikimedia.org/T410765#11410526 (10SHust) @Lars, I created CID 69158152: The country was originally set to... [18:29:06] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: US addresses not displaying correctly on Summary page after updating - https://phabricator.wikimedia.org/T410765#11410614 (10Lars) p:05Triage→03Medium @SHust Thanks, I think we are seeing the s... [18:58:32] (03PS2) 10Ejegg: Add VideoJS player [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211306 [19:36:24] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fix civicrm/report/wmffraud/fredge results - https://phabricator.wikimedia.org/T411127 (10jgleeson) 03NEW [19:36:33] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Fix civicrm/report/wmffraud/fredge results - https://phabricator.wikimedia.org/T411127#11410922 (10jgleeson) [19:37:55] (03PS1) 10Ejegg: Turn on minfraud for Apple and GPay [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211761 [19:38:15] fr-tech ^^^ is what mbeat wanted [19:38:31] dwisehaupt: could you please unlock payments db for me? [19:38:44] sure thing. [19:39:13] (03CR) 10Jgleeson: [C:03+2] "LGTM" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211761 (owner: 10Ejegg) [19:41:32] ejegg: unlocked and puppet runs completed on the payments hosts. [19:41:58] (03Merged) 10jenkins-bot: Turn on minfraud for Apple and GPay [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211761 (owner: 10Ejegg) [19:58:33] (03CR) 10Ejegg: [C:03+2] WP25 appeal templates for 3 more languages [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1211208 (https://phabricator.wikimedia.org/T411031) (owner: 10Ejegg) [20:08:01] (03PS16) 10Wfan: Donor Portal: Add coming soon model [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1203180 (https://phabricator.wikimedia.org/T396271) [20:21:58] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 10FR-payments-general, 13Patch-For-Review: WP25 Payments Wiki - Localization (fr, pt-br, es-419) - https://phabricator.wikimedia.org/T411031#11411002 (10Ejegg) [20:25:45] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 10FR-payments-general, 13Patch-For-Review: WP25 Payments Wiki - Localization (fr, pt-br, es-419) - https://phabricator.wikimedia.org/T411031#11411006 (10Ejegg) OK @ppenloglou these should be live. I had to slightly modify the text of the third paragr... [20:46:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: CIDs in November Direct Mail list are not found in Civi - Cannot import Activity Update - https://phabricator.wikimedia.org/T410789#11411036 (10MDemosWMF) Thank you @Lars ! These are now in Civi. [21:02:17] (03PS5) 10Wfan: Reuse the popup modal for wikiminute [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1208481 (https://phabricator.wikimedia.org/T407645) [21:02:44] (03CR) 10Wfan: [V:04-1] "we can just add controls to video https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/1208481" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211306 (owner: 10Ejegg) [21:05:55] 06Fundraising-Backlog: Figure out phone numbers from other sources - https://phabricator.wikimedia.org/T411133 (10Cstone) 03NEW [21:09:18] (03PS1) 10Eileen: Add Venmo as source [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211800 [21:11:41] (03CR) 10Lars SG: [C:03+2] Add Venmo as source [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211800 (owner: 10Eileen) [21:22:23] (03CR) 10Wfan: [V:03+2] Add Venmo as source [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211800 (owner: 10Eileen) [21:23:58] ejegg: just let me know when you are all done with the payments updates and i can relock the db. [21:25:38] ok dwisehaupt, I'm not seeing the styles on mobile as I'd expected, so I'm checking what might be different from the en appeal [21:26:50] 06Fundraising-Backlog: Add fees to braintree audit - https://phabricator.wikimedia.org/T411136 (10Cstone) 03NEW [21:28:14] (03Abandoned) 10Ejegg: Add VideoJS player [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1211306 (owner: 10Ejegg) [21:28:57] (03Merged) 10jenkins-bot: Add Venmo as source [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211800 (owner: 10Eileen) [21:29:39] sure thing. no rush, just didn't want to forget. [21:30:14] dwisehaupt: oh huh, I don't actually see a color change on mobile for en either. Maybe I'm misremembering [21:30:37] dwisehaupt: might as well lock it back up for now and if there are edit requests by monday I'll do them then [21:30:54] ok. will do. [21:39:40] 06Fundraising-Backlog: Put a different favicon on frdev civi - https://phabricator.wikimedia.org/T411142 (10Cstone) 03NEW [21:40:42] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Missing Braintree transactions - https://phabricator.wikimedia.org/T411143 (10Eileenmcnaughton) 03NEW [21:45:13] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Missing Braintree transactions - https://phabricator.wikimedia.org/T411143#11411261 (10Cstone) These are all our gravy venmo test transactions linking {T409599} [21:46:46] (03CR) 10Ejegg: [V:03+2 C:03+2] WP25 appeal templates for 3 more languages [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1211208 (https://phabricator.wikimedia.org/T411031) (owner: 10Ejegg) [21:47:15] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Prevent any @lists.wikimedia.org email addresses from becoming Civi contacts (or at least opted out of all communication) - https://phabricator.wikimedia.org/T411042#11411282 (10greg) For the record, the list of emails in my screenshot above have been mar... [21:52:00] 03Fundraising Sprint WWaally hoo, 06Fundraising-Backlog, 07Fundraising-FRUP, 13Patch-For-Review: Remove Fundraiseup from donatewiki CSP - https://phabricator.wikimedia.org/T410737#11411295 (10Ejegg) All set, the headers are gone. [22:14:24] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Feature Request: Add a “Recurring Active” Indicator on the Manual Merging Screen - https://phabricator.wikimedia.org/T408703#11411384 (10Lars) The additional request here is also to add something similar in Deduper (we'll just add t... [22:29:30] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Merging Screen Feature request - https://phabricator.wikimedia.org/T403626#11411404 (10SHust) [22:29:33] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Feature Request: Add a “Recurring Active” Indicator on the Manual Merging Screen - https://phabricator.wikimedia.org/T408703#11411407 (10SHust) →14Duplicate dup:03T403626 [22:35:20] 06Fundraising-Backlog, 06FR-donorrelations: Optimize donation form 'maximum amount' messaging for method selected - https://phabricator.wikimedia.org/T410790#11411418 (10MBeat33) Adyen is still in the process of raising the SEPA limit to $1k, I'll share here when it's confirmed [22:53:11] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Civi Merging Screen Feature request - https://phabricator.wikimedia.org/T403626#11411459 (10Lars) The additional request here is also to add something similar in Deduper (we'll just add these to the ext... [22:56:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1575) = 95.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [23:01:15] PROBLEM - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1575) = 95.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [23:05:40] ACKNOWLEDGEMENT - check_memory on fransw2003 is CRITICAL: CRIT Memory 96% used. Largest process: trino-server-co (1575) = 95.5% Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [23:11:13] RECOVERY - check_memory on fransw2003 is OK: OK Memory 71% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2003&service=check_memory [23:18:19] (03PS1) 10Eileen: Add caching [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211834 (https://phabricator.wikimedia.org/T402837) [23:26:39] (03PS1) 10Eileen: Cache whole setting fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211839 (https://phabricator.wikimedia.org/T402837) [23:28:03] (03CR) 10Lars SG: [C:03+2] Cache whole setting fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211839 (https://phabricator.wikimedia.org/T402837) (owner: 10Eileen) [23:28:24] (03PS2) 10Eileen: Cache whole setting fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211839 (https://phabricator.wikimedia.org/T402837) [23:32:49] (03PS1) 10Ejegg: Update querylog parser for new date format [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211843 [23:53:21] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211839 (https://phabricator.wikimedia.org/T402837) (owner: 10Eileen) [23:57:08] (03PS1) 10Eileen: Add Venmo phone source, if true [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1211849 (https://phabricator.wikimedia.org/T411133)