[01:21:04] FIRING: DiskSpaceWarn: Disk space WARNING for (instance frmon1002.frack.eqiad.wmnet:9100):/srv - https://alerts.wikimedia.org/?q=alertname%3DDiskSpaceWarn [02:30:57] FIRING: MariadbReplicationLag: MariaDB replica frdb1006.frack.eqiad.wmnet:9004 behind origin: 368 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [02:39:57] FIRING: MariadbReplicationLag: MariaDB replica frdb1006.frack.eqiad.wmnet:9004 behind origin: 908 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [02:49:57] RESOLVED: MariadbReplicationLag: MariaDB replica frdb1006.frack.eqiad.wmnet:9004 behind origin: 1328 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [02:50:57] RESOLVED: MariadbReplicationLag: MariaDB replica frdb1006.frack.eqiad.wmnet:9004 behind origin: 1328 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [04:20:57] FIRING: MariadbReplicationLag: MariaDB replica frdb1007.frack.eqiad.wmnet:9004 behind origin: 559 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [04:25:57] RESOLVED: MariadbReplicationLag: MariaDB replica frdb1007.frack.eqiad.wmnet:9004 behind origin: 619 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMariadbReplicationLag [04:55:11] FIRING: NodeDown: Node frav1003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [04:57:24] ^^^ the newstyle alerts we should be seeing now after the transition from icinga to prometheus/alertmanager for our monitoring. the text of alerts still needs to be fleshed out. [04:57:33] FIRING: RsyslogBacklog: 3 ryslog files queued up on on frauth1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [05:00:11] RESOLVED: NodeDown: Node frav1003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frav1003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [05:02:33] RESOLVED: RsyslogBacklog: 3 ryslog files queued up on on frauth1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DRsyslogBacklog [05:17:11] FIRING: NodeDown: Node frdb1007 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdb1007 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [05:19:20] PROBLEM - Host frdb1007 is DOWN: PING CRITICAL - Packet loss = 100% [05:21:04] FIRING: DiskSpaceWarn: Disk space WARNING for (instance frmon1002.frack.eqiad.wmnet:9100):/srv - https://alerts.wikimedia.org/?q=alertname%3DDiskSpaceWarn [05:21:57] FIRING: MariadbNoReplicas: No MariaDB replicas connected to : frdb2003.frack.codfw.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [05:22:11] FIRING: [2x] NodeDown: Node frdb1007 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [05:22:19] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_44) - 10https://gerrit.wikimedia.org/r/1283357 (owner: 10L10n-bot) [05:23:09] sorry for the noise. still figuring out how to do the preemptive downtimes for restarts. [05:23:12] RECOVERY - Host frdb1007 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms [05:26:57] RESOLVED: MariadbNoReplicas: No MariaDB replicas connected to : frdb2003.frack.codfw.wmnet:9004 - https://alerts.wikimedia.org/?q=alertname%3DMariadbNoReplicas [05:32:08] restarting frmon1002 for kernel update. grafana will be unavailable for a vew. [05:32:11] few even. [05:34:49] ok. it's back. [06:23:41] RESOLVED: [2x] NodeDown: Node payments1009 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:25:57] FIRING: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [06:30:11] FIRING: NodeDown: Node franio2004 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio2004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:34:42] FIRING: MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [06:35:11] RESOLVED: NodeDown: Node franio2004 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=franio2004 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:35:41] FIRING: [2x] NodeDown: Node franio2003 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:35:57] RESOLVED: HostMemoryUtilizationWarn: Host high memory utilization (instance fransw2003.frack.codfw.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DHostMemoryUtilizationWarn [06:39:42] RESOLVED: MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [06:40:41] FIRING: [3x] NodeDown: Node franio2002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:41:42] FIRING: [2x] MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [06:45:41] RESOLVED: [3x] NodeDown: Node franio2002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:46:42] RESOLVED: [2x] MinIONodesOffline: Instance franio2001.frack.codfw.wmnet:9000 has 1 offline node/s - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DMinIONodesOffline [06:50:41] FIRING: [2x] NodeDown: Node franio2001 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [06:55:41] RESOLVED: [3x] NodeDown: Node franio2001 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [07:28:23] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Campaign mix in local black out hours - https://phabricator.wikimedia.org/T425508 (10greg) 03NEW [07:45:11] FIRING: NodeDown: Node fransw2003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [07:45:47] FIRING: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [07:50:11] RESOLVED: NodeDown: Node fransw2003 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw2003 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [07:50:47] RESOLVED: HAProxyExporterDown: HAProxy on fransw2003.frack.codfw.wmnet is down - https://alerts.wikimedia.org/?q=alertname%3DHAProxyExporterDown [07:55:11] FIRING: NodeDown: Node fransw2002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=fransw2002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:00:11] RESOLVED: [2x] NodeDown: Node fransw2002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [08:15:40] (03PS1) 10Ejegg: Use trixie / PHP8.4 images for CiviCRM [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1283626 (https://phabricator.wikimedia.org/T424774) [08:20:18] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: Campaign mixin - local blackout hours - https://phabricator.wikimedia.org/T425508#11893189 (10Ejegg) a:03Ejegg [08:24:09] (03CR) 10Eileen: [C:03+2] Review access change [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1281817 (https://phabricator.wikimedia.org/T423961) (owner: 10Thcipriani) [08:24:28] (03CR) 10Eileen: [V:03+2 C:03+2] Review access change [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1281817 (https://phabricator.wikimedia.org/T423961) (owner: 10Thcipriani) [08:28:36] 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Gerrit, and 2 others: Figure out how to maintain cv repo - given it is primarily committed to by non-WMF - https://phabricator.wikimedia.org/T423961#11893239 (10Eileenmcnaughton) thanks heaps for that -... [08:28:37] (03PS1) 10Ejegg: Civix upgrade for matching-gifts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283635 (https://phabricator.wikimedia.org/T417702) [08:33:37] (03PS2) 10Ejegg: Suppress selected recurring charge alerts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282400 [08:40:00] (03PS1) 10Eileen: Push out id repair date threshold [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283639 [08:40:36] (03CR) 10Ejegg: [C:03+2] Push out id repair date threshold [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283639 (owner: 10Eileen) [08:41:57] (03PS2) 10Ejegg: Map paypal payer ID to backendProcessorContactID [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1277613 (https://phabricator.wikimedia.org/T424213) [08:51:36] (03PS1) 10Ejegg: Refactor: class to map Adyen ID and errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283643 (https://phabricator.wikimedia.org/T423982) [08:52:10] (03CR) 10CI reject: [V:04-1] Refactor: class to map Adyen ID and errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283643 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [08:54:32] (03PS2) 10Ejegg: Refactor: class to map Adyen ID and errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283643 (https://phabricator.wikimedia.org/T423982) [08:56:15] (03CR) 10Eileen: [C:03+2] Suppress selected recurring charge alerts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282400 (owner: 10Ejegg) [08:56:40] (03CR) 10Eileen: [C:03+2] Civix upgrade for matching-gifts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283635 (https://phabricator.wikimedia.org/T417702) (owner: 10Ejegg) [09:03:33] (03Merged) 10jenkins-bot: Push out id repair date threshold [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283639 (owner: 10Eileen) [09:07:37] (03CR) 10Eileen: [C:03+2] Refactor: class to map Adyen ID and errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283643 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [09:11:02] (03PS2) 10Ejegg: Civix upgrade for matching-gifts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283635 (https://phabricator.wikimedia.org/T417702) [09:11:02] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283655 [09:12:04] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283655 (owner: 10Eileen) [09:15:53] !log civicrm upgraded from 38dcf7a8 to 4d9c8600 [09:15:54] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [09:18:23] (03Merged) 10jenkins-bot: Suppress selected recurring charge alerts [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1282400 (owner: 10Ejegg) [09:22:02] (03PS1) 10Eileen: Also repair native adyen capture & auth -> backend_trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283659 [09:32:06] (03CR) 10Ejegg: [C:03+2] Also repair native adyen capture & auth -> backend_trxn_id [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283659 (owner: 10Eileen) [09:39:38] (03PS1) 10Ejegg: Subclass Adyen response mapper for capture response [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) [09:40:08] (03CR) 10CI reject: [V:04-1] Subclass Adyen response mapper for capture response [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) (owner: 10Ejegg) [09:46:10] (03CR) 10Sergio Gimeno: [C:03+1] "Tested along with the GrowthExperiments change, works as expected. Leaving room for maintainers to approve." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1282396 (https://phabricator.wikimedia.org/T422082) (owner: 10Ejegg) [09:53:53] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11893509 (10Lars) @AStein-... [09:54:52] (03PS2) 10Ejegg: Subclass Adyen response mapper for capture response [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) [09:55:33] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283678 [09:55:58] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283678 (owner: 10Eileen) [09:57:56] (03CR) 10CI reject: [V:04-1] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283678 (owner: 10Eileen) [09:58:42] (03Merged) 10jenkins-bot: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1283678 (owner: 10Eileen) [09:59:20] !log civicrm upgraded from 4d9c8600 to 24ac90e7 [09:59:21] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [10:03:04] (03PS1) 10Ejegg: Add chariot to generic ParseAuditFile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283685 [10:14:27] (03PS12) 10Eileen: Get chariot tests working & various updates in this pre-alpha stage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1281749 (https://phabricator.wikimedia.org/T419044) [10:14:30] (03CR) 10Ejegg: [C:03+2] Get chariot tests working & various updates in this pre-alpha stage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1281749 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [10:15:35] (03CR) 10Eileen: [C:03+2] Add chariot to generic ParseAuditFile [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283685 (owner: 10Ejegg) [10:17:36] (03Merged) 10jenkins-bot: Get chariot tests working & various updates in this pre-alpha stage [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1281749 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [10:19:12] (03PS3) 10Ejegg: Subclass Adyen response mapper for capture response [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283671 (https://phabricator.wikimedia.org/T423982) [10:21:04] FIRING: DiskSpaceWarn: Disk space WARNING for (instance frmon1002.frack.eqiad.wmnet:9100):/srv - https://alerts.wikimedia.org/?q=alertname%3DDiskSpaceWarn [10:22:41] FIRING: [2x] NodeDown: Node frlog2002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [10:27:41] RESOLVED: [2x] NodeDown: Node frlog2002 is down. - TODO - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [10:36:41] (03PS1) 10Ejegg: Subclass Adyen RefundPayment response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 [10:37:14] (03CR) 10CI reject: [V:04-1] Subclass Adyen RefundPayment response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 (owner: 10Ejegg) [10:39:52] (03PS2) 10Ejegg: Subclass Adyen RefundPayment response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 [10:40:40] (03PS3) 10Ejegg: Subclass Adyen RefundPayment response mapper [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283705 [10:53:35] (03PS1) 10Eileen: Merge branch 'master' of https://github.com/wikimedia/wikimedia-fundraising-SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1283712 [10:54:08] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://github.com/wikimedia/wikimedia-fundraising-SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1283712 (owner: 10Eileen) [10:54:40] (03Merged) 10jenkins-bot: Merge branch 'master' of https://github.com/wikimedia/wikimedia-fundraising-SmashPig into deployment [wikimedia/fundraising/SmashPig] (deployment) - 10https://gerrit.wikimedia.org/r/1283712 (owner: 10Eileen) [10:56:01] !log SmashPig upgraded from 4201ef56 to bb833986 [10:56:02] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [11:25:16] FIRING: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:00:16] RESOLVED: ContextSwitchingSpike: Host context switching high (instance frdb1007.frack.eqiad.wmnet:9100) - https://alerts.wikimedia.org/?q=alertname%3DContextSwitchingSpike [12:22:52] (03CR) 10Wfan: [C:03+2] Map paypal payer ID to backendProcessorContactID [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1277613 (https://phabricator.wikimedia.org/T424213) (owner: 10Ejegg) [12:23:28] (03Merged) 10jenkins-bot: Map paypal payer ID to backendProcessorContactID [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1277613 (https://phabricator.wikimedia.org/T424213) (owner: 10Ejegg) [12:29:33] !log config revision changed from 41cfd677 to 00752f91 [12:29:34] Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL [12:36:04] FIRING: [2x] DiskSpaceWarn: Disk space WARNING for (instance civi1002.frack.eqiad.wmnet:9100):/tmp - https://alerts.wikimedia.org/?q=alertname%3DDiskSpaceWarn [12:44:15] 10fundraising-tech-ops: Finish DNS/NAT setup for frdata1003 - https://phabricator.wikimedia.org/T425539 (10Jgreen) 03NEW [12:45:49] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install frdata1003, frmx1002, frqueue100[5-6] - https://phabricator.wikimedia.org/T416249#11893942 (10Jgreen) [12:46:58] 10fundraising-tech-ops: Finish DNS/NAT setup for frdata1003 - https://phabricator.wikimedia.org/T425539#11893956 (10Jgreen) [12:47:00] 10fundraising-tech-ops, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:rack/setup/install frdata1003, frmx1002, frqueue100[5-6] - https://phabricator.wikimedia.org/T416249#11893955 (10Jgreen) [13:02:39] FIRING: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [11.2] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [13:03:49] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Chariot import - get api access and investigate pulling in DAF (in particular PayPal ... - https://phabricator.wikimedia.org/T415825#11894059 [13:05:39] FIRING: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [10.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [13:05:39] FIRING: HighPaymentMethodFailures: Average cc payment method failures are high [10.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [13:05:39] FIRING: HighPaymentSchemeFailures: Average payment method scheme failures are high [10.6] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [13:10:39] RESOLVED: HighPaymentGatewayFailures: Average gravy payment gateway failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentGatewayFailures [13:10:39] RESOLVED: HighPaymentSchemeFailures: Average payment method scheme failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentSchemeFailures [13:10:39] RESOLVED: HighPaymentMethodFailures: Average cc payment method failures are high [7.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentMethodFailures [13:10:58] (03PS1) 10Eileen: Upgrading wikimedia/omnimail-silverpop (1.31 => 1.32) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283759 [13:19:22] (03CR) 10Ejegg: [C:03+2] Upgrading wikimedia/omnimail-silverpop (1.31 => 1.32) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283759 (owner: 10Eileen) [13:33:55] (03CR) 10CI reject: [V:04-1] Upgrading wikimedia/omnimail-silverpop (1.31 => 1.32) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283759 (owner: 10Eileen) [13:44:52] 10fundraising-tech-ops, 13Patch-For-Review: Finish DNS/NAT setup for frdata1003 - https://phabricator.wikimedia.org/T425539#11894215 (10Jgreen) DNS is updated [13:49:36] 10fundraising-tech-ops, 13Patch-For-Review: Finish DNS/NAT setup for frdata1003 - https://phabricator.wikimedia.org/T425539#11894240 (10Jgreen) [13:49:55] 10fundraising-tech-ops, 13Patch-For-Review: Finish DNS/NAT setup for frdata1003 - https://phabricator.wikimedia.org/T425539#11894247 (10Jgreen) 05Open→03Resolved a:03Jgreen [13:56:24] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283780 [13:56:48] (03PS2) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283780 [14:02:39] RESOLVED: HighPaymentFraudMessages: More than 10 payment fraud messages in the past hour [13.1] - https://alerts.wikimedia.org/?q=alertname%3DHighPaymentFraudMessages [14:16:00] 10fundraising-tech-ops, 10observability: Deprecate Fundraising nsca icinga alert collection - https://phabricator.wikimedia.org/T425424#11894498 (10hnowlan) a:03tappof [14:19:18] (03CR) 10CI reject: [V:04-1] Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283780 (owner: 10Eileen) [14:23:39] (03Abandoned) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283780 (owner: 10Eileen) [14:26:24] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283795 [14:26:59] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283795 (owner: 10Eileen) [14:34:55] (03PS4) 10Eileen: Use fileName (only) to track file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) [14:34:55] (03PS3) 10Eileen: Extract queuing of audit message [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) [14:34:55] (03PS6) 10Eileen: Fix assumption that the batch is initiated before the date is added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281754 (https://phabricator.wikimedia.org/T419044) [14:34:56] (03PS2) 10Eileen: Move setting Campaign to somewhere less opinionated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281995 (https://phabricator.wikimedia.org/T419044) [14:36:44] (03CR) 10CI reject: [V:04-1] Use fileName (only) to track file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [14:40:23] (03PS1) 10Eileen: Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) [14:48:39] (03PS2) 10Eileen: Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) [14:51:31] (03CR) 10CI reject: [V:04-1] Upgrading wikimedia/smash-pig (v1.2.4.6 => v1.2.4.7) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283795 (owner: 10Eileen) [14:59:56] (03CR) 10CI reject: [V:04-1] Move setting Campaign to somewhere less opinionated [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281995 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:00:31] (03CR) 10CI reject: [V:04-1] Fix assumption that the batch is initiated before the date is added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281754 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:01:35] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:04:11] (03CR) 10CI reject: [V:04-1] Extract queuing of audit message [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281988 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:07:11] FIRING: NodeDown: Node frdev1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdev1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:12:11] RESOLVED: NodeDown: Node frdev1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=frdev1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [15:17:30] (03CR) 10CI reject: [V:04-1] Extract getImportTimerName, allow for more logic to be added [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1283802 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:35:34] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1281753 (https://phabricator.wikimedia.org/T419044) (owner: 10Eileen) [15:41:15] (03PS1) 10Ejegg: Mixin to suppress banners during certain hours [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1283818 (https://phabricator.wikimedia.org/T425508) [16:13:08] 14Fundraising Sprint - GNU England Shaker dresser, 03Fundraising Sprint Hutch Ado About Nothing, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add link in Zendesk to contact in CiviCRM & unsubscribe form by email - https://phabricator.wikimedia.org/T415378#11895327 (10Lars) @KHansen... [16:15:20] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations: Provide API for key CiviCRM data by email address to load in Zendesk - https://phabricator.wikimedia.org/T421245#11895363 (10Lars) @KHansen also requests that we also implement this for Wikipedia store ZD, not a huge priority, but w... [16:41:32] (03CR) 10Ejegg: [C:04-1] "Oops, needs a small correction. If blackoutCrossesMidnight, then nowIsBetweenStartAndEnd is never going to be true." [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1283818 (https://phabricator.wikimedia.org/T425508) (owner: 10Ejegg) [16:53:32] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Improve matching gift verification & thank you process - https://phabricator.wikimedia.org/T425577 (10Lars) 03NEW [16:54:01] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface: Update fundraising code for PHP 8.4 - https://phabricator.wikimedia.org/T425578 (10Ejegg) 03NEW [16:54:10] (03PS1) 10Ejegg: Update php-queue and predis dependencies [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1283843 (https://phabricator.wikimedia.org/T425578) [18:06:29] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Smashpig, 10MediaWiki-extensions-DonationInterface, and 2 others: Update fundraising code for PHP 8.4 - https://phabricator.wikimedia.org/T425578#11895787 (10Reedy) [18:34:11] FIRING: NodeDown: Node civi1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:35:28] FIRING: OOM: OOM killer active on civi1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DOOM [18:39:11] RESOLVED: NodeDown: Node civi1002 is down. - TODO - https://frmon.wikimedia.org/d/000000377/host-overview?orgId=1&var-host=civi1002 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [18:40:28] RESOLVED: OOM: OOM killer active on civi1002.frack.eqiad.wmnet:9100 - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DOOM