[00:07:18] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for mritch - https://phabricator.wikimedia.org/T415428 (10Dwisehaupt) 03NEW
[00:10:48] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific channel - https://phabricator.wikimedia.org/T415389#11550632 (10bd808) Would https://wikitech.wikimedia.org/wiki/Fundraising/SAL be a reasonable place to record the `!log` messages?  Would you like these `!log` messages...
[00:14:12] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific channel - https://phabricator.wikimedia.org/T415389#11550633 (10greg) Both are fine. And I assume the separate view on https://sal.toolforge.org/ is just going to be via having a `fundraising` entry on the /projects list?
[00:19:34] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific channel - https://phabricator.wikimedia.org/T415389#11550634 (10bd808) >>! In T415389#11550633, @greg wrote: > Both are fine. And I assume the separate view on https://sal.toolforge.org/ is just going to be via having a...
[00:20:02] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389#11550638 (10bd808)
[00:21:14] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389#11550639 (10greg) 👍
[00:21:34] <greg-g>	 heh, that thumb looks big in my The Lounge view
[00:24:28] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: gravy MISSING_PREDECESSOR - https://phabricator.wikimedia.org/T415430 (10AnnWF) 03NEW
[00:31:10] <bd808>	 greg-g: one last question: Do you expect to want to transclude that new SAL on any other page? The difference for that is that I would set the daily headers to H3 instead of H2 in that case. This is also a thing that can be changed in the future if you say no now and decide to transclude later.
[00:35:55] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:install (1) SSD each into franio200[1-3] - https://phabricator.wikimedia.org/T405982#11550681 (10Dwisehaupt) @Jhancock.wm Could we do this some time the week of 1/26? Any time that works for you would be ok, just let us know and we'll keep an eye. So...
[00:41:30] <greg-g>	 bd808: not planning on it
[00:41:37] <greg-g>	 bd808: thanks for letting me nerd snipe you
[00:42:01] <bd808>	 I am very easy to nerd snipe when I'm supposed to working hypothesis wording ;)
[00:43:07] <greg-g>	 hah, you're welcome :)
[00:43:39] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389#11550684 (10bd808) 05Open→03In progress a:03bd808
[00:45:20] <bd808>	 "Setting up a new SAL for FR-Tech will increase social bonds between FR-Tech and bd808 while reducing search toil for FR-Tech by 17% or more"
[00:45:48] <cstone>	 Hah bd808 
[00:48:58] <wikibugs>	 (03PS1) 10Cstone: Revert "Specify the type of trustly transactions to send chargeback messages for." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231077
[00:50:46] <wikibugs>	 (03CR) 10Eileen: "Short term revert seems safe enough - the noise seems to be failed payments coming through as refunds - but the originals were never final" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231077 (owner: 10Cstone)
[00:55:04] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389#11550709 (10bd808) Stashbot's config is kept in private files on Toolforge because I have been too lazy to update things to read secrets from envvars instead of embedding t...
[00:55:14] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022 (owner: 10Eileen)
[00:55:26] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Stop if host mismatch [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231029 (owner: 10Eileen)
[00:55:41] <bd808>	 !log Testing #wikimedia-fundraising SAL integration (T415389)
[00:55:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[00:55:44] <stashbot>	 T415389: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389
[00:55:45] <wikibugs>	 (03Merged) 10jenkins-bot: Deal with false-negative errors on host key check [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231022 (owner: 10Eileen)
[00:55:56] <wikibugs>	 (03Merged) 10jenkins-bot: Stop if host mismatch [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231029 (owner: 10Eileen)
[00:56:21] <greg-g>	 neat! https://sal.toolforge.org/fundraising
[00:58:29] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops, 10Stashbot: Create a Fundraising specific SAL - https://phabricator.wikimedia.org/T415389#11550713 (10bd808) 05In progress→03Resolved * https://wikitech.wikimedia.org/w/index.php?title=Fundraising/SAL&oldid=2377054 * https://wikimedia.social/@sal/1159473...
[01:00:15] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Revert "Specify the type of trustly transactions to send chargeback messages for." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231077 (owner: 10Cstone)
[01:00:49] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Specify the type of trustly transactions to send chargeback messages for." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1231077 (owner: 10Cstone)
[01:01:30] <greg-g>	 dwisehaupt: ^ now all we need is to have tools announce in here with !log when we want them to (do so publicly)
[01:01:40] <dwisehaupt>	 nice.
[01:01:58] <cstone>	 what is this fancy
[01:02:28] <greg-g>	 <3 to bd808 
[01:04:09] <bd808>	 You can also setup some kind of `dologmsg` stuff on your servers to make logging from an ssh session easier. The prod integration with that works with logmsgbot, but in Toolforge we let wm--bot do the work of talking on the IRC side.
[01:05:10] <bd808>	 What we use in Toolforge: https://github.com/wikimedia/operations-puppet/blob/aaa83bc8d82608ecace538bd4a1081be9efde1ca/modules/profile/manifests/toolforge/dologmsg.pp#L4
[01:05:21] <dwisehaupt>	 ok. we'll have to look into how to do this, either with our existing irc logger or in some other way.
[01:06:08] <wikibugs>	 (03PS1) 10Cstone: Upgrade smash-pig to v1.1.1.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231088
[01:06:13] <bd808>	 The prod version is https://github.com/wikimedia/operations-puppet/blob/aaa83bc8d82608ecace538bd4a1081be9efde1ca/modules/scap/files/dologmsg
[01:06:54] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Upgrade smash-pig to v1.1.1.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231088 (owner: 10Cstone)
[01:07:26] <dwisehaupt>	 thanks. i'll have a look.
[01:08:30] <greg-g>	 dwisehaupt: obviously no rush, it was an idle thought that turned into a nerd snipe :)
[01:08:46] <greg-g>	 (a recurring idle thought, but still)
[01:09:56] <dwisehaupt>	 yeah, i've been thinking about how to better get info from tools for a bit.
[01:10:29] <dwisehaupt>	 if we can get it to prometheus we can overlay "events" on graphs. and if we can get it into there, it should be easy to get into SAL now also.
[01:11:06] <wikibugs>	 (03PS1) 10Cstone: Revert "Do not skip Trustly refund when Contribution is not found." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231089
[01:12:48] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07fr-payments-ACH: Recurring ACH looks like a success but then is actually failed - https://phabricator.wikimedia.org/T409471#11550728 (10Cstone) I'm reverting the two patches for the weekend as they are causing failmail of all the incomplete transactio...
[01:12:59] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Revert "Do not skip Trustly refund when Contribution is not found." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231089 (owner: 10Cstone)
[01:22:06] <wikibugs>	 (03Merged) 10jenkins-bot: Upgrade smash-pig to v1.1.1.2 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231088 (owner: 10Cstone)
[01:31:23] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Do not skip Trustly refund when Contribution is not found." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1231089 (owner: 10Cstone)
[01:45:36] <wikibugs>	 (03PS1) 10Cstone: Update smash-pig to v1.1.1.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1231091
[01:46:46] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Update smash-pig to v1.1.1.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1231091 (owner: 10Cstone)
[01:59:11] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 0725 Dec Q3 Planning: Create Recurring Upgrade and Recurring Downgrade activities when completed by DR through CiviCRM UI - https://phabricator.wikimedia.org/T409097#11550752 (10Lars) @CHudson-WMF From the donor portal, we...
[02:07:56] <wikibugs>	 (03Merged) 10jenkins-bot: Update smash-pig to v1.1.1.2 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1231091 (owner: 10Cstone)
[02:09:18] <wikibugs>	 (03PS1) 10Cstone: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1231092
[02:09:55] <wikibugs>	 (03CR) 10Cstone: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1231092 (owner: 10Cstone)
[02:10:46] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1231092 (owner: 10Cstone)
[02:19:08] <cstone>	 !log civicrm upgraded from f7064a46 to 19f94835
[02:19:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[02:19:45] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Chariot import fails when there is no exact match found for organization name - https://phabricator.wikimedia.org/T415295#11550755 (10Lars) @MDemosWMF Thanks for this, having the error details...
[02:51:54] <wikibugs>	 (03CR) 10CI reject: [V:04-1] build: Updating npm dependencies [extensions/FundraisingTranslateWorkflow] (REL1_43) - 10https://gerrit.wikimedia.org/r/1157359 (owner: 10Libraryupgrader)
[06:27:11] <icinga-wm>	 PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1567) = 96.3% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[06:32:09] <icinga-wm>	 PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1567) = 96.3% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[06:37:09] <icinga-wm>	 PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 95% used. Largest process: trino-server-co (1567) = 93.9% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[06:47:09] <icinga-wm>	 RECOVERY - check_memory on fransw2001 is OK: OK Memory 75% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[20:35:54] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 0725 Dec Q3 Planning: Create Recurring Upgrade and Recurring Downgrade activities when completed by DR through CiviCRM UI - https://phabricator.wikimedia.org/T409097#11551409 (10CHudson-WMF) Hi, @Lars -- from what I underst...