[00:18:16] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed bc1d49c202 (better CSRF error handling, T242573)
[00:18:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[00:21:35] <chicocvenancio>	 ottomata: I'm currently maintaining PAWS, it wasn't clear from backscroll if there was any doubt still
[01:39:06] <ottomata>	 thanks!
[11:19:37] <Urbanecm>	 Hi, would it be possible to send a PY 2.7 deprecation email in PWB to cloud-announce? A lot of PWB users use it from Toolforge, and I'm thinking about best way to reach them. 
[11:20:19] <Urbanecm>	 I have an email drafted, just thinking about targets
[12:34:13] <arturo>	 yup, probably sync with bryan/brooke about that Urbanecm 
[12:34:44] <arturo>	 and anyway cloud@ should be open for you to start a thread
[13:33:38] <Urbanecm>	 Sure, but I feel this is more an announcement
[16:59:33] <bd808>	 !log admin Changed the config for cloud-announce mailing list so that lsit admins do not get bounce unsubscribe notices
[16:59:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[17:23:22] <jeh>	 !log cloudstore restart cloudstore-puppetmaster01 lots of OOM events and "Cannot allocate memory" errors T242893
[17:23:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudstore/SAL
[17:23:26] <stashbot>	 T242893: puppetmaster broken in the cloudstore project - https://phabricator.wikimedia.org/T242893
[17:26:01] <Reedy>	 Cyberpower678: 
[17:26:07] <Reedy>	 [17:24:13] <Cymbopetalum> Anyone know who maintains reFill now?  https://en.wikipedia.org/wiki/User_talk:Zhaofeng_Li#Blue_whale_reFill_errors
[17:26:31] <Cyberpower678>	 Yes?
[17:26:51] <Reedy>	 refill seems borked
[17:26:56] <Cyberpower678>	 reFill is going to be merged with IABot
[17:27:09] <Cyberpower678>	 But that still is a ways away
[17:27:09] <Reedy>	 That's great
[17:27:17] <Reedy>	 But that doesn't help people now ;)
[17:27:30] * Cyberpower678 is getting ready for classes now.
[17:29:08] <Cymbopetalum>	 I've never seen that error before using the refill tool.
[17:29:26] <Cymbopetalum>	 "Submitting your task... Internal Server Error "
[17:30:31] <Reedy>	 Possibly broken yesterday in migration stuff? Service(s) just need a restart?
[17:31:26] <Cymbopetalum>	 Flickr2Commons was also having issues yesterday but got fixed today, so I think something happened on WMFLabs
[17:33:09] <Cyberpower678>	 I can't check on it right now.
[17:33:13] <Cyberpower678>	 College
[17:34:23] <Cymbopetalum>	 it's cool.  Long as it's on your radar
[18:38:11] <hauskatze>	 bd808: re T242898 - did updating the ssh key helped?
[18:38:11] <stashbot>	 T242898: MIrroring Diffusion repositories to GitHub seems to be broken - https://phabricator.wikimedia.org/T242898
[18:56:24] <bd808>	 hauskatze: no, or I would not have filed the bug :)
[18:56:55] <hauskatze>	 bd808: can you try granting access to the github account 'wmfphab' so I can test?
[18:57:06] <hauskatze>	 it's an account from wmf-operations
[18:58:04] <bd808>	 hauskatze: I invited that github account to the project years ago and it has never accepted. I don't know how to actually get it connected.
[18:58:25] <hauskatze>	 I think it's twentyafterfour's
[18:58:51] <bd808>	 GitHub makes this a 2 part process so that it can't be abused by force joining someone to a repo and spamming them with abuse.
[18:59:12] <hauskatze>	 so if the ssh key was updated both on github and on passphrase...
[18:59:50] <bd808>	 I do not see any sign that Phab is trying to mirror. I don't know what magic config might gate that
[19:00:06] <hauskatze>	 maybe they put a restriction so only wmfphab can mirror stuff?
[19:00:24] <bd808>	 *shrug* That's why I opened the bug report :)
[19:00:31] * bd808 needs to eat some food
[19:19:14] <hauskatze>	 I'm browsing puppet conf to see if there's any limits in place but found none so far
[19:19:23] <hauskatze>	 but 20after4 has been pinged already
[19:19:30] <hauskatze>	 he will surely know
[19:36:57] <mutante>	 !log git shutting down and deleting instance 'gerrit-test' (T236569)
[19:37:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[19:37:00] <stashbot>	 T236569: "git" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236569
[19:38:04] <mutante>	 !log git shutting down and deleting instance 'jenkins-slave-02' (T236569)
[19:38:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[20:30:07] <halfak>	 Hey folks.  Did the CloudVPS statsd server go away? 
[20:30:32] <halfak>	 Looks like I was trying to connect to labmon1001.eqiad.wmnet
[20:42:11] <bd808>	 halfak: it was renamed. T224585 T241284 T241285
[20:42:12] <stashbot>	 T241284: Service name needed for things wanting to write data to WMCS's statsd service - https://phabricator.wikimedia.org/T241284
[20:42:12] <stashbot>	 T224585: Migrate labmon* to Buster - https://phabricator.wikimedia.org/T224585
[20:42:12] <stashbot>	 T241285: Deployment-prep should host its own statsd/graphite server - https://phabricator.wikimedia.org/T241285
[20:42:42] <bd808>	 halfak: for now, you can change to using cloudmetrics1001.eqiad.wmnet
[20:43:42] <bd808>	 It turns out that nobody on the WMCS team knew that multiple projects were treating that statsd/graphite endpoint as a "service"
[20:56:54] <mutante>	 !log git deleting instance gerrit-mysql (used be icinga2 for ores, but ores has prod icinga checks at least) T236569
[20:56:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[20:56:56] <stashbot>	 T236569: "git" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236569
[20:57:14] <halfak>	 thanks bd808!
[21:11:12] <bd808>	 Urbanecm: nice announce email on the pywikibot py2 deprecation. I may steal some of that when I finally get around to setting an EOL for py2 on the Toolforge Kubernetes cluster
[21:12:14] <hauskatze>	 that reminds me that I have yet to find some (God give me strenght) time to migrate stewardbots
[23:42:14] <mutante>	 !log devtools deploy-1002 mkdir /srv/deployments/.git ; chown trebuchet:wikidev .git ; manually run "scap deploy --init" as trebuchet user in an attempt to fix initial puppet run on deployment_server
[23:42:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:44:43] <mutante>	 !log devtools deploy-1002 sudo git init in /srv/deployment ; scap deploy --init (now fails with 'fatal: Not a valid object name HEAD')
[23:44:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:51:09] <paladox>	 !log devtools deploy-1002 rm -rf /srv/deployment
[23:51:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL