[00:18:05] <MacFan4000>	 !log tools.zppixbot-test restart for new module/config changes and created MySQL database for quotes
[00:18:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[00:20:57] <MacFan4000>	 !log tools.zppixbot-test restarting again
[00:20:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[00:37:21] <MacFan4000>	 !log tools.zppixbot-test restarting for config updates
[00:37:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[07:53:22] <wm-bot>	 !log tools.majavah-bot <taavi> set task 4 for sqwiki and sqquote as quiet in crontab
[07:53:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL
[10:00:13] <wm-bot>	 !log wikispeech <lokal-profil> New Speechoid services deployed on https://wikispeech-tts-dev.wmflabs.org, built on Blubber
[10:00:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL
[10:29:42] <legoktm>	 !log tools.newusers switch to Rust-powered version
[10:29:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.newusers/SAL
[10:30:02] <legoktm>	 $ curl -I "https://newusers.toolforge.org/" | grep rust
[10:30:02] <legoktm>	 x-rust: newusers/0.1.1
[10:40:10] <arturo>	 !log tools created VM tools-legacy-redirector, with the corresponding puppet prefix (T247236, T234617)
[10:40:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:40:14] <stashbot>	 T247236: Toolforge: introduce a system to preserve old tools.wmflabs.org URLs - https://phabricator.wikimedia.org/T247236
[10:40:14] <stashbot>	 T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617
[12:28:51] <nokib>	 I cannot edit my custom json pages frmo pywikibot
[12:29:13] <nokib>	 OthersPageError
[12:29:18] <nokib>	 raises
[12:32:51] <nokib>	 !help
[12:32:51] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[12:33:19] <arturo>	 nokib: I suggest you send an email to the cloud@ mailing list explaining your case and requesting help
[12:37:05] <nokib>	 My user-config.py raises error that it doesn't find '*' 
[12:37:33] <nokib>	 I want to use my bot username in all the wiki
[12:37:43] <nokib>	 What the syntax would be?
[12:38:05] <nokib>	 username["wikipedia"]["*"]?
[12:39:00] <Reedy>	 https://www.mediawiki.org/wiki/Manual:Pywikibot/login.py#Login_as_a_bot
[12:39:25] <Reedy>	 https://www.mediawiki.org/wiki/Manual:Pywikibot/user-config.py
[12:39:32] <arturo>	 thanks Reedy !!
[12:49:11] <nokib>	 Thanks Reedy
[12:49:27] <nokib>	 Though didn't work
[12:50:43] <Reedy>	 What didn't work?
[12:51:00] <Reedy>	 You're asking questions about pywikibot rather than cloud specific questions
[12:51:08] <Reedy>	 You'd probably be better asking in #pywikibot
[12:51:55] <nokib>	 Ok thanks for reference
[13:18:35] <MacFan4000>	 !log tools.zppixbot restarting for new module/code changes and created mysql database for quotes
[13:18:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[14:16:04] <wm-bot>	 !log tools.zppixbot <zppixbot> auto-update@website: Synced website repo in 45.s
[14:16:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[14:23:21] <CurbSafeCharmer_>	 @bstorm_: reFill has stopped working again. Please could you take a look?
[14:31:17] <wm-bot>	 !log tools.zppixbot <zppixbot> auto-update@website: Synced website repo in 50.s
[14:31:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[14:35:04] <Naypta>	 Hey folks, I'm trying to debug SMTP sending through mail.tools.wmflabs.org - is there somewhere I can see logs for mail sent from my tool account? Thanks!
[14:40:59] <arturo>	 Naypta: no, please open a phab task describing your problem
[14:41:42] <Naypta>	 arturo: no worries, cheers :) I think it's probably an issue with my software, so I'll hold off on a phab task for now
[14:42:33] <arturo>	 we could start a live debugging session, but I dont have time now
[14:43:03] <Naypta>	 nah, don't sweat it - I'll just drop some breakpoints into the SMTP library I'm using and try and pull out the raw server responses
[14:43:04] <Naypta>	 cheers :)
[14:43:23] <arturo>	 ok :-)
[14:59:34] <bd808>	 Naypta: depending on where you are sending to, you may run into issues related to SPF and DKIM. We are slowly working on that. See T249237
[14:59:34] <stashbot>	 T249237: Fix Cloud VPS and Toolforge mail servers to work with the modern internet - https://phabricator.wikimedia.org/T249237
[15:23:20] <bstorm_>	 !log tools.refill-api restarting the pods in case it needs it
[15:23:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL
[15:26:10] <Naypta>	 bd808: turns out the issue wasn't actually that, but that the mailserver treats tool email addresses as case sensitive, it seems - I know that's technically RfC compliant but it's not what I expected :p
[15:26:31] <Naypta>	 do you know if this was done deliberately? if it was, I'll add it to the wikitech documentation; if not, I'll raise a phab ticket
[15:28:44] <bd808>	 Naypta: I'm not exactly sure, but my guess is that the behavior is whatever the exim mail server does. You could open a task to poke us to look into it and see if that guess is true or if there is some regex or something that was just not done well.
[15:30:05] <Naypta>	 cheers, will do! am reading that exiqgrep defaults to case sensitivity, so it may be connected to that, but idk what's being used internally to process the mail out of the exim queue so that might be totally unrelated
[15:45:13] <CurbSafeCharmer_>	 @bstorm_: that fixed it, thanks
[15:47:41] <bstorm_>	 👍🏻
[16:01:03] <CurbSafeCharmer_>	 @bstorm_: there's some discussion about the recurring issue with Refill at the Village Pump (technical) - do you want me to say what needed fixing or would you like to chip in with a bit more narrative?
[16:12:20] <bstorm_>	 I honestly don't know this time. I just restarted it during the meeting. What I suspect it would benefit from is a liveness probe.
[16:12:47] <bstorm_>	 I'm considering adding a default liveness probe to all webservices that will restart things that stop responding well, but that's something to be careful with :) 
[16:13:48] <bstorm_>	 That particular service seems to have been hitting its quota at one point
[16:14:29] <bstorm_>	 hmm, no that was just during the restart
[16:16:04] <bstorm_>	 In the past it tended to use up ram and then crash
[17:02:06] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 25325355b20a2e2453238d78ddea8d0cd18c8774 Update zppixbot project name
[17:02:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[18:02:55] <MacFan4000>	 !log tools.zppixbot removing reminder that I just added and restarting
[18:02:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[19:31:05] <andre__>	 Does anyone remember if we intentionally shut down phab.wmflabs.org at some point (e.g. due to new Debian versions or such)?
[19:47:54] <bd808>	 andre__: https://openstack-browser.toolforge.org/project/devtools is still claiming the proxy. Maybe paladox or mutante know what's up the the backend service?
[19:48:53] <andre__>	 ah thanks
[19:49:11] * andre__ bookmarks
[19:50:51] <paladox>	 hi
[19:51:04] <paladox>	 It was up and running when we switched backends.
[19:55:29] <paladox>	 fixed
[19:55:32] <paladox>	 was firewall
[19:55:52] <paladox>	 !log devtools ran `iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT` on phabricator-prod-1001
[19:55:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[19:56:34] <bd808>	 paladox: missing a ferm rule?
[19:57:14] <andre__>	 paladox, uh, thanks a lot, that was quick! :) As I was thinking of using that instance for a workshop...
[19:57:28] <paladox>	 bd808 i think so.
[19:57:51] <paladox>	 ferm was on "active existed" so i stopped it. That didn't fix it, so i tried the iptables approach.
[20:00:20] <paladox>	 i see the ferm rule &R_SERVICE(tcp, 80, $DEPLOYMENT_HOSTS);
[20:10:31] <mutante>	 paladox: port 80 ist not needed for it anymore in prod
[20:10:55] <paladox>	 mutante should we change https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/phabricator/main.pp#L113 to LABS_NETWORKS?
[20:10:56] <mutante>	 also... i was about to say "but this used to work" 
[20:12:18] <mutante>	 paladox: the other day we found that "cache_hosts" just works
[20:12:26] <mutante>	 for the proxy
[20:12:31] <paladox>	 oh
[20:12:51] <mutante>	 does it have to be port 80 at all, though?
[20:13:01] <mutante>	 i guess so
[20:13:14] <mutante>	 it's different in cloud from prod because of the proxy /caching layer
[20:13:35] <mutante>	 we only speak https even behind caching there
[20:13:53] <mutante>	 TLS termination is at envoy on the phab host itself
[20:14:23] <paladox>	 mutante https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/606255/
[20:16:40] <mutante>	 paladox: unfortunately no, we don't want to remove that hole in prod
[20:16:57] <mutante>	 it's there to be able to use httpbb.. though maybe nobody does that
[20:17:03] <paladox>	 ok
[20:18:34] <paladox>	 mutante fixed
[20:18:36] <mutante>	 the best fix would be to be able to use 443 like in prod, but that's unlikely to happen i think
[20:18:58] <bd808>	 mutante: the front proxy can be pointed to any port on the instance, so if there is normally a listener on another port that can be used. The front proxy doesn't expect to speak TLS to the backend though.
[20:19:00] <mutante>	 also.. how did it work before?
[20:19:35] <mutante>	 bd808: yea, it would have to speak TLS to the backend on 443
[20:19:47] <mutante>	 then it would actually be like prod with envoy behind ATS
[20:20:53] <mutante>	 second best we can do is allow port 80 from the proxies
[20:21:00] <mutante>	 but that needs a $realm check
[20:21:17] <bd808>	 or hiera something
[20:21:38] <bd808>	 I know y'all are allergic to $::realm conditionals :)
[20:21:46] <mutante>	 that would work if it was the same port but just a different srange
[20:22:07] <mutante>	 but it's not port 80 at all in prod
[20:22:27] <mutante>	 besides the extra rule for testing from deployment_servers
[20:22:32] <bd808>	 right, but is a port and that could be made hiera configurable
[20:23:01] <bd808>	 the app runs on port 80 in prod, it is just not exposed
[20:23:27] <mutante>	 yea, the port could be added to Hiera, that's right
[20:23:49] <mutante>	 latest version from paladox would also work.. kind of "by chance" but it would
[20:29:37] <mutante>	 trying to rebase and amend.. not sure if the rebasing issue is due to zuul changes or not
[20:40:28] <mutante>	 yea.. so let's do it the right way and add that port in hiera... and keep the existing rule for deployment_servers as it is in both prod and cloud
[20:40:35] <mutante>	 will look at that
[20:47:08] <mutante>	 the ferm::service we care about is inside the profile:tlsproxy::envoy, not phabricator itself
[21:02:32] <mutante>	 paladox: bd808: yep, so i'd say this way https://gerrit.wikimedia.org/r/c/operations/puppet/+/606255
[21:03:40] <bd808>	 mutante: yeah, that looks like a pretty clean way to do it :)
[21:04:24] <paladox>	 ^
[21:10:05] <mutante>	 cool! compiled on prod and cloud instance and merged
[21:10:12] <mutante>	 checking prod
[21:11:33] <mutante>	 paladox: yep, noop in prod. let's remove the manual rule in cloud and run puppet
[21:12:42] <paladox>	 great!
[21:12:59] <paladox>	 Error: Systemd start for ferm failed!
[21:13:11] <paladox>	 oh
[21:13:37] <paladox>	 Jun 17 21:13:20 phabricator-prod-1001 ferm[4362]: & R_SERVICE
[21:13:37] <paladox>	 Jun 17 21:13:20 phabricator-prod-1001 ferm[4362]: (
[21:13:37] <paladox>	 Jun 17 21:13:20 phabricator-prod-1001 ferm[4362]:     tcp , 80 , $ DEPLOYMENT_HOSTS $ <--
[21:13:37] <paladox>	 Jun 17 21:13:20 phabricator-prod-1001 ferm[4362]: "," expected
[21:13:38] <paladox>	 Jun 17 21:13:20 phabricator-prod-1001 ferm[4362]:  failed!
[21:13:40] <paladox>	 mutante ^
[21:14:37] <mutante>	 paladox: we should check inside /etc/ferm/conf.d/10_phabmain_http
[21:14:37] <paladox>	 and cache_hosts doesn't exist
[21:14:47] <mutante>	 but i compiled it.. grrr
[21:14:47] <paladox>	 changing that to CACHES would
[21:14:56] <paladox>	 and also putting brackets around it
[21:15:31] <mutante>	 paladox: yea, confirmed. it is just $CACHES
[21:16:22] <paladox>	 mutante https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/606267/
[21:16:57] <mutante>	 yep:) and i like how now it does not touch prod at all
[21:17:07] <mutante>	 merging
[21:17:49] <paladox>	 :)
[21:51:27] <bstorm_>	 !log paws upgraded chart in the new cluster to include resource limits
[21:51:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[21:51:48] <bstorm_>	 !log paws upgraded chart in the new cluster to include resource limits T251298
[21:51:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[21:51:50] <stashbot>	 T251298: Design the resource limits, RBAC and PSP needed for the PAWS Kubernetes cluster - https://phabricator.wikimedia.org/T251298
[23:21:17] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 9b7349c602 (update a Bengali template)
[23:21:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL