[00:44:54] <Josve05a>	 Krenair: Could you please help and restart a tool/service which is 503'ing? The operator has not been around for  a week https://tools.wmflabs.org/citations/ (context if needed https://en.wikipedia.org/wiki/User_talk:Citation_bot#Bot_down)
[00:46:21] <Krenair>	 sure
[00:49:13] <Krenair>	 !log tools.citations Started webservice that was not running per request from Josve05a and discussion at https://en.wikipedia.org/wiki/User_talk:Citation_bot#Bot_down
[00:49:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citations/SAL
[00:49:44] <Krenair>	 Josve05a, is that better?
[00:49:49] <Josve05a>	 Hmm...now the "landing page" is loaidng, but when using the service/tool, it just kills/blank screen
[00:49:55] <Josve05a>	 loading*
[00:50:07] <Josve05a>	 (such as https://tools.wmflabs.org/citations/process_page.php?edit=toolbar&slow=1&page=User%3AJosve05a%2Fsandbox)
[00:50:19] <Krenair>	 PHP Parse error:  syntax error, unexpected '', '' (T_CONSTANT_ENCAPSED_STRING), expecting ')' in /mnt/nfs/labstore-secondary-tools-project/citations/public_html/constants/capitalization.php on line 44
[00:50:20] <Krenair>	 nice
[00:50:28] <Josve05a>	 yay
[00:51:16] <Josve05a>	 thanks for your help though, I'm sure that's somehting which the maintainers can fix, since it is a code at GitHub which is erroring
[00:51:48] <Krenair>	 .... nice
[00:51:57] <Krenair>	 It seems the bot killed itself with a syntax error.
[00:52:15] <Krenair>	 commit c5d6c7b0090db7c3392d0249a90aa10fbb477651
[00:52:15] <Krenair>	 Author: tools.citations <tools.citations@tools-sgewebgrid-lighttpd-0914.tools.eqiad.wmflabs>
[00:52:15] <Krenair>	 Date:   Thu Dec 26 16:37:50 2019 +0000
[00:52:15] <Krenair>	     Automated file maintenance
[00:52:36] <Josve05a>	 Yeah, going to debug that now :) Thanks for restarting it, since i could not have figured that out 
[00:52:50] <Krenair>	 Josve05a, try now
[00:55:32] <Krenair>	 found another error
[00:55:35] <Krenair>	 Josve05a, try again
[00:55:55] <Josve05a>	 it's filled with bugs it seems, that not a good sing
[00:56:36] <Josve05a>	 Still getting the same thing...(I'm grabbig a cuppa coffee)
[00:56:37] <Krenair>	 and again
[00:57:26] <Krenair>	 okay you know what, I can run php -l on this to find syntax errors, I don't need to wait for user traffic :D
[00:59:24] <Krenair>	 hm but the error is less obvious now :(
[01:00:56] <Krenair>	 Josve05a, okay, what about now
[01:01:13] <Josve05a>	 it is taking me to the AUth-page (so promising....!)
[01:01:20] <Josve05a>	 YAY
[01:01:23] <Josve05a>	 Thank you! :D
[01:01:34] <Krenair>	 Josve05a, okay, so
[01:01:40] <Krenair>	 basically something in the tool automatically updates this file
[01:01:48] <Krenair>	 so it's possible it'll break itself again
[01:03:20] <Josve05a>	 Krenair: Did you edit the code to fix the errors? I think it is automatically fetching the code from GitHub https://github.com/ms609/citation-bot so it will most likely revert then
[01:03:55] <Krenair>	 I committed to the git repository in /data/project/citations/public_html
[01:04:27] <Krenair>	 suspect the next automated file maintenance commit will break stuff again
[01:23:38] <Krenair>	 Josve05a, I have no idea where the code is that does those automatic commits
[01:30:31] <Josve05a>	 Krenair: Well,  https://tools.wmflabs.org/citations/gitpull.php caused it to brake again (sorry)...As I've said previously, I'm not good with this. (At least you got the too working long enough for me to do what I needed to do with it :D )
[01:30:58] <Josve05a>	 https://www.irccloud.com/pastebin/76BrfyzH/
[01:31:56] <Krenair>	 well that was predictable I guess
[01:32:48] <Krenair>	 Josve05a, try not to do that
[01:32:53] <Krenair>	 it should work again
[01:33:37] <Krenair>	 I would like to look at the code that does the automatic update, but don't know where it is yet
[01:34:47] <Josve05a>	 the way to fix it is to simply copy the git pull to the source repo at github and merge that there instead, no
[01:35:49] <Krenair>	 it's not under the wikimedia org I won't be able to push to it
[01:37:12] <Josve05a>	 Well, The maintainer at the talk page (AManWithNoPan) is able to merge to that repo at GitHub, he just isn't  a operator of the tool "here", so a pul request and it would be "fixed" as soon as he was online tomorrow
[01:37:48] <Josve05a>	 (Guessing citation-bot/maintain_files.php https://github.com/ms609/citation-bot/blob/5e8b05c9c0b91dc6c7cd8343f0fd2b322f17a1b2/maintain_files.php  is doing some of the automated update)
[01:38:29] <Krenair>	 interesting
[01:39:00] <Krenair>	 the bit that does the committing etc. is  missing from the version in labs
[01:39:27] <Krenair>	 no wonder I couldn't find it
[01:39:50] <Krenair>	 oh  I see it no longer does this as  of https://github.com/ms609/citation-bot/pull/2354/files
[01:39:59] <Krenair>	 so it's that script generating the thing still but without committing
[01:41:19] <Josve05a>	 I see on that that Travis is doing the gitpull as part ofit's things
[01:41:33] <Josve05a>	 in travis.yml 
[01:47:48] <Krenair>	 Josve05a, where is that file?
[01:48:07] <Krenair>	 oh, .travis.yml
[01:48:55] <Josve05a>	 (this almost feel like a treasure hunt)
[01:50:23] <AntiComposite>	 THe build is failing on Travis as well
[01:50:31] <AntiComposite>	 https://travis-ci.org/ms609/citation-bot/jobs/631706743
[01:51:23] <Krenair>	 Josve05a, so I think that the problem with pushing my commit as a fix is really all I'm fixing is commits that do not exist on github
[01:51:39] <Krenair>	 they are only on the repo on NFS
[01:52:39] <Krenair>	 that said looking at `git diff 8d755d36a9e5e023c690c47be7bf10bd5422f00`
[01:53:19] <Krenair>	 I think it's fine to drop the local commits
[01:55:31] <Krenair>	 !log tools.citations Resetting repo in public_html from ef1ea17a4d1d2bc0adbcce6032a768f91b53ec40 to 8d755d36a9e5e023c690c47be7bf10bd5422f00 to attempt to drop broken automatic local commits to constants/capitalization.php - it looks like the automatic system is being removed in the upstream repo anyway
[01:55:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citations/SAL
[01:55:45] <Krenair>	 Josve05a, can you see if it's still working?
[01:56:12] <Josve05a>	 yeah it's working on my end at least Krenair
[01:56:17] <Krenair>	 alright
[01:58:07] <Krenair>	 Josve05a, try now?
[01:58:57] <Josve05a>	 Krenair: yup, still working
[01:59:00] <Krenair>	 ok
[01:59:09] <Krenair>	 so that's after running gitpull.php
[01:59:21] <Krenair>	 and it's all still going without any local hacks from me
[01:59:30] <Krenair>	 will update the user talk page
[01:59:50] <Josve05a>	 Thanks for your help and hard work!
[02:06:04] <andrewbogott>	 !log cyberbot moving cyberbot-db-01 to cloudvirt1024
[02:06:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cyberbot/SAL
[10:37:24] <arturo>	 !log etytree increase MEM quota T241716
[10:37:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etytree/SAL
[10:37:26] <stashbot>	 T241716: Request increased quota for etytreee Cloud VPS project - https://phabricator.wikimedia.org/T241716
[10:40:05] <dcausse>	 !log deployment-prep: created missing elastic indices: T241487
[10:40:06] <stashbot>	 dcausse: Unknown project "deployment-prep:"
[10:40:06] <stashbot>	 T241487: deployment-mediawiki-07: Search backend error during {queryType} search for '{query}' after {tookMs}: {error_message} - https://phabricator.wikimedia.org/T241487
[10:40:24] <dcausse>	 !log deployment-prep created missing elastic indices: T241487
[10:40:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[17:12:11] <AntiComposite>	 Ugh, tools-sgebastion-07 is lagging again
[17:13:58] <AntiComposite>	 (seriously people, `webservice --backend kubernetes <container> shell`, then run your stuff. Please.)
[17:14:34] * arturo wants to broadcast AntiComposite message
[17:26:18] <wyllie>	 \disconnect
[19:38:36] <Cyberpower678>	 andrewbogott: reporting that the DB VM is happy now.
[19:39:00] <Cyberpower678>	 Now about that quota ticket I opened on Phabricator a little bit ago. ;-)
[20:17:59] <andre__>	 Cyberpower678, hi! If you find some time, could you take a look at https://phabricator.wikimedia.org/T241152 ? Thanks in advance!
[20:18:19] <Cyberpower678>	 I saw it.  I just haven't had much opportunity to action it.
[20:18:57] <Cyberpower678>	 Holidays, RL stuff, college prep, and a big overhaul that I'm working on for IABot.
[20:22:01] <andre__>	 Ah, okay :) Thanks!
[20:22:46] * bd808 waves to andre__ 
[20:22:53] <andre__>	 morn bd808!
[20:24:33] <bd808>	 If I was in Hawaii, it actually would be morning. I should work on a plan for that.
[20:25:44] <hauskatze>	 bd808: Hawai'i has some active volcanos, better stay in ID :P
[20:27:53] <bd808>	 I'm well inside the blast zone for Yellowstone when it finally blows. I think I could take the risk of being nearer to Kīlauea  
[20:28:51] <bd808>	 And to my west is Mt Hood and St Helens which are also prime for a catastrophic blow. Stupid live planets
[20:29:51] <hauskatze>	 Geisers are kinda fun bd808 
[20:30:03] <hauskatze>	 free hot water
[20:30:07] <paladox>	 :O
[20:30:13] <bd808>	 hot mud gets boring pretty fast in my experience
[20:30:22] <hauskatze>	 that's true too
[20:30:37] <hauskatze>	 hot mud == called 'coffee' in some places :P
[20:30:47] <bd808>	 "Look! Its more bubbling smelly mud with some skeletons next to it!"
[21:04:33] <andre__>	 I parse "ID" as Indonesia. TLDs FTW.
[22:19:48] <bstorm_>	 !log toolsbeta Changed the ingress-admission ValidatingWebhookConfiguration to check extensions as well as networking API groups
[22:19:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[22:37:08] <bstorm_>	 !log toolsbeta Deleting the massive number of test ingresses for tool-fourohfour so the ingress controllers aren't moving so slowly.
[22:37:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[23:34:19] <mutante>	 !log cloudinfra cloud-puppetmaster-01 puppet cert clean puppetmaster-1001.devtools.eqiad.wmflabs
[23:34:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra/SAL
[23:35:10] <mutante>	 !log devtools attempting to create local puppetmaster (formerly puppetmaster::self)
[23:35:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:39:26] <mutante>	 paladox: symlink it?
[23:40:17] <paladox>	 !log devtools puppetmaster-1001: ln -s /var/lib/puppet/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem
[23:40:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:40:39] <mutante>	 sigh @ the issue. thanks for logging the fix!
[23:41:09] <paladox>	 andrewbogott it seems puppet master seems to be broken in the cloud (due to the path). Symnlinking /var/lib/puppet/server/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem to /var/lib/puppet/ssl/certs/puppetmaster-1001.devtools.eqiad.wmflabs.pem  fixes it.
[23:42:00] <mutante>	 note: "server" isn't running but the path is used in the prod .erb template
[23:42:13] <mutante>	 (We are trying to setup a local puppetmaster)
[23:42:22] <paladox>	 And also symnlinking /var/lib/puppet/server/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem to /var/lib/puppet/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem
[23:42:51] <paladox>	 !log devtools puppetmaster-1001: ln -s /var/lib/puppet/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem /var/lib/puppet/server/ssl/private_keys/puppetmaster-1001.devtools.eqiad.wmflabs.pem
[23:42:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:52:51] <bd808>	 paladox: that sounds like some weird regression? I know buster puppetmasters were working 2 months ago.
[23:53:08] <paladox>	 bd808 i believe i did the same hack in 2017
[23:53:12] <paladox>	 it seems
[23:53:49] <paladox>	 i just forgot about it until now (since it was working).
[23:53:54] <bd808>	 that sounds like maybe your hiera settings are not quite right then? Is that symlink needed to make the puppetmaster a client of itself?
[23:54:10] <paladox>	 Nope, it's needed for apache to start.
[23:54:53] <bd808>	 !log tools Rebuiliding all buster-ssd Docker images to pick up busybox
[23:54:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:56:53] <bd808>	 paladox: hmmm... I have built a lot of puppetmasters and the steps in https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_1:_Setup_a_standalone_puppetmaster are all I've needed (except when working on support for a new distro)
[23:57:04] <paladox>	 oh
[23:57:28] <paladox>	 oh! that's where the doc was :P
[23:57:46] <paladox>	 i tried searching "puppetmaster" but that didn't bring up what i wanted.
[23:58:14] <mutante>	 ah !:)
[23:58:35] <bd808>	 yeah, I always just go to [[wikitech:Puppet]] and then get the link from the See Also
[23:58:47] <mutante>	 thanks bd808 we were looking for just that