[01:58:08] <Sveta>	 bd808: it gives 503, then i need to restart
[13:43:26] <milimetric>	 hey can someone add me to the meetbot group? https://wikitech.wikimedia.org/wiki/Tool:Meetbot
[13:44:09] <milimetric>	 everyone keeps making Tim reboot the thing and I wanna help out if I'm around
[13:44:25] <Adithya>	 Is anybody here?
[13:46:11] <Krenair>	 hi Adithya 
[13:46:37] <Adithya>	 I need a help
[13:46:55] <Adithya>	 I am currently working on a project that is present in incubator
[13:47:40] <Adithya>	 I am facing a script error in a module in Khowar Wikipedia
[13:48:16] <Krenair>	 ok
[13:48:25] <Krenair>	 this is an lua module?
[13:48:34] <Adithya>	 I have asked the question in the community portal of Incubator (https://incubator.wikimedia.org/wiki/Incubator:Community_Portal#Script_error)
[13:50:32] <Adithya>	 It's actually a template named Module rating that causes the issue
[13:50:41] <Adithya>	 There is no module currently present for that
[13:50:55] <Krenair>	 unfortunately I don't really know much lua
[13:50:56] <Adithya>	 I have detailed my problem in the link provided above
[13:51:46] <Adithya>	 My actual question is: Where can I post my doubt if I didn't get a help from the community portal of Incubator for a project related to Incubator?
[13:52:21] <Adithya>	 I have also asked the same in Village Pump (Technical) of English wikipedia where also I didn't get a reply
[13:53:41] <Krenair>	 the error seems to be coming from within {{wp/khw/module rating|protected}}
[13:54:40] <Krenair>	 but since the page is not actually protected it should be okay to remove that for now?
[13:56:35] <Adithya>	 https://incubator.wikimedia.org/wiki/Module:Wp/mni/Citation/CS1/doc
[13:56:43] <Adithya>	 Please check that above link
[13:57:12] <Adithya>	 It is of another test wikipedia where a similar error has occurred
[13:57:43] <Krenair>	 I don't think this is similar
[13:57:51] <Adithya>	 The above link also uses the same template Module rating
[13:58:55] <Adithya>	 But https://incubator.wikimedia.org/wiki/Module:Wp/khw/Citation/CS1/doc also contains the same script erro
[13:58:57] <Adithya>	 *error
[13:59:36] <Krenair>	 we don't know what the error is yet
[13:59:44] <Krenair>	 I'm not sure I'm the best person to help with thihs
[13:59:46] <Krenair>	 this
[14:00:05] <Adithya>	 Ok
[14:00:13] <Krenair>	 you might try #wikimedia-tech or https://meta.wikimedia.org/wiki/Tech
[14:00:59] <Adithya>	 Thank you. Let me post it there
[14:04:56] <Krenair>	 :/
[15:27:01] <paladox>	 !log upgrading gerrit-test3 to debian 9 (stretch)
[15:27:02] <stashbot>	 paladox: Unknown project "upgrading"
[15:27:06] <paladox>	 !log git upgrading gerrit-test3 to debian 9 (stretch)
[15:27:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[16:26:51] <paladox>	 !log git rebuilding gerrit-test3 as gerrit-test5 due to removal of a kernel during a upgrade to stretch.
[16:26:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL
[16:56:26] <pintoch>	 noob question: is it possible to run mysqldump in a kubernetes pod? it does not seem to be available in the python images at least.
[16:57:03] <bd808>	 !lot tools.meetbot Added milimetric as a co-maintainer
[16:57:23] <bd808>	 !log tools.meetbot Added milimetric as a co-maintainer
[16:57:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meetbot/SAL
[16:57:39] <pintoch>	 in other words: is there a preferred way to backup large tool databases? I assume running mysqldump on bastion is not desirable as it is IO intensive.
[16:58:03] <pintoch>	 (I am happy to add the answer to the wiki, as this is not mentioned there yet)
[16:58:11] <bd808>	 pintoch: I don't think we install any of the mysql shell clients in the Kubernetes containers. You can pretty easily run a mysqldump job on the job grid
[16:58:25] <pintoch>	 ah, the SGE grid?
[16:58:38] * pintoch forgot about that one
[16:59:38] <bd808>	 Yes. You'll need to write a little shell script to do it so that you can redirect output to a file
[16:59:59] <pintoch>	 makes sense, thanks
[17:00:04] <pintoch>	 adding that to the wiki
[17:00:22] <bd808>	 but that script could be about as simple as `exec mysqldump ... > my_file.sql`
[17:01:43] <pintoch>	 yeah, with a "| gzip" in my case, because it's quite big
[17:40:31] <tgr>	 what do striker error ids refer to?
[17:40:37] <tgr>	 not logstash, apparently
[17:43:09] <milimetric>	 thanks vm bd808
[17:50:25] <bd808>	 tgr: they should be in logstash, but... we have a log shipping problem where the json blob gets truncated at 2048 bytes. So data gets to the ELK stack but in partial form that prevents desired indexing
[17:50:47] * bd808 has not had time to figure out where the truncation happens
[17:51:42] <tgr>	 bd808: do you remember the type/channel name?
[17:52:07] <bd808>	 uwsgi-striker I think ...
[17:54:59] <bd808>	 tgr: https://logstash.wikimedia.org/goto/c39c354d6fd5043383517aceef081638
[17:55:48] <bd808>	 I see some errors there about ssh key manipulation actions failing
[17:56:32] <bd808>	 probably a bug from the Django 2 upgrade that I missed
[18:03:26] <tgr>	 hm, I guess I'm confusing toolsadmin with tools.wmflabs.org/admin which is not striker
[18:09:04] <paladox>	 Does anyone know what hiera value i need to set so that ldapauth wiki will work?
[18:09:10] <paladox>	 (in mwvagrant)
[18:38:59] <bd808>	 paladox: `vagrant roles enable ldapauth; vagrant provision`
[18:39:20] <paladox>	 Yup, though https://ldapauth-gitldap.wmflabs.org/wiki/Special:Version appears to be using the default configured wiki
[18:40:05] <bd808>	 paladox: that's probably related to https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS#Run_a_wikifarm which is maybe what you were asking about?
[18:40:17] <paladox>	 ah
[18:40:20] <paladox>	 yup
[18:41:32] <paladox>	 i've setup local.yaml so it's like https://github.com/wikimedia/mediawiki-vagrant/blob/57ef729a34620611b170d88d56afc337f0813290/puppet/hieradata/environment/labs.yaml#L4
[18:42:04] <paladox>	 which correctly sets ldapauth-gitldap.wmflabs.org in the php files.
[18:43:00] <paladox>	 I've also configured the proxy
[18:43:06] <paladox>	 so that it does 8085:8080
[18:43:50] <bd808>	 paladox: hmm... something is not working right. You can see the wiki name in the footer: "About devwiki". That tells me that multiversion (the php code that picks the db and config based on vhost name) is not working
[18:44:01] <paladox>	 oh
[18:45:29] <bd808>	 I am running multiple deploys of MediaWiki-Vagrant using that role, so I'm pretty confident that the puppet and multiversion code works generally.
[18:46:19] <paladox>	 yeh it was working before, but i had to rebuild the instance due to me removing the kernel by mistake.
[18:46:38] <paladox>	 (So it's fresh but i have forgotten how i did it so that it would work with https://ldapauth-gitldap.wmflabs.org/wiki/Special:Version )
[18:47:16] <bd808>	 YOu should be able to see the expected hostname in settings.d/wikis/ldapauthwiki/wgConf.php
[18:47:53] <paladox>	 oh
[18:47:55] <paladox>	 i see what it did
[18:48:50] <paladox>	 thank you bd808!
[18:48:52] <paladox>	 works now!
[18:49:00] <bd808>	 cool :)
[18:52:26] <andrewbogott>	 !role wikistats deleting some project- and prefix-wide puppet config to try to get puppet running again on VMs
[19:05:52] <bd808>	 !log wikistats deleting some project- and prefix-wide puppet config to try to get puppet running again on VMs (re-logged for andrewbogott)
[19:05:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL
[19:06:52] <andrewbogott>	 bd808: thanks
[19:15:59] <paladox>	 bd808 hmm, changing the password for the admin user is failing with "authmanager-authplugin-setpass-failed-title", "authmanager-authplugin-setpass-failed-message"
[19:22:36] <paladox>	 hmm "[091d1879a597772ad1442c4e] [no req]   PasswordError from line 62 of /vagrant/mediawiki/maintenance/changePassword.php: Passwords must be at least {{PLURAL:10|1 character|10 characters}}."
[19:27:56] <bd808>	 paladox: yeah... there is an override in LocalSettings.php that lets the default "Admin" user have a shorter password (search for isValidPassword to see it). Otherwise MediaWiki has changed its default for password strength
[19:28:07] <paladox>	 ah ok
[19:28:09] <paladox>	 thanks
[19:28:25] <paladox>	 though using the shorter pass dosen't seem to be taking affect
[19:28:34] <bd808>	 I proposed a patch at one point to MediaWiki-Vagrant's config to disable all the password strength checks but others talked me out of mergin it :)
[19:28:49] <paladox>	 (even though the cli says it was successfully changed)
[19:28:56] <paladox>	 heh
[19:30:03] <bd808>	 There may still be some bugs in changing LDAP passwords using createAndPromote.php. I hacked in some fixes for that a few months ago, but they might have become broken again
[19:30:36] <bd808>	 The LdapAuthentication plugin is basically in life-support only maintenance mode these days
[19:30:42] <Wurgl>	 Seems there is a problem opening the database, I am running into a timeout? mysql --defaults-file=$HOME/replica.my.cnf --host=tools-db --database=s51512__data
[19:32:04] <Wurgl>	 Same behaviour with database s51412__data
[19:32:08] <bd808>	 Wurgl: where are you trying that from? I'm not having problems connecting
[19:32:22] <Wurgl>	 tools-sgebastion-07
[19:32:30] <paladox>	 I guess it's using the password from openldap?
[19:32:51] <Wurgl>	 My scripts seem to have the same problem, and they are running inside the magic cloud
[19:33:21] <Wurgl>	 since about 4 hours
[19:33:46] <bd808>	 Wurgl: waht's your tool name? I can connect easily with `sql toolsdb` as multiple users. I'm wondering if the user you are using has too many connections open or something.
[19:34:14] <Wurgl>	 persondata is s51412__data and wikihistory is s51512__data
[19:34:30] <paladox>	 bd808 is there a way to disable the admin user for now?
[19:34:52] <Wurgl>	 Problem startet at ~ 15:14 UTC
[19:35:32] <bd808>	 Wurgl: hmm.. `sql toolsdb` as persondata works fine too
[19:38:02] <bd808>	 Wurgl: do you have some error logs for this I can look at?
[19:38:30] <Wurgl>	 Aha! tools-db has now 10.64.37.9 whereas tools.db.svc.eqiad.wmflabs was address 172.16.7.153
[19:39:00] <MaxSem>	 Tools main page is down for me
[19:39:06] <andrewbogott>	 tarrow: it looks to me like elasticsearch-01, elasticsearch-20, elasticsearch-21.wikifactmine.eqiad.wmflabs are crashed and unreachable… has that been true for a while?  Should I try to reboot them?
[19:39:16] <bd808>	 MaxSem: yeah, looking at that
[19:39:36] <bd808>	 !log tools.admin Restarting webservice, seems to have lost connectivity to the proxy
[19:39:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL
[19:39:51] <bd808>	 MaxSem: better now I think
[19:40:15] <ebernhardson>	 andrewbogott: iirc those have been broken for some time, i think i remember poking them before we switched puppet
[19:40:23] <MaxSem>	 Weee
[19:40:29] <bd808>	 Wurgl: Ah ha! Let me see if I can find the alias that needs updating
[19:40:49] <andrewbogott>	 ebernhardson: what do you mean by 'switched puppet'?
[19:41:01] <ebernhardson>	 andrewbogott: we removed elasticsearch 2.x support
[19:41:13] <ebernhardson>	 andrewbogott: we currently support 5.x and 6.x in puppet
[19:42:20] <andrewbogott>	 ebernhardson: so what's the plan for getting those back online?
[19:42:33] <andrewbogott>	 They're not just failing puppet runs, they're literally unreachable.  Can't even get a local console.
[19:42:48] <ebernhardson>	 andrewbogott: not sure, in https://phabricator.wikimedia.org/T198690 tarrow said he unapplied the role so they would basically stop updating anything elasticsearch related
[19:42:49] <bd808>	 Wurgl: I found the config that we missed updating when that database moved. I bet the recent breakage for you happened as a result of us changing the dns servers
[19:46:47] <Wurgl>	 Problem fixed, so everything is fine
[19:56:02] <lucaswerkmeister>	 !log tools.quickcategories deployed 1d4e4fb16b
[19:56:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL
[20:10:08] <andrewbogott>	 Hydriz: dumps-1.dumps is having various troubles… I can't get dpkg to work there, which is blocking puppet and other things.  Think it can be replaced, or are you interested in trying to fix?
[20:13:47] <andrewbogott>	 !log dumps rebooting dumps-1 to try to workaround nfs issues
[20:13:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Dumps/SAL
[20:18:22] <bd808>	 !log redirects Manually fixed broken /etc/puppet/puppet.conf on redirects-nginx01
[20:18:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Redirects/SAL
[20:28:32] <andrewbogott>	 !log mcr-dev deleting old log files on mcr-full because the drive is full
[20:28:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mcr-dev/SAL
[20:30:14] <andrewbogott>	 paladox: puppet is broken on at least three VMs in the 'phabricator' project.  those are yours, right?
[20:30:22] <paladox>	 Oh yup
[20:30:24] <paladox>	 which one?
[20:31:14] <andrewbogott>	 phab-tin, phabricator
[20:31:24] <andrewbogott>	 I guess I only see those two for now
[20:31:38] <paladox>	 andrewbogott puppet works
[20:31:43] <paladox>	 on phab-tin "The last Puppet run was at Thu Apr 18 20:08:02 UTC 2019 (23 minutes ago). "
[20:31:54] <andrewbogott>	 is syncing broken on your puppetmaster then?
[20:32:00] <andrewbogott>	 I patch I merged this morning still isn't applied there
[20:32:01] <paladox>	 likley
[20:33:04] <paladox>	 im not sure why syncing is broken
[20:33:20] <paladox>	 but git fetch origin && git rebase origin has updated the puppet repo
[20:33:57] <andrewbogott>	 ok, so things should catch up eventually
[20:34:09] <andrewbogott>	 mutante, are you the defacto owner of the 'planet' project?
[20:36:28] <mutante>	 andrewbogott: yep
[20:36:46] <mutante>	 is there a puppet problem?
[20:36:49] <andrewbogott>	 mutante: the puppetmaster is shut down but there's a running VM trying to use that puppetmaster
[20:36:59] <mutante>	 andrewbogott: ok, looking
[20:37:03] <andrewbogott>	 thanks
[20:37:48] * andrewbogott has cut the broken puppet list from about 120 to about 60.  60 still feels like a lot :(
[20:43:16] <andrewbogott>	 mutante: it's also worth mentioning that VMs left in a 'shutdown' state will probably rot and stop working
[20:43:22] <andrewbogott>	 since they don't get basic infra updates
[20:49:32] <andrewbogott>	 !log recommendation-api deleting vagrant logfiles on rec-wiki because the drive is 100% full
[20:49:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Recommendation-api/SAL
[20:52:59] <andrewbogott>	 chasemp: can I enable puppet on gerrit-sizzle?
[20:55:03] <mutante>	 !log planet - coverting instance pk8s back to labs-puppetmaster, fixed puppet runs
[20:55:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL
[20:55:12] <andrewbogott>	 thanks mutante 
[20:57:04] <mutante>	 !log planet - deleting instance planet-hotdog , deprecated
[20:57:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL
[20:57:35] <mutante>	 !log planet - booting local puppetmaster that was shutdown
[20:57:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Planet/SAL
[21:00:01] <mutante>	 andrewbogott: ack!  so.. one instance back to labs-puppetmaster, one instance deleted and one ..my own puppetmaster.. booted up again.  nothing shutdown or broken run now
[21:01:17] <andrewbogott>	 mutante: thank you!
[21:07:37] <chasemp>	 andrewbogott: yes sorry, I can't remember why it's disabled
[21:07:42] <chasemp>	 I think maybe tyler did that
[21:07:44] <andrewbogott>	 thanks
[21:48:11] <danmichaelo>	 Hi, I'm trying to see if there is a way I can setup a websocket server with the uwsgi-plain gridengine backend and python3. With my first attempt, I'm trying the example in https://uwsgi-docs.readthedocs.io/en/latest/WebSockets.html , but both the http-websockets and http-raw-body configuration options yield "unknown config directive", so I guess there is something that's not enabled on the gridengine uwsgi. Is it something I can
[21:48:11] <danmichaelo>	  request on Phabricator?
[22:04:18] <bd808>	 danmichaelo: sounds like its worth a task to look into. The answer might be that the uwsgi we get from Debian upstream is older than you need for this, but that's just a guess based on the error you are mentioning
[22:08:03] <danmichaelo>	 Right. What's a simple way to find out which version it is?
[22:09:26] <danmichaelo>	 If it's the default debian stretch version, I think it should be 2.0.14
[22:13:17] <bd808>	 danmichaelo: yes. looks like it is 2.0.14+20161117-3+deb9u2
[22:16:31] <danmichaelo>	 bd808: Weird, that should be new enough.
[23:11:28] <brion>	 agh, i either haven't set up my ssh config right or i've forgotten my hostnames :D
[23:13:52] <brion>	 seems to be a keys problem
[23:14:18] <bd808>	 brion: your keys or the host keys?
[23:14:30] <brion>	 bd808: i think my personal keys
[23:16:01] <brion>	 i guess i just log in with the old keyh and add my new one? or should there be a more managed way to do this
[23:16:07] * brion pokes around horizon
[23:16:38] <bd808>	 brion: add your ssh keys at https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack
[23:16:51] <brion>	 ah it's on wikitech :D that explains it
[23:16:52] <brion>	 thanks
[23:17:15] <bd808>	 umm... why is that page not listing my existing keys?
[23:17:21] <brion>	 yeah me too
[23:17:22] <brion>	 O_O
[23:17:38] <bd808>	 I can see my keys in LDAP...
[23:18:01] <brion>	 i tried adding one and it says "Failed to import keypair."
[23:18:03] <bd808>	 I see 2 keys for you too brion (in the directory)
[23:18:24] <brion>	 curiouser and curiouser
[23:18:46] <bd808>	 let's see if Striker can see the keys...
[23:19:15] <bd808>	 Ok, I see my keys at https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/
[23:19:24] <bd808>	 brion: ^ you can try that UI next :)
[23:19:43] <bd808>	 I'll file a bug about the wikitech key hiding problem
[23:19:59] <paladox>	 my key shows in wikitech's.
[23:20:01] <paladox>	 *ui
[23:20:14] <bd808>	 It might be fallout from the changes I made to the config to enforce username casing
[23:23:00] <brion>	 bd808: woohoo, that one worked :) I added my wikimedia public key, got my settings in shape, and can now log into my virts
[23:23:15] <bd808>	 sweet
[23:24:04] <brion>	 let's see.... my wikitech account is cased as 'Brion VIBBER', and the toolsadmin let me log in with it cased that way too
[23:25:29] <bd808>	 Yeah. toolsadmin actually should still be case-insensitive for logins. Only Wikitech had to be changed to case-sensitive because of weird remote account attachment bugs
[23:25:46] <brion>	 fun
[23:25:49] <bd808>	 But "Brion VIBBER" is your official cn/username in LDAP
[23:25:56] <brion>	 curious then!
[23:26:36] <brion>	 (i love how apt prompts me with questions about package upgrades like i know what they heck it's talking about)
[23:26:49] <brion>	 let's hope i don't break the vm ;)
[23:28:58] <brion>	 oh good god i have one on jessie still
[23:29:14] <brion>	 ok time ot update that
[23:32:14] * brion launches backup and wanders off
[23:36:39] <Krenair>	 bd808, I don't see any keys in wikitech and I don't think I have any case sensitivity problems around my account
[23:36:47] <Krenair>	 I definitely have keys in LDAP though
[23:37:00] <bd808>	 yeah, something got broken somewhere :/
[23:37:51] <bd808>	 Its going to take some poking to debug since it is not crashing. I'll probably have to do some hot patching of the OSM extension to figure out where things are going wrong