[01:28:43] <harej>	 What is my sudo password on a VPS instance?
[01:37:09] <paladox>	 harej you shoulden't have one
[01:37:16] <paladox>	 unless it's root?
[01:50:17] <harej>	 Well awkwardly, I'm not a sudoer
[01:55:32] <paladox>	 harej is it asking you for a password? Did you add your ssh key to wikitech?
[02:11:14] <harej>	 Though I checked in Horizon and I should be one
[04:55:36] <zhuyifei1999_>	 srish_aka_tux: `app.use( "/nodejs-mw-oauth-tool", router );` <= you might want to tell the user to change this line
[04:56:55] <zhuyifei1999_>	 otherwise, really cool :)
[04:58:36] <srish_aka_tux>	 @zhuyifei1999_ thanks! you meant tell the user to change the name of the tool or something else? 
[05:01:07] <zhuyifei1999_>	 I mean, change the line to refer to the name of the new tool
[05:10:14] <srish_aka_tux>	 @zhuyifei1999_ Okay! sounds good :) 
[05:10:26] <zhuyifei1999_>	 :)
[05:28:15] <wikibugs>	 (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/442762 (owner: 10L10n-bot)
[09:46:15] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: hieradata: add profile::openstack::eqiad1::neutron::db_pass [labs/private] - 10https://gerrit.wikimedia.org/r/442814 (https://phabricator.wikimedia.org/T196633)
[09:46:39] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [C: 032] hieradata: add profile::openstack::eqiad1::neutron::db_pass [labs/private] - 10https://gerrit.wikimedia.org/r/442814 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez)
[09:46:42] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [V: 032 C: 032] hieradata: add profile::openstack::eqiad1::neutron::db_pass [labs/private] - 10https://gerrit.wikimedia.org/r/442814 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez)
[12:36:37] <wikibugs>	 (03PS1) 10Arturo Borrero Gonzalez: hieradata: add profile::openstack::eqiad1::glance::db_pass [labs/private] - 10https://gerrit.wikimedia.org/r/442837 (https://phabricator.wikimedia.org/T196633)
[12:38:12] <wikibugs>	 (03CR) 10Arturo Borrero Gonzalez: [V: 032 C: 032] hieradata: add profile::openstack::eqiad1::glance::db_pass [labs/private] - 10https://gerrit.wikimedia.org/r/442837 (https://phabricator.wikimedia.org/T196633) (owner: 10Arturo Borrero Gonzalez)
[15:50:59] <pintoch>	 hi! I am trying to understand why my web service https://tools.wmflabs.org/openrefine-wikidata/ is suddenly so slow
[15:51:07] <pintoch>	 I have restarted the service but it does not help. Here is an excerpt of the logs (no user info is there, so I think it is fine to share it): http://pintoch.ulminfo.fr/5dffa4f521/uwsgi.log.txt
[15:52:41] <pintoch>	 these lines "DAMN ! worker 3 (pid: 12) died, killed by signal 9 :( trying respawn ..." are suspicious (and they are new, I cannot find them in a backup of the previous logs)
[15:53:55] <pintoch>	 in what context would a uWSGI worker normally get killed by signal 9? taking up too much memory maybe?
[16:01:30] <pintoch>	 let's add that even loading the main page is very slow (https://tools.wmflabs.org/openrefine-wikidata/), even if that is basically just a static HTML page (nothing expensive server-side)
[16:23:49] <pintoch>	 hmm, it seems that it's a wmflabs-wide issue
[16:24:03] <chicocvenancio>	 pintoch: how so?
[16:24:28] <pintoch>	 most tools I try load suspiciously slowly
[16:24:37] <pintoch>	 for instance, https://tools.wmflabs.org/quickstatements/#
[16:25:19] <pintoch>	 and other tools I run, such as https://tools.wmflabs.org/editgroups/ or https://tools.wmflabs.org/oabot/
[16:25:45] <pintoch>	 chicocvenancio: how fast do they load for you?
[16:27:04] <chicocvenancio>	 slowly...
[16:27:34] <pintoch>	 should we ping someone in particular?
[16:28:21] <harej>	 chicocvenancio: do you know why I can’t sudo into toolsbeta-flynn-01?
[16:28:41] <chicocvenancio>	 harej: yes
[16:28:59] <chicocvenancio>	 add yourself to the root rule in toolsbeta on horizion
[16:29:10] <harej>	 I think I’m already there?
[16:31:10] <pintoch>	 bd808: there seems to be a widespread performance issue on most web-based tools. For instance, https://tools.wmflabs.org/openrefine-wikidata/ and https://tools.wmflabs.org/quickstatements/ are very slow to load. It seems that web workers get killed (for instance: http://pintoch.ulminfo.fr/5dffa4f521/uwsgi.log.txt)
[16:31:31] <chicocvenancio>	 harej: did you add yourself?
[16:31:38] <chicocvenancio>	 i did not
[16:32:20] <chicocvenancio>	 pintoch: I'm not sure both issues are related (they may be, but I'm not seeing how atm)
[16:32:54] <chicocvenancio>	 openrefine is just (sometimes) loading pages slowly, other tools seem to be slow due to tools-static loading slowly or not at all 
[16:34:19] <chicocvenancio>	 !log toolsbeta adding harej as root for flynn testing
[16:34:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[16:34:50] <chicocvenancio>	 harej: for reference that is defined in https://horizon.wikimedia.org/project/sudo/ (choosing toolsbeta as the project)
[16:35:14] <chicocvenancio>	 you probably have to wait for puppet to run, let me see if I can speed that up
[16:37:45] <pintoch>	 chicocvenancio: interesting… let me know if there is anything I can do on my side to debug this, I am a bit clueless
[16:38:47] <paladox>	 chicocvenancio i wonder do tools use the same labvirts as normal vps?
[16:39:02] <paladox>	 chicocvenancio could this be related to https://phabricator.wikimedia.org/T198348 ?
[16:40:23] <andrewbogott>	 !log tools rebooting tools-worker-1012 and tools-worker-1015 to get their nfs mounts unstuck
[16:40:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:42:44] <chicocvenancio>	 pintoch: the uwsgi killing workers thing is hard to debug. I'd wait a bit for the team to figure out if/what are the cloud wide issues before debugging, but the main culprit is likely them taking too long to read the files, not memory
[16:42:54] <andrewbogott>	 !log tools rebooting tools-worker-<everything> to get NFS unstuck
[16:42:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:43:45] <pintoch>	 chicocvenancio: okay, thanks a lot
[16:43:50] <chicocvenancio>	 paladox: I'm unsure what the underlying infra is, haven't worked on that
[16:44:11] <arturo>	 paladox: yes, same labvirts
[16:44:30] <arturo>	 i.e, labvirts are shared for all cloudvps
[16:44:59] <paladox>	 hmm
[16:45:00] <paladox>	 https://wmbot.wmflabs.org/logs/%23wikimediacloud/
[16:45:03] <paladox>	 is very slow
[16:45:05] <paladox>	 arturo ^^
[16:45:11] <paladox>	 that is usually fast to load
[16:45:43] <paladox>	 https://grafana-labs.wikimedia.org/dashboard/db/top-instances?orgId=1&from=now-5m&to=now
[16:45:54] <arturo>	 we have an ongoing operation regarding NFS that is affecting toolforge
[16:48:02] <arturo>	 !log tools rebooting tools-docker-registry-01
[16:48:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:50:59] <Cyberpower678>	 anomie: what API call is the best call to get all available subpages of a given page?
[17:16:25] <Cyberpower678>	 musikanimal: ping
[17:17:57] <musikanimal>	 https://www.mediawiki.org/w/api.php?action=help&modules=query%2Ballpages
[17:20:46] <chasemp>	 !log tools tools-worker-1007:~# /sbin/reboot
[17:20:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:43:33] <paladox>	 hmm https://wmbot.wmflabs.org/logs/%23wikimediacloud/ still is not loading
[17:53:19] <chasemp>	 !log tools tools-clushmaster-01:~$ clush -w @all "sudo puppet agent --disable 'labstore1007 outage'"
[17:53:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[18:02:13] <chasemp>	 !log tools tools-clushmaster-01:~$ clush -w @all "sudo umount -fl /mnt/nfs/dumps-labstore1007.wikimedia.org"
[18:02:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:50:36] <chasemp>	 !log tools tools-clushmaster-01:~$ clush -w @all 'sudo umount -fl /mnt/nfs/dumps-labstore1006.wikimedia.org'
[19:50:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[20:55:21] <paladox>	 !help https://wmbot.wmflabs.org/logs/%23wikimediacloud/ is not loading for me
[20:55:21] <wm-bot>	 paladox: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[20:57:00] <bd808>	 paladox: I'll see if the cause is obvious, but we may need petan to take a look
[20:57:29] <paladox>	 thanks bd808 :)
[21:00:31] <bd808>	 !log wm-bot Restarted apache2 on wm-bot2.wm-bot.eqiad.wmflabs; web interface to logs was unresponsive
[21:00:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL
[21:00:39] <bd808>	 paladox: works for me now
[21:01:00] <paladox>	 bd808 thanks!
[21:01:07] <paladox>	 hmm loading slowly
[21:01:27] <paladox>	 hmm not loading for me
[21:01:49] <bd808>	 paladox: https://wm-bot.wmflabs.org/logs/%23wikimedia-cloud/
[21:02:03] <paladox>	 aha
[21:02:07] <paladox>	 that's why
[21:02:08] <paladox>	 thanks
[21:02:14] <paladox>	 bd808 topic has the wrong url :)
[21:02:58] <bd808>	 yeah... trying to fix that
[21:03:12] <paladox>	 ok thanks!
[21:04:04] <bd808>	 apparently we have topic lock on... /me is not a fan
[21:04:48] <chicocvenancio>	 we had spam a while back, iirc
[21:17:53] * bd808 stops piddling
[21:30:26] <Cyberpower678>	 musikanimal: I've got a question regarding the edit filters
[21:30:33] <Cyberpower678>	 Feel free to ping me in a PM.
[21:30:46] <Cyberpower678>	 musikanimal: and thanks for the above answer
[23:44:41] <MarioFinale>	 Hi everyone, i have a question... There are any way to check what version of the TLS encryption protocol my .net/MONO bot is negotiating with the server?.
[23:58:18] <MarioFinale>	 !help 
[23:58:18] <wm-bot>	 MarioFinale: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team
[23:59:50] <bd808>	 MarioFinale: hmmm... I think you would have to check that from inside your mono code itself, but I have no idea how to actually do that. Do you remember the task number of the task about mono and TLS?