[09:40:18] <arturo>	 !log toolsbeta livehacking toolsbeta-sgebastion-04 (https://gerrit.wikimedia.org/r/c/566045 and https://gerrit.wikimedia.org/r/c/565575) and toolsbeta-proxy-1 (https://gerrit.wikimedia.org/r/c/565556) for testing T234617
[09:40:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[09:40:21] <stashbot>	 T234617: Toolforge. introduce new domain toolforge.org - https://phabricator.wikimedia.org/T234617
[10:24:02] <arturo>	 !log admin running `sudo systemctl restart apache2.service` in both labweb servers to try mitigating T240852
[10:24:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:24:05] <stashbot>	 T240852: CloudVPS: horizon giving http/500 intermitently - https://phabricator.wikimedia.org/T240852
[12:49:36] <arturo>	 !log toolsbeta cleanup livehackings in toolsbeta-sgebastion-04 and toolsbeta-proxy-1
[12:49:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[17:21:26] <bstorm_>	 !log tools rebooting toolschecker to recover stale nfs handle
[17:21:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:43:15] <bstorm_>	 !log admin remounting /mnt/nfs/dumps-labstore1007.wikimedia.org/ on all dumps-mounting projects
[17:43:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[18:30:54] <bstorm_>	 !log tools depooling and rebooting tools-sgeexec-[0911,0913,0919,0921,0924,0931,0933,0935,0939,0941].tools.eqiad.wmflabs
[18:30:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:17:54] <bstorm_>	 !log tools depooled and rebooted tools-sgeexec-0914 because it was acting funny
[19:17:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:25:02] <bstorm_>	 !log tools hard rebooting tools-sgeexec-0913/14/35 because they aren't even on the network
[19:25:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:46:19] <AntiComposite>	 Cyberpower678, (slightly dumb question, but the VPT thread is getting long) Have you tried turning reFill off and back on again?
[19:46:32] <Cyberpower678>	 Yes
[19:46:47] <Cyberpower678>	 I only know of the webservice though.
[19:47:34] <Cyberpower678>	 The documentation of the tool is less than helpful, so it leaves me effectively unaware of any backends that should be rebooted.
[19:53:25] <Krenair>	 Cyberpower678, AntiComposite what is the problem?
[19:54:19] <mutante>	 !log puppet-diffs syncing puppet facts from local puppet repo
[19:54:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL
[19:55:30] <AntiComposite>	 The https://tools.wmflabs.org/refill/ landing page wasn't loading at all for me before, it is now, so that's a start
[19:56:12] <AntiComposite>	 https://phabricator.wikimedia.org/T213515 says that refill-api is the backend
[19:56:27] <Krenair>	 that appears up too: https://tools.wmflabs.org/refill-api/
[19:57:25] <AntiComposite>	 Swagger is, but POSTs to https://tools.wmflabs.org/refill-api/fixWikipage 500
[20:32:08] <isaacj>	 question: for tools hosted on toolforge, if I as the tool developer don't do any data collection -- i.e. no analytics tracking scripts etc. -- is there a privacy policy to point to? i presume some information is being logged somewhere even if I haven't done it explicitly? the best I can think of is essentially saying "No additional personal data is collected per https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use"
[20:34:36] <Krenair>	 AntiComposite, yeah I don't know what's up with that one
[20:34:57] <Krenair>	 app is clearly serving HTTP 500 in response to a POST to /refill-api/fixWikipage - it's not happening at the proxy or anything
[20:38:12] <AntiComposite>	 Yeah
[20:39:18] <AntiComposite>	 The status endpoint doesn't error out immediately, can't test it properly because there are no jobs
[20:39:47] <AntiComposite>	 https://github.com/zhaofengli/refill/blob/115a5c42e31b65f841f30e16ffce5c81eee59036/backend/app.py#L116 is the deployed code right now
[20:42:07] <Krenair>	 the thing is this app is clearly serving some endpoints just fine
[20:42:10] <AntiComposite>	 There should be a ton of exceptions in /data/project/refill-api/uwsgi.log , I can't see it because of file permissions
[20:42:54] <Krenair>	 AntiComposite, not when I try to make the request
[20:43:44] <Krenair>	 oh there we go
[20:43:47] <Krenair>	 just had to be patient
[20:43:51] * Krenair shakes tiny fist at NFS
[20:44:05] <AntiComposite>	 Oh hey, I'm getting a 202 now
[20:44:55] <Krenair>	 AntiComposite, https://phabricator.wikimedia.org/P10239
[20:45:28] <Krenair>	 sounds like maybe it's actually supposed to be serving 4xx to this request, but there's some bug that makes it give HTTP 500 :)
[20:48:50] <AntiComposite>	 Yeah, it's expecting api.payload to always be dict-like, but doesn't handle the case where api.payload is None
[20:51:54] <bd808>	 isaacj: that sounds fine to me. As you probably found we don't really have any strong guidance for tools about their local privacy/TOU statements
[20:52:47] <Krenair>	 AntiComposite, that even is the payload attribute though?
[20:52:58] <Krenair>	 what*
[20:54:16] <isaacj>	 thanks bd808 : yeah, kinda wild-westy out there but also I appreciate the flexibility of the ecosystem :) just wanted to make sure i wasn't misleading users
[20:55:42] <AntiComposite>	 Krenair, it /should/ be the data from the POST, decoded into a Python dict
[20:55:56] <AntiComposite>	 by flask_restplus
[20:57:38] <Krenair>	 AntiComposite, well, I still got HTTP 500 when  sending it empty dict
[20:57:57] <AntiComposite>	 I get a 400
[20:59:19] <Krenair>	 AntiComposite, $ curl -X POST 'https://tools.wmflabs.org/refill-api/fixWikipage' --data {}
[20:59:19] <Krenair>	 {"message": "Internal Server Error"}
[21:01:12] <AntiComposite>	 Hmm...it looks like it wants a "Content-Type: application/json" header attached
[21:02:16] <Krenair>	 $ curl -X POST 'https://tools.wmflabs.org/refill-api/fixWikipage' --data {} -H "Content-Type: application/json"
[21:02:16] <Krenair>	 {"message": "The browser (or proxy) sent a request that this server could not understand."}
[21:02:23] <Krenair>	 that is 400 though
[21:02:40] <AntiComposite>	 Which is the expected behavior
[21:02:44] <Krenair>	 yes
[21:02:46] <Krenair>	 cool
[21:03:21] <AntiComposite>	 The lack of a header issue is an upstream problem, but it could be handled better
[21:05:14] <AntiComposite>	 Was that the only recent traceback in the log, or were there different ones?
[21:05:26] * AntiComposite still isn't sure why it stopped working and is working now
[21:11:02] <Krenair>	 AntiComposite, I don't know either, while fiddling around with it I did delete the pod so maybe it getting started again helped in some way
[21:11:32] <AntiComposite>	 Probably
[21:12:58] <AntiComposite>	 https://www.youtube.com/watch?v=rksCTVFtjM4
[21:13:22] <Krenair>	 lol
[21:41:24] * AntiComposite wonders out loud why reFill appears to be targeting python3.6
[21:53:20] <Krenair>	 AntiComposite, as opposed to 3.7/3.8/3.9?
[21:54:04] <AntiComposite>	 I'd usually expect 3.5 or 3.7 on Toolforge
[22:03:51] <AntiComposite>	 oic, celery didn't support 3.7 1-2 years ago
[22:53:31] <jeh>	 !log openstack deactivated image debian-9.1-stretch (deprecated  2017-11-16)
[22:53:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL