[00:12:29] <wikibugs>	 (03PS1) 10Awight: Add scoring repo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/425930
[00:41:52] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Add scoring repo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/425930 (owner: 10Awight)
[00:42:00] <awight>	 :)
[00:42:20] <wikibugs>	 (03Merged) 10jenkins-bot: Add scoring repo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/425930 (owner: 10Awight)
[00:42:36] <wikibugs>	 (03CR) 10jenkins-bot: Add scoring repo [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/425930 (owner: 10Awight)
[00:53:12] <awight>	 !log deployment-prep ORES: Test large file in LFS
[00:53:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL
[01:04:06] <legoktm>	 Krinkle: https://tools.wmflabs.org/nagf/?project=codesearch 404 project not found?
[01:09:07] <musikanimal>	 bd808: can you enable `page_assessments` and `page_assessments_projects` for frwiki and huwiki?
[01:09:13] <musikanimal>	 and that will be the last of it for a while
[01:12:26] <musikanimal>	 now I might should mention that for huwiki, the data isn't there yet (but the tables are). So there will be a surge of data coming in during a short timeframe once they add the parser function (400,000 or so rows)
[01:12:30] <musikanimal>	 not sure if that matters
[01:24:23] <Krinkle>	 legoktm: Hm..
[01:25:56] <Krinkle>	 legoktm: problem is upstream
[01:25:56] <Krinkle>	 https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novainstances&niregion=eqiad&niproject=codesearch
[01:25:58] <Krinkle>	 https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novainstances&niregion=eqiad&niproject=cvn
[01:26:03] <Krinkle>	 it seems to return empty array for all possible values
[01:26:07] <Krinkle>	 bd808: ^
[01:26:26] <Krinkle>	 https://github.com/wikimedia/nagf/blob/master/inc/Nagf.php#L18-L25
[01:26:37] <Krinkle>	 The project list still works though https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novaprojects
[03:28:56] <zhuyifei1999_>	 Krinkle bd808: fwiw, the same thing is happening to the video project
[04:07:27] <quiddity>	 a bunch of them. admin, analytics, art-recs.  just from the a's.
[06:13:41] <revi>	 s1 on web.db seem to be sick on wikireplica? https://usercontent.irccloud-cdn.com/file/0lIvI2JL/IMG_0879.PNG
[06:14:35] <zhuyifei1999_>	 jynus: ^
[06:21:00] <revi>	 ok seems to caught up
[07:28:34] <sebastian-wmse>	 !log wikispeech Deploy latest from Git master: 81fca40 (T149302), 2a69b6b (T182205, T174799)
[07:28:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL
[07:28:39] <stashbot>	 T174799: failed stubbing in tests. - https://phabricator.wikimedia.org/T174799
[07:28:39] <stashbot>	 T182205: Refactor JS code - https://phabricator.wikimedia.org/T182205
[07:28:39] <stashbot>	 T149302: Settings page - https://phabricator.wikimedia.org/T149302
[07:39:35] <jynus>	 revi: there is category collation maintenance ongoing on all wikis on production, specially enwiki, which makes replicas not able to guarantee 0 lag
[07:40:24] <revi>	 ok!
[07:40:29] <jynus>	 you can see the higher write load at: https://grafana.wikimedia.org/dashboard/db/mysql?panelId=2&fullscreen&orgId=1&var-dc=eqiad%20prometheus%2Fops&var-server=db1052&var-port=9104&from=now-7d&to=now
[07:40:58] <jynus>	 when that finishes, things will go back to normal, there is not much we can do at wikirreplicas
[14:15:36] <Krinkle>	 andrewbogott: It seems the Nova API is still not responding corretly, is this related to "VPS upgrade", and so, which upgrade is that?
[14:15:48] <andrewbogott>	 'still not responding correctly'?
[14:15:54] <andrewbogott>	 Can you tell me more about what you mean by 'still'?
[14:15:57] <andrewbogott>	 I haven't started the upgrade yet
[14:16:32] <Krinkle>	 Since yesterday, most projects returned by https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novaprojects are not returning instances at urls like https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novainstances&niregion=eqiad&niproject=codesearch or https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novainstances&niregion=eqiad&niproject=cvn
[14:16:49] <Krinkle>	 I know it does validate it because providing a bogus project name does result in a proper error instead of empty array
[14:16:58] <Krinkle>	 This has as result that https://tools.wmflabs.org/nagf/?project=cvn doesn't work for most projects.
[14:17:31] <paladox>	 Krinkle ah, i reported problems with that tool too, i just thought it was the tool, but grafana worked.
[14:17:32] <Krinkle>	 Although a few specific ones to come through (catgraph, deployment-prep)
[14:17:55] <Krinkle>	 paladox: Those are not related. Grafana does not use the Nova API.
[14:18:03] <paladox>	 oh
[14:18:08] <Krinkle>	 The problem is not with Nagf.
[14:18:29] <andrewbogott>	 Krinkle: I wouldn't have thought that the wikitech api had worked in many months.  It was replaced by https://tools.wmflabs.org/openstack-browser/
[14:18:47] <Krinkle>	 Does that have non-html endpoint?
[14:19:35] <andrewbogott>	 Krinkle: I'm sorry, I'm in the middle of an upgrade window and can't think about this right now
[14:19:40] <Krinkle>	 Okay
[14:19:53] <andrewbogott>	 But, do please make a phab ticket
[14:22:05] <Krinkle>	 I didn't think of using openstack-browser as I saw it as a replacement for the UI, but while undocumented (afaik), I did find some api endpoints in its Flask code
[14:22:07] <Krinkle>	 https://tools.wmflabs.org/openstack-browser/api/projects.json
[14:22:09] <Krinkle>	 https://tools.wmflabs.org/openstack-browser/api/dsh/project/cvn
[14:22:12] <Krinkle>	 andrewbogott: Thanks :)
[14:32:40] <darkblue_b>	 hi all - I admit I have not visited in some time, but.. I got "labs" credentials some time ago, and I want to make them current.. how shall I proceed ?
[14:33:26] <darkblue_b>	 .. I am a shell user here in Berkeley, on various Debian-style machines mainly
[14:35:27] <darkblue_b>	 in the wikitech world the login is darkblueb -- same as this one but no hyphen
[14:36:08] <eddiegp>	 darkblue_b: What does "want to make them current" mean?
[14:36:13] <eddiegp>	 Do you still have your password?
[14:36:18] <chicocvenancio>	 Hey darkblue_b, what is your objective with the access? 
[14:36:32] * chicocvenancio checks the users
[14:36:36] <darkblue_b>	 I have three physical pieces of paper in front of me, that have not been updated for some time
[14:36:52] <darkblue_b>	 I dont know what the world looks like here, lately... 
[14:36:57] <chicocvenancio>	 Paper? 
[14:37:06] <darkblue_b>	 yes with my account info
[14:37:28] <darkblue_b>	 when I signed up, it was to be able to run shell jobs in a VM..
[14:37:42] <darkblue_b>	 not sure how you run this now.. I work on VMs daily 
[14:38:18] <chicocvenancio>	 What shell jobs? What were their purpose? 
[14:38:41] <darkblue_b>	 http://blog.light42.com/wordpress/wp-content/uploads/2015/09/clark.pdf
[14:38:55] <darkblue_b>	 I am Brian Hamlin  :-)
[14:39:31] <darkblue_b>	 thats a VM we built.. that was the same era as when I signed up here
[14:39:48] <darkblue_b>	 I have not been keeping up.. you have been busy no doubt
[14:46:10] <darkblue_b>	 (I am subscribed to cloud@lists.wikimedia.org as maplabs, if that matters also)
[14:46:36] <chicocvenancio>	 darkblue_b: Do you have access to wikitech? 
[14:47:11] <darkblue_b>	 wikitech.wikimedia.org - is that right ?
[14:48:16] <chicocvenancio>	 Yes
[14:48:18] <darkblue_b>	 ok yes - I am at https://wikitech.wikimedia.org/wiki/Main_Page  and logged in
[14:51:25] <chicocvenancio>	 darkblue_b: your shell username is dbb
[14:51:33] <darkblue_b>	 oohhh
[14:51:37] <darkblue_b>	 yes I use that
[14:52:00] <darkblue_b>	 I might want a reset on that .. I do not see it here
[14:52:07] <darkblue_b>	 the email is unchanged
[14:52:12] <chicocvenancio>	 You private key can be set in the Wikitech preferences 
[14:52:23] <darkblue_b>	 yes I have a key to use
[14:53:09] <chicocvenancio>	 That is is for credentials, Wikitech username and password, shell username and private key
[14:53:27] <darkblue_b>	 hm I could hunt around here for notes on that.. it will take a few minutes.. I realize I started this.. thx for your patience
[14:53:43] <darkblue_b>	 is this the right channel ?
[14:53:47] <chicocvenancio>	 I don't see you participating in any project though, darkblue_b 
[14:53:55] <darkblue_b>	 yes right - I am not current
[14:54:13] <chicocvenancio>	 Yes, this is the right channel, and I'm the right person 
[14:54:17] <darkblue_b>	 aha thx
[14:54:29] <chicocvenancio>	 You're welcome 
[14:54:45] <darkblue_b>	 I signed up early when this project was forming.. it would not surprise me if I am not in groups that are ordinary or even required now
[14:55:09] <darkblue_b>	 I will look for shell things now.. 
[15:00:52] <eddiegp>	 andrewbogott: toolforge reported down in -tech
[15:01:06] <arturo>	 eddiegp: we are working on it
[15:01:13] <eddiegp>	 Pinging you because your mail said running instances/tools shouldn't be affected
[15:01:22] <arturo>	 which kind of down? BTW?
[15:01:30] <andrewbogott>	 'toolforge down' is pretty broad
[15:01:39] <eddiegp>	 Error 500 for various tools
[15:01:49] <arturo>	 !log tools clush run with puppet agent for T145919 to apply https://gerrit.wikimedia.org/r/#/c/426063/
[15:01:58] <andrewbogott>	 https://tools.wmflabs.org/openstack-browser/ seems fine
[15:02:23] <arturo>	 no /home directories in some instances
[15:02:30] <arturo>	 aborrero@tools-clushmaster-01:/$ ls /home
[15:02:30] <arturo>	 ls: cannot access /home: Stale file handle
[15:02:45] <eddiegp>	  ToBeFree-mobile | Internal Server Error at https://tools.wmflabs.org/copyvios/    --- 500 error at https://tools.wmflabs.org/afdstats
[15:03:05] <chicocvenancio>	 darkblue_b: Well, take a look at our getting started documentation, we have 2 main products, Toolforge and Cloud Vps https://wikitech.wikimedia.org/wiki/Help:Getting_Started
[15:03:21] <eddiegp>	 And I know 'toolforge down' is very broad, I'm just relaying what users reported in -tech ;)
[15:03:56] <chicocvenancio>	 We also have https://paws.wmflabs.org that is a web interface for python and R notebooks 
[15:04:06] <darkblue_b>	 ah that -- definitely
[15:06:13] <eddiegp>	 Openstack browser doesn't look fine if you look at e.g. https://tools.wmflabs.org/openstack-browser/project/ by the way. I was expecting that though, as it probably relies on some api that is part of the maintenance.
[15:06:49] <darkblue_b>	 chicocvenancio: I have Ryan Lane's email here for some reason also.. no recall why
[15:07:41] * chicocvenancio is new here, other team members may know the historical reason? 
[15:08:24] <darkblue_b>	 no worries -- looking
[15:08:53] <ragesoss>	 my tool is down too, and I can't get to it via `become`.
[15:08:54] <chicocvenancio>	 eddiegp: could you point the users here for support? 
[15:09:13] <eddiegp>	 Yeah, I'm going to do that.
[15:09:19] <chicocvenancio>	 ragesoss: become is filling? 
[15:09:35] <chicocvenancio>	 *failing? 
[15:10:13] <ragesoss>	 chicocvenancio: actually, just started working.
[15:10:28] <ragesoss>	 (and tool is back up, no longer serving 500s)
[15:10:47] <ragesoss>	 it previously said there was no such tool when I tried 'become'
[15:11:00] <eddiegp>	 Others reported the same, probably part of the NFS issues the team is working on :)
[15:11:12] <ragesoss>	 cool beans.
[15:11:36] <chasemp>	 things went crazy, we have a temp fix in place, in theory road to recovery 
[15:11:47] * chasemp waves at ragesoss 
[15:12:08] * ragesoss waves back
[15:28:59] <eddiegp>	 http://shinken.wmflabs.org/ reports '502 - Bad Gateway'
[15:30:55] <paladox>	 i think they stopped shinken
[15:33:53] <bd808>	 eddiegp, paladox: yes, shinken is down to reduce alert spam
[15:34:00] <paladox>	 thanks.
[15:35:55] <eddiegp>	 Okay.
[15:37:56] <zhuyifei1999_>	 the `become` issue is probably the home directory not found I think. (there's a phab ticket about that command's error messages being unimformative
[15:52:40] <darkblue_b>	 curious .. I listed 'Servers' on the openstack browser, then picked on in the (long) list and clicked on it.. I get a response "does not exist, are you guessing ?"  .. which is alarming.. 
[15:52:48] <darkblue_b>	 https://tools.wmflabs.org/openstack-browser/server/wikilabels.wmf-research-tools.eqiad.wmflabs
[15:53:11] <darkblue_b>	 I did not guess and I am not trying to
[15:53:41] <bd808>	 darkblue_b: its a poor error message. If anything goes wrong loading the data it gives that response
[15:54:08] <darkblue_b>	 ok thx, I have not been here in a long time, and I do not want to do anything inappropriate 
[15:54:24] <bd808>	 darkblue_b: we are doing maintenance right now that will make a lot of random errors show up
[15:54:29] <darkblue_b>	 ooohhhh
[15:54:31] <darkblue_b>	 thx
[15:54:47] <bd808>	 darkblue_b: so sadly, you piked a bad day to come back :)
[15:54:49] <bd808>	 *picked
[15:55:09] <darkblue_b>	 chicocvenancio: is that true? should we wait to get the reset thing going ?
[15:55:34] <chicocvenancio>	 For a project request, you shouldn't be affected
[15:55:46] <darkblue_b>	 at your convenience
[15:55:59] <chicocvenancio>	 https://phabricator.wikimedia.org/project/view/2875/ will get you htere
[15:56:04] <chicocvenancio>	 *there
[15:56:52] <chicocvenancio>	 The other option is to use Toolforge, and in that case it might be better to learn in a time with less likelyhood of things breaking 
[15:57:08] <darkblue_b>	 aha - maybe that is what I did
[15:57:12] <darkblue_b>	 that sounds familiar
[15:57:56] <chicocvenancio>	 But it would not be a VM for you, but a platform with kubernetes and a sun grid engine
[15:58:21] <darkblue_b>	 sure - I can read a bit later.. I want to get the basics reset if we can
[15:58:52] <chicocvenancio>	 Your account is working
[16:00:30] <chicocvenancio>	 You can test that by connecting to bastion.wmflabs.org, but without a project it is not a very useful server
[16:01:03] <chicocvenancio>	 darkblue_b: ^
[16:01:32] <darkblue_b>	 ok thats sounds familiar too
[16:02:39] <darkblue_b>	 chicocvenancio: you have an ECDSA host key ?  :-)
[16:03:54] <darkblue_b>	 I am not a key expert, but I thought that was widely frowned upon, in favor of something newer
[16:06:16] <musikanimal>	 bd808: thanks!
[16:08:16] <bd808>	 darkblue_b: actually ECDSA is the new better type of key. 
[16:08:49] <bd808>	 the EC part that is the betterness, vs old DSA keys
[16:09:15] <darkblue_b>	 hm
[16:17:52] <darkblue_b>	 ok chicocvenancio  - logged in at bastion
[16:18:16] <darkblue_b>	 Debian 4 9 65
[16:19:37] <darkblue_b>	 .. with mosh enabled  !!
[16:23:53] <chicocvenancio>	 Cool
[16:24:30] <chicocvenancio>	 darkblue_b: that is a bastion host only for login to other VPS instances 
[16:25:09] <chicocvenancio>	 To do actual work you need a tool in Toolforge or a cloud vps project 
[16:26:50] <Krinkle>	 Hm.. after restarting the webservice for 'nagf', it came back up with the face of tools-admin
[16:26:55] <Krinkle>	 https://tools.wmflabs.org/nagf/
[16:26:57] * Krinkle is very confused
[16:27:51] <Krinkle>	 Hm.. may've been a glitch, it's fixd now
[16:28:37] <chicocvenancio>	 Yeah, it looks OK to me, except for the bugs on projects
[16:29:01] <chicocvenancio>	 That are expected atm
[16:30:29] <paladox>	 https://tools.wmflabs.org/nagf/?project=git
[16:34:43] * darkblue_b notes
[16:47:15] <Krinkle>	 I've added caching so Nagf will remember the correct responses, but it does seem like the openstack-browser API is spotty for some reason
[16:47:27] <Krinkle>	 I filed a bug at https://phabricator.wikimedia.org/T192160
[16:47:49] <darkblue_b>	 chicocvenancio: before we finish for today, is there something you can say about the Jupyter environment here?
[16:48:20] <darkblue_b>	 thx in advance, seems like a busy day here
[16:51:17] <chicocvenancio>	 darkblue_b: Well, it is not suited for all use cases atm, but it is an excellent point of first contact for the community and can be a nice, user friendly way, to demonstrate scripts
[16:52:03] <darkblue_b>	 yes certainly - I am in the BIDS group with some of your hackers
[16:52:13] <darkblue_b>	 .. at Berkeley.. work in progress.. thats fine
[16:52:25] <darkblue_b>	 everyone is figuring this out
[16:53:09] <darkblue_b>	 there are many things here I can (re)learn for search.. as you know I am keen on geo
[16:53:50] <darkblue_b>	 your choices of kernel and stack will influence my own work, etc
[16:55:38] <chicocvenancio>	 Well, yuvi was the one set it up when he was working here, I think he is at Berkeley as well atm
[16:55:51] <darkblue_b>	 yes Yuvi exactly
[16:56:15] <darkblue_b>	 they move fast :-)
[16:57:11] <darkblue_b>	 what should I bookmark to learn/use the Notebooks here?
[16:57:57] <darkblue_b>	 and, I am writing down the Phab link to make a project request, for later
[16:58:00] <chicocvenancio>	 Https://paws.wmflabs.org 
[16:58:05] <darkblue_b>	 super
[16:59:00] <chicocvenancio>	 I would love to discuss jupyterhub if you want, but yuvi can speak about it in a lot more detail than I can
[16:59:49] <darkblue_b>	 I cant keep up with Yuvi, most days :-)
[17:00:10] <darkblue_b>	 I am editing the Jupyter setup for our Linux distro in the next months
[17:00:16] <darkblue_b>	 osgeolive
[17:00:56] <darkblue_b>	 I would like to solve things for users.. they are doing more plumbing and auth
[17:18:34] <darkblue_b>	 thanks for your help chicocvenancio -- good so far, have a great day
[17:18:40] <darkblue_b>	 :-)
[17:19:06] <chicocvenancio>	 You're welcome 
[17:38:21] <legoktm>	 andrewbogott: I got a "Host DOWN alert for codesearch3!" - would that be related to any of the maintenance you were doing?
[17:38:32] <legoktm>	 and it does appear to actually be down
[17:38:35] <andrewbogott>	 legoktm: shouldn't be related
[17:38:44] <andrewbogott>	 um… let me look where it is though
[17:40:10] <legoktm>	 https://horizon.wikimedia.org/project/instances/6d3b44d5-8d4f-48a7-aac5-3ec1da17cd29/ is the instance
[17:40:53] <legoktm>	 the "Action Log" tab has " req-e66c8170-45ea-4b51-935f-12405f5e90d4 	Stop 	April 13, 2018, 5:07 p.m. "
[17:40:57] <legoktm>	 but no user attributed?
[17:41:58] <legoktm>	 !log codesearch booted up codesearch3, it was down for some reason
[17:41:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL
[17:43:40] <andrewbogott>	 legoktm: all I can think is that that host went OOM during the upgrades and killed some VMs.  I don't know why though, it has plenty of RAM
[17:43:42] <andrewbogott>	 right now at least
[17:44:14] <andrewbogott>	 there are 11 VMs in 'shutoff' state but they're scattered about, not enough to really seem like a pattern
[17:44:20] <andrewbogott>	 did it come back up ok?
[17:45:12] <legoktm>	 yep, it's all fine now
[17:45:24] <legoktm>	 I'm skimming through /var/log and don't see anything obvious
[18:10:42] <eddiegp>	 Just as a note, in case anyone else experiences this: The 'Overview' and 'Instances' pages in Horizon wouldn't load for me after the update. Logging out and back in fixed that.
[18:11:33] * chicocvenancio guesses sessions has something to do with that
[18:43:24] <wikibugs>	 (03PS1) 10Legoktm: Include full repository name in logs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426135
[18:43:26] <wikibugs>	 (03PS1) 10Legoktm: Bump container cache [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426136
[18:43:42] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Bump container cache [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426136 (owner: 10Legoktm)
[18:43:45] <wikibugs>	 (03CR) 10Legoktm: [C: 032] Include full repository name in logs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426135 (owner: 10Legoktm)
[18:43:59] <wikibugs>	 (03Merged) 10jenkins-bot: Include full repository name in logs [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426135 (owner: 10Legoktm)
[18:44:01] <wikibugs>	 (03Merged) 10jenkins-bot: Bump container cache [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/426136 (owner: 10Legoktm)
[20:01:31] <zhuyifei1999_>	 !log video bind mount /srv/zhuyifei1999 /srv/v2c/ssu on encoding01 so I can copy the videos to a publicly-accessible (v2c.wmflabs.org) instance local storage rather than the slooowww NFS T191572
[20:01:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL
[20:01:33] <stashbot>	 T191572: Please upload large file to Wikimedia Commons - https://phabricator.wikimedia.org/T191572
[20:36:18] <eddiegp>	 If I click on any project at https://tools.wmflabs.org/openstack-browser/project/ I still see 'Unknown project '<projectname>'. Are you just guessing?' for almost all of them.
[20:36:40] <eddiegp>	 As concrete examples, everything starting with 'a' has that issue for me.
[20:36:50] <eddiegp>	 deployment-prep is the only one I've found so far that doesn't.
[20:38:03] <bd808>	 eddiegp: hmmm... I think that app has a redis cache layer. I wonder if the testing during outage cached a bunch of junk data.
[20:38:07] <bd808>	 I'll take a look
[20:38:25] <eddiegp>	 bd808: Thanks.
[20:40:29] <bd808>	 eddiegp: looks like that's what's going on
[20:41:00] * bd808 tries to figure out an easy purge method
[20:41:29] * eddiegp points out that 'easy' and 'cache purging' is a contradiction by itself
[20:47:29] * bd808 slips and falls into a rabbit hole on the way to a solution
[20:51:48] <bd808>	 my python 3.4 local install is busted for doing tls with pypi. Fun times
[20:52:04] * bd808 shakes fist at the world
[20:56:15] <eddiegp>	 Did you try turning it off and on again?
[20:56:19] <eddiegp>	 S,CNR
[21:17:17] <eddiegp>	 bd808: Looks fixed for me now.
[21:17:30] <eddiegp>	 Thanks! :)
[21:17:44] <bd808>	 things are either aging out or you picked one I manually purged :)
[21:17:57] * bd808 is still in a python34 rathole
[21:24:25] <legoktm>	 just change the cache key?
[21:27:17] <bd808>	 legoktm: yeah. I have the local commit. I got stuck trying to run tox locally with a busted python3.4 interpreter (version we are still running on k8s)
[21:35:54] <andrewbogott>	 bd808: is that https://mail.python.org/pipermail/distutils-sig/2018-April/032114.html maybe?
[21:36:08] <andrewbogott>	 (bd808 I can't claim to know anything other than having seen that roll by on twitter last week)
[21:36:24] <andrewbogott>	 oh, sorry, I read more context, nevermind :)
[21:37:04] <bd808>	 andrewbogott: yeah. the local python3.4 I have installed has the same core problem. Lack of TLS1.2 support.
[21:37:20] * bd808 is hacking a new homebrew build formula
[21:37:59] <bd808>	 I'd better actually check that the version in our k8s containers works too :/
[21:38:46] <bd808>	 it does :)
[21:49:48] <bd808>	 !log tools.openstack-browser Upgrade to b71a890 and restart
[21:49:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL
[21:50:46] <bd808>	 hmmm... did that make it all worse?
[21:52:12] <andrewbogott>	 It doesn't look great :(
[21:52:42] <eddiegp>	 Pages that previously worked stopped to do so. E.g. deployment-prep and analytics
[21:52:48] <bd808>	 'designateclient.exceptions.Forbidden: forbidden'
[21:52:49] <eddiegp>	 So I'd say yes.
[21:53:05] <andrewbogott>	 wait, what?
[21:53:15] <andrewbogott>	 bd808: you're only working on openstack-browser, right?
[21:53:20] <bd808>	 yeah
[21:53:42] <bd808>	 and designate lookups are failing
[21:53:46] <eddiegp>	 Another issue, I just noticed I can't use the 'DNS/Zones' page in Horizon.
[21:54:00] <bd808>	 possibly related
[21:54:20] <eddiegp>	 Designate was the Openstack DNS thing?
[21:54:32] <andrewbogott>	 eddiegp: just noticed, like, how long ago?
[21:54:41] <andrewbogott>	 'cause I fixed that half an hour ago
[21:54:48] <bd808>	 yes, its the API to OpenStack's dns
[21:54:52] <bd808>	 andrewbogott: https://phabricator.wikimedia.org/P6990
[21:55:07] <bd808>	 it was working and then it stopped I think
[21:55:10] <eddiegp>	 andrewbogott: I can reproduce it right now.
[21:55:22] <andrewbogott>	 huh, works for me :(
[21:55:26] * andrewbogott bangs on it a bit harder
[21:55:51] <eddiegp>	 I'll try to re-login, that worked for the instance list and overview earlier
[21:56:41] <andrewbogott>	 eddiegp: what project?
[21:56:51] <eddiegp>	 deployment-prep
[21:57:36] <bd808>	 working for me (fresh session). maybe only fixed for folks with super powers?
[21:57:59] <eddiegp>	 No, logging out, clearing all cookies for horizon.wikimedia.org, logging in again, still doesn't work.
[21:58:09] <andrewbogott>	 eddiegp: what's your username on wikitech?
[21:58:24] <eddiegp>	 I get a red box at the top right saying "Error: Unable to retrieve the zone."
[21:58:27] <eddiegp>	 Twice, actually
[21:58:35] <eddiegp>	 andrewbogott: EddieGP , in that casing
[21:58:46] <eddiegp>	 just 'eddie' as shell name
[21:58:53] <andrewbogott>	 yeah, you're not project admin… so it seems like maybe you shouldn't be able to see those things.
[21:58:55] <bd808>	 the error in openstack-browser is a 403 on requests for zone lookups as the novaobserver user
[21:58:58] <andrewbogott>	 But you could e.g. yesterday?
[21:59:10] <andrewbogott>	 bd808: hm, ok, let me check that
[21:59:13] <eddiegp>	 Yeah, I definitely could before.
[21:59:15] <andrewbogott>	 maybe the read-only stuff broke
[21:59:29] <bd808>	 andrewbogott: that was what I was wondering too
[22:00:23] <andrewbogott>	 hm, the policies didn't change so the code interpreting them must've
[22:00:27] <andrewbogott>	 I'll see what I can do
[22:00:27] <bd808>	 andrewbogott: but it was working at some point today... at 20:40Z (~ an hour ago)
[22:00:46] <andrewbogott>	 that's surprising
[22:01:09] <andrewbogott>	 since the designate API wasn't working at all an hour ago :)
[22:01:25] <andrewbogott>	 it was fixed in https://gerrit.wikimedia.org/r/#/c/426237/
[22:01:35] <bd808>	 heh. maybe it handles that failure mode better....
[22:05:12] <eddiegp>	 I'm under the impression that the read-only things in horizon are mostly an afterthought. First T191433, then I noticed the "Launch instance" interface isn't hidden from me (I can walk through all of selecting a name, image, flavor and it fails at the last step denying permissions as expected) and now this.
[22:05:33] <eddiegp>	 I actually wanted to create a task for the lauch instance thing, but forgot ...
[22:09:48] <andrewbogott>	 eddiegp: it's not too late
[22:10:25] <eddiegp>	 yes yes, I'm already doing it ... ;)
[22:18:41] <eddiegp>	 T192179
[22:18:42] <stashbot>	 T192179: Horizon should not show 'Lauch' button on 'Images' page for non-project admins - https://phabricator.wikimedia.org/T192179
[22:22:48] <andrewbogott>	 bd808, eddiegp any different now?
[22:23:29] <eddiegp>	 andrewbogott: Seems to work now
[22:23:36] <andrewbogott>	 great
[22:24:28] <bd808>	 andrewbogott: yeah. random purge seems good now.
[22:24:35] <andrewbogott>	 (backstory:  They keystone people started using the word "domain" for a completely arbitrary thing, and the designate people were all "wait, we already use the word domain for actual domains" and the keystone people somehow won the fight and now I'll be renaming things for years)
[22:24:59] <bd808>	 heh. zone is more technically correct for DNS, but yea
[22:25:06] <andrewbogott>	 yeah, I guess it is
[22:25:19] <andrewbogott>	 I just hate these Great Arbitrary Renamings
[22:25:28] * bd808 coughs
[22:25:59] <andrewbogott>	 you could make a case that ours is less arbitrary
[22:26:17] <bd808>	 I did! Not everyone was convinced, but so it goes
[22:27:27] * eddiegp tries to remember who posted the comment saying 'A rose by another name ...' somewhere in phab ...
[22:27:34] <eddiegp>	 ;)
[22:27:51] <bd808>	 andrewbogott: the failing novainstances API query that Krinkle was dealing with seems magically fixed now too
[22:28:10] <andrewbogott>	 huh
[22:28:31] <andrewbogott>	 I do my best to ignore/avoid the nova-via-wikitech api so have no idea what that does
[22:28:50] <bd808>	 this made me realize that we may have to put out a deprecation notice about it though
[22:29:11] <bd808>	 It somehow never dawned on me that people were using those api modules
[22:29:12] <andrewbogott>	 that or instruct me to start caring :)
[22:29:13] <Krinkle>	 Confirmed that the openstack-browser API seems reliable now
[22:29:21] <Krinkle>	 But the novainstances API still returns empty array for everything
[22:29:30] <Krinkle>	 Which worked until last week afaik
[22:29:42] <Krinkle>	 but I've already updated nagf today to use openstack-browser's API instead
[22:29:55] <Krinkle>	 https://wikitech.wikimedia.org/w/api.php?format=jsonfm&action=query&list=novainstances&niregion=eqiad&niproject=cvn
[22:30:46] <bd808>	 I guess I need to look at the code. We ripped the local storage of the instance data out a long time ago now
[22:30:51] <andrewbogott>	 Krinkle: I think that's all backed by a giant cache, so it might yet recover
[22:31:05] <andrewbogott>	 oh, ok, I guess maybe there's not a giant cache after all :)
[22:31:28] <bd808>	 list=novaprojects is working now. it was empty when I bookmarked it earlier
[22:31:52] <bd808>	 there is a lot of memcache in the OSM code
[22:32:09] <bd808>	 to keep from hammering the backing OpenStack apis
[22:32:55] * bd808 curses Ryan for OpenStackManager->OSM; OpenStreetMaps->OSM confusion
[22:33:52] <andrewbogott>	 I'm going to take a stab at being done for the day, but have IRC on my phone if new breakages appear.
[22:33:58] <andrewbogott>	 Good evening all!
[22:34:12] <eddiegp>	 andrewbogott: Have a great weekend!
[22:34:13] <bd808>	 cool deal. thanks for all your fixes today andrewbogott 
[23:00:22] <bd808>	 !log tools.csp-report Restart to deploy 5a9fc76 (didn't I do this before?)
[23:00:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.csp-report/SAL