[01:20:54] <wm-bot>	 !log tools.pbbot <peterbowman> Deploy 465b0a8: support new and legacy Toolforge URL schemes
[01:20:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL
[12:54:38] <CustosLimen>	 hi
[12:54:48] <CustosLimen>	 so I did the steps here: https://tools.wmflabs.org/admin/
[12:54:57] <CustosLimen>	 including https://toolsadmin.wikimedia.org/tools/membership/status/795
[12:55:08] <CustosLimen>	 "Status: Approved"
[12:55:20] <CustosLimen>	 Now if I click on: https://toolsadmin.wikimedia.org/tools/create
[12:55:33] <CustosLimen>	 I get "You are not currently authorized to view the URI /tools/create."
[12:57:46] <Majavah>	 CustosLimen: log out and back in
[12:58:31] <CustosLimen>	 ah, thanks
[13:02:29] <CustosLimen>	 So I'm trying to get this to work: https://www.mediawiki.org/wiki/OAuth/For_Developers#OAuth_2
[13:02:35] <CustosLimen>	 I dunno if this is right place to ask
[13:02:38] <CustosLimen>	 but I dunno where else
[13:02:55] <CustosLimen>	 But it does not work: https://www.mediawiki.org/wiki/Topic:Vmwiwu073xo33eh5
[13:03:08] <CustosLimen>	 "Application Connection Error: Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method)"
[17:45:47] <Jayprakash12345>	 !log tools.indic-wscontest Migrate indic-wscontest to new toolforge URL (T253456)
[17:45:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-wscontest/SAL
[17:45:50] <stashbot>	 T253456: Migrate indic-wscontest to new toolforge URL - https://phabricator.wikimedia.org/T253456
[21:25:34] <danilo>	 tools-sgebastion-07 is very slow
[21:30:44] <Krenair>	 looking
[21:31:41] <Krenair>	 load avg is around 10
[21:36:12] <Krenair>	 doesn't look like anything particularly CPU heavy is going on
[21:36:46] <Krenair>	 lots of iowait sometimes
[21:38:24] <Krenair>	 forgotten exactly what I'm looking at but nfsiostat looks... ok...
[21:40:15] <danilo>	 I am trying to edit a small file in vim, and it takes a long time only to open the file
[21:44:04] <Krenair>	 something's up with this VM and it's not immediately obvious to me what
[21:48:17] <Krenair>	 looks like whatever this is started around 1845
[21:48:38] <Krenair>	 am looking at https://grafana-labs.wikimedia.org/d/7fTGpvsWz/toolforge-vm-details?orgId=1&var-VM=tools-sgebastion-07
[21:58:58] <Krenair>	 er, 1845 is BST
[21:59:07] <Krenair>	 https://prometheus.wmflabs.org/cloud/graph?g0.range_input=6h&g0.expr=node_load1%7Binstance%3D%22tools-sgebastion-07%22%7D&g0.tab=0 shows 1745 UTC
[22:01:58] <danilo>	 now it is fast again
[22:02:27] <Krenair>	 yeah whatever was going on has stopped
[22:05:56] <Krenair>	 looking at the graph, this has happened earlier for a while but then it resumed
[22:47:21] <AmandaNP>	 So I am on debian 9.6 and trying to do an apt-get upgrade and it's asking me to downgrade a lot of packages that are forcing 404s
[22:48:19] <AmandaNP>	 47 packages it wants to downgrade
[23:17:51] <AmandaNP>	 also is there an automatic backup of cloud VPS instances, or is that our responsibility?
[23:22:56] <Krenair>	 AmandaNP, that's your responsibility
[23:23:07] <AmandaNP>	 ok, fair enough
[23:23:12] <Krenair>	 AmandaNP, what instance is this?
[23:23:28] <AmandaNP>	 it was utrs-database2
[23:27:16] <Krenair>	 it looks like your package downgrades (or at least the couple I checked) relate to a special wikimedia apt component - jessie-wikimedia/openstack-mitaka-jessie
[23:28:23] <Krenair>	 not sure if that component's existence still makes sense actually
[23:28:36] <Krenair>	 heh
[23:28:47] <Krenair>	 ooh
[23:28:50] <Krenair>	 apt-get update fails
[23:28:56] <Krenair>	 Err:6 http://apt.wikimedia.org/wikimedia jessie-wikimedia InRelease
[23:28:56] <Krenair>	   The following signatures were invalid: DB3DC2BD4CD504EF2D908FC509DBD9F93F6CD44A
[23:34:30] <Krenair>	 ah yes removed in https://gerrit.wikimedia.org/r/c/operations/puppet/+/593223
[23:35:42] <Krenair>	 and removed from clients puppet catalogs in https://gerrit.wikimedia.org/r/c/operations/puppet/+/585814/1/modules/openstack/manifests/commonpackages/vms/mitaka.pp - but not absented
[23:39:25] <Krenair>	 looks like /etc/apt/sources.list.d/openstack-mitaka-jessie.list still exists across 106 cloud VPS instances
[23:41:04] <Krenair>	 !log utrs Manually removed /etc/apt/sources.list.d/openstack-mitaka-jessie.list on utrs-database2 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/585814/1/modules/openstack/manifests/commonpackages/vms/mitaka.pp (missing absent)
[23:41:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL
[23:41:16] <Krenair>	 AmandaNP, apt-get upgrade should look better now
[23:41:37] <AmandaNP>	 ok, if this happens on another instance, what can I do that you just did?
[23:41:54] <Krenair>	 yeah
[23:42:28] <Krenair>	 might also happen on utrs-beta-production and utrs-production2
[23:43:06] <AmandaNP>	 yep, i'm just wondering what you did though. I'm not too familiar with the game of apt
[23:43:40] <Krenair>	 just removed one of its sources that probably should not have been there for a while
[23:44:39] <Krenair>	 I think the component was there to bring jessie-like packages forward into stretch instances and pin them at a higher priority than the stretch ones, providing compatibility with the older OpenStack Mitaka APIs that were in use until more recently
[23:44:43] <Krenair>	 or something along those lines
[23:45:31] <Krenair>	 I think some code that runs on all wikimedia puppet clients relies on connecting to the OpenStack APIs so it ends up with a bunch of dependencies like this