[08:47:13] !log admin restarting nova-compute on cloudvirt2001-dev after upgrading ceph libraries to 15.2.11 [08:47:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:59:21] !log admin manually force stopping the server exploding-head on codfw, to try cold migration [08:59:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:14:47] !log admin manually force stopping the server puppetmaster-01 to unblock migration [09:14:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:14:54] !log admin manually force stopping the server puppetmaster-01 to unblock migration (in codfw1) [09:14:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:54:40] Hey everyone, I have a question about prometheus in the Cloud VPS. How can I make it scrape metrics from a service we are testing in maps-tile-server01.maps-experiments? There is a public DNS to access the service so you can see the endpoint here https://tegola.wmflabs.org/metrics [09:55:09] I assume that once this is configured properly I can access the metrics from https://grafana-labs.wikimedia.org, is that right? [10:01:12] thesocialdev: we don't have a self-service prometheus server in CloudVPS or any other prometheus-based offering. We do have an internal prometheus server that we use to monitor some basic stuff like VM instances being down, puppet agent failures etc, but only for selected CloudVPS projects [10:01:28] thesocialdev: you will need to deploy your own prometheus server [10:01:59] but I may be wrong, so it would be a good idea to open a phabricator ticket, or send an email to the mailing list. Others may know better [10:09:28] Thanks arturo [10:12:08] you are welcome [10:48:55] !log admin ceph.eqiad: Tweaked the target_size_ratio of all the pools, enabling autoscaler (it will increase cinder pool only) (T273783) [10:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:48:59] T273783: [ceph] disable autoscale and ramp up cinder pool - https://phabricator.wikimedia.org/T273783 [10:51:42] !log admin ceph.eqiad: cinder pool got it's pg_num increased to 1024, re-shuffle started (T273783) [10:51:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:00:13] !log admin codfw.openstack cloudvirt2001-dev back online, taking cloudvirt2002-dev out to upgrade ceph libraries (T280641) [13:00:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:00:17] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [13:07:41] !log admin codfw.openstack cloudvirt2002-dev done, taking cloudvirt2003-dev out to upgrade ceph libraries (T280641) [13:07:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:07:45] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [14:10:40] !log admin codfw.openstack upgraded ceph libraries to 15.2.11 (T280641) [14:10:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:10:45] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [14:21:32] How does one increase the resources for the kubernetes pod which runs a web tool? Cf. https://github.com/ms609/citation-bot/blob/master/.lighttpd.conf [14:22:52] Nemo_bis: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Quotas_and_Resources check if this works [14:33:12] Hm, right. So I can see the current config of the webservice pod with e.g. kubectl get pod -o json oabot-5d788d549b-m5gp7 [14:35:34] Is it supposed to agree with ~/service.manifest ? [14:49:20] arturo: seems to work, thanks! How I checked: [14:49:20] $ kubectl get pod -o jsonpath={.spec.containers[].resources} oabot-d6dd4fb9c-bb46h ; echo [14:49:23] map[limits:map[cpu:500m memory:512Mi] requests:map[cpu:150m memory:256Mi]] [14:49:36] $ webservice stop ; webservice --backend=kubernetes --mem 1G --cpu 1 python2 start [14:49:43] $ kubectl get pod -o jsonpath={.spec.containers[].resources} oabot-6f594f7d9-z8jsd ; echo [14:49:46] map[limits:map[cpu:1 memory:1G] requests:map[cpu:500m memory:500M]] [14:50:07] Nemo_bis: 👍 [14:50:16] also, I strongly suggest you move away from python2 [14:50:19] :-P [14:50:36] why, python3 is only 15 years old or something [14:50:41] I'll wait for the age of consent ^_^ [14:51:02] wait for python4 [14:51:09] or python7, as for PHP [14:51:47] jokes aside, we may completely drop python2 support some day soon [14:52:01] debian bullseye won't have any python2 [14:52:39] yeah I know [14:52:52] I actually have a branch for python3 in this tool but but but [14:53:01] :-) [14:55:36] arturo: do you think it would be worth adding the example above to the help page? for me "kubectl get" is always a mystery [14:55:53] Nemo_bis: sure [14:56:00] ok I'll do it this evening, thanks [14:58:24] arturo: what does the change in bullseye mean for #! lines? Will #!/usr/bin/python invoke python3? Will /usr/bin/python3 still exist? [14:58:47] both I think :) [14:59:05] andrewbogott: I have no idea (yet). I think `/usr/bin/python` has been discouraged for years [14:59:15] (regardless of py2 vs py3) [14:59:19] arturo: oh? In favor of what? [14:59:27] python2 or python3? [14:59:36] python3 [14:59:41] well yeah :) [14:59:50] I think the nice thing to do today is to specify the version in the shebang [15:00:06] sure [15:00:07] actually, imo #!/usr/bin/env pythonX would be the most portable [15:00:17] yeah, or env ^^^ [15:00:56] but I wonder about what the world of existing #!/usr/bin/python things will do on bullseye... They could fail explicitly with a "fix your script" warning or they could just silently run on python3 and just break or fail at random points... [15:01:22] Seems like the former might be better, which implies installing an intentionally broken /usr/bin/python rather than just linking it to python3 [15:01:32] Fedora switched /usr/bin/python to /usr/bin/python3 and AFAIK nobody died [15:01:39] (or did I miss something?) [15:01:50] Yeah, I tripped over a system recently that was set up that way, I think it was maybe alpine? [15:02:10] I'm glad I don't have to deal with that migration [15:02:12] So probably that's the consensus approach, regardless of what I might think :) [15:02:23] We're ALL going to have to deal with it pretty soon [15:02:40] I mean, from the owner of python perspective :-) [15:02:46] oh yeah [15:03:11] I assume that in 2007 they wrote a 15-year migration timeline and have stuck with it [15:03:16] :D [15:04:58] that would be actually pretty impressive :} [15:06:07] What debian release will discontinue ipv4 support? [15:07:13] after redhat :-) [15:07:18] so we can share the blame [15:08:22] it is 2021 and my home ISP doesn't have IPv6 even in the upstream backbone. [15:08:46] mine does not either [15:08:57] but 6in4 tunnels are fun! :) [15:09:38] there will be a tesla car in mars and I wont have IPv6 [15:10:40] bd808: I'm sorry I don't do network engineering in my spare time :-D [15:11:59] seems both my home isps have IPv6, though I haven't bothered to setup ipv6 in the home lan [15:12:10] that sounds like a good work/life separation. :) I did the hard work for mine in 2011 in preparation for World IPv6 Day... which obviously changed very little. [15:20:59] sounds like today is a bad day to poke you cloudy folks about things ;) [15:21:35] !log tools rebooting tools-sgebastion-07 [15:22:18] !log tools rebooting tools-sgebastion-08 and 09 [15:22:25] (no stashbot yet) [15:22:34] I can try :) [15:22:52] That suggests the whole grid is likely down still [15:27:54] * chicocvenancio following the discussion on telegram, confused the channel wen quiet 8 minutes ago [15:29:19] NFS is still mostly broken (and, therefore, the grid). We're working on it but the cause of the problem is not yet clear [15:33:54] I've also shut down the bastions to prevent local disk from filling [16:16:27] !log tools cleared E status on grid queues to get things flowing again [16:16:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:18:44] * Nemo_bis sees oabot working fine [16:40:42] !log tools deleted all the errored out grid jobs stuck in queue wait [16:40:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:41:09] bd808: my friends and I will be driving through southern Idaho while traversing the massive desert that is the eastern half of the Pacific Northwest. They want to know if there are any, specifically, tacky tourist attractions they could visit. [19:21:08] harej: both tacky and interesting -- http://www.shoshoneicecaves.com/ [19:22:46] They also have this preoccupation with eating Idaho potatoes [19:25:52] harej: https://www.airbnb.com/rooms/32011367 [19:26:34] ...Wow [19:46:25] By the way, having driven through Idaho a few times now, I enjoy it a lot, it's nice to drive through and it is like being on vacation from the rest of civilization [19:46:39] Coeur d'Alene is very pretty to drive through