[00:59:41] !log tools Live hacked the "nginx-configuration" ConfigMap for T245426 (done several hours ago, but I forgot to !log it) [00:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:59:45] T245426: Earwig's copyvio tool 504 gateway time-out issues - https://phabricator.wikimedia.org/T245426 [01:43:27] bd808: Things still aren't looking good.. [01:43:50] I've had a copyvio query running for well over 10 minutes.. They usually finish in a minute or two max. [01:44:01] Plus, reFill is hung at "Waiting for an available worker". [01:53:36] login toolforge seems to be running a little slower than usual, also [02:23:46] Yeah, it seems like the refill-api Celery worker is down [02:27:52] I believe last time someone restarted it with kubectl, but I don't remember. [02:28:08] Cyberpower678, could you poke at refill-api when you get a chance? [02:55:06] for setting the default backend for a tool to kubernetes, would I just edit service.manifest ? and what value? [03:02:01] guess that's now how you do it, it just overwrote it again after starting the webservice with no arguments ... [03:02:08] not * how [03:13:44] never mind, think I found it [03:23:12] ok so I got the backend setting to work in the rc file ... but - how do I set the default type? [03:25:10] tried using "type=php7.3" in the file, but that didn't work [11:50:34] !log tools fix invalid yaml format in horizon puppet prefix 'tools-k8s-haproxy' that prevented clean puppet run in the VMs [11:50:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:48:15] !log codesearch restarting hound_proxy to pick up the new changes [14:48:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [15:22:25] hi, I need to remove a VM puppet certificate from the labs puppetmaster, is there an SOP or just 'puppet cert clean' on the right (which?) host(s) ? [15:25:49] godog: yeah, `puppet cert clean` on cloud-puppetmaster-01.cloudinfra.eqiad.wmflabs [15:26:22] jeh: thanks! will do [15:26:53] out of curiosity is there a cleanup process for nonexistant VMs certs and I just raced it by deleting and recreating the same VM ? [15:27:38] there's a cleanup process, it usually works fine but I think there are some race conditions (like you found) [15:29:17] ack, thanks! yeah I figured delete + create in horizon in short succession did it [15:35:22] jeh: looks like in this case the instance can't recover by itself if the puppetmaster refused the cert and puppet failed, even on reboot, expected? if not I'm happy to open a task if there isn't one already [15:35:35] on first run that is [15:36:10] when that happens you'll need to delete the VM, clean the old certs, then recreate it. The firstboot process wont re-run on an existing VM [15:36:52] !log tools setting 'puppetmaster: tools-puppetmaster-02.tools.eqiad.wmflabs' on tools-sgeexec-0942 to test new puppetmaster on grid T245365 [15:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:36:55] T245365: Replace tools-puppetmaster-01 (jessie) with a buster puppetmaster - https://phabricator.wikimedia.org/T245365 [15:37:14] indeed, deleted the vm and recreated it and it is back now, thanks1 [15:39:07] so does anyone have an answer from my question about what option to use in the webservice configuration file to set the service type? [15:40:34] neither type, nor adding it to the --backend= line works [15:46:01] DSquirrelGM: I don't think you can set the default type in webservicerc, looks like it's only a command line argument `webservice --backend=kubernetes php7.3 ` [15:58:37] ok so let me put it this way then - why is the builtin default still being set to php5.6 if it's deprecated? [19:11:28] DSquirrelGM: to avoid breaking things that rely on the default. It'll get changed as part of the deprecation process [19:14:59] beginning to think it'd be easier just to set up the start line in a script [22:05:02] !log tools Project-wide hiera change to swap puppetmaster to tools-puppetmaster-02 T245365 [22:05:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:05:05] T245365: Replace tools-puppetmaster-01 (jessie) with a buster puppetmaster - https://phabricator.wikimedia.org/T245365 [23:34:16] is it possible for me to create a tool that invokes ia-upload to import something to wikimedia commons, then does something else? [23:34:28] or does the wiki user have to go to ia-upload themselves, then go to my tool? [23:48:32] guess I can live with 2 extra files and using sh and the filename for webservice start/stop