[00:05:11] !log wikistats re-enabling puppet (disabled without reason) on wikistats-wild-tiger [00:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [01:42:48] Good evening. Does anyone know how to submit a job from a webservice? I tried to call jsub but it seems the command is not found like it is from bash. [01:42:56] I cannot run the job directly from the webservice since Labs now kills my script after running for one minute [01:49:11] Return status: 127 [01:49:19] \/usr/bin/jsub -mem 6g -N UploadReport-1607564890-22869 /usr/bin/timeout 1200 /usr/bin/php /data/project/magog/exe/UploadReport.php 20201203000000 --update [01:57:45] if it's a kubernetes webservice, you can't [01:58:00] only the bastions and grid jobs can talk to the grid [02:01:40] OK is there a way to increase the timeout on my webservice then? [02:03:52] otherwise I will need to create a temp file and create a continuous job which polls for the temp file and submits it [08:58:35] !log toolsbeta starting a new etcd instance completely from ansible playbook (etcd-8) (T267412) [08:58:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:58:39] T267412: Provisioning/maintenance scripts - https://phabricator.wikimedia.org/T267412 [09:17:31] !log srwiki-dev removing leaked dns record discordwiki.srwiki-dev.eqiad.wmflabs (clinic duty) [09:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Srwiki-dev/SAL [09:17:55] !log admin removing leaked dns record discordwiki.eqiad.wmflabs (clinic duty) [09:17:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:49:17] !log tools upgrading tesseract on tools-sgewebgrid-lighttpd-0914 (clinic duty) [09:49:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:58:53] !log tools successful tesseract upgrade on tools-sgewebgrid-lighttpd-0914, upgrading the rest of nodes (clinic duty) [09:58:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:44:47] !log tools disabled puppet in all k8s nodes in preparation for version upgrade (T263284) [11:45:07] stashbot ? [11:50:32] !log tools disabled puppet in all k8s nodes in preparation for version upgrade (T263284) [11:50:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:50:37] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [11:50:55] !log tools.stashbot restarted tool, the bot left all IRC channels... [11:51:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [11:56:09] !log admin Freed some space on cloudvirt1024 by running the purge script (T269419) [11:56:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:56:13] T269419: cloudvirt1024: /srv full 99% - https://phabricator.wikimedia.org/T269419 [12:41:57] !log tools set hiera `profile::wmcs::kubeadm::component: thirdparty/kubeadm-k8s-1-17` in project & tools-k8s-control prefix (T263284) [12:42:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:42:01] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [15:32:43] !log tools Upgrading puppet from 4 to 5 on tools-sgebastion-09 (clinic duty) [15:32:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:35:17] !log tools Puppet 5 on tools-sgebastion-09 ran well and without issues, upgrading the other sge nodes (clinic duty) [15:35:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:41:04] !log tools icinga-downtime toolschecker for 2h (T263284) [15:41:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:41:13] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [15:49:57] !log tools puppet upgraded to 5.5.10 on the hosts, ping me if you see anything weird (clinic duty) [15:50:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:50:12] * dcaro stops upgrading things until the kubernetes one ;) [17:00:51] !log paws fixing /etc/kubernetes/kublet.conf and restarting kubelet in paws-k8s-control-1 (T269865) [17:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [17:00:54] T269865: k8s: kubelet.conf embeds cert data in k8s < 1.17 - https://phabricator.wikimedia.org/T269865 [17:11:53] What would you recommend if I wanted to execute the same SQL query across every wiki replica? [17:12:44] (very simple ones) [17:16:15] !log tools k8s control nodes were all upgraded to 1.17, now upgrading worker nodes (T263284) [17:16:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:16:19] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [17:16:59] hare: a for loop? [17:17:09] arturo: but what would I be iterating over [17:17:38] hare: the list of wikis? [17:17:46] I think there's a way to pull up every existing wikimedia wikis but I don't remember. sitematrix or something like that? [17:18:17] Rather than generate my own list I'd prefer to use something that is automatically updated [17:18:33] hare: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Metadata_database might be what you'd need [17:18:54] that looks really interesting, thank you [17:19:09] That's how tools like GUC find all the wikis [17:19:27] I've seen other folks mentioning parsing https://noc.wikimedia.org/db.php but that sounds horrible [17:21:54] Coolest Tool Award 2020 starts in ~23 hours! https://www.youtube.com/watch?v=zYM4k_LD_9w [17:30:53] ^ \o/! [17:35:29] !log tools k8s-control nodes upgraded to 1.17.13 T263284 [17:35:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:35:33] T263284: Upgrade Toolforge K8s to 1.17 - https://phabricator.wikimedia.org/T263284 [17:43:16] parsing https://noc.wikimedia.org/conf/dblists/all.dblist would be slightly less horrible I guess [17:43:29] or another dblist file if you only need one section or somtehing [23:10:21] @seen cyberpower [23:10:22] bd808: I have never seen cyberpower [23:10:41] @seen cyberpower678 [23:10:41] bd808: Last time I saw Cyberpower678 they were changing the nickname to Skynet and Skynet is still in the channel #wikimedia-rename at 11/30/2020 11:54:20 PM (9d23h16m20s ago) [23:13:44] Skynet: hi :) [23:36:42] !log admin cleaned up the logs for haproxy on cloudcontrol1003 by deleting all the gzipped ones and truncating the .1 file [23:36:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL