[11:19:41] !log admin [codfw1dev] rebooting labtestvirt2003 [11:19:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:56:09] hi i have a problem with tool etytree [13:56:20] the website says 404 not found [13:56:44] it's my tool and I', trying to restart the webservice [14:03:00] !log admin draining cloudvirt1022 so I can rebuild it with backy support [14:03:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:13:04] epantaleo: that means the web server couldn't find the requested files [14:13:41] but I didn't move anything [14:15:35] arturo: i need nodejs could that be the problem? some server settings changed? [14:16:21] epantaleo: is this problem in the toolforge tool or your CloudVPS project? [14:16:43] I believe it'is in the toolforge tool [14:16:59] did you check the logs? [14:18:41] arturo: i the error.log I don't see anything helpful [14:18:58] epantaleo: what about `kubectl logs` [14:19:26] if i type that i get no such file or directory [14:19:58] !log admin rebuilding cloudvirt1022 with backy support [14:20:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:20:53] arturo: sorry I am no expert, but I remember a while ago, but I don't remember exactly why, they recommended I do url.rewrite-if-not-file += ( "(.*)" => "/etytree/index.html" ) [14:21:01] maybe this can help? [14:22:55] what about this? [14:22:57] https://www.irccloud.com/pastebin/a9jamPOY/ [14:24:45] I am not sure, I didn't change anything [14:28:04] epantaleo: I think I fixed it for you. Your $HOME/.lighttpd.conf needed to be updated for https://wikitech.wikimedia.org/wiki/News/Toolforge.org [14:28:22] !log tools.notwikilambda bumped update deployment to 3 GiB RAM [14:28:24] thanks bd808 ! [14:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL [14:28:47] arturo: great works now! [14:28:56] epantaleo: the fix was `url.rewrite-if-not-file += ( "(.*)" => "/index.html" )` -- removing the tool name from the redirect [14:29:25] err... rewrite [14:30:02] epantaleo: If you are not subscribed to https://lists.wikimedia.org/mailman/listinfo/cloud-announce, now would be a good time to do that :) [14:30:28] arturo: awesome! thanks a lot. the problem is I am not sure I can understand much of those messages [14:31:10] I'm glad bd808 (bryan) found the issue so fast! [14:31:13] We try to give warning when we make breaking changes, but they do happen. [14:31:34] bd808: thanks a lot ! [14:33:12] I opened a task https://phabricator.wikimedia.org/T265363. should I close it? [14:38:01] epantaleo: yes please [14:38:35] perhaps you can put some hint message in the task before closing it [14:42:59] done thanks [15:22:56] !log admin draining cloudvirt1021 so I can rebuild it with backy support [15:22:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:33:50] According to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Running_Locally it is possible to run docker's locally. [15:34:35] Running the command as given returns "permission denied" [15:34:41] acre:[/home/greenc] docker run --name toolforge -p 8888:80 -v "${PWD}:/var/www/html:cached" -d docker-registry.tools.wmflabs.org/toolforge-php73-sssd-web sh -c "lighty-enable-mod fastcgi-php && lighttpd -D -f /etc/lighttpd/lighttpd.conf"docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: [15:34:41] Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/containers/create?name=toolforge: dial unix /var/run/docker.sock: connect: permission denied. [15:35:27] GreenC: that sounds like issues with your local Docker setup? [15:36:02] Could be I don't enough about docker to say. I am able to run dockers from other sites. [15:42:26] GreenC: that command works locally for me -- https://phabricator.wikimedia.org/P12976 -- your error message sounds a bit like you do not have your docker service setup locally to work for non-root users. [15:42:32] https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user might help [15:45:59] ahh duh I'm logged into a term that is old and it doesn't reflect the docker group I added a few weeks ago..just needed to log out and in.. thank you [15:46:50] nice find. that sort of thing has made me waste too much time in the past :) [15:49:39] @@bd808 thanks for your quick help! [15:49:50] yw [17:53:58] !log rebuilding cloudvirt1021 for backy support [17:53:59] andrewbogott: Unknown project "rebuilding" [17:54:06] !log admin rebuilding cloudvirt1021 for backy support [17:54:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [22:53:30] is there a way to get SHOW EXPLAIN FOR to work more reliably? [22:54:01] AntiComposite: https://sql-optimizer.toolforge.org/ usually works [22:54:16] that's having the same issue for me [22:54:26] SHOW EXPLAIN failed: Unknown thread id: 95/100 times [22:54:33] it is a racy sort of system unfortunately [22:54:58] hmmm... yeah maybe something funky is going on right now? [22:57:55] sometimes I can wait and/or try again and it will work [22:58:14] but for others, it won't start working [22:59:17] like https://quarry.wmflabs.org/query/49019 (https://quarry.wmflabs.org/query/49028) and https://quarry.wmflabs.org/query/49032, (https://quarry.wmflabs.org/query/49039) [23:01:07] I don't know how the sql-optimizer tool works exactly, but one guess is that if it opens 2 sql connections there may be a "luck of the draw" issue with the load balancer in front of the wiki replica cluster sending both connections to the same wiki replica instance [23:05:10] AntiComposite: tailing the logs in the tool doesn't help me understand any better :/ [23:05:49] I can see it print the query and the connection id in the same way for the ones that work and the ones that fail to find the thread [23:07:34] ok, looking at the source I can see that it does open 2 connections (one to run the query and one to run the explain). My hunch is that this is a load balancer caused problem [23:09:26] AntiComposite: I think this is worth a bug report. I'm not sure there is a quick fix, but having the potential problem with the load balancing layer documented would be useful [23:10:16] I'll type one up [23:22:41] https://phabricator.wikimedia.org/T265430 [23:24:13] thanks, I'll add some info about why I think this could be the load balancer