[00:12:04] musikanimal: that url reads to me as w sex port [00:27:27] Zppix: you are not the first or last to parse "W(iki)S(ource)Export" in that way [00:27:54] bd808: i mean they kinda ask for it [00:28:20] naming is hard and I would pretty much guarantee the original author did not make that connection [00:28:27] heh [00:29:56] we are pretty globally bad at naming in this movement -- https://www.mediawiki.org/wiki/Wikimpedia [00:50:31] !log tools rebuilt all docker images to include webservice 0.64 [00:50:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [00:50:49] naming is hard [01:20:24] and coming up with unique usernames - even more so [01:22:29] speaking of - seems to be pretty much back to normal on activity now - so can probably go ahead and change the status message [01:33:15] maybe apg would generate unique usernames... it's a password generator but I guess it could be used for usernames as well [01:50:26] was making a joke about what we were dealing with earlier today, zhuyifei1999_ [01:51:01] ok? [01:51:12] but that was no laughing matter [02:00:21] one thing I've been wondering about lately - why does wikitech have Special:BotPasswords disabled? [03:40:52] DSquirrelGM: probably just never turned on. https://www.mediawiki.org/wiki/OAuth/Owner-only_consumers is a better way to run a bot though than using the hack of BotPasswords and there are honestly very few active bots on wikitech. [03:41:44] DSquirrelGM: oh, actually I bet it is related to LDAPAuthentication [03:42:41] I meant more for AWB or PWB editing tasks, than continuous monitoring/fix jobs [03:43:21] if PWB is pywikibot then its better to use owner only OAuth tokens for that [03:44:08] BotPasswords are honestly a bit of a hack that we had to implement along with the AuthManager refactor [03:44:30] they really are for legacy code that for whatever reason can't be updated to use OAuth [03:44:48] (I only use AutoWikiBrowser for automated edit tasks) [03:45:31] just seemed rather odd to have that considering how many other types of bots are routinely used [03:46:37] If AWB doesn't support Owner-only OAuth then we should create a task for the upcoming hackathon. :) Magioladitis can do it or find someone else to. [03:48:59] !log tools Joined tools-k8s-worker-36 to 2020 Kubernetes cluster [03:48:59] also, even if it IS supported in AWB, last I heard it required multiple extensions and other account setup procedures to use [03:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [03:49:51] the OAuth I mean [03:50:48] It is a mediawiki extension, yes. And yes it requires filling out the form at https://wikitech.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose?wpownerOnly=1 [04:06:16] !log tools Joined tools-k8s-worker-37 to 2020 Kubernetes cluster [04:06:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:07:20] !log tools Joined tools-k8s-worker-38 to 2020 Kubernetes cluster [04:07:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:08:27] !log tools Joined tools-k8s-worker-39 to 2020 Kubernetes cluster [04:08:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:09:21] !log tools Joined tools-k8s-worker-40 to 2020 Kubernetes cluster [04:09:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:10:49] !log tools Joined tools-k8s-worker-41 to 2020 Kubernetes cluster [04:10:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:12:17] !log tools Joined tools-k8s-worker-42 to 2020 Kubernetes cluster [04:12:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:13:24] !log tools Joined tools-k8s-worker-43 to 2020 Kubernetes cluster [04:13:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:14:28] !log tools Joined tools-k8s-worker-44 to 2020 Kubernetes cluster [04:14:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:14:37] The intro here could probably use a bit of a rewrite, but not entirely sure if this is what's intended - "to different use-cases" -> "for different project types" - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Host_aggregates [04:16:52] project types isn't quite right. it really is use-cases. The special ones right now are also related to particular projects, but that is coincidental [04:18:22] a more common use-case in the future is likely to be related to disk IO speed [04:19:48] !log tools Joined tools-k8s-worker-45 to 2020 Kubernetes cluster [04:19:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:21:05] !log tools Joined tools-k8s-worker-46 to 2020 Kubernetes cluster [04:21:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:21:59] !log tools Joined tools-k8s-worker-47 to 2020 Kubernetes cluster [04:22:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:23:22] !log tools Joined tools-k8s-worker-48 to 2020 Kubernetes cluster [04:23:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:24:22] !log tools Joined tools-k8s-worker-49 to 2020 Kubernetes cluster [04:24:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:28:02] !log tools Joined tools-k8s-worker-50 to 2020 Kubernetes cluster [04:28:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:31:38] !log tools Joined tools-k8s-worker-51 to 2020 Kubernetes cluster [04:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:32:39] !log tools Joined tools-k8s-worker-52 to 2020 Kubernetes cluster [04:32:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:34:05] !log tools Joined tools-k8s-worker-53 to 2020 Kubernetes cluster [04:34:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:35:11] !log tools Joined tools-k8s-worker-54 to 2020 Kubernetes cluster [04:35:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [04:53:38] !log tools.stashbot Restarting bot to migrate back to the 2020 Kubernetes cluster [04:53:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [05:46:50] !log tools.sammour Stopped webservice stuck in CrashLoopBackoff state. It doens't look like this ever actually was working. [05:46:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sammour/SAL [05:55:21] !log tools.entsuggest Deleted custom Kubernetes deployment. Created 1 year ago and creating pods with a non-existent entry point which always end in CrashLoopBackoff state [05:55:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.entsuggest/SAL [06:44:46] !log tools.simplewd Fixed webservice stuck in CrashLoopBackoff due to webservice bug [06:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.simplewd/SAL [06:51:16] !log tools.speedpatrolling Hard restart of webservice to fix incorrect python runtime version [06:51:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.speedpatrolling/SAL [07:07:10] !log tools.unpkg Stopped broken webservice; edited package.json to have a valid entrypoint, restarted with proper node10 runtime. Still needs some work to fix redirect to invalid URL. [07:07:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.unpkg/SAL [14:20:01] !log tools create new puppet prefixes for existing (no change in data) and new elasticsearch VMs [14:20:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:09:37] !log tools create 3 new elasticsearch VMs tools-elastic-[1,2,3] T236606 [15:09:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:09:41] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [15:28:48] !log tools create OpenStack server group tools-elastic with anti-affinty policy enabled T236606 [15:28:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:28:51] T236606: Rebuild Toolforge elasticsearch cluster with Stretch or Buster - https://phabricator.wikimedia.org/T236606 [15:32:01] Hi, [15:33:01] hello anmolwassan [15:34:41] I need some help. I am trying to deploy a next.js and node.js based tool on toolforge. The problem with next.js is it sets up cache (_next) directory in root and there is no way to disable it unless I have a nginx server set up. The url for the tool is: https://tools.wmflabs.org/bub2/ [15:37:20] Open developer tools > Go to Network tab > Refresh the page. You will see it gives 404 for every file that is served. I suspect next.js is trying to write in the root directory, as you can see in the network logs and since it doesn't have permission to do so, the files were never written, so it gives a 404 error. [15:38:10] Been scratching my head over this for 2 days now. Any help/lead on this will be really appreciated. Thank you! [15:42:34] anmolwassan: I am not sure I completely understand the problem. Can you explain more about what an nginx server would do to fix the problem you are having? [15:43:05] Your tool is running on the 2020 Kubernetes cluster. One thing that adds which we have not really documented on wikitech yet is an Ingress object. That Ingress object is configuration for an nginx reverse proxy that sites between your tool and the internet. So maybe we can use it to solve the problem if it is a matter of some rewrites or something like that. [15:44:44] bd808: Thanks a lot for replying. I read an article on internet that mentions deploying the next js app on a different path other than root. This is the article: https://levelup.gitconnected.com/deploy-your-nextjs-application-on-a-different-base-path-i-e-not-root-1c4d210cce8a [15:46:20] Can you help with the ingress object? I need to setup nginx reverse proxy. [15:46:25] ah! I understand now. This may surprise you because it is also an unannounced feature, but try https://bub2.toolforge.org/ [15:46:56] Whaaat? OMG. [15:47:05] I can see this live. But how? [15:47:41] it's set up - just not ... announced publicly yet [15:47:44] This is part of what the new Ingress objects are enabling. I have some early adopter information on how to make the old URL redirect here too -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Redirect_requests_for_tools.wmflabs.org/my-tool_to_my-tool.toolforge.org [15:49:51] Alright. Got it. This is really cool. I will keep this URI lowkey for now. Just to clarify, I can test using this url and it won't cause any problems, right? [15:50:03] anmolwassan: so there are two different things that you could do: 1) apply custom annotations to the bub2-legacy to redirect to the bub2.toolforge.org URL; 2) apply custom annotations to bub2-legacy to do the rewrites that the next.js framework expects for path based routing [15:52:01] anmolwassan: yes. It is fine to use this url and advertise it to people. :) I have been playing with the new domain in preparation for documenting and announcing it. Try https://tools.wmflabs.org/bash/ to see a tool that has been setup to redirect folks to the new domain. [16:00:20] !log tools Devoicing stashbot in #wikimedia-cloud to reduce irc spam while migrating tools to 2020 Kubernetes cluster [16:00:31] !log tools.admin Running k8s-2020-migrate.sh with prefix=q [16:00:55] !log tools.qrcode-generator Migrated to 2020 Kubernetes cluster [16:01:07] bd808: is it possible to run webserver using a non-default public_html folder? [16:01:18] !log tools.quarrybot-enwiki Migrated to 2020 Kubernetes cluster [16:01:27] hauskatze: symlinks work for that [16:01:41] !log tools.query Migrated to 2020 Kubernetes cluster [16:01:57] so ln from ~/public_html to the "right place" would do the trick right? [16:01:58] I have a couple of tools where $HOME/public_html is a symlink to something like $HOME/git_clone/public [16:02:05] yeah, it should work fine [16:02:05] !log tools.quickpreset-migrate Migrated to 2020 Kubernetes cluster [16:02:19] bd808: thanks a lot, I'll try that [16:04:01] !log tools.admin Running k8s-2020-migrate.sh with prefix=r [16:04:24] !log tools.railways Migrated to 2020 Kubernetes cluster [16:04:35] !log tools.random-featured Migrated to 2020 Kubernetes cluster [16:04:58] !log tools.rangeblockfinder Migrated to 2020 Kubernetes cluster [16:05:09] !log tools.rang Migrated to 2020 Kubernetes cluster [16:05:20] !log tools.rank Migrated to 2020 Kubernetes cluster [16:05:34] !log tools.raun Migrated to 2020 Kubernetes cluster [16:05:46] !log tools.readmore Migrated to 2020 Kubernetes cluster [16:06:02] !log tools.recitation-bot Migrated to 2020 Kubernetes cluster [16:06:12] !log tools.redirtest Migrated to 2020 Kubernetes cluster [16:06:36] !log tools.refill-api Migrated to 2020 Kubernetes cluster [16:06:47] !log tools.refill Migrated to 2020 Kubernetes cluster [16:06:57] !log tools.refswikipedia Migrated to 2020 Kubernetes cluster [16:07:20] !log tools.remarkup2wikitext Migrated to 2020 Kubernetes cluster [16:07:45] !log tools.reviewtools Migrated to 2020 Kubernetes cluster [16:08:08] !log tools.rfastats Migrated to 2020 Kubernetes cluster [16:08:31] !log tools.ricordisamoa Migrated to 2020 Kubernetes cluster [16:08:54] !log tools.ri-diff-fixture-updater Migrated to 2020 Kubernetes cluster [16:09:11] !log tools.rightstool Migrated to 2020 Kubernetes cluster [16:09:21] !log tools.rm-stats Migrated to 2020 Kubernetes cluster [16:09:34] !log tools.rmstats Migrated to 2020 Kubernetes cluster [16:09:45] !log tools.robin Migrated to 2020 Kubernetes cluster [16:09:56] !log tools.roundtripping Migrated to 2020 Kubernetes cluster [16:10:19] !log tools.ruarbcom-js Migrated to 2020 Kubernetes cluster [16:10:30] !log tools.ruarbcom Migrated to 2020 Kubernetes cluster [16:10:57] !log tools.rxy Migrated to 2020 Kubernetes cluster [16:11:21] Thank you bd808. Will look into this. Finally happy to see it live! :) [16:12:21] anmolwassan: I would hope that by the end of March there will be good documentation and better tools for using the *.toolforge.org hostnames. Glad to see this helped with your project! [16:14:01] !log tools rebooted tools-k8s-worker-7 to clear some puppet issues [16:20:52] !log tools.admin Running k8s-2020-migrate.sh with prefix=s [16:21:02] !log fastcci removed dump and scratch mounts from /etc/fstab and unmounted on all instances. T208404 [16:21:17] !log tools.sau226test Migrated to 2020 Kubernetes cluster [16:21:40] !log tools.scholia Migrated to 2020 Kubernetes cluster [16:22:03] !log tools.scribe Migrated to 2020 Kubernetes cluster [16:22:22] !log tools.sdbot Migrated to 2020 Kubernetes cluster [16:22:34] !log tools.section-links Migrated to 2020 Kubernetes cluster [16:22:56] !log tools.secwatch Migrated to 2020 Kubernetes cluster [16:23:09] !log tools.serviceawards Migrated to 2020 Kubernetes cluster [16:23:31] !log tools.sge-jobs Migrated to 2020 Kubernetes cluster [16:23:43] And the wikitech registration spammer is back :/ [16:23:53] !log tools.shexia Migrated to 2020 Kubernetes cluster [16:24:03] !log tools.shex-simple Migrated to 2020 Kubernetes cluster [16:24:25] !log tools.shextranslator Migrated to 2020 Kubernetes cluster [16:24:36] !log tools.shields Migrated to 2020 Kubernetes cluster [16:24:45] !log tools.shortnames Migrated to 2020 Kubernetes cluster [16:24:51] ugh [16:24:54] !log tools.shorturls Migrated to 2020 Kubernetes cluster [16:25:04] !log tools.sibu Migrated to 2020 Kubernetes cluster [16:25:13] !log tools.sibutest Migrated to 2020 Kubernetes cluster [16:25:23] !log tools.similarity Migrated to 2020 Kubernetes cluster [16:25:45] !log tools.simplewd Migrated to 2020 Kubernetes cluster [16:25:58] !log tools.sistercities Migrated to 2020 Kubernetes cluster [16:26:20] !log tools.slow-parse Migrated to 2020 Kubernetes cluster [16:26:31] !log tools.smv-description-translations Migrated to 2020 Kubernetes cluster [16:26:41] !log tools.snapshots Migrated to 2020 Kubernetes cluster [16:26:56] !log tools.sonarqubebot Migrated to 2020 Kubernetes cluster [16:27:18] !log tools.sowhy Migrated to 2020 Kubernetes cluster [16:27:23] [17:23:43] And the wikitech registration spammer is back :/ <-- let me know if I can help [16:27:40] !log tools.spacemedia Migrated to 2020 Kubernetes cluster [16:27:50] !log tools.spdx Migrated to 2020 Kubernetes cluster [16:27:59] !log tools.speed-patrolling Migrated to 2020 Kubernetes cluster [16:28:12] !log tools.speedpatrolling Migrated to 2020 Kubernetes cluster [16:28:21] !log tools.sphinxcapt-leaderboard Migrated to 2020 Kubernetes cluster [16:28:22] hauskatze: Let's talk in PM just in case the jerk is lurking here [16:28:42] !log tools.spiarticleanalyzer Migrated to 2020 Kubernetes cluster [16:28:53] !log tools.sqid Migrated to 2020 Kubernetes cluster [16:29:12] bd808: I had umode+g, but you should be able to PM now if you want [16:29:16] !log tools.sql-optimizer Migrated to 2020 Kubernetes cluster [16:29:24] !log tools.squirrelnest-upf Migrated to 2020 Kubernetes cluster [16:29:42] !log tools.srwiki Migrated to 2020 Kubernetes cluster [16:29:53] !log tools.statistics-api Migrated to 2020 Kubernetes cluster [16:30:04] !log tools.status Migrated to 2020 Kubernetes cluster [16:30:14] !log tools.stemmeberettigelse Migrated to 2020 Kubernetes cluster [16:30:23] !log tools.stewardbots Migrated to 2020 Kubernetes cluster [16:30:33] !log tools.strephit Migrated to 2020 Kubernetes cluster [16:30:44] !log tools.stylize Migrated to 2020 Kubernetes cluster [16:30:54] !log tools.supercount Migrated to 2020 Kubernetes cluster [16:31:08] !log tools.svgcheck Migrated to 2020 Kubernetes cluster [16:31:17] !log tools.svgedit Migrated to 2020 Kubernetes cluster [16:31:27] !log tools.swviewer Migrated to 2020 Kubernetes cluster [16:36:12] !log tools rebooting k8s workers 1-35 on the 2020 cluster to clear a strange nologin condition that has been there since the NFS maintenance [16:44:20] !log tools drained tools-k8s-worker-15 and hard rebooting it because it wasn't happy [16:44:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:51:47] !log tools node/tools-k8s-worker-15 uncordoned [16:51:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:49:49] !log tools.admin Running k8s-2020-migrate.sh with prefix=t [17:49:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [17:50:01] !log tools.tabulist Migrated to 2020 Kubernetes cluster [17:50:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tabulist/SAL [17:50:12] !log tools.teg Migrated to 2020 Kubernetes cluster [17:50:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.teg/SAL [17:50:34] !log tools.templatecheck Migrated to 2020 Kubernetes cluster [17:50:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.templatecheck/SAL [17:50:47] !log tools.templatetiger Migrated to 2020 Kubernetes cluster [17:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.templatetiger/SAL [17:50:58] !log tools.templatetransclusioncheck Migrated to 2020 Kubernetes cluster [17:50:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.templatetransclusioncheck/SAL [17:51:22] !log tools.tesseract-ocr-service Migrated to 2020 Kubernetes cluster [17:51:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tesseract-ocr-service/SAL [17:51:34] !log tools.test-webservice-generic Migrated to 2020 Kubernetes cluster [17:51:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.test-webservice-generic/SAL [17:51:44] !log tools.textcatdemo Migrated to 2020 Kubernetes cluster [17:51:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.textcatdemo/SAL [17:51:55] !log tools.tfaprotbot Migrated to 2020 Kubernetes cluster [17:51:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tfaprotbot/SAL [17:52:18] !log tools.thankyou Migrated to 2020 Kubernetes cluster [17:52:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.thankyou/SAL [17:52:27] !log tools.thibaut120094 Migrated to 2020 Kubernetes cluster [17:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.thibaut120094/SAL [17:52:40] !log tools.thibtools Migrated to 2020 Kubernetes cluster [17:52:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.thibtools/SAL [17:52:49] !log tools.threed2commons Migrated to 2020 Kubernetes cluster [17:52:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.threed2commons/SAL [17:52:59] !log tools.tilde Migrated to 2020 Kubernetes cluster [17:53:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tilde/SAL [17:53:08] !log tools.timescale Migrated to 2020 Kubernetes cluster [17:53:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.timescale/SAL [17:53:30] !log tools.toolforge-gallery Migrated to 2020 Kubernetes cluster [17:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-gallery/SAL [17:53:39] !log tools.toolschecker-k8s-ws Migrated to 2020 Kubernetes cluster [17:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolschecker-k8s-ws/SAL [17:53:44] looks like stashbot unquieted? [17:53:49] !log tools.toolserver-home-archive Migrated to 2020 Kubernetes cluster [17:53:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolserver-home-archive/SAL [17:53:58] !log tools.toolserver Migrated to 2020 Kubernetes cluster [17:53:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolserver/SAL [17:54:00] hmmm. indeed [17:54:07] !log tools.tools-info Migrated to 2020 Kubernetes cluster [17:54:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tools-info/SAL [17:54:19] !log tools.topviews-test Migrated to 2020 Kubernetes cluster [17:54:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.topviews-test/SAL [17:54:30] !log tools.totoazero Migrated to 2020 Kubernetes cluster [17:54:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.totoazero/SAL [17:54:40] !log tools.tour Migrated to 2020 Kubernetes cluster [17:54:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tour/SAL [17:54:53] !log tools.tptools Migrated to 2020 Kubernetes cluster [17:54:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.tptools/SAL [17:55:03] !log tools.traffic-grapher Migrated to 2020 Kubernetes cluster [17:55:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.traffic-grapher/SAL [17:55:06] didn't you say something about voice overriding q or something before? [17:55:12] !log tools.translate-bot Migrated to 2020 Kubernetes cluster [17:55:34] !log tools.translatemplate Migrated to 2020 Kubernetes cluster [17:55:38] that's better [17:55:44] !log tools.translate Migrated to 2020 Kubernetes cluster [17:55:53] !log tools.translation-server Migrated to 2020 Kubernetes cluster [17:56:02] !log tools.trusty-deprecation Migrated to 2020 Kubernetes cluster [17:56:05] thanks :) [17:56:20] I see what happened too. Stashbot restarted and the mode here autovoices it [17:56:24] !log tools.tsbot Migrated to 2020 Kubernetes cluster [17:56:34] !log tools.tts-comparison Migrated to 2020 Kubernetes cluster [17:56:43] !log tools.twl17 Migrated to 2020 Kubernetes cluster [17:56:56] !log tools.twltools Migrated to 2020 Kubernetes cluster [17:57:08] !log tools.typoscan Migrated to 2020 Kubernetes cluster [17:57:58] I think I should actually update the config for stashbot to keep it from responding to mw-bot in channel. [17:58:12] * bd808 looks to see if that is easy config or needs code changes [17:59:54] here's an idea, maybe have it just send back queries for confirmation/status messages [18:04:01] most of the time it's useful [18:08:06] anyone else having issues sshing into the toolforge bastion, or is it just my network [18:08:40] It took a long time to get the motd, and I still haven't gotten to a prompt yet [18:09:42] hold on a sec I can try to log in [18:10:52] yea - this is definitely not normal - almost instant for me usually [18:11:17] bd808: bstorm_ ^ Did I see NFS issues mentioned earlier? [18:11:34] Looks like load is high [18:11:46] Guessing that someone is running something nasty on the bastion [18:12:22] sftp stuff running that may be the cause [18:12:29] Reedy: not today [18:13:27] load might be NFS and caused by qstat runs from some tool [18:14:04] nethogs is flipping back and forth on qstat runs and scp processes [18:14:22] qstat runs? [18:14:26] That's odd. [18:14:37] Scp, on the other hand will hang everything up often [18:15:43] some tool has a `watch qstat` running. Will kill that [18:16:22] I see. I'm fixing a network issue here 😛 [18:17:13] load avg is low now [18:17:37] normal responsiveness now, thanks [18:17:51] If you didn't touch the scp, qstat did it, which would be new! :) [18:18:19] scp isn't running now, though, so ... [18:19:42] nfs home dirs...not a fan [18:29:34] !log tools.admin Running k8s-2020-migrate.sh with prefix=u [18:29:44] !log tools.universalviewer Migrated to 2020 Kubernetes cluster [18:30:05] !log tools.unpkg Migrated to 2020 Kubernetes cluster [18:30:14] !log tools.uploadhelper-ir Migrated to 2020 Kubernetes cluster [18:30:24] !log tools.upload-stats-bot Migrated to 2020 Kubernetes cluster [18:30:34] !log tools.usage Migrated to 2020 Kubernetes cluster [18:30:44] !log tools.user-contributions-feed Migrated to 2020 Kubernetes cluster [18:30:55] !log tools.usernamesearch Migrated to 2020 Kubernetes cluster [18:31:05] !log tools.userrank Migrated to 2020 Kubernetes cluster [18:31:14] !log tools.usrd-tools Migrated to 2020 Kubernetes cluster [18:40:31] !log tools.admin Running k8s-2020-migrate.sh with prefix=v [18:40:40] !log tools.vendor Migrated to 2020 Kubernetes cluster [18:40:51] !log tools.video2commons-test Migrated to 2020 Kubernetes cluster [18:41:01] !log tools.video-cat-bot Migrated to 2020 Kubernetes cluster [18:41:10] !log tools.visualcategories Migrated to 2020 Kubernetes cluster [19:08:03] !log tools.admin Running k8s-2020-migrate.sh with prefix=x [19:08:12] !log tools.xfd-stats Migrated to 2020 Kubernetes cluster [19:08:23] !log tools.xtools-autoedits Migrated to 2020 Kubernetes cluster [19:08:45] !log tools.xtools-dev Migrated to 2020 Kubernetes cluster [19:09:01] !log tools.xtools-pages Migrated to 2020 Kubernetes cluster [19:09:11] !log tools.xyzbot Migrated to 2020 Kubernetes cluster [19:09:17] !log tools.admin Running k8s-2020-migrate.sh with prefix=y [19:09:28] !log tools.yabbr Migrated to 2020 Kubernetes cluster [19:09:40] !log tools.yadkard Migrated to 2020 Kubernetes cluster [19:09:50] !log tools.yellowbot Migrated to 2020 Kubernetes cluster [19:10:01] !log tools.yifeibot Migrated to 2020 Kubernetes cluster [19:10:12] !log tools.youtube-channel Migrated to 2020 Kubernetes cluster [19:12:27] !log tools.admin Running k8s-2020-migrate.sh with prefix=z [19:12:37] !log tools.zhaofeng-test Migrated to 2020 Kubernetes cluster [19:12:48] !log tools.zhwiki-username-check Migrated to 2020 Kubernetes cluster [19:12:59] !log tools.zoomable-images Migrated to 2020 Kubernetes cluster [19:13:13] !log tools.zoomproof Migrated to 2020 Kubernetes cluster [19:13:20] !log tools.zppixbot Migrated to 2020 Kubernetes cluster [19:13:31] !log tools.zppixbot-test Migrated to 2020 Kubernetes cluster [20:01:16] !log tools.admin Running k8s-2020-migrate.sh with prefix=w [20:01:39] !log tools.wakt Migrated to 2020 Kubernetes cluster [20:01:45] This will be a big batch of migrations, but it is the last one! [20:01:48] !log tools.wam-article-suggestions Migrated to 2020 Kubernetes cluster [20:01:58] hopefully it'll work this time [20:02:01] (with some manual migrations to follow) [20:02:09] !log tools.wam Migrated to 2020 Kubernetes cluster [20:02:18] !log tools.warped-to-iframe Migrated to 2020 Kubernetes cluster [20:02:24] cluster is looking good so far -- https://grafana-labs.wikimedia.org/d/toolforge-kubernetes/toolforge-kubernetes?orgId=1&refresh=1m&from=now-12h&to=now [20:02:40] !log tools.watch-translations Migrated to 2020 Kubernetes cluster [20:02:50] !log tools.watroles Migrated to 2020 Kubernetes cluster [20:02:59] !log tools.wb2rdf Migrated to 2020 Kubernetes cluster [20:03:09] !log tools.wcam-bot Migrated to 2020 Kubernetes cluster [20:03:19] !log tools.wcna-2018-registration Migrated to 2020 Kubernetes cluster [20:03:31] !log tools.wd-analyst Migrated to 2020 Kubernetes cluster [20:03:52] !log tools.wdbeoupdate Migrated to 2020 Kubernetes cluster [20:04:02] !log tools.wd-depicts Migrated to 2020 Kubernetes cluster [20:04:13] !log tools.wdprop Migrated to 2020 Kubernetes cluster [20:04:25] !log tools.wdq2sparql Migrated to 2020 Kubernetes cluster [20:04:34] !log tools.wdq-checker Migrated to 2020 Kubernetes cluster [20:04:42] !log tools.wd-rank Migrated to 2020 Kubernetes cluster [20:04:52] !log tools.wdvaliditycheck Migrated to 2020 Kubernetes cluster [20:05:04] !log tools.wdvd Migrated to 2020 Kubernetes cluster [20:05:13] !log tools.weapon-of-mass-description Migrated to 2020 Kubernetes cluster [20:05:23] !log tools.weapon-of-mass-description-test Migrated to 2020 Kubernetes cluster [20:05:47] !log tools.wiki13 Migrated to 2020 Kubernetes cluster [20:05:56] !log tools.wiki2email Migrated to 2020 Kubernetes cluster [20:06:08] !log tools.wiki3d Migrated to 2020 Kubernetes cluster [20:06:23] !log tools.wikibase-termbox-storybook Migrated to 2020 Kubernetes cluster [20:06:33] !log tools.wikicite-dashboard Migrated to 2020 Kubernetes cluster [20:06:44] !log tools.wikicup Migrated to 2020 Kubernetes cluster [20:41:36] !log tools.admin Running k8s-2020-migrate.sh with prefix=w [20:43:26] !log tools.nearby-places-viewer Migrated to 2020 Kubernetes cluster [20:44:54] !log tools.nodejs-mw-oauth-tool Migrated to 2020 Kubernetes cluster [20:51:14] !log tools.osm-gadget-leaflet Migrated to 2020 Kubernetes cluster [20:52:11] !log tools.pageviews-test Migrated to 2020 Kubernetes cluster [20:52:55] !log tools.potd-viewer Migrated to 2020 Kubernetes cluster [20:53:48] !log tools.seealsology Migrated to 2020 Kubernetes cluster [20:59:51] !log tools.statistics Migrated to 2020 Kubernetes cluster [21:02:04] !log tools.yadfa Migrated to 2020 Kubernetes cluster [21:03:04] !log tools.wd-image-positions Migrated to 2020 Kubernetes cluster [21:04:22] !log tools.wdmm Migrated to 2020 Kubernetes cluster [21:05:34] !log tools.wembedder Migrated to 2020 Kubernetes cluster [21:14:01] !log tools.abbreviso Migrated to 2020 Kubernetes cluster [21:25:33] !log tools.algo-news Migrated to 2020 Kubernetes cluster [21:30:39] !log tools.anon Migrated to 2020 Kubernetes cluster [21:33:14] !log tools.autolist Migrated to 2020 Kubernetes cluster [21:35:11] !log tools.commons-app-stats Migrated to 2020 Kubernetes cluster [21:36:08] !log tools.admin Running k8s-2020-migrate.sh with prefix=? [21:38:10] !log tools.gerrit-newcomer-bot Migrated to 2020 Kubernetes cluster [21:42:15] !log tools.mdann52bot Migrated to 2020 Kubernetes cluster [21:43:31] !log tools.portal-stats Migrated to 2020 Kubernetes cluster [21:43:54] !log tools.position-holder-history Migrated to 2020 Kubernetes cluster [21:44:18] !log tools.prompter Migrated to 2020 Kubernetes cluster [21:44:40] !log tools.proneval-gsoc17 Migrated to 2020 Kubernetes cluster [21:44:51] !log tools.proxies Migrated to 2020 Kubernetes cluster [21:45:11] !log tools.ptable Migrated to 2020 Kubernetes cluster [21:45:22] !log tools.pub Migrated to 2020 Kubernetes cluster [21:45:33] !log tools.pyshexy Migrated to 2020 Kubernetes cluster [21:45:42] !log tools.pywikibot-testwiki Migrated to 2020 Kubernetes cluster [21:45:52] !log tools.pywikipedia Migrated to 2020 Kubernetes cluster [21:52:23] !log tools.commons-video-clicks Deleted orphan kubernetes webservice pod [21:56:46] !log tools.congressedits Migrated to 2020 Kubernetes cluster [21:59:25] !log tools.contact Migrated to 2020 Kubernetes cluster [22:00:18] !log tools.contentcontributor Migrated to 2020 Kubernetes cluster [22:02:34] bd808 or andrewbogott or anyone really, i cannot run puppet on @deployment-eventlog05:~$ sudo puppet agent -tv, did we changed the way that is done? (we might have changed it like last year cause i think i do this once ayer) [22:02:40] * once a year [22:02:55] that should work [22:02:56] nuria: that's how i would do it; what happens? [22:03:07] https://www.irccloud.com/pastebin/FAzY7INU/ [22:04:15] andrewbogott: see above [22:04:46] nuria: I can look in a bit -- I think that the puppet infra in that project may be in the process of upgrade [22:05:00] andrewbogott: k, will be here let me know [22:05:04] nuria: that looks like a puppetdb error. If Krenair is around he might be able to help troublshoot [22:13:11] andrewbogott, nah that's done [22:13:30] !log tools.countcountbot Migrated to 2020 Kubernetes cluster [22:13:37] looks like one of the new instances is broken though [22:14:03] Krenair: regarding eventlog05.deployment? or something else? [22:14:32] !log tools shutting down the old maintain-kubeusers and taking the gloves off the new one (removing --gentle-mode) [22:14:39] nuria, well it's not eventlog05 is it? eventlog05 just sees the symptoms of the puppetdb host being broken [22:15:29] Krenair: i see, let me know when you think it can be fixed [22:15:30] These last 2 redirects make zero sense - intentionally making a double redirect?... - https://wikitech.wikimedia.org/wiki/Special:Contributions/RLazarus [22:15:54] nuria, I logged into the puppetdb host and ran `service puppetdb start`, it seems to work now [22:16:05] Krenair: k let me reetry [22:19:14] as for what went wrong, oom-killer killed it at Feb 25 03:15:18 [22:19:45] huh, puppet doesn't enforce that it's running? [22:20:03] puppet can't run when it's not though [22:20:21] the next thing in syslog is puppet running and it not being able to retrieve its catalog [22:25:00] that's a fun combo :/ [22:25:23] Krenair: working now thanks [22:25:43] Krenair: may i ask waht command i can use to list processes as >ps -aux -fw does not seem to work? [22:26:18] `ps auxfw` seems to work for me [22:26:58] Krenair: ah ya , ok no '-' [22:31:54] I've seen puppetdb get OOMKilled like this before [22:32:06] may need to replace it with a larger VM [22:32:27] annoyingly really as the old one ran happily with the same amount of RAM [22:39:38] oh, it's managed by itself, of course [22:40:14] It's fine if you need a bigger VM but agreed, the ram inflation is disappointing. [22:52:44] hooray for the JVM [23:59:14] !log tools.dewikigreetbot Migrated to 2020 Kubernetes cluster