[06:21:33] !log tools.openstack-browser-dev apply https://phabricator.wikimedia.org/R2073:12a3fc4d4f16a76bb313d78443f4579f8a5c5531 [06:21:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser-dev/SAL [08:26:11] !log admin draining and rebooting coludvirt1018 (T280641) [08:26:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [08:26:15] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [09:12:39] !log admin draining and rebooting coludvirt1021 (T280641) [09:12:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:12:43] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [09:29:15] dcaro: s/coludvirt/cloudvirt on your copy-paste message [09:30:00] xd, I make that type very often [10:23:09] !log admin (from a cookbook) [10:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:23:24] almost :) [10:23:30] :-) [10:24:01] !log wmflabsdotorg created PTR records for cloudgw100{1,2}.eqiad1.wikimediacloud.org. (T270704) [10:24:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmflabsdotorg/SAL [10:24:03] T270704: cloud: introduce new edge network architecture for eqiad1 and codfw1dev - https://phabricator.wikimedia.org/T270704 [10:31:29] !log admin Safe rebooting 'cloudvirt1021.eqiad.wmnet'. (T280641 - cookbook ran by dcaro@vulcanus) [10:31:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:31:35] \o/ [10:31:54] (it crashed doing the drain, but the log is there :)) [10:32:59] see it does not like the task format though [10:33:17] it added the entry in the task anyhow [10:33:28] (T280641 [10:33:29] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [10:33:34] oh, there it is :) [10:34:08] (T280641 - something) [10:34:21] hmmm [10:34:51] ยก+ [10:34:59] (cat typing) [10:35:03] * arturo brb [10:35:41] hahahaha [11:07:29] !log tools.openstack-browser update to fix cinder quota display [11:07:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [13:13:40] Hi everybody [13:14:56] I'm running a bot with pywikibot and since april, 30th, it's not working [13:15:13] please define "not working" [13:15:24] ;) [13:15:53] It does not start when launched [13:16:07] the text says: Traceback (most recent call last): [13:16:08] File "/data/project/shared/pywikibot/core/scripts/replace.py", line 154, in [13:16:09] from pywikibot.exceptions import NoPageError [13:16:11] ImportError: cannot import name 'NoPageError' [13:16:13] CRITICAL: Exiting due to uncaught exception [13:16:39] Could anybody help me? [13:18:07] what script are you trying to run, and how are you launching it? [13:18:57] Thanks, Majavah for your interest [13:19:09] the script is replace.py [13:20:24] the exact syntax is sh /data/project/rebot/robot.sh [13:23:21] !log toolsbeta livehacking puppetmaster with https://gerrit.wikimedia.org/r/c/operations/puppet/+/684032 (T278109) [13:23:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:23:27] T278109: setup email for toolforge.org and wmcloud.org like on {tools.,}wmflabs.org - https://phabricator.wikimedia.org/T278109 [13:25:26] @Pau: in the sh file, try replacing "python3 /data/project/shared/pywikibot/core/scripts/replace.py" with "python3 /data/project/shared/pywikibot/core/pwb.py replace.py" [13:27:32] Thanks [13:27:40] I'll change it [13:29:17] change made [13:30:08] It works! [13:30:21] thanks [13:55:21] !log deployment-prep enable new search features for the template dialog (T271802) [13:55:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [13:55:26] T271802: Better search - search for keywords out of order in template search (Template findability) - https://phabricator.wikimedia.org/T271802 [14:01:00] Hi ๐Ÿ‘‹, I am a toolforge newbie. I've created a tool on toolforge called pilot-flag-lab based on https://labelstud.io/guide/. Got all the python dependencies up and running. Whenever I start my tool, it starts fine. But when I visit https://pilot-flag-lab.toolforge.org/, I get a "504 Gateway Time-out" error. PS: I've followed the same steps on my local machine and the tool runs fine without the 504 error. Could I be missing somethi [14:01:01] ng with the toolforge environment? [14:03:03] hi kevinbazira, how did you start it on toolforge? [14:05:20] Hi Majavah.I run "label-studio start nineth_labeling_campaign --init --host https://pilot-flag-lab.toolforge.org/" inside a Kubernetes shell. [14:07:25] !log tools depooling tols-sgeexec-0908/7 to be able to restart the VMs as they got stuck during migration (T280641) [14:07:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:07:30] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [14:11:30] kevinbazira: `webservice shell` is designed for developing and debugging tools running in kubernetes, it does not allow you to actually start a webservice inside the interactive container. webservices can be started either with the `webservice start` command (which might not work in this case) or by writing some complex kubernetes definition files [14:19:07] Majavah: Thank you for the clarification. Any ideas on how I can host https://labelstud.io/guide/ so that my team members can easily access it without having to set it up locally on their machines? [14:24:19] have you double checked what info that software collects? according to the website they seem to collect data even when running your own instance [14:24:50] https://labelstud.io/guide/#Information-collected-by-Label-Studio [14:31:56] kevinbazira: it's a django app so you might be able to set it up in a way that it works with `webservice start`, but that would require some tweaking too (but likely much less than messing with kubernetes directly), so no easy solutions unfortunately [14:31:56] and yes, please check what information it collects as d.caro suggested above, the cloud services terms of use is fairly strict on what information tools running there can collect [14:33:35] kevinbazira: https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Django_OAuth_tool should have some info that could be helpful [14:54:41] Great, thank you for your help chicocvenancio, dcaro and Majavah. [14:54:41] Re: data collection, they mention that they collect anonymous usage statistics about the number of page visits and data types being used in labeling configurations. [14:54:41] The end users of this tool are mostly my team members who are are exploring this software as we plan to develop are similar one (ofcourse without the data collection part): https://toolsadmin.wikimedia.org/tools/id/pilot-flag-lab [15:03:10] kevinbazira: you might find this useful though: https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#What_information_should_I_provide_to_users? I'm not very familiar with the current terms and conditions details though [15:13:12] !log admin Safe rebooting 'cloudvirt1022.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [15:13:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:41:23] !log admin Safe reboot of 'cloudvirt1022.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [15:41:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:41:50] !log admin Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [15:41:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:23:57] !log tools started tools-sgeexec-0907, was stuck on initramfs due to an unclean fs (/dev/vda3, root), ran fsck manually fixing all the errors and booted up correctly after (T280641) [16:24:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:24:03] T280641: ceph: Upgrade to latest Nautilus/Octopus to fix CVE-2021-20288 - https://phabricator.wikimedia.org/T280641 [16:29:08] !log admin Safe rebooting 'cloudvirt1023.eqiad.wmnet'. (T280641) - cookbook ran by dcaro@vulcanus [16:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:29:15] ^try 2 [16:31:36] Hi, I have a question. Can mail servers be run on WMCS? [16:32:47] To give some context, Wikinews admins have been running a web cum mail server for years privately, and they have indicated a desire to switch to WMF-based options. For the web server that's accomplished by running a WMCS instance, but I'm not sure what the best option for the mail server would be. [16:32:59] I was asked to come here by a sysadmin during business hours, which I hope this is. [16:34:02] !log admin Safe reboot of 'cloudvirt1023.eqiad.wmnet' finished successfully. (T280641) - cookbook ran by dcaro@vulcanus [16:34:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:35:44] Leaderboard: do you mean they have an instance in Cloud VPS? https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS , WMCS is all the cloud services, both Toolforge, Cloud VPS, and others [16:35:58] joakino: to clarify, I mean Cloud VPS [16:37:55] Leaderboard: aha! Cloud VPS are very flexible and self managed so I think it is a good option. when you say "running a web cum mail server for years privately", what do you mean? what mail server are they running? [16:39:28] if it is a mailing list, maybe it would be worth moving to one in https://lists.wikimedia.org/mailman/listinfo ? [16:42:18] joakino: I assume this is related to https://phabricator.wikimedia.org/T281600 and https://phabricator.wikimedia.org/T281520#7048316 [16:42:55] joakino: so from what I've understood, all accredited reporters on Wikinews get an email address - I don't think a mailing list would do that. [16:43:43] They currently run it on their own, and was wondering if this can be achieved on the Cloud VPS instance (which hasn't yet been created) [16:44:21] technically speaking, yes you can set up your own mailserver in Cloud VPS. whether you should is another matter entirely [16:44:46] legoktm is correct on the context, but no mention of the mail server plan was given at that ticket (because the focus was on the wiki) [16:46:23] legoktm: how would it work in that case? Like what would be the send/receive address in that case? If an alternative would suit better let me know and I'll feed it back [16:47:26] it would be @.wmcloud.org [16:47:59] couldn't you run the smtp server in the cloud vps instance, and point the mx records of the domain you use (wn-reporters.org) to the instance's address? [16:49:09] Leaderboard: who is running this currently? Is it hired as a service or is it a mail server some volunteer is managing? [16:49:10] You could theoretically bring your own domain too... Theoretically [16:49:25] you'll need a floating ip allocated to the project in any case [16:49:58] joakino: Ideally it would be nice if "wn-reporters.org" could be superseded by the Cloud VPS instance. The admin (acagastya) is currently spending ~$100/year for this, which I felt was ridiculous and suggested moving to WMF servers [16:50:37] chicovenancio: it's a volunteer admin currently managing the web/mail server - before that it was a different admin whose dH/dt = 0 [16:50:43] currently [16:53:16] Leaderboard: my biggest concern with mail servers is managing mail servers is a pain. If you have volunteers to manage it, create the ticket and discuss it with the team there. [16:54:00] chicovenancio: which team? [16:56:15] T281600 is the ticket, but not a lot of information there, I'd recommend you copy the information there. The cloud services team is what I meant. [16:56:16] T281600: Request creation of wikinewsie VPS project - https://phabricator.wikimedia.org/T281600 [16:56:18] cloud services, which runs the infrastructure for cloud vps, toolforge and some other projects [16:57:53] yep, if you can fill the description, and talk about how you want to run run a wiki, and a mail server, and also mention if you want or not to use your external domain, that in itself would be helpful to evaluate the request [16:58:17] I'll expand it (most of the background was in the old ticket) [16:58:47] Leaderboard: so to answer your initial questions, yes, mail servers can be run on cloud vps instances, it is a server administrators' work, so it is not easy or straightforward [16:59:02] Also one more thing. The ToS appears to indicate that using Cloud VPS is not supposed to be used as a VPN. However, one of the admins has told me that (supposedly because of government restrictions), he is unable to access essential sites for reviewing, and asked me whether it was OK to use Wireguard as a VPN for this reason [17:01:30] (currently I'm told that they have to use "Pornhub's VPN", not that I am sure why that exactly) [17:02:34] Leaderboard: that item in the ToS is to not use the WMCS servers as proxies for other traffic, if he is using an external VPN for anonymity reasons I don't think that is the same [17:03:27] OK, I interpreted it that way and could well be wrong. I'll feed it back to him [17:03:41] damn https://www.vpnhub.com/ it is true heh [17:05:25] oki, as long as you don't use the servers to run proxies like Tor nodes, peer-to-peer network servers, or VPNs servers, it should be good to go. If you have more doubts the ticket is a good place to ask too with specific details about what they may be doing [17:06:02] joakino: to clarify, I think it's a proxy that I'm thinking about - after all, the idea is that he would like to be able to use the Cloud VPS instance to access the site rather than having to using a Pornhub service. Sounds like a proxy to me, though I'll clarify in the phabricator task [17:07:29] yep, better to get more specific details about what exactly he wants to do [17:27:40] joakino: I've updated T281600, let me know if you need more details [17:27:41] T281600: Request creation of wikinewsie VPS project - https://phabricator.wikimedia.org/T281600 [17:30:29] thanks Leaderboard, I think running Wireguard on the instance is not going to be allowed, it will need to be reviewed by the team, but the rest sound feasible to me. They'll have a look soon [17:31:47] joakino: to clarify, is the problem with Wireguard or with the proxy plan (which can also be achieved using a SOCKS proxy for instance) [17:31:52] to repeat the point, there is no managed mail servers or domain configuration in cloud vps, configuring, setting up and running the mail server and domains will need to be done by the sysadmins of the instance, you or your partner [17:32:15] joakino: I got that, noted in the phab task for completeness [17:32:39] generally using the instance as a proxy, regardless of software, that is what is forbidden by the ToS explicitly [17:35:03] there are much better VPN options out there than Cloud VPS [17:42:02] Also to clarify, where exactly are the servers located? Can't find the location (my understanding is that it's only in US)? [17:45:30] yep, US right now, I believe the cloud services are in eqiad https://wikitech.wikimedia.org/wiki/Eqiad_cluster [17:47:37] Is "Eqsin cluster" used only for production then? [17:47:48] And "Essams cluster"? [17:47:50] there are some things too in https://wikitech-static.wikimedia.org/wiki/Codfw_cluster [17:48:55] Leaderboard: those are not used for cloud services afaict, https://wikitech-static.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Deployments [17:55:43] Leaderboard: eqsin, esams, and ulsfo only have caching for the wikis, everything else happens on eqiad or codfw. WMCS currently has all production machines in eqiad, codfw has a small cluster for testing cloud vps upgrades and config changes. [17:56:16] (IRC does not have a thumbsup option) [17:57:08] you can copy/paste from emojipedia. https://emojipedia.org/thumbs-up/ [19:35:31] !log tools.lexeme-forms deployed b159dd1060 (l10n updates) [19:35:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [20:03:07] I think bridge bot is missing some messages [20:03:52] Like it missed Lucas' !log just now, but caught stashbot's response [20:22:23] @Alex could be. I think I really need to setup multiple copies of bridegbot. It works to add tons of different channels to one, but I think it does lose some messages as a result [20:43:24] RAIBB [21:09:11] maybe it just ignores !log [21:09:15] or all ! commands [21:14:40] !test [21:19:03] Huh well that came through [21:25:29] Maybe just log? [21:25:33] !log blah [21:25:34] RhinosF1: Missing project or message? Expected !log [21:25:41] Yes I am on purpose [21:34:07] That went though [21:34:10] Through [21:34:24] Might be ones that get to IRC via wm-bot [21:38:18] Iโ€™d guess that wm-bb likely ignores wm-bot [22:00:03] MacFan4000: yes. I had to look at the config to be sure, but I have the IRC side of the bridge set to ignore all messages from wm-bot. The comment in the config says "Ignore some spammy irc bots" [22:01:39] that config will keep all the !log messages sent via the wm-bot relay from a bastion from being seen on Telegram [22:02:03] that's not why wm-bot was set to be ignored, but it is a side effect [22:21:56] bd808: Did you change that recently? The Telegram group feels a lot quieter lately (I'm not complaining) [22:23:21] @jhsoby: I think that's been in place for at least a year. Long enough that I forgot it was there. :) [22:25:07] aha, i must be thinking of something else then ๐Ÿ˜Š [23:50:32] !log tools.bridgebot Hard restart to see if that fixes some missing bridges [23:50:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [23:51:02] \o/ wm-bb is back on https://chat.wmcloud.org/wikimedia/channels/wikimedia-cloud [23:51:05] !log admin running `maintain-dbusers harvest-replicas` on labstore1004 [23:51:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:53:04] actually, to get that on the right task [23:53:18] !log admin running `maintain-dbusers harvest-replicas` on labstore1004 T281287 [23:53:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:53:21] T281287: Now that labsdb1012 is clouddb1021, connect it to maintain-dbusers - https://phabricator.wikimedia.org/T281287