[10:01:37] !log gitlab-test Increased quota +1 instance, +4 cores, +8G ram (T274458) [10:01:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-test/SAL [10:01:40] T274458: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 [10:02:06] I think that was the wrong task xd [10:02:21] !log gitlab-test Increased quota +1 instance, +4 cores, +8G ram (T278197) [10:02:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-test/SAL [10:02:25] T278197: Request increased quota for gitlab-test Cloud VPS project: Extra floating IP, capacity for extra g3.cores4.ram8.disk20 - https://phabricator.wikimedia.org/T278197 [11:31:25] !log tools.openstack-browser-dev deploy cinder quota display [11:31:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser-dev/SAL [11:32:57] !log tools.openstack-browser deploy cinder quota display [11:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [12:21:38] !log tools shutdown tools-package-builder-02 (stretch), we keep -03 which is buster (T275864) [12:21:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:21:45] T275864: Toolforge: migrate to Debian Buster - https://phabricator.wikimedia.org/T275864 [14:25:42] The Parliament Diagram Generator at https://parliamentdiagram.toolforge.org/parlitest.php uses OAuth to upload to Wikimedia Commons, but I've just gotten a bug report saying that uploads are failing because they're IP blocked. [14:25:50] I tested that, and can confirm that this is happening. [14:26:00] https://github.com/slashme/parliamentdiagram/issues/91 [14:26:29] How can I find out what IP is blocked here and why? [14:27:47] !help [14:27:47] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [14:28:36] The precise error text is: "Error: Your IP address has been blocked automatically, because it was used by a blocked user." [14:31:43] slashme_irc, you get that error on commons then after oauth yes? [14:31:47] which wiki? [14:31:58] ah, commons, oops missed that [14:32:02] Yes, commons. [14:32:06] OAuth works, [14:32:13] but uploading using oauth doesn't. [14:32:27] does the error message specify a block id number or anything similar? [14:32:49] I see the last successful upload from the tool was 08:17 https://commons.wikimedia.org/w/index.php?tagfilter=OAuth+CID%3A+538&limit=1000&days=30&title=Special:RecentChanges&urlversion=2 [14:33:19] Majavah: no, just that. [14:33:44] I'm not sure how to get the full error information. [14:36:02] Oops, I got disconnected just now because my desktop lost power :-/ [14:36:37] Not sure whether I missed any suggestions or questions. [14:36:45] nope, I'm looking at a few things [14:37:00] Thanks!! [14:37:29] Oh, it's working again! [14:38:19] oh? [14:38:36] slashme_irc, nice! I didn't see any recent changes to blocks from commons [14:38:55] either the tool pod was moved to a different k8s node or the autoblock expired [14:39:38] Yes, my browser was restarted due to the reboot, and when I reloaded the tab with the failed upload, it first gave me the expected warning that the file I was trying to upload already existed (because I keep a test file), and then when I clicked on the upload link, it worked. [14:40:18] https://commons.wikimedia.org/wiki/File:My_Parliament.svg [14:40:45] did you do anything on the toolforge side? [14:40:52] Majavah, ahh.. is that the autoblock message on commons? [14:40:54] Nothing. [14:41:05] balloons: that's the standard autoblock message [14:41:17] 👍 I didn't know that [14:41:31] What is this "autoblock message"? [14:41:53] perhaps https://en.wikipedia.org/wiki/Wikipedia:Autoblock ? [14:42:03] "autoblock" means that an IP address is automatically blocked for 24 hours when a blocked user tries to use it [14:42:08] I'm not sure on commons, but that's the gist [14:42:28] Ah, OK. That makes sense. [14:43:12] the full message should look something like https://commons.wikimedia.org/wiki/MediaWiki:Autoblockedtext, so I'm curious why the message you see was so short and does not contain any useful information to debug it [14:43:32] The message I see is somewhat filtered by my PHP. [14:44:49] I would have expected the whitelist would prevent an autoblock [14:46:40] To get the warning message to display to the user, I do: [14:46:42] if (isset($last_res->upload) && isset($last_res->upload->warnings) ) { [14:46:43]         echo "
"; [14:46:43]         foreach ( $last_res->upload->warnings as $k => $v ) { [14:46:50] and then look at $k. [14:46:57] don't copy paste huge amounts of test, use a pastebin instead [14:47:00] If I recognise it, I have specific warnings, [14:47:15] and if not, I do "echo "Warning \"".$k."\": ".$v."
";" [14:47:37] So I guess some of the warning text doesn't make it to the user? [14:48:34] Oh, was that huge? Sorry! [14:50:35] balloons: based on another user report on a private channel I now know that tools-sgewebgrid-lighttpd-0913 has "multiple blocks against your account and/or IP address", and I'd assume that would be the standard WMCS anon block and something like this [14:50:49] let me see if I can get it to tell the another block message [14:58:00] balloons: at least https://commons.wikimedia.org/wiki/Special:BlockList?wpTarget=%23405016&blockType=&limit=1000&wpFormIdentifier=blocklist affects a toolforge web grid node [14:59:47] Majavah, ahh. You got the block id from the reporter I presume [15:00:51] balloons: nope, some magic tricks - got a report of the crop tool being broken in another channel, found out which web grid node it was running on, ssh'd in (grid exec nodes can be ssh'd in by non-admins), curl "https://commons.wikimedia.org/w/index.php?title=User_talk:Majavah&action=edit" > edit.html and then look at that HTML for the full message [15:01:23] Majavah, clever! Thanks for sharing [15:01:34] I'm somewhat surprised to see those would affect the grid nodes, but apparently they do :-) [15:01:36] https://commons.wikimedia.org/wiki/MediaWiki:Autoblock_whitelist has 172.16.0.0/12 in it, so I'm not sure how this happened [15:01:47] Yes, exactly [15:03:17] and on https://commons.wikimedia.org/wiki/MediaWiki:Block-autoblock-exemptionlist, so https://gerrit.wikimedia.org/r/c/mediawiki/core/+/673654 should not be the cause here [18:03:40] !log tools.wikidata-primary-sources truncated the 27G error.log file T276525 [18:03:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikidata-primary-sources/SAL [18:03:43] T276525: 2021-03-05: tools nfs share cleanup - https://phabricator.wikimedia.org/T276525 [18:06:49] !log tools.request truncated 20G error log file T276525 [18:06:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.request/SAL [18:10:38] !log tools.khanamalumat truncated the 80+GB qaus.err and purawiki.err files T278199 [18:10:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.khanamalumat/SAL [18:10:42] T278199: khanamalumat has a job that puts a lot of text in a log file when not doing any changes - https://phabricator.wikimedia.org/T278199 [19:27:37] !log quarry deploying changes to the replica class and restarting things T278544 [19:27:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [19:27:42] T278544: Queries left in "running" state for hours - https://phabricator.wikimedia.org/T278544 [20:05:59] Following up on this - I think I asked before and the answer was no - but can I use that statsd endpoint from a Toolforge tool ? I would like to track a couple of metrics and that would be a very easy way for me to achieve that. (re @wmtelegram_bot: [irc] dancy: I just made a "bd808.test" metric by doing functionally the same thing although I used telnet rather than nc. So I guess one thing to work out is if there are so [20:10:00] @JeanFred: you can use it, but also be aware that the service comes with no uptime or stability guarantees and may be completely removed at any time. There is an effort to remove Diamond as a data collector for system metrics and once that is done for Cloud VPS instances we may remove the statsd servers that are currently available. [20:34:53] legoktm: I'd love your review and thoughts on my writeup at T278584 on the SASL idea. [20:34:54] T278584: Promote use of SASL for Cloud VPS/Toolforge hosted Freenode IRC bots - https://phabricator.wikimedia.org/T278584 [20:45:39] bd808: in your checklist, should we set a deadline before making announcements? otherwise lgtm [20:45:58] thank you for pushing this forward :) [20:47:20] legoktm: yeah, a deadline before a wide announce seems reasonable. But I would love to have feedback by anyone other than you and me on what seems "reasonable" too. :) [20:47:43] any ideas on who else to ping into the task now? [20:48:21] let me check the IRC bot list [20:48:50] thanks! I forget that there is a whole page on meta for this stuff [20:52:19] I wonder if freenode staff could give us a list of connections from our IPs and their nicks [20:52:52] one of the toolforge-hosted IRC bots I run isn't on that list :p [20:53:17] there's a friendly staffer lurking on -admin but not here, ask them :P [20:55:25] !log tools.integraality Deploy latest from Git master: bcc4fe41 (T240312) [20:55:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:57:53] !log tools.integraality Reinstalled requirements in both virtual environments for T240312 [20:57:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [21:11:48] !log mailman deleted mailman01 VM. Not needed anymore. Mailman02 handles the cloud test setup. [21:11:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mailman/SAL [21:23:27] what is the IP range that would cover all possible IPs used in toolforge and cloud VPS [21:27:06] mutante: from what frame of reference? (inside our SDN, from the "prod" network, from internet) [21:27:34] bd808: from the Internet, all IPs that could in theory be an IRC bot [21:29:11] mutante: per wikitech 185.15.56.240/29 + 185.15.56.0/25 -- https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Neutron#Topology_data_example [21:29:22] bd808: thank you [21:30:07] it might be worth double checking netbox to see if there are any other allocations but I think that's it [21:47:56] !log tools.lexeme-forms deployed 78a5c9a10a (indicate optional forms) [21:48:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:50:18] !log tools.lexeme-forms deployed 5b44b44f52 (Malayalam verbs) [21:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [22:34:31] so I'm trying to recreate the lists.wmcloud.org MX record in horizon and I'm getting "Error: Unable to create the record set." [22:36:24] bd808, bstorm: ^ do you know why that would happen? there used to be an MX record, but I think it got wiped when artur.o moved the zone into the mailman project [22:37:11] The project you are creating it in probably doesn't have that domain delegated to it [22:37:24] Where are we talking about? [22:37:38] lists.wmcloud.org / mailman project [22:37:48] BUT Amir1 is telling me that we might not need the MX record [22:38:17] yeah, if there's A record, it should be fine [22:38:34] lists.wmcloud.org. 3600 IN A 185.15.56.28 should be enough [22:39:32] huh ok :) [22:39:53] If it's fine, then I will not try to attack designate [22:40:06] I have very small confidence that I won't break something anyway 😁 [22:41:23] false alarm, https://lists.wmcloud.org/hyperkitty/list/test2@lists.wmcloud.org/thread/WXYGT7VQ22QOV4M2HZRRICBGMXSFYYF4/ went through [22:41:32] bstorm: thanks for looking though [23:13:43] :) [23:16:08] it's still not letting me set a TXT record, but I've just re-opened T278358 [23:16:09] T278358: Delegate lists.wmcloud.org domain to be able to add DNS DKIM records - https://phabricator.wikimedia.org/T278358 [23:19:11] legoktm: I think I see the problem. In Designate the "name" needs to be the FQDN for the record. Their goofy database setup does not inherit anything from the zone the record is placed in. [23:21:39] * legoktm tries [23:22:05] bd808: still getting an error :( [23:22:11] poop [23:22:51] fyi on a scale of importance, this is "not very important" [23:26:42] * bd808 pokes in logstash a bit to see if there is an error trace [23:36:48] legoktm: I think it may have something to do with the length of the DKIM nonce. I can make a txt record with "testing", but not with the full string [23:37:29] ah :| [23:37:47] that may just mean that the db schema needs some fixing [23:39:28] ed25519 was added to DKIM in 2018, so we can work around the max length thing, but I wanted to use RSA because that's what prod has [23:45:02] legoktm: I think this may just need a schema change in the Designate db. That's an a.ndrewbogott ask imo though. Not something I'm willing to poke on Friday of a 3 day weekend. [23:45:19] ack, no worries [23:45:27] I verified that I can make a 255 char TXT record but not a 256 char one [23:46:09] https://bugs.launchpad.net/designate/+bug/1595265 [23:47:16] ah wonderful [23:48:08] https://opendev.org/openstack/designate/src/branch/master/designate/objects/rrdata_txt.py#L29 :(( [23:49:36] * bd808 throws another log on the Designate is goofy pile