[00:09:19] !log tools.suha truncated the err output file /project/data/suha/stubnat.err because it was exceeding 26GB T276525 [00:09:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.suha/SAL [00:09:23] T276525: 2021-03-05: tools nfs share cleanup - https://phabricator.wikimedia.org/T276525 [01:43:56] OK, this is *very* odd -- I found a random URL that *does* work with the Mix'n'Match scraper: http://www.emeraldrose.com/consuite/dannyinthejar.htm [02:01:21] and the same page from the Wayback Machine doesn't work: https://web.archive.org/web/20080703173135id_/www.emeraldrose.com/consuite/dannyinthejar.htm [08:14:30] Hmm, is quarry (login) down? I get an internal server error [08:15:24] even just by visiting https://quarry.wmflabs.org/Nemo%20bis [08:19:35] even the front page fails [08:20:09] Majavah: it doesn't if you have no cookie [08:20:31] this suggests the user session backend is down (was it Redis?) [11:07:21] !log toolsbeta drop and build again the VM toolsbeta-sgregrid-shadow (T277653) [11:07:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:07:25] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [11:22:04] !log toolsbeta drop and build again the VM toolsbeta-sgregrid-master (T277653) [11:22:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:22:08] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [11:33:44] !log admin root@cloudcontrol1005:~# wmcs-novastats-dnsleaks --delete [11:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:13:25] !log tools created server group 'tools-grid-master-shadow' with anty-affinity policy [12:13:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:14:51] !log tools created puppet prefix 'tools-sgegrid-shadow' and migrated puppet configuration from VM-puppet [12:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:15:58] !log tools delete & re-create VM tools-sgegrid-shadow as Debian Buster (T277653) [12:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:16:03] T277653: Toolforge: migrate grid to Debian Buster - https://phabricator.wikimedia.org/T277653 [12:46:28] !log tools aborrero@tools-sgegrid-master:~$ sudo systemctl restart gridengine-master.service [12:46:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:35:33] !log tools.majavah-bot deploy change to use new wiki replicas [13:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.majavah-bot/SAL [14:24:43] does anyone here have quarry access? the tool is completely broken, https://phabricator.wikimedia.org/T278230 [14:36:17] it crashes for me too [14:53:20] !log service mariadb start on quarry-db-01 [14:53:21] andrewbogott: Unknown project "service" [14:53:29] !log quarry service mariadb start on quarry-db-01 [14:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [15:08:07] !log quarry systemctl restart mariadb on quarry-db-01 brought it back to life (T278230) [15:08:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [15:08:11] T278230: Quarry: HTTP 500 errors for all authenticated requests - https://phabricator.wikimedia.org/T278230 [15:12:20] Majavah: can you verify that it works for you too? [15:12:33] dcaro: yes [15:12:47] 👍 [18:15:53] Hey folks, I'm finding that url "tools.db.eqiad.wmflabs" works for Tools DB connection but the new url (which I have from the documentation as "tools.db.svc.wikimedia.cloud") does not.  Do I have the new url wrong, or is the tools db url not updating until later? (The new replica DB url works fine for me.) [18:18:14] we may not have the tools alias for the new domain yet suriname0, i'm going to check [18:21:41] suriname0: I don't think we have the alias for tools set yet (from looking at https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/openstack/files/util/wikireplica_dns.yaml), thanks for catching it. I'll amend the docs and make a phab task [18:22:03] ty joakino [18:22:04] did you find it from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database or somewhere else? [18:22:15] yup, that help page [18:35:41] cool, thanks suriname0, made T278252 to track this [18:35:41] T278252: Make alias for tools.db.svc.wikimedia.cloud - https://phabricator.wikimedia.org/T278252 [18:43:56] !log quarry `git stash`ing the in-place test of sentinel in the code checkout [18:43:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [18:45:22] !log quarry stopping the quarry web service for the upgrade process [18:45:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [18:51:51] !log quarry running the multiinstance migration script T264254 [18:51:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [18:51:55] T264254: Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 [19:17:11] !log quarry finished updating quarry for multiinstance replicas T264254 [19:17:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [19:17:16] T264254: Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 [19:24:50] run /cs deop #wikimedia-cloud [19:25:11] yeah, no kidding...stupid pasting around [19:27:12] !log tools.lexeme-forms deployed c0b2c473ff (add language code as ID on index page, suggested by jhsoby) [19:27:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:20:57] bstorm: thanks for working on quarry! I noticed that I get a "bad database name" error on https://quarry.wmflabs.org/query/53474 (which worked until few hours ago), is that expected? [21:21:04] Or is it the same problem as with meta_p [21:21:15] It's a problem with meta_p [21:21:31] Nemo_bis: https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/674427 [21:21:39] I just have to deploy that, I'm pretty sure [21:21:53] Ok thanks [21:22:02] Oh wait, you may have something else. I misread quickly :) [21:22:44] That should work. [21:23:26] Nemo_bis: When re-running, did it blank the database field and make it None? [21:23:31] That could be a different problem [21:23:58] I think it's running fine for me now, which makes me think it's a re-running issue [21:24:26] bstorm: I don't know, the "Submit query" button doesn't do anything [21:25:11] I mean, I get the "bad database name" alert when clicking "Submit query" so I don't think it's even trying [21:25:24] I mean what is in this box https://usercontent.irccloud-cdn.com/file/vlRBSn1t/Screen%20Shot%202021-03-23%20at%202.25.01%20PM.png [21:25:34] Is it metawiki or is it something else? [21:25:44] Like None, which would throw that error [21:25:56] I see "None" in that box [21:26:04] You have to change that to the database name [21:26:38] It *should* fill that in for you on re-runs. [21:26:56] Oh. I never used this database box [21:27:13] its brand new! :) [21:27:14] That's new, so it will likely cause a lot of people some trouble [21:27:26] It was the best way I could come up with in order to make all this work right [21:27:34] Ok, fine. :) [21:27:56] My first thought seeing it was that I had failed to load some CSS because it's unstyled :D [21:28:24] Glad to know it's not my uMatrix breaking something [21:32:43] !log tools.lexeme-forms deployed 253aed283c (Latvian nouns) [21:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [21:45:50] !log quarry restarting quarry services for the meta_p and centralauth issue T264254 [21:45:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [21:45:55] T264254: Prepare Quarry for multiinstance wiki replicas - https://phabricator.wikimedia.org/T264254 [22:31:38] ragesoss: your 200GB cinder quota has been granted. https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances#Cinder attempts to tell you how you can make use of it.