[03:36:08] Looks like the OAuth bug in ipcheck is squashed, and the old/new URL's are passing args now. [03:47:00] ...Not really the channel I meant, but yay in any case! LOL [08:54:56] bd808: the commonswiki lag is now 13 hours, do you know what's happening? I don't know what to search for in phabricator [08:58:50] don-vip: what kind of lag? a user edits wiki and another user still sees the old page? or something else [08:59:19] or you mean db replicas in cloud or something [08:59:28] mutante: db replica [09:00:06] I see only files imported at least 13 hours ago, yesterday evening it was only 8 hours, the lag increased this night [09:01:07] don-vip: i see. so like the old ticket https://phabricator.wikimedia.org/T75980 [09:01:24] don-vip: can you create a new ticket just like that and maybe link it? [09:01:51] i was about to say reopen it but it's from 2014 and then it would have the wrong assignee [09:02:38] i asked about the wiki lage because that was a thing .. but entirely different thing then [09:02:42] lag [09:03:20] mutante: I think DBAs are already aware of it, it might just be normal maintenance. bd808 said they were closely monitoring this kind of things [09:04:33] don-vip: i can't see any ticket about it tagged with DBA [09:05:40] let me search server admin log about maintenance [09:06:09] since you asked what to search for in phab.. the fix for that would be to make one and then get comments there [09:07:17] the closest i can see so far is https://phabricator.wikimedia.org/T250055 [09:07:57] or you can ask in #wikimedia-databases [09:13:41] don-vip: i asked the DBAs in their channel: [09:13:45] 09:12 < marostegui> we are going to try to upgrade it next week and see if it helps [09:13:48] 09:12 < marostegui> but yes, labsdb hosts are overloaded [09:13:50] 09:12 < marostegui> specially quarry's one [09:13:58] don-vip: it's a known issue related to quarry [09:14:17] but it's not maintenance [09:35:32] mutante: thanks! [09:48:28] don-vip: yw [10:21:24] bd808: Outstanding work, thank you so much! [11:00:55] mutante: lag is quickly decreasing right now :) [11:01:58] don-vip: great! :) [14:31:33] What are the criterias for having a subproject under Tools at phabricator? [14:58:51] iridescence: basically just asking for one. :) See https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects for instructions [14:59:40] someday™ I will get around to making this a one-click process for tools using https://toolsadmin.wikimedia.org/ [15:01:49] iridescence: T144111 is the feature request to make this self-service [15:01:51] T144111: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111 [15:02:23] "only" 3.5 years old. Practically a brand new idea! ;) [15:18:23] bd808: Component alone should be sufficient? [15:19:16] * Majavah hates installing python-ldap for Windows [15:20:50] iridescence: yes, the "component" type is the right one to ask for. See https://phabricator.wikimedia.org/project/subprojects/703/ for existing tool components. We recommend "tool-" as the name as well. That comes from discussion at T144111 [15:20:52] T144111: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111 [15:21:28] But as you can see there are a number of components that do not follow that pattern for various reasons [15:24:34] bd808: how does one override values of default striker/striker.ini in a local env? the comments seem to suggest that I can place a striker.ini in project root but that file is not in .gitignore [15:29:48] Majavah: /etc/striker/striker.ini is how we do it in the MediaWiki-Vagrant role for Striker development. I would highly encourage you to use the MWV setup for local development. It takes care of a whole lot of things that would be a huge pain to setup manually. [15:30:09] https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org#Local_development [15:30:22] oh mw-vagrant also has striker? I never knew that it supported anything else than MW itself :D [15:30:42] If I have written code for it, it probably has an MWV role ;) [15:31:19] that Wikitech would have been really helpful to include in the project readme.rst or on Phab project description or anywhere where else :D [15:37:45] Majavah: maybe a good first patch! [15:38:08] The readme predates the wiki docs by many months [15:45:27] bd808: {{done}} at https://gerrit.wikimedia.org/r/#/c/labs/striker/+/591369, want a phab task for it? [15:46:43] bd808: Should it be under vps-projects? https://phabricator.wikimedia.org/T250819 [15:46:49] Majavah: no task needed. I'll +2 as soon as zuul/jerkins wake up :) [15:48:10] iridescence: no, not for a Toolforge tool. vps-projects is for things that are directly hosted on Cloud VPS. This is all a bit confusing because Toolforge is itself a Cloud VPS project, but provides another layer of abstraction from that to the tools it hosts. [15:48:32] iridescence: I'll correct andre_ :) [15:48:47] bd808: I was faster correcting andre :D [15:49:37] * bd808 will just create the project :) [15:51:57] iridescence: https://phabricator.wikimedia.org/tag/tool-moedata/ [15:52:07] start your backlog! :) [15:52:18] bd808: Nice, thanks again :) [15:54:01] bd808: the patch seems to be failing on some dependency trying to use python 3.6 format strings [15:54:31] Majavah: ugh [15:55:00] that's my problem to fix I suppose. The core is py3, but probably py3.5 only at the moment [15:55:27] I should have done better pinning in the top level requirements.txt. This has happened before [15:56:03] * bd808 is not a fan of f-strings yet [15:56:16] boo, fstrings rule [15:56:54] (in many cases, not always. {} based formatting gets annoying with wikitext) [15:57:39] my only experience with them was when pycharm suggested that I should use them and then my bot broke because grid engine only has 3.5 [15:58:17] * AntiComposite decided to only write tools for 3.7+ at this point [15:59:50] * bd808 likes .format() for explicit over implicit reasons [16:01:24] * Majavah is patiently waiting for mediawiki-vagrant to install phabricator [16:01:27] !log admin restart cloudceph mon and osd services for openssl upgrades [16:01:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:02:29] Majavah: heh. the initial provision for that role is s l o w [16:21:31] "default: Notice: /Stage[main]/Phabricator/Phabricator::Config[metamta.mail-adapter]/Exec[phab_set_metamta.mail-adapter]/returns: Usage Exception: Configuration key "metamta.mail-adapter" is unknown. Use "bin/config list" to list all known keys." [16:21:37] hmmm [16:23:12] Majavah: *nod* those kinds of drifts happen in the puppet role. I don't redeploy that from scratch very often [16:23:43] * Majavah comments that out [16:23:54] I was going to suggest that :) [16:39:27] bd808: exact same error on vagrant than on jenkins :/ [16:43:45] Majavah: I think the failing lib is django-reversion-compare. I guess we just need to tighten up the version pinning for that in requirements.txt [16:46:03] Anyways I have to leave now so I'll continue working on T144111 later today/tomorrow [16:46:04] T144111: Allow self-service creation of Maniphest projects for Tools - https://phabricator.wikimedia.org/T144111 [16:56:36] This is a test [16:56:38] !help [16:56:38] bstorm_: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [16:58:07] I trust: bd808!.*@wikimedia/BDavis-WMF (2admin), .*@wikimedia/andrew-bogott (2admin), .*@wikimedia/mviswanathan-wmf (2admin), .*@wikimedia/.* (2trusted), .*@mediawiki/.* (2trusted), .*@wikipedia/.* (2trusted), [16:58:07] @trusted [17:00:46] !help is If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [17:00:46] This key already exists - remove it, if you want to change it [17:01:03] !help del [17:01:04] Successfully removed help [17:01:06] !help is If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [17:01:07] Key was added [17:01:18] !help | bstorm_ [17:01:19] bstorm_: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [17:01:25] better? [17:08:47] neat [17:11:10] Cool! [17:11:21] That'll probably help quite a bit [17:11:27] thanks bd808 [21:07:40] !log tools.lexeme-forms deployed 6634452b4c (increase uWSGI buffer) [21:07:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [22:01:58] !log draining tools-k8s-worker-38 and tools-k8s-worker-52 as they are on the crashed host T250869 [22:01:59] bstorm_: Unknown project "draining" [22:02:00] T250869: cloudvirt1004 failed RAID controller - https://phabricator.wikimedia.org/T250869 [22:02:20] oops [22:02:31] !log tools draining tools-k8s-worker-38 and tools-k8s-worker-52 as they are on the crashed host T250869 [22:02:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:09:16] !log tools depooling tools-sgewebgrid-lighttpd-0918/9 and tools-sgeexec-0901 T250869 [22:09:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:09:21] T250869: cloudvirt1004 failed RAID controller - https://phabricator.wikimedia.org/T250869 [22:12:37] !log admin moving cloudvirt1004 out of the 'standard' aggregate and into the 'maintenance' aggregate [22:12:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:01:44] Majavah: I figured out the requirements.txt change that was needed. Sneaky library added f-string dependency between 0.8.x and 0.9.0 with no release notes. [23:06:34] !log tools repooled tools-k8s-worker-38/52, tools-sgewebgrid-lighttpd-0918/9 and tools-sgeexec-0901 T250869 [23:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:06:39] T250869: cloudvirt1004 failed RAID controller - https://phabricator.wikimedia.org/T250869 [23:14:51] I really wish there was a good way to include python versions as a dependency [23:15:05] Pipfiles have it, but they're not supported by pip yet