[00:12:07] I think I can depool one or two servers during the migration [00:16:15] is it possible to request a rename on wikitech? [00:33:57] apap: see https://wikitech.wikimedia.org/wiki/LDAP/Renaming_users [00:34:11] thanks [00:34:50] I don't have that "super special permissions" unfortunately so you may have to create a phab ticket [01:03:07] eh i probably wont do it because of the consequences... [08:13:02] !log clouddb-services enable puppet in clouddb1001/clouddb1002 to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/546102 (T236384 , T236420) [08:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [08:13:08] T236420: ToolsDB unstable following unplanned software upgrade - https://phabricator.wikimedia.org/T236420 [08:13:08] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [10:37:09] !log clouddb-services clouddb1002 downgrading wmf-mariadb101 from 10.1.41-1 to 10.1.39-1 (T236384 , T236420) [10:37:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [10:37:18] T236420: ToolsDB unstable following unplanned software upgrade - https://phabricator.wikimedia.org/T236420 [10:37:19] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [10:45:21] !log admin icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) (T236384 , T236420) [10:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:45:27] T236420: ToolsDB unstable following unplanned software upgrade - https://phabricator.wikimedia.org/T236420 [10:45:27] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [10:45:29] !log clouddb-services icinga downtime toolschecker for 1 to upgrade clouddb1002 mariadb (toolsdb secondary) (T236384 , T236420) [10:45:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [10:46:36] !log clouddb-services (jynus) clouddb1002 mariadb (toolsdb secondary) being upgraded from 10.1.38 to 10.1.39 (T236384 , T236420) [10:46:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [10:49:25] !log clouddb-services (jynus) clouddb1002 mariadb (toolsdb secondary) being upgraded from 10.1.38 to 10.1.39 is done !(T236384 , T236420) [10:49:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [13:01:46] join #wikipedia-de-admins [13:01:49] Argh [13:24:31] !log tools.integraality Deploying from master, will break things. [13:24:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [13:57:44] !log wikispore created wikispore-prod for wikispore.wmflabs.org [13:57:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispore/SAL [13:58:33] is Horizon dead? [14:00:51] tgr: looks like there's an issue, I'll look into it and get back to you [14:00:52] yes, it looks like it [14:00:59] thanks, jeh [14:01:05] thx [14:03:51] !log openstack graceful restart apache on labweb1001 to resolve horizon 503 [14:03:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:08:21] !log tools.integraality Deploy latest from Git master: 6a055c2 (T228405) [14:08:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [14:10:20] !log openstack graceful restart apache on labweb1002 to resolve horizon 503 [14:10:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:12:10] tgr thanks for letting us know. Horizon should be available now. Looks like this might have been effected by some upgrade work on labweb1002. [14:21:21] thanks for the quick fix! [14:24:52] !log openstack `systemctl reset-failed` on labweb1001 and labweb1002 to cleanup hhvm service [14:24:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openstack/SAL [14:51:28] !log tools.wd-image-positions deployed 2d28f55912 (optimize cropper image loading) [14:51:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:02:42] !log tools.integraality Deploy latest from Git master: 82de2957 (T224212, T223930) [15:02:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [15:10:23] !log tools.wd-image-positions deployed 7a4982705c (improve headings) [15:10:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [15:18:49] tgr (or anyone) , what was the tool in cloud to be able to search mediawiki codebase? [15:19:26] cc bd808 [15:19:33] nuria: https://codesearch.wmflabs.org/search/ [15:19:43] OOHHH Reedy superstar [15:20:18] someday™ we will have a nicer directory to find these things... someday. [15:22:11] Reedy IS the directory [15:22:19] !log wikispore made wikispore-prod temporarily its own puppetmaster to work around T236455 [15:22:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispore/SAL [15:22:23] T236455: Cloud-vagrant: lxc.net must not have a value - https://phabricator.wikimedia.org/T236455 [15:26:33] nuria: heh, he might be one of those Telephone operators from when I was young. "Operadora, conferencia con xxx por favor" :P [15:27:45] bd808: Hay's directory and the one at /admin is nice [15:27:59] but if you have a better one in mind :D [15:31:37] hauskater: https://meta.wikimedia.org/wiki/Toolhub [15:31:55] Hay's is nice, but not a lot of people publish to it [15:32:09] * hauskater whisles innocently [15:32:45] fact is I don't have a tool that people can use so I don't think I'm in violation here [15:32:55] I just use TF to run a bot [16:15:02] !log toolsbeta rebooting toolsbeta-test-k8s-worker-1 and -2 [16:15:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [16:47:49] !log tools.deadlinks Reverted local hack that disabled /api/ route (T236423) [16:47:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.deadlinks/SAL [16:47:53] T236423: Restore db write functionality for https://tools.wmflabs.org/deadlinks/api/ - https://phabricator.wikimedia.org/T236423 [16:48:36] musikanimal, Cyberpower678: ^ the deadlinks api should be working again now. I'm going to leave the task open for a bit longer just incase things go badly and we need to turn it back off. [16:56:04] matanya: if you're around, I'd like to talk briefly about your video rebuild plans [17:44:50] bd808: around two days ago this happened: https://commons.wikimedia.org/wiki/Commons:Village_pump#video2commons . afaict instances behind the cloud NAT (with exceptions like encoding0[1-3] as of yesterday, no idea why) seem to get 'HTTP Error 429: Too Many Requests' when trying to access youtube with youtube-dl. I PM'ed matanya last night about it and he suggested to get us whitelisted and that the foundation might have some [17:44:50] contacts in google. do you know whom to ask? [17:46:52] zhuyifei1999_: hmmm... I think I would start with kaldari. I know he has talked with Google about other quotas and things before on behalf of the Community Tech team. [17:47:28] ok [17:47:48] @seen kaldari [17:47:48] zhuyifei1999_: Last time I saw kaldari they were talking in the channel, they are still in the channel #wikisource at 10/7/2019 5:51:58 PM (17d23h55m50s ago) [18:08:27] bd808: I don't think CommTech is using it anymore, so that'd be all Cyberpower678. I believe in the beginning we just wanted to see the impact IABot had, but I think now it's obvious that it's very big! [18:08:38] Cyberpower or Internet Archive might still be using it though, not sure. [21:06:53] !log tools.wd-image-positions deployed temporary experiment (clear session on OAuth callback error) [21:07:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [23:41:32] !log toolsbeta Deployed custom webhook controllers for registry and ingress checking to toolsbeta-test kubernetes cluster T215531 T215678 T234231 [23:41:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [23:41:39] T215531: Deploy upgraded Kubernetes to toolsbeta - https://phabricator.wikimedia.org/T215531 [23:41:40] T234231: Toolforge ingress: decide on how ingress configuration objects will be managed - https://phabricator.wikimedia.org/T234231 [23:41:40] T215678: Replace each of the custom controllers with something in a new Toolforge Kubernetes setup - https://phabricator.wikimedia.org/T215678