[02:14:13] !log admin downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping T281045 [02:14:13] T281045: labstore1007 crashed after storage controller errors--replace disk? - https://phabricator.wikimedia.org/T281045 [02:14:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [02:14:16] T281045: labstore1007 crashed after storage controller errors--replace disk? - https://phabricator.wikimedia.org/T281045 [02:23:24] Anyone knows what's up with FontCDN on wmflabs? CSS files (https://tools-static.wmflabs.org/fontcdn/css?family=Roboto) are being served properly but font files (e.g. https://tools-static.wmflabs.org/fontcdn/s/roboto/v27/KFOmCnqEu92Fr1Mu72xKOzY.woff2) are 502'ing. [02:24:21] The mess can be visualized on https://fontcdn.toolforge.org/ where everything is just browser fallback font (for me, at least) [02:39:00] chlod: I'll see if I can figure anything out quickly. We may need to open a bug and look more tomorrow (/me is getting tired) [02:39:25] bd808: Got it. Thanks! [02:42:47] "no live upstreams while connecting to upstream" in the nginx error log. Now to check what it's really doing [02:51:55] !log tools Restarted nginx on tools-static-14 to see if that clears up the fontcdn 502 errors [02:51:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [02:52:29] chlod: turning it off and on seemed to work. It's like I'm a pro SRE or something :) [02:53:02] thank you so much! :) [04:33:28] !log tools.bridgebot Added #countervandalism bridge (T283297) [04:33:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [04:34:56] I wonder what we need to change to make the wm-bot on Libera.Chat relay the dologmsg notices instead of the Freenode bot? [05:16:50] !log tools.bridgebot Restarted to pick up higher cpu and ram limits [05:16:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [09:48:32] !log tools.lexeme-forms deployed 44812d4446 (add Portuguese modal adverb) [09:48:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [09:52:52] !log tools.lexeme-forms deployed 31e85bafcf (l10n updates) [09:52:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [10:32:47] !log tools.ranker deployed 38b8f68887 (cross-links between batch modes) [10:32:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [12:59:41] !log tools.integraality Deploy d4fa363 (T283287) [12:59:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [13:00:34] !log tools.integraality Deploy 633eeb0f [13:00:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [14:02:04] !log tools.integraality Deploy 69048ef [14:02:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [14:33:21] !log tools.pbbot Migrate webservice to k8s (445ea07) [14:33:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pbbot/SAL [14:52:36] !log tools.translate-link deployed 2e020b7d3e (first version with useful index page; first-ever version was about twenty minutes earlier, forgot to log it) [14:52:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [15:59:58] !log tools.integraality Deploy 9727a01c [16:00:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [16:06:15] !log tools.integraality Deploy 0966be04 (T279236) [16:06:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [16:44:49] !log tools.stashbot Restart Freenode bot to part from non-priority channels (T283263) [16:46:59] stashbot: ?? [16:48:13] non-priority? [16:50:12] majavah: no I broke it :) [16:50:41] oh, non-priority means channels where !log doesn't work [16:53:25] !log tools.translate-link deployed 1b957ac4e0 (more robust message matching) [16:53:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [16:57:45] !log tools.integraality Deploy 6de2308 and c436760 (T279236) [16:57:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [17:28:16] !log tools.integraality Deploy 8ffbad82c3 (T279236) [17:28:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [19:00:17] test [19:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [19:34:49] there we go ^.^ [20:03:37] i'm trying to mount a cinder volume but it just doesn't work. could be a volume deleted not long ago be a problem here? [20:03:58] +with the same name [20:18:21] no dice, wth [20:21:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:24:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:33:38] gifti: hmmm... I haven't done much debugging with cinder volumes yet. andrewbogott or bstorm are either of you about? [20:35:18] gifti: can you be more specific than "doesn't work"? how are you doing that? any errors? [20:40:06] majavah: i had a 100G volume attached (it was mounted properly), i detached it, deleted it, created a new 80G one, attached it, forgot to edit fstab, ran wmcs-prepare-cinder-volume, edited fstab, ran the command again, it was successful, no errors, but it wasn't mounted, i tried it one more time, this time properly, same results, then i tried it with another volume name, same results again [20:47:27] i can mount it on another instance, so i think i broke something on the instance i want to attach it on [20:47:43] what does /etc/fstab look like? [20:48:42] UUID=5eb86d23-f988-42e6-affd-9cc9409a40cb/ext4defaults11 [20:48:42] /dev/vd/second-local-disk/srvext4defaults00 [20:48:43] UUID=308ee617-48e2-4ac9-8a38-a3821cb61793 /data ext4 nofail,x-systemd.device-timeout=2s 0 2 [20:49:26] uh [20:49:55] does that vm also have a lvm-based extra disk? [20:50:01] yes [20:50:09] hmm [20:50:19] i wanted to migrate some data off of it [20:50:43] it worked two months ago [20:57:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:58:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:58:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:59:12] gifti: sorry I missed the ping before, I can have a look if you're still here [20:59:54] can you tell me the fqdn of the affected instance? [21:02:18] andrewbogott: it's taxonbota-b.dwl.eqiad1.wikimedia.cloud [21:04:46] gifti: ok, I'm going to start by removing this line from fstab: 'UUID=308ee617-48e2-4ac9-8a38-a3821cb61793 /data ext4 nofail,x-systemd.device-timeout=2s 0 2' that seems like a relic from a previous cinder mount, does that sound right? [21:05:46] yes, that was my last failed attempt [21:06:18] ok [21:06:30] I'm looking at 'lsblk' and I don't see the volume presenting to the OS [21:06:41] it's currently somewhere else [21:06:47] should i reattach it? [21:07:11] this is the 'taxonbot' volume? Horizon shows it as attached [21:07:32] 'attached to: /dev/vdb on taxonbot-b' [21:07:40] it's taxonbota-test [21:08:59] horizon shows that one as '/dev/sdb on taxonbota-c' [21:09:18] that's right [21:09:37] ok, you've lost me then [21:09:54] i want to attach it to taxonbota-b [21:10:52] which i have done now [21:10:57] stop please [21:11:01] ok [21:11:22] do you not see in Horizon that the 'taxonbot' volume is already attached to taxonbot-b? [21:11:40] i do [21:11:46] Is it your intent to have both volumes attached at once? [21:11:48] how is that relevant? [21:11:54] no [21:12:20] it's all named pretty similarly [21:12:41] i want taxonbot on taxonbot-b and taxonbota on taxonbota-b [21:12:44] oh shit taxonbota-b and taxonbot-b are two different things [21:12:46] * wm-bb sights [21:12:56] ok, so now I see the volume in lsblk, that seems good [21:13:53] go ahead and try running wmcs-prepare-cinder-volume and let's see what it does :) [21:14:38] it asks me where i want to mount it [21:14:48] that seems good [21:15:16] i will say /data [21:15:49] ok [21:15:55] it's ready to prepare and mount and i continue [21:16:12] yep, go ahead [21:16:24] it says it mounts and updates fstab [21:16:27] and is done [21:17:04] there is no mountpoint in lsblk, no corresponding output in mount and nothing in /data [21:17:07] hm, fstab looks right but it didn't mount [21:18:28] yeah [21:18:38] and 'mount /data' returns success but doesn't do anything [21:19:39] did wmcs-prepare-cinder-volume say it was going to format things? [21:20:05] no [21:20:10] it was already formatted [21:20:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL [21:20:34] ok [21:21:41] I tried mounting it as /waffle instead of /data and it worked [21:21:50] huh [21:21:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL [21:21:55] so... that's interesting! No idea why it cares unless there's some residual state from something being mounted there before [21:22:20] Have you tried rebooting at any point in all this? (Is this a server that's safe to reboot?) [21:22:42] i did not and it's not safe to reboot it at all [21:23:08] i would have to coordinate with doctaxon for that [21:23:17] ok [21:23:27] then... I guess we treat /data as cursed for now :( [21:23:32] if it works with another path, i'm happy [21:23:45] I'm sorry I don't have an actual explanation [21:23:52] that's ok [21:23:59] thank you for your idea [21:24:31] the server will be scrapped soon hopefully [21:38:03] when I was mounting stuff I had to fiddle with systemd because it wanted to control the mounts [21:38:09] I don't remember what I actually did [21:38:25] mmmm systemd [21:38:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [22:04:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL [22:05:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL [22:35:10] chicocvenancio: do you think I can get rights to PAWS again if I wanted? [22:35:14] (not that I'm asking for it now...) [22:35:28] but sort of testing willingness of the cloud team :) [23:28:59] bd808: what do you think I need to do to request more rights for the PAWS deployment? [23:30:59] yuvipanda: file a phabricator task and have no-one with similae rights oppose within a week, basically [23:31:02] https://wikitech.wikimedia.org/wiki/Help:Access_policies [23:31:11] makes sense [23:31:14] yuvipanda: open a phab task, tag it with #wmcs-kanban, and let me know. :) [23:31:43]

this is a part of IRC you can't do anywhere else [23:31:51] god-damn-sucker: :) [23:32:14] god-damn-sucker: reminded me of https://bash.toolforge.org/quip/AVQ_K14_1EdYBLP_7qWd [23:33:36]

hahahahahahahah [23:37:09] bd808: https://phabricator.wikimedia.org/T283443 [23:39:39] no promises on actually doing anything tho [23:53:23] and hopefully https://github.com/toolforge/paws/pull/70 will setup automatic build + push of the user image, making chicocvenancio's life eaiser :) [23:56:26] yuvipanda: if you can trick^Wconvince madhu to come around too I'll give you root everywhere. ;) [23:56:39] * wm-bb misses the "old" gang [23:56:41] bd808: I'm 100% convinced that isn't going to happen :) [23:57:19] she's engineering management at hashicorp now, but really she wants to work on making clothes for a living eventually... [23:58:46] clothes are cool. bootstrapping physical manufacturing sounds meh, but I'm sure madhu can do it if she decides to do it