[09:56:29] <arturo>	 !log tools make jhernandez (IRC joakino) projectadmin (T278975)
[09:56:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:56:37] <stashbot>	 T278975: Grant Joaquin Hernandez tools & clouddb-services admin rights - https://phabricator.wikimedia.org/T278975
[09:56:42] <arturo>	 !log clouddb-services make jhernandez (IRC joakino) projectadmin (T278975)
[09:56:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL
[11:29:39] <Majavah>	 !log tools.wikibugs restart irc relay to get it to re-join #wikimedia-releng
[11:29:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[11:29:58] <Majavah>	 denied: host "tools-sgebastion-08.tools.eqiad.wmflabs" is not an admin host
[11:30:31] <Majavah>	 !help did someone break the grid? ^
[11:30:31] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[11:30:50] <arturo>	 hopefully not :-)
[11:31:04] <arturo>	 Majavah: let me check
[11:34:19] <arturo>	 Majavah: I just submitted a job form that host, no problem :-S
[11:34:53] <arturo>	 submitted/run/deleted without problems
[11:35:00] <Majavah>	 arturo: it does not let me do that, tested both dev. and login.toolforge.org
[11:35:12] <arturo>	 wait, admin host?
[11:35:46] <arturo>	 that host is not supposed to be an admin host
[11:35:59] <arturo>	 it should be a simple submit host
[11:36:03] <arturo>	 what operation are you running Majavah ?
[11:36:26] <arturo>	 the error message is legit 
[11:37:33] <Majavah>	 arturo: I tried "qmod -rj wb2-irc" based off wikibugs documentation, just manual qdel seems to be working fine
[11:37:52] <arturo>	 👍
[11:38:01] <Majavah>	 sorry for the false alarm
[11:44:54] <arturo>	 np
[13:40:41] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed f5439f66a2 (l10n updates)
[13:40:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[15:06:54] <bstorm>	 Majavah: You probably figured it out already, but qmod commands no longer work outside of the grid masters because of fiddling I did with the security model. We updated wikitech docs, but I'm sure there's still things recommending the command (or qconf, which also won't work in most cases).
[15:07:07] <bstorm>	 bastions used to be admin hosts
[15:39:21] <chicocvenancio>	 hey zhuyifei1999_  a few users complaining of errors in v2c in the past few days
[15:40:04] <chicocvenancio>	 looking at logs in `enconding06` I see some OOMs
[16:37:32] <Urbanecm>	 zhuyifei1999_: chicocvenancio: +1 to that,  there is a large spike of server side upload requests. Not sure if related.
[16:39:21] <chicocvenancio>	 Urbanecm: the ssu were a result of users noticing google stoped blocking v2c
[16:39:40] <Urbanecm>	 oh
[16:39:44] <chicocvenancio>	 not sure if it broke for the same reasons, seems the ooms don't correlate with the errors... 
[16:40:26] <Urbanecm>	 so that's why it's a large bunch of youtube video requests came through...
[16:40:28] <Urbanecm>	 makes sense
[16:41:20] <chicocvenancio>	 yeah, a few users had noticed a few months ago. A few power users noticed it a few weeks ago
[16:41:31] <Urbanecm>	 chicocvenancio: tbh if the requests keep adding with the same speed it'll quickly get to an unsustainable state. Right now I have 16 pending requests (and 1 request is about ~30 videos)
[16:41:44] <Urbanecm>	 do you expect the number of SSU reqs will get back to normal soon?
[16:42:18] <chicocvenancio>	 I expect there to be a very large backlog of freely licensed videos in youtube
[16:42:54] <chicocvenancio>	 I think if they end up being larger than 1GB video2commons fails to upload and goes for the ssu
[16:43:42] <Urbanecm>	 ...
[16:44:04] <Urbanecm>	 do we know why video2commons fails to upload files larger than ~1 GB chicocvenancio ?
[16:44:36] <chicocvenancio>	 I don't remember, but I think it is/was a pywikibot limitation
[16:45:46] <chicocvenancio>	 we could try to figure out other methods of upload that would be more reliable, but the whole of video2commons is a bit janky and brittle 
[17:02:57] <bstorm>	 !log tools chowned the data volume for the docker registry to docker-registry:docker-registry
[17:03:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[17:49:09] <Urbanecm>	 chicocvenancio: I would really apreciate if you can figure that out, or at least linking the blocker task(s) to me, so I'm aware of it
[18:02:32] <brennen>	 so i have an instance (gitlab, in the gitlab-test project) that i'd like to resize for more VCPU.  complication: it's a g2.cores4.ram8.disk80, not disk20, i think because i used the deprecated method here for adding disk space: https://wikitech.wikimedia.org/wiki/Help:Adding_Disk_Space_to_Cloud_VPS_instances#With_LVM_(deprecated_as_of_February,_2021)
[18:03:00] <brennen>	 is there any sensible method for resizing that, or am i better off just recreating from scratch?
[18:17:08] <brennen>	 (ah, seems like i can use a different VM in that project, disregard.)
[19:07:39] <Majavah>	 !log tools.wikibugs restart phab listener, had crashed due to invalid json message
[19:07:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[19:08:10] <RhinosF1>	 Majavah: no auto recover?
[19:08:54] <Majavah>	 RhinosF1: no idea on how it works, I just have access so I can restart things if they fail, and first time doing that
[19:09:35] <RhinosF1>	 Majavah: heh, I'm surprised. Normally with json it's easy to just try except and log/ignore if fail then crash
[20:11:38] <legoktm>	 Majavah: can you file a bug please? it really should catch those kinds of issues
[20:17:43] <legoktm>	 !log tools.wikibugs restarted wb2-phab
[20:17:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[20:17:59] <Majavah>	 legoktm: https://phabricator.wikimedia.org/T279383
[20:22:21] <legoktm>	 Majavah: now that I read scrollback, there's a bug filed that `qmod -rj` no longer works and we need to update wikibugs to jstop,jstart instead but haven't gotten around to it yet
[20:45:38] <bd808>	 `qmod -rj` hasn't worked for like 2 years :)
[20:48:19] <wm-bot>	 !log tools.jouncebot <bd808> Restarting to test cherry-pick of [[gerrit:677017]]
[20:48:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL
[20:53:50] <wm-bot>	 !log tools.jouncebot <bd808> Restarting after updating to e956d87
[20:53:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jouncebot/SAL
[21:40:32] <fourty-six>	 hello i have a question about wikiproject watchlists
[21:40:43] <fourty-six>	 so https://toolserver.org/~dispenser/cgi-bin/transcluded_changes.py/Template:WikiProject_Wisconsin is down and i need help
[21:40:55] <zhuyifei1999_>	 Urbanecm: how big are the files that went for SSU?
[21:41:41] <mutante>	 fourty-six: the error message seems to imply they are not running it on cloud infra anymore but external
[21:41:55] <bd808>	 fourty-six: Dispenser's tools run somewhere random. You will have to track them down for help.
[21:42:03] <fourty-six>	 ah that sucks
[21:42:08] <fourty-six>	 thanks for your help
[21:43:29] <Dispenser>	 I just haven't found the time lately
[22:19:38] <chicocvenancio>	 zhuyifei1999_: arround 1GB
[22:25:19] <zhuyifei1999_>	 I see lots and lots of "APIError: stashfailed: Internal error: Server failed to publish temporary file." in the logs
[22:25:53] <zhuyifei1999_>	 eg:
[22:25:53] <zhuyifei1999_>	 [2021-04-02 05:10:49,892: VERBOSE/ForkPoolWorker-17] API Error: query=
[22:25:53] <zhuyifei1999_>	 u"{u'ignorewarnings': [False], u'maxlag': ['5'], u'format': [u'json'], 'filekey': [u'188hwbltwq4g.wtnm03.3726108.webm'], u'watch': [False], u'assert': [u'user'], 'token': [u'7f0dc2e2de4
[22:25:53] <zhuyifei1999_>	 e128e3803a92d2549d9c76066a1d2+\\\\'], 'checkstatus': [True], 'action': [u'upload']}"
[22:25:53] <zhuyifei1999_>	 [2021-04-02 05:10:49,892: VERBOSE/ForkPoolWorker-17]            response=
[22:25:53] <zhuyifei1999_>	 {u'servedby': u'mw1290', u'error': {u'info': u'Internal error: Server failed to publish temporary file.', u'code': u'stashfailed', u'help': u'See https://commons.wikimedia.org/w/api.php
[22:25:53] <zhuyifei1999_>	  for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and br
[22:25:54] <zhuyifei1999_>	 eaking changes.'}}
[22:25:59] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/jBhx4jTS/
[22:36:00] <chicocvenancio>	 zhuyifei1999_: which log is that?
[22:36:24] <zhuyifei1999_>	 this is /var/log/v2ccelery/celery1.log on encoding04
[22:36:49] <zhuyifei1999_>	 the logs are distributed across /var/log/v2ccelery/celery[12].log on encoding0[456]
[22:47:26] <chicocvenancio>	 I think we're getting a lot more ffmpeg errors, but not sure they're properly logged
[22:48:10] <chicocvenancio>	 https://www.irccloud.com/pastebin/Q9s24xSn/
[23:03:46] <Urbanecm>	 zhuyifei1999_: from 1 GB to 4GB
[23:11:15] <zhuyifei1999_>	 I'll look into ffmpeg errors later tonight
[23:11:25] <zhuyifei1999_>	 might be some ffmpeg parameter