[01:11:42] <Reedy>	 !log tools.wikibugs restarted wikibugs because of apparent irc split brain (wikibugs and wikibugs____)
[01:11:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[08:54:33] <arturo>	 !log tools merged several patches by bryan for toolforge front proxy (cleanups, etc) example: https://gerrit.wikimedia.org/r/c/operations/puppet/+/622435
[08:54:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[10:02:33] <Iluvatar_>	 Hello, world! I got a message for my bot: "Someone (probably you) recently logged in to your account from a new device". Has the software configuration changed on Toolforge or has my account been hacked?
[10:31:20] <wm-bot>	 !log tools.wdmm <lucaswerkmeister-wmde> deployed f58fa33946 (wuuwiki update)
[10:31:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wdmm/SAL
[10:32:53] <arturo>	 Iluvatar_: perhaps joakino can help you navigate that
[10:50:41] <joakino>	 Iluvatar_: have you logged out and in, or had to log in again recently, or logged in from a different browser, computer, or device like a phone?
[11:00:16] <joakino>	 if you haven't, I would strongly recommend changing the password as soon as possible
[11:01:13] <joakino>	 the message looks like the one from LoginNotify for the Wikimedia accounts, so think if you have logged in into any wikis around the time when you got the email
[15:24:29] <zhuyifei1999_>	 andrewbogott: could you hard reboot video-{dev,redis}-buster.video.eqiad.wmflabs? I can't ssh in and the dev one reports insane load (NFS?) and the redis one reports down
[15:35:11] <andrewbogott>	 zhuyifei1999_: in a meeting but I'll have a look in maybe 30 minutes?
[15:35:18] <zhuyifei1999_>	 ok sure
[16:16:27] <andrewbogott>	 zhuyifei1999_: want me to investigate anything first or just reboot?
[16:17:10] <andrewbogott>	 heh, load average: 174.07, 174.02, 174.04
[16:17:31] * andrewbogott reboots
[16:30:12] <zhuyifei1999_>	 ikr
[16:32:11] <zhuyifei1999_>	 andrewbogott: can you reboot the redis instance too?
[16:32:25] <andrewbogott>	 zhuyifei1999_: I did but it didn't come up right, trying again now
[16:32:51] <andrewbogott>	 or possibly it was too broken to honor my reboot
[16:33:00] <andrewbogott>	 seems ok now?
[16:33:15] <andrewbogott>	 I'm doing some puppet repair on those hosts too while I'm here...
[16:34:14] <andrewbogott>	 zhuyifei1999_: since you're here… do you have an intuition about whether the encoding instances are throttled on CPU, disk IO, something else?
[16:34:28] <zhuyifei1999_>	 CPU
[16:34:30] <andrewbogott>	 I'm interested in cloud-vps workloads that are sensitive to disk IO
[16:34:36] <andrewbogott>	 (apart from databases which almost certainly are)
[16:34:48] <andrewbogott>	 ok, so slower disk access would
[16:34:55] <andrewbogott>	 wouldn't mess with you a whole lot?
[16:35:30] <andrewbogott>	 Do you have consistent enough performance numbers that you could tell if it got worse or better?  If we created an experimental encoding node on a different system?
[16:38:02] <zhuyifei1999_>	 I don't think I saw a lot of D waits iirc
[16:38:22] <zhuyifei1999_>	 sure we can get a different node running
[16:38:56] <zhuyifei1999_>	 but I don't really know performance counters :/
[16:48:19] <andrewbogott>	 ok — I'll hold off on that for now then
[16:48:30] <andrewbogott>	 I think your broken VMs are un-broken for the moment
[16:51:44] <zhuyifei1999_>	 thanks :)
[17:12:48] <bstorm>	 !log admin Running 'ionice -c 3 nice -19 find /srv/tools -type f -size +100M -printf "%k KB %p\n" > tools_large_files_20200826.txt' on labstore1004 T261336
[17:12:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[17:12:51] <stashbot>	 T261336: 2020-08-26: tools NFS share cleanup - https://phabricator.wikimedia.org/T261336
[17:41:59] <bstorm>	 !log tools.paws deleting data in the old NFS location of paws/userhomes T261338
[17:42:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws/SAL
[17:42:01] <stashbot>	 T261338: Remove PAWS data in Tools volume - https://phabricator.wikimedia.org/T261338
[18:12:39] <wm-bot>	 !log tools.zoomviewer <root> Added `-o /dev/null -e /dev/null` to jsub commands generated by index.php (T248188)
[18:12:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoomviewer/SAL
[19:24:32] <mutante>	 for completeness, the "308" redirect question came from https://phabricator.wikimedia.org/T256276 which said to use 308
[20:22:13] <bd808>	 thanks for the follow up mutante :)
[20:35:43] <wm-bot>	 !log tools.zoomviewer <root> Disabled crontab for "jlocal echo ${HOME}/check.sh >> /tmp/debugomat"
[20:35:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoomviewer/SAL
[20:36:38] <wm-bot>	 !log tools.zoomviewer <root> Changed schedule of check.sh crontab to once per 5 minutes
[20:36:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoomviewer/SAL
[20:39:08] <wm-bot>	 !log tools.zoomviewer <root> Deleted old access.log files (6G total size)
[20:39:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoomviewer/SAL
[21:08:00] <bd808>	 !log tools Disabled puppet on tools-proxy-06 to test fixes for a bug in the new T251628 code
[21:08:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[21:08:05] <stashbot>	 T251628: Serve some default well known files for Toolforge webservices - https://phabricator.wikimedia.org/T251628
[21:29:08] <wm-bot>	 !log tools.heritage <jeanfred> Deploy latest from Git master: 7b0a9d7
[21:29:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL
[21:29:39] <wm-bot>	 !log tools.heritage <jeanfred> jsub new categorize_images job to test 7b0a9d7
[21:29:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL
[22:16:13] <bstorm>	 !log tools.robokobot truncated file virgule.err since it had reached 97G in size and was just repeated output from a job (not actually errors)
[22:16:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.robokobot/SAL