[00:41:35] bd808: nobody read auth.log anyways :P [00:42:25] (when we really need the logs they are... sometimes rotated out) [00:43:32] there's a not-well-known syslog aggregation server that I think hangs on to stuff for longer than the user-facing instances [00:45:17] hmm.. /me didn't know that [06:36:02] bstorm_: I'd like to restart dump-torrents at some point...what's the best way for me to monitor that I'm not overloading NFS? I added in a small sleep (https://phabricator.wikimedia.org/R2037:74f53c88a509ac0a8874fe4edeaa48b7d7243ae9) and nice to hopefully reduce the impact caused by the script [06:41:06] Hmm. Poking around for a good resource [06:43:46] It's tricky because the numbers are terrible as is right now. [06:43:55] https://grafana.wikimedia.org/d/000000568/labstore1004-1005?orgId=1 <-- can you see that one? [07:27:42] grafana is public [14:38:43] !log yandex-proxy shutting down yandex-proxy01 as discussed in T212306 [14:38:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Yandex-proxy/SAL [14:38:46] T212306: Can we delete the VPS project 'yandex-proxy'? - https://phabricator.wikimedia.org/T212306 [15:16:47] legoktm: probably the best indicator of NFS health is iowait across Toolforge, even though that's a pretty crappy indicator [18:39:15] I agree with that. You can use that grafana link to watch for a numerical jump, but ultimately, everything will see iowait if it's broke [20:53:07] !log zoranzoki21wiki Installed mediawiki again, webservice started [20:53:08] Zoranzoki21: Unknown project "zoranzoki21wiki" [20:53:08] Zoranzoki21: Did you mean to say "tools.zoranzoki21wiki" instead? [20:53:17] !log tools.zoranzoki21wiki Installed mediawiki again, webservice started [20:53:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL [20:58:13] !log tools.zoranzoki21wiki Installed extension Duplicator and updated database via update.php - for resolving and testing T212044 [20:58:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL [20:58:15] T212044: Convert Duplicator to use extension registration - https://phabricator.wikimedia.org/T212044 [20:59:20] !log tools.zoranzoki21wiki update.php: Done in 5.8 s. [20:59:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL [22:11:10] !log tools.zoranzoki21wiki Installed ConfirmAccount extension on tools.zoranzoki21wiki [22:11:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL [22:29:37] did the US tech help irc hour get cancelled? [22:33:00] tgr: it did. I need to send out an email to you and other past hosts about that. The new plan if for WMF folks to help co-host in the earlier time slot [22:33:31] ...which is at seven in the morning. [22:34:15] if you are in SF yes... which is something like 10% of our developer workforce [22:35:10] there are 2 reasons for the change: 1) low volume of questions at the newer time, 2) changes at WMDE that will give them less capacity to staff the weekly event [22:35:47] the ideal plan from Birgit is that we will have a 50/50 split of WMDE and WMF folks for the weekly meeting [22:36:08] yeah, I can understand the reasons (and there wasn't such huge interest from the host side either) [22:36:51] for hosts it doesn't really matter how many time zones are covered as long as there are enough people to keep it running [22:36:58] for the askers it does though [22:38:54] tgr: I honestly think we would be better served by investing in discourse and async Q&A [22:39:43] I don't disagree, but they are somewhat similar use cases [22:39:48] real time help is nice, but it costs a lot and when it happens in a forum that is bad at searchable history that cost gets paid over and over [22:40:09] being able to have a conversation instead of a back and forth over a week can be very useful sometimes [22:41:12] a good async forum has way more impact for sure, scales better, is more accessible wrt devices and languages, etc [22:43:40] hi! was there just any kind of puppet config change or maintenance on VPS? [22:47:10] musikanimal: vary vague questions get very vague answers [22:47:46] there are Puppet changes all the time, but no maintenance that I am aware of [22:47:51] I was still trying to figure what the heck was going wrong, all things XTools broke [22:47:54] but I think I figured it out [22:48:02] `MariaDB [enwiki_p]> SELECT * FROM meta_p.wiki;` [22:48:08] `Empty set (0.00 sec)` [22:48:13] ?? [22:48:39] hmmm... andrewbogott was running some db update scripts. Is that still happening now? [22:49:10] hmmm [22:49:14] the process of updating the meta_p view is to drop and then recreate the data [22:49:33] which means that it does interrupt things briefly [22:50:05] looks still empty to me [22:50:24] on 1009 [22:50:27] it's filled on 1010 [22:50:34] yeah, the web cluster one is empty [22:50:54] andrewbogott: wanna kick of a maintain-meta_p on 1009? [22:51:00] *kick off [22:51:01] ok, let's see if I can convince it to regenerate [22:51:56] * bd808 imagines a secure chat-ops interface that would let him type maintain-meta_p@labsdb1009 and see the bot do the needful [22:53:38] bd808, musikanimal, I think that bit is fixed now [22:53:45] thank you! :D [22:54:03] andrewbogott: thanks [22:54:20] that was really hard to debug! I thought it was all on my end at first, but then saw xtools-dev was broken too [23:12:40] !log tools.zoranzoki21wiki Removing extension Duplicator from zoranzoki21wiki [23:12:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL [23:13:16] !log tools.zoranzoki21wiki Removing extension Duplicator from zoranzoki21wiki - done at 2.1 seconds [23:13:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zoranzoki21wiki/SAL