[00:05:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.62, 6.25, 5.28 [00:05:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.23, 6.29, 5.24 [00:07:25] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.88, 5.28, 5.05 [00:07:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.42, 5.64, 5.13 [01:02:57] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.11, 7.39, 4.04 [01:06:56] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.74, 3.95, 3.35 [01:08:55] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.32, 2.67, 2.94 [01:10:02] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 187s [01:11:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.48, 4.20, 2.36 [01:12:02] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 285s [01:13:27] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 2.27, 3.30, 2.25 [01:37:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is CRITICAL: CRITICAL - load average: 5.67, 4.36, 3.16 [01:42:04] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 33s [01:45:26] PROBLEM - dbbackup1 Current Load on dbbackup1 is WARNING: WARNING - load average: 2.05, 3.57, 3.47 [01:47:26] RECOVERY - dbbackup1 Current Load on dbbackup1 is OK: OK - load average: 1.00, 2.71, 3.17 [01:51:53] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [02:00:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.08, 2.75 [02:02:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.70, 2.84, 2.70 [04:40:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 219s [04:41:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.28, 2.75, 1.54 [04:47:32] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.92, 3.75, 2.41 [04:55:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.91, 4.10, 3.09 [04:59:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.18, 3.86, 3.25 [05:05:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.03, 3.88, 3.45 [05:06:09] [02mw-config] 07Universal-Omega opened pull request 03#3827: Disable wgCosmosEnabledRailModules['recentchanges'] on some wikis - 13https://git.io/JOTAj [05:07:08] miraheze/mw-config - Universal-Omega the build passed. [05:07:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.03, 3.61, 3.41 [05:09:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.01, 3.82, 3.51 [05:11:35] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.80, 3.54, 3.45 [05:19:32] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.68, 3.89, 3.62 [05:21:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.99, 3.56, 3.53 [05:27:35] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.43, 3.02, 3.32 [05:37:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.58, 3.49, 3.42 [05:39:34] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.57, 3.26, 3.36 [05:45:32] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.98, 3.35, 3.31 [05:47:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.23, 3.68, 3.45 [05:49:32] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.66, 3.32, 3.34 [05:56:26] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.91, 3.71, 3.51 [05:58:22] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.17, 3.80, 3.56 [06:00:17] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.57, 3.33, 3.42 [06:08:00] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.28, 3.64, 3.51 [06:09:55] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.22, 3.48, 3.47 [06:11:53] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.10, 3.25, 3.38 [06:21:39] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.49, 3.41, 3.40 [06:23:36] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.17, 2.97, 3.24 [06:33:34] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.38, 3.66, 3.41 [06:35:34] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.42, 3.22, 3.29 [07:03:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.50, 3.24, 2.96 [07:05:32] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.43, 3.30, 3.02 [07:07:34] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.35, 3.36, 3.08 [07:52:08] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [09:20:20] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:21:05] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 6.76, 5.48, 2.47 [13:23:05] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.17, 3.79, 2.22 [13:25:05] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.88, 2.78, 2.03 [14:02:10] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki zoneminderwiki /home/reception/delbackups2/zoneminderwiki.xml [14:02:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:04:55] [02puppet] 07paladox reviewed pull request 03#1726 commit - 13https://git.io/JOIee [14:05:32] [02puppet] 07paladox reviewed pull request 03#1726 commit - 13https://git.io/JOIet [14:06:17] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki glitchcitywiki --username-prefix="old-glitchcity" /home/reception/glitchcity.xml [14:06:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:06:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch-1 [+0/-0/±3] 13https://git.io/JOIel [14:06:50] [02miraheze/puppet] 07paladox 0358a0ee2 - Switch to x-real-ip [14:06:52] [02puppet] 07paladox synchronize pull request 03#1726: Introduce trafficserver module - 13https://git.io/JYonQ [14:07:29] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 464.22 ms [14:08:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03patch-1 [+0/-0/±1] 13https://git.io/JOIeK [14:08:27] [02miraheze/puppet] 07paladox 0376df1a9 - Remove new lines in logging.yaml [14:08:28] [02puppet] 07paladox synchronize pull request 03#1726: Introduce trafficserver module - 13https://git.io/JYonQ [14:17:12] [02puppet] 07paladox reviewed pull request 03#1726 commit - 13https://git.io/JOIvR [14:18:03] [02puppet] 07paladox reviewed pull request 03#1726 commit - 13https://git.io/JOIva [14:29:45] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 263.08 ms [14:33:57] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 447.69 ms [14:34:38] [02mw-config] 07R4356th opened pull request 03#3828: Add new import sources for snapwikiwiki, batmanwiki and batfamilywiki - 13https://git.io/JOIJz [14:35:41] miraheze/mw-config - R4356th the build passed. [14:36:02] [02mw-config] 07R4356th edited pull request 03#3828: Add new import source for snapwikiwiki, batmanwiki and batfamilywiki - 13https://git.io/JOIJz [14:44:04] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 272.17 ms [14:48:19] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 570.42 ms [14:54:21] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 344.08 ms [14:56:22] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 561.19 ms [14:59:58] [02miraheze/jobrunner-service] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOIkZ [14:59:59] [02miraheze/jobrunner-service] 07JohnFLewis 037e6175d - Remove rootjobs regularly [15:04:10] [02miraheze/jobrunner-service] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOIkb [15:04:11] [02miraheze/jobrunner-service] 07JohnFLewis 038809dab - tabbing [15:09:52] Reception123: ^ hopefully should start to see a decrease in the jobqueue size with that removing duplicated jobs that have been abandoned from the stats [15:11:01] that would be good [15:11:03] jobqueue has already gone down by 77% [15:11:14] wow, that's impressive! [15:11:31] guess all that was needed were some tweaks [15:11:54] [02mw-config] 07Reception123 closed pull request 03#3828: Add new import source for snapwikiwiki, batmanwiki and batfamilywiki - 13https://git.io/JOIJz [15:11:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOILt [15:11:57] [02miraheze/mw-config] 07R4356th 03753fc60 - Add new import sources for snapwikiwiki, batmanwiki and batfamilywiki (#3828) [15:12:07] Yeah, as RhinosF1 kept saying, we were running unmaintained software - so now we're maintaining it :P [15:12:29] [02mw-config] 07Reception123 closed pull request 03#3827: Disable wgCosmosEnabledRailModules['recentchanges'] on some wikis - 13https://git.io/JOTAj [15:12:31] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOILs [15:12:32] [02miraheze/mw-config] 07Universal-Omega 03aa9f514 - Disable wgCosmosEnabledRailModules on some wikis (#3827) [15:12:39] heh yes [15:12:55] at least we decided to look into it :) [15:12:58] miraheze/mw-config - Reception123 the build passed. [15:13:33] miraheze/mw-config - Reception123 the build passed. [15:13:56] https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?viewPanel=40&orgId=1&from=now-3h&to=now says it all [15:13:57] [ Grafana ] - grafana.miraheze.org [15:14:22] Well only reason we looked into it was because we had statistics that said there even was a problem [15:14:32] JohnLewis: I did say that about a million and one times [15:14:43] If we had these stats months ago, we'd have looked into it before [15:15:44] Yeah [15:38:50] and to be fair looking at these stats, I'm not seeing that we have ever had a need to run jobs, as the jobrunner system actually seems overpowered currently [15:42:45] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 255.49 ms [15:44:54] > and to be fair looking at these stats, I'm not seeing that we have ever had a need to run jobs, as the jobrunner system actually seems overpowered currently [15:44:54] yeah, so essentially, this is mainly a lack of system resource issue? It's a question of which resources and, of course, budget to allocate additional resources? [15:45:33] in effect, running jobs manually taxes the already strained jobrunner more [15:45:58] "strained jobrunner"? [15:46:23] perhaps bad wording, I guess I meant the server on which the jobrunner runs [15:46:37] oh, you said overpowered [15:46:40] not underpowered [15:46:46] hrm, I'm not sure then [15:48:50] Sure about what? :) [15:51:50] well, I guess what the solution is [15:52:54] like why so GUP page cache jobs go unclaimed and how to resolve that [15:52:56] JohnLewis: let's see what happens with your changes the next time loginwiki goes crazy [15:53:04] dmehus: that's been resolved [15:53:13] RhinosF1, ack, oh okay, cool yeah [15:53:13] That's why the dashboard is clean now [15:53:42] I could make an update to my global user page (a single edit) and see what happens? [15:54:02] dmehus: https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?orgId=1&from=now-3h&to=now [15:54:03] [ Grafana ] - grafana.miraheze.org [15:54:07] I mean you can [15:54:14] But it should be sensible now [15:54:38] yeah, that would be awesome if John's changes have resolved things :) [15:55:06] We'll only see under real load but the graphs look good [15:55:29] The only other thing I'd like to see is the rate of new jobs v rate of processing [15:55:43] JohnLewis said that should be easy to put in a graph [15:57:48] RhinosF1, updated, and yep, it seems to be operating perfectly now, per https://grafana.miraheze.org/d/3L3WYylMz/mediawiki-job-queue?orgId=1&from=now-3h&to=now. Great work, JohnLewis. What did your change(s) change to the jobqueue? [15:57:49] [ Grafana ] - grafana.miraheze.org [15:59:35] All I did was remove abandoned jobs :P [15:59:59] That seems steady at 59 for now [16:00:11] So let's see if it stays that way [16:00:18] JohnLewis, ah, cool. Interesting. [16:00:35] JohnLewis: Any chance you could do them stats I mentioned? [16:02:03] RhinosF1: abandoned claimed jobs - abandoned jobs are already pruned regularly [16:02:05] *unclaimed [16:03:18] Okay [16:03:38] JohnLewis: can you do the stats about rate of processing and rate of new jobs? [16:03:54] RhinosF1: yeah, doing it now [16:04:29] Ty [16:05:26] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.83, 7.03, 5.52 [16:07:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.37, 6.45, 5.49 [16:10:04] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 414.67 ms [16:18:07] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 271.65 ms [16:23:21] RhinosF1: looking at the data returns, I'm unsure what value it adds that the graph doesn't - if anything the graph seems to add much more value [16:29:31] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 375.46 ms [16:35:34] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 257.97 ms [16:37:57] JohnLewis: the graph only shows the rate of change. It wouldn't tell us whether more jobs were coming in or whether processing has stopped [16:38:22] (We have seen issues caused by it not processing before) [16:42:55] The graph is the total number of jobs [16:44:05] The data would be just a rate of change calculation on the graph, which can also just be seen by the eye [16:44:48] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 352.18 ms [16:46:16] JohnLewis: I didn't say just rate of change. I said a seperate number showing rate of processing and rate of new [16:46:35] Which is a rate of change [16:46:49] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 286.63 ms [16:47:35] JohnLewis: yes but I want to be able to see which of them is affecting rate of change during an incident [16:47:49] They're two numbers not one [16:49:09] The information on the graph is the only information you can get [16:49:31] So unless you want two separate metrics detailing how much the graph goes up and down by, I'm unsure what you're exactly asking for [16:50:57] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 514.31 ms [16:53:12] JohnLewis: a metric showing the number of new jobs that have come in the last x minutes and a metric show how many jobs were processed seperately [16:54:46] I want to know whether jobs are simply high because there's been a spike in jobs or Reception123 made a jobrunner crash doing wikibackups and have no space [16:55:26] That information isn't available [16:56:59] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 312.57 ms [16:57:29] JohnLewis: can we make it? [16:58:56] Not without making changes to the JobQueueRedis class [16:59:00] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 417.42 ms [16:59:49] JohnLewis: is it worth a task [17:00:07] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [17:00:20] It would be a MW-SRE task because it's MediaWiki code [17:01:02] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 334.77 ms [17:01:46] Who in mw-sre is gonna do that [17:02:43] Allocation/resourcing of MW-SRE tasks isn't my job so, Reception123 would be the one to ask about that [17:03:04] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 357.78 ms [17:03:12] Well the issue is I don't really see who currently in our team would be able to work on that [17:03:26] as neither of us would be able to really [17:04:49] That'll probably answer whether it's task worthy then [17:05:05] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 324.72 ms [17:07:05] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 518.04 ms [17:07:18] https://phabricator.miraheze.org/T7127 [17:07:20] [ ⚓ T7127 Add more jobrunner rate tasks to Grafana ] - phabricator.miraheze.org [17:08:22] JohnLewis: we really need to look into MediaWiki development now Universal_Omega is off as our team is a Python developer with very little formal [17:08:24] Why has the jobrunner software been tagged? [17:08:25] Teaching [17:08:30] And a Sysadmin we trained [17:08:44] Feel free to move it as I have no clue how we'd get the stats [17:09:21] MediaWiki needs to expose them to Redis, then it’ll be a Prometheus change [17:10:27] Right [17:10:44] Feel free to edit tasks and put that on [17:11:07] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 256.58 ms [17:11:34] I’ll support general development whenever I have time, but it becomes a low priority compared to other work, and so far that time is non existent as doing this Redis job for MW-SRE was already a push [17:13:51] Anything wrong with the JBR? I approved a wiki request at 15:49 (UTC) but the wiki still has not been created. [17:14:40] I don't see any jobs according to grafana [17:15:18] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 330.71 ms [17:15:21] Hmm... https://meta.miraheze.org/wiki/Special:RequestWikiQueue/17679 [17:15:22] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [17:15:37] JohnLewis: I understand but we are without any proper php developers on sre [17:16:05] No jobs are currently queued @R4356tg [17:16:15] Not sure what you mean by "Hmm...". [17:16:36] That wiki hasn't been created [17:17:08] Look up "hmm" in a dictionary. [17:17:20] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 561.42 ms [17:17:39] RhinosF1 already answered it for me. [17:17:39] While I understand, it’s not Infra’s responsibility to fix the development gap at the expense of our own work [17:18:27] JohnLewis: it's more a general concern [17:18:29] Sure, and I provided my answer. [17:19:09] We have 3 PHP developers on SRE, just all are in the Infra team [17:19:15] However, I felt that your response to mine was rather rude. I only asked what you were talking about in that link you provided. [17:20:01] May I ask you if you read my message above? [17:20:06] (Before replying) [17:20:11] #miraheze-offtopic-relay [17:20:42] I did [17:21:36] JohnLewis: yes but all 3 also have a million other tasks. You have it low on the list, paladox is busy with the traffic server stuff and SPF|Cloud has director stuff to do. [17:21:46] Anyways, moving on from that. [17:23:40] Mhm, which is my point - the experience is there if needed - just not readily available to do all the MW-SRE tasks that require it. This is a problem solving exercise [17:25:24] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 334.53 ms [17:25:30] JohnLewis Exactly what do you mean by that, if I may ask? [17:27:10] darkmatterman450: I was replying to Rhinos rather than the conversation you were having with R4356th. If anything in particular in my message contains something you were confused about, please ask specifically other the messages above should hopefully provide the context [17:27:24] Alright. [17:27:24] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 449.57 ms [17:28:08] JohnLewis: yes, my point is we need someone who can properly dedicate to developement [17:28:41] which unfortunately is part of our lack of volunteer and recruitment problem :( [17:29:00] Not just hope someone in infra eventually gets time [17:29:11] But that comes with our lack of volunteers [17:29:17] As reception just said [17:29:54] Yeah, it's always been very difficult to find them [17:29:59] So what I'm getting from your conversations here, you're saying you want more volunteers to come in and get whatever needs to be done? [17:30:02] and the word 'find' isn't right either as they've all found us [17:30:21] darkmatterman450: yes, of course! We're always in need of new volunteers, though now especially a new MediaWiki/PHP developer [17:30:27] ^ [17:30:43] Yeah most of our volunteers just stumble upon us [17:31:13] RhinosF1: of course we do, any team which has gaps should make all efforts to make such recruitment drives to find that experience or take steps to fill it [17:31:57] the issue is we don't really have where to "recruit" people and everyone who came after Orain just stumbled on the project by accident and eventually became interested [17:32:00] JohnLewis: that needs to be done for sure [17:32:06] it's been mostly luck [17:32:23] Infra as a team ensures that any new software deployed as a reasonable PoC who either has sufficient experience with the software or prior to deployment has given sufficient time to gain an understanding enough to deploy [17:32:31] *has a reasonable [17:32:33] So far, there's only 4 Stewards, which nearly isn't enough, don't you think? In my personal opinion (in which you don't have to agree with me on, but just want to hear your opinions), I think there should be around 8 to 9 Stewards. [17:32:34] and heh, I originally started out at Orain as just wanting to request a wiki but being disappointed that no one bothered creating it in like a month so I ran for wiki creator, and so on [17:32:56] darkmatterman450: well we're not really talking about stewards here, it's more the tech side. Though yes more Stewards would also be welcome of course [17:33:25] I could talk to Hasan next time he's on irc about his job sites but when I tried to get him to sign an NDA, it didn't exactly work out so not sure what his contacts would be like [17:33:26] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 349.08 ms [17:34:14] First question to answer the recruitment problem is, what steps have been done to sufficient advertise how to start volunteering in a technical sense and then what further steps have been done to advertise any gaps in technical expertise [17:34:41] @Reception123 I mean, we could try and welcome more volunteers into the group more than ever. [17:34:57] well the only attempt was https://meta.miraheze.org/wiki/Miraheze_Vacancies and well, it's definitely not satisfactory [17:34:58] [ Miraheze Vacancies - Miraheze Meta ] - meta.miraheze.org [17:35:23] darkmatterman450: well that's the hard part, finding people who are interested in volunteering and who have some knowledge [17:35:27] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 256.15 ms [17:36:14] It's sad that The Pioneer retired as a Steward. If he was still a Steward he would be unstoppable while appropriately handling tough spots and sticky situations. [17:36:15] JohnLewis: I tried to get a developer to work with us on bots who would happily help but couldn't get him to sign an NDA after a debate with Owen [17:36:26] But I'm gonna keep that out of a public channel [17:37:21] That would unfortunately be an unmovable barrier [17:37:34] Unmovable barrier? :/ [17:38:23] One that's not going to change [17:41:17] The roles? Or something else? [17:41:43] No, requiring an NDA [17:42:39] NDA? [17:43:40] Non disclosure agreement [17:45:22] hey, so, for some reason, @GR (he/him) [SocDemWiki]'s wiki (https://socdemwiki.miraheze.org/) is very slow and he's getting constantly 503's [17:45:25] [ SocDemWiki ] - socdemwiki.miraheze.org [17:45:28] Sorry about forgetting whatever they mean. Putting this into tuition should've been a feat. [17:45:44] I don't get the 503s but it takes almost 10 seconds to load the pages for some reason [17:46:00] 1393ms [17:52:38] [02miraheze/jobrunner-service] 07JohnFLewis pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JOIwf [17:52:39] [02miraheze/jobrunner-service] 07JohnFLewis 03d41607e - Only handle global user page mass-abandonment [17:52:41] [02miraheze/jobrunner-service] 07JohnFLewis 03e13a358 - Merge remote-tracking branch 'origin/master' [17:52:42] [02miraheze/jobrunner-service] 07JohnFLewis 03efb7f14 - Only handle global user page mass-abandonment [17:53:14] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 5.69, 5.00, 2.42 [17:55:08] @Lake thanks for taking care of this problem ^^ [17:55:14] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.02, 3.43, 2.15 [17:57:14] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.57, 2.41, 1.93 [18:11:17] Reception123: RhinosF1: realistic solution going forward might end up being what the WMF did in the early days, using the wiki to put a job description (well rather a 'Volunteer Description') and treat it like a job by advertising it sufficiently across the platforms to both make existing but also external users aware. [18:11:47] Ack [18:16:06] We could give that a try [18:16:20] But where would we advertise it? [18:17:04] Twitter, Facebook, other platforms we have a presence on? [18:18:16] I guess do something like that then, it's worth a try [18:18:31] Redesign that vacancies page [19:03:28] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.55, 6.35, 4.88 [19:03:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 6.06, 4.88 [19:05:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.87, 6.32, 5.07 [19:05:40] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.75, 6.22, 5.12 [19:36:57] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JOIH2 [19:36:58] [02miraheze/mediawiki] 07Reception123 03f2b1195 - Update Cosmos [20:00:20] RECOVERY - wiki.mlpwiki.net - reverse DNS on sslhost is OK: rDNS OK - wiki.mlpwiki.net reverse DNS resolves to cp11.miraheze.org [20:11:06] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:12:12] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [20:19:46] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.83, 3.88, 1.96 [20:21:47] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.68, 4.68, 2.51 [20:23:45] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.59, 3.48, 2.32 [20:25:44] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.60, 2.52, 2.10 [20:34:17] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:35:05] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:19:18] [02MirahezeMagic] 07joritochip opened pull request 03#245: Add i18n for Special:ManageWiki/extensions tabs - 13https://git.io/JOIhZ [21:20:17] miraheze/MirahezeMagic - joritochip the build passed. [21:23:01] [02MirahezeMagic] 07paladox closed pull request 03#245: Add i18n for Special:ManageWiki/extensions tabs - 13https://git.io/JOIhZ [21:23:02] [02miraheze/MirahezeMagic] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/JOIhM [21:23:04] [02miraheze/MirahezeMagic] 07joritochip 0368be629 - Add en i18n for Special:ManageWiki/extensions tabs [21:23:05] [02miraheze/MirahezeMagic] 07joritochip 039702363 - Add qqq i18n for Special:ManageWiki/extensions tabs [21:23:07] [02miraheze/MirahezeMagic] 07paladox 034fb6501 - Merge pull request #245 from joritochip/master [21:24:02] miraheze/MirahezeMagic - paladox the build passed. [21:40:03] [02mediawiki] 07Southparkfan reviewed pull request 03#1446 commit - 13https://git.io/JOLeC [21:40:04] [02mediawiki] 07Southparkfan reviewed pull request 03#1446 commit - 13https://git.io/JOLeW [21:50:15] [02ManageWiki] 07Universal-Omega opened pull request 03#267: A little maintenance to make things easier to read - 13https://git.io/JOLvc [21:51:17] miraheze/ManageWiki - Universal-Omega the build passed. [22:03:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.29, 6.58, 5.54 [22:05:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.09, 7.07, 5.70 [22:05:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.92, 5.81 [22:07:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.61, 6.18, 5.54 [22:07:41] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.69, 6.66, 5.87 [22:28:09] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.17, 1.94, 1.40 [22:30:09] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.97, 1.59, 1.33 [22:37:29] [02mw-config] 07joritochip opened pull request 03#3829: Add extension categories (T7125) - 13https://git.io/JOLTv [22:38:31] miraheze/mw-config - joritochip the build passed. [22:44:07] [02mw-config] 07Universal-Omega commented on pull request 03#3829: Add extension categories (T7125) - 13https://git.io/JOLTr [22:52:58] [02mw-config] 07paladox closed pull request 03#3829: Add extension categories (T7125) - 13https://git.io/JOLTv [22:53:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JOLkn [22:53:01] [02miraheze/mw-config] 07joritochip 03e68ac86 - Add extension categories (T7125) (#3829) [22:53:51] !log rebuild lc on mw* and jobrunner* [22:53:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:03] miraheze/mw-config - paladox the build passed. [23:04:07] PROBLEM - wiki.mlpwiki.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mlpwiki.net reverse DNS resolves to 192-185-16-85.unifiedlayer.com [23:19:22] SPF|Cloud: you ask a good question, why do we do that? [23:21:14] MirahezeMagic needs to be updated, the i18n messages for the new ManageWiki/Extension tabs are not included currently [23:22:03] I believe the pull request to add them got merged it just needs to get updated on miraheze/mediawiki [23:22:55] Why is this localization done in MirahezeMagic instead of ManageWiki? [23:23:17] Oh, I think I understand [23:23:18] They are Miraheze specific [23:23:38] Hmm, i've ran lc [23:23:55] but still doesn't show the i18n messages [23:24:47] MirahezeMagic was last updated 3 days ago [23:25:53] oh [23:25:55] i know why [23:25:56] yeh [23:26:36] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JOLtZ [23:26:38] [02miraheze/mediawiki] 07paladox 03b2280d8 - Update MirahezeMagic [23:27:22] !log rebuild lc on mw* and jobrunner* [23:27:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:28:49] Fixed now, thanks [23:36:53] [02mw-config] 07Universal-Omega opened pull request 03#3830: Reorder extension sections - 13https://git.io/JOLtF [23:37:48] miraheze/mw-config - Universal-Omega the build passed. [23:59:08] [02ManageWiki] 07Universal-Omega edited pull request 03#267: A little maintenance to make things easier to read - 13https://git.io/JOLvc