[00:36:08] [02mw-config] 07MacFan4000 labeled pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [00:36:17] [02mw-config] 07MacFan4000 assigned pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [00:48:54] [02mw-config] 07MacFan4000 synchronize pull request 03#2270: Switch to Swift for file uploads / viewing files - 13https://git.io/f4SjT [00:48:56] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/f49TQ [00:48:57] [02miraheze/mw-config] 07MacFan4000 0346f73e0 - consistency [00:50:07] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f49T7 [00:50:08] [02miraheze/mw-config] 07MacFan4000 03c996d1e - Revert "Fix "Undefined index: editor"" This reverts commit ea75c6dbfc7c8120a9580cef242d09230a54aae0. [00:51:34] MacFan4000: uh why did you do that? [00:51:38] Please revert that back [00:52:30] Because you fixed with Void suggestion [00:52:40] s/fixed/fixed it [00:52:40] MacFan4000 meant to say: Because you fixed it with Void suggestion [00:52:44] MacFan4000: yes but still need that fix for future [00:52:49] That was a pointless revert [00:52:52] No we don't [00:53:37] MacFan4000: why doint we? [00:53:44] Its really never been an issue [00:53:46] What does not having that check do? [00:54:24] When this happens, check syntax first [00:54:26] bounce [00:54:50] I rather this fix to prevent spamming the jobqueue [00:55:03] * paladox bounces NeoBogan [00:55:04] Lol [00:55:15] * MacFan4000 bounces paladox [00:55:27] Heh [01:22:56] !nini [01:22:56] Good night, sweet dreams and dream of the miraheze servers without errors! [01:24:03] sad that they rely on bots to bid them a dusken farewell [01:24:25] * NeoBogan swipes MacFan4000’s beloved crapple collection and sets it adrift for a viking funeral [01:24:50] Heh [01:40:08] * NeoBogan sends paladox headfirst over niagra falls [01:40:37] Lol [01:40:47] NeoBogan: we are about to have a heatwave [01:40:51] Starting from today [01:41:00] Sunny for at least over week [01:41:09] 27 degrees+ [01:41:10] things that haven’t been used for a while tend to gather dust [01:41:13] Celsius [01:41:22] dust is not known for its lubricative properties... [01:41:50] NeoBogan: let me guess you used dust as your lubricat [01:41:56] so... by not getting you to think so much, we wouldn’t have as much friction from all the dust, therefore no additional heat... :P [01:42:54] : NeoBogan: let me guess you used dust as your lubricat [01:43:43] let me put it this way, paladox: when was the last time you saw someone use sawdust instead of motor oil in their car? [01:43:57] yokels don’t count [01:44:09] I doint think I’ve seen either [01:44:24] At least not in the last 5-6 years [01:46:54] NeoBogan: ^^ [01:47:03] precisely :) [01:47:45] * paladox sends NeoBogan to antartica [01:47:55] It’s as easy as asking the queen to deport you lol [01:48:39] you’re going to deport me... from the country where I was born? [01:48:42] good luck with that [01:49:03] NeoBogan: not me the queen! [01:52:34] NeoBogan: jk! [01:52:49] drop that “not” and that declaration takes on a *whole* new meaning [01:53:14] Lol [01:53:32] NeoBogan: I could have sworn you were in the tv a few days ago [01:53:40] New Zealand pm baby! [01:53:48] yeah, I was the one making your picture fuzzy [01:54:32] NeoBogan: Lol hope [01:54:37] Was on the news! [01:54:46] NeoBogan: you were playing the baby [01:54:58] And apparently have learned very very fast [01:55:11] Can use the computer, speak and talk [01:55:32] to the point where this less-than-a-week-old knows more than a twenty year old [01:55:40] just how sheltered is that twenty year old?! [01:55:47] * NeoBogan glares at paladox quizzingly [01:56:29] NeoBogan: your 20? [01:56:38] You started work as a child then! [01:57:15] According to you, I’m only mere days old [01:57:20] but _you_ are 20 [01:57:28] Ah I see [01:57:45] NeoBogan: how do you know more then me? [01:58:08] well, for one, I know when to use “than” in contexts that need it [01:58:19] I can also spell “lubricant” without the need of my backspace key [01:58:47] That means you can speak English clearly [01:59:06] NeoBogan: Dosent me you know more :) [01:59:12] I speak english, bad engish, and drunkenese [01:59:38] NeoBogan: bad English? [01:59:41] Swearing? [02:00:00] NeoBogan: do you speak kiwi? [02:01:03] ya I do, bro [02:01:09] how’s it hangin, cuz? [02:01:15] Lol what [02:01:22] That’s not kiwii [02:01:23] you asked if I spoke kiwi [02:01:37] That’s really kiwii? [02:01:45] Because that sound like teenage talk [02:01:47] and what would kiwi sound like if ^ that ^ isn’t kiwi? [02:02:08] or just relaxed english [02:02:15] which, basically, is kiwi [02:02:21] Have no idea [02:02:28] then how do you know it’s not kiwi? [02:02:32] if you have no idea what kiwi is [02:02:33] But that sounds almost like what teenagers over here speak like [02:02:51] I’ve heard that kind of talk (my examples above) from people in the 50-60 age bracket [02:03:25] Heh [02:03:47] NeoBogan: Māori [02:04:06] What’s the difference between British English and New Zealand English I wonder [02:08:05] yanno, there’s a wikipedia article on NZ english [02:08:08] ever read it? [02:11:36] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [02:13:37] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 12% [02:15:27] NeoBogan: nope! [02:15:48] Best to ask someone who actually speaks nz English [02:28:18] well, good luck to ewe [02:28:33] hopefully it won’t colour your views on english [02:55:07] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2544 [02:55:09] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2544 [03:45:06] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [03:45:08] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [04:13:06] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2553 [05:49:57] [02miraheze/services] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f49OR [05:49:58] [02miraheze/services] 07Reception123 035865268 - add "bloowiki" to services [06:18:55] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f493y [06:18:56] [02miraheze/ManageWiki] 07translatewiki 03e0199ff - Localisation updates from https://translatewiki.net. [06:18:58] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f493S [06:18:59] [02miraheze/CreateWiki] 07translatewiki 0388065e5 - Localisation updates from https://translatewiki.net. [06:19:01] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4939 [06:19:02] [02miraheze/WikiDiscover] 07translatewiki 0372cf852 - Localisation updates from https://translatewiki.net. [06:55:06] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [06:55:08] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [07:01:07] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 10803 [12:04:26] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [12:12:26] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:47:07] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [14:34:25] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [14:42:26] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:24] .weather [14:54:26] Yahoo! Weather - London, England, GB: Sunny, 27°C (80°F), Humidity: 27%, Light air 1.8m/s (→) [15:04:25] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [15:11:08] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2578 [16:11:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4HY5 [16:11:23] [02miraheze/mediawiki] 07paladox 033614d18 - Update DynamicPageList3 to f2caf0e667b44a37aac63465b93b33519b44d90c [18:24:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4H2j [18:24:25] [02miraheze/mediawiki] 07paladox 03d474fb5 - Update Comments to master [18:25:03] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw1 [18:25:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:25:53] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw2 [18:26:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:26:33] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on m3 [18:26:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:36:34] paladox: could just put it in one [18:36:54] JohnLewis yep [18:38:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4HVW [18:38:30] [02miraheze/puppet] 07paladox 03986a41f - Increase JobRunner runner processes to 2 [18:39:51] tbh I'm tempted to revert [18:40:03] I know I gave you the go ahead but thinking about it, I'm withdrawing it [18:40:40] JohnLewis we will have to Remove GlobalUserPage then [18:40:44] we don't have an issue with the job queue most of the time, but that will now leave a permanent increase in usage, unnecessarily [18:40:45] that's far to many jobs [18:41:01] no we don't, we just have to acknowledge it and ignore it for now until a solution can be thought about it [18:41:26] until it's proven it's causing harm, I'm not calling it a real problem requiring strong action either way [18:41:30] JohnLewis what would be the side affects of a large queue [18:41:52] very little tbh [18:42:30] hmm [18:43:09] then we are going to have to update the icinga check to expect over 13707 jobs [18:43:20] because it being red means bad for me :) [18:43:31] hello paladox JohnLewis :) [18:43:34] hi [18:43:36] no, 300 is perfectly fine [18:43:55] if we're changing icinga to suit us, let's kill all the checks ;P [18:43:58] uh https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw1.miraheze.org&var-port=9100 [18:43:59] Title: [ Grafana ] - grafana.miraheze.org [18:44:06] how does mw1 have such as high load? [18:44:39] i did run lc, but that's finished now [18:46:58] JohnLewis ^^ [18:47:11] look at *top [18:47:30] php-fpm7.2 [18:47:43] and nginx [18:47:55] !log restarting php7.2-fpm and nginx on mw1 [18:48:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:49:54] JohnLewis feel free to revert my job runner increase [18:50:37] JohnLewis mw1 seems to have a higher cpu usage then mw2 and mw3 [18:50:47] dun dun dun [18:50:49] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw1.miraheze.org&var-port=9100&panelId=77&fullscreen [18:50:49] Title: [ Grafana ] - grafana.miraheze.org [18:51:55] 1.86 [18:51:59] back up to ^^ [18:52:24] 21.3% of cpu for php-fpm7.2 [18:52:32] thats just for one process [18:59:39] JohnLewis should i depool mw1 from varnish? [19:00:03] uh, what are you trying to achieve there? [19:00:22] JohnLewis trying to reduce load / investigate why there's high load [19:00:25] all i can tell is [19:00:29] it is php7.2-fpm [19:00:40] which means we are getting alot of traffic for mw1 [19:00:55] you'll reduce load on mw1 but increase it on mw2 and 3 [19:01:18] hmm ok, then ignore what i said :) [19:04:12] JohnLewis im guessing mw1 high load is because cp* keeps sending most of the traffic to mw1 [19:04:20] mw2 and mw3 cpu usage is lower [19:04:27] and load [19:04:30] varnish should be using a RR system [19:05:41] JohnLewis what's a RR system? [19:05:52] round robin [19:06:01] ah ok [19:07:04] JohnLewis then lets try to do that :) [19:07:44] JohnLewis seems we do alrrady [19:07:46] already [19:07:47] https://github.com/miraheze/puppet/blob/master/modules/varnish/templates/default.vcl#L59 [19:07:47] Title: [ puppet/default.vcl at master · miraheze/puppet · GitHub ] - github.com [19:07:59] I know... [19:08:02] that's why I said it [19:09:09] what about random director [19:10:26] directors.random [19:10:28] directors.random() [19:11:17] there's no system there [19:11:30] random is poor because you can accidentally push all the traffic to one server [19:11:42] especially with small numbers [19:12:04] hmm [19:12:17] JohnLewis it seems it is already doing that [19:12:18] already [19:12:21] with round the robin [19:12:26] not really [19:12:58] why is php-fpm causing high cpu? [19:12:58] it works in strategy but the issue is, intensive requests can be sent to mw1. it's not something you can avoid - it's how it goes [19:13:06] oh [19:13:10] it could equally as likely be mw2 [19:13:13] or even mw3 [19:13:15] but it's mw1 [19:16:52] JohnLewis would adding another mw have a affect (ie reduce some of mw1 cpu usage)? [19:17:06] in theory, yes [19:17:19] JohnLewis should i file a request for this? [19:17:25] but adding more will always have an affect [19:17:26] no [19:17:34] it won't be approved and it isn't justified [19:18:14] !log reboot mw1 [19:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:18:19] !log depool mw1 [19:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:20:27] !log repool mw1 [19:20:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:21:45] JohnLewis it's back up to 1.40 [19:22:01] there has to be a reason [19:22:11] mw1 must be different to the others [19:22:23] there is no way this is varnish and traffic [19:23:08] JohnLewis the only thing i can think of is mw1 is at the top [19:25:26] cp2 is sending traffic equally [19:27:40] so it's cp4? [19:27:47] I'm going to restart all three varnish's and monitor their traffic over a 10 minute period [19:27:55] so aka, cp* is mine, touch it and die [19:28:30] ok [19:28:34] and lol [19:29:28] though cp5 has been up for nearly 18 days and that shows my point the wrong way around [19:29:48] mw1 received 579k requests, mw2 and mw3 588k requests [19:29:58] JohnLewis cp5 has more resources then cp4 and cp2 though [19:30:01] so mw2 and mw3 are getting more traffic from cp5 [19:30:06] oh [19:30:17] the resources are irrelevant on cache server [19:30:36] is mw2 and mw3 closer to cp5 [19:30:52] mw* is in the same DC... [19:30:59] probably in the same rack [19:31:24] they can't be closer to cp5, since cp5 is in Asia while cp4 is in Europe [19:31:57] well this is strange then [19:32:05] they should all be using mw* equally [19:32:17] not one sending traffic more to one mw and then the other to another mw [19:35:03] what should happen and what does are two different things. if you believe otherwise, then you're gullible :P [19:38:54] heh [19:39:41] 10 minutes over, let's start analysing this in detail then. will post on ticket [19:40:52] ok [19:40:58] JohnLewis your creating a task right? [19:41:05] no [19:41:19] wait, there isn't a task for this nevermind [19:41:22] paste then [19:41:29] i will create one then [19:41:36] for what? [19:42:25] JohnLewis https://phabricator.miraheze.org/T3272 [19:42:27] Title: [ ⚓ T3272 Investigate mw1 high cpu and load usage ] - phabricator.miraheze.org [19:48:24] varnish is legit 33% split [19:48:27] it's working [19:49:04] hmm [19:55:21] JohnLewis i wonder if it was memcached [19:55:25] i just stopped it [19:55:29] and load went down to 0.80 [19:55:31] yeah [19:55:41] I didn't see that and just closed the task [19:56:01] * paladox is still monitoring mw1 through top [19:56:44] JohnLewis i doin't think we will be able to use memcache on mw then [19:56:56] im going to uninstall it [19:57:07] I could have guessed that anyway [19:57:17] more processes on small resources is bad [19:57:44] JohnLewis well the resources are tied to terms of use by ramnode [19:57:55] we are only limited to 1.00 even though we have 4 processors [19:58:03] no, it's respective [19:58:06] !log apt-get --purge remove memcached on mw1 [19:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:58:20] that 1 core represents 4 vcores [19:58:39] hmm [19:58:41] oh [19:59:45] JohnLewis load hasen't gone back up to 1.00+ for over a few mins now since i stopped memcached [19:59:52] so i can now say this is resloved [19:59:58] resolved [20:00:28] !log apt-get --purge remove memcached on mw* [20:00:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:01:58] meh [20:02:03] memcached got installed again [20:02:08] i a going to remove it from puppet [20:03:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4HM4 [20:03:06] [02miraheze/puppet] 07paladox 03282a402 - Update init.pp [20:25:16] !log running jobs on all wikis [20:25:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:25:48] !log that is also in a screen [20:25:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:26:00] JohnLewis i wonder if this https://code.uplex.de/uplex-varnish/libvmod-vslp would be useful for us? [20:26:01] Title: [ uplex-varnish / libvmod-vslp · GitLab ] - code.uplex.de [20:26:47] a screen seems pointless for that [20:27:51] but that will keeps requests to one server [20:28:08] which makes it more likely for one server to have more intensive requests than others [20:28:14] ok [20:32:25] Huh, suddenly got multiple notifications from hours ago (that were not visible before to me). [20:36:06] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [20:39:14] * NeoBogan feels a need to stay warm this frosty morning... [20:39:41] * NeoBogan ignites paladox with the grandfather paradox theory [20:39:48] lol [20:39:53] .weather NeoBogan [20:39:55] Yahoo! Weather - Nabagan, Cagayan Valley, PH: Scattered Showers, 25°C (77°F), Humidity: 95%, Light air 1.8m/s (↑) [20:39:57] .weather [20:39:57] Yahoo! Weather - London, England, GB: Sunny, 27°C (80°F), Humidity: 27%, Gentle breeze 4.9m/s (↙) [20:40:07] .weather wellington nz [20:40:08] Yahoo! Weather - Wellington, Wellington, NZ: Partly Cloudy, 7°C (44°F), Humidity: 73%, Moderate breeze 6.3m/s (→) [20:41:00] .weather scott base [20:41:00] Yahoo! Weather - Mascoutah, IL, US: Mostly Cloudy, 28°C (82°F), Humidity: 70%, Light breeze 3.1m/s (↖) [20:41:08] ... [20:41:12] .weather mcmurdo station [20:41:13] None: unknown, unknown, unknown, unknown [20:41:21] .weather antarctica [20:41:22] None: unknown, unknown, unknown, unknown [20:41:29] huh [20:41:30] .weather zppix [20:41:31] Yahoo! Weather - Spixhe, Liege, BE: Clear, 16°C (60°F), Humidity: 59%, Light breeze 3.1m/s (↓) [20:41:34] lol [20:41:43] .weather paladox [20:41:44] Yahoo! Weather - London, England, GB: Sunny, 27°C (80°F), Humidity: 27%, Gentle breeze 4.9m/s (↙) [20:41:50] .weather Vermont [20:41:51] Yahoo! Weather - Vermont, US: Mostly Sunny, 22°C (71°F), Humidity: 30%, Fresh breeze 9.8m/s (↘) [20:44:25] .weather NeoBogan [20:44:25] Yahoo! Weather - Nabagan, Cagayan Valley, PH: Scattered Showers, 25°C (77°F), Humidity: 95%, Light air 1.8m/s (↑) [20:48:36] .weather labster [20:48:39] Yahoo! Weather - Lybster, Scotland, GB: Mostly Cloudy, 12°C (53°F), Humidity: 88%, Light breeze 3.1m/s (↗) [20:49:34] lol [20:49:39] it's cold in scotland! [20:49:55] around 15 degrees colder (thats a best guess) [20:50:02] .weather [20:50:02] Yahoo! Weather - Portland, ME, US: Showers, 17°C (62°F), Humidity: 69%, Light breeze 3.1m/s (↙) [21:41:07] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2396 [21:41:09] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2396 [21:43:05] if some one edits there user page per wiki that's going to be alot of jobs [21:43:45] yep https://meta.miraheze.org/w/index.php?title=User:%E9%96%8B%E6%8B%93%E8%80%85&curid=11124&diff=52464&oldid=52171 [21:43:45] Title: [ Difference between revisions of "User:開拓者" - Miraheze Meta ] - meta.miraheze.org [21:44:04] ok [21:46:31] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4HNT [21:46:32] [02miraheze/services] 07paladox 03ff11ff3 - Enable services for checkinwiki fixes T3274 [21:49:51] Hello sysadmin_wannabe! If you have any questions feel free to ask and someone should answer soon. [21:52:06] hi guys [21:55:23] hi [21:57:26] hi paladox, thank you for enabling VE on my wiki. [21:57:36] your welcome :) [22:05:06] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [22:13:19] how do I get permissions to translate the main page? [23:22:52] paladox: found out how to resolve the GUP issue [23:23:08] JohnLewis oh how? [23:23:18] opt-in, aka none default [23:25:05] JohnLewis oh, you set the default as none? [23:25:19] no... [23:25:25] opt-in = none default [23:26:03] ok [23:30:32] hm. nevermind even when disabled it creates the jobs [23:33:06] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2439 [23:33:08] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2439 [23:57:07] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs