[00:01:23] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 1601 MB (8% inode=93%); [00:04:25] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 2837 MB (14% inode=93%); [01:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvbpA [01:20:11] [02miraheze/services] 07MirahezeSSLBot 030e2d9ff - BOT: Updating services config for wikis [02:03:05] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.69, 1.71, 1.54 [02:05:52] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.58, 1.62, 1.53 [05:45:19] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.06, 1.79, 1.60 [05:48:05] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.88, 1.79, 1.63 [05:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNLG [05:50:10] [02miraheze/services] 07MirahezeSSLBot 03d4d5b80 - BOT: Updating services config for wikis [05:50:48] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.53, 1.61, 1.58 [06:27:30] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2913 MB (12% inode=93%); [07:23:58] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.72, 1.65, 1.54 [07:26:39] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.64, 1.61, 1.54 [08:04:38] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±2] 13https://git.io/JvNm8 [08:04:40] [02miraheze/mw-config] 07Reception123 039e106bb - add wgCategoryTreeDefaultMode setting [08:04:41] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [08:05:12] [02mw-config] 07Reception123 opened pull request 03#2966: add wgCategoryTreeDefaultMode setting - 13https://git.io/JvNm4 [08:05:26] [02mw-config] 07Reception123 commented on pull request 03#2961: Add new setting - 13https://git.io/JvNmB [08:05:28] [02mw-config] 07Reception123 closed pull request 03#2961: Add new setting - 13https://git.io/JvFJZ [08:07:51] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JvNmE [08:07:52] [02miraheze/mw-config] 07Reception123 0355c3541 - syntax [08:07:54] [02mw-config] 07Reception123 synchronize pull request 03#2966: add wgCategoryTreeDefaultMode setting - 13https://git.io/JvNm4 [08:13:01] Reception123: thx [08:17:48] Reception123: LGTM [08:25:38] hi [08:26:05] I'll maybe test it out on test2 first and then deploy [08:27:33] Good idea [08:35:05] [02puppet] 07Reception123 commented on pull request 03#1086: Prometheus: Set query.max-samples to 10000000 - 13https://git.io/JvNm7 [08:35:07] [02puppet] 07Reception123 closed pull request 03#1086: Prometheus: Set query.max-samples to 10000000 - 13https://git.io/JeOcM [08:35:32] [02puppet] 07Reception123 commented on pull request 03#1296: mediawiki: Introduce support for apache as a backend - 13https://git.io/JvNmd [08:35:34] [02puppet] 07Reception123 closed pull request 03#1296: mediawiki: Introduce support for apache as a backend - 13https://git.io/JvSQn [08:41:48] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200402213524.xml [08:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:44:14] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200402160456.xml [08:44:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:44:28] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki aeschyluswiki /home/reception/Wikipedia-20200403182927.xml [08:44:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:02:41] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.27, 4.96, 4.39 [09:05:24] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.09, 4.42, 4.29 [09:06:09] Morning JohnLewis [09:09:30] Morning [09:10:20] JohnLewis: is it as warm down south? [09:10:38] Boiling today [09:23:17] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 12.44, 9.25, 6.49 [09:28:39] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.60, 5.71, 5.77 [09:37:02] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 2.04, 3.58, 4.75 [09:55:46] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.90, 5.32, 4.75 [09:58:32] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 9.67, 7.72, 5.79 [10:36:51] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2637 MB (10% inode=93%); [10:37:23] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2116 MB (10% inode=93%); [10:57:59] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.44, 1.82, 1.71 [11:06:10] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.61, 1.63, 1.66 [11:34:07] [02mw-config] 07Reception123 synchronize pull request 03#2966: add wgCategoryTreeDefaultMode setting - 13https://git.io/JvNm4 [11:34:08] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JvN3l [11:34:10] [02miraheze/mw-config] 07Reception123 0328e1d81 - Update ManageWikiSettings.php [11:37:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JvN34 [11:37:22] [02miraheze/mw-config] 07Reception123 0348166f5 - fix indents [11:37:23] [02mw-config] 07Reception123 synchronize pull request 03#2966: add wgCategoryTreeDefaultMode setting - 13https://git.io/JvNm4 [12:00:33] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123 [12:00:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:06:47] !log deleted and dropped https://phabricator.miraheze.org/P290 (DP) [13:06:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:06:53] [ ✎ P290 Deleted wikis 05/04/2020 ] - phabricator.miraheze.org [13:12:00] [02mw-config] 07Reception123 closed pull request 03#2966: add wgCategoryTreeDefaultMode setting - 13https://git.io/JvNm4 [13:12:02] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvNGp [13:12:03] [02miraheze/mw-config] 07Reception123 0364c4f7f - add wgCategoryTreeDefaultMode setting (#2966) * add wgCategoryTreeDefaultMode setting * syntax * Update ManageWikiSettings.php * fix indents [13:12:05] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [13:12:06] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [13:12:13] [02miraheze/mw-config] 07Reception123 deleted branch 03paladox-patch-2 [13:12:15] [02mw-config] 07Reception123 deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [13:55:08] PROBLEM - wiki.mastodon.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mastodon.kr' expires in 15 day(s) (Tue 21 Apr 2020 13:49:25 GMT +0000). [13:56:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNnP [13:56:31] [02miraheze/ssl] 07MirahezeSSLBot 032138d33 - Bot: Update SSL cert for wiki.mastodon.kr [14:00:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNn7 [14:00:11] [02miraheze/mw-config] 07Reception123 036ad004a - rm pathofexilewiki (dropped) [14:04:30] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvNcT [14:04:31] [02miraheze/CreateWiki] 07JohnFLewis 03161d2d2 - add migrateServerName.php maint script [14:06:02] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Sat 04 Jul 2020 12:56:23 GMT +0000. [14:19:30] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNco [14:19:31] [02miraheze/ManageWiki] 07JohnFLewis 03488f329 - convert migration script to only add existing extensions (cleanup step) [14:21:40] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNcK [14:21:41] [02miraheze/ManageWiki] 07JohnFLewis 03233733f - missing global from fixes yesterday [14:26:17] Reception123: I’ve just changed the test2 main page to say test2 instead of 1 but it’s stuck in moderation [14:26:31] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNcD [14:26:32] [02miraheze/CreateWiki] 07JohnFLewis 03aa831dd - use isset over is_null for safety [14:27:06] RhinosF1: let me see if I find out how to fix that [14:28:17] Reception123: whatever you did worked! [14:29:15] RhinosF1: clicked accept :P [14:29:32] That would make sense [15:03:52] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 5.25, 4.64, 3.87 [15:06:29] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 2.88, 3.49, 5.49 [15:06:39] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 1.39, 3.42, 3.55 [15:09:12] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 2.83, 3.24, 5.07 [15:21:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNCN [15:21:36] [02miraheze/puppet] 07paladox 0398be53f - jobrunner1: Set php-fpm child to 1 [15:28:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNWY [15:28:40] [02miraheze/puppet] 07paladox 030b5a2b1 - Update jobrunner.json.erb [15:29:52] RhinosF1: around? [15:34:12] Reception123: yep [15:34:29] RhinosF1: can you edit test2? [15:35:22] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 10.94, 7.17, 5.31 [15:35:59] Reception123: yep [15:36:04] JohnLewis: ^ [15:36:14] oh nevermind [15:36:17] pls ignore, my bad [15:36:25] RhinosF1: try now [15:37:43] Reception123: still [15:37:57] paladox: ^ do you see why? I removed edit from *, user and sysop on test2 [15:38:04] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.14, 5.65, 5.01 [15:39:03] I get "View source for Main Page" [15:39:17] oh [15:39:22] protected main page [15:39:28] paladox: I can edit from my test account, my main and Rhinos can [15:39:31] wait no [15:40:48] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 9.67, 6.19, 5.25 [15:41:16] hmm, nope [15:41:21] JohnLewis maybe a bug ^? [15:41:35] I don't know [15:41:37] JohnLewis: yeah it seems after removing edit from user, *, sysop we can still edit regardless [15:41:42] Cache says 'edit' isn't assign [15:41:56] But wgGroupPermissions says it is [15:42:38] Reception123: ‘edit’ is assigned to * on Special:ListGroupRights [15:43:06] well I guess John is looking into it [15:43:29] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.88, 5.73, 5.20 [15:45:23] > var_dump( $wgConf->settings['wgGroupPermissions']['test2wiki']['*']['edit'] ); [15:45:23] PHP Notice: Undefined index: edit in /srv/mediawiki/w/maintenance/eval.php(78) : eval()'d code on line 1 [15:46:03] well that works [15:46:07] JohnLewis try users [15:46:15] huh? [15:48:12] JohnLewis try 'user' [15:48:20] What do you mean? [15:48:47] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.77, 5.06, 5.08 [15:48:51] JohnLewis it works when logged out [15:48:51] JohnLewis: /cdb gets an internal error [15:49:02] And the view link needs i18n [15:49:10] paladox: yes..? [15:49:12] yeah now it seems to work for anon [15:49:18] RhinosF1: huh? [15:49:35] Reception123 well i already said it worked a while ago :) [15:49:45] JohnLewis: try viewing Special:ManageWiki/cdb [15:49:56] Yes, it doesn't exist [15:50:05] It's testing->prod remants [15:50:12] Ah [15:50:33] Reception123: can still edit logged in [15:50:44] same [15:51:04] People are really starting to confuse me now, paladox what do you mean? I've asked 3 times and you've still not clarified what "try users" means [15:51:27] I think he meant do var_dump( $wgConf->settings['wgGroupPermissions']['test2wiki']['user']['edit'] ); [15:51:28] and if it's not relevant to what I'm debugging, please don't mention it [15:51:40] JohnLewis well Reception123 said he dropped "edit" from "users" and the sysadmin group, yet he says Reception123 and RhinosF1 can still edit. [15:51:54] paladox: Yes. Because * has edit [15:51:55] oh wait not the sysadmin group [15:52:06] how? [15:52:09] it's still on there but I can edit with my test account [15:52:15] https://test2.miraheze.org/wiki/Special:ManageWiki/permissions/* does not list edit [15:52:16] [ Manage this wiki's permissions - Test2 ] - test2.miraheze.org [15:52:27] paladox: This is what I am trying to debug... [15:52:47] ok [15:52:48] That was the issue as mentioned at the very beginning [15:52:58] * paladox goes to look at something else [15:54:03] PROBLEM - puppet2 Current Load on puppet2 is WARNING: WARNING - load average: 5.11, 4.60, 3.81 [15:54:57] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 8.57, 6.60, 5.69 [15:56:46] RECOVERY - puppet2 Current Load on puppet2 is OK: OK - load average: 0.72, 3.07, 3.37 [16:03:03] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.02, 5.76, 5.69 [16:04:14] Reception123: I feel like 'edit' is being added by something after we initialise [16:04:32] Because $wgConf isn't returning 'edit' => true anywhere from what I can see [16:04:41] JohnLewis: what could it be? an interfering extension? [16:04:52] Potentially [16:05:16] JohnLewis: I'll try disabling all extensions then and see [16:05:24] No, please don;t [16:05:33] Makes my debugging even harder [16:05:54] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 11.33, 7.50, 6.32 [16:05:58] JohnLewis: oh ok [16:06:04] JohnLewis: then what could I do? [16:06:19] Test other stuff? :) [16:06:29] I'll go for namespaces then [16:08:41] JohnLewis: something broke [16:08:45] JohnLewis: https://test2.miraheze.org/wiki/ [16:08:50] don't know if it's you or me with namespaces [16:08:55] someone did var_dump [16:08:56] :P [16:11:24] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.84, 5.79, 5.99 [16:14:10] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 10.14, 6.92, 6.33 [16:15:08] Namespaces seem to work, same for settings and disabling/enabling extensions [16:15:14] so only permissions is problematic [16:16:51] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.04, 5.19, 5.77 [16:19:05] Reception123: It is an extension, I'm not too interested in which because it's kind of standard practise though. Let me see if I can 'cleanly' work around it [16:19:28] JohnLewis: ok :) [16:19:31] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.02, 6.25, 6.09 [16:19:47] * Reception123 wonders which one it is [16:22:11] Reception123: now that should be all fine... [16:22:30] JohnLewis: let's have a look. As for extensions/settings/namespaces that seems to be working fine [16:22:39] JohnLewis: main page = [f9a560b82a063f1adf2d8247] 2020-04-05 16:22:33: Fatal exception of type "MWException" [16:23:24] "Invalid scripting engine is specified in $wgScribuntoDefaultEngine" [16:24:28] hm, I didn't test anything with scribunto [16:25:22] maybe it's when I created then deleted a namespace? [16:25:33] No, it's a config variable [16:25:42] oh, well I don't think I touched that [16:26:06] aha [16:26:22] The clean fix breaks stuff, because now we want CW to be the 'master' while before, we didn't care [16:26:48] We use $wgX in LS, we can no longer do this, we have to use $wgConf (or $settings) [16:27:02] JohnLewis: ah, [16:27:16] well weren't we supposed to move to only one file (ManageWikiSettings.php)? [16:27:22] So if $wgX is in $wgConf, $wgConf now overrides it [16:27:28] oh [16:28:06] Which is kind of a blessing and a curse [16:28:37] Blessing: Extension X can't override ManageWiki now. Curse: Extension X has to have it's required Config known to CreateWiki before it can be enabled [16:29:00] Which won't be a problem for production extensions - it might be a problem for testing [16:30:35] But actually, the curse is kind of a blessing in disguise now, because we want to kill LE, so extension config *has* to be in $wgConf regardless [16:30:56] But then it begs whether we should go straight to end game where we kill LE now, or convert all of LE to use $wgConf [16:31:54] Convert would be less risky and less time consuming, end game is risky and would probably require a few more days worth of work and make this too big a chance to safely too as once with other stuff we're doing [16:32:01] Reception123 paladox ^ thoughts [16:33:04] JohnLewis: wow that makes it way more complicated than I thought it would be. I think we should definitely go for a safe option that ensures we don't cause any issues in prod [16:33:17] ^ [16:33:37] (more the lets go the safe way) [16:34:04] and if it's less time consuming that's good too [16:34:04] I'll probably just branch this all to mw-config as its too much to do on test2 and then reproduce later [16:34:18] yeah probably a better idea and easier to track [17:04:54] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.03, 4.91, 5.73 [17:10:15] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.11, 5.88, 5.96 [17:13:00] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.90, 6.00, 5.99 [17:18:48] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 1.62, 3.55, 4.96 [17:59:32] Zppix: already doing that [17:59:34] Zppix: forgot to say [17:59:40] Oh ok [17:59:47] Reception123: ill reassign to you then [17:59:56] Zppix: thanks :) [18:00:28] Reception123: np i was just trying to find things i could do safely from a mobile ssh connection [18:03:13] Reception123: xml in your home dir? [18:03:39] Zppix: homedir, but I've already started to rerun [18:03:44] Ok [18:03:45] Zppix: and the others both had a problem with actor [18:04:00] Reception123: the actor is a db change that needs done [18:04:02] Zppix: which I had to file upstream and their answer was it's fixed with 1.34.1 which it's obviously not [18:04:11] I thought paladox said everything was up to date? [18:04:14] Reception123: its trying to assign a edit to a user using a different uid iirc [18:04:53] It wants to assign probably something to flow, but it already exists but the uid is different so import doesnt use existing [18:05:16] Reception123: check sal and it should have the fix weve done before [18:05:35] paladox didn't mention that [18:06:19] Reception123: maybe its not the same issue then [18:15:33] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.07, 5.85, 3.92 [18:15:59] RF1: fyi i do not have access to any security tasks unless im involved in some way... even if i did since I do have a NDA on file I wouldnt be able to disclose anything w/o permission [18:16:19] (For wikimedia atleast) [18:16:35] k [18:16:40] Martin sorted iy [18:18:23] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.13, 5.67, 4.18 [18:24:22] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.86, 7.87, 5.63 [18:27:59] Reception123 huh? [18:34:12] !log increase puppet2 to 4 cores [18:34:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:35:09] paladox: zppix says we had a fix for that importdump problem [18:35:29] !log increase jobrunner1 ram to 3gb [18:35:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:35:36] If its the same issue [18:35:41] That i think [18:35:51] Reception123 it doesn't appear that fixed it, because 1.34.1 is rolled out [18:36:24] I guess I'll wait for the person who asked me to answer then [18:36:31] paladox: is it an error due to the flow talk manager? [18:36:48] no [18:36:48] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 0.48, 0.14, 0.05 [18:37:00] Zppix wasn't me who did the import, it was Reception123 [18:37:04] Reception123: nm its not the issue i thought [18:37:06] but as far as i'm aware it was not flow [18:39:25] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 56 failures. Last run 4 minutes ago with 56 failures. Failed resources (up to 3 shown) [18:39:30] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 5 minutes ago with 33 failures. Failed resources (up to 3 shown) [18:39:48] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 80 failures. Last run 5 minutes ago with 80 failures. Failed resources (up to 3 shown) [18:40:10] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 265 failures. Last run 4 minutes ago with 265 failures. Failed resources (up to 3 shown) [18:40:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvN00 [18:40:28] [02miraheze/puppet] 07paladox 033dc8ad1 - Update site.pp [18:40:33] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 71 failures. Last run 5 minutes ago with 71 failures. Failed resources (up to 3 shown) [18:41:29] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 103 failures. Last run 6 minutes ago with 103 failures. Failed resources (up to 3 shown) [18:41:40] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 107 failures. Last run 6 minutes ago with 107 failures. Failed resources (up to 3 shown) [18:43:36] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:44:22] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:31] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:44:51] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 160 failures. Last run 6 minutes ago with 160 failures. Failed resources (up to 3 shown): File[espiral.org_private],File[wiki.lbcomms.co.za],File[wiki.lbcomms.co.za_private],File[www.openonderwijs.org] [18:45:11] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:45:22] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:45:37] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:45:41] Weeeee [18:46:04] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:47:32] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:59:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 4 minutes ago with 7 failures. Failed resources (up to 3 shown) [19:07:39] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 5 minutes ago with 13 failures. Failed resources (up to 3 shown) [19:10:13] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 139 failures. Last run 5 minutes ago with 139 failures. Failed resources (up to 3 shown): File[www.wikimicrofinanza.it],File[www.wikimicrofinanza.it_private],File[documentation.aqfer.com],File[documentation.aqfer.com_private] [19:13:47] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:06] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:15] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:16] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:27] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:28] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:28] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:36] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:38] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:14:41] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:44] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:11] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:11] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:11] PROBLEM - dbt1 Puppet on dbt1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:13] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:23] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:23] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:24] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:32] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:47] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:15:56] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:16:13] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:16:13] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:16:25] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:16:53] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:17:06] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:17:07] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [19:17:17] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:17:17] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:17:19] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:17:31] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:17:32] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [19:17:36] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:05] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:06] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:14] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:15] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:18:15] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:19] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [19:18:35] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:18:41] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [19:19:04] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:19:19] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:21:50] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [19:22:04] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:22:26] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:24:04] RECOVERY - dbt1 Puppet on dbt1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:24:10] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:24:50] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:29:40] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.43, 5.08, 3.64 [19:38:06] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.36, 4.70, 4.15 [19:44:15] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 10.04, 7.82, 5.59 [19:49:42] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.62, 5.89, 5.37 [19:52:33] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.88, 6.31, 5.61 [19:55:13] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.79, 5.92, 5.57 [19:57:50] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.03, 5.51, 5.43 [20:00:31] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 2.39, 4.18, 4.94 [20:07:00] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03new-cache [+0/-0/±7] 13https://git.io/JvNzJ [20:07:01] [02miraheze/mw-config] 07JohnFLewis 03d07673e - messily convert stuff to wgConf [20:07:03] [02mw-config] 07JohnFLewis created branch 03new-cache - 13https://git.io/vbvb3 [20:12:12] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 8.63, 6.81, 5.71 [20:14:37] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03new-cache [+0/-0/±2] 13https://git.io/JvNzc [20:14:39] [02miraheze/mw-config] 07JohnFLewis 0354badaf - more changes [20:17:52] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 2.58, 5.15, 5.46 [20:23:30] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 2.60, 4.08, 4.93 [20:29:42] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.05, 5.51, 5.45 [20:32:27] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.42, 5.93, 5.61 [20:35:17] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.90, 5.76, 5.59 [20:43:40] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.94, 5.70, 5.52 [20:54:57] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.60, 5.79, 5.87 [21:00:52] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.38, 6.24, 5.97 [21:03:44] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.68, 5.45, 5.73 [21:09:24] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.00, 5.15, 5.38 [21:10:36] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03new-cache [+0/-0/±2] 13https://git.io/JvNg6 [21:10:37] [02miraheze/mw-config] 07JohnFLewis 0381b3050 - fixes with messy changes [21:12:13] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.75, 4.95, 5.25 [21:14:55] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.20, 5.41, 5.36 [21:27:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvN2L [21:27:08] [02miraheze/mw-config] 07paladox 03cbd20b0 - new-cache: Set wgScribuntoDefaultEngine to luaautodetect [21:27:09] JohnLewis ^ [21:27:52] oh [21:27:55] i see the problem [21:27:57] paladox: when I skimmed the error was because that was the value [21:28:23] yeh [21:29:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvN2s [21:29:24] [02miraheze/mw-config] 07paladox 03d7c3395 - Add wgScribuntoEngineConf [21:29:28] grr [21:29:40] i hate it when you accidentally press enter [21:29:45] though basically that commit [21:32:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvN24 [21:32:06] [02miraheze/mw-config] 07paladox 036b654e0 - Tweek wgScribuntoEngineConf [21:32:18] JohnLewis done ^ [21:33:28] Thanks [21:34:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvN2r [21:34:27] [02miraheze/mw-config] 07paladox 030a12519 - Fix syntax [21:35:15] JohnLewis works now [21:35:46] though https://test2.miraheze.org/ redirects to https://$lang.miraheze.org/ and https://test2.miraheze.org/wiki/Main_Page whilst works, shows no sitename. I presume this is knwon? [21:35:47] [ No sitename set! ] - test2.miraheze.org [21:35:48] *known [21:36:48] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.64, 5.50, 5.95 [21:42:16] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.68, 5.83, 5.88 [21:45:00] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.30, 5.83, 5.88 [21:47:39] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 11.73, 7.17, 6.31 [21:58:32] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 3.05, 4.70, 5.61 [22:06:40] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 7.90, 6.26, 5.94 [22:09:31] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 5.34, 5.75, 5.79 [22:12:19] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 5.84, 6.18, 5.96 [22:20:24] PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: WARNING - load average: 4.67, 5.82, 5.98 [22:22:39] !log increase jobrunner1 core by ` [22:22:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:22:45] !log increase jobrunner1 core by 1 [22:22:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:23:51] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 0.56, 0.12, 0.04 [22:27:12] PROBLEM - jobrunner1 JobChron Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [22:27:46] PROBLEM - jobrunner1 MirahezeRenewSsl on jobrunner1 is CRITICAL: connect to address 51.89.160.135 and port 5000: Connection refused [22:28:18] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [22:29:57] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [22:30:34]