[00:08:35] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 3745 files, 7.213MB, 2019-05-26 00:05:00 (1.1 weeks ago) [00:12:33] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:20:30] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=94%); [06:26:30] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5487 MB (22% inode=94%); [07:10:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjED1 [07:10:15] [02miraheze/services] 07MirahezeSSLBot 03653bd23 - BOT: Updating services config for wikis [08:40:17] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjES3 [08:40:19] [02miraheze/services] 07MirahezeSSLBot 034a3fd92 - BOT: Updating services config for wikis [10:03:33] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.38, 1.83, 0.95 [10:05:33] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.22, 1.23, 0.84 [11:02:37] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 6.82, 3.67, 1.73 [11:03:20] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 4.01, 2.39, 0.98 [11:05:14] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.73, 1.72, 0.90 [11:07:13] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.10, 1.15, 0.79 [11:08:28] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.23, 1.60, 1.45 [13:03:13] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.24, 1.69, 0.77 [13:03:50] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.19, 5.20, 2.68 [13:05:13] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.42, 1.18, 0.69 [13:09:41] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 0.38, 1.78, 1.89 [13:11:38] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.21, 1.27, 1.69 [15:42:41] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjENr [15:42:43] [02miraheze/CreateWiki] 07translatewiki 0300e1ffc - Localisation updates from https://translatewiki.net. [15:42:44] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/fjENo [15:42:46] [02miraheze/ManageWiki] 07translatewiki 031353c6e - Localisation updates from https://translatewiki.net. [15:42:47] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjENK [15:42:49] [02miraheze/WikiDiscover] 07translatewiki 036087272 - Localisation updates from https://translatewiki.net. [15:59:01] Paladox (or anyone): see https://phabricator.miraheze.org/T4436 - Whose up for doing the sysadmin side of things? [15:59:02] [ ⚓ T4436 Import of Twinkle and Userlinks templates to QuIRC wiki ] - phabricator.miraheze.org [16:00:49] Im around, but i carn't commit to a time. [16:07:28] Paladox: That's cool - I'll ping when we're ready. [16:18:28] Hello StephaneTest! If you have any questions feel free to ask and someone should answer soon. [16:19:28] How can I use publictestwiki, I don't understand [16:20:41] StephaneTest https://publictestwiki.com/wiki/ [16:20:42] [ TestWiki ] - publictestwiki.com [16:21:48] Yes, ok, I know the link for the site, but how can I create a new page to see how a project could look like ? [16:22:39] I don't success to obtain this result [16:23:22] I don't succeed sorry [16:26:45] Sorry, I have to go, I will try later, thank you [20:37:49] Sysadmins, about half way through getting my import files ready for you guys [20:57:31] Paladox, JohnLewis: Can one of you do the honours? https://phabricator.miraheze.org/T4436#84410 [20:57:32] [ ⚓ T4436 Import of Twinkle and Userlinks templates to QuIRC wiki ] - phabricator.miraheze.org [20:58:04] paladox: ^ [20:58:12] ok [20:58:15] * JohnLewis is trying to get code ready to deploy [21:01:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuvN [21:01:43] [02miraheze/mw-config] 07paladox 03813882d - Compress quircwiki db Change-Id: Ic03b6982b01b3b37ca76c5c1a217e10bd1ed851d [21:02:34] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03single-page [+1/-4/±4] 13https://git.io/fjuvp [21:02:35] [02miraheze/ManageWiki] 07JohnFLewis 031c9df66 - merge ManageWiki special pages [21:02:37] [02ManageWiki] 07JohnFLewis created branch 03single-page - 13https://git.io/vpSns [21:03:28] paladox, tf is this? https://github.com/miraheze/mw-config/compare/0dd3500b0e8a...813882dbb293#diff-04824105610acf2174db11a5522c2772R8 [21:03:29] [ Comparing 0dd3500b0e8a...813882dbb293 · miraheze/mw-config · GitHub ] - github.com [21:03:39] uh [21:03:47] miraheze/mw-config/master/813882d - paladox The build was broken. https://travis-ci.org/miraheze/mw-config/builds/540950524 [21:03:49] 503 [21:03:54] 503 [21:04:00] reverted on mw2 [21:04:27] :b [21:04:32] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [21:04:32] Service is back [21:04:54] eh, Did I cause you to break something? [21:04:57] miraheze/ManageWiki/single-page/1c9df66 - John Lewis The build passed. https://travis-ci.org/miraheze/ManageWiki/builds/540950868 [21:05:30] uh, i ment reverted on mw1 (not mw2) [21:05:32] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [21:05:37] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw2 mw3 [21:05:46] paladox: correcting server name < fixing it [21:06:32] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [21:06:57] fixed already! (deploying) [21:07:27] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.26, 5.05, 2.85 [21:08:49] deployed JohnLewis [21:09:12] deployed? [21:09:15] * JohnLewis sees no commit [21:09:19] yes? [21:09:27] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 4.74, 5.22, 3.20 [21:09:33] JohnLewis https://github.com/miraheze/mw-config/commit/adb7f06cfd09b5c51771577418809be15c15bc98 [21:09:33] [ Fix syntax · miraheze/mw-config@adb7f06 · GitHub ] - github.com [21:09:40] GH is being slow then [21:09:49] Paladox: I'm still 503 [21:09:59] yeah, that isn't a full fix that commit [21:10:47] Is there an eta? I can't believe this is just from me asking for some templates importing for userlinks and Twinkle [21:11:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:11:05] reverted again [21:11:08] grr. [21:11:15] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [21:11:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:11:32] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [21:11:46] deployed fix, except mw1 because conflict [21:12:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjufk [21:12:12] [02miraheze/mw-config] 07paladox 03adb7f06 - Fix syntax Change-Id: Ia2b49bc26fa0ddcadd83add0136d8867f9941e3f [21:12:31] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [21:12:58] JohnLewis reverted in the repo now [21:13:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:13:04] im just going to re do the commit. [21:13:06] ..? [21:13:14] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 3% [21:13:25] ohh [21:13:26] you're going to redo a commit, when the code you've commited now is fine? [21:13:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:13:28] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 5% [21:13:32] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:13:34] nvm [21:13:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:13:38] just saw your fix [21:13:53] you'd have seen the issue if you checked the logs [21:14:30] because "reverting after pushing a 2 line change after fixing half the code" is a disproportionate way to handle something [21:14:31] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuft [21:14:32] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:14:32] [02miraheze/mw-config] 07JohnFLewis 0375b73c6 - fix syntax [21:15:08] JohnLewis ok [21:15:45] miraheze/mw-config/master/75b73c6 - John Lewis The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/540954098 [21:16:43] [02miraheze/mw-config] 07paladox pushed 035 commits to 03master [+0/-0/±5] 13https://git.io/fjufZ [21:16:44] [02miraheze/mw-config] 07paladox 037f85528 - Revert "Fix syntax" This reverts commit adb7f06cfd09b5c51771577418809be15c15bc98. [21:16:46] [02miraheze/mw-config] 07paladox 0357921d9 - Revert "Compress quircwiki db" This reverts commit 813882dbb2932a6f1079fd19c46e97792d7ce994. [21:16:47] [02miraheze/mw-config] 07paladox 0369052d0 - Revert "Revert "Compress quircwiki db"" This reverts commit 57921d9e890c0490286ec52493c9d85095b0e948. [21:16:49] [02miraheze/mw-config] ... and 2 more commits. [21:17:00] that's me fixing the repo on gerrithub so ignore (i integrated your fix) [21:17:24] fixing the repo? [21:17:37] Yes? I had to pull in your fix. [21:17:45] after reverting back the a known good state. [21:18:03] honestly, I don't care at this point [21:18:17] that's about 5 commits that is unnecessary [21:18:48] yup [21:18:49] in fact, tbh that's 8 commits for 2 lines of code, averaging 4 commits for 1 line [21:19:02] if you agree it's unnecessary, why did you do it? [21:19:38] Only because i wanted to stop the 503. [21:20:00] but you did 5 commits AFTER the 503s were solved by my commit [21:20:47] miraheze/mw-config/master/adb7f06 - paladox The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/540952785 [21:22:54] JohnLewis ok, well im sorry and i didn't see your commit but i belive you said you fixed it. [21:23:39] I did say that, so now the question is, why the need to push commits ~6 minutes after I said that [21:23:52] quircwiki is back up - per PM with paladox the imports are not fully showing [21:24:30] Your imports are not showing as it failed due to redis restarting. [21:24:52] paladox, okay, keep me updated. [21:26:11] JohnLewis at the time, it didn't click that you fixed it. Only after re reading it did i realise (after seeing your commit). [21:26:30] okay, but you had to implement by fix [21:26:33] so you know I fixed it [21:26:41] so yet again, the question stands and remains unanswered [21:27:00] JohnLewis i answered though? [21:27:07] you did? [21:27:10] yes? [21:27:37] and the valid reason for needing 5 additional commits is..? [21:28:17] After the fact it was unnecessary. But when i did it, it was correct to me (reverting). [21:28:40] okay so essentially, it wasn't necessary at all [21:28:41] I had not realised you fixed it. Only after checking mw-config, did i realise you did. [21:29:01] but you included my fix in your commits. You knew it. [21:29:11] no, i reverted first [21:29:12] pushed [21:29:19] then i pulled your fix in [21:29:31] well i reverted my revert [21:29:35] and then pushed with your fix. [21:29:38] * JohnLewis sighs [21:29:39] okay [21:29:49] look, whatever this isn't a useful discussion in any way [21:44:57] !log compressing quircwiki wiki [21:45:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:46:40] !log hacking Redis.php on mw1 to disable redis if using cli [21:46:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:19:42] !log starting more imports for quircwiki [22:19:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:36:15] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03single-page [+1/-4/±4] 13https://git.io/fjuJw [22:36:16] [02miraheze/ManageWiki] 07JohnFLewis 03bb29aa0 - merge ManageWiki special pages [22:37:28] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03single-page [+0/-4/±4] 13https://git.io/fjuJ6 [22:37:30] [02miraheze/ManageWiki] 07JohnFLewis 038871078 - merge ManageWiki special pages [22:38:04] [02ManageWiki] 07JohnFLewis opened pull request 03#102: merge ManageWiki special pages - 13https://git.io/fjuJP [22:39:45] miraheze/ManageWiki/single-page/bb29aa0 - John Lewis The build has errored. https://travis-ci.org/miraheze/ManageWiki/builds/540982430 [22:47:12] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03single-page [+0/-4/±4] 13https://git.io/fjuJF [22:47:14] [02miraheze/ManageWiki] 07JohnFLewis 0305ec896 - merge ManageWiki special pages [22:47:15] [02ManageWiki] 07JohnFLewis synchronize pull request 03#102: merge ManageWiki special pages - 13https://git.io/fjuJP [22:48:47] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03single-page [+0/-4/±4] 13https://git.io/fjuJN [22:48:49] [02miraheze/ManageWiki] 07JohnFLewis 033e8a857 - merge ManageWiki special pages [22:48:50] [02ManageWiki] 07JohnFLewis synchronize pull request 03#102: merge ManageWiki special pages - 13https://git.io/fjuJP [22:54:19] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjuJx [22:54:20] [02miraheze/ManageWiki] 07paladox 03887bbf1 - Add namespaces support to the api [22:54:22] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [22:54:23] [02ManageWiki] 07paladox opened pull request 03#103: Add namespaces support to the api - 13https://git.io/fjuJp [22:54:55] [02ManageWiki] 07paladox synchronize pull request 03#103: Add namespaces support to the api - 13https://git.io/fjuJp [22:54:56] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjuJj [22:54:58] [02miraheze/ManageWiki] 07paladox 0322ab76d - Update ApiQueryWikiConfig.php [22:55:50] [02miraheze/ManageWiki] 07JohnFLewis pushed 033 commits to 03master [+0/-8/±20] 13https://git.io/fjuUf [22:55:51] [02miraheze/ManageWiki] 07JohnFLewis 030f9df98 - Merge branch 'master' into single-page [22:55:53] [02miraheze/ManageWiki] 07JohnFLewis 036c8eb01 - Merge branch 'single-page' [22:55:54] [02ManageWiki] 07JohnFLewis closed pull request 03#102: merge ManageWiki special pages - 13https://git.io/fjuJP [22:56:32] [02ManageWiki] 07paladox closed pull request 03#103: Add namespaces support to the api - 13https://git.io/fjuJp [22:56:34] [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns [22:56:36] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1 [22:57:25] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjuUJ [22:57:26] [02miraheze/mediawiki] 07JohnFLewis 0368d3854 - update MW [22:57:48] miraheze/ManageWiki/paladox-patch-1/22ab76d - paladox The build was broken. https://travis-ci.org/miraheze/ManageWiki/builds/540988582 [23:09:46] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:10:34] bug en ManageWiki Namespaces paladox JohnLewis [23:10:56] Hispano76_: yep, investigating... it worked literally 5 mintues ago [23:11:24] okay [23:22:57] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuU4 [23:22:59] [02miraheze/ManageWiki] 07JohnFLewis 03374a2d6 - fixes to new changes [23:23:22] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjuUB [23:23:23] [02miraheze/mediawiki] 07JohnFLewis 0311cf5ba - update MW [23:23:44] Hispano76: should be working now [23:55:48] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjuUd [23:55:49] [02miraheze/ManageWiki] 07JohnFLewis 0395a432d - check if int [23:56:12] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjuUF [23:56:14] [02miraheze/mediawiki] 07JohnFLewis 03104eb6b - MW