[00:08:49] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 47% [00:10:55] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 34% [00:28:37] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [00:30:37] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15174 bytes in 0.403 second response time [00:43:11] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [00:44:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvJ24 [00:44:18] [02miraheze/puppet] 07paladox 0313432f5 - mediawiki: Add global headers to local {} Setting headers in global will prevent the global headers from taking effect, we have to add it to the local {} block for it to take effect again. Also switch off server_tokens. [00:44:19] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:44:21] [02puppet] 07paladox opened pull request 03#1185: mediawiki: Add global headers to local {} - 13https://git.io/JvJ2B [00:45:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvJ2z [00:45:17] [02miraheze/puppet] 07paladox 030f08ad7 - Update nginx.conf.erb [00:45:19] [02puppet] 07paladox synchronize pull request 03#1185: mediawiki: Add global headers to local {} - 13https://git.io/JvJ2B [00:47:24] [02puppet] 07paladox edited pull request 03#1185: mediawiki: Add global headers to local {} - 13https://git.io/JvJ2B [00:47:49] [02puppet] 07paladox closed pull request 03#1185: mediawiki: Add global headers to local {} - 13https://git.io/JvJ2B [00:47:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvJ2V [00:47:52] [02miraheze/puppet] 07paladox 0323ed665 - mediawiki: Add global headers to local {} (#1185) * mediawiki: Add global headers to local {} Setting headers in global will prevent the global headers from taking effect, we have to add it to the local {} block for it to take effect again. Also switch off server_tokens. * Update nginx.conf.erb [00:47:53] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:47:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [00:49:09] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:12:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvJat [01:12:19] [02miraheze/puppet] 07paladox 03072554c - Fix X-XSS-Protection header [01:12:20] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [01:12:22] [02puppet] 07paladox opened pull request 03#1186: Fix X-XSS-Protection header - 13https://git.io/JvJaq [01:13:27] [02puppet] 07paladox synchronize pull request 03#1186: Fix X-XSS-Protection header - 13https://git.io/JvJaq [01:13:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvJam [01:13:30] [02miraheze/puppet] 07paladox 0344cd1ae - Update nginx.conf.erb [01:21:45] !log MariaDB [sagan4wiki]> ALTER TABLE /*_*/cargo_tables ADD field_helper_tables text NOT NULL; [01:21:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:32:50] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 22983 MB (6% inode=95%); [01:40:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvJVv [01:40:35] [02miraheze/mediawiki] 07paladox 0361ce1b9 - Update Cargo [01:43:27] !log sudo -u www-data php update.php --wiki=sagan4wiki --quick --force - mw1 [01:43:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:50:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJVs [01:50:05] [02miraheze/puppet] 07paladox 035e08017 - mediawiki: Increase opcache to 312 [01:57:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJVB [01:57:10] [02miraheze/puppet] 07paladox 0303339f9 - Update mediawiki.conf [02:54:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:54:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [02:56:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:56:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJ6O [05:55:09] [02miraheze/services] 07MirahezeSSLBot 03cb9135d - BOT: Updating services config for wikis [06:14:56] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21979 MB (5% inode=95%); [06:27:15] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3143 MB (13% inode=94%); [06:30:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJ6d [06:30:09] [02miraheze/services] 07MirahezeSSLBot 0301a0d7f - BOT: Updating services config for wikis [07:12:09] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.73, 2.27, 1.41 [07:14:05] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.69, 1.58, 1.33 [07:41:38] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 4.90, 3.34, 1.71 [07:45:31] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.15, 1.33, 1.50 [08:24:10] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CRITICAL - load average: 3.31, 7.00, 3.70 [08:30:00] PROBLEM - cp2 Current Load on cp2 is WARNING: WARNING - load average: 0.33, 0.91, 1.87 [08:31:58] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 1.07, 0.78, 1.59 [09:01:06] chip having irc problems? :P [09:01:22] uh [09:01:29] I shouldn’t be [09:01:40] does it show me leaving and joining or something [09:02:01] No, just shows you joining then changing your nick [09:02:43] oh [09:02:51] that’s weird [09:03:27] I’m leaving and joining my bouncer, testing out some modules, but the bouncer isn’t disconnecting from IRC [09:03:39] I wonder why it’s sending joins [09:04:14] Heh [09:58:27] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Thu 30 Jan 2020 09:54:37 AM GMT +0000). [09:58:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJME [09:58:42] [02miraheze/ssl] 07MirahezeSSLBot 037287623 - Bot: Update SSL cert for vmcodex.net [09:59:58] PROBLEM - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bh.gyaanipedia.co.in' expires in 15 day(s) (Thu 30 Jan 2020 09:56:20 AM GMT +0000). [10:00:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJMw [10:00:16] [02miraheze/ssl] 07MirahezeSSLBot 03c657cdd - Bot: Update SSL cert for bh.gyaanipedia.co.in [10:02:26] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Mon 13 Apr 2020 08:58:34 AM GMT +0000. [10:12:03] RECOVERY - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'bh.gyaanipedia.co.in' will expire on Mon 13 Apr 2020 09:00:08 AM GMT +0000. [14:23:06] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2641 MB (10% inode=94%); [14:27:00] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2654 MB (11% inode=94%); [14:33:04] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%); [16:42:36] paladox: which is MF https://phabricator.miraheze.org/T5101#97151 curiosity [16:42:38] [ ⚓ T5101 Tabs on Mobile ] - phabricator.miraheze.org [16:42:47] MobileFrontend [16:42:59] ah, ok thanks paladox [16:59:51] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvJbE [16:59:53] [02miraheze/mediawiki] 07paladox 03a797606 - Update MobileFrontend [17:36:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:37:17] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 14.17, 21.98, 13.12 [17:38:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:39:17] !log apt-get upgrade - lizardfs6 [17:39:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:41:13] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 6.71, 13.43, 11.58 [19:24:20] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJhG [19:24:21] [02miraheze/MirahezeMagic] 07paladox 0365b9e28 - Remove duplicate $c configs [19:26:47] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvJhc [19:26:48] [02miraheze/MirahezeMagic] 07paladox 03b452c64 - GlobalNewFiles: Introduce new column "files_timestamp" [19:26:50] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [19:26:51] [02MirahezeMagic] 07paladox opened pull request 03#96: GlobalNewFiles: Introduce new column "files_timestamp" - 13https://git.io/JvJhC [19:32:30] [02MirahezeMagic] 07paladox edited pull request 03#96: GlobalNewFiles: Use "files_timestamp" column - 13https://git.io/JvJhC [19:35:11] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvJjz [19:35:12] [02miraheze/MirahezeMagic] 07paladox 036bfa466 - Update GlobalNewFilesPager.php [19:35:14] [02MirahezeMagic] 07paladox synchronize pull request 03#96: GlobalNewFiles: Use "files_timestamp" column - 13https://git.io/JvJhC [19:37:55] [02MirahezeMagic] 07paladox closed pull request 03#96: GlobalNewFiles: Use "files_timestamp" column - 13https://git.io/JvJhC [19:37:56] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvJj2 [19:37:57] [02miraheze/MirahezeMagic] 07paladox 03e17f078 - GlobalNewFiles: Use "files_timestamp" column (#96) * GlobalNewFiles: Introduce new column "files_timestamp" * Update GlobalNewFilesPager.php [19:37:59] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [19:38:01] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [19:38:28] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvJjV [19:38:30] [02miraheze/mediawiki] 07paladox 03e178543 - Update MirahezeMagic [19:38:35] paladox: uh [19:38:41] You've not really review that file [19:38:51] files_timestamp is being used twice [19:38:52] I did? [19:39:04] Where [19:39:13] https://github.com/miraheze/MirahezeMagic/blob/6bfa466b3ebde9b47fb0093efe3587b72098833c/includes/GlobalNewFiles/GlobalNewFilesHooks.php#L17 [19:39:14] [ MirahezeMagic/GlobalNewFilesHooks.php at 6bfa466b3ebde9b47fb0093efe3587b72098833c · miraheze/MirahezeMagic · GitHub ] - github.com [19:39:15] https://github.com/miraheze/MirahezeMagic/blob/6bfa466b3ebde9b47fb0093efe3587b72098833c/includes/GlobalNewFiles/GlobalNewFilesHooks.php#L22 [19:39:16] oh [19:39:21] [ MirahezeMagic/GlobalNewFilesHooks.php at 6bfa466b3ebde9b47fb0093efe3587b72098833c · miraheze/MirahezeMagic · GitHub ] - github.com [19:39:43] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvJjr [19:39:45] [02miraheze/MirahezeMagic] 07paladox 0325b944a - Remove dupe [19:40:03] I'm curious though. why remove that and not the one you added? [19:40:22] When the one I added is the one which is being used for all existing entrys [19:40:27] JohnLewis i put it in order :) [19:40:33] oh [19:40:35] damn [19:40:45] * paladox knew he should have got you to review [19:41:12] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvJji [19:41:13] [02miraheze/MirahezeMagic] 07paladox 03bda28dd - Fix reordering [19:41:15] [02MirahezeMagic] 07paladox created branch 03paladox-patch-3 - 13https://git.io/fQRGX [19:41:16] [02MirahezeMagic] 07paladox opened pull request 03#97: Fix reordering - 13https://git.io/JvJjP [19:41:21] JohnLewis https://github.com/miraheze/MirahezeMagic/pull/97 better? [19:41:50] [ Fix reordering by paladox · Pull Request #97 · miraheze/MirahezeMagic · GitHub ] - github.com [19:42:08] no because all you've done is reordered it, when my comment was much rather "remove your line, leave mine in" [19:43:56] JohnLewis i'm now confused. [19:44:13] You added line 17 [19:44:18] I added line 22 [19:44:29] yup [19:44:30] The correct thing would have been to remove L17, not L22 [19:44:39] You remove the dupe, not the original [19:45:09] oh [19:45:15] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 43% [19:45:46] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvJjF [19:45:48] [02miraheze/mediawiki] 07paladox 0369efea5 - Update MirahezeMagic [19:46:10] Why have you updated it? [19:47:14] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 32% [19:47:45] Because you said "<+JohnLewis> You remove the dupe, not the original" (e.g i removed it) and you also said "<+JohnLewis> no because all you've done is reordered it, when my comment was much rather "remove your line, leave mine in"". And also because it updates to the latest commit i did. [19:48:08] But you have to make another commit to actually do what I asked, then make another commit to update it [19:48:24] uh [19:48:28] JohnLewis i did [19:48:30] and you said no [19:48:50] https://github.com/miraheze/MirahezeMagic/pull/97 is the commit i did, which you said no. [19:48:51] yes, no to your change, because the correct thing, as I just said, was remove L17 not L22 [19:48:59] Because your change is still wrong [19:49:05] that is removing L17 [19:49:13] [ Fix reordering by paladox · Pull Request #97 · miraheze/MirahezeMagic · GitHub ] - github.com [19:49:19] and moving the SAME L17 to L22 [19:49:22] Not adding back in L22 [19:49:26] which is what I said. [19:49:49] [02MirahezeMagic] 07paladox synchronize pull request 03#97: Fix reordering - 13https://git.io/JvJjP [19:49:50] done [19:49:51] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvUef [19:49:52] [02miraheze/MirahezeMagic] 07paladox 036f0edd8 - Update GlobalNewFilesHooks.php [19:50:09] dbw->timestamp [19:50:13] That is my point [19:50:24] You should have not even change that function at all [19:50:29] oh [19:50:59] done [19:50:59] [02MirahezeMagic] 07paladox synchronize pull request 03#97: Fix reordering - 13https://git.io/JvJjP [19:51:01] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvUeJ [19:51:02] [02miraheze/MirahezeMagic] 07paladox 03dd48448 - Update GlobalNewFilesHooks.php [19:51:19] Yes [19:51:29] ok, merging [19:51:38] [02MirahezeMagic] 07paladox closed pull request 03#97: Fix reordering - 13https://git.io/JvJjP [19:51:39] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUeU [19:51:41] [02miraheze/MirahezeMagic] 07paladox 03dc181cd - Fix reordering (#97) * Fix reordering * Update GlobalNewFilesHooks.php * Update GlobalNewFilesHooks.php [19:53:18] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JvUek [19:53:20] [02miraheze/MirahezeMagic] 07paladox 037ff098b - Fix so that the date is correctly formatted [19:53:21] [02MirahezeMagic] 07paladox created branch 03paladox-patch-4 - 13https://git.io/fQRGX [19:53:23] [02MirahezeMagic] 07paladox opened pull request 03#98: Fix so that the date is correctly formatted - 13https://git.io/JvUeI [19:53:37] JohnLewis can you review ^ please? [19:53:49] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/fQRGX [19:53:51] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-3 [19:53:57] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [19:53:59] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [19:54:00] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [19:54:02] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [19:54:03] [02miraheze/MirahezeMagic] 07paladox deleted branch 03Southparkfan-patch-1 [19:54:05] [02MirahezeMagic] 07paladox deleted branch 03Southparkfan-patch-1 - 13https://git.io/fQRGX [19:54:06] [02miraheze/MirahezeMagic] 07paladox deleted branch 03RhinosF1-patch-1 [19:54:08] [02MirahezeMagic] 07paladox deleted branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [19:54:20] yes [19:54:23] miraheze/MirahezeMagic/paladox-patch-4/7ff098b - paladox The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/637072629 [19:55:38] JohnLewis so looks good to you [19:55:40] ? [19:55:44] yes [19:55:51] thanks! [19:56:02] [02MirahezeMagic] 07paladox closed pull request 03#98: Fix so that the date is correctly formatted - 13https://git.io/JvUeI [19:56:04] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUeq [19:56:05] [02miraheze/MirahezeMagic] 07paladox 0324d16ef - Fix so that the date is correctly formatted (#98) [19:56:07] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-4 [19:56:09] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/fQRGX [19:56:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvUem [19:56:33] [02miraheze/mediawiki] 07paladox 033006274 - Update MirahezeMagic [19:59:12] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45% [20:07:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [20:08:28] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvUeV [20:08:30] [02miraheze/MirahezeMagic] 07paladox 03e5e6ad9 - GlobalNewFile: order by timestamp [20:08:31] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [20:08:33] [02MirahezeMagic] 07paladox opened pull request 03#99: GlobalNewFile: order by timestamp - 13https://git.io/JvUer [20:13:13] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 40% [20:21:13] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 39% [20:26:12] [02MirahezeMagic] 07paladox synchronize pull request 03#99: GlobalNewFile: order by timestamp - 13https://git.io/JvUer [20:26:14] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvUv3 [20:26:15] [02miraheze/MirahezeMagic] 07paladox 0307b8a9c - Update GlobalNewFilesPager.php [20:26:23] [02MirahezeMagic] 07paladox edited pull request 03#99: GlobalNewFile: Fix ordering by timestamp - 13https://git.io/JvUer [20:27:27] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 42% [20:27:39] [02MirahezeMagic] 07paladox edited pull request 03#99: GlobalNewFile: Fix ordering by timestamp - 13https://git.io/JvUer [20:27:49] [02MirahezeMagic] 07paladox closed pull request 03#99: GlobalNewFile: Fix ordering by timestamp - 13https://git.io/JvUer [20:27:50] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUvs [20:27:52] [02miraheze/MirahezeMagic] 07paladox 03dacf32f - GlobalNewFile: Fix ordering by timestamp (#99) * GlobalNewFile: order by timestamp * Update GlobalNewFilesPager.php [20:27:53] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [20:27:55] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [20:28:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvUvC [20:28:32] [02miraheze/mediawiki] 07paladox 0367e17f0 - Update MirahezeMagic [20:29:33] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [20:33:45] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 42% [20:35:46] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [20:45:05] paladox, GlobalNewFiles appears to be broken, as you can only view the first page of results [20:45:12] oh [20:46:08] and you get a dbquery error if you attempt to switch the sort direction [20:48:08] thanks [21:07:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvUJT [21:07:35] [02miraheze/mediawiki] 07paladox 03e1a05a7 - Revert "Update MirahezeMagic" This reverts commit 67e17f081ff404e06ddf6383c03d88d230799bca. [21:08:18] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45% [21:12:25] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 35% [21:17:40] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:23:46] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [21:31:44] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [21:44:03] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 45% [21:52:21] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [21:56:30] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 41% [22:00:34] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 36% [22:06:36] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 40% [22:07:03] Zppix: turn scary transclude on [22:07:24] JohnLewis, Voidwalker: around? [22:08:00] kinda [22:08:36] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 14% [22:08:51] Voidwalker: can you look at my talk page, it’s clearly a user evading a ban based on edit summary that 8ight is complaing about and actions [22:12:18] RhinosF1: hmm? [22:12:43] Zppix: it’ll fix that error on the page you just used to test [22:13:10] RhinosF1: i see [22:14:08] they both don't look like they're being very constructive [22:15:27] Voidwalker: I’m not sure what 8ight did exactly but RandomUser seems to be evading a ban as they refer to avoiding impersonation and stop trying to get me banned at only 16 global edits. [22:15:44] You’re free to take whatever action needed [22:16:59] I'm going to leave it for now and see what happens [22:17:09] Ill keep an eye out too [22:17:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUTK [22:17:12] [02miraheze/puppet] 07paladox 037bdc7af - varnish: Fix variable to correctly find the status code [22:20:41] Voidwalker: I’ve just redacted 3 edits for giving an age. They’ve also made a threat of police action. [22:25:28] RhinosF1 apologies, that alert is actually correct and it was wrong on my part to brush that off. Fixed the script above. [22:25:58] paladox: ok [22:29:13] Voidwalker: I’m not sure whether that’s a personal attack now and to block 8ight [22:29:15] https://meta.miraheze.org/w/index.php?diff=94425&oldid=94358&rcid=371878 [22:29:18] [ Difference between revisions of "Stewards' noticeboard" - Miraheze Meta ] - meta.miraheze.org [22:29:25] Zppix: ? [22:32:36] RhinosF1: replied [22:32:58] Zppix: good, slightly worried about 8ight’s civilty though [22:33:35] RhinosF1: if either of them interact i will enforce blocks [22:33:46] 2 way interaction block [22:34:13] Zppix: Great, I’m going to sleep. If 8ight even dreams about them, I’ll support a block. [22:53:33] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:55:05] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvUkB [22:55:07] [02miraheze/MirahezeMagic] 07paladox 039a2dc24 - Fix pagination including ordering by timestamp [22:55:08] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [22:55:10] [02MirahezeMagic] 07paladox opened pull request 03#100: Fix pagination including ordering by timestamp - 13https://git.io/JvUkR [22:56:08] Voidwalker fixed ^ [22:57:06] [02MirahezeMagic] 07paladox closed pull request 03#100: Fix pagination including ordering by timestamp - 13https://git.io/JvUkR [22:57:08] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUkz [22:57:09] [02miraheze/MirahezeMagic] 07paladox 033c11b56 - Fix pagination including ordering by timestamp (#100) [22:57:11] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [22:57:12] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [22:57:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvUka [22:57:44] [02miraheze/mediawiki] 07paladox 0397a13e3 - Update MirahezeMagic [22:59:18] !log rebuild lc on mw* and lizardfs6 [22:59:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:59:57] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUkr [22:59:59] [02miraheze/dns] 07paladox 03e5ae495 - Depool cp2 Reimaging as debian buster [23:00:26] ^^ tests will error, known. That's due to openstream removing that download link (you have to log in to download due to a law change in california) [23:01:08] miraheze/dns/master/e5ae495 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/637151936 [23:02:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUkD [23:02:32] [02miraheze/puppet] 07paladox 037576c99 - Revert "mediawiki: Increase opcache to 312" This reverts commit 5e080176c38d0fe1401ba3260d5dcfa23abb17dc. [23:04:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:04:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [23:06:17] !log restart php7.3-fpm on mw3 for config change [23:06:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:08:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:09:11] !log turn TUN off on cp2 [23:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:09:35] !log restart php7.3-fpm on mw2 for config change [23:09:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:10:05] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.96, 7.76, 6.79 [23:10:21] !log reinstall cp2 as debian buster [23:10:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:10:54] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:49] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:12:07] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:12:20] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.98, 7.98, 7.08 [23:12:39] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:12:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [23:12:48] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address thesimswiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:13:26] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.counterculturelabs.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:13:29] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: connect to address 107.191.126.23 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:13:30] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:13:39] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:14:02] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:14:04] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:14:11] PROBLEM - cp2 Current Load on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:14:14] PROBLEM - cp2 Puppet on cp2 is CRITICAL: connect to address 107.191.126.23 port 5666: Connection refusedconnect to host 107.191.126.23 port 5666: Connection refused [23:14:21] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.63, 6.47, 6.71 [23:19:17] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is UNKNOWN: NRPE: Unable to read output [23:19:46] PROBLEM - cp2 Varnish Backends on cp2 is UNKNOWN: NRPE: Unable to read output [23:19:56] RECOVERY - cp2 Current Load on cp2 is OK: OK - load average: 1.59, 1.07, 0.57 [23:20:16] PROBLEM - cp2 Puppet on cp2 is UNKNOWN: NRPE: Unable to read output [23:20:45] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 41613 MB (96% inode=97%); [23:20:46] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.403 second response time [23:21:20] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 1.406 second response time [23:21:27] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15289 bytes in 0.402 second response time [23:21:53] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43680 bytes in 0.643 second response time [23:22:03] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15301 bytes in 0.296 second response time [23:23:15] https://commons.miraheze.org/wiki/Special:GlobalNewFiles throwing dbquery error [23:23:20] [ Database error - Miraheze Commons ] - commons.miraheze.org [23:24:50] uh [23:25:42] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [23:26:06] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUI0 [23:26:08] [02miraheze/MirahezeMagic] 07paladox 0383d10e5 - Fix [23:26:32] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvUIE [23:26:34] [02miraheze/mediawiki] 07paladox 0374fad25 - Update MirahezeMagic [23:26:49] Voidwalker fixed [23:26:57] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [23:26:57] PROBLEM - db5 Disk Space on db5 is WARNING: DISK WARNING - free space: / 20072 MB (10% inode=99%); [23:27:04] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 97% [23:27:04] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Wed 01 Apr 2020 04:43:14 AM GMT +0000. [23:27:09] yeah, that did it [23:28:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:29:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:29:09] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUI2 [23:29:11] [02miraheze/dns] 07paladox 03946471d - Revert "Depool cp2" This reverts commit e5ae49594ab67212010d1a5975faedc87e8aa4c5. [23:29:21] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1536 bytes in 0.502 second response time [23:30:23] miraheze/dns/master/946471d - Paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/637162276 [23:31:05] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 6% [23:34:16] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:35:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvUI1 [23:35:40] [02miraheze/puppet] 07paladox 0369925e1 - Update mediawiki.conf