[00:02:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JLoV5 [00:02:49] [02miraheze/puppet] 07paladox 03aba0bd7 - varnish: Do not cache sitemaps [00:02:50] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [00:02:52] [02puppet] 07paladox opened pull request 03#1561: varnish: Do not cache sitemaps - 13https://git.io/JLoVF [00:02:57] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.75, 8.65, 8.13 [00:03:03] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.69, 21.29, 20.34 [00:03:42] RhinosF1: check py3.9 shell [00:03:43] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 9.56, 7.92, 7.54 [00:06:12] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [00:06:22] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:07:03] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.64, 22.63, 21.20 [00:08:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JLowW [00:08:10] [02miraheze/puppet] 07paladox 032550065 - mediawiki: Fix private location Some wikis contained private in their name so fix this by adding a slash. [00:08:11] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:08:16] [02puppet] 07paladox opened pull request 03#1562: mediawiki: Fix private location - 13https://git.io/JLow8 [00:08:48] [02puppet] 07paladox closed pull request 03#1562: mediawiki: Fix private location - 13https://git.io/JLow8 [00:08:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLow0 [00:08:51] [02miraheze/puppet] 07paladox 0308137dc - mediawiki: Fix private location (#1562) Some wikis contained private in their name so fix this by adding a slash. [00:08:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:08:54] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:09:10] [02puppet] 07paladox closed pull request 03#1561: varnish: Do not cache sitemaps - 13https://git.io/JLoVF [00:09:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLowg [00:09:13] [02miraheze/puppet] 07paladox 03507a416 - varnish: Do not cache sitemaps (#1561) [00:09:15] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [00:09:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [00:13:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.63, 19.35, 20.38 [00:13:24] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.14, 6.86, 6.12 [00:13:51] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.82, 7.23, 6.53 [00:14:27] paladox: I just had another spam of 502/503 [00:14:58] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 5.32, 7.22, 7.82 [00:15:46] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.80, 6.74, 6.43 [00:16:58] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.03, 7.44, 7.81 [00:17:16] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.22, 7.68, 6.67 [00:17:43] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.24, 7.74, 7.92 [00:18:32] !log root@gluster1:/home/paladox# gluster volume set mvol performance.nl-cache off [00:18:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:58] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.40, 7.64, 7.85 [00:21:07] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.11, 6.08, 6.23 [00:21:46] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLoov [00:21:48] [02miraheze/MirahezeMagic] 07RhinosF1 03cb502d6 - gsi: be even more verbose and try again if MH seems down [00:22:03] paladox: if you deploy that then it'll be fine for most stuff [00:22:13] should also backoff in a 5xx [00:22:13] ok [00:22:48] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLook [00:22:50] [02miraheze/mediawiki] 07paladox 037260744 - Update MirahezeMagic [00:22:54] i'm actually sleeping [00:22:59] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 8.77, 7.46, 7.66 [00:22:59] and shutting laptop down [00:24:26] !log depool mw[45] and repool, also increase core by 1 (so 5 instead of 4) experimenting to seem if that'll help with the load [00:24:58] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 6.89, 7.56, 7.70 [00:25:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:25:53] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 10.12, 7.86, 6.85 [00:26:01] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:23] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [00:27:28] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb [00:27:42] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [00:28:01] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 1.36, 0.44, 0.16 [00:28:05] PROBLEM - mw4 Check Gluster Clients on mw4 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:28:08] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [00:28:23] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [00:28:26] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 10.13, 8.84, 7.38 [00:28:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb [00:28:54] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:29:28] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:29:48] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:08] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:10] RECOVERY - mw4 Check Gluster Clients on mw4 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:30:21] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:29] PROBLEM - cp9 Stunnel Http for mw6 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:30:57] PROBLEM - monarchists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for monarchists.wiki could not be found [00:30:58] PROBLEM - innersphere.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for innersphere.tk could not be found [00:31:03] PROBLEM - www.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fraterniterebellion.com could not be found [00:31:25] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 0.92, 0.48, 0.19 [00:31:32] PROBLEM - mw5 Check Gluster Clients on mw5 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [00:32:09] PROBLEM - cp9 Stunnel Http for mw7 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:18] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [00:32:20] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 9.680 second response time [00:32:31] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:32:34] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [00:32:40] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:33:04] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 20069 bytes in 3.234 second response time [00:33:25] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [00:33:26] RECOVERY - mw5 Check Gluster Clients on mw5 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [00:33:43] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [00:33:58] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 0.948 second response time [00:34:08] RECOVERY - cp9 Stunnel Http for mw7 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 1.888 second response time [00:34:35] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 6.272 second response time [00:34:44] RECOVERY - cp9 Stunnel Http for mw6 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 6.371 second response time [00:35:57] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [00:36:13] !log depool mw[67] and repool, also increase core by 1 (so 5 instead of 4) experimenting to seem if that'll help with the load [00:36:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:31] PROBLEM - wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikilukas.tk could not be found [00:36:34] PROBLEM - www.museummiddelland.nl - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.museummiddelland.nl could not be found [00:36:34] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 0.020 second response time [00:36:35] PROBLEM - wiki.yunachannel.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.yunachannel.com could not be found [00:36:49] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:36:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:36:53] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 16642 bytes in 1.043 second response time [00:37:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:37:54] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: rDNS OK - monarchists.wiki reverse DNS resolves to cp7.miraheze.org [00:37:55] RECOVERY - innersphere.tk - reverse DNS on sslhost is OK: rDNS OK - innersphere.tk reverse DNS resolves to cp7.miraheze.org [00:38:03] RECOVERY - www.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - www.fraterniterebellion.com reverse DNS resolves to cp6.miraheze.org [00:39:25] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [00:39:27] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [00:39:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [00:39:42] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw6 [00:40:18] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.61, 8.56, 4.27 [00:40:24] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 0.78, 0.32, 0.12 [00:41:25] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [00:41:25] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [00:41:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [00:42:16] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 9.04, 8.59, 4.78 [00:43:17] RECOVERY - www.museummiddelland.nl - reverse DNS on sslhost is OK: rDNS OK - www.museummiddelland.nl reverse DNS resolves to cp6.miraheze.org [00:43:20] RECOVERY - wiki.yunachannel.com - reverse DNS on sslhost is OK: rDNS OK - wiki.yunachannel.com reverse DNS resolves to cp7.miraheze.org [00:43:29] RECOVERY - wikilukas.tk - reverse DNS on sslhost is OK: rDNS OK - wikilukas.tk reverse DNS resolves to cp7.miraheze.org [00:43:51] PROBLEM - mw6 Puppet on mw6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [00:45:22] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw7 [00:45:22] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw7 [00:45:34] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [00:45:49] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.47, 1.46, 0.50 [00:46:10] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.79, 8.24, 5.46 [00:47:19] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [00:47:21] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [00:47:39] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLo6H [00:47:40] [02miraheze/dns] 07paladox 036d3995a - depool gb (cp6/cp7) [00:51:42] !log set cp6/7 cores to 3 [00:51:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:56] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [00:53:30] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLoiR [00:53:31] [02miraheze/dns] 07paladox 030e7d698 - Revert "depool gb (cp6/cp7)" This reverts commit 6d3995a5600a45d551aa1e7dbb5489be30cd3035. [00:56:20] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [00:56:57] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:57:36] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [00:57:58] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:57:59] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:58:16] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [01:01:30] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [01:13:34] miraheze/MirahezeMagic/paladox-patch-1/d8e8c65 - paladox The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/750933101 [01:19:51] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:23:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLoMJ [01:23:16] [02miraheze/mw-config] 07paladox 03e4b0408 - Remove duplication and also increase limit to 200 for post [01:32:31] [02mediawiki] 07Universal-Omega closed pull request 03#729: Bump extensions/ManageWiki from `696cc07` to `932e361` - 13https://git.io/JLaFM [01:32:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#729: Bump extensions/ManageWiki from `696cc07` to `932e361` - 13https://git.io/JLoDt [01:32:40] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ManageWiki-932e361 [01:32:41] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/ManageWiki-932e361 - 13https://git.io/vbL5b [01:37:47] [02mediawiki] 07Universal-Omega closed pull request 03#738: Bump extensions/PageForms from `75103fb` to `d2e48e5` - 13https://git.io/JLabe [01:37:48] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLoDD [01:37:50] [02miraheze/mediawiki] 07Universal-Omega 03bd8300f - Merge pull request #738 from miraheze/dependabot/submodules/REL1_35/extensions/PageForms-d2e48e5 Bump extensions/PageForms from `75103fb` to `d2e48e5` [01:37:51] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-d2e48e5 - 13https://git.io/vbL5b [01:37:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-d2e48e5 [01:37:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JLoDy [01:37:56] [02miraheze/mw-config] 07paladox 035b8ecfd - Move timeout code to a seperate file loaded using auto_prepend_file [01:37:57] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:38:03] [02mw-config] 07paladox opened pull request 03#3580: Move timeout code to a seperate file loaded using auto_prepend_file - 13https://git.io/JLoD5 [01:38:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JLoDb [01:38:36] [02miraheze/mw-config] 07paladox 03946fdec - Update LocalSettings.php [01:38:37] [02mw-config] 07paladox synchronize pull request 03#3580: Move timeout code to a seperate file loaded using auto_prepend_file - 13https://git.io/JLoD5 [01:39:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JLoyf [01:39:56] [02miraheze/puppet] 07paladox 03bdd3e58 - mediawiki: Set auto_prepend_file to /srv/mediawiki/config/PhpAutoPrepend.php [01:39:58] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:39:59] [02puppet] 07paladox opened pull request 03#1563: mediawiki: Set auto_prepend_file to /srv/mediawiki/config/PhpAutoPrep… - 13https://git.io/JLoyJ [01:40:06] [02puppet] 07paladox edited pull request 03#1563: mediawiki: Set auto_prepend_file to /srv/mediawiki/config/PhpAutoPrepend.php - 13https://git.io/JLoyJ [01:40:11] [02puppet] 07paladox edited pull request 03#1563: mediawiki: Set auto_prepend_file to /srv/mediawiki/config/PhpAutoPrepend.php - 13https://git.io/JLoyJ [01:40:17] [02puppet] 07paladox edited pull request 03#1563: mediawiki: Set auto_prepend_file to PhpAutoPrepend.php - 13https://git.io/JLoyJ [01:41:03] [02mw-config] 07paladox closed pull request 03#3580: Move timeout code to a seperate file loaded using auto_prepend_file - 13https://git.io/JLoD5 [01:41:04] [02mediawiki] 07Universal-Omega closed pull request 03#753: Bump extensions/SocialProfile from `504501c` to `88ae8b1` - 13https://git.io/JLabw [01:41:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JLoyt [01:41:07] [02miraheze/mw-config] 07paladox 031650e45 - Move timeout code to a seperate file loaded using auto_prepend_file (#3580) * Move timeout code to a seperate file loaded using auto_prepend_file * Update LocalSettings.php [01:41:09] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLoyq [01:41:10] [02miraheze/mediawiki] 07Universal-Omega 03437fdc8 - Merge pull request #753 from miraheze/dependabot/submodules/REL1_35/extensions/SocialProfile-88ae8b1 Bump extensions/SocialProfile from `504501c` to `88ae8b1` [01:41:12] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:41:13] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:41:15] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-88ae8b1 - 13https://git.io/vbL5b [01:41:16] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-88ae8b1 [01:41:18] [02puppet] 07paladox closed pull request 03#1563: mediawiki: Set auto_prepend_file to PhpAutoPrepend.php - 13https://git.io/JLoyJ [01:41:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLoyY [01:41:21] [02miraheze/puppet] 07paladox 03dce16fe - mediawiki: Set auto_prepend_file to /srv/mediawiki/config/PhpAutoPrepend.php (#1563) [01:44:40] miraheze/MirahezeMagic/master/cb502d6 - RhinosF1 The build passed. https://travis-ci.org/miraheze/MirahezeMagic/builds/750940820 [01:49:09] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw7 [01:51:09] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [01:56:28] miraheze/mw-config/paladox-patch-1/946fdec - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/750949220 [02:03:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JLoHL [02:03:49] [02miraheze/puppet] 07paladox 0326633c3 - base::puppet: Use nice for puppet round [02:03:51] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [02:03:52] [02puppet] 07paladox opened pull request 03#1564: base::puppet: Use nice for puppet round - 13https://git.io/JLoHq [02:04:07] [02puppet] 07paladox closed pull request 03#1564: base::puppet: Use nice for puppet round - 13https://git.io/JLoHq [02:04:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLoH3 [02:04:10] [02miraheze/puppet] 07paladox 03acd9be6 - base::puppet: Use nice for puppet round (#1564) [02:04:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:04:13] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [02:28:35] [02mediawiki] 07paladox closed pull request 03#741: Bump extensions/Cargo from `56f6e35` to `def3670` - 13https://git.io/JLabt [02:28:36] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLo7x [02:28:38] [02miraheze/mediawiki] 07paladox 0318a7bb3 - Merge pull request #741 from miraheze/dependabot/submodules/REL1_35/extensions/Cargo-def3670 Bump extensions/Cargo from `56f6e35` to `def3670` [02:28:39] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/Cargo-def3670 - 13https://git.io/vbL5b [02:28:41] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/Cargo-def3670 [02:30:13] [02mediawiki] 07paladox closed pull request 03#731: Bump extensions/FlaggedRevs from `ad63882` to `fa8f929` - 13https://git.io/JLaF9 [02:30:14] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLo5Y [02:30:16] [02miraheze/mediawiki] 07paladox 0304b292c - Merge pull request #731 from miraheze/dependabot/submodules/REL1_35/extensions/FlaggedRevs-fa8f929 Bump extensions/FlaggedRevs from `ad63882` to `fa8f929` [02:30:17] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/FlaggedRevs-fa8f929 - 13https://git.io/vbL5b [02:30:19] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/FlaggedRevs-fa8f929 [02:30:37] [02mediawiki] 07paladox closed pull request 03#732: Bump extensions/ArticleFeedbackv5 from `722e655` to `daafa9e` - 13https://git.io/JLaFH [02:30:38] [02mediawiki] 07dependabot[bot] commented on pull request 03#732: Bump extensions/ArticleFeedbackv5 from `722e655` to `daafa9e` - 13https://git.io/JLo5Z [02:30:40] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/ArticleFeedbackv5-daafa9e - 13https://git.io/vbL5b [02:30:41] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/ArticleFeedbackv5-daafa9e [02:31:02] [02mediawiki] 07paladox closed pull request 03#734: Bump extensions/UniversalLanguageSelector from `555664d` to `c9902ca` - 13https://git.io/JLaFN [02:31:04] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLo5l [02:31:05] [02miraheze/mediawiki] 07paladox 03de637f9 - Merge pull request #734 from miraheze/dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-c9902ca Bump extensions/UniversalLanguageSelector from `555664d` to `c9902ca` [02:31:07] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-c9902ca [02:31:08] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-c9902ca - 13https://git.io/vbL5b [02:31:45] [02mediawiki] 07paladox closed pull request 03#750: Bump skins/Refreshed from `dfc6e59` to `1634a33` - 13https://git.io/JLabu [02:31:46] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLo5u [02:31:48] [02miraheze/mediawiki] 07paladox 039578ff6 - Merge pull request #750 from miraheze/dependabot/submodules/REL1_35/skins/Refreshed-1634a33 Bump skins/Refreshed from `dfc6e59` to `1634a33` [02:31:49] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/skins/Refreshed-1634a33 - 13https://git.io/vbL5b [02:31:51] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/skins/Refreshed-1634a33 [02:32:27] [02mediawiki] 07paladox closed pull request 03#749: Bump skins/Femiwiki from `218e021` to `d330189` - 13https://git.io/JLab0 [02:32:29] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLo5V [02:32:30] [02miraheze/mediawiki] 07paladox 032268550 - Merge pull request #749 from miraheze/dependabot/submodules/REL1_35/skins/Femiwiki-d330189 Bump skins/Femiwiki from `218e021` to `d330189` [02:32:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Femiwiki-d330189 - 13https://git.io/vbL5b [02:32:38] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Femiwiki-d330189 [02:32:59] [02miraheze/ssl] 07paladox deleted branch 03JohnFLewis-patch-1 [02:33:00] [02ssl] 07paladox deleted branch 03JohnFLewis-patch-1 - 13https://git.io/vxP9L [02:37:04] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLodk [02:37:06] [02miraheze/ManageWiki] 07JohnFLewis 0320df8cb - Bring group array up one [02:38:33] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLodC [02:38:35] [02miraheze/mediawiki] 07paladox 03384511c - Update ManageWiki [02:41:37] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:41:49] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:42:46] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:04:48] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [03:05:45] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:05:52] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:34:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLKkB [04:34:16] [02miraheze/puppet] 07paladox 038b3c257 - base::puppet: Use nice for puppet round part 2 [05:56:02] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Thu 07 Jan 2021 05:52:38 GMT +0000). [05:58:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLKWU [05:58:22] [02miraheze/ssl] 07MirahezeSSLBot 03760620b - Bot: Update SSL cert for history.estill.org [06:21:21] PROBLEM - wiki.arnab.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.arnab.at' expires in 15 day(s) (Thu 07 Jan 2021 06:12:33 GMT +0000). [06:22:54] [02ssl] 07Universal-Omega opened pull request 03#373: Remove redirect for deleted wiki - 13https://git.io/JLKBk [06:24:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLKB3 [06:24:02] [02miraheze/ssl] 07MirahezeSSLBot 0352cd18b - Bot: Update SSL cert for wiki.arnab.at [06:24:57] [02ssl] 07Reception123 closed pull request 03#373: Remove redirect for deleted wiki - 13https://git.io/JLKBk [06:24:58] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLKBB [06:25:00] [02miraheze/ssl] 07Universal-Omega 032efe642 - Remove redirect for deleted wiki (#373) [06:27:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://git.io/JLKBd [06:27:59] [02miraheze/mw-config] 07Universal-Omega 031eb713c - Remove config for deleted wiki [06:28:00] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [06:28:02] [02mw-config] 07Universal-Omega opened pull request 03#3581: Remove config for deleted wiki - 13https://git.io/JLKBF [06:29:46] miraheze/mw-config/Universal-Omega-patch-7/1eb713c - Universal Omega The build passed. https://travis-ci.org/miraheze/mw-config/builds/750976536 [06:30:56] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Mon 22 Mar 2021 04:58:14 GMT +0000. [07:02:03] RECOVERY - wiki.arnab.at - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.arnab.at' will expire on Mon 22 Mar 2021 05:23:55 GMT +0000. [07:13:55] [02mw-config] 07Pix1234 closed pull request 03#3581: Remove config for deleted wiki - 13https://git.io/JLKBF [07:13:57] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLKV4 [07:13:58] [02miraheze/mw-config] 07Universal-Omega 03c230182 - Remove config for deleted wiki (#3581) [07:14:00] [02miraheze/mw-config] 07Pix1234 deleted branch 03Universal-Omega-patch-7 [07:14:01] [02mw-config] 07Pix1234 deleted branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [13:02:06] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL6KC [13:02:08] [02miraheze/CreateWiki] 07translatewiki 03c18a169 - Localisation updates from https://translatewiki.net. [13:02:09] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JL6KW [13:02:11] [ Main page - translatewiki.net ] - translatewiki.net [13:02:11] [02miraheze/MirahezeMagic] 07translatewiki 03a8d448e - Localisation updates from https://translatewiki.net. [13:02:12] [ Main page - translatewiki.net ] - translatewiki.net [13:02:12] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JL6Kl [13:02:14] [02miraheze/ManageWiki] 07translatewiki 035b82f2e - Localisation updates from https://translatewiki.net. [13:02:15] [ Main page - translatewiki.net ] - translatewiki.net [13:38:49] [02mw-config] 07JohnFLewis closed pull request 03#3574: Show Different Sitenotice for Inactive Private Wikis (Fixes T6604) - 13https://git.io/JL3ym [13:38:50] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL6Ss [13:38:52] [02miraheze/mw-config] 07R4356th 03c4f67fe - Show different message for inactive private wikis (#3574) [13:56:49] It's a long shot considering the time I know, but are there any users here in the Americas/Asia who have experienced slowness and might potentially be accessing us via cp9.miraheze.org? (https://phabricator.miraheze.org/T6431) [13:56:51] [ ⚓ T6431 cp9 extremely latent for a number of users ] - phabricator.miraheze.org [14:00:50] JohnLewis: https://tools.keycdn.com/ping shows a bit [14:01:15] But then it's as bad for Asia via Asia cp [14:01:23] [ Ping Test - Simultaneously Ping From 10 Global Locations | KeyCDN Tools ] - tools.keycdn.com [14:01:43] I need MTR data though, ideally for different users in different physical locations :) [14:03:06] RhinosF1; though, that site does provide some relatively useful information so thanks! [14:05:28] JohnLewis: Does this help- ping meta.miraheze.org Pinging meta.miraheze.org [128.199.139.216] with 32 bytes of data: Reply from 128.199.139.216: bytes=32 time=144ms TTL=53 Reply from 128.199.139.216: bytes=32 time=155ms TTL=53 Reply from 128.199.139.216: bytes=32 time=168ms TTL=53 Reply from 128.199.139.216: bytes=32 time=166ms TTL=53 Ping statistics for 128.199.139.216: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), [14:05:29] Approximate round trip times in milli-seconds: Minimum = 144ms, Maximum = 168ms, Average = 158ms ? I am in Asia but connected via Cp3. [14:07:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JL6dM [14:07:44] [02miraheze/puppet] 07paladox 03c99e0ce - mediawiki: Change git clone depth for cloning core to 1 Should speed things up, and also use less disk space [14:07:45] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [14:07:47] [02puppet] 07paladox opened pull request 03#1565: mediawiki: Change git clone depth for cloning core to 1 - 13https://git.io/JL6dS [14:07:56] [02puppet] 07paladox closed pull request 03#1565: mediawiki: Change git clone depth for cloning core to 1 - 13https://git.io/JL6dS [14:07:57] R4356th: I need an mtr, ideally 50-100 cycles directly to cp9.miraheze.org please :) [14:07:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JL6dH [14:07:59] [02miraheze/puppet] 07paladox 031ec4be8 - mediawiki: Change git clone depth for cloning core to 1 (#1565) Should speed things up, and also use less disk space [14:08:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [14:08:02] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [14:08:58] !log depool/repool mw4 (recloning mediawiki) [14:09:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:10:08] JohnLewis: will traceroute help? I use Windows 10 and do not have mtr downloaded. [14:10:57] JohnLewis: I can try from the bots server [14:11:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [14:11:59] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [14:12:05] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [14:12:34] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.026 second response time [14:12:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [14:16:18] R4356th: traceroute also works [14:16:31] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.016 second response time [14:19:28] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [14:19:32] Okay. [14:20:04] JohnLewis: see email [14:20:17] will do [14:20:21] thanks [14:20:31] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.089 second response time [14:20:53] ping me if you need any more [14:20:59] or comment on the phab task [14:22:31] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 20055 bytes in 0.152 second response time [14:23:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [14:23:39] Tracing route to cp9.miraheze.org [51.222.27.129] over a maximum of 30 hops: 1 105 ms 199 ms 2 ms router.asus.com [REDACTED] 2 6 ms 8 ms 4 ms 10.64.0.1 3 797 ms 609 ms 4 ms 172.31.31.101 4 * * * Request timed out. 5 648 ms 4 ms 4 ms 103.12.172.217 6 6 ms 378 ms 202 ms 103.12.172.201 7 164 ms 5 ms 6 ms 103-16-155-89-noc.bsccl.com [14:23:39] [103.16.155.89] 8 12 ms 119 ms 11 ms 103-16-152-34-noc.bsccl.com [103.16.152.34] 9 219 ms 200 ms 203 ms xe2-3-3.marsiglia3.mar.seabone.net [213.144.176.194] 10 204 ms 202 ms 202 ms ae21.parigi52.par.seabone.net [195.22.210.107] 11 * * * Request timed out. 12 590 ms 408 ms 406 ms par-gsw-sbb1-nc5.fr.eu [54.36.50.228] 13 * * * Request timed out. 14 427 ms 406 ms [14:23:40] 407 ms par-gsw-sbb1-nc5.fr.eu [54.36.50.228] 15 409 ms 407 ms 406 ms be103.rbx-g4-nc5.fr.eu [54.36.50.229] 16 * * * Request timed out. 17 404 ms 407 ms 407 ms be100-1298.nwk-5-a9.nj.us [192.99.146.133] 18 289 ms 324 ms 407 ms be102.bhs-g2-nc5.qc.ca [192.99.146.138] 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * [14:23:40] Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * * * Request timed out. 25 * * * Request timed out. 26 384 ms 407 ms 406 ms cp9.miraheze.org [51.222.27.129] [14:23:58] The trace, JohnLewis. [14:24:37] R4356th: Thanks :) [14:28:01] !log depool/repool mw5 (recloning mediawiki) [14:28:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:24] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.011 second response time [14:29:04] !log recloning mediawiki on jobrunner[12] [14:29:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:29:35] paladox: why recloning [14:30:49] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.007 second response time [14:33:04] Zppix because we've changed the depth of the git clone [14:33:44] Oh [14:34:23] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2500 bytes in 0.034 second response time [14:34:48] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.016 second response time [14:36:26] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 20069 bytes in 0.821 second response time [14:37:55] @RhinosF1 https://phabricator.miraheze.org/T4352, if that is a MH-Bots thing, can you create a task on the other Phab and then we can close that one? [14:37:56] [ ⚓ T4352 Create the Miraheze-feed-es channel ] - phabricator.miraheze.org [14:38:21] I can [14:38:49] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.102 second response time [14:40:49] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 20055 bytes in 0.141 second response time [14:42:23] Done [14:46:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [14:47:28] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [14:47:58] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [14:48:20] !log depool/repool mw6 (recloning mediawiki) [14:48:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:50:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [14:51:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [14:51:33] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.005 second response time [14:51:38] PROBLEM - jobrunner2 MediaWiki Rendering on jobrunner2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.870 second response time [14:51:58] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw6 [14:55:37] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.017 second response time [14:55:38] RECOVERY - jobrunner2 MediaWiki Rendering on jobrunner2 is OK: HTTP OK: HTTP/1.1 200 OK - 1265 bytes in 0.532 second response time [15:04:48] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 8.55, 7.12, 4.87 [15:06:47] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 5.43, 6.40, 4.88 [15:07:23] !log depool/repool mw7 (recloning mediawiki) [15:07:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:28] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [15:08:25] [02puppet] 07Universal-Omega opened pull request 03#1566: add minotar.net to CSP whitelist (T6588) - 13https://git.io/JLikI [15:11:28] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw7 [15:12:03] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.006 second response time [15:12:13] [02puppet] 07paladox closed pull request 03#1566: add minotar.net to CSP whitelist (T6588) - 13https://git.io/JLikI [15:12:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLikA [15:12:16] [02miraheze/puppet] 07Universal-Omega 03c6e6d60 - add minotar.net (#1566) [15:14:03] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.007 second response time [15:23:58] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [15:24:05] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [15:24:50] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [15:25:28] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [15:31:04] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [15:53:24] https://phabricator.miraheze.org/T6457 - Deal with travis-ci.org shutdown, authored by revi, assigned to Universal_Omega, Priority: High, Status: Open [15:56:06] wut [15:56:17] It's a weekly thing [15:56:41] Reminders [15:56:45] .uptime [15:56:45] I've been sitting here for 14 days, 0:03:26 and I keep going! [15:57:55] JohnLewis: what you find about cp9 [15:59:02] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [16:00:15] RhinosF1: there's quite a bit of loss on some links, so filed it with OVH [16:00:19] but user -> cp9 is fine [16:00:29] cp9 -> cloud* isn't [16:00:44] JohnLewis: I see [16:00:54] Is cp9 OVH owned? [16:01:49] I though the high prio task thing had a message that announced what it was doing [16:02:00] No [16:02:05] It used too [16:02:08] But I'll fix that after Xmas [16:02:19] It's a very old regression then [16:02:41] But I plan to remodel it in the Xmas -> new year break [16:05:16] RhinosF1: yes [16:06:08] JohnLewis: must be their internal network [16:06:16] Zppix: it's a 7 month old regression [16:06:24] Yeah [16:06:31] File a task and if we don't have one create a regression project [16:07:40] Deal with jt later [16:12:12] For future note, https://github.com/MirahezeBots/MirahezeBots/commit/09a56b311e2983ebd1fdc37692d7cae99eed042b [16:12:13] [ Rebuild mhphab script (#135) · MirahezeBots/MirahezeBots@09a56b3 · GitHub ] - github.com [16:16:37] Made a task [16:16:47] Will deal with when I redo the code [16:16:54] A week more won't harm [16:17:07] We can do a backport then to 9.1.x if needed [16:39:00] Yay! IRCCloud outage lasting about 16 hours has been resolved. ;) [16:39:45] Only affected an apparently small "subset of users," though. Unfortunately, I seem to have been in that subset. :( [16:57:39] * hispano76 greetings [17:01:45] Alert to Miraheze Staff: It looks like the MirahezeRC bot has stopped! Recent Changes are no longer available from IRC. [17:02:02] Eh [17:02:08] Log it too [17:02:10] * RhinosF1 here [17:02:44] JohnLewis, paladox: can you check mon1? [17:03:07] no, not unless its an emergency. I'm busy with the task i've been assinged. [17:03:45] paladox: log bot is dead so you need to restart it anyway to log anything you change [17:03:54] Oh there it is [17:04:01] Welcome back MirahezeLogbot [17:05:31] !log are you alive? [17:05:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:06:01] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:06:09] Undone ^ [17:06:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:21] PROBLEM - Host ns1 is DOWN: CRITICAL - Time to live exceeded (192.184.82.120) [17:06:25] Oh f you icinga-miraheze [17:06:38] Why's ns1 upset [17:08:02] PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [17:08:17] JohnLewis: ^ [17:08:23] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 102.55 ms [17:08:23] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 50%, RTA = 98.34 ms [17:08:29] Jbr is prob my import causing puppet to freak out [17:08:31] It came back [17:08:57] Zppix: seems wider [17:09:01] Just run puppet again [17:09:12] Ns1 and mon1 just seem to be crying too [17:10:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:10:08] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 99.38 ms [17:12:38] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:13:02] Also paladox that git fix can't have worked [17:13:14] what do you mean [17:13:44] Notice: /Stage[main]/Mediawiki/Git::Clone[MediaWiki core]/Exec[git_pull_MediaWiki core]/returns: fatal: unable to access 'https://github.com/wikimedia/mediawiki-extensions-3D.git/': Failed to connect to github.com port 443: Connection timed out [17:13:44] [ GitHub - wikimedia/mediawiki-extensions-3D: Github mirror of MediaWiki extension 3D - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [17:13:48] network issue [17:14:25] network issue would explain mon1 in general [17:19:12] JohnLewis: can we get onto someone to get it fixed? [17:20:00] Nothing stands out on https://mobile.twitter.com/ovh_status [17:20:20] it's working fine for me currently [17:21:21] Try kicking puppet then [17:23:39] puppet is fine [17:24:39] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:26:32] There good [17:27:26] [02mw-config] 07Universal-Omega commented on pull request 03#3554: Convert from travis to GitHub actions - 13https://git.io/JLigh [17:27:34] [02mw-config] 07Universal-Omega closed pull request 03#3554: Convert from travis to GitHub actions - 13https://git.io/JIwUl [17:27:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+2/-2/±0] 13https://git.io/JLigj [17:27:37] [02miraheze/mw-config] 07MacFan4000 03777691e - Convert from travis to GitHub actions (#3554) * Delete .travis.yml * Rename .travis.phplint.sh to phplint.sh * Create workflow.yml * Update workflow.yml [17:34:01] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:36:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-test [+0/-0/±1] 13https://git.io/JLi2p [17:36:14] [02miraheze/mw-config] 07Universal-Omega 035332e1d - test (DO NOT MERGE) [17:36:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-test - 13https://git.io/vbvb3 [17:36:17] [02mw-config] 07Universal-Omega opened pull request 03#3582: test (DO NOT MERGE) - 13https://git.io/JLi2j [17:36:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+6/-16/±5] 13https://git.io/JLiaf [17:36:55] [02miraheze/puppet] 07paladox 037b9045d - Move profile back to role Also add some documentation Bug: T6290 [17:37:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-test [+0/-0/±1] 13https://git.io/JLiaI [17:37:48] [02miraheze/mw-config] 07Universal-Omega 0343c9828 - test [17:37:50] [02mw-config] 07Universal-Omega synchronize pull request 03#3582: test (DO NOT MERGE) - 13https://git.io/JLi2j [17:46:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://git.io/JLiVZ [17:46:27] [02miraheze/mw-config] 07Universal-Omega 03537b1b0 - Add workflow name [17:46:28] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [17:46:31] [02mw-config] 07Universal-Omega opened pull request 03#3583: Add workflow name - 13https://git.io/JLiVn [17:49:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JLiV9 [17:49:58] [02miraheze/mw-config] 07paladox 034e1020a - Use rectalogic/notify-irc to send irc events [17:50:00] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [17:50:01] [02mw-config] 07paladox opened pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLiVH [17:51:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JLiVN [17:51:04] [02miraheze/mw-config] 07paladox 032079f1f - Update workflow.yml [17:51:06] [02mw-config] 07paladox synchronize pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLiVH [17:51:49] paladox pushed refs/heads/paladox-patch-1 https://github.com/miraheze/mw-config/compare/4e1020a447ef...2079f1f8a47b [17:51:49] Update workflow.yml [17:51:49] [17:51:49] Hello miraheze-github-! If you have any questions, feel free to ask and someone should answer soon. [17:51:50] [ Comparing 4e1020a447ef...2079f1f8a47b · miraheze/mw-config · GitHub ] - github.com [17:51:56] paladox opened PR [17:51:56] [17:52:03] paladox opened PR [17:52:03] [17:52:09] uh [17:52:11] paladox pushed refs/heads/paladox-patch-1 https://github.com/miraheze/mw-config/compare/4e1020a447ef...2079f1f8a47b [17:52:11] Update workflow.yml [17:52:11] [17:52:12] [ Comparing 4e1020a447ef...2079f1f8a47b · miraheze/mw-config · GitHub ] - github.com [17:53:13] Bot fight? [17:55:46] @QuIRC Seems to be in error. It should be whitelisting miraheze-github-, imho, but could be a larger code issue, I suspect [17:57:23] https://phabricator.miraheze.org/R9:2079f1f8a47b370d77ee59a7285f56cabbc3dd2a ^^ [17:57:24] [ R9:2079f1f8a47b ] - phabricator.miraheze.org [17:57:38] Not in error, dmehus. [17:58:37] @R4356th Well, still, the MirahezeBot team needs to whitelist that username from the open proxy/VPN detection service, so this doesn't happen again. [17:58:54] hmm not the message i really want, just want it to send if the test fails or pass [17:59:00] Universal_Omega maybe you want to look? [17:59:10] paladox: looking now [17:59:10] Universal_Omega: 2020-12-21 - 04:24:43UTC tell Universal_Omega DM me when you're around [17:59:18] thanks! [18:00:06] @Universal_Omega ^ that tell was to remind me to tell you @Naleksuh wanted to ask you something [18:00:23] [02mw-config] 07MacFan4000 commented on pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLirI [18:00:42] RhinosF1: how do we whitelist bots on MirahezeBot i forget [18:04:08] Zppix: what you mean whitelist? [18:04:17] Oh that [18:04:22] It's in the code [18:04:25] There's a list [18:05:16] Add whatever needs done [18:05:49] Look in Sygnal [18:05:59] It's just an if statement [18:06:16] can you add whatever needs added [18:06:27] Mobile and getting Xmas order [18:08:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JLiof [18:08:33] [02miraheze/mw-config] 07Universal-Omega 036f6086d - Update workflow.yml [18:08:34] [02mw-config] 07Universal-Omega synchronize pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLiVH [18:08:45] Zppix: ask MacFan [18:09:19] Universal-Omega the build has passed [18:09:19] [18:09:26] Universal-Omega the build has passed [18:09:26] [18:09:33] Universal-Omega the build has passed [18:09:33] [18:09:34] Ugh [18:09:40] Universal-Omega the build has passed [18:09:40] [18:09:48] .deop @MirahezeBot [18:09:48] Attempting to OP... [18:09:49] paladox: Worked, but does for every check [18:09:53] Will someone stop until it's been whitelisted by someone with access [18:09:59] It's designed to reop [18:10:03] and includes empty space [18:10:03] ergh [18:10:05] what's with that extra empty line [18:10:07] I had it part for neo [18:10:09] Now* [18:10:55] Okay [18:10:57] .unban 52.247.247.80 [18:10:59] Shout a MacFan [18:11:06] He knows the code too [18:11:31] Can someone unban the IPs in #miraheze-bot-alerts? I'm not familiar with the correct syntax. [18:11:55] ```Alert: Sygnal triggered in #miraheze by miraheze-github- (52.247.247.80). Open proxy or VPN detected.. Actions taken: Automatic kickban. Ping JohnLewis SPF|Cloud PuppyKun paladox Reception123 Zppix RhinosF1 dmehus [18:11:55] 10:09 Alert: Sygnal triggered in #miraheze by miraheze-github- (52.151.3.228). Open proxy or VPN detected.. Actions taken: Automatic kickban. Ping JohnLewis SPF|Cloud PuppyKun paladox Reception123 Zppix RhinosF1 dmehus [18:11:55] 10:09 Alert: Sygnal triggered in #miraheze by miraheze-github- (52.151.46.18). Open proxy or VPN detected.. Actions taken: Automatic kickban. Ping JohnLewis SPF|Cloud PuppyKun paladox Reception123 Zppix RhinosF1 dmehus [18:11:55] 10:09 Alert: Sygnal triggered in #miraheze by miraheze-github- (52.151.52.214). Open proxy or VPN detected.. Actions taken: Automatic kickban. Ping JohnLewis SPF|Cloud PuppyKun paladox Reception123 Zppix RhinosF1 dmehus``` [18:12:06] Why [18:12:07] Did [18:12:10] You do that [18:12:23] 🤦🏻‍♂️ [18:12:40] @Zppix because until it's fixed, MirahezeBot is going to keep banning the open proxy IPs used by miraheze-github- [18:12:41] dmehus: you can /mode #miraheze +b to get the ban list [18:12:52] @Sario ah, thanks I always forget that [18:12:53] You legit just pinged ppl 4 times dmehus [18:13:09] thanks @Voidwalker :) [18:14:04] Also, the real issue is miraheze-githib needs to be whitelisted so it's exempt from the open proxy rule [18:14:11] @Zppix Oops, sorry. I don't think IRC pings are that problematic, to be clear, as one isn't emailed every time they're pinged on IRC [18:14:20] @Sario yes, which @Zppix could do :P [18:14:23] Sario: well aware [18:14:34] Just needs someone to do it who isn't picking turkey up [18:15:22] dmehus i'm notified each time someone pings [18:15:25] We are aware of the issue and are fixing it. [18:16:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JLiKf [18:16:44] [02miraheze/puppet] 07paladox 030fc2f12 - Remove nagiosre user as not needed [18:16:45] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [18:16:47] [02puppet] 07paladox opened pull request 03#1567: Remove nagiosre user as not needed - 13https://git.io/JLiKJ [18:18:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JLiKZ [18:18:40] [02miraheze/puppet] 07paladox 0324e23c2 - Update init.pp [18:18:41] [02puppet] 07paladox synchronize pull request 03#1567: Remove nagiosre user as not needed - 13https://git.io/JLiKJ [18:18:52] [02puppet] 07paladox closed pull request 03#1539: puppet: Tweak settings - 13https://git.io/JIOVA [18:18:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [18:18:55] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [18:19:38] @paladox, okay, fair enough, the trickiness with IRC is that because you don't need an @ to ping someone, it's easily to accidentally mention someone when mentioning them by name [18:19:51] yup [19:35:51] @dmehus since it's been......13 days now, consider replying (yes, I know I'm a bit annoying, but it's been literally 2 weeks. How long can it take to write a quick message?) [20:06:16] @blackwidowmovie0 ✅ {{Done}} [20:07:25] thank you [20:18:58] dmehus: check the cabal discord [20:22:18] @dmehus looking [20:53:20] @RhinosF1 okay [20:54:07] @dmehus submitted a reply, as I seem to have overlooked a flaw in the request [21:13:18] @dmehus See your Discord for a moment [21:21:46] @blackwidowmovie0 I don't know what you mean by "flaw in the request," but in any case, seems dealt with. Also, please do not ping me on IRC to check my DMs on Discord. I will respond to Discord DMs according to my internal prescribed methodology of priority. Thanks [21:23:18] dmehus, what's your "internal prescribed methodology of priority" [21:23:59] @blackwidowmovie0 It varies, is non-codified, and proprietary in the sense it is unique to my needs [21:24:46] @dmehus - Probably not important, just wanted to ask which extension was which [21:29:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://git.io/JLiAj [21:29:18] [02miraheze/mw-config] 07Universal-Omega 03eef7429 - Add IRC notifications and workflow name [21:29:20] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://git.io/vbvb3 [21:29:21] [02mw-config] 07Universal-Omega opened pull request 03#3585: Add IRC notifications and workflow name - 13https://git.io/JLixe [21:30:21] [02mw-config] 07Universal-Omega commented on pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLixt [21:30:23] [02mw-config] 07Universal-Omega closed pull request 03#3584: Use rectalogic/notify-irc to send irc events - 13https://git.io/JLiVH [21:30:24] miraheze/mw-config - Universal-Omega The build passed. [21:30:24] [21:30:24] miraheze/mw-config - Universal-Omega The build passed. [21:30:24] [21:30:24] Hello miraheze-github_! If you have any questions, feel free to ask and someone should answer soon. [21:30:24] Hello miraheze-github! If you have any questions, feel free to ask and someone should answer soon. [21:30:28] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:30:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-1 [21:30:32] miraheze/mw-config - Universal-Omega The build passed. [21:30:32] [21:30:53] [02mw-config] 07Universal-Omega commented on pull request 03#3583: Add workflow name - 13https://git.io/JLixY [21:30:54] [02mw-config] 07Universal-Omega closed pull request 03#3583: Add workflow name - 13https://git.io/JLiVn [21:30:57] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [21:30:58] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [21:32:13] Aware [21:32:23] Not sure why MacFan's fix didn't work [21:32:53] Did he deploy it? [21:33:17] Yes [21:35:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLixD [21:35:18] [02miraheze/mediawiki] 07paladox 03c766d97 - Update Comments to a638ae8d70b9dd71ee52a9eab58de47a1ee39a82 [21:37:16] RhinosF1: Maybe because it is adding the _ to it, so the username is not always the same? [21:37:38] Universal_Omega: the username needs to be the same [21:37:57] We can only current exempt single nicks or single IPs [21:38:24] If it auth'd to NickServ via SASL then we could do account based [21:39:00] When the bot sends the messages to close together, it adds _ to the next one, so maybe white list miraheze-github and miraheze-github_? [21:43:23] [02mw-config] 07MacFan4000 commented on pull request 03#3585: Add IRC notifications and workflow name - 13https://git.io/JLipM [21:50:54] Universal_Omega: tell MacFan [21:52:16] RhinosF1: Already did [21:53:19] why is there an autoban? [21:53:44] VPN [21:54:19] ah, yes [21:54:31] I sometimes join with a VPN, but not often [22:07:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+1/-1/±0] 13https://git.io/JLPe1 [22:07:14] [02miraheze/mw-config] 07Universal-Omega 03cd6b95b - Rename workflow.yml to continuousIntegration.yml [22:07:15] [02mw-config] 07Universal-Omega synchronize pull request 03#3585: Add IRC notifications and workflow name - 13https://git.io/JLixe [22:08:32] miraheze/mw-config - Universal-Omega The build passed. [22:08:32] [22:08:33] miraheze/mw-config - Universal-Omega The build passed. [22:08:33] [22:12:25] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLPvC [22:12:26] [02miraheze/mediawiki] 07paladox 03043a753 - Update Widgets [22:13:50] .op [22:13:50] Attempting to OP... [22:14:16] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLPvo [22:14:18] [02miraheze/mediawiki] 07paladox 034d5785f - Update PollNy [22:14:36] RhinosF1 Universal_Omega ^ [22:14:44] still broken [22:14:58] We know. [22:15:09] @MacFan4000 and @RhinosF1 ^ We need to look into modifying the whitelist for miraheze-github_ to include anything following "github", I think [22:15:20] dmehus: macfan [22:15:45] I'm dealing with important stuff [22:15:48] @RhinosF1 Why can't I ping you too? Aren't you back from picking up xmas / turkey order? [22:15:58] Oh okay, fair enough. [22:16:02] sorry [22:16:04] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLPvH [22:16:06] [02miraheze/mediawiki] 07paladox 03a823ddb - Update GlobalUsage [22:16:17] .tell MacFan4000 ^ We need to look into modifying the whitelist for miraheze-github_ to include anything following "github", I think [22:16:18] dmehus: I'll pass that on when MacFan4000 is around. [22:16:18] dmehus: see -staff [22:16:29] @RhinosF1 okey dokey will do [22:18:05] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLPvx [22:18:07] [02miraheze/mediawiki] 07paladox 037b60542 - Update RandomGameUnit [22:19:09] .deop [22:19:43] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JLPfI [22:19:44] [02miraheze/mediawiki] 07paladox 038cdf153 - Update QuizGame [22:22:18] @RhinosF1, see #miraheze-missionimpossiblewiki as well [22:22:44] blackwidowmovie0: busy [22:24:15] ok [22:25:01] !log rm -rf /srv/mediawiki/w/extensions/Widgets/compiled_templates/** - mw*/jobrunner*/test2 [22:25:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:50] does anyone have a sec to help me with something CSS-related? [22:27:27] no [22:27:34] not sysadmins at least [22:27:43] ah, ok [22:27:52] what are the sysadmins doing? [22:28:19] Working [22:29:10] how very descriptive of you [22:31:01] !log rebuild lc on mw* and jobrunner* [22:31:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:03] blackwidowmovie0: important stuff [22:34:27] again, verrry descriptive [22:34:29] but ok [22:35:08] We cannot disclose what we're working on as its security related. [22:35:19] @blackwidowmovie0 Observation noted. Let's let system administrators do their important work, as they are quite busy. Thanks. [22:35:37] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 21.24, 18.08, 14.20 [22:35:52] .op [22:36:11] ok [22:36:16] thanks :) [22:36:20] np [22:37:38] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.57, 15.99, 13.91 [22:39:48] Hello Guest48! If you have any questions, feel free to ask and someone should answer soon. [22:40:34] hi Guest48 [22:46:35] Can you help me make a working wiget for discord enbed? [22:47:20] Yes, I can help [22:47:26] which wiki? [22:47:29] I need this https://pastebin.com/Tfv5ssZa embedded into a wiget page [22:47:30] [