[00:02:40] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 36.08, 25.18, 18.16 [00:02:45] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.64, 17.24, 11.84 [00:04:01] PROBLEM - wiki.senety.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.senety.com could not be found [00:04:02] PROBLEM - wiki.cyberepo.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cyberepo.org could not be found [00:04:02] PROBLEM - wiki.inebriation-confederation.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.inebriation-confederation.com could not be found [00:04:06] PROBLEM - marvelcinematicuniverse.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marvelcinematicuniverse.miraheze.org could not be found [00:04:07] PROBLEM - theatlas.pw - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for theatlas.pw could not be found [00:04:07] PROBLEM - wiki.campaign-labour.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.campaign-labour.org could not be found [00:04:08] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mobilityengineer.com could not be found [00:04:40] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.87, 22.52, 18.07 [00:04:45] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 8.76, 13.83, 11.26 [00:06:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.88, 19.84, 17.62 [00:10:45] RECOVERY - wiki.senety.com - reverse DNS on sslhost is OK: rDNS OK - wiki.senety.com reverse DNS resolves to cp6.miraheze.org [00:10:49] RECOVERY - wiki.cyberepo.org - reverse DNS on sslhost is OK: rDNS OK - wiki.cyberepo.org reverse DNS resolves to cp7.miraheze.org [00:10:49] RECOVERY - wiki.inebriation-confederation.com - reverse DNS on sslhost is OK: rDNS OK - wiki.inebriation-confederation.com reverse DNS resolves to cp6.miraheze.org [00:10:54] RECOVERY - marvelcinematicuniverse.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - marvelcinematicuniverse.miraheze.org reverse DNS resolves to cp7.miraheze.org [00:10:55] RECOVERY - wiki.campaign-labour.org - reverse DNS on sslhost is OK: rDNS OK - wiki.campaign-labour.org reverse DNS resolves to cp7.miraheze.org [00:10:55] RECOVERY - theatlas.pw - reverse DNS on sslhost is OK: rDNS OK - theatlas.pw reverse DNS resolves to cp6.miraheze.org [00:10:57] RECOVERY - wiki.mobilityengineer.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mobilityengineer.com reverse DNS resolves to cp6.miraheze.org [05:59:09] !log renamed animatedfeetwiki to drawnfeetwiki [05:59:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:59:41] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTiIE [05:59:43] [02miraheze/ssl] 07Reception123 030e07351 - animatedfeetwiki redirects per T4778 [06:00:20] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTiIz [06:00:21] [02miraheze/services] 07MirahezeSSLBot 0371fcba9 - BOT: Updating services config for wikis [06:00:29] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTiI2 [06:00:31] [02miraheze/ssl] 07Reception123 03b004666 - correct [06:02:47] [02mw-config] 07Reception123 commented on pull request 03#3319: Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) - 13https://git.io/JTiIP [06:03:12] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [06:57:28] RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 12 packages available for upgrade (0 critical updates). [11:28:54] Hi, Is there any way for intercriaturaswiki to notify that a file is being used in external wikis? The sections "use" and "global use" are in the target wiki, but not in the repository. For example this file is a transclusion: https://tuscriaturas.miraheze.org/wiki/Archivo:Icon_revisado.png of this file: https://intercriaturas.miraheze.org/wiki/Archivo:Icon_revisado.png [11:28:54] [ Archivo:Icon revisado.png - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [11:28:55] [ Archivo:Icon revisado.png - InterCriaturas ] - intercriaturas.miraheze.org [11:33:25] It comes from instant commons being on [11:33:37] Any public wiki with instant commons on has it turned on [11:43:23] And how i turn it on instant commons on wiki https://intercriaturas.miraheze.org/? [11:43:24] [ Página principal - InterCriaturas ] - intercriaturas.miraheze.org [11:55:41] In ManageWiki/Settings [12:13:21] in which tab of MW settigs [12:13:56] https://intercriaturas.miraheze.org/wiki/Especial:ManageWiki/settings#mw-section-media [12:13:58] [ Administra los ajustes adicionales de este wiki - InterCriaturas ] - intercriaturas.miraheze.org [12:14:42] I have it enable before today, but i don't see the usage on some of the files of Intercriaturas [12:22:04] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 12.90, 7.75, 4.18 [12:26:26] For example, this image only sees the usage on inter: https://intercriaturas.miraheze.org/wiki/Archivo:DarenGa_by_nosoyridley.png and this one sees the global usage: https://tuscriaturas.miraheze.org/wiki/Archivo:DarenGa_by_nosoyridley.png [12:26:26] [ Archivo:DarenGa by nosoyridley.png - InterCriaturas ] - intercriaturas.miraheze.org [12:26:27] [ Archivo:DarenGa by nosoyridley.png - Bestiario del Hypogripho ] - tuscriaturas.miraheze.org [12:35:51] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:01] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:36:42] PROBLEM - mw4 HTTPS on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:05] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [12:37:24] PROBLEM - cp9 Stunnel Http for mw4 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:27] Reception123: ^ [12:37:29] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:37:32] paladox: ^ [12:37:39] JohnLewis: SPF|Cloud [12:37:43] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [12:37:50] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [12:37:56] * Spookreeeno no access - at DHSC test site [12:38:05] I’m mobile see staff channel [12:38:24] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [12:38:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [12:38:31] Ty [12:52:10] RECOVERY - cp9 Stunnel Http for mw4 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 0.326 second response time [12:52:20] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 17313 bytes in 0.228 second response time [12:52:20] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 0.938 second response time [12:52:20] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.58, 19.79, 17.28 [12:52:24] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [12:52:28] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 0.003 second response time [12:52:29] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [12:52:56] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 0.004 second response time [12:53:05] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [12:53:15] RECOVERY - mw4 HTTPS on mw4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.006 second response time [12:53:49] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [12:54:16] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.24, 17.71, 16.86 [12:54:48] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [13:02:27] !log rebooted mw4 [13:02:30] ^ paladox done [13:02:48] Reception123 oh, we resolved the problem. [13:02:49] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [13:03:11] paladox: oh ok, how did you fix it? [13:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:03:23] See -staff Reception123 [13:03:30] ok [13:04:03] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 3.82, 1.45, 0.53 [13:04:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTiHd [13:04:19] [02miraheze/puppet] 07paladox 03912e01f - Update init.pp [13:05:26] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [13:07:26] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:08:52] RECOVERY - gluster2 GlusterFS port 49152 on gluster2 is OK: TCP OK - 0.012 second response time on 51.68.201.37 port 49152 [13:09:29] PROBLEM - gluster1 GlusterFS port 49152 on gluster1 is CRITICAL: connect to address 51.77.107.209 and port 49152: Connection refused [13:13:00] [02puppet] 07paladox commented on commit 03912e01f7b0f1e1a5dd8bc906f608da3142b06692 - 13https://git.io/JTi7i [14:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTix4 [14:05:13] [02miraheze/services] 07MirahezeSSLBot 0393f378e - BOT: Updating services config for wikis [14:31:08] [02mw-config] 07Universal-Omega synchronize pull request 03#3319: Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) - 13https://git.io/JT66n [14:33:29] [02mw-config] 07Universal-Omega synchronize pull request 03#3319: Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) - 13https://git.io/JT66n [14:34:46] hi [14:35:09] @Hispano76 : Hello! [14:38:48] revi: does Tegal do IRC? [14:38:54] https://meta.miraheze.org/wiki/Special:Contributions/63.143.227.40 [14:38:55] [ User contributions for 63.143.227.40 - Miraheze Meta ] - meta.miraheze.org [14:39:02] .whois 63.143.227.40 [14:39:04] Tegel? [14:39:13] revi: the steward [14:39:16] Might been seen at -stew [14:39:27] .ip 63.143.227.40 [14:39:27] [IP/Host Lookup] Hostname: ool-3f8fe328.dyn.optonline.net | Location: United States | Region: New Jersey | City: Paterson | ISP: AS6128 CABLE-NET-1 [14:39:42] revi: see if they know why that IP vandalised their userpage [14:40:59] [02miraheze/SpriteSheet] 07paladox pushed 03194 commits to 03master [+60/-22/±457] 13https://git.io/JTPv6 [14:41:00] [02miraheze/SpriteSheet] 07Alexia 036a81b69 - Create README.md [14:41:02] [02miraheze/SpriteSheet] 07Alexia 03c53e86a - Initial setup. [14:41:03] [02miraheze/SpriteSheet] 07Alexia 035aa080c - More skeleton work. [14:41:05] [02miraheze/SpriteSheet] ... and 191 more commits. [14:41:14] [02SpriteSheet] 07paladox created branch 03master - 13https://git.io/JTPvX [14:44:09] [02miraheze/SpriteSheet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JTPfJ [14:44:10] [02miraheze/SpriteSheet] 07paladox 03a0b9188 - Fix an issue where SpriteSheet::newFromTitle can return false We cannot use internal functions if the function returns false, thus check for this. [14:44:12] [02miraheze/SpriteSheet] 07paladox 032f8a178 - Fix "PHP Notice: Undefined index: success" [14:45:21] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±2] 13https://git.io/JTPfq [14:45:22] [02miraheze/mediawiki] 07paladox 03a20f171 - Update SpriteSheet to fork [15:01:56] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 25.60, 22.53, 19.73 [15:03:53] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.02, 19.47, 18.95 [15:10:49] PROBLEM - ping6 on misc1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 94.99 ms [15:12:47] RECOVERY - ping6 on misc1 is OK: PING OK - Packet loss = 0%, RTA = 13.57 ms [15:13:27] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTPTD [15:13:29] [02miraheze/mediawiki] 07paladox 03ad29f56 - Update SimpleBlogPage to d696443f87cf568960e15013a11dc23ac328d94e [15:24:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTPIK [15:24:45] [02miraheze/mw-config] 07paladox 03608d4ff - Remove CrossReference Extension not supported by 1.35 and also extension is severally unmaintained. [15:24:47] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [15:24:49] [02mw-config] 07paladox opened pull request 03#3320: Remove CrossReference - 13https://git.io/JTPIi [15:25:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTPI1 [15:25:12] [02miraheze/mw-config] 07paladox 032e59a66 - Update LocalExtensions.php [15:25:14] [02mw-config] 07paladox synchronize pull request 03#3320: Remove CrossReference - 13https://git.io/JTPIi [15:25:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JTPIy [15:25:25] [02miraheze/mw-config] 07paladox 03d736d9c - Update extension-list [15:25:27] [02mw-config] 07paladox synchronize pull request 03#3320: Remove CrossReference - 13https://git.io/JTPIi [15:28:28] [02mw-config] 07paladox closed pull request 03#3320: Remove CrossReference - 13https://git.io/JTPIi [15:28:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±3] 13https://git.io/JTPLO [15:28:31] [02miraheze/mw-config] 07paladox 0334c7348 - Remove CrossReference (#3320) * Remove CrossReference Extension not supported by 1.35 and also extension is severally unmaintained. * Update LocalExtensions.php * Update extension-list [15:28:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [15:28:34] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [15:28:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-1/±1] 13https://git.io/JTPL3 [15:28:43] [02miraheze/mediawiki] 07paladox 03f170beb - Remove CrossReference extension [15:28:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JTPLG [15:28:52] [02miraheze/mw-config] 07Reception123 030968718 - 1.35 sitenotice [15:28:53] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:28:55] [02mw-config] 07Reception123 opened pull request 03#3321: 1.35 sitenotice - 13https://git.io/JTPLn [15:35:46] [02mw-config] 07Reception123 closed pull request 03#3321: 1.35 sitenotice - 13https://git.io/JTPLn [15:35:47] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPtm [15:35:49] [02miraheze/mw-config] 07Reception123 0321cba63 - 1.35 sitenotice (#3321) [15:35:50] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [15:35:52] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:38:16] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.51, 21.44, 18.61 [15:40:12] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.76, 20.76, 18.69 [15:43:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTPqs [15:43:36] [02miraheze/mediawiki] 07paladox 03aaea766 - Update TreeAndMenu to 14ff544cf04f34d9d6d20e667acba846a0bed81c [15:44:09] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 30.70, 24.68, 20.58 [15:44:15] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.00, 18.99, 13.83 [15:46:13] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 11.62, 16.21, 13.45 [15:48:02] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.14, 22.19, 20.62 [15:48:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JTPq5 [15:48:55] [02miraheze/mw-config] 07Reception123 03c7de87d - sitenotice for wikis that have CrossReference [15:48:57] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:48:58] [02mw-config] 07Reception123 opened pull request 03#3322: sitenotice for wikis that have CrossReference - 13https://git.io/JTPqd [15:49:58] [02mw-config] 07Reception123 synchronize pull request 03#3322: sitenotice for wikis that have CrossReference - 13https://git.io/JTPqd [15:49:59] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.84, 19.20, 19.73 [15:50:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JTPmf [15:50:01] [02miraheze/mw-config] 07Reception123 033832080 - Update Sitenotice.php [15:50:05] [02mw-config] 07Reception123 closed pull request 03#3322: sitenotice for wikis that have CrossReference - 13https://git.io/JTPqd [15:50:07] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPmT [15:50:08] [02miraheze/mw-config] 07Reception123 03f8386dc - sitenotice for wikis that have CrossReference (#3322) * sitenotice for wikis that have CrossReference * Update Sitenotice.php [15:50:10] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [15:50:11] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:52:55] Reception123 that's going to error for wikis [15:53:34] paladox: why? it worked before [15:53:39] when I did it with Comments [15:54:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPm9 [15:54:08] [02miraheze/mw-config] 07paladox 03f7dd1cf - fix [15:54:20] ohh the double definition, my bad :( [15:55:05] paladox: https://github.com/miraheze/mw-config/commit/60658620d68031a76e085a023386bd03f064cfda#diff-0ebce748027f36364059daa8eb9847c933aa7562798d2624f259264c2d2014fd ohh you fixed it that time too [15:55:06] [ Fix · miraheze/mw-config@6065862 · GitHub ] - github.com [15:55:14] I never noticed so I had no idea it didn't work like that [16:03:11] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPOI [16:03:13] [02miraheze/mw-config] 07Reception123 03267ff93 - bump sitenotice id [16:04:37] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 26.12, 20.40, 18.97 [16:08:29] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 19.61, 20.29, 19.27 [16:31:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.56, 17.75, 17.17 [16:33:43] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.67, 18.02, 17.34 [16:35:27] miraheze/mw-config/paladox-patch-1/d736d9c - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/739326526 [16:43:32] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 17.38, 21.51, 19.62 [16:49:21] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.89, 20.37, 19.77 [17:03:25] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 28.19, 21.57, 19.66 [17:03:37] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 145s [17:04:10] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:04:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:15] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100% [17:05:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.61, 21.51, 19.88 [17:05:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:32] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:38] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [17:05:43] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:05:50] PROBLEM - Host ns1 is DOWN: PING CRITICAL - Packet loss = 100% [17:05:55] Why is ns1 upset paladox Reception123 JohnLewis [17:06:04] huh [17:06:09] looking [17:06:21] does it really have to be upset right before the 1.35 upgrade? [17:06:29] seems up to me [17:06:34] works for me [17:07:01] Refresh icinga then [17:07:27] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 97.21 ms [17:07:28] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 97.27 ms [17:07:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:07:31] I got hund emails [17:07:31] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 11362 MB (88% inode=92%); [17:07:35] Yeah there we go [17:07:39] RECOVERY - ns1 APT on ns1 is OK: APT OK: 6 packages available for upgrade (0 critical updates). [17:07:43] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:43] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.0008253753185 secs [17:07:48] RECOVERY - ns1 Current Load on ns1 is OK: OK - load average: 0.37, 0.36, 0.30 [17:08:11] seems like it had a network issue [17:08:23] Ack [17:08:34] Oct 27 17:03:14 ns1 puppet-agent[7347]: (/Stage[main]/Dns/Git::Clone[dns]/Exec[git_pull_dns]/returns) fatal: unable to access 'https://github.com/miraheze/dns.git/': Could not resolve host: github.com [17:08:35] [ GitHub - miraheze/dns: Production DNS configuration ] - github.com [17:09:20] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.92, 18.75, 19.16 [17:10:18] !log upgrade mariadb-client-10.3 mariadb-client-core-10.3 mariadb-common npm - mail1 [17:10:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:10:42] RECOVERY - mail1 APT on mail1 is OK: APT OK: 7 packages available for upgrade (0 critical updates). [17:13:36] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:50:33] PROBLEM - devwiki.anders.moldin.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'devwiki.anders.moldin.com' expires in 15 day(s) (Thu 12 Nov 2020 17:41:34 GMT +0000). [17:51:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPRL [17:51:44] [02miraheze/ssl] 07MirahezeSSLBot 03dd75d6b - Bot: Update SSL cert for devwiki.anders.moldin.com [18:04:05] RECOVERY - devwiki.anders.moldin.com - LetsEncrypt on sslhost is OK: OK - Certificate 'devwiki.anders.moldin.com' will expire on Mon 25 Jan 2021 16:51:35 GMT +0000. [18:39:18] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JTPzN [18:39:19] [02miraheze/mediawiki] 07paladox 030ecb803 - Add extension Video Bug: T5906 [18:43:54] [02mw-config] 07paladox closed pull request 03#3317: Add Video Extension (T5906) - 13https://git.io/JT6gY [18:43:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±4] 13https://git.io/JTPg3 [18:43:57] [02miraheze/mw-config] 07Universal-Omega 03b8df7af - Add Video Extension (T5906) (#3317) * add video ext (T5906) * add video ext (T5906) * add video ext (T5906) * add video ext (T5906) [18:45:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JTPgc [18:45:44] [02miraheze/mw-config] 07paladox 03640523d - Set $wgHtml5 to true for JavascriptSlideshow [19:09:07] [02mw-config] 07paladox commented on pull request 03#3318: hard-enable HAWelcome ext on csydeswiki (T6272) - 13https://git.io/JTP2y [19:09:09] [02mw-config] 07paladox closed pull request 03#3318: hard-enable HAWelcome ext on csydeswiki (T6272) - 13https://git.io/JT6KG [19:12:00] [02mw-config] 07Universal-Omega commented on pull request 03#3318: hard-enable HAWelcome ext on csydeswiki (T6272) - 13https://git.io/JTP2h [19:13:38] [02mw-config] 07paladox reopened pull request 03#3318: hard-enable HAWelcome ext on csydeswiki (T6272) - 13https://git.io/JT6KG [19:13:43] [02mw-config] 07paladox closed pull request 03#3318: hard-enable HAWelcome ext on csydeswiki (T6272) - 13https://git.io/JT6KG [19:13:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPaI [19:13:46] [02miraheze/mw-config] 07Universal-Omega 032dc40dc - hard-enable HAWelcome ext on csydeswiki (T6272) (#3318) [19:17:04] [02mw-config] 07Universal-Omega opened pull request 03#3323: Bring REL1_35 branch up-to-date with master - 13https://git.io/JTPaC [19:20:22] [02mw-config] 07Universal-Omega closed pull request 03#3323: Bring REL1_35 branch up-to-date with master - 13https://git.io/JTPaC [19:20:25] .op [19:20:26] Attempting to OP... [19:21:34] paladox: ^ [19:21:46] ok [19:21:48] thanks [19:22:11] paladox: I think that broke Code of Conduct link but will fix after [19:22:19] ok [19:26:00] miraheze/mw-config/Reception123-patch-1/3832080 - Reception123 The build has errored. https://travis-ci.org/miraheze/mw-config/builds/739337791 [19:30:00] [02mw-config] 07paladox synchronize pull request 03#3319: Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) - 13https://git.io/JT66n [19:30:15] [02mw-config] 07paladox closed pull request 03#3319: Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) - 13https://git.io/JT66n [19:30:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JTPVs [19:30:18] [02miraheze/mw-config] 07Universal-Omega 03e2fa4b5 - Add $wgWBRepoSettings['federatedPropertiesEnabled'] for paikallishistoriawikiwiki (T6353) (#3319) * add federatedPropertiesEnabled config * add $wmgFederatedPropertiesEnabled * move to ManageWikiSettings * move to ManageWikiSettings * Update LocalSettings.php Co-authored-by: paladox [19:39:56] [02mw-config] 07Universal-Omega opened pull request 03#3324: $wmgFederatedPropertiesEnabled default to false - 13https://git.io/JTPwn [19:41:11] [02mw-config] 07paladox closed pull request 03#3324: $wmgFederatedPropertiesEnabled default to false - 13https://git.io/JTPwn [19:41:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JTPwE [19:41:14] [02miraheze/mw-config] 07Universal-Omega 03158304d - $wmgFederatedPropertiesEnabled default to false (#3324) * $wmgFederatedPropertiesEnabled default to false * $wmgFederatedPropertiesEnabled default to false [19:41:37] PROBLEM - vsfan.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:01:10] [02miraheze/mw-config] 07paladox pushed 0319 commits to 03master [+0/-0/±38] 13https://git.io/JTPo8 [20:01:12] [02miraheze/mw-config] 07paladox 03133d68d - Merge branch 'REL1_35' into master * REL1_35: Set $wgHtml5 to true for JavascriptSlideshow Add Video Extension (T5906) (#3317) Remove CrossReference (#3320) Add SimpleBlogPage ext (#3316) wgCreateWikiSQLfiles: Update sql location for 1.35 Update path of sql files (#3291) Fix syntax Update LocalSettings.php Replace deprecated hook SkinTemplateOutputPageBeforeExec [20:01:12] with SkinAddFooterLinks (#3308) Switch SubpageFun to extension registration (#3307) Switch MassEditRegex to Extension Registration (#3306) WBL_VERSION constant was removed, replace with wgVersion (#3304) Update wgPasswordPolicy to be compatible with 1.35 (#3303) Properly migrate config for 1.35 support part 1 (#3301) [20:01:26] :) [20:01:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPoE [20:01:42] [02miraheze/puppet] 07paladox 03edf0256 - jobrunner1: Upgrade to mediawiki 1.35 [20:02:02] PROBLEM - jobrunner2 Puppet on jobrunner2 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 9 minutes ago with 0 failures [20:02:07] !log stop jobrunner and jobchron on jobrunner[12] [20:02:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:22] PROBLEM - jobrunner1 Puppet on jobrunner1 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 10 minutes ago with 0 failures [20:02:31] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 10 minutes ago with 0 failures [20:02:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPor [20:02:58] [02miraheze/puppet] 07paladox 03b45fa4a - jobrunner2: upgrade to mediawiki 1.35 [20:03:17] PROBLEM - mw7 Puppet on mw7 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 1 minute ago with 0 failures [20:03:22] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 10 minutes ago with 0 failures [20:03:52] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.007 second response time [20:03:58] PROBLEM - mw6 Puppet on mw6 is WARNING: WARNING: Puppet is currently disabled, message: mediawiki upgrade to 1.35 - paladox, last run 2 minutes ago with 0 failures [20:03:59] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:04:22] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 2 backends are down. mw6 mw7 [20:04:27] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw6 mw7 [20:04:28] hmm why has mw gone down? [20:05:00] hmmm [20:05:05] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 2 backends are down. mw6 mw7 [20:05:08] PROBLEM - jobrunner2 JobRunner Service on jobrunner2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:05:24] PROBLEM - jobrunner2 MediaWiki Rendering on jobrunner2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.005 second response time [20:05:26] PROBLEM - jobrunner2 JobChron Service on jobrunner2 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:05:26] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.005 second response time [20:05:32] PROBLEM - mw6 MediaWiki Rendering on mw6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 284 bytes in 0.006 second response time [20:05:37] PROBLEM - jobrunner1 JobChron Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:05:48] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 2 backends are down. mw6 mw7 [20:07:24] guess it's because of jobrunner1 causing issues due to i/o [20:07:25] PROBLEM - jobrunner2 MediaWiki Rendering on jobrunner2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.112 second response time [20:08:12] oh [20:09:27] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.065 second response time [20:10:22] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.39, 7.08, 4.78 [20:12:49] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 8.01, 6.74, 4.79 [20:13:26] RECOVERY - jobrunner2 MediaWiki Rendering on jobrunner2 is OK: HTTP OK: HTTP/1.1 200 OK - 19269 bytes in 0.691 second response time [20:13:27] RECOVERY - jobrunner2 JobChron Service on jobrunner2 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:13:28] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 1265 bytes in 0.341 second response time [20:13:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb [20:13:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [20:14:01] RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:14:21] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:14:45] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 7.08, 7.02, 5.13 [20:15:08] RECOVERY - jobrunner2 JobRunner Service on jobrunner2 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [20:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPK0 [20:15:13] [02miraheze/services] 07MirahezeSSLBot 0309617fa - BOT: Updating services config for wikis [20:15:44] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:16:07] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [20:16:17] !log depool mw4 [20:16:18] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 7.67, 7.61, 5.84 [20:16:42] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 6.29, 6.61, 5.19 [20:16:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPKr [20:16:55] [02miraheze/puppet] 07paladox 03485826d - mw4: upgrade mediawiki to 1.35 [20:18:12] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 9.62, 8.42, 6.35 [20:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:05] PROBLEM - mw4 MediaWiki Rendering on mw4 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.005 second response time [20:22:44] PROBLEM - cp9 Stunnel Http for mw5 on cp9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:22:48] paladox: seeing 502 [20:23:05] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.024 second response time [20:23:16] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:33] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:24:47] RECOVERY - cp9 Stunnel Http for mw5 on cp9 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 5.992 second response time [20:25:23] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 9.324 second response time [20:27:02] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.010 second response time [20:27:48] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15662 bytes in 5.917 second response time [20:27:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:30:35] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:31:15] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php crossreference --disable [20:31:53] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.222.27.129/cpweb, 2607:5300:205:200::2ac4/cpweb [20:32:56] PROBLEM - mw4 MediaWiki Rendering on mw4 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2500 bytes in 0.008 second response time [20:33:54] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 18360 bytes in 0.081 second response time [20:35:34] RECOVERY - mw6 MediaWiki Rendering on mw6 is OK: HTTP OK: HTTP/1.1 200 OK - 18361 bytes in 0.197 second response time [20:35:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:35:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:37:38] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.55, 6.79, 7.46 [20:38:50] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.095 second response time [20:39:14] !log repool mw4 [20:39:18] !log depool mw5 [20:39:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:39:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:41:37] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 1.23, 3.90, 6.13 [20:42:30] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.94, 19.87, 15.35 [20:43:22] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 8.52, 6.07, 3.12 [20:43:34] PROBLEM - mw5 MediaWiki Rendering on mw5 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.010 second response time [20:44:04] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.61, 6.87, 3.63 [20:45:22] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 7.91, 6.42, 3.59 [20:46:04] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.90, 7.26, 4.17 [20:46:27] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.96, 20.81, 16.83 [20:47:22] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 6.41, 6.28, 3.88 [20:47:30] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.011 second response time [20:48:03] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 8.02, 7.44, 4.60 [20:48:23] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.60, 19.26, 16.73 [20:50:04] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 5.27, 6.62, 4.65 [20:51:26] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.017 second response time [20:57:22] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:59:15] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 469 bytes in 0.007 second response time [21:00:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTPX2 [21:00:02] [02miraheze/puppet] 07paladox 03d215ef0 - mw5: upgrade mediawiki to 1.35 [21:01:21] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.022 second response time [21:01:25] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 39.91, 28.47, 21.86 [21:02:06] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.77.107.210/cpweb, 51.222.27.129/cpweb [21:02:57] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 9.00, 7.28, 5.62 [21:03:29] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 2607:5300:205:200::2ac4/cpweb [21:04:02] !log repool mw5 [21:04:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:04:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:54] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.16, 6.08, 5.39 [21:05:11] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [21:05:18] paladox: rendering completely broke here [21:05:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:05:35] Spookreeeno screenshot [21:05:38] as it works for me [21:05:48] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [21:05:49] paladox: cleared cache and it renders but a mess [21:05:52] !log depool mw6 [21:05:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:05] oh [21:06:12] https://usercontent.irccloud-cdn.com/file/69K692lV/1603832765.JPG [21:06:25] mw7 is serving that [21:06:41] It seems to a horrible cache on proxy end maybe? [21:07:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 11.54, 21.86, 21.71 [21:07:32] mw7 is the last one i need to depool [21:07:33] @Spookareeeno do you think the CSS issues could be deprecated CSS classes in MW 1.35 that will need to be updated? [21:07:43] but cannot till mw6 is done, otherwise everything will go down [21:07:52] Though yes [21:07:56] i think this may be cache [21:08:04] s/Spookareeeno/Spookreeeno [21:08:04] dmehus meant to say: @Spookreeeno do you think the CSS issues could be deprecated CSS classes in MW 1.35 that will need to be updated? [21:08:08]