[00:03:06] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:20] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:48] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:00] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:12] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:26] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:34] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:07] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [00:11:21] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [00:11:47] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [00:12:01] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [00:12:27] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:14:11] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:20:02] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fpLyL [00:20:04] [02miraheze/mediawiki] 07paladox 038941ae1 - Update composer.json [00:25:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [00:31:18] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:42:02] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+0/-0/±2] 13https://git.io/fpLSm [00:42:03] [02miraheze/mediawiki] 07legoktm 0348884db - Update git submodules * Update vendor from branch 'REL1_32' to 90fef8279a44622b14cf9d84de7f051a4c50b9bd - Remove non-tarball dependencies from release branch Just like we did in REL1_31. Change-Id: Ib52116409112a10a7f7c587660eb060b17f33d12 [00:42:05] [02miraheze/mediawiki] 07paladox 03bc55822 - Merge branch 'REL1_32' of https://github.com/wikimedia/mediawiki into REL1_32 [00:42:05] Title: [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [01:38:31] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:38:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpLHg [01:38:34] [02miraheze/mw-config] 07paladox 038d94158 - Add wgPageLanguageUseDB to ManageWiki [01:38:35] [02mw-config] 07paladox opened pull request 03#2552: Add wgPageLanguageUseDB to ManageWiki - 13https://git.io/fpLH2 [01:41:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpLH1 [01:41:20] [02miraheze/mw-config] 07paladox 0385fe75b - Update ManageWiki.php [01:41:21] [02mw-config] 07paladox synchronize pull request 03#2552: Add wgPageLanguageUseDB to ManageWiki - 13https://git.io/fpLH2 [01:41:38] miraheze/mw-config/paladox-patch-1/8d94158 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/453149820 [01:44:05] [02mw-config] 07paladox closed pull request 03#2552: Add wgPageLanguageUseDB to ManageWiki - 13https://git.io/fpLH2 [01:44:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fpLHp [01:44:08] [02miraheze/mw-config] 07paladox 0398a7085 - Add wgPageLanguageUseDB to ManageWiki (#2552) * Add wgPageLanguageUseDB to ManageWiki * Update ManageWiki.php [01:44:10] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [01:44:11] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [01:52:23] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/fpLQs [01:52:24] [02miraheze/mw-config] 07paladox 0395fdbe5 - Revert "Add wgPageLanguageUseDB to ManageWiki (#2552)" This reverts commit 98a7085d34b3fd2b59ce8971ce32c356d2d3e393. [01:52:26] [02miraheze/mw-config] 07paladox 03936c454 - Enable wgPageLanguageUseDB for guiaslocaiswiki [01:52:50] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/fpLQG [01:52:51] [02miraheze/mw-config] 07paladox 03ccdaa06 - Revert "Enable wgPageLanguageUseDB for guiaslocaiswiki" This reverts commit 936c45407a6935c4d6109d21b4c19cffa014660c. [01:52:53] [02miraheze/mw-config] 07paladox 038374e81 - Revert "Revert "Add wgPageLanguageUseDB to ManageWiki (#2552)"" This reverts commit 95fdbe5787550ee43b6147c0a7e2d6f06030ebe3. [01:56:18] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fpLQB [01:56:20] [02miraheze/ManageWiki] 07paladox 035fdc075 - Move the is_null check around [01:56:21] [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns [01:56:23] [02ManageWiki] 07paladox opened pull request 03#38: Move the is_null check around - 13https://git.io/fpLQR [02:12:42] [02ManageWiki] 07JohnFLewis commented on pull request 03#38: Move the is_null check around - 13https://git.io/fpLQF [02:14:23] [02ManageWiki] 07paladox commented on pull request 03#38: Move the is_null check around - 13https://git.io/fpLQj [02:16:28] [02ManageWiki] 07JohnFLewis commented on pull request 03#38: Move the is_null check around - 13https://git.io/fpL7f [02:21:26] [02ManageWiki] 07JohnFLewis closed pull request 03#38: Move the is_null check around - 13https://git.io/fpLQR [03:55:30] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 76918 MB (20% inode=96%); [08:00:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpLxs [08:00:15] [02miraheze/services] 07MirahezeSSLBot 03123e2fb - BOT: Updating services config for wikis [09:04:27] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:04:35] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:12:25] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:12:35] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:06:53] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fpLjt [10:06:55] [02miraheze/ssl] 07Reception123 03a1f0bc0 - add en.gyaanipedia.co.in cert [10:15:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpLj0 [10:15:13] [02miraheze/services] 07MirahezeSSLBot 03e3cd5b6 - BOT: Updating services config for wikis [12:50:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fptJa [12:50:14] [02miraheze/services] 07MirahezeSSLBot 03b90ebe8 - BOT: Updating services config for wikis [13:33:03] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [13:33:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [13:35:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:35:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:38:29] PROBLEM - bacula1 Bacula Private Git on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: puppet1-fd [13:40:27] PROBLEM - bacula1 Bacula Private Git on bacula1 is WARNING: WARNING: Full, 1613 files, 1.858MB, 2018-10-28 00:08:00 (1.9 weeks ago) [13:42:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:42:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:42:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:43:17] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:43:21] PROBLEM - lizardfs1 Puppet on lizardfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:43:29] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:43:53] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:43:59] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:01] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:03] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:05] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:11] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:13] PROBLEM - lizardfs2 Puppet on lizardfs2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:17] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:44:19] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:46:17] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:49:41] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:51:25] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [13:51:43] PROBLEM - misc3 Parsoid on misc3 is CRITICAL: connect to address 185.52.1.144 and port 8142: Connection refused [13:51:47] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [13:51:55] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:51:57] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:51:59] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [13:52:01] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [13:52:03] RECOVERY - lizardfs1 Puppet on lizardfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:52:09] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:52:11] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:52:21] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:52:25] RECOVERY - lizardfs2 Puppet on lizardfs2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:55:16] !log upgraded redis on misc2 to 5.0 [13:55:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:55:27] !log updated debian on misc* to 9.6 [13:55:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:55:51] !log upgraded bacula1 to debian 9.6 [13:55:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:59:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [13:59:12] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [13:59:20] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3801 bytes in 0.008 second response time [13:59:26] PROBLEM - misc4 Lizardfs Master Port 3 on misc4 is CRITICAL: connect to address 185.52.3.121 and port 9421: Connection refused [13:59:36] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [13:59:46] PROBLEM - misc4 Lizardfs Master Port 1 on misc4 is CRITICAL: connect to address 185.52.3.121 and port 9419: Connection refused [14:00:20] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3814 bytes in 1.084 second response time [14:00:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:00:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:01:02] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:01:10] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [14:01:16] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23556 bytes in 0.022 second response time [14:01:18] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 57% [14:01:20] RECOVERY - misc4 Lizardfs Master Port 3 on misc4 is OK: TCP OK - 0.003 second response time on 185.52.3.121 port 9421 [14:01:30] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 16% [14:01:32] RECOVERY - misc3 Parsoid on misc3 is OK: TCP OK - 0.001 second response time on 185.52.1.144 port 8142 [14:01:44] RECOVERY - misc4 Lizardfs Master Port 1 on misc4 is OK: TCP OK - 0.002 second response time on 185.52.3.121 port 9419 [14:01:52] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:02:18] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23556 bytes in 1.513 second response time [14:02:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:02:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:02:40] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [14:04:08] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 78713 MB (21% inode=96%); [14:21:29] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fptIB [14:21:31] [02miraheze/mediawiki] 07paladox 03526fb96 - Update Wikibase [14:23:59] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3140 [14:25:34] Hi [15:58:46] [02mediawiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbL5b [15:58:48] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fptm9 [15:58:49] [02miraheze/mediawiki] 07paladox 03a5fb05e - Update composer.json [15:58:51] [02mediawiki] 07paladox opened pull request 03#113: Update composer.json - 13https://git.io/fptmH [15:58:54] [02mediawiki] 07paladox closed pull request 03#113: Update composer.json - 13https://git.io/fptmH [15:58:55] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_32 [+0/-0/±2] 13https://git.io/fptmQ [15:58:57] [02miraheze/mediawiki] 07paladox 0318a9991 - Merge pull request #113 from miraheze/paladox-patch-1 Update composer.json [15:59:56] [02mediawiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbL5b [15:59:57] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-1 [16:13:08] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:36] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:06] PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:30] PROBLEM - test1 SSH on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:14:44] PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100% [16:16:46] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:16:54] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 22759 bytes in 0.009 second response time [16:17:06] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:17:30] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.19, 0.09, 0.03 [16:17:54] RECOVERY - test1 php7.2-fpm on test1 is OK: PROCS OK: 5 processes with command name 'php-fpm7.2' [16:55:36] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 75145 MB (20% inode=96%); [17:01:05] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.96, 6.70, 4.24 [17:14:07] uh [17:15:35] how did the load manage to spike like that? [17:23:01] huh the storage is going up by that i mean being used [17:27:04] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 5.68, 7.82, 7.57 [17:27:36] RECOVERY - db4 Disk Space on db4 is OK: DISK OK - free space: / 78850 MB (21% inode=96%); [17:29:02] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [17:29:04] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 1.64, 5.58, 6.78 [17:29:06] !log restarted db4 to reduce load and cpu as it was quite high (warnings from icinga too) [17:29:24] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [17:29:40] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 3799 bytes in 0.011 second response time [17:30:02] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [17:30:10] is that what's causing the issues with cp*? [17:30:58] Voidwalker yup [17:31:02] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [17:31:10] Voidwalker i did a emergency restart since the load was very high [17:31:21] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [17:31:23] and i had no idea how it the storage was going down so fast. [17:31:34] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 50% [17:31:39] !log restarted db4 to reduce load and cpu as it was quite high (warnings from icinga too) [17:31:40] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 23556 bytes in 0.012 second response time [17:31:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:32:02] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 3% [17:32:48] can you deploy the most recent change to CW once everything is in order? [17:33:34] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 28% [17:33:43] Voidwalker yup [17:34:20] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fptGM [17:34:21] [02miraheze/mediawiki] 07paladox 0358685a1 - Update CW [17:35:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fptGD [17:35:33] [02miraheze/mediawiki] 07paladox 03f38c9da - Update CW [17:41:57] Voidwalker should be deployed now [17:44:02] looks like it works [17:47:11] :) [18:22:36] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 28% [18:34:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fptc3 [18:34:59] [02miraheze/puppet] 07paladox 037c5c8f8 - Run gpg with --no-tty [18:40:03] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [18:40:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fptc0 [18:40:14] [02miraheze/services] 07MirahezeSSLBot 039670846 - BOT: Updating services config for wikis [18:41:22] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fptc2 [18:41:23] [02miraheze/puppet] 07paladox 034200d6d - Revert "Run gpg with --no-tty" This reverts commit 7c5c8f80ad1c3a130cad57c64e1243fbb0604165. [18:41:25] [02miraheze/puppet] 07paladox 034fe4129 - Update grafana apt key