[03:04:00] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.36, 3.98, 3.22 [03:05:59] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.28, 3.64, 3.19 [03:07:59] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 2.23, 3.19, 3.07 [04:32:47] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.53, 3.53, 3.11 [04:34:47] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 3.05, 3.35, 3.09 [04:38:44] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.70, 3.61, 3.26 [04:50:39] PROBLEM - misc2 Current Load on misc2 is CRITICAL: CRITICAL - load average: 4.31, 3.80, 3.52 [04:54:38] PROBLEM - misc2 Current Load on misc2 is WARNING: WARNING - load average: 3.73, 3.90, 3.62 [04:56:37] RECOVERY - misc2 Current Load on misc2 is OK: OK - load average: 1.01, 2.82, 3.26 [10:00:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjrWz [10:00:18] [02miraheze/services] 07MirahezeSSLBot 0366f7b40 - BOT: Updating services config for wikis [11:56:11] if someone in staff team needs some http://atlas.ripe.net credit to run measurements for Miraheze I can give them some [11:56:12] [ RIPE Atlas - RIPE Network Coordination Centre ] - atlas.ripe.net [12:14:12] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 372 bytes in 0.010 second response time [12:14:48] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:15:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:15:28] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:38] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:15:43] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:15:44] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [12:16:22] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [12:16:49] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [12:21:10] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [12:21:23] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:21:38] oh [12:21:39] db4 [12:22:19] !log restart mysql on db4 [12:23:16] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 347 bytes in 0.012 second response time [12:24:21] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (111 "Connection refused") [12:25:15] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [12:25:22] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4227 bytes in 0.029 second response time [12:27:09] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 49% [12:29:58] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41136 bytes in 0.199 second response time [12:30:21] RECOVERY - db4 MySQL on db4 is OK: Uptime: 160 Threads: 60 Questions: 2331 Slow queries: 232 Opens: 196 Flush tables: 1 Open tables: 190 Queries per second avg: 14.568 [12:30:42] !log [13:22:19] <+paladox> !log restart mysql on db4 [12:30:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:30:48] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [12:30:49] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is WARNING: WARNING - NGINX Error Rate is 44% [12:31:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:31:09] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [12:31:15] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17725 bytes in 0.275 second response time [12:31:22] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19074 bytes in 0.235 second response time [12:31:38] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:31:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:31:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [12:31:52] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5805 bytes in 0.040 second response time [12:32:22] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 5% [12:32:49] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 7% [13:12:26] [02CreateWiki] 07paladox closed pull request 03#116: Fix race condition where sometimes you get a 404 just as the list files are updating - 13https://git.io/fjwIP [13:12:27] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjr4p [13:12:29] [02miraheze/CreateWiki] 07paladox 03929e25f - Fix race condition where sometimes you get a 404 just as the list files are updating (#116) [13:12:30] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [13:12:32] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [13:13:16] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjr4h [13:13:18] [02miraheze/mediawiki] 07paladox 03831ed51 - Update CW [13:15:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+0/-0/±1] 13https://git.io/fjrBT [13:15:19] [02miraheze/mediawiki] 07paladox 03f79776e - Update CW [13:15:54] [02miraheze/mediawiki] 07paladox pushed 033 commits to 03REL1_33 [+0/-0/±3] 13https://git.io/fjrBL [13:15:55] [02miraheze/mediawiki] 07urbanecm 0371a1f93 - [bugfix] Fetch tag ID before calling undefineTag() undefineTag() also deletes the tag if it's not used anywhere, which is breaking the rest of deleteTagEverywhere() in that case. Bug: T225564 Change-Id: I7ca5db9efd0088b266e33c0a9ce78d73a4fa87c9 (cherry picked from commit 27cbfaa54f1e699b1b2f61ef2f33eff1c999819b) [13:15:57] [02miraheze/mediawiki] 07jenkins-bot 03dba14a8 - Merge "[bugfix] Fetch tag ID before calling undefineTag()" into REL1_33 [13:15:58] [02miraheze/mediawiki] 07paladox 03d900baf - Merge branch 'REL1_33' of https://github.com/wikimedia/mediawiki into REL1_33 [13:15:59] [ GitHub - wikimedia/mediawiki: 🌻 The collaborative editing software that runs Wikipedia. This is a mirror from gerrit.wikimedia.org. See https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [13:40:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjrB9 [13:40:15] [02miraheze/services] 07MirahezeSSLBot 03c8ae470 - BOT: Updating services config for wikis [15:21:41] Zppix, same issue with relay bot again? [16:15:44] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±4] 13https://git.io/fjrut [16:15:45] [02miraheze/mw-config] 07paladox 0300e42cc - Revert "Add some cirrus/elastic configs to ManageWiki" This reverts commit a8424df01593d9d8adaca75948d03e61d6f1a2f3. Change-Id: I0c0247ebf3316c0a0fb81b4e24e211773058d3c1 [16:15:47] [02miraheze/mw-config] 07paladox 03e2a1428 - Add missing wikis to wmgSearchType Change-Id: I8b70cfd0138f83e613de5e5f15c98ff60346fe9b [16:47:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjrui [16:47:51] [02miraheze/mw-config] 07paladox 03d58680d - Add missing wikis to CirrusSearch Change-Id: I6b88229465bd15ae70945207cae1fb5131a7df33 [16:49:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjruD [16:49:21] [02miraheze/mw-config] 07paladox 03d60196c - Enable CirrusSearch on buswiki and set wmgDisableSearchUpdate Change-Id: I978ff1fbdc1d921626a31248d33d8d3c8ed355f0 [16:50:51] !log running updateSearchIndexConfig.php for buswiki [16:50:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:51:51] !log running forceSearchIndex.php for buswiki [16:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:53:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjruH [16:53:40] [02miraheze/mw-config] 07paladox 035d9a206 - Disable wmgDisableSearchUpdate for buswiki Change-Id: I6274062e87dd76eb4b9e80a148f3c5d41a2a2abf [16:58:07] miraheze/mw-config/master/5d9a206 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/550395743 [18:29:16] !log apt-get install nginx postfix postfix-sqlite puppet-agent rsync tzdata unzip on elasticsearch1 [18:29:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:45:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjr2f [19:45:17] [02miraheze/services] 07MirahezeSSLBot 03ac82155 - BOT: Updating services config for wikis [20:00:16] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjr23 [20:00:17] [02miraheze/services] 07MirahezeSSLBot 038ac8537 - BOT: Updating services config for wikis [20:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjr2o [20:25:13] [02miraheze/services] 07MirahezeSSLBot 03f323a51 - BOT: Updating services config for wikis [22:36:20] !log deleted CleavagesCwars from phabricator (per email) [22:36:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master