[00:04:28] !log reboot test1 [00:04:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:07:36] RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.013 second response time on 54.36.165.161 port 49152 [00:11:36] PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 49152: Connection refused [00:17:36] RECOVERY - lizardfs6 GlusterFS port 49152 on lizardfs6 is OK: TCP OK - 0.013 second response time on 54.36.165.161 port 49152 [00:22:52] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 58 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/var/lib/glusterd/secure-access] [00:24:51] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:35:42] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [00:43:41] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:17:28] !log installed rsync on gluster1 [01:17:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:32:16] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [01:46:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWRz [01:46:10] [02miraheze/puppet] 07paladox 032de04cb - gluster: Add port 49154 to firewall [01:48:16] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:51:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvWRo [01:51:21] [02miraheze/puppet] 07paladox 03548d098 - mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm [01:51:23] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:51:24] [02puppet] 07paladox opened pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:51:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvWRi [01:51:40] [02miraheze/puppet] 07paladox 0348a0bd7 - Update mw5.yaml [01:51:41] [02puppet] 07paladox synchronize pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:51:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvWRX [01:51:55] [02miraheze/puppet] 07paladox 03b7296e7 - Update mw6.yaml [01:51:57] [02puppet] 07paladox synchronize pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:52:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvWR1 [01:52:06] [02miraheze/puppet] 07paladox 0356966f3 - Update mw7.yaml [01:52:08] [02puppet] 07paladox synchronize pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:52:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvWRM [01:52:16] [02miraheze/puppet] 07paladox 0357f625d - Update test2.yaml [01:52:18] [02puppet] 07paladox synchronize pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:52:25] [02puppet] 07paladox closed pull request 03#1227: mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm - 13https://git.io/JvWRK [01:52:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JvWRD [01:52:28] [02miraheze/puppet] 07paladox 03e456670 - mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm (#1227) * mw[4567] & test2: Set gluster_volume to gluster1.miraheze.org:/mm * Update mw5.yaml * Update mw6.yaml * Update mw7.yaml * Update test2.yaml [01:52:29] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:52:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:59:33] Hey, I need help with CreateWiki and ManageWiki on my own wikis. [04:17:16] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2087 MB (10% inode=93%); [05:03:03] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 4395 MB (22% inode=93%); [06:26:59] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3178 MB (13% inode=94%); [14:54:59] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2643 MB (10% inode=94%); [15:14:56] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.74, 1.84, 1.18 [15:16:56] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.29, 0.96, 0.97 [15:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWyI [15:25:09] [02miraheze/services] 07MirahezeSSLBot 0306b5aba - BOT: Updating services config for wikis [16:32:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:34:47] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:37:51] PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Sat 29 Feb 2020 04:34:39 PM GMT +0000). [16:38:06] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvW9b [16:38:08] [02miraheze/ssl] 07MirahezeSSLBot 03874141e - Bot: Update SSL cert for programming.red [16:38:12] PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Sat 29 Feb 2020 04:34:39 PM GMT +0000). [16:44:22] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:44:23] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:44:54] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:45:38] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:46:05] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:46:10] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:46:11] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:47:05] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [16:52:15] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [16:52:18] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [16:53:03] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [16:53:49] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:53:59] JohnLewis: ^ [16:54:00] If you can’t deal with that now I can later [16:54:13] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:54:16] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:54:22] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:54:59] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:32:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 51.161.32.127/cpweb [17:34:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:20:37] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvW5L [18:20:39] [02miraheze/ssl] 07paladox 03e98f06f - Update programming.red.crt [18:24:12] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:24:19] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:24:45] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:25:36] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:26:08] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:26:28] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:26:40] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:26:59] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [18:32:07] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 13 May 2020 05:20:10 PM GMT +0000. [18:32:12] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:32:14] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [18:32:15] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [18:32:44] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [18:33:06] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [18:33:34] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 13 May 2020 05:20:10 PM GMT +0000. [18:33:42] !log restarted nginx on mw3 [18:33:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:33:56] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:34:30] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:35:07] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:55:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb [18:59:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:21:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvWFh [19:21:23] [02miraheze/mw-config] 07paladox 033f94c4f - New server migration announcement [19:21:24] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [19:21:26] [02mw-config] 07paladox opened pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:22:00] [02mw-config] 07paladox synchronize pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:22:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvWbU [19:22:03] [02miraheze/mw-config] 07paladox 03fb5f65d - Update Sitenotice.php [19:26:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb [19:27:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [19:27:18] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [19:27:24] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:27:29] [02mw-config] 07paladox synchronize pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:27:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvWbC [19:27:32] [02miraheze/mw-config] 07paladox 037cbe5c6 - Update Sitenotice.php [19:27:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [19:27:51] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3 [19:29:12] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [19:29:24] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [19:29:28] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.760 second response time [19:29:38] [02mw-config] 07paladox synchronize pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:29:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvWb4 [19:29:41] [02miraheze/mw-config] 07paladox 031bdf81a - Update Sitenotice.php [19:29:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [19:29:52] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [19:30:59] [02mw-config] 07paladox synchronize pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:31:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvWbR [19:31:02] [02miraheze/mw-config] 07paladox 039ac022f - Update Sitenotice.php [19:31:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:32:13] [02mw-config] 07paladox closed pull request 03#2879: New server migration announcement - 13https://git.io/JvWFj [19:32:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWba [19:32:16] [02miraheze/mw-config] 07paladox 03c96e6a2 - New server migration announcement (#2879) * New server migration announcement * Update Sitenotice.php * Update Sitenotice.php * Update Sitenotice.php * Update Sitenotice.php [19:33:28] PROBLEM - wiki.conworlds.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.conworlds.org' expires in 15 day(s) (Sat 29 Feb 2020 07:30:16 PM GMT +0000). [19:33:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWbi [19:33:44] [02miraheze/ssl] 07MirahezeSSLBot 03a5e6dc6 - Bot: Update SSL cert for wiki.conworlds.org [19:38:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [19:40:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [19:44:29] RECOVERY - wiki.conworlds.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.conworlds.org' will expire on Wed 13 May 2020 06:33:36 PM GMT +0000. [19:44:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [19:44:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:47:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:47:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:14:24] PROBLEM - cp8 Stunnel Http for mw3 on cp8 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.279 second response time [20:15:06] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:18] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:32] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3 [20:15:47] !log restart php7.3-fpm on mw3 [20:15:47] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [20:15:48] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [20:16:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:16:21] RECOVERY - cp8 Stunnel Http for mw3 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.879 second response time [20:17:06] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 3.442 second response time [20:17:17] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 3.448 second response time [20:17:31] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [20:17:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [20:17:48] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [20:37:33] paladox|UKInEU: the migration date is correct (sitenotice)? [20:37:40] yup [20:38:23] ok :) [20:43:30] !log stop phd & nginx on misc4. Dump phabricator db, migrate to db6 and migrate phabricator to phab1 [20:43:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:45:31] !log install mariadb-client-10.1 on misc4 (to backup the db) [20:45:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:46:28] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: connect to address 185.52.3.121 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:46:28] PROBLEM - misc4 phd on misc4 is CRITICAL: PROCS CRITICAL: 0 processes with args 'phd' [20:46:31] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: connect to address 185.52.3.121 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:46:48] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [20:51:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWAo [20:51:28] [02miraheze/puppet] 07paladox 036734bc2 - phabricator: Switch to db6 [20:53:00] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvWA6 [20:53:01] [02miraheze/dns] 07paladox 037795927 - phabricator: Switch backend to phab1 [20:53:03] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbQXl [20:53:04] [02dns] 07paladox opened pull request 03#123: phabricator: Switch backend to phab1 - 13https://git.io/JvWAi [20:53:26] [02dns] 07paladox synchronize pull request 03#123: phabricator: Switch backend to phab1 - 13https://git.io/JvWAi [20:53:28] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvWAP [20:53:29] [02miraheze/dns] 07paladox 03890b3ae - Update miraheze.org [20:59:45] [02dns] 07paladox closed pull request 03#123: phabricator: Switch backend to phab1 - 13https://git.io/JvWAi [20:59:46] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWAd [20:59:48] [02miraheze/dns] 07paladox 0352067d7 - phabricator: Switch backend to phab1 (#123) * phabricator: Switch backend to phab1 * Update miraheze.org [20:59:49] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [20:59:51] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbQXl [21:04:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWxT [21:04:49] [02miraheze/puppet] 07paladox 037664ead - phabricator: Add port 80/443 to firewall [21:05:49] paladox|UKInEU: phab1 and db6 live then? [21:06:04] phab1 is, and db6 is for phabricator. [21:06:43] paladox|UKInEU: great! Cp8+test2 are the only others right? [21:06:52] yes [21:07:34] paladox|UKInEU: have still got test1 pooled? [21:07:40] yes [21:09:06] paladox|UKInEU: https://meta.miraheze.org/w/index.php?title=Tech:Cloud_Server_Comparison&diff=96517&oldid=96510 [21:09:09] [ Difference between revisions of "Tech:Cloud Server Comparison" - Miraheze Meta ] - meta.miraheze.org [21:09:41] paladox|UKInEU, SPF|Cloud: phab is unreachable [21:09:50] that's expected [21:09:55] Ok [21:10:09] For how long paladox|UKInEU [21:10:37] My estimate maybe 10-30 mins [21:10:42] Kk [21:31:43] !log increased phab1 cores to 2 [21:32:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:02:38] !log increase phab1 disk to 20g [22:02:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:05:33] !log install parted on phab1 [22:05:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:17:30] Is anyone else getting "Error loading data from server" errors? [22:17:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWh8 [22:17:35] [02miraheze/puppet] 07paladox 03c190c04 - base: Install lvm2 and parted [22:17:42] MikeV is that VS? [22:17:46] *VE [22:18:23] paladox, What is *VE? [22:18:28] VisualEditor [22:19:30] paladox, VE is involved. [22:20:25] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 1139 bytes in 0.038 second response time [22:20:46] MikeV should work now [22:23:15] It does. Thanks. But come on... What happened? [22:26:52] I stoped nginx on misc4 for the migration of phabricator to our new server. [22:26:58] i forgot we used parsoid on misc4 [22:34:26] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:34:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWh1 [22:34:58] [02miraheze/puppet] 07paladox 035785ee9 - Remove phabricator from misc4 [22:36:23] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 1.820 second response time [22:36:59] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:39:16] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvWh9 [22:39:18] [02miraheze/dns] 07paladox 03cb0571f - Switch parsoid/restbase backend to services1 [22:39:19] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbQXl [22:39:20] [02dns] 07paladox opened pull request 03#124: Switch parsoid/restbase backend to services1 - 13https://git.io/JvWhH [22:41:01] [02dns] 07paladox closed pull request 03#124: Switch parsoid/restbase backend to services1 - 13https://git.io/JvWhH [22:41:02] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWh5 [22:41:04] [02miraheze/dns] 07paladox 03de34319 - Switch parsoid/restbase backend to services1 (#124) [22:41:05] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbQXl [22:41:07] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [22:47:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWjf [22:47:40] [02miraheze/puppet] 07paladox 0342a21ba - Remove parsoid from misc4 [22:48:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWjI [22:48:57] [02miraheze/puppet] 07paladox 03e2c6dc5 - Change from backing up misc4 to phab1 [22:53:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWj3 [22:54:01] [02miraheze/dns] 07paladox 030b672bc - Remove cloud.miraheze.org Not going to be used [22:54:58] PROBLEM - bacula1 Bacula Phabricator Static on bacula1 is CRITICAL: CRITICAL: Timeout or unknown client: misc4-fd [22:56:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvWjC [22:56:45] [02miraheze/puppet] 07paladox 030bd4e9a - misc4 -> phab1 [22:58:32] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvWjE [22:58:34] [02miraheze/dns] 07paladox 039a03916 - Add services2 as a backend to restbase/parsoid [22:58:35] [02dns] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbQXl [22:58:37] [02dns] 07paladox opened pull request 03#125: Add services2 as a backend to restbase/parsoid - 13https://git.io/JvWju [22:59:34] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvWjz [22:59:35] [02miraheze/dns] 07paladox 03bba791c - Update config [22:59:37] [02dns] 07paladox synchronize pull request 03#125: Add services2 as a backend to restbase/parsoid - 13https://git.io/JvWju [23:01:00] [02dns] 07paladox closed pull request 03#125: Add services2 as a backend to restbase/parsoid - 13https://git.io/JvWju [23:01:02] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvWjw [23:01:03] [02miraheze/dns] 07paladox 0332ccab9 - Add services2 as a backend to restbase/parsoid (#125) [23:01:05] [02dns] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbQXl [23:01:06] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-3 [23:13:10] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:15] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 5.314 second response time [23:22:34] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:22:40] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:24:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:24:39] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 3.596 second response time