[00:01:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3tV [00:02:01] [02miraheze/puppet] 07paladox 03bd72257 - Update mediawiki.pp [00:07:20] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:07:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je3tr [00:07:23] [02miraheze/mw-config] 07paladox 03f306dbb - Move allthetropeswiki and metawiki to use the new mount point [00:07:25] [02mw-config] 07paladox opened pull request 03#2762: Move allthetropeswiki and metawiki to use the new mount point - 13https://git.io/Je3to [00:07:36] paladox: new mount point? [00:07:52] We are making att and meta use gluster [00:08:09] paladox: any downtime? [00:08:12] nope [00:08:34] paladox: how are you moving file systems without downtime :O hacks [00:08:42] :D [00:08:49] not but really how? [00:32:07] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [00:32:22] PROBLEM - cp4 Stunnel Http for test1 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [00:32:27] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [00:32:38] PROBLEM - test1 HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:44:02] RECOVERY - cp4 Stunnel Http for test1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.019 second response time [00:44:03] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.496 second response time [00:44:10] RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 1.037 second response time [00:44:39] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.009 second response time [06:03:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.98, 1.64, 1.33 [06:05:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.17, 1.48, 1.31 [06:26:36] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3020 MB (12% inode=94%); [11:45:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3nI [11:45:10] [02miraheze/services] 07MirahezeSSLBot 030ab182f - BOT: Updating services config for wikis [11:55:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3na [11:55:11] [02miraheze/services] 07MirahezeSSLBot 03cab1cc8 - BOT: Updating services config for wikis [12:00:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3ny [12:00:11] [02miraheze/services] 07MirahezeSSLBot 03b36032e - BOT: Updating services config for wikis [12:18:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [12:19:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [12:19:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:19:13] Looking [12:19:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [12:20:32] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [12:20:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:21:05] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:21:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [12:33:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:33:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:34:03] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [12:34:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [12:34:43] Reception123: ^ [12:35:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:35:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:36:03] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [12:36:32] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [12:36:50] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3cB [12:36:51] [02miraheze/CreateWiki] 07translatewiki 03a415fda - Localisation updates from https://translatewiki.net. [12:36:52] [ Main page - translatewiki.net ] - translatewiki.net. [12:58:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [12:58:43] Again [12:59:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:59:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [12:59:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [12:59:51] Reception123, paladox, SPF|Cloud: it keeps doing this ^ [12:59:59] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [13:01:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:01:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:01:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [13:01:59] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [13:02:33] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:41:58] I'd have no idea about that [13:42:41] Reception123: ok, load times are a bit slow for meta [13:44:56] I’m mobile [13:45:51] I'll send an email out if SPF|Cloud doesn't respond soon [13:46:12] I'm also mobile but anyway wouldn't be able to do much [13:46:35] Hmm [13:50:47] JohnLewis: we keep getting backend down errors for mw* and slow loading times [13:51:01] That’ll be lizardfs... [14:06:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:06:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:08:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:08:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:25:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.78, 7.31, 6.05 [14:29:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.05, 6.55, 6.02 [14:36:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:38:12] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:42:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [14:44:02] PROBLEM - glusterfs1 GlusterFS port 49152 on glusterfs1 is CRITICAL: connect to address 81.4.100.90 and port 49152: Connection refused [14:44:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:44:18] PROBLEM - glusterfs2 GlusterFS port 49152 on glusterfs2 is CRITICAL: connect to address 81.4.100.77 and port 49152: Connection refused [14:45:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static-new] [14:45:42] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static-new] [14:46:41] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static-new] [14:56:35] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:56:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:58:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:58:34] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:07:28] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [16:09:47] !log gluster volume set mvolume performance.io-thread-count "12" - on gluster [16:09:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:10:02] RECOVERY - glusterfs1 GlusterFS port 49152 on glusterfs1 is OK: TCP OK - 0.007 second response time on 81.4.100.90 port 49152 [16:12:18] RECOVERY - glusterfs2 GlusterFS port 49152 on glusterfs2 is OK: TCP OK - 0.001 second response time on 81.4.100.77 port 49152 [16:12:35] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:12:49] hmm, why didn't gluster auto recover... [16:12:51] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:13:14] !log restarted glusterd on glusterfs[12] [16:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:14:34] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:23:40] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [16:23:42] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Je34w [16:23:43] [02miraheze/dns] 07paladox 035399412 - Add static-temp to dns Will be used to evaluate gluster. [16:23:45] [02dns] 07paladox opened pull request 03#112: Add static-temp to dns - 13https://git.io/Je34r [16:26:43] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [16:26:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Je34P [16:26:47] [02miraheze/puppet] 07paladox 030133987 - mediawiki: Add static-temp Will be used to trial glusterfs for a few wikis. [16:26:48] [02puppet] 07paladox opened pull request 03#1089: mediawiki: Add static-temp - 13https://git.io/Je34X [16:28:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je34D [16:28:24] [02miraheze/puppet] 07paladox 036d12a44 - gluster: use hasrestart and hasstatus for service [16:32:34] [02dns] 07paladox closed pull request 03#112: Add static-temp to dns - 13https://git.io/Je34r [16:32:35] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je34d [16:32:37] [02miraheze/dns] 07paladox 03ab8b945 - Add static-temp to dns (#112) Will be used to evaluate gluster. [16:32:38] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [16:32:40] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [16:34:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je34b [16:34:09] [02miraheze/puppet] 07paladox 03b26c43b - Update default.vcl [16:34:22] [02puppet] 07paladox closed pull request 03#1089: mediawiki: Add static-temp - 13https://git.io/Je34X [16:34:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je34N [16:34:25] [02miraheze/puppet] 07paladox 03ed28192 - mediawiki: Add static-temp (#1089) Will be used to trial glusterfs for a few wikis. [16:34:27] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [16:34:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [16:35:56] [02mw-config] 07paladox synchronize pull request 03#2762: Move allthetropeswiki and metawiki to use the new mount point - 13https://git.io/Je3to [16:35:57] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Je34j [16:35:59] [02miraheze/mw-config] 07paladox 03f4bca3a - Update LocalSettings.php [16:36:37] !log gluster volume set mvolume network.ping-timeout "20" [16:36:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:38:10] !log gluster volume set mvolume performance.parallel-readdir "on" [16:38:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:39:23] [02mw-config] 07paladox closed pull request 03#2762: Move allthetropeswiki and metawiki to use the new mount point - 13https://git.io/Je3to [16:39:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3Bt [16:39:26] [02miraheze/mw-config] 07paladox 037b1096f - Move allthetropeswiki and metawiki to use the new mount point (#2762) * Move allthetropeswiki and metawiki to use the new mount point * Update LocalSettings.php [16:41:02] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [16:41:04] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [16:42:33] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.72, 7.93, 6.71 [16:44:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 3.36, 6.31, 6.27 [16:45:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw2 mw3 [16:46:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:46:14] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [16:46:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:46:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:46:44] reverting [16:47:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3BR [16:47:23] [02miraheze/mw-config] 07paladox 03a51b544 - Revert "Move allthetropeswiki and metawiki to use the new mount point (#2762)" This reverts commit 7b1096f7d1b47542766fc97e384187d858298745. [16:48:25] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:55] paladox: meta 503 [16:50:05] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [16:50:06] Good you've reverted [16:50:06] yes see my message above [16:50:07] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [16:50:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3Bg [16:50:09] [02miraheze/services] 07MirahezeSSLBot 03e14efab - BOT: Updating services config for wikis [16:50:19] I need to go for 2-3 hours [16:50:24] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.390 second response time [16:50:51] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:39] hmm, everything is still slow [16:51:45] even though everything is reverted [16:52:26] !log reboot mw1 [16:54:09] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.389 second response time [16:54:09] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.062 second response time [16:54:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:54:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:55:03] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 1.624 second response time [16:56:12] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [16:56:32] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:57:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:59:00] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.19, 7.21, 6.27 [17:00:56] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.56, 6.89, 6.26 [17:01:44] !log manually reverting https://git.io/Je3BR on mw (each by each) testing new configuation change gluster side. [17:01:45] [ Comparing 7b1096f7d1b4...a51b544aecd3 · miraheze/mw-config · GitHub ] - git.io [17:02:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:02:51] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.07, 5.94, 5.96 [17:04:46] !log revert is working! [17:04:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:05:42] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/Je3B5 [17:05:44] [02miraheze/mw-config] 07paladox 03f29568d - Revert "Revert "Move allthetropeswiki and metawiki to use the new mount point (#2762)"" This reverts commit a51b544aecd3de0a39b18cbf8476c2b77bf22866. [17:05:45] [02miraheze/mw-config] 07paladox 03ec757e8 - Move loginwiki to new mount Change-Id: I2130e5c31b5ba159c23350421682d92f73ec45c7 [17:15:43] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [17:17:21] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.60, 6.84, 6.45 [17:19:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3RU [17:19:02] [02miraheze/mw-config] 07paladox 038595fd0 - Update LocalSettings.php [17:19:17] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.76, 6.44, 6.34 [17:20:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3Rt [17:20:27] [02miraheze/mw-config] 07paladox 033f14bc4 - Fix overriding $wgUploadPath [17:27:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.31, 7.03, 6.58 [17:29:15] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.38, 7.25, 6.69 [17:30:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3Rl [17:30:13] [02miraheze/services] 07MirahezeSSLBot 0394b6b08 - BOT: Updating services config for wikis [17:35:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.06, 7.03, 6.91 [17:37:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.13, 6.34, 6.67 [18:13:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.15, 6.67, 6.00 [18:17:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.88, 7.45, 6.44 [18:19:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.79, 7.21, 6.48 [18:29:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.49, 7.53, 6.90 [18:33:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.59, 7.73, 7.13 [18:39:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.48, 6.13, 6.57 [18:58:58] paladox: so what exactly did doing the new mount do? [18:59:10] It moves us to a new file system. [18:59:13] called gluster [18:59:24] i'm trialing it for a few big wikis [18:59:41] paladox: do I get new superpowers? [18:59:47] nope [18:59:55] then I dont want it xD [19:01:19] lol [19:41:03] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.83, 1.43, 1.11 [19:41:53] .gethelp test [19:41:53] RhinosF1, needs help. Pinging Reception123, Zppix, PuppyKun, Voidwalker, RhinosF1. [19:41:57] Worked [19:42:01] Sorry for the ping [19:43:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.04, 1.26, 1.08 [19:57:03] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.27, 1.73, 1.37 [19:59:03] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.57, 1.65, 1.38 [20:01:41] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [20:01:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [20:02:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:03:29] hmm [20:03:40] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:03:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:03:46] that's not gluster [20:03:48] paladox: back [20:03:56] Meta and QuIRC we're up throughout [20:04:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:04:20] Ah [20:04:26] hmm [20:16:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:16:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:17:09] ... [20:17:17] well at least meta stays up now :P [20:18:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:18:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:19:30] PROBLEM - test1 Disk Space on test1 is WARNING: DISK WARNING - free space: / 4367 MB (10% inode=97%); [20:54:26] paladox: How hard is it to migrate wikis to gluster? [20:54:35] not that hard [20:54:50] paladox: good [21:05:16] well... he says that 10 days after migration started :P [21:05:40] JohnLewis well how do you define hard then? :) [21:05:47] because all i did was type a command :D [21:06:07] and wait 10+ days :P [21:06:50] it's easy as 1... 2... 3... 4... [...] 5000000593... 5000000594... [...] [21:12:07] JohnLewis: please wikimedia's easy is 300 reverts later a blown up router and some operations team member running from the issue so no one will blame them [21:12:08] xD [21:12:39] basically what paladox does, except the blown up router because we don't own one :P [21:12:50] Zppix: have you seen the train tasks recently? [21:13:30] PROBLEM - test1 Disk Space on test1 is CRITICAL: DISK CRITICAL - free space: / 2445 MB (5% inode=97%); [21:15:48] RhinosF1: No but i get the emails [21:16:20] JohnLewis: idk hes gotten better at not doing that, either that or ramnode finally hired a full time Fire extinguisher for Miraheze [21:16:45] Zppix: that's been so unreliable recently [21:17:01] RhinosF1: i mean they do have to legally give him lunch and breaks [21:17:14] I mean he does get a hazard pay of the entire wmf staff combined [21:17:30] Zppix: no, the train [21:17:43] it seems to be blocked or rolled back constantly [21:18:02] JohnLewis i do what?!?!?! [21:18:28] paladox: what Zppix said :P [21:18:44] "300 reverts later a blown up router and some operations team member running from the issue so no one will blame them" [21:18:44] ? [21:19:02] * paladox pushes the blame onto his manager :P [21:20:15] Thats JohnLewis for anyone that didn't know :D [21:20:33] nah [21:20:41] I don't do more than 2 reverts :P [21:20:43] yes :P [21:20:58] Gerrit has a new revert all the changes button comming soon :P [21:21:14] that means if you create a stack of changes, you can revert them all in one go i think [21:22:18] https://gerrit-review.googlesource.com/c/homepage/+/233996 [21:41:30] RECOVERY - test1 Disk Space on test1 is OK: DISK OK - free space: / 8762 MB (21% inode=98%); [21:57:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [21:57:56] paladox: ^ [21:58:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3gg [21:58:01] [02miraheze/mw-config] 07paladox 03d278311 - Move nonciclopediawiki to use new mount point [21:58:15] hmm [21:58:45] both mount points are up [21:59:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:59:49] seems resolved [22:02:08] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3gP [22:02:10] [02miraheze/mw-config] 07paladox 03baad548 - Move anotheredenwiki to new mount point [22:19:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3gH [22:19:02] [02miraheze/mw-config] 07paladox 03b9de200 - Revert "Move nonciclopediawiki to use new mount point" This reverts commit d2783115c4620b8fc9763a1366faa407bfd88346. Change-Id: Id654ce2197b109b5b1d1725a4f559fc210518a45 [22:20:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je3gQ [22:21:00] [02miraheze/mw-config] 07paladox 0376cfb6e - Revert "Revert "Move nonciclopediawiki to use new mount point"" This reverts commit b9de200353ab320cde13e6961575d2109364f029. [22:21:25] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:39:14] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 11.63, 8.73, 6.35 [22:45:15] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 5.69, 7.54, 6.71 [22:45:28] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 9.66, 6.55, 4.72 [22:49:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.37, 6.12, 6.35 [22:49:26] PROBLEM - mw1 Current Load on mw1 is WARNING: WARNING - load average: 7.40, 7.33, 5.49 [22:53:24] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 3.39, 5.76, 5.30 [23:13:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.81, 6.73, 5.84 [23:15:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.65, 6.14, 5.72 [23:42:39] Hello Spiffidimus! If you have any questions, feel free to ask and someone should answer soon. [23:43:01] I'd like to know who owns the data I post to Miraheze. [23:43:28] I ask because I'd like to build a Wikibook in private and then later post it to Wikibooks. [23:57:35] Spiffidimus: https://creativecommons.org/licenses/by-sa/4.0/ [23:57:36] [ Creative Commons — Attribution-ShareAlike 4.0 International — CC BY-SA 4.0 ] - creativecommons.org [23:57:49] thast is the terms of any contribution on a miraheze wiki by defualt [23:57:50] default*