[00:06:49] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:19:55] !log restart lizardfs-master [01:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:25:28] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [01:28:04] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:49:06] paladox: is it possible to give CVT commentadmin globally? [01:52:51] I’m not too sure Zppix [01:52:53] https://meta.miraheze.org/wiki/Special:GlobalGroupPermissions [01:52:53] Dosent show it and I’m not sure if we changed that on the wiki that has that, that it would be overwritten if we changed it on meta where that right is not available. [01:52:54] [ Global group management - Miraheze Meta ] - meta.miraheze.org [01:53:44] Also I’m about to go. [01:55:27] ok [02:04:49] it doesn't get overwritten [06:26:58] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2999 MB (12% inode=94%); [08:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeC4L [08:40:11] [02miraheze/services] 07MirahezeSSLBot 0333f1ed9 - BOT: Updating services config for wikis [10:07:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [10:13:07] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [10:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCBS [10:25:09] [02miraheze/services] 07MirahezeSSLBot 03cb7b28b - BOT: Updating services config for wikis [11:00:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCR8 [11:00:11] [02miraheze/services] 07MirahezeSSLBot 034bc761b - BOT: Updating services config for wikis [12:05:33] Hello Guest30634! If you have any questions, feel free to ask and someone should answer soon. [12:37:36] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown) [12:38:21] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 5 minutes ago with 8 failures. Failed resources (up to 3 shown) [12:38:34] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 107 failures. Last run 5 minutes ago with 107 failures. Failed resources (up to 3 shown) [12:38:58] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 156 failures. Last run 5 minutes ago with 156 failures. Failed resources (up to 3 shown): File[drag.lgbt_private],File[wikiverte.pl],File[wikiverte.pl_private],File[nonciclopedia.org] [12:39:25] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown) [12:44:36] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [12:45:19] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:46:24] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:46:59] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:47:28] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:51:10] !log restarting php-fpm on mw3 [13:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:06:25] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [14:06:45] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:06:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:09:21] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [14:09:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:09:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:13:16] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2643 MB (10% inode=94%); [14:29:07] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [14:31:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:34:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:35:25] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [15:08:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [15:10:54] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [15:11:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:13:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:22:21] !log a test [15:22:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:22:41] paladox: third of the speed [15:22:56] 1/6 th even [15:23:00] hmm? [15:23:06] It's 12s to login and 12 to edit [15:23:07] it logged it in 5s [15:23:09] For ZppixBot [15:23:10] which is not bad. [15:23:21] paladox: no but watch this [15:23:31] .status mhtest paladox [15:23:33] RhinosF1updating User:RhinosF1/Status! [15:23:41] RhinosF1: Done! [15:23:49] 7s [15:23:58] paladox: ^ it's getting quicker so maybe it was just high load [15:24:11] you can use grafana to confirm that [15:24:21] but even with highload, it would have to be extreamly high [15:24:31] since the cores are clocked at a high speed [15:24:49] paladox: you can see #ZppixBot 's wm-bot logs for how bad it was earlier [15:24:57] .status mhtest around [15:25:00] RhinosF1updating User:RhinosF1/Status! [15:25:08] RhinosF1: Done! [15:26:20] .mh SAL [15:26:20] https://meta.miraheze.org/wiki/SAL [18:50:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:50:45] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [18:53:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:53:17] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:39:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeCaj [19:39:16] [02miraheze/puppet] 07paladox 035eb0866 - Remove vpncloud from misc3 [19:49:29] !log stop electron on misc3 [19:49:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:04:29] How much ram and cpu does your wikis use? [20:08:18] Hi, mw[123] have 4cpu's each and 1gb of ram each. [20:08:33] our caching proxy (varnish) has 1gb of ram and 1 cpu. [20:12:47] Thanks [20:13:37] Are wikis more cpu heavy or ram hevy or what? [20:38:17] BurningPrincess i'm not entirely sure, i haven't looked into that. [20:38:43] but we are getting high load on the cpu sometimes so i'm thinking more cpu. [20:45:38] * BurningPrincess nods. [21:09:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [21:09:55] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [21:12:42] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [21:15:07] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:53:48] how do you pronounce miraheze? [23:55:20] apap: I pronounce it Mirrah (mirror but with ah instead of or) then hease (like hes with the word ease combined) [23:58:08] i always pronounced it mirrah-hazeh [23:58:10] weird [23:58:44] the pronounciation should be put on the front page.. because why not ;) [23:59:17] (if there should be a preferred pronounciation)