[00:13:50] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:45:15] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9253],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9113],Package[php7.3-apcu],Package[php7.3-redis] [00:53:20] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [00:53:43] see the new message in or see the default? https://commons.miraheze.org/wiki/Special:Upload (I don't see it) https://commons.miraheze.org/wiki/MediaWiki:Uploadtext [00:53:44] [ Login required - Miraheze Commons ] - commons.miraheze.org [00:53:45] [ MediaWiki:Uploadtext - Miraheze Commons ] - commons.miraheze.org [00:55:47] hispano76: clear your cache [00:56:35] oh, ok :) Zppix [01:05:11] Um, it hasn't been updated. :/ [01:07:02] paladox: repool It affects languages, doesn't it? [01:07:10] repooled** [01:07:11] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:07:12] oh! [01:07:14] right [01:07:14] yes [01:07:16] i forgot lc [01:07:55] !log building lc on mw2 [01:08:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:16:40] hispano76 done! [01:16:43] (fixed) [01:18:50] to you paladox :) [01:18:57] :) [01:23:38] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:35:39] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-443] [01:46:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6zg [01:46:05] [02miraheze/dns] 07paladox 032d709ad - Remove lizardfs[45] from dns [01:53:56] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [02:25:27] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis] [02:26:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [02:29:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:30:51] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:31:17] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:31:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [02:32:52] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.417 second response time [02:33:13] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.632 second response time [02:33:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:33:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [02:55:26] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-443] [03:52:41] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:26:40] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2672 MB (11% inode=94%); [06:54:42] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [07:15:31] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [07:15:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:17:39] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [07:20:14] Testing relay [07:20:30] Test done [07:20:38] Still working [07:22:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:31:04] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 51651 MB (10% inode=99%); [10:03:22] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:05:36] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 42074 MB (8% inode=99%); [10:16:33] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [10:16:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [10:20:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.08, 6.90, 6.11 [10:20:34] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:21:04] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[panadev.ir_private] [10:24:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [10:24:56] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:42:13] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.34, 6.69, 6.70 [10:52:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [10:52:44] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:53:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [10:55:01] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 6.494 second response time [10:58:10] !log reboot mw3 [10:58:19] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:59:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:59:24] !log reboot mw3 [10:59:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:00:17] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.398 second response time [11:01:12] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [11:03:09] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [11:03:14] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:03:57] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:31:03] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: DISK CRITICAL - free space: / 28413 MB (5% inode=99%); [13:57:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [13:58:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [14:04:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:05:08] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:18:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:18:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [14:21:00] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:23:00] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 5.430 second response time [14:26:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:26:42] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:28:41] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:28:42] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 1.141 second response time [14:32:36] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [14:32:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [14:35:26] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:36:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:36:54] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:37:23] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.393 second response time [15:02:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [15:04:35] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:08:41] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [15:08:41] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [15:10:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:14:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:21:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [15:21:29] Hello IRCCloud! If you have any questions, feel free to ask and someone should answer soon. [15:22:26] Hello ex20sjd! If you have any questions, feel free to ask and someone should answer soon. [15:22:52] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:53] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:23:40] Hello Wolf20482! If you have any questions, feel free to ask and someone should answer soon. [15:24:06] Hi [15:24:10] I'm Wolf [15:24:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [15:24:28] hmmm [15:24:37] someone is impersonating me [15:24:59] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 8.134 second response time [15:24:59] Wolf20482: in what way? [15:25:00] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 7.919 second response time [15:25:00] https://usercontent.irccloud-cdn.com/file/Ll461tKs/Screenshot_20191121-222448_IRCCloud.jpg [15:25:09] who is this guy [15:25:35] That's coming from your IRC Cloud accouunt [15:25:58] no [15:26:04] someone is using irccloud [15:26:28] You mean you're not in control of the account? [15:26:31] joking [15:26:39] I forgot my password [15:26:43] So I make a new one [15:26:47] Okay [15:26:54] Please keep this on topic [15:27:12] We have #miraheze-offtopic but let's keep it reasonably sensible [15:27:15] Ok sorry [15:28:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:29:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:30:22] know a simpler template to use than this template? has too many calls to other templates and modules that in turn call others. And that causes errors. https://commons.wikimedia.org/wiki/Template:Autotranslate [15:30:22] [ Template:Autotranslate - Wikimedia Commons ] - commons.wikimedia.org [15:33:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:34:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:35:44] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:36:14] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:37:40] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [15:37:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:41:35] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:42:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:43:05] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:43:42] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 8.973 second response time [15:45:09] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 5.998 second response time [15:50:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:57:46] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:58:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:00:11] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:35] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [16:01:49] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.390 second response time [16:02:13] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 0.641 second response time [16:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6D2 [16:10:10] [02miraheze/services] 07MirahezeSSLBot 0373dba93 - BOT: Updating services config for wikis [16:10:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:11:38] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:11:58] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:50] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24640 bytes in 4.455 second response time [16:14:05] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 7.569 second response time [16:14:52] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:15:09] RhinosF1: https://phabricator.miraheze.org/T4922#93520 :P [16:15:11] [ ⚓ T4922 can you please enable the visual editor and infobox ? ] - phabricator.miraheze.org [16:15:50] !log on mw2 access files were owned by nginx:adm, changed to www-data:adm [16:16:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:18:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [16:18:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:25:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:26:11] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6Dx [16:26:12] [02miraheze/puppet] 07JohnFLewis 03d4ae15c - board alias; req request of secretary [16:29:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:32:30] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [16:33:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [16:34:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:35:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:42:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:42:50] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:51:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:55:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [16:59:56] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1353 bytes in 0.025 second response time [17:00:55] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:09] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:54] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:02:03] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19047 bytes in 6.571 second response time [17:02:22] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [17:02:50] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 0.643 second response time [17:03:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:04:46] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [17:05:40] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19046 bytes in 6.932 second response time [17:06:14] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.743 second response time [17:06:27] RhinosF1: ok [17:06:40] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:08:43] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 6.669 second response time [17:10:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6S4 [17:10:38] [02miraheze/puppet] 07paladox 0310c311c - Revert "varnish: Remove lizardfs6" This reverts commit 4f8c2ed05a6705d04fc4c21e0131eaab2c62ca59. [17:14:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:17:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [17:18:14] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [17:19:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:20:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [17:20:57] !log depool mw2 temporarily [17:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:07:31] JohnLewis: some of our mail aliases are double redirects, any reason? [19:23:03] I still don't see the new message in https://commons.miraheze.org/wiki/Special:Upload Do you really see it? [19:23:03] [ 503 Backend fetch failed ] - commons.miraheze.org [19:24:04] Oh, when the wiki's working. :D [19:29:11] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[glusterfs.pem] [19:29:12] :/ [19:29:31] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/nginx/fastcgi_params],File[private.revi.wiki_private],File[wiki.dwplive.com_private] [19:29:46] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20170718/luasandbox.so],File[/usr/lib/php/20180731/tideways_xhprof.so] [19:31:59] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [19:33:10] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [19:33:36] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:33:55] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:47:16] Still not updated after cleaning https://commons.miraheze.org/wiki/MediaWiki:Uploadtext https://commons.miraheze.org/wiki/Special:Upload [19:47:20] [ MediaWiki:Uploadtext - Miraheze Commons ] - commons.miraheze.org [19:47:22] [ Login required - Miraheze Commons ] - commons.miraheze.org [19:48:45] hispano76: has for me [19:52:42] ? [19:53:37] You see? RhinosF1 [19:57:00] It looks updated to me [20:06:00] I still get the pre-terminated message even after deleting all the content in my browser and restoring settings. https://usercontent.irccloud-cdn.com/file/tD51wrKx/hhhhhhhhhhh.png [20:06:07] RhinosF1: [20:06:48] hispano76: that’s a translated version though, is the translation up to date? [20:07:45] It's got nothing to do [20:10:30] I’ll look [20:11:49] the contents of the page are being grabbed from https://commons.miraheze.org/wiki/MediaWiki:Uploadtext/es [20:11:51] [ MediaWiki:Uploadtext/es - Miraheze Commons ] - commons.miraheze.org [20:13:05] Voidwalker: what page shows a list of translatable pages and let’s you remove/mark them for translation [20:14:27] Found it [20:14:31] i got it RhinosF1 [20:15:12] Special:PageTranslation or something [20:15:18] hispano76: fixing [20:15:24] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_checkout_landing],Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [20:15:28] Voidwalker: surprisingly yes [20:15:55] shouldnt be in the page right? [20:16:14] RhinosF1: Voidwalker [20:17:26] Zppix: I’m fixing it now [20:18:08] RhinosF1: you legit just did what i had ready to save just was making sure :P [20:18:33] hum, Um, yesterday I didn't see it when I changed the language. [20:19:15] hispano76: it should be translateable now [20:19:34] https://commons.miraheze.org/w/index.php?title=Special:Translate&group=page-MediaWiki%3AUploadtext&action=page&filter=&language=es [20:19:35] [ Translate - Miraheze Commons ] - commons.miraheze.org [20:19:43] and Repool, hum [20:20:52] Sorry, that wasn't updated either when the repooled thing happened. [20:21:16] All right, thanks you. [20:21:27] if you translate it i make sure it updates [20:22:21] Zppix: it wasn’t remarked for translation [20:23:53] Translating and the translation is automatically approved. ;) [20:24:33] hispano76: the es version looks fine now :) [20:24:34] hispano76: good [20:24:42] Take advantage of Spam -> know a simpler template to use than this template? has too many calls to other templates and modules that in turn call others. And that causes errors. https://commons.wikimedia.org/wiki/Template:Autotranslate [20:24:42] [ Template:Autotranslate - Wikimedia Commons ] - commons.wikimedia.org [20:26:59] Anything from wikimedia is that way hispano76 [20:27:44] I know but I think I'd seen a simpler one but I don't remember. [20:28:04] Oh [20:32:40] Hey everyone, Miraheze has an exciting announcement: We’re now an official not-for-profit in the UK. See https://meta.miraheze.org/wiki/Incorporation for more information. [20:32:42] [ Incorporation - Miraheze Meta ] - meta.miraheze.org [20:38:18] oh [20:41:34] Zppix: because that's how stuff goes it seems, It's not too big of an issue really [20:43:39] Voidwalker: I translated the ad [20:46:35] JohnLewis: the RFC you suggested I should do in Meta or Commonswiki? [20:46:53] I'm already confused. :( [20:48:48] hispano76: Commonswiki preferably, but since it has an impact on every wiki (in theory), you could do it on meta and states participation on the wiki itself would be low so you'd prefer to do it on meta [20:58:00] ok [21:23:05] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [21:34:58] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-enable],Package[php7.3-apcu],Package[php7.3-redis],Service[rsyslog] [21:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je65D [21:40:08] [02miraheze/services] 07MirahezeSSLBot 039b4a537 - BOT: Updating services config for wikis [21:42:58] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [22:02:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6de [22:02:48] [02miraheze/puppet] 07paladox 035ce24eb - roundmail: Update to 1.4.0 stable [22:04:50] hey dross [22:05:00] yo [22:05:39] dross: just here to hang round or anything we can help with? [22:05:47] Mostly just hanging around [22:06:11] okay, have you seen our big announcement on meta [22:06:31] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis] [22:06:57] I have. Some things of concern to me, but we'll see if I can even get my thoughts together enough to provide input on the RfC [22:07:22] dross: ask away [22:07:52] I haven't opened the Community Directors RfC yet officially so now is the best time [22:08:06] In short, a lot of it seems very bureaucratic [22:08:34] I've been a part of several nonprofits before, and the whole closed door model always scares me [22:08:57] dross: We will attempt to be as transparent as possible [22:10:12] Well, part of it is the fact that the number of board appointments is far higher than that of community members as a whole [22:10:58] That would be fine with me, assuming some other empowered community representative existed, like an ombudsman [22:11:11] the max number of board members is only 9 people, we have over 92,000 users [22:11:30] Instead, it feels a lot like a clique that is very difficult to communicate with or be a part of [22:12:07] Sorry, I should clarify. Maximum 5 appointments vs maximum 3 community doesn't feel comfortable [22:12:12] I believe you know about Miraheze Meetings, our idea behind that is the community trustees can chat directly with the community in real time [22:12:28] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [22:12:41] I'd have to ask Owen about where 5 appointed v 3 community comes from [22:12:52] Hopefully [22:13:08] What is the accountability method though? [22:13:25] The community itself has zero powers, especially without membership [22:13:30] s/trustees/directors [22:13:30] RhinosF1 meant to say: I believe you know about Miraheze Meetings, our idea behind that is the community directors can chat directly with the community in real time [22:14:12] Community Directors are accountable to the community and with valid reason can be removed [22:14:26] In theory [22:14:32] They can [22:14:34] There is no guarantee [22:14:45] Nothing in AoA displays that has legal standing [22:15:30] If we had valid reason, then we'd ask Owen to present a resolution to remove them which would be voted on and is legally binding if the board approve it. [22:15:53] Why shouldn't it exist from the start? [22:16:14] It can even be a pre-defined list of reasons for the community to be empowered to unseat a director [22:16:55] The only reason the community can't just remove someone is board members by law can only be removed and appointed by the board as far as I know. [22:17:25] Hm... [22:17:37] Well, I will admit my knowledge lies only with US nonprofits [22:17:44] If we asked for it, with valid reasons, I see no reason why we wouldn't be working with the board throughout to ensure it is enacted properly [22:18:27] I know that USA law empowers anybody with membership and voting as a right to membership can vote on matters such as election, appointment, or vacation [22:18:41] Members can even be empowered to make motions [22:18:54] Under the AoA, Membership is only given to Directors [22:19:04] That's my point precisely [22:19:35] that's something you'd have to ask Owen who I will poke again and is waiting on Discord to answer Questions [22:21:02] 5 v 3 is arbitrary [22:21:38] and Membership is as it-is currently just to get things rolling. There's a lot of pressure to get things done without any time to actually do things [22:21:53] How easy are amendments then? [22:22:02] Extremely [22:22:06] It's just a resolution [22:24:02] Good. I ask mostly because nonprofits in America tend to be difficult to modify. It also seems that AoA behaves more like bylaws than Articles of Incorporation. In California at least, amendments to the AoI need to fit certain conditions and be approved by the Secretary of State before they legally take effect. [22:24:43] Not in the UK, AoA are essentially bylaws here [22:26:29] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ops_ensure_members],Exec[mediawiki-admins_ensure_members],Service[nginx],Service[nginx-prometheus-exporter] [22:29:52] https://phabricator.miraheze.org/T4918 This has me confused. [22:29:53] [ ⚓ T4918 Problems setting up my headings ] - phabricator.miraheze.org [22:52:06] .status mhmeta off [22:52:08] RhinosF1: Updating User:RhinosF1/Status to off! [22:52:16] RhinosF1: Updated! [22:53:10] .status mhmeta Off [22:53:12] RhinosF1: Updating User:RhinosF1/Status to Off! [22:53:20] RhinosF1: Updated! [23:35:29] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [23:35:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [23:35:51] PROBLEM - mw2 Disk Space on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:36:04] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:36:23] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:05] PROBLEM - mw2 SSH on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:37:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [23:37:16] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:23] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [23:37:24] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [23:37:32] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: connect to address 185.52.2.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:37:38] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24656 bytes in 0.502 second response time [23:37:53] RECOVERY - mw2 Disk Space on mw2 is OK: DISK OK - free space: / 57125 MB (70% inode=94%); [23:38:03] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 0.00, 0.00, 0.00 [23:38:20] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures [23:39:00] RECOVERY - mw2 SSH on mw2 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u1 (protocol 2.0) [23:39:00] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [23:39:16] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.004 second response time [23:39:19] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24662 bytes in 0.667 second response time [23:39:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [23:39:27] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.007 second response time [23:39:41] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:41:25] !log repool mw2 [23:41:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:42:39] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [23:46:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Je6FX [23:46:28] [02miraheze/puppet] 07paladox 037709576 - varnish: Remove lizardfs6 This reverts commit 10c311cc33f79fb9afb4efe066e6280708e1adb2.