[00:03:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [00:04:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:07:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:17:20] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 4.88, 11.03, 15.74 [00:21:33] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.32, 7.35, 13.07 [00:21:46] [02mw-config] 07paladox closed pull request 03#2814: Move all wikis that start with the letter s to the new mount - 13https://git.io/JeowR [00:21:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo6y [00:21:49] [02miraheze/mw-config] 07paladox 031665479 - Move all wikis that start with the letter s to the new mount (#2814) [00:22:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [00:22:41] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:27:28] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:30] https://meta.miraheze.org/wiki/Community_noticeboard#Define_license_policie_for_Miraheze_Commons Participate! [00:38:32] [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org [00:40:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jeoiv [00:40:30] [02miraheze/mw-config] 07paladox 03fd015c6 - Move all wikis that start with the letter t to the new mount [00:40:31] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:40:33] [02mw-config] 07paladox opened pull request 03#2815: Move all wikis that start with the letter t to the new mount - 13https://git.io/Jeoif [00:42:51] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf s* [00:43:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:45:31] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 25.18, 16.26, 11.69 [00:49:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeoim [00:49:03] [02miraheze/puppet] 07paladox 03bd25025 - Update mfsmaster.cfg.erb [00:54:12] !log restart lizardfs-master on lizardfs6 [00:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:57:52] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 8.81, 15.13, 15.40 [01:01:49] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 19.36, 15.41, 15.31 [01:03:58] !log restarted lizardfs-chunkserver on lizardfs6 [01:04:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 9.05, 13.72, 14.77 [01:04:58] !log restart lizardfs-cgiserv on lizardfs6 [01:05:37] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4112 bytes in 0.021 second response time [01:06:09] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:06:19] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.46, 10.18, 13.31 [01:07:36] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19026 bytes in 0.559 second response time [01:08:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [01:12:24] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 21.82, 17.42, 15.23 [02:23:27] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:24:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:26:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:28:02] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24681 bytes in 0.699 second response time [02:28:33] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:28:51] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:31:25] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 19013 bytes in 0.479 second response time [02:32:48] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:34:45] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19020 bytes in 0.957 second response time [03:28:13] [02mw-config] 07paladox closed pull request 03#2815: Move all wikis that start with the letter t to the new mount - 13https://git.io/Jeoif [03:28:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoP9 [03:28:16] [02miraheze/mw-config] 07paladox 0327141bf - Move all wikis that start with the letter t to the new mount (#2815) [03:28:17] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [03:28:19] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [03:31:54] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 8.20, 9.88, 15.85 [03:32:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeoPQ [03:32:51] [02miraheze/mw-config] 07paladox 030be8605 - Move all wikis that start with the letter u to the new mount [03:32:53] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [03:32:54] [02mw-config] 07paladox opened pull request 03#2816: Move all wikis that start with the letter u to the new mount - 13https://git.io/JeoP7 [03:33:50] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.91, 12.72, 16.19 [04:57:01] Hello gaddman! If you have any questions, feel free to ask and someone should answer soon. [05:00:44] Is there some work going on at the moment? I notice in Special:ListFiles that the links have changed from static.miraheze to static-temp.miraheze [05:01:11] are those links functional? [05:02:19] so I get 404 for files I've previously linked at static.miraheze.org/wikiname/n/nn/filename.png, but if I go to static-temp it's available [05:03:19] well, images should not be on static-temp for very long, this should only be a temporary issue [05:03:56] figured as much. Is this the right place to watch for notifications of that sort of thing? [05:04:26] for now, yes [05:04:33] cool, thanks a lot [05:04:45] although we'll probably post an announcement about it when it's done [06:27:25] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2724 MB (11% inode=94%); [07:25:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 14.31, 14.75, 15.96 [07:33:16] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.48, 16.35, 16.11 [07:37:14] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 11.31, 15.17, 15.82 [07:43:15] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.89, 16.30, 16.10 [07:45:12] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 15.81, 15.87, 15.95 [07:47:09] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 17.47, 16.34, 16.10 [07:57:04] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.13, 15.09, 15.81 [07:59:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 15.75, 16.22, 16.19 [08:00:23] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [08:01:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 13.07, 14.89, 15.70 [08:03:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 24.66, 17.19, 16.35 [08:07:03] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 12.53, 15.35, 15.82 [08:13:03] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.49, 8.63, 12.86 [09:40:37] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki testwiki /home/reception/Wikipedia-20191116181322.xml on mw3 [09:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [09:42:25] !log add vnederbotwiki webhook [09:42:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:00:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo94 [12:00:09] [02miraheze/services] 07MirahezeSSLBot 03388ee9c - BOT: Updating services config for wikis [14:18:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jeo7a [14:18:02] [02miraheze/mw-config] 07paladox 036b29904 - Switch things around [14:18:04] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [14:18:05] [02mw-config] 07paladox opened pull request 03#2817: Switch things around - 13https://git.io/Jeo7V [14:18:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/Jeo7w [14:18:41] [02miraheze/puppet] 07paladox 0361bedaf - mediawiki: Switch things around [14:18:43] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [14:18:44] [02puppet] 07paladox opened pull request 03#1148: mediawiki: Switch things around - 13https://git.io/Jeo7r [14:20:18] [02mw-config] 07paladox synchronize pull request 03#2817: Switch things around - 13https://git.io/Jeo7V [14:20:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jeo76 [14:20:21] [02miraheze/mw-config] 07paladox 032bb0531 - Update LocalSettings.php [14:20:59] [02mw-config] 07paladox synchronize pull request 03#2817: Switch things around - 13https://git.io/Jeo7V [14:21:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jeo7P [14:21:02] [02miraheze/mw-config] 07paladox 03b730113 - Update LocalSettings.php [14:21:16] [02puppet] 07paladox closed pull request 03#1148: mediawiki: Switch things around - 13https://git.io/Jeo7r [14:21:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo7M [14:21:19] [02miraheze/puppet] 07paladox 032e96712 - mediawiki: Switch things around (#1148) [14:22:58] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [14:22:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [14:23:10] [02mw-config] 07paladox closed pull request 03#2817: Switch things around - 13https://git.io/Jeo7V [14:23:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo7D [14:23:13] [02miraheze/mw-config] 07paladox 037a73bf1 - Switch things around (#2817) * Switch things around * Update LocalSettings.php * Update LocalSettings.php [14:27:33] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [14:27:34] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [14:28:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jeo7d [14:28:13] [02miraheze/mw-config] 07paladox 03e6ea03e - Move all wikis that start with the letter u to the new mount [14:28:14] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3 [14:28:16] [02mw-config] 07paladox opened pull request 03#2818: Move all wikis that start with the letter u to the new mount - 13https://git.io/Jeo7F [14:28:24] [02mw-config] 07paladox closed pull request 03#2818: Move all wikis that start with the letter u to the new mount - 13https://git.io/Jeo7F [14:28:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo7b [14:28:27] [02miraheze/mw-config] 07paladox 03fa5613f - Move all wikis that start with the letter u to the new mount (#2818) [14:28:43] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [14:28:45] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [14:29:13] miraheze/mw-config/paladox-patch-3/e6ea03e - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/613106442 [14:43:30] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf u* [14:43:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:47:08] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.87, 11.65, 6.36 [14:47:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:49:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:59:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeo5a [14:59:20] [02miraheze/mw-config] 07paladox 03b0035d5 - Fix [15:03:48] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:06:22] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24675 bytes in 0.639 second response time [15:11:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:13:29] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:16:48] PROBLEM - endlesssea.lucyawrey.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [15:22:49] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [15:24:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:32:22] Hello doctawho! If you have any questions, feel free to ask and someone should answer soon. [15:34:28] I'm working on a miraheze wiki page and trying to get collapsible elements to work for mobile devices. I found the following: https://en.wikipedia.org/wiki/Template:Collapse-mobile; however I don't know how to get it to work. Anyone have any tips? The reason I'm using collapsible elements is to hide spoilers. Is there a different way I can do that? [15:46:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeodn [15:46:22] [02miraheze/mw-config] 07paladox 03fe4088e - Move all wikis that start with the letter v to the new mount [15:48:42] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf v* [15:48:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:50:44] doctawho: it sounds like it might be missing some of the css and js needed, which wiki are you working on? [15:51:15] this one: https://taintedgrailfaq.miraheze.org/wiki/Main_Page [15:51:16] [ Tainted Grail Wiki ] - taintedgrailfaq.miraheze.org [15:51:44] I'm new at this, so it may be missing stuff :). [15:52:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb [15:54:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [15:55:32] paladox, Reception123: do I still have the info box files in my homedir on mw1 [15:56:14] yes [15:56:15] drwxr-xr-x 2 rhinos rhinos 4.0K Sep 27 21:29 infoboxes [15:56:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:56:22] paladox: thx [15:56:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:56:53] doctawho: I should be able to have a look and get that working later. Although, I see you’ve got some css added already [15:57:33] RhinosF1 Thanks! Yeah, I just figured that part out. [16:00:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeodV [16:00:08] [02miraheze/services] 07MirahezeSSLBot 0317db503 - BOT: Updating services config for wikis [16:04:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [16:06:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:07:23] JohnLewis: https://meta.miraheze.org/wiki/Community_noticeboard#Define_license_policie_for_Miraheze_Commons Would any Sitenotice be necessary? Um, if you ask. I will open an RFC if necessary, when I have comments on the board. [16:07:25] [ Community noticeboard - Miraheze Meta ] - meta.miraheze.org [16:17:21] hispano76: it's a policy which affects only 1 wiki in theory, it's up to said wiki to decide. [16:17:43] I'm not involved in the administration of that wiki anyway in any regard so I'm not the best person to ask eitherway [16:19:39] JohnLewis: There isn't a community in Commons yet. That's why I decided to bring the topic to Meta. There are some editors but... it's not enough for me. JohnLewis [16:20:21] hispano76: currently, If it’s legal, we’ll keep it up [16:20:40] https://commons.miraheze.org/wiki/Special:ActiveUsers [16:20:43] [ Active users list - Miraheze Commons ] - commons.miraheze.org [16:20:48] Can RfC it if you want, but my last replies stands unfortunately - I'm not involved in any regard :/ [16:22:47] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:24:20] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [16:26:19] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:28:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:32:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:34:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:48:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoFR [16:48:07] [02miraheze/puppet] 07paladox 0321629c6 - Update config.yaml [16:48:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [16:51:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoF2 [16:51:13] [02miraheze/puppet] 07paladox 03c8a71bc - Update parsoid [16:53:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoFV [16:53:04] [02miraheze/mw-config] 07paladox 039748aa2 - Move all wikis that start with the letter w to the new mount [16:54:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoFP [16:55:09] [02miraheze/services] 07MirahezeSSLBot 0367c8d63 - BOT: Updating services config for wikis [16:58:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoFH [16:58:02] [02miraheze/puppet] 07paladox 037c6dc79 - Update mfsmaster.cfg.erb [16:58:48] !log restart lizardfs-master on lizardfs6 [16:58:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:00:49] Hello ppap! If you have any questions, feel free to ask and someone should answer soon. [17:01:48] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf w* [17:02:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:04:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeoFb [17:04:19] [02miraheze/mw-config] 07paladox 03aa75ebe - Move all wikis that start with the letter x to the new mount [17:05:21] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf x* [17:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:15:56] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:16:40] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:01] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb [17:20:06] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24659 bytes in 4.438 second response time [17:20:46] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24681 bytes in 5.853 second response time [17:22:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:04:35] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 6.89, 10.48, 14.92 [18:08:28] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 4.88, 7.20, 12.56 [21:03:47] [02mw-config] 07paladox closed pull request 03#2816: Move all wikis that start with the letter u to the new mount - 13https://git.io/JeoP7 [21:03:50] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [21:03:51] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [21:04:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeojT [21:04:14] [02miraheze/mw-config] 07paladox 03cfe5264 - Move all wikis that start with the letter y to the new mount [21:06:21] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf y* [21:06:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:20:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeojl [21:20:30] [02miraheze/mw-config] 07paladox 0387c6bbd - Move all wikis to new mount [21:21:16] !log root@lizardfs6:/mnt/mediawiki-static# rm -rf z* [21:21:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:22:15] Yey [21:22:31] Well done paladox [21:22:52] :) [21:27:10] [02miraheze/landing] 07paladox pushed 031 commit to 03revert-18-paladox-patch-1 [+0/-0/±1] 13https://git.io/Jeojr [21:27:11] [02miraheze/landing] 07paladox 03398c826 - Revert "Temporarily changing to static-temp (#18)" This reverts commit 7f6bbc3a48e01b276773206745cb62745113014c. [21:27:13] [02landing] 07paladox created branch 03revert-18-paladox-patch-1 - 13https://git.io/fxEQA [21:27:14] [02landing] 07paladox opened pull request 03#19: Revert "Temporarily changing to static-temp" - 13https://git.io/Jeojo [21:27:23] [02landing] 07paladox closed pull request 03#19: Revert "Temporarily changing to static-temp" - 13https://git.io/Jeojo [21:27:24] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeoj6 [21:27:26] [02miraheze/landing] 07paladox 0364ceb1d - Revert "Temporarily changing to static-temp (#18)" (#19) This reverts commit 7f6bbc3a48e01b276773206745cb62745113014c. [21:27:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeoji [21:27:39] [02miraheze/puppet] 07paladox 03aeb1c97 - Remove static-temp [21:27:48] paladox: one day I’ll buy everyone a drink [21:27:55] heh [21:28:07] miraheze/landing/revert-18-paladox-patch-1/398c826 - paladox The build passed. https://travis-ci.org/miraheze/landing/builds/613215552 [21:28:15] [02landing] 07paladox deleted branch 03revert-18-paladox-patch-1 - 13https://git.io/fxEQA [21:28:17] [02miraheze/landing] 07paladox deleted branch 03revert-18-paladox-patch-1 [21:28:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeojM [21:28:57] [02miraheze/puppet] 07paladox 032e666c4 - Stop installing lizard on lizardfs6 [21:29:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [21:30:05] paladox: how many of use even drink? [21:30:11] s/use/us [21:30:12] RhinosF1 meant to say: paladox: how many of us even drink? [21:30:25] if you mean alcohol then i'm out :P [21:30:36] i doin't like alohol so i doin't drink it :) [21:30:39] *alcohol [21:30:41] JohnLewis ^? [21:30:42] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:30:51] paladox: I don't either so it'll be a cheap night! [21:30:57] lol [21:31:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeoj9 [21:31:07] [02miraheze/puppet] 07paladox 034489cad - mediawiki: Stop installing lizardfs-client on mw* [21:31:24] My age is a factor but last time I had a sip I threw up everywhere so [21:31:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:32:44] I'm off for the night [21:32:57] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 22.11, 15.55, 11.88 [21:33:32] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 19015 bytes in 1.879 second response time [21:34:54] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 8.14, 12.43, 11.19 [21:35:43] ssh!log depool mw3, reboot and repool [21:36:58] !log depool mw3, reboot and repool [21:37:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:38:00] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:38:23] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [21:38:47] PROBLEM - mw3 JobRunner Service on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:38:49] PROBLEM - mw3 SSH on mw3 is CRITICAL: connect to address 81.4.121.113 and port 22: Connection refused [21:38:55] PROBLEM - mw3 Puppet on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:39:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:39:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:39:49] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [21:39:51] PROBLEM - mw3 HTTPS on mw3 is CRITICAL: connect to address 81.4.121.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:39:52] PROBLEM - mw3 php-fpm on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:39:54] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [21:39:54] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: connect to address 81.4.121.113 port 5666: Connection refusedconnect to host 81.4.121.113 port 5666: Connection refused [21:39:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [21:40:03] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [21:40:23] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.006 second response time [21:40:30] known ^ [21:40:46] RECOVERY - mw3 JobRunner Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:40:49] RECOVERY - mw3 SSH on mw3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u7 (protocol 2.0) [21:40:55] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [21:41:23] !log depool mw2, reboot and repool [21:41:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:42:18] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.692 second response time [21:42:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [21:43:16] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.75, 1.91, 0.71 [21:43:34] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 25740 MB (33% inode=98%); [21:43:35] !log depool mw1, reboot and repool [21:43:47] RECOVERY - mw3 php-fpm on mw3 is OK: PROCS OK: 7 processes with command name 'php-fpm7.2' [21:43:48] RECOVERY - mw3 HTTPS on mw3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 442 bytes in 0.007 second response time [21:43:50] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [21:43:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [21:43:59] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24655 bytes in 4.380 second response time [21:44:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:44:24] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:44:27] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:05] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [21:46:35] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.364 second response time [21:46:39] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.368 second response time [21:46:59] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [21:47:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:47:49] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 6 backends are healthy [21:48:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:51:53] !log remove lizardfs* from lizardfs6 [21:52:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:53:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKeR [21:53:30] [02miraheze/puppet] 07paladox 03e387b3f - Remove lizard.miraheze.org [21:55:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKeu [21:55:58] [02miraheze/puppet] 07paladox 03806fc25 - Temporarily remove lizardfs6 [21:58:30] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:59:34] PROBLEM - lizardfs6 Current Load on lizardfs6 is CRITICAL: CRITICAL - load average: 16.21, 20.52, 13.71 [21:59:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:59:50] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4108 bytes in 0.115 second response time [22:01:02] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:01:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:01:48] PROBLEM - lizardfs6 Current Load on lizardfs6 is WARNING: WARNING - load average: 3.08, 13.84, 12.13 [22:02:13] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 19019 bytes in 2.089 second response time [22:03:07] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 1.175 second response time [22:03:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKeD [22:03:14] [02miraheze/puppet] 07paladox 0368a50ab - Revert "Temporarily remove lizardfs6" This reverts commit 806fc25e1296dfcd110776475f8f9a923ef32feb. [22:04:07] RECOVERY - lizardfs6 Current Load on lizardfs6 is OK: OK - load average: 2.57, 9.62, 10.78 [22:05:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:05:20] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19013 bytes in 0.995 second response time [22:06:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:13:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeKeA [22:13:20] [02miraheze/puppet] 07paladox 03c882692 - Remove old nodes and vpncloud from all hosts [22:16:20] PROBLEM - lizardfs6 Lizardfs Chunkserver Port on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 9422: Connection refused [22:16:41] PROBLEM - lizardfs6 Lizardfs Master Port 3 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 9421: Connection refused [22:17:07] !log revoke lizardfs[45] puppet cert, remove from salt (deleting salt key) and delete instance [22:17:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:17:21] PROBLEM - lizardfs6 Lizardfs Master Port 2 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 9420: Connection refused [22:17:41] PROBLEM - lizardfs6 Lizardfs Master Port 1 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 9419: Connection refused [22:19:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeKvO [22:19:45] [02miraheze/puppet] 07paladox 03589ecd0 - mediawiki-static-new -> mediawiki-static [22:19:46] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [22:19:48] [02puppet] 07paladox opened pull request 03#1149: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv3 [22:22:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeKvc [22:22:27] [02miraheze/puppet] 07paladox 03c8477d9 - Update mediawiki.pp [22:22:29] [02puppet] 07paladox synchronize pull request 03#1149: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv3 [22:22:42] PROBLEM - lizardfs5 SSH on lizardfs5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeKvC [22:23:02] [02miraheze/puppet] 07paladox 03695fe7e - Update init.pp [22:23:03] [02puppet] 07paladox synchronize pull request 03#1149: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv3 [22:23:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JeKvW [22:23:20] [02miraheze/puppet] 07paladox 033f45f0d - Update mediawiki.conf [22:23:21] PROBLEM - lizardfs4 SSH on lizardfs4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:22] [02puppet] 07paladox synchronize pull request 03#1149: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv3 [22:23:27] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:34] PROBLEM - lizardfs5 Disk Space on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:40] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:55] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeKv8 [22:24:00] [02miraheze/mw-config] 07paladox 0386be20e - mediawiki-static-new -> mediawiki-static [22:24:02] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:24:03] [02mw-config] 07paladox opened pull request 03#2819: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv4 [22:24:12] PROBLEM - Host lizardfs5 is DOWN: PING CRITICAL - Packet loss = 100% [22:24:27] PROBLEM - lizardfs4 Disk Space on lizardfs4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:24:39] PROBLEM - Host lizardfs4 is DOWN: PING CRITICAL - Packet loss = 100% [22:24:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeKvR [22:24:47] [02miraheze/mw-config] 07paladox 0365e7bd0 - Update LocalWiki.php [22:24:48] [02mw-config] 07paladox synchronize pull request 03#2819: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv4 [22:25:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeKv0 [22:25:16] [02miraheze/mw-config] 07paladox 03bfb4814 - Update LocalExtensions.php [22:25:17] [02mw-config] 07paladox synchronize pull request 03#2819: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv4 [22:25:25] [02puppet] 07paladox closed pull request 03#1149: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv3 [22:25:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JeKvE [22:25:28] [02miraheze/puppet] 07paladox 03d6aebe0 - mediawiki-static-new -> mediawiki-static (#1149) * mediawiki-static-new -> mediawiki-static * Update mediawiki.pp * Update init.pp * Update mediawiki.conf [22:25:40] [02mw-config] 07paladox closed pull request 03#2819: mediawiki-static-new -> mediawiki-static - 13https://git.io/JeKv4 [22:25:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JeKvu [22:25:43] [02miraheze/mw-config] 07paladox 0356e1279 - mediawiki-static-new -> mediawiki-static (#2819) * mediawiki-static-new -> mediawiki-static * Update LocalWiki.php * Update LocalExtensions.php [22:27:04] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:27:05] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [22:27:08] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [22:27:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [22:34:36] !log deleted lizardfs[45] from puppetdb [22:34:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:42:09] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:49:03] !log reboot test1 [22:49:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:54:08] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:03:17] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:23] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:24] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:30] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:03:40] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 2 minutes ago with 14 failures. Failed resources (up to 3 shown) [23:03:48] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 2 minutes ago with 17 failures. Failed resources (up to 3 shown) [23:04:17] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 229 failures. Last run 2 minutes ago with 229 failures. Failed resources (up to 3 shown) [23:04:18] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 233 failures. Last run 2 minutes ago with 233 failures. Failed resources (up to 3 shown) [23:04:19] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:26] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:30] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 226 failures. Last run 3 minutes ago with 226 failures. Failed resources (up to 3 shown) [23:04:31] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 3 minutes ago with 31 failures. Failed resources (up to 3 shown) [23:04:37] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 212 failures. Last run 3 minutes ago with 212 failures. Failed resources (up to 3 shown) [23:04:38] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [23:04:42] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:46] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:04:52] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 3 minutes ago with 15 failures. Failed resources (up to 3 shown) [23:13:21] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:13:24] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [23:13:26] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [23:13:48] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:14:07] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:14:12] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:14:31] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:35] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [23:14:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [23:14:43] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [23:14:45] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:52] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:14:54]