[00:03:31] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:07] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.48, 6.78, 5.81 [00:15:08] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 7.61, 7.08, 6.04 [00:17:07] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.85, 7.76, 6.41 [00:19:05] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.40, 6.89, 6.25 [00:21:03] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.96, 6.72, 6.27 [00:25:54] !log rsync /mnt/test (lizardfs) to /mnt/mediawiki-static (gluster) on test1 [00:25:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:28:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.17, 1.80, 1.36 [00:33:53] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 9.08, 7.94, 6.97 [00:37:48] PROBLEM - mw2 Current Load on mw2 is WARNING: WARNING - load average: 5.94, 7.81, 7.21 [00:41:45] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.57, 5.96, 6.57 [01:00:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.86, 1.90, 1.96 [01:10:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.74, 2.25, 2.06 [01:22:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.90, 1.92, 1.98 [01:24:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.02, 1.96, 1.98 [01:26:55] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.83, 1.96, 1.98 [01:38:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.01, 1.92, 1.92 [01:42:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.87, 1.97, 1.94 [01:48:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.40, 2.12, 2.00 [01:54:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.90, 1.98, 1.97 [01:58:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.04, 2.01, 1.98 [02:02:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.56, 1.86, 1.94 [02:06:55] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 1.91, 2.01, 1.98 [02:08:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.88, 1.98, 1.97 [02:10:55] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.16, 2.02, 1.98 [02:24:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.76, 2.00, 2.00 [02:28:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.32, 2.10, 2.02 [02:34:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.55, 1.91, 1.98 [02:46:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.11, 1.90, 1.90 [02:47:28] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 10.37, 7.16, 5.48 [02:48:34] paladox: expected/ [02:48:38] ?* [02:48:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.75, 1.82, 1.86 [02:49:25] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 5.51, 6.45, 5.43 [02:50:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.14, 1.92, 1.89 [02:54:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.60, 1.90, 1.89 [02:58:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.23, 1.97, 1.91 [03:06:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.94, 1.98, 1.96 [03:20:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.07, 1.82, 1.84 [04:24:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.75, 1.82, 1.94 [04:30:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.22, 1.99, 1.96 [04:32:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.81, 1.90, 1.93 [04:40:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.02, 1.86, 1.86 [04:42:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.76, 1.84, 1.85 [04:50:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.25, 1.93, 1.86 [04:52:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.88, 1.91, 1.86 [04:58:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.16, 2.05, 1.92 [05:06:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.68, 1.98, 1.96 [05:14:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.96, 1.35, 1.69 [05:42:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:25:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3119 MB (12% inode=94%); [06:33:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:33:37] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:33:41] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:34:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [06:35:34] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:35:40] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 7.642 second response time [06:35:41] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.599 second response time [06:35:56] Reception123: ^ [06:37:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:37:16] Nothing I can do about that... [06:37:31] PROBLEM - mw6 SSH on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:37:36] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24500 bytes in 1.772 second response time [06:38:02] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 17.77, 8.07, 3.28 [06:38:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:38:28] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 6.35, 4.30, 1.87 [06:39:13] K [06:39:28] RECOVERY - mw6 SSH on mw6 is OK: SSH OK - OpenSSH_7.9p1 Debian-10 (protocol 2.0) [06:40:28] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 1.09, 2.93, 1.66 [06:43:48] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 0.18, 2.79, 2.39 [07:02:47] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [07:03:45] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:13:45] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:42:47] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [07:43:46] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 2 minutes ago with 12 failures. Failed resources (up to 3 shown) [08:22:34] Zppix: high load on test1 is expected [08:43:46] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:53:31] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:14:08] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:26:09] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:53:31] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:26:55] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.04, 1.79, 1.43 [10:28:55] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.67, 1.77, 1.46 [10:30:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.49, 1.66, 1.46 [11:42:40] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [11:43:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [11:43:17] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host [11:43:21] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: HTTP CRITICAL - No data received from host [11:44:40] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 3% [11:44:54] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:44:57] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [11:45:03] PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:45:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [11:45:13] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:45:15] PROBLEM - mw1 php-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [11:45:38] PROBLEM - Host mw1 is DOWN: PING CRITICAL - Packet loss = 100% [12:16:28] Um [12:16:30] Reception123: ^ [12:17:14] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24516 bytes in 3.589 second response time [12:17:48] Not bandwith this time [12:17:49] !log rebooted mw1 through the control panel (was showing offline) [12:17:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:18:02] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 1.232 second response time [12:18:28] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24522 bytes in 0.004 second response time [12:18:56] Reception123: your going to have to do the thing we do to get mw1 fully up [12:18:58] RECOVERY - Host mw1 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [12:19:13] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 2.41, 0.81, 0.28 [12:20:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jembj [12:20:09] [02miraheze/services] 07MirahezeSSLBot 0329e161e - BOT: Updating services config for wikis [12:23:41] :O it recovered! [13:09:44] paladox: damn i didnt break it hard enough [13:33:59] paladox: Ok, though what was "the thing we do"? [13:34:03] Not sure what you mean [13:42:58] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:55:01] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:02:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.77, 1.81, 1.53 [14:03:03] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:04:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.15, 1.57, 1.48 [14:15:05] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [14:42:47] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:13:24] should there be a global notice for the rfc, considering its scope is global? [15:13:36] Voidwalker: I'd say so [15:22:47] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [15:23:19] trying to make this translatable :P [15:33:27] Voidwalker: yeah, thought I'd give a day before [15:33:28] so will do soon [15:34:17] I'm already working on one [15:34:36] Voidwalker: oh okay then :D [15:36:17] RhinosF1: making an amendment to the warning proposal per The Pioneer [15:36:25] Reception123: k [15:39:19] how long should the banner last? [15:40:41] gonna put it up for a week, can adjust later [15:41:32] Voidwalker: that seems fine, though maybe a bit long IMO (though that's just my opinion) [15:41:52] well, it's up now [15:41:59] and should include a link to translate it [15:42:25] ok [15:42:30] great [15:45:53] for anyone who likes stats; [15:46:35] !log rhinos@test1 sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki oecumenewiki /home/rhinos/oecumene/ [15:46:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:47:49] Over the 24 hour period (September 11 from 02:00 UTC to September 12 02:00 UTC), Miraheze MediaWiki servers handled around 2 million requests (1.98 million), averaging around 82.5 thousand an hour, down to around 23 a second. Which meant each induvidual server handled just over 8 requests a second [15:48:17] JohnLewis: congratz? [15:48:34] JohnLewis: that's amazing [15:48:54] Zppix: just a stat flash :P [15:49:04] wow [15:49:28] JohnLewis: im happy to know i have “global admin” on a project with 1.98 requests i guess [15:50:54] !log import failed [15:51:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:51:11] RhinosF1: no need to log an import failed [15:51:22] * RhinosF1 will revert [15:51:54] just being through :P [15:54:09] * RhinosF1 has manually removed the failed log and repointed it to a working server [15:59:07] I suspect the import might be working now :P [16:00:00] fyi, if anyone translates the RfC notice, please be sure to ping me, as I have to approve changes before they go live [16:08:55] Voidwalker: moved to a new server [16:13:34] hello [16:15:46] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown) [16:20:05] Hi hispano76_ [16:23:27] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:25:52] Voidwalker approve my translation of the rfc global announcement [16:27:27] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:35:41] hispano76_, should be done, let me know if the changes don't show up for you [16:36:35] thanks, if I see them. Voidwalker [16:40:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 8 backends are healthy [16:41:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [16:41:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 8 backends are healthy [16:57:10] Hello ARZ100123! If you have any questions feel free to ask and someone should answer soon. [16:58:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw5 [16:58:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYed [16:58:27] [02miraheze/puppet] 07paladox 0304c7ac0 - Update config.yaml [16:58:32] So this is pretty much a live help page? [16:58:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYeF [16:58:36] [02miraheze/puppet] 07paladox 035cb448c - Update storage_firewall.yaml [16:59:11] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [16:59:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw5 [16:59:28] ARZ100123, in a way, although it's used for a bit more than that [16:59:50] general chat, service alerts, notices for code changes, etc [17:00:47] @+ Voidwalker Cool. [17:02:14] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 3 failures [17:03:46] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:13] Hi o/ I'd like to have country flag icons in a Miraheze wiki. I tried to export Template:Flagicon from Wikipedia and import it to our wiki, but now it just shows "Template:Country data ". What else should I do or is it even possible in Miraheze? I played around with it in a local Mediawiki installation before importing it in Miraheze, and it worked. The versions are not exactly the [17:05:19] same, though, so that may be one issue. [17:06:54] weird, ideally templates on one wiki should be functional on others [17:07:22] did you use any specific extensions, modules, or gadgets? [17:07:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.29, 2.98, 1.60 [17:10:51] Voidwalker: I didn't create the wiki but the creator gave me Administrator rights for this stuff. But I don't think there are any special stuff. [17:11:14] I exported this one https://en.wikipedia.org/wiki/Template:Flagicon [17:11:15] [WIKIPEDIA] Template:Flagicon | "..." [17:11:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.37, 3.54, 2.17 [17:11:55] now I wonder whether I should have exported the one in Wikimedia Commons instead.. I don't really know about the differences https://commons.wikimedia.org/wiki/Template:Flagicon [17:11:55] [ Template:Flagicon - Wikimedia Commons ] - commons.wikimedia.org [17:12:07] oh, I had meant if you had used any extensions or the like in order to get it to work on your local instance [17:13:36] Voidwalker: oh, yes, I did (I also had to upgrade my virtual Debian to Buster to get newer versions). I installed Scribunto and TemplateStyles. Both of those seem to be installed in the Miraheze wiki, according Special:Version. [17:14:24] however, from the looks of things, Template:Flagicon works by calling Template:Country_data_, which must exist in order to do anything [17:15:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.14, 3.07, 2.32 [17:15:58] hmm, maybe I used the Wikimedia Commons version? I remember trying both of those so maybe I mixed up.. [17:16:13] in the local installation, that is [17:17:27] yeah, that one is much more simple, although every country does need to be hard-coded into the template [17:18:03] that's what I thought as well, but it seems to have "= Flag of {{{1|}}}" in the end [17:18:22] oh, ok, that simplifies things :) [17:19:52] blah, looks like I really used the Wikimedia Commons one in the local wiki, now that I look at the source. So if I export that to Miraheze, I guess it works. But first I probably need to delete the stuff I imported, or does importing override? [17:19:53] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.49, 3.78, 2.77 [17:20:34] Tegu: probably best to delete what you imported to not mess up histories [17:20:55] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.88, 1.65, 1.35 [17:21:47] RhinosF1: yea, that's what I thought as well. Let's see how to delete those. And actually it doesn't seem to override! And that's why I thought I had the Wikipedia one working, as I had imported it _after_ importing the Wikimedia one. [17:22:48] any quick way to undo last import? [17:22:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.31, 1.55, 1.35 [17:24:44] of course, I can open them one by one and hit "Delete" :D [17:25:03] if there aren't too many, that's probably your best option [17:25:22] 40 [17:25:31] due to the dependencies [17:25:44] so, doable [17:27:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [17:27:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 8 backends are healthy [17:27:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 0.85, 2.99, 3.01 [17:28:14] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki core],Exec[/mnt/mediawiki-static] [17:31:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [17:31:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw6 [17:55:36] finally deleted all of them (and accidentally one module that was needed by another template). Now it works, thank you! [17:57:09] cheers, and good luck with your project [18:01:35] thanks, I'm quite a newbie with wikis. And oops, now Recent changes is full of my (Import log) and (Deletion log) stuff. :D But it's fine for transparency. [18:03:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [18:03:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 8 backends are healthy [18:04:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 8 backends are healthy [18:19:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYf3 [18:19:12] [02miraheze/puppet] 07paladox 0360ee1f3 - nodejs: Ping nodejs to the official nodejs apt repo This will also install npm. [18:20:59] okay [18:21:03] wrong window :P [18:26:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYfl [18:26:22] [02miraheze/puppet] 07paladox 03c2c8eac - Update init.pp [18:33:43] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:33:45] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:33:55] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:02] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:05] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:26] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:34:39] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:38:48] PROBLEM - browndust.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'browndust.wiki' expires in 15 day(s) (Sat 28 Sep 2019 06:36:24 PM GMT +0000). [18:39:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYfw [18:39:04] [02miraheze/ssl] 07MirahezeSSLBot 03fbeec4b - Bot: Update SSL cert for browndust.wiki [18:42:49] RECOVERY - browndust.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'browndust.wiki' will expire on Wed 11 Dec 2019 05:38:57 PM GMT +0000. [19:09:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.26, 2.35, 1.11 [19:09:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.53, 2.39, 1.11 [19:11:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.47, 2.85, 1.44 [19:13:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.97, 3.37, 1.84 [19:15:38] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 3.03, 3.19, 1.96 [19:17:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.25, 3.43, 2.12 [19:19:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.96, 3.26, 2.22 [19:27:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.71, 3.47, 2.76 [19:28:42] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.70, 3.58, 2.82 [19:30:39] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.01, 3.80, 3.00 [19:32:35] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.93, 3.79, 3.09 [19:34:32] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.37, 4.01, 3.25 [19:35:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [19:36:29] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.07, 3.51, 3.15 [19:37:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.40, 3.99, 3.36 [19:37:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [19:40:23] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.89, 3.38, 3.19 [19:41:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.34, 3.92, 3.51 [19:44:13] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.74, 3.82, 3.39 [19:46:10] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.73, 3.78, 3.43 [19:47:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 5.16, 4.04, 3.65 [19:49:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.16, 3.75, 3.60 [19:52:01] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.46, 4.11, 3.67 [19:52:20] Testing MH-Discord [19:52:44] Works (bot restarted) [19:55:55] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.89, 3.93, 3.70 [19:57:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.70, 4.04, 3.75 [19:57:52] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.42, 4.20, 3.83 [19:59:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.80, 3.98, 3.77 [19:59:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 2.92, 3.71, 3.69 [20:03:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.04, 3.76, 3.70 [20:05:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.41, 3.94, 3.76 [20:07:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.82, 3.99, 3.83 [20:07:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.56, 3.81, 3.73 [20:08:13] Who's messing with mw[56]? [20:08:31] john and me [20:08:37] john's off [20:08:42] so it'll be me atm [20:08:54] but that load is because of the traffic [20:10:09] Okay paladox: and see -staff for a incinga survey [20:11:03] ok [20:17:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.54, 2.99, 3.35 [20:19:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.05, 3.64, 3.61 [20:21:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.50, 3.53, 3.57 [20:25:38] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.26, 3.86, 3.68 [20:25:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.84, 3.52, 3.42 [20:27:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.93, 3.74, 3.65 [20:27:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.13, 3.69, 3.49 [20:29:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.54, 3.25, 3.35 [20:35:38] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 1.81, 2.74, 3.25 [20:37:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+37/-0/±0] 13https://git.io/JeYTG [20:37:47] [02miraheze/puppet] 07paladox 03572d71b - Add translate module, required by apt module [20:40:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYTC [20:40:35] [02miraheze/puppet] 07paladox 03c98c45e - fix class [20:41:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:43:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:44:31] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.84, 3.87, 3.52 [20:45:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.10, 4.05, 3.58 [20:46:27] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.03, 3.22, 3.32 [20:47:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 2.87, 3.73, 3.53 [20:48:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYTH [20:48:14] [02miraheze/puppet] 07paladox 036dadb2c - Update init.pp [20:53:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 1.87, 2.89, 3.27 [20:57:03] paladox: why required? [20:57:23] JohnLewis since i want to make sure before apt-get update, that the priority is set [20:57:38] it's causing a dependacy cycle though.. [20:57:44] but why is the whole thing required? [20:57:55] why couldn't you just add whats need or adapt [20:58:47] well nodejs_pin is required to be set first [20:59:02] okay..? [20:59:11] that doesn't answer my question though? [20:59:58] JohnLewis what do you mean by 'whats need or adapt'? [21:00:18] My question hasn't been answered first yet [21:00:29] JohnLewis i did answer though? [21:00:52] why is it required, you answered "needs to be pinned first" [21:01:05] why's a module called translate needed, what does that actually do? [21:02:42] ohh [21:02:45] JohnLewis it's required by apt [21:02:48] apt uses it [21:02:53] (the module not the command) [21:03:11] again, where does it use and why? [21:03:14] https://github.com/miraheze/puppet/blob/master/modules/apt/manifests/pin.pp#L84 [21:03:15] [ puppet/pin.pp at master · miraheze/puppet · GitHub ] - github.com [21:03:16] JohnLewis ^ [21:03:22] if we blindly import moduels we might as well just copy and paste Wikimedia's [21:04:02] so essentially instead of removing that line, you import 37 new files and 1.5k lines of code [21:04:11] or even worst case, you do that for 14 lines of actual code? [21:04:49] JohnLewis so your suggesting to revert that and remove translate from apt? [21:06:12] I'm suggesting minimal footprint [21:06:47] ok [21:10:39] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-37/±9] 13https://git.io/JeYkO [21:10:41] [02miraheze/puppet] 07paladox 039f30d67 - Revert "Add translate module, required by apt module" This reverts commit 572d71b0fd18d3703cd81f2618c652de2d72dc25. [21:10:42] [02miraheze/puppet] 07paladox 03094f272 - Remove translate from apt module [21:10:49] JohnLewis ^ [21:10:58] ok [21:13:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYkG [21:13:37] [02miraheze/puppet] 07paladox 032029e3d - mediawiki: Require the class nodejs instead of package [21:15:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYkW [21:15:27] [02miraheze/puppet] 07paladox 03440e968 - Apt: Fix syntax error [21:26:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYkE [21:26:44] [02miraheze/puppet] 07paladox 03dadd8af - Update init.pp [21:28:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYkz [21:28:04] [02miraheze/puppet] 07paladox 03747998d - Update init.pp [21:29:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.74, 3.37, 3.08 [21:31:52] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 2.97, 3.14, 3.03 [21:32:39] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:33:43] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:33:46] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:33:55] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:34:02] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:34:05] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:34:15] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:34:26] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:35:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.92, 3.73, 3.30 [21:41:53] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.54, 4.17, 3.63 [21:42:13] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [21:44:05] @steward [21:44:23] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:46:20] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:52:31] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [21:53:34] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [21:54:26] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [21:54:52] RhinosF1: im not a stewards but whats up [21:55:08] Zppix: void has done it (see meta) [21:55:16] ok [21:57:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 2.47, 3.70, 3.91 [21:58:55] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 4.59, 3.74, 3.22 [21:59:51] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 4.22, 3.95, 3.97 [22:04:26] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:06:23] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:08:33] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 2.97, 3.64, 3.53 [22:09:51] PROBLEM - mw5 Current Load on mw5 is WARNING: WARNING - load average: 3.23, 3.67, 3.91 [22:10:29] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 1.79, 2.98, 3.30 [22:16:17] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.25, 3.44, 3.43 [22:16:27] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:17:51] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 1.98, 2.72, 3.40 [22:18:12] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.00, 2.91, 3.24 [22:33:06] !log rebooted mw5 [22:33:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:34:18] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw5 [22:35:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [22:35:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw5 [22:35:38] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.43, 3.25, 3.05 [22:37:38] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.78, 3.11, 3.03 [22:42:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 8 backends are healthy [22:43:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [22:43:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 8 backends are healthy [22:44:31] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 3.62, 3.60, 3.27 [22:46:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw5 [22:46:26] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 2.89, 3.32, 3.20 [22:47:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw5 [22:47:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw5 [22:49:45] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 3.90, 3.47, 3.03 [22:53:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 8 backends are healthy [22:53:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 8 backends are healthy [22:53:41]