[00:09:16] JohnLewis: You told me yesterday to verify what my IRC bot does before adding it to the channel, what do you mean by that? [00:16:02] paladox, Voidwalker, Zppix: ^ [00:16:12] Examknow: he wants to know what it does [00:16:39] I assume he is not on right now [00:19:09] .help tell [00:19:10] Zppix: Give someone a message the next time they're seen [00:19:11] Zppix: e.g. ZppixBot, tell Embolalia he broke something again. [00:19:12] Examknow: ^ [00:19:18] use .tell [00:20:15] .tell JohnLewis I need to talk with you about my IRC bot [00:20:15] Examknow: I'll pass that on when JohnLewis is around. [00:20:46] you could of just told him what it does... [00:21:25] oop [00:21:46] you can help me contribute if you want [00:21:57] I assume you know a bit about IRC bots [00:22:37] I already have 2 IRC bots i gotta maintain, plus cvt and thats just for Miraheze xD [00:22:53] guess your busy than [00:22:57] yep [00:23:42] if you want I can just add you as a sysadmin for it so you can issue admin commands although I don't know what you would need them for [00:25:16] Is your bot built in python? [00:25:40] yes [00:25:50] Examknow: go ahead, but i would like to know what your bot is for [00:26:59] It basically logs conversations, can get news from RSS feeds, weather, ca links for miraheze and wmf stuff like that [00:27:29] So basically an infobot like ZppixBot... i highly doubt we need another infobot [00:27:35] we already have ZppixBot and wm-bot5 [00:28:58] does ZppixBot use ca links? [00:29:27] That can be added quite easily [00:30:41] yeah I didn't have much trouble doing that [00:30:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG2l [00:31:01] [02miraheze/puppet] 07paladox 03d4f0456 - Increase NR_OF_NETWORK_WORKERS to 3 [00:52:08] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:52:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:53:01] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:56:16] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.004 second response time [00:56:17] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:56:46] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [05:00:46] Zppix: Zppix's 5th year on English Wikipedia [05:37:45] .tell Examknow Read the request comments section of wiki requests please [05:37:45] RhinosF1: I'll pass that on when Examknow is around. [06:05:37] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:06:20] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:06:36] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 399 failures. Last run 4 minutes ago with 399 failures. Failed resources (up to 3 shown) [06:06:45] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 5 minutes ago with 13 failures. Failed resources (up to 3 shown) [06:06:47] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 5 minutes ago with 13 failures. Failed resources (up to 3 shown) [06:06:48] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 50 failures. Last run 5 minutes ago with 50 failures. Failed resources (up to 3 shown) [06:07:14] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 220 failures. Last run 5 minutes ago with 220 failures. Failed resources (up to 3 shown) [06:07:42] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 6 minutes ago with 16 failures. Failed resources (up to 3 shown) [06:07:42] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 6 minutes ago with 14 failures. Failed resources (up to 3 shown) [06:07:54] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 6 minutes ago with 12 failures. Failed resources (up to 3 shown) [06:07:58] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 155 failures. Last run 4 minutes ago with 155 failures. Failed resources (up to 3 shown): File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades],File[/root/ufw-fix],File[/usr/local/bin/gen_fingerprints] [06:08:02] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:08:12] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 201 failures. Last run 6 minutes ago with 201 failures. Failed resources (up to 3 shown): File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades],File[/root/ufw-fix] [06:08:13] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 378 failures. Last run 6 minutes ago with 378 failures. Failed resources (up to 3 shown) [06:08:17] PROBLEM - glusterfs1 Puppet on glusterfs1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:08:22] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 6 minutes ago with 20 failures. Failed resources (up to 3 shown) [06:08:35] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 7 minutes ago with 12 failures. Failed resources (up to 3 shown) [06:09:20] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 392 failures. Last run 7 minutes ago with 392 failures. Failed resources (up to 3 shown) [06:09:32] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:10:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGwi [06:10:11] [02miraheze/services] 07MirahezeSSLBot 03f2956bf - BOT: Updating services config for wikis [06:12:53] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:12:58] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:13:39] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:13:47] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:13:52] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:00] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:17] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [06:14:21] RECOVERY - glusterfs1 Puppet on glusterfs1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:27] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:45] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:15:00] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:15:35] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:15:43] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:15:46] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:15:57] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:16:03] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:16:30] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:16:55] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:17:19] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [06:27:19] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3035 MB (12% inode=94%); [08:53:06] PROBLEM - unrecnations.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:58:47] RECOVERY - unrecnations.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'unrecnations.wiki' will expire on Wed 04 Dec 2019 09:56:04 PM GMT +0000. [09:13:49] PROBLEM - unrecnations.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:58:00] PROBLEM - glusterfs2 Current Load on glusterfs2 is CRITICAL: CRITICAL - load average: 4.85, 3.83, 2.60 [10:11:37] PROBLEM - glusterfs2 GlusterFS port 49152 on glusterfs2 is CRITICAL: connect to address 81.4.100.77 and port 49152: Connection refused [10:15:30] PROBLEM - glusterfs2 Current Load on glusterfs2 is WARNING: WARNING - load average: 0.09, 1.90, 3.54 [10:18:23] RECOVERY - glusterfs2 Current Load on glusterfs2 is OK: OK - load average: 0.00, 1.07, 2.94 [11:25:26] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.52, 1.84, 1.62 [11:30:12] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [11:38:17] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.13, 1.95, 1.77 [11:44:29] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.72, 1.99, 1.86 [11:50:26] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [11:50:56] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.30, 2.20, 1.96 [11:57:04] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.30, 1.71, 1.82 [12:03:31] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.64, 1.50, 1.66 [14:03:34] RhinosF1: https://meta.miraheze.org/wiki/Stewards%27_noticeboard#Wiki_deletion_request [14:03:35] [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org [14:13:55] Zppix: needs operations or a steward [14:14:07] RhinosF1: you can't delete wikis? [14:15:23] Stewards deal with that I think [14:15:39] Zppix: nah, should be left to someone who can drop the dB or stewards do it (policy) [14:16:03] * Zppix shrugs [14:34:24] Yes, not ops [14:34:28] Stewards deal with on-wiki deletion [14:34:32] Ops just do the final ones [14:36:38] Reception123: see -staff ToU? [14:37:46] ok [14:59:32] PROBLEM - glusterfs2 Current Load on glusterfs2 is CRITICAL: CRITICAL - load average: 8.10, 6.07, 2.93 [15:00:34] PROBLEM - glusterfs2 Puppet on glusterfs2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:04:33] RECOVERY - glusterfs2 Puppet on glusterfs2 is OK: OK: Puppet is currently enabled, last run 21 minutes ago with 0 failures [15:07:58] RECOVERY - glusterfs2 GlusterFS port 49152 on glusterfs2 is OK: TCP OK - 0.001 second response time on 81.4.100.77 port 49152 [15:10:25] PROBLEM - glusterfs2 Current Load on glusterfs2 is WARNING: WARNING - load average: 0.48, 3.10, 3.64 [15:13:35] RECOVERY - glusterfs2 Current Load on glusterfs2 is OK: OK - load average: 0.24, 1.75, 3.00 [15:45:05] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Sat 12 Oct 2019 03:38:57 PM GMT +0000). [15:45:22] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGDQ [15:45:23] [02miraheze/ssl] 07MirahezeSSLBot 03cfe7b01 - Bot: Update SSL cert for wiki.zymonic.com [15:52:35] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [15:54:21] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 25 Dec 2019 02:45:15 PM GMT +0000. [16:07:50] !log reboot glusterfs[12] [16:07:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:17:48] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static-new] [16:24:03] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:24:59] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:25:58] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:35:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG9C [17:35:13] [02miraheze/services] 07MirahezeSSLBot 03302d198 - BOT: Updating services config for wikis [18:33:19] [ANNOUNCEMENT] ZppixBot v5 was just released! https://github.com/Pix1234/ZppixBot-Source/releases/tag/v5 [18:34:53] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGHo [18:34:54] [02miraheze/ManageWiki] 07translatewiki 031125bd0 - Localisation updates from https://translatewiki.net. [18:34:55] [ Main page - translatewiki.net ] - translatewiki.net. [19:39:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGQM [19:39:13] [02miraheze/puppet] 07paladox 03178fb8f - Set NR_OF_HDD_WORKERS_PER_NETWORK_WORKER to 4 [19:39:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGQy [19:39:28] [02miraheze/puppet] 07paladox 03885b3d1 - Set NR_OF_NETWORK_WORKERS to 4 [19:44:58] !log restart lizardfs-chunkserver on lizardfs[45] [19:54:26] !log apt-get upgrade - misc3 [19:54:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:55:35] !log apt-get upgrade - lizardfs[45] [19:55:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:05:56] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:05:57] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:05:58] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw2 mw3 [20:05:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:07:13] Eek, 503s! [20:07:15] paladox: 593 [20:07:20] oh [20:07:22] S9/9 [20:07:27] paladox: 593 [20:07:30] 503 [20:07:34] There we go [20:07:42] that would be the restart i did. [20:07:47] (logged above) [20:08:01] though i dunno why things went down [20:08:07] meta/login doin't use lfs atm [20:08:14] paladox: Well they are [20:08:20] how? [20:08:38] paladox: 503 backend fetch failed [20:08:44] paladox: did you unplug stuff to charge your phone again? [20:08:56] yes, but i dunno how. I switched meta/login to the new mount [20:09:02] which varnish uses login as a healthcheck [20:09:06] Zppix lol, ofc :P [20:09:28] paladox: is it possible that if loginwiki cant reach the wikis on lizard that it causes some sort of erorr and takes it down too? [20:09:41] Nope [20:09:52] well something with lizard is affecting loginwiki [20:09:56] or vise versa [20:09:59] unless i need to put the new mount first (as default) but disable lizardfs for the wikis i migrated [20:10:09] paladox: try it [20:10:18] paladox: are all new wikis created going on new mount? [20:10:30] Zppix: stop pinging me (loginwiki does) [20:10:54] RhinosF1: im not pinging you? [20:11:20] well.. i just did but besides that [20:11:32] Zppix: 'loginwiki', 'metawiki' and various other wikis ping me [20:11:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG7l [20:11:34] [02miraheze/mw-config] 07paladox 03ec22111 - Switch mounts around [20:11:36] done [20:11:44] RhinosF1: well thats your fault xD [20:11:51] paladox: can you force puppet [20:12:26] paladox: back [20:13:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:13:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:13:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:13:51] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.822 second response time [20:27:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG7V [20:27:04] [02miraheze/mw-config] 07paladox 03188612b - Fix check [20:35:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG7P [20:35:11] [02miraheze/services] 07MirahezeSSLBot 036d752cc - BOT: Updating services config for wikis [21:08:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:11:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeG5G [21:11:19] [02miraheze/mw-config] 07paladox 03b3e5f46 - Update LocalSettings.php [21:11:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:50:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdT [21:50:38] [02miraheze/mw-config] 07paladox 032eaf639 - Remove metawiki from new mount [21:52:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdI [21:52:24] [02miraheze/mw-config] 07paladox 03dcc5a2b - Remove anotheredenwiki from new mount [22:08:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGd0 [22:08:16] [02miraheze/mw-config] 07paladox 038cb2c57 - Move nonciclopediawiki off the new mount point [22:11:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGd2 [22:11:45] [02miraheze/mw-config] 07paladox 032da0910 - Move allthetropeswiki off new mount point [22:18:07] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [22:19:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdK [22:19:04] [02miraheze/mw-config] 07paladox 03295fca9 - Update LocalSettings.php [22:20:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdP [22:20:36] [02miraheze/mw-config] 07paladox 035703df4 - Move browndustwiki off new mount point [22:21:17] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:23:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdS [22:23:11] [02miraheze/mw-config] 07paladox 03bbb922f - Move fapceowiki off new mount point [22:30:44] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGdh [22:30:45] [02miraheze/mw-config] 07paladox 03c3579fd - Move breedersofthenephelymwiki off new mount point [22:32:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGFe [22:32:38] [02miraheze/mw-config] 07paladox 03e34622f - Move sidemwiki and cwarswiki off new mount point [22:52:39] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/JeGFC [22:52:41] [02miraheze/mediawiki] 07paladox 034905df5 - Add HeaderFooter extension - T4712 [22:53:35] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [22:53:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeGFW [22:53:38] [02miraheze/mw-config] 07paladox 035d400d2 - Add HeaderFooter extension to MW [22:53:39] [02mw-config] 07paladox opened pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:54:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeGF8 [22:54:23] [02miraheze/mw-config] 07paladox 033d28219 - Update LocalSettings.php [22:54:24] [02mw-config] 07paladox synchronize pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:55:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeGF4 [22:55:05] [02miraheze/mw-config] 07paladox 030db896b - Update LocalExtensions.php [22:55:06] [02mw-config] 07paladox synchronize pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:55:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeGFB [22:55:29] [02miraheze/mw-config] 07paladox 03efe4f4e - Update extension-list [22:55:30] [02mw-config] 07paladox synchronize pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:56:07] [02mw-config] 07paladox edited pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:57:41] [02mw-config] 07paladox closed pull request 03#2769: Add HeaderFooter extension to MW - 13https://git.io/JeGFl [22:57:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JeGF0 [22:57:44] [02miraheze/mw-config] 07paladox 03f13d7e1 - Add HeaderFooter extension to MW (#2769) * Add HeaderFooter extension to MW * Update LocalSettings.php * Update LocalExtensions.php * Update extension-list [22:57:46] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [22:57:47] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [23:02:48] !log running lc on mw* [23:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:05:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGFr [23:05:10] [02miraheze/services] 07MirahezeSSLBot 032660159 - BOT: Updating services config for wikis [23:07:39] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [23:08:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGFK [23:08:02] [02miraheze/puppet] 07paladox 03587e622 - Raise NR_OF_NETWORK_WORKERS to 6 [23:08:38] [02ManageWiki] 07JohnFLewis closed pull request 03#118: only check perms on install - 13https://git.io/JeGmK [23:08:39] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JeGF6 [23:08:41] [02miraheze/ManageWiki] 07The-Voidwalker 03acaf5a4 - only check perms on install (#118) feels like a hack, but just adding a parameter to ManageWikiRequirements::process to ignore the permissions check seems like a decent temporary solution, barring a rewrite [23:09:12] !log restart lizardfs-chunkserver on lizardfs[45] for config change [23:09:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGFP [23:09:52] [02miraheze/puppet] 07paladox 035575875 - Raise NR_OF_HDD_WORKERS_PER_NETWORK_WORKER to 6 [23:11:38] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:11:53] [02ManageWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpSns [23:11:54] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JeGF1 [23:11:56] [02miraheze/ManageWiki] 07paladox 0384bf029 - Fix travis [23:11:57] [02ManageWiki] 07paladox opened pull request 03#119: Fix travis - 13https://git.io/JeGFM [23:12:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:12:34] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:12:36] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:13:36] should recover [23:13:44] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:16] miraheze/ManageWiki/paladox-patch-3/84bf029 - paladox The build passed. https://travis-ci.org/miraheze/ManageWiki/builds/590187022 [23:14:28] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:31] figures, if you're restarting lizardfs [23:15:08] * paladox trying to improve uptime [23:15:34] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 50% [23:15:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:16:33] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:16:42] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:16:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:17:10] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.747 second response time [23:17:42] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24570 bytes in 0.746 second response time [23:18:03] [02ManageWiki] 07paladox closed pull request 03#119: Fix travis - 13https://git.io/JeGFM [23:18:04] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeGFH [23:18:06] [02miraheze/ManageWiki] 07paladox 037b74eec - Fix travis (#119) [23:18:07] [02ManageWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpSns [23:18:09] [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-3 [23:18:21] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 7% [23:18:22] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:18:28] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:18:40] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:24:17] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:27:57] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.76, 1.86, 1.45 [23:30:42] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.18, 1.93, 1.53 [23:33:36] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.45, 1.64, 1.47 [23:41:15] Hello Examknow_! If you have any questions, feel free to ask and someone should answer soon.