[00:04:37] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 18.47, 21.18, 16.72 [00:06:37] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.59, 18.55, 16.30 [00:55:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJvr6 [00:55:14] [02miraheze/services] 07MirahezeSSLBot 03690aca6 - BOT: Updating services config for wikis [01:56:46] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 7.87, 7.42, 3.78 [01:58:48] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.79, 5.22, 3.43 [02:08:22] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.43, 1.76, 1.37 [02:10:20] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.69, 1.70, 1.39 [02:14:24] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.25, 1.81, 1.53 [02:16:23] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.20, 2.04, 1.64 [02:18:24] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.06, 1.72, 1.57 [02:20:25] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 1.12, 1.45, 1.49 [02:25:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJvKB [02:25:17] [02miraheze/services] 07MirahezeSSLBot 035915556 - BOT: Updating services config for wikis [04:19:03] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 4.41, 3.22, 1.88 [04:24:54] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.92, 1.73, 1.62 [04:26:56] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.95, 1.49, 1.55 [04:35:56] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Wed 15 Jul 2020 04:32:07 GMT +0000). [04:36:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJviP [04:36:03] [02miraheze/ssl] 07MirahezeSSLBot 031a9d2f0 - Bot: Update SSL cert for wiki.villagecollaborative.net [04:41:57] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Sun 13 Sep 2020 10:40:41 GMT +0000. [05:05:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJvP5 [05:05:15] [02miraheze/services] 07MirahezeSSLBot 03076cf07 - BOT: Updating services config for wikis [05:07:05] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:09:04] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [05:37:54] !log deleted and dropped doctorwhofanfilmsdatabasewiki [05:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:51:16] Let me know if I need to do anything with T5808 in Phabricator: https://phabricator.miraheze.org/T5808 [05:51:17] [ ⚓ T5808 Error 503 ] - phabricator.miraheze.org [05:52:25] @icon256 no I just triaged it as we don't like leaving non-triaged tasks [05:54:40] Ah, gotcha, no problem. Sounds good. [05:54:41] thx [05:55:12] Hello icon256! If you have any questions, feel free to ask and someone should answer soon. [06:00:14] Hello ZI_Jony[m]! If you have any questions, feel free to ask and someone should answer soon. [06:17:05] PROBLEM - jobrunner1 Disk Space on jobrunner1 is WARNING: DISK WARNING - free space: / 4190 MB (9% inode=84%); [06:23:02] PROBLEM - jobrunner1 Disk Space on jobrunner1 is CRITICAL: DISK CRITICAL - free space: / 2599 MB (5% inode=84%); [07:34:58] RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 6101 MB (13% inode=84%); [07:55:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJvSQ [07:55:16] [02miraheze/services] 07MirahezeSSLBot 035e5420f - BOT: Updating services config for wikis [07:55:52] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [07:58:52] Reception123: ^ [07:59:09] Reception123: and the status can be Up when you fix that alert [07:59:28] RhinosF1: urgh again [07:59:43] Reception123: I think we need to look at that closer [08:00:08] RhinosF1: by we you mean paladox :) [08:00:14] !log umount /mnt/mediawiki-static on mw4 [08:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:00:27] Reception123: I mean file a task and stare at Paladox [08:01:43] https://phabricator.miraheze.org/T5833 [08:01:43] [ ⚓ T5833 mediawiki-static keeps getting unmounted on mw* ] - phabricator.miraheze.org [08:02:16] Reception123: thanks [08:02:36] Reception123: can you run puppet so we can see it's working [08:02:45] * RhinosF1 is waiting for his grandad to arrive [08:02:48] k [08:03:57] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [08:10:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJv92 [08:10:23] [02miraheze/services] 07MirahezeSSLBot 03a923ee5 - BOT: Updating services config for wikis [09:43:08] PROBLEM - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:14] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Sun 16 Aug 2020 23:10:01 GMT +0000. [10:41:06] Reception123: around? [10:56:55] !log importDump.php for T5829 [10:56:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:01:38] !log importDump.php for T5831 [11:01:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:02:09] ^ me will check them after lunch [11:33:06] Morning JohnLewis [11:35:46] Morning [11:36:24] JohnLewis: AF is global? https://phabricator.miraheze.org/T5835 [11:36:25] [ ⚓ T5835 Extension:AbuseFilter ] - phabricator.miraheze.org [11:38:18] It is enabled [11:44:29] PROBLEM - cp7 Current Load on cp7 is WARNING: WARNING - load average: 7.13, 6.77, 3.31 [11:46:30] RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 1.63, 4.88, 3.06 [11:48:28] It is global yes [11:49:00] JohnLewis: they were looking in the wrong section [11:49:35] JohnLewis: how do we remove the empty right from moviepediawiki? [11:53:28] RhinosF1: still need me? [11:53:53] Reception123: there's an ssl cert waiting [11:54:24] Someone needs to do the private part. I don't mind doing the jobrunner1 part [11:58:23] ok [11:58:51] RhinosF1: well you don't need to ask first you can do the jobrunner part, make an SSL PR and then I can do the key [12:09:38] Reception123: will do at some point today. Just dealing with the best customer support ever. [12:10:02] RhinosF1: ok. which one is that? [12:10:43] Reception123: Amazon's A-Z Guarntee team. From complaint to refund to about 30s [12:10:59] ah [12:11:00] wow [12:12:24] Reception123: My grandad wasn't hearing back from a seller so I filed the claim and it was granted straight away. [12:22:53] that's nice [12:25:48] RhinosF1: since it’s a blank string, I don’t know if ManageWikiPermissions() would work for it [12:33:23] JohnLewis: okay [12:50:09] JohnLewis: should we use a set user (maybe with Global Bot for maint script) [12:58:17] Maint script actions should ideally be logged on wiki on so [12:58:25] *so no [12:59:07] JohnLewis: Okay, Tbh I've only ever twice enabled it running as bot. Once on request and once for commonswiki which was based on my own local request. [12:59:30] But a script can forcefully Mark it as a bot edit anyway [12:59:43] It doesn’t require bot rights to do so in the backend [13:01:40] JohnLewis: hmm [13:02:27] !log run createLocalAccount.php on all wikis for ZppixBot to hopefully make it less likely as wikis are added to say login error if there's no account. [13:02:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:07:54] JohnLewis: what should we do about granting autopatrolled on loginwiki to users that edit often? [13:08:13] Should we just add autopatrol to autoconfirmed? [13:08:21] Yeah [13:08:34] JohnLewis: I'll add it then [13:09:45] {{done}} [13:10:11] That saves me a huge job [14:05:47] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 22.27, 19.32, 14.50 [14:07:47] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 17.22, 18.84, 14.95 [14:07:50] Good morning [14:19:10] [02mw-config] 07Amanda-Catherine opened pull request 03#3132: Misc. config for dcwiki T5838 - 13https://git.io/JJfJG [14:21:36] Amanda-Catherine/mw-config/patch-3/61b6b3e - Amanda The build passed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/173486630 [14:25:38] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JJfJ2 [14:25:39] [02miraheze/MirahezeMagic] 07translatewiki 0354a66ab - Localisation updates from https://translatewiki.net. [14:25:40] [ Main page - translatewiki.net ] - translatewiki.net [14:33:30] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.72, 22.01, 19.27 [14:37:41] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 14.65, 18.87, 18.65 [15:11:19] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [15:14:50] [02mw-config] 07paladox reviewed pull request 03#3132 commit - 13https://git.io/JJfTi [15:15:48] Paladox: I didn't see where the variable was defined elsewhere? [15:16:03] AmandaCath you can use github search [15:16:17] AmandaCath https://github.com/miraheze/mw-config/blob/4a8862c0fd2a4316fbc1f6df755bfa9199b89423/LocalSettings.php#L2576 [15:16:18] [ mw-config/LocalSettings.php at 4a8862c0fd2a4316fbc1f6df755bfa9199b89423 · miraheze/mw-config · GitHub ] - github.com [15:16:46] Oh... [15:16:56] * AmandaCath has never scrolled down that far in LS [15:19:26] I'll fix it in a minute, busy with something else right now [15:22:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/JJfTx [15:22:31] [02miraheze/puppet] 07paladox 03ecd864d - salt: Refactor and only use salt-ssh We no longer use salt-(master|minion), instead we replace with salt-ssh. We create a salt-user for full root for all servers. [15:22:32] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [15:22:34] [02puppet] 07paladox opened pull request 03#1415: salt: Refactor and only use salt-ssh - 13https://git.io/JJfTp [15:23:23] [02mw-config] 07Amanda-Catherine synchronize pull request 03#3132: Misc. config for dcwiki T5838 - 13https://git.io/JJfJG [15:25:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-1/±0] 13https://git.io/JJfkJ [15:25:16] [02miraheze/puppet] 07paladox 03bdcdc07 - Delete minions.pp [15:25:18] [02puppet] 07paladox synchronize pull request 03#1415: salt: Refactor and only use salt-ssh - 13https://git.io/JJfTp [15:25:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-1/±0] 13https://git.io/JJfkT [15:25:49] [02miraheze/puppet] 07paladox 03c3947df - Update and rename master.pp to init.pp [15:25:50] [02puppet] 07paladox synchronize pull request 03#1415: salt: Refactor and only use salt-ssh - 13https://git.io/JJfTp [15:28:20] !log upgrade ca-certificates on puppet2 [15:28:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:29:06] !log install salt-ssh on puppet2 [15:29:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:29:59] Why does the logbot call everybody "Master"? [15:30:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfkl [15:30:23] [02miraheze/services] 07MirahezeSSLBot 03f85eba6 - BOT: Updating services config for wikis [15:31:25] PROBLEM - puppet2 Puppet on puppet2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [15:32:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJfku [15:32:51] [02miraheze/puppet] 07paladox 03f3be880 - logbot: Stop calling users master [15:32:52] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [15:32:54] [02puppet] 07paladox opened pull request 03#1416: logbot: Stop calling users master - 13https://git.io/JJfkz [15:33:11] [02puppet] 07paladox closed pull request 03#1416: logbot: Stop calling users master - 13https://git.io/JJfkz [15:33:13] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJfkV [15:33:14] [02miraheze/puppet] 07paladox 03e439779 - Merge pull request #1416 from miraheze/paladox-patch-8 logbot: Stop calling users master [15:33:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:33:17] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [15:33:17] AmandaCath ^ [15:34:15] I was just curious why it did that [15:34:29] Since it seems it's been doing it forever [15:36:12] [02puppet] 07JohnFLewis commented on commit 03f3be880ba5b58de6a856a8bba52f401e965ceba4 - 13https://git.io/JJfkX [15:36:54] I'd assume that the bot only listens to sysadmins/SRE? [15:37:23] (If not we should make sure that not just anyone can use the command while we're at it) [15:38:18] [02puppet] 07paladox commented on commit 03f3be880ba5b58de6a856a8bba52f401e965ceba4 - 13https://git.io/JJfkQ [15:38:56] [02puppet] 07JohnFLewis commented on commit 03f3be880ba5b58de6a856a8bba52f401e965ceba4 - 13https://git.io/JJfkF [15:39:52] [02puppet] 07paladox commented on commit 03f3be880ba5b58de6a856a8bba52f401e965ceba4 - 13https://git.io/JJfkN [15:43:13] JohnLewis / paladox: I'm assuming the bot knows to only log messages that come from a sysadmin? [15:43:23] no, anyone can log [15:44:53] Maybe that should be addressed [15:45:33] Not really that important I suppose, but since the Server Admin Log Page on Meta is protected by an abuse filter, it doesn't really make sense that anyone can log [15:46:03] Kind of defeats the purpose of having the SAL protection IMO [15:52:31] !log gluster volume set mvol performance.cache-size 32MB - gluster1 [15:52:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:02] !log gluster volume set mvol performance.cache-refresh-timeout 1 - gluster1 [15:54:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:54:43] AmandaCath; the protection isn’t to stop people logging who aren’t sysadmins, it to stop people editing it on-wiki without using the IRC bot unless they really need to [15:55:14] it encourages people to log their actions in a venue which is live and open to immediate discussion rather than on-wiki where neither are true easily [15:57:14] !log gluster volume set mvol performance.io-thread-count 12 - gluster1 [15:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:52] But what purpose would a non-sysadmin have for logging anyway? [15:58:23] !log gluster volume set mvol network.inode-lru-limit 100000 - gluster1 [15:58:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:59:49] AmandaCath; generally, very few. But can be useful for incident response, collaboration between sysadmins/non-Sysadmins etc. [16:42:24] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.19, 2.34, 1.41 [16:44:20] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.04, 1.92, 1.36 [16:46:20] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.49, 1.42, 1.25 [17:02:58] !log gluster volume set mvol server.event-threads 5 - gluster1 [17:03:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:02] !log gluster volume set mvol client.event-threads 5 - gluster1 [17:03:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:08] !log gluster volume set mvol features.cache-invalidation-timeout 60 - gluster1 [17:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:19] [02puppet] 07RhinosF1 opened pull request 03#1417: Fix abuse - 13https://git.io/JJfmd [17:12:29] Reception123: ^ [17:12:44] RhinosF1: I'd love to be able to "fix" abuse forever xD [17:13:02] [02puppet] 07Reception123 closed pull request 03#1417: Fix abuse - 13https://git.io/JJfmd [17:13:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfmA [17:13:05] [02miraheze/puppet] 07RhinosF1 03971dd68 - Fix abuse (#1417) [17:13:39] Reception123: that's a whole other can of worms [17:14:44] RhinosF1: shouldn't have moved it [17:15:08] abuse is a standard email aliases, so should have stayed in that section [17:15:10] JohnLewis: made a lot more sense to put the abuse aliases together to me [17:15:19] But I can re-put it there [17:15:29] but keeping RFC defined aliases together makes more sense ;) [17:16:39] JohnLewis: One sec [17:17:25] [02puppet] 07RhinosF1 opened pull request 03#1418: Update aliases - 13https://git.io/JJfYG [17:17:31] JohnLewis: ^ [17:18:02] [02puppet] 07JohnFLewis closed pull request 03#1418: Update aliases - 13https://git.io/JJfYG [17:18:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfYl [17:18:05] [02miraheze/puppet] 07RhinosF1 03145ad12 - Move abuse back to RFC defined section (#1418) [17:27:24] !log gluster volume set mvol diagnostics.count-fop-hits off - gluster1 [17:27:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:48] !log gluster volume set mvol diagnostics.latency-measurement off - gluster1 [17:27:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:42] !log gluster volume set mvol server.event-threads 7 - gluster1 [17:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:28:46] !log gluster volume set mvol client.event-threads 7 - gluster1 [17:28:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:05] !log gluster volume set mvol diagnostics.latency-measurement on ; gluster volume set mvol diagnostics.count-fop-hits on - gluster1 [17:32:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:14] PROBLEM - netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Wed 15 Jul 2020 18:08:07 GMT +0000). [18:12:06] PROBLEM - www.netazar.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.netazar.org' expires in 15 day(s) (Wed 15 Jul 2020 18:08:07 GMT +0000). [18:18:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfsU [18:18:59] [02miraheze/ssl] 07MirahezeSSLBot 03cb8fd66 - Bot: Update SSL cert for www.netazar.org [18:21:57] RECOVERY - www.netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Sun 27 Sep 2020 17:18:51 GMT +0000. [18:22:13] RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Sun 27 Sep 2020 17:18:51 GMT +0000. [18:46:35] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/dumpBackup.php --wiki nltramswiki --full > /home/reception/nltramswiki29062020.xml (part of wikibackups) [18:46:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:50:56] PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lbcomms.co.za' expires in 15 day(s) (Wed 15 Jul 2020 19:48:28 GMT +0000). [19:51:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfcl [19:51:05] [02miraheze/ssl] 07MirahezeSSLBot 0376d86e7 - Bot: Update SSL cert for wiki.lbcomms.co.za [19:55:21] !log added a discord webhook [19:55:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:28] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:28] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:31] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:32] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:39] my bad... fixed [20:02:40] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:41] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:46] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:48] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:02:53] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:06] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:08] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:11] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:13] PROBLEM - ldap1 Puppet on ldap1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:13] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:18] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:18] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:24] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:28] PROBLEM - db9 Puppet on db9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:48] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:03:56] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:01] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:06] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:11] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:13] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:13] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:15] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:04:25] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:07:41] Hello Superleaf1995! If you have any questions, feel free to ask and someone should answer soon. [20:07:56] ok so [20:08:09] ok [20:08:45] hi Superleaf1995 [20:09:01] hi [20:11:27] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:11:28] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [20:11:34] RECOVERY - db9 Puppet on db9 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [20:11:48] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:12:03] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:12:09] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [20:12:10] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:12:12] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:12:14] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [20:12:15] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [20:12:27] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:31] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:37] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:37] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:12:41] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:43] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:44] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:48] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:59] RECOVERY - wiki.lbcomms.co.za - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lbcomms.co.za' will expire on Mon 14 Sep 2020 09:10:41 GMT +0000. [20:13:02] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:13:05] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:12] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:13] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:15] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:17] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:18] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:51] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:40] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:11:10] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJf4j [22:11:11] [02miraheze/dns] 07paladox 034e3c5cb - Add bots.miraheze.wiki [22:11:13] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl [22:11:14] [02dns] 07paladox opened pull request 03#152: Add bots.miraheze.wiki - 13https://git.io/JJfBe [22:13:30] [02dns] 07RhinosF1 commented on commit 034e3c5cb01d37321795a0b951e0d2dcaafe4196bf - 13https://git.io/JJfBU [22:24:20] paladox: we're down [22:24:31] looking [22:24:37] db9 probably [22:25:00] db9 is down [22:25:03] unless the whole cloudx that it's on is down too [22:25:27] i can ping db9 [22:25:30] but cannot ssh in [22:25:58] PROBLEM - db9 APT on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:18] PROBLEM - db9 Puppet on db9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:26:37] "" this should probably be a critical [22:27:02] oh [22:27:06] mysql oom [22:27:14] db9 - CRITICAL - load average: 125.97, 76.46, 34.14 [22:27:16] oh my [22:27:35] oof [22:27:42] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [22:27:57] and i don't know what the root pass is... [22:28:06] so i guess we'll have to wait for puppet to restart ssh [22:28:09] PROBLEM - db9 Current Load on db9 is CRITICAL: CRITICAL - load average: 36.73, 59.70, 31.56 [22:28:11] RECOVERY - db9 APT on db9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:28:17] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [22:28:17] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [22:28:20] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [22:28:20] PROBLEM - db9 MySQL on db9 is CRITICAL: Can't connect to MySQL server on '51.77.109.151' (115) [22:28:26] RECOVERY - db9 Puppet on db9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:28:28] * RhinosF1 here [22:28:29] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [22:28:32] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [22:28:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [22:28:40] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 46% [22:28:48] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [22:29:01] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [22:29:31] i can ssh in no [22:29:32] now [22:30:40] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [22:32:24] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 51% [22:34:14] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 37% [22:38:11] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [22:38:37] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [22:40:09] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 39% [22:42:39] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 50% [22:42:44] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 23% [22:44:09] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 58% [22:46:10] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 39% [22:46:46] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [22:48:45] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 47% [22:50:09] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 49% [22:50:44] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [22:52:04] PROBLEM - db9 Current Load on db9 is WARNING: WARNING - load average: 2.02, 1.78, 7.61 [22:52:52] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 13% [22:53:29] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [22:53:46] RECOVERY - db9 MySQL on db9 is OK: Uptime: 1608 Threads: 59 Questions: 36594 Slow queries: 1203 Opens: 5495 Flush tables: 1 Open tables: 5489 Queries per second avg: 22.757 [22:54:06] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [22:54:09] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 5% [22:54:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:54:23] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [22:54:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:54:29] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [22:54:40] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 4% [22:54:55] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 26% [22:56:07] PROBLEM - db9 Current Load on db9 is CRITICAL: CRITICAL - load average: 8.56, 4.77, 7.42 [22:58:06] PROBLEM - db9 Current Load on db9 is WARNING: WARNING - load average: 6.52, 5.48, 7.36 [23:00:56] PROBLEM - espiri.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'espiri.wiki' expires in 15 day(s) (Wed 15 Jul 2020 22:57:29 GMT +0000). [23:02:05] RECOVERY - db9 Current Load on db9 is OK: OK - load average: 3.45, 4.54, 6.57 [23:02:14] hi RhinosF1 [23:02:17] ho [23:02:19] hi [23:03:46] :) [23:04:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJfRN [23:04:43] [02miraheze/ssl] 07MirahezeSSLBot 03ef193ef - Bot: Update SSL cert for espiri.wiki [23:10:32] what is the purpose of miraheze commons? is the goal not to have collected centers? why not use from wikimedia commons? same reason that is bundled with mw [23:12:37] RECOVERY - espiri.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'espiri.wiki' will expire on Sun 27 Sep 2020 22:04:35 GMT +0000. [23:15:22] License, Miraheze Commons allows images under NoDerivate and NoCommercial license [23:15:45] (CC BY ND, CC BY NC for example) [23:16:19] Wikimedia only allows licenses in the public domain, CC BY and CC BY SA [23:18:26] And that the same image can be used on multiple wikis without the need for "special settings"