[00:02:52] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3834 MB (19% inode=93%); [00:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMGR [00:35:14] [02miraheze/services] 07MirahezeSSLBot 037f52b49 - BOT: Updating services config for wikis [00:46:44] PROBLEM - cp3 APT on cp3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [01:12:01] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.37, 21.18, 18.56 [01:14:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.29, 19.43, 18.24 [01:27:43] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [01:33:56] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.82, 24.89, 20.95 [01:35:57] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.96, 21.35, 20.13 [01:37:56] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.45, 20.10, 19.81 [01:43:56] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.34, 21.42, 20.36 [01:45:56] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.48, 19.86, 19.93 [01:59:14] PROBLEM - cp8 APT on cp8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [02:00:34] PROBLEM - db7 APT on db7 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [02:04:07] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 30.78, 24.58, 21.22 [02:06:01] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.93, 21.14, 20.36 [02:09:57] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.17, 19.61, 19.95 [02:16:41] PROBLEM - cp6 APT on cp6 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [02:16:43] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (2 critical updates). [02:29:44] PROBLEM - cloud2 APT on cloud2 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (2 critical updates). [02:30:51] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.95, 21.66, 20.51 [02:32:47] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.97, 21.87, 20.77 [02:34:41] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.67, 20.07, 20.26 [02:42:32] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.52, 21.05, 20.38 [02:44:29] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 16.70, 19.96, 20.10 [02:54:22] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.87, 20.57, 20.04 [02:55:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMlE [02:55:14] [02miraheze/services] 07MirahezeSSLBot 0318a3563 - BOT: Updating services config for wikis [02:56:17] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.58, 18.50, 19.35 [03:14:10] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 18.02, 20.48, 19.78 [03:16:06] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.15, 19.31, 19.43 [03:24:02] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.97, 20.85, 19.87 [03:25:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMlh [03:25:13] [02miraheze/services] 07MirahezeSSLBot 030cc162a - BOT: Updating services config for wikis [03:25:56] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.36, 19.03, 19.34 [03:29:01] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfM8e [03:29:03] [02miraheze/mediawiki] 07paladox 03f335722 - Update CheckUser [03:32:54] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [03:34:10] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.89, 24.35, 21.47 [03:35:27] !log regenerating backups on bacula2 [03:35:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:36:05] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.68, 23.42, 21.43 [03:40:03] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 13.21, 18.37, 19.88 [03:40:46] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [03:42:29] !log apt-get upgrade - bacula2 (libgnutls30 puppet-agent) [03:42:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:43:18] !log apt-get upgrade - mail1 (libgnutls30) [03:43:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:44:04] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.71, 21.50, 20.87 [03:44:46] RECOVERY - mail1 APT on mail1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:46:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 13.67, 19.02, 20.08 [03:52:45] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [03:54:04] musikanimal i've deployed the fix for CU. [03:55:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfMBk [03:55:31] [02miraheze/mediawiki] 07paladox 03ddab613 - Update PortableInfobox [04:04:05] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.61, 21.71, 20.09 [04:09:33] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:10:04] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:10:43] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:12:15] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [04:17:55] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 15.83, 22.25, 22.63 [04:19:56] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.73, 22.69, 22.69 [04:22:44] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:23:31] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:03] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:24:27] PROBLEM - cp7 APT on cp7 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [04:25:54] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 14.01, 21.65, 22.89 [04:34:06] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:54] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.95, 17.54, 20.14 [04:53:06] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.81, 21.88, 20.03 [04:55:04] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.70, 21.31, 20.04 [04:57:00] PROBLEM - services2 APT on services2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [04:57:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.51, 19.86, 19.67 [05:22:52] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.68, 18.92, 18.47 [05:24:49] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 15.53, 17.07, 17.82 [05:38:01] PROBLEM - gluster1 APT on gluster1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:49:03] PROBLEM - puppet2 APT on puppet2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:49:54] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (2 critical updates). [05:54:15] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [05:59:56] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [06:04:29] RECOVERY - mon1 APT on mon1 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [06:09:52] RECOVERY - cloud2 APT on cloud2 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [06:10:52] RECOVERY - services1 APT on services1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:11:42] PROBLEM - db6 Current Load on db6 is WARNING: WARNING - load average: 7.91, 6.25, 5.06 [06:13:39] RECOVERY - db6 Current Load on db6 is OK: OK - load average: 5.67, 5.97, 5.09 [06:20:26] RECOVERY - gluster1 APT on gluster1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:26:34] RECOVERY - db7 APT on db7 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:29:22] PROBLEM - db6 APT on db6 is CRITICAL: APT CRITICAL: 30 packages available for upgrade (1 critical updates). [06:29:25] RECOVERY - phab1 APT on phab1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:33:44] RECOVERY - puppet2 APT on puppet2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:37:12] RECOVERY - cloud1 APT on cloud1 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [06:39:16] RECOVERY - db6 APT on db6 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:43:57] RECOVERY - ns2 APT on ns2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:44:39] RECOVERY - cp6 APT on cp6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMah [06:45:12] PROBLEM - mw7 APT on mw7 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [06:45:13] [02miraheze/services] 07MirahezeSSLBot 036426da4 - BOT: Updating services config for wikis [06:49:00] RECOVERY - cp7 APT on cp7 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:51:06] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:55:56] RECOVERY - ns1 APT on ns1 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:58:44] RECOVERY - services2 APT on services2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:59:50] RECOVERY - cp8 APT on cp8 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [07:50:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMwP [07:50:20] [02miraheze/services] 07MirahezeSSLBot 030f7d24e - BOT: Updating services config for wikis [09:02:30] Hey [09:02:38] I created wiki https://meta.miraheze.org/wiki/Special:RequestWikiQueue/12535 [09:02:38] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [09:02:59] But it isnt marked as approved [09:06:14] Same for https://meta.miraheze.org/wiki/Special:RequestWikiQueue/12534 [09:06:15] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [09:06:21] I got this error [09:06:22] https://i.nuuls.com/LP7gs.png [09:41:22] RECOVERY - bacula2 Bacula Databases db6 on bacula2 is OK: OK: Full, 22675 files, 97.02GB, 2020-06-07 09:40:00 (1.4 minutes ago) [10:04:02] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:07:28] PROBLEM - rdb2 APT on rdb2 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [10:15:45] @MrJaroslavik: File a phab task [10:30:02] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [10:45:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMiC [10:45:15] [02miraheze/services] 07MirahezeSSLBot 03c96c08a - BOT: Updating services config for wikis [10:52:05] revi: looks like Wikimedia community joined us in banning PlavorSeol [10:52:21] yeah, happy day [10:58:22] revi: yep! [10:58:51] * RhinosF1 noticed the other ToU banned Miraheze user once had a GBan request open (that you actually closed) [11:11:18] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMPq [11:11:19] [02miraheze/CreateWiki] 07JohnFLewis 033a178a9 - Change from true to null in returns [11:19:01] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:22:00] PROBLEM - gluster2 APT on gluster2 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [11:29:42] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates). [11:30:18] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMPi [11:30:20] [02miraheze/services] 07MirahezeSSLBot 0390d2fc2 - BOT: Updating services config for wikis [11:36:46] Hi JohnLewis [11:36:58] hi [11:39:37] How is JohnLewis today? [11:42:11] Busy as usual :P You? [11:42:57] Not bad so far [11:44:20] JohnLewis: did you see my stewards@ email [11:44:52] Not yet [11:48:17] RhinosF1: and done [11:49:02] JohnLewis: ty, I should probably wake up fully before reading the request query [11:49:05] Queue [11:51:23] JohnLewis: FYI, based on meta+testwiki's account creation log. There's at least two socks. [11:51:35] * RhinosF1 waves at sario [11:52:21] Good morning [12:24:58] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfM1I [12:24:59] [02miraheze/mediawiki] 07JohnFLewis 03aaf2923 - CW [12:30:08] PROBLEM - dbt1 APT on dbt1 is CRITICAL: APT CRITICAL: 29 packages available for upgrade (1 critical updates). [12:34:05] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.71, 17.90, 12.81 [12:34:19] PROBLEM - mw4 APT on mw4 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [12:36:01] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.80, 16.85, 13.06 [12:41:50] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:41:56] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [12:43:45] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:43:53] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:11:49] [02mw-config] 07Amanda-Catherine opened pull request 03#3099: $wgAllowExternalImages from for nonbinarywiki T5708 - 13https://git.io/JfMMt [13:13:24] Amanda-Catherine/mw-config/patch-2/21c6a7a - Amanda The build passed. https://travis-ci.com/Amanda-Catherine/mw-config/builds/170148843 [13:19:54] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/JfMM0 [13:19:56] [02miraheze/mediawiki] 07paladox 03d0231c0 - Revert "Update PortableInfobox" This reverts commit ddab613588b1e59eb5ea70cc803252bd0bd889dd. [13:19:57] [02miraheze/mediawiki] 07paladox 0323d85d8 - Merge branch 'REL1_34' of github.com:miraheze/mediawiki into REL1_34 [13:20:18] paladox: why [13:20:48] it broke 1.34 support? [13:37:24] Ok [13:39:10] paladox: can you check that exception Id I gave on discord? I don't think Reception123 got round to it [13:39:34] i did, and i've reverted the change that caused it [13:39:45] paladox: oh that was it! [13:39:47] Ty then [14:07:07] thanks to both paladox and rhinos 🙂 [14:08:35] .task T2740#65738 [14:08:35] RhinosF1: I can't find task with id 274065738 [14:08:48] .task 2740 [14:08:50] https://phabricator.miraheze.org/T2740 - Install Html2Wiki Extension [Declined] authored by Ha-Mim, assigned to None [14:09:46] @Cleavages: Paladox did all the work. I just got an execption Id and the requesting server (from a script that Paladox mostly built) [14:10:05] what script? [14:10:24] paladox: the backend response time one [14:10:39] That's how I knew the server (I said on mw6) [14:10:56] oh [14:11:02] that wasn't built by me [14:11:05] that was built by SPF|Cloud [14:13:01] PSA: Google wants longer HSTS duration on miraheze.org https://hstspreload.org/?domain=miraheze.org [14:13:02] [ HSTS Preload List Submission ] - hstspreload.org [14:15:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JfMy0 [14:15:28] [02miraheze/puppet] 07paladox 03bcf0534 - Increase hst to 31536000 See https://hstspreload.org/?domain=miraheze.org [14:15:28] [ HSTS Preload List Submission ] - hstspreload.org [14:15:29] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [14:15:31] [02puppet] 07paladox opened pull request 03#1390: Increase hst to 31536000 - 13https://git.io/JfMyE [14:17:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JfMya [14:17:04] [02miraheze/puppet] 07paladox 0386f97a2 - Update mediawiki.conf [14:17:05] [02puppet] 07paladox synchronize pull request 03#1390: Increase hst to 31536000 - 13https://git.io/JfMyE [14:17:08] paladox: ah [14:22:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfMyF [14:22:02] [02miraheze/puppet] 07paladox 03848e96d - varnish: Add missing headers to *.miraheze.org block [14:22:03] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [14:22:05] [02puppet] 07paladox opened pull request 03#1391: varnish: Add missing headers to *.miraheze.org block - 13https://git.io/JfMyb [14:22:23] [02puppet] 07paladox closed pull request 03#1391: varnish: Add missing headers to *.miraheze.org block - 13https://git.io/JfMyb [14:22:24] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfMyN [14:22:26] [02miraheze/puppet] 07paladox 030f6dcb2 - Merge pull request #1391 from miraheze/paladox-patch-6 varnish: Add missing headers to *.miraheze.org block [14:22:27] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [14:22:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [14:23:08] paladox: wiki subdomains show green on the site Revi linked [14:23:16] (I check QuIRC.mh.o) [14:23:42] So does icinga. As well [14:23:50] Just not Miraheze.org itself [14:24:07] subdomains doesn't really matter cuz they are not on hsts preload list [14:24:20] They not? [14:24:27] they're inferred from mh.o [14:24:32] Ah [14:24:35] so they're not directly listed [14:24:50] They shows a currently listed but mh.o says it could be removed [14:27:00] [02miraheze/puppet] 07paladox pushed 033 commits to 03paladox-patch-5 [+0/-0/±2] 13https://git.io/JfMSk [14:27:02] [02miraheze/puppet] 07paladox 03031a232 - Merge branch 'master' into paladox-patch-5 [14:27:03] [02puppet] 07paladox synchronize pull request 03#1390: Increase hst to 31536000 - 13https://git.io/JfMyE [14:54:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfM9E [14:54:30] [02miraheze/puppet] 07paladox 032cb5ce3 - Revert "varnish: Add missing headers to *.miraheze.org block" This reverts commit 848e96d2c9e555d0ced4b3722eb055221f4b5071. [15:06:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfM95 [15:06:01] [02miraheze/puppet] 07paladox 030ac42d5 - mediawiki: Update strict firewall rules [15:06:03] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [15:06:04] [02puppet] 07paladox opened pull request 03#1392: mediawiki: Update strict firewall rules - 13https://git.io/JfM9d [15:07:30] [02puppet] 07paladox synchronize pull request 03#1392: mediawiki: Update strict firewall rules - 13https://git.io/JfM9d [15:07:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfM9x [15:07:33] [02miraheze/puppet] 07paladox 031850e35 - Update mediawiki.pp [15:09:48] [02puppet] 07paladox closed pull request 03#1392: mediawiki: Update strict firewall rules - 13https://git.io/JfM9d [15:09:49] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JfMHU [15:09:51] [02miraheze/puppet] 07paladox 03840a5d4 - Merge pull request #1392 from miraheze/paladox-patch-6 mediawiki: Update strict firewall rules [15:09:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [15:09:54] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [15:10:48] RECOVERY - bacula2 Bacula Databases dbt1 on bacula2 is OK: OK: Full, 983315 files, 25.71GB, 2020-06-07 15:10:00 (47.0 seconds ago) [15:11:18] !log depool mw4, and repool (deploying strict firewall) [15:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:11:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMHs [15:11:42] [02miraheze/puppet] 07paladox 0324761a4 - mw4: enable strict firewall [15:16:04] !log depool mw5, and repool (deploying strict firewall) [15:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:16:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMHB [15:16:31] [02miraheze/puppet] 07paladox 033a9dff1 - mw5: enable strict firewall [15:20:34] RECOVERY - bacula2 Bacula Phabricator Static on bacula2 is OK: OK: Full, 82779 files, 3.261GB, 2020-06-07 15:19:00 (1.6 minutes ago) [15:21:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMHK [15:22:01] [02miraheze/puppet] 07paladox 03d61b59d - mw6: enable strict firewall [15:22:02] !log depool mw6, and repool (deploying strict firewall) [15:22:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:23:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMHi [15:23:26] [02miraheze/puppet] 07paladox 03614646d - mw7: enable strict firewall [15:24:26] !log depool mw7, and repool (deploying strict firewall) [15:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:26:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMHQ [15:26:27] [02miraheze/puppet] 07paladox 03866ff99 - jobrunner1: enable strict firewall [15:40:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfMQR [15:40:07] [02miraheze/puppet] 07paladox 03da8ef65 - varnish: Also add the ipv6 address to the firewall [15:40:09] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [15:40:10] [02puppet] 07paladox opened pull request 03#1393: varnish: Also add the ipv6 address to the firewall - 13https://git.io/JfMQ0 [15:40:21] [02puppet] 07paladox synchronize pull request 03#1393: varnish: Also add the ipv6 address to the firewall - 13https://git.io/JfMQ0 [15:40:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfMQu [15:40:24] [02miraheze/puppet] 07paladox 0339dacbc - Update mediawiki.pp [15:40:33] [02puppet] 07paladox closed pull request 03#1393: varnish: Also add the ipv6 address to the firewall - 13https://git.io/JfMQ0 [15:40:34] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JfMQz [15:40:36] [02miraheze/puppet] 07paladox 034ab6ef6 - Merge pull request #1393 from miraheze/paladox-patch-6 varnish: Also add the ipv6 address to the firewall [15:40:37] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [15:40:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [15:44:27] !log upgrade ca-certificates libgnutls30 on jobrunner1 [15:44:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:45:00] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:49:20] !log upgrade ca-certificates libgnutls30 on mw[4567] [15:49:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:50:27] RECOVERY - mw5 APT on mw5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:50:41] RECOVERY - mw4 APT on mw4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:51:15] !log upgrade ca-certificates libgnutls30 on dbt1, db[67] [15:51:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:51:27] RECOVERY - mw7 APT on mw7 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:52:20] RECOVERY - dbt1 APT on dbt1 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [15:52:21] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:54:43] !log upgrade ca-certificates libgnutls30 on cp[3678] [15:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:55:48] !log package upgrade not available on cp3 [15:55:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:59:07] Upgrade paladox to version 6.5.9 [15:59:34] lol [15:59:35] !log recreate slave on db7 [15:59:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:59:38] PROBLEM - db7 MySQL on db7 is CRITICAL: Can't connect to MySQL server on '51.89.160.143' (115) [15:59:57] Why is the slave recreating slaves xD jk ❤️ u paladox [16:00:10] heh [16:00:54] @Zppix: come on irc [16:21:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfM7K [16:21:56] [02miraheze/puppet] 07paladox 03f2e3222 - base: Restrict port 5666 to the icinga host only [16:21:58] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [16:21:59] [02puppet] 07paladox opened pull request 03#1394: base: Restrict port 5666 to the icinga host only - 13https://git.io/JfM76 [16:23:01] [02puppet] 07paladox closed pull request 03#1394: base: Restrict port 5666 to the icinga host only - 13https://git.io/JfM76 [16:23:02] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfM7P [16:23:04] [02miraheze/puppet] 07paladox 034743ece - Merge pull request #1394 from miraheze/paladox-patch-6 base: Restrict port 5666 to the icinga host only [16:23:05] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [16:23:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [16:24:00] [02puppet] 07Southparkfan commented on pull request 03#1394: base: Restrict port 5666 to the icinga host only - 13https://git.io/JfM7y [16:29:11] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:29:25] PROBLEM - cp3 Stunnel Http for test2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:02] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:41] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:48] PROBLEM - cp6 APT on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:55] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:56] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:09] PROBLEM - cp3 Stunnel Http for mon1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:12] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:27] paladox: ^ [16:31:37] PROBLEM - cp6 Stunnel Http for mw7 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:31:49] !s [16:31:55] Please wait while I check the status of Miraheze Services. [16:31:56] RhinosF1: Status report finished. There are currently 0 dead services and 6 alive services. To view the full report, say !status. [16:32:02] Good [16:32:13] yeh, i'm aware [16:32:19] it's the firewall [16:32:29] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [16:32:35] RECOVERY - cp3 Stunnel Http for test2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15527 bytes in 1.047 second response time [16:32:47] paladox: ack [16:32:59] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 1.085 second response time [16:33:19] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3282 MB (13% inode=93%); [16:33:20] RECOVERY - cp6 APT on cp6 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:33:26] RECOVERY - cp3 APT on cp3 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:33:27] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [16:33:35] RECOVERY - cp3 Stunnel Http for mon1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 30248 bytes in 1.044 second response time [16:33:39] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 1.081 second response time [16:33:47] RECOVERY - cp6 Stunnel Http for mw7 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.004 second response time [16:34:45] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.01, 23.11, 20.56 [16:42:56] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.97, 23.92, 21.51 [16:44:55] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 19.76, 21.96, 21.07 [16:45:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfM5C [16:45:41] [02miraheze/puppet] 07paladox 0367023f9 - Update nginx-prometheus-exporter to version 0.7.0 [16:45:45] PROBLEM - rdb1 APT on rdb1 is CRITICAL: APT CRITICAL: 3 packages available for upgrade (1 critical updates). [16:46:59] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.22, 18.16, 19.78 [16:47:50] !log upgrade ca-certificates libgnutls30 puppet-agent on rdb[12] [16:48:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:48:23] RECOVERY - rdb2 APT on rdb2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:50:13] RECOVERY - rdb1 APT on rdb1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:50:24] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfM5u [16:50:26] [02miraheze/services] 07MirahezeSSLBot 03f0b12bc - BOT: Updating services config for wikis [16:51:27] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 27.52, 23.00, 21.31 [16:51:32] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.44, 21.59, 19.07 [16:53:34] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 15.44, 20.36, 18.99 [16:55:36] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 14.30, 21.19, 21.34 [17:01:41] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.19, 20.99, 21.02 [17:03:45] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 20.12, 21.18, 21.14 [17:11:08] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.64, 21.62, 19.41 [17:12:26] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 29.49, 22.76, 21.43 [17:13:12] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 18.74, 20.05, 19.07 [17:14:31] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 16.78, 20.68, 20.87 [17:16:30] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.46, 18.72, 20.14 [17:18:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfMdi [17:18:52] [02miraheze/puppet] 07paladox 03b184b9d - services: Simplify firewall rules [17:18:54] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:18:55] [02puppet] 07paladox opened pull request 03#1395: services: Simplify firewall rules - 13https://git.io/JfMdP [17:20:04] [02puppet] 07paladox synchronize pull request 03#1395: services: Simplify firewall rules - 13https://git.io/JfMdP [17:20:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JfMdD [17:20:07] [02miraheze/puppet] 07paladox 034952f1c - Update services.pp [17:22:34] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.72, 20.69, 20.23 [17:24:31] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 17.94, 19.19, 19.71 [17:50:15] [02puppet] 07paladox closed pull request 03#1395: services: Simplify firewall rules - 13https://git.io/JfMdP [17:50:16] [02miraheze/puppet] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/JfMFN [17:50:18] [02miraheze/puppet] 07paladox 03cfaee3b - Merge pull request #1395 from miraheze/paladox-patch-6 services: Simplify firewall rules [17:50:19] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [17:50:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [17:52:02] love this firewall work :P [17:55:17] failed [17:55:47] :( [17:55:53] PROBLEM - db7 Puppet on db7 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [17:55:54] Hey SPF|Cloud [17:56:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMbT [17:56:08] [02miraheze/puppet] 07paladox 03478db71 - services: Fix duplicates [17:56:55] works! [17:57:02] Hello dmehus! If you have any questions, feel free to ask and someone should answer soon. [17:57:41] Great paladox [17:57:45] Hi dmehus [17:58:25] Hey RhinosF1. [17:59:04] dmehus: if you want a Miraheze cloak, wait for PuppyKun or JohnLewis [17:59:19] * RhinosF1 pokes PuppyKun [17:59:47] Sure, sounds good. I assume -testwiki is like a *room* on the miraheze **server**? [18:00:17] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 31.02, 22.80, 18.76 [18:00:56] dmehus: no, Freenode is a server [18:01:05] Each room is called a channel [18:01:30] And #miraheze and #miraheze-* are in the Miraheze namespace [18:01:55] Ah, RhinosF1...thanks, I wondered if Freenode was the actual server, or just a web-based distribution platforrm (like Google Groups is to newsgroups, sort of). [18:02:12] dmehus: Freenode is a server [18:03:04] Thanks, that makes sense. Looks like we've moved away from people running IRC servers on old computers in their basement? ;) [18:03:19] dmehus: yep [18:04:52] Cool. It's too bad that RfC for NordVPN didn't pass. I would've supported that, if all that was going to happen was the miraheze.org main website had an unobstrusive NordVPN sponsor logo like Freenode does with PIA. [18:05:07] That was Zppix's idea [18:05:32] Yeah [18:08:25] I mean, a NordVPN banner on every wiki site would be counter to our mission, but I suspect that's not what Zppix was proposing. Maybe we can find another sponsor...or work out similar deal for a small sponsor logo on the main site with a host. Or apply for charitable tax status in the UK/EU, though that would add lots of filings and paperwork. [18:08:51] We are a LTD dmehus [18:10:23] we a limited company in the UK [18:10:29] *are [18:10:45] Not familiar with that structure, can you issue tax receipts to UK/EU residents? If so, then that would work. My thinking was we could then reach out to a charitable community foundation in each major country that accepts flow-through donations to foreign charitable organizations, so residents in those countries could receive a tax receipt from [18:10:46] that foundation. [18:10:46] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.33, 22.80, 21.53 [18:10:56] Nope [18:11:09] we tried applying for the charitable status, but that failed [18:11:21] that, paladox, that sucks :( [18:11:30] we had to restrict what we hosted in order to fulfil the charitable status, which is not what we wanted. [18:11:36] so we went with limited by gurentee [18:12:48]