[00:43:19] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4KI [00:43:21] [02miraheze/dns] 07JohnFLewis 03404a833 - geo re-analysis reordering [00:43:29] continually looking to improve performance :P [01:15:47] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4KC [01:15:49] [02miraheze/CreateWiki] 07JohnFLewis 038aa8a35 - add SonarCloud stuff [01:17:36] miraheze/CreateWiki/master/8aa8a35 - John Lewis The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/112693309 [01:18:41] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4KB [01:18:43] [02miraheze/CreateWiki] 07JohnFLewis 03933544f - add SonarCloud stuff [01:21:00] miraheze/CreateWiki/master/933544f - John Lewis The build is still failing. https://travis-ci.com/miraheze/CreateWiki/builds/112693459 [01:21:42] grr [01:26:49] miraheze/CreateWiki/master/933544f - John Lewis The build is still failing. https://travis-ci.com/miraheze/CreateWiki/builds/112693782 [01:28:09] hopefully this time it works... [01:30:02] miraheze/CreateWiki/master/933544f - John Lewis The build was fixed. https://travis-ci.com/miraheze/CreateWiki/builds/112693928 [01:30:35] well, Travis was fixed - SonarCloud marks the extension as a failure ;) [03:06:37] [02miraheze/CreateWiki] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±22] 13https://git.io/fj468 [03:06:38] [02miraheze/CreateWiki] 07JohnFLewis 03e15eec7 - state visibility of methods [03:06:40] [02miraheze/CreateWiki] 07JohnFLewis 038d3566a - fix unused variable usage, NULL->null, clean comments and arrays [03:08:40] miraheze/CreateWiki/master/8d3566a - John Lewis The build was broken. https://travis-ci.com/miraheze/CreateWiki/builds/112699141 [03:08:52] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±13] 13https://git.io/fj46B [03:08:53] [02miraheze/CreateWiki] 07JohnFLewis 03c34d87c - fix unused variable usage, NULL->null, clean comments and arrays [03:10:56] miraheze/CreateWiki/master/c34d87c - John Lewis The build was fixed. https://travis-ci.com/miraheze/CreateWiki/builds/112699281 [03:11:45] miraheze/CreateWiki/master/8d3566a - John Lewis The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/112699294 [04:05:38] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=95%); [04:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj46F [04:35:13] [02miraheze/services] 07MirahezeSSLBot 03e148fb6 - BOT: Updating services config for wikis [06:25:38] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5456 MB (22% inode=95%); [07:06:28] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Diff, 52856 files, 61.57GB, 2019-05-22 07:05:00 (1.5 minutes ago) [07:14:32] PROBLEM - theatlas.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'theatlas.pw' expires in 15 day(s) (Fri 07 Jun 2019 07:11:27 AM GMT +0000). [07:19:50] PROBLEM - knuxwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'knuxwiki.com' expires in 15 day(s) (Fri 07 Jun 2019 07:17:10 AM GMT +0000). [10:07:23] PROBLEM - bacula1 Bacula Static on bacula1 is CRITICAL: CRITICAL: Diff, 306439 files, 21.30GB, 2019-04-21 10:07:00 (4.4 weeks ago) [10:12:39] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.45, 1.75, 0.96 [10:14:36] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.18, 1.60, 1.01 [11:01:48] miraheze/CreateWiki/master/c34d87c - John Lewis The build was fixed. https://travis-ci.com/miraheze/CreateWiki/builds/112699281 [14:14:56] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [14:14:58] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown) [14:15:28] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown) [14:15:29] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 192 failures. Last run 2 minutes ago with 192 failures. Failed resources (up to 3 shown) [14:15:48] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown) [14:16:42] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 177 failures. Last run 3 minutes ago with 177 failures. Failed resources (up to 3 shown): File[pwiki.arkcls.com_private],File[kunwok.org],File[kunwok.org_private],File[www.splat-teams.com] [14:23:29] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:23:45] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [14:24:39] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [14:24:50] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:11] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:28] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:25:30] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:38:23] * Hispano76 saluda [14:43:29] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:32:20] PROBLEM - wiki.contraao.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [18:30:49] !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='forumadmin'; [18:30:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:31:14] !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='chatmod'; [18:31:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:33:50] !log DELETE FROM mw_permissions WHERE perm_dbname='metawiki' AND perm_group='comentadmin'; [18:33:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:43:12] is there a reason you can't do it on wiki? [18:45:35] JohnLewis: the related exts are not/have never been enabled on meta [18:45:50] that doesn't answer my question [18:45:53] MacFan4000 he's refering to the fact you can delete them through the ui [18:46:21] Not when all the perms In said group are added by the ext [18:46:30] you can remove the perms [18:47:15] When the ext isn’t enabled wgAvailablePermissions isn’t set for the perms [18:47:34] Thus they won’t appear in managewiki [18:48:24] paladox: ^^ [18:48:47] MacFan4000 are you sure? Since saving then would remove it from the group. [18:49:16] Last time I did it DB side [18:49:25] if non are select, saving deletes the group [18:49:42] because count ( $perms ) = 0 therefore -> deletion [18:50:09] Plus some of the perms were added to other groups that we use [18:50:18] i just tested [18:50:23] and works [18:50:23] https://test1.miraheze.org/wiki/Special:Log/managewiki [18:50:25] [ ManageWiki log - Test1 ] - test1.miraheze.org [18:50:28] i added then removed flow [18:50:55] which added flow-bot and removed it. [18:51:20] Don’t know why this is happening then, but it is [18:58:25] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40521 MB (11% inode=97%); [19:20:36] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4AV [19:20:38] [02miraheze/ManageWiki] 07JohnFLewis 03e15fc98 - add sonarcloud [19:22:44] [02miraheze/DataDump] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4AP [19:22:45] [02miraheze/DataDump] 07JohnFLewis 0338f93b7 - add sonarcloud [19:23:21] JohnLewis: what's sonarcloud? [19:24:22] Reception123: https://sonarcloud.io [19:24:23] [ SonarCloud ] - sonarcloud.io [19:24:56] Reception123 allows us to find vulns and bugs quicker :P [19:25:04] [02miraheze/DataDump] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4A9 [19:25:06] [02miraheze/DataDump] 07JohnFLewis 036dbf278 - add script to list [19:25:08] sounds nice [19:25:32] I'll have to get familiar with how it works [19:28:44] [02miraheze/DataDump] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4AA [19:28:45] [02miraheze/DataDump] 07JohnFLewis 035edeb8c - 2 spaces not 4 [19:29:32] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4Ap [19:29:33] [02miraheze/ManageWiki] 07JohnFLewis 0392a54e7 - 2 not 4 [19:32:28] these commits are really really lagged [19:34:54] https://www.githubstatus.com/ that's why [19:34:55] [ GitHub Status ] - www.githubstatus.com [19:36:25] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40291 MB (10% inode=97%); [19:36:35] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.teessidehackspace.org.uk and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [19:38:35] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.teessidehackspace.org.uk' expires in 12 day(s) (Tue 04 Jun 2019 07:34:37 PM GMT +0000). [19:43:45] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4xl [19:43:46] [02miraheze/ManageWiki] 07JohnFLewis 035ce527e - update README.md [19:48:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xE [19:48:05] [02miraheze/puppet] 07paladox 03373971f - Add frame-ancestors to varnish CSP [19:48:07] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:48:08] [02puppet] 07paladox opened pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:48:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xa [19:48:38] [02miraheze/puppet] 07paladox 030551860 - Update init.pp [19:48:39] [02puppet] 07paladox synchronize pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:49:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-1/±0] 13https://git.io/fj4xV [19:49:06] [02miraheze/puppet] 07paladox 0352cb291 - Rename whitelist.yaml to csp_whitelist.yaml [19:49:07] [02puppet] 07paladox synchronize pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:49:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/fj4xr [19:49:28] [02miraheze/puppet] 07paladox 03333d763 - Create frame_whitelist.yaml [19:49:29] [02puppet] 07paladox synchronize pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:49:55] [02miraheze/IncidentReporting] 07JohnFLewis pushed 032 commits to 03master [+2/-0/±1] 13https://git.io/fj4xo [19:49:56] [02miraheze/IncidentReporting] 07JohnFLewis 032a5d2d9 - add sonarcloud [19:49:58] [02miraheze/IncidentReporting] 07JohnFLewis 03806313c - Merge branch 'master' of github.com:miraheze/IncidentReporting [19:51:38] [02miraheze/MatomoAnalytics] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4xX [19:51:40] [02miraheze/MatomoAnalytics] 07JohnFLewis 03a1c2347 - add sonarcloud [19:52:03] [02puppet] 07paladox synchronize pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:52:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xM [19:52:06] [02miraheze/puppet] 07paladox 03580569d - Update default.vcl [19:52:15] [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4xD [19:52:17] [02miraheze/RottenLinks] 07JohnFLewis 036ab50fc - add sonarscanner [19:52:48] [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fj4xy [19:52:49] [02miraheze/WikiDiscover] 07JohnFLewis 03ae8d27f - add sonarcloud [19:53:35] [02puppet] 07paladox closed pull request 03#1002: Add frame-ancestors to varnish CSP - 13https://git.io/fj4xu [19:53:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-1/±2] 13https://git.io/fj4x9 [19:53:38] [02miraheze/puppet] 07paladox 0374a7a17 - Add frame-ancestors to varnish CSP (#1002) * Add frame-ancestors to varnish CSP * Update init.pp * Rename whitelist.yaml to csp_whitelist.yaml * Create frame_whitelist.yaml * Update default.vcl [19:53:40] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [19:53:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [19:54:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4xH [19:54:40] [02miraheze/puppet] 07paladox 03900ecac - Remove X-Frame-Options as superseded by CSP [20:04:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xb [20:04:08] [02miraheze/puppet] 07paladox 032308d1f - Update frame, also add X-Frame-Options header to nginx [20:04:09] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [20:04:11] [02puppet] 07paladox opened pull request 03#1003: Update frame, also add X-Frame-Options header to nginx - 13https://git.io/fj4xN [20:04:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xA [20:04:36] [02miraheze/puppet] 07paladox 030367bbb - Update init.pp [20:04:38] [02puppet] 07paladox synchronize pull request 03#1003: Update frame, also add X-Frame-Options header to nginx - 13https://git.io/fj4xN [20:04:38] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is CRITICAL: connect to address astrapedia.ru and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:39] PROBLEM - drag.lgbt - LetsEncrypt on sslhost is CRITICAL: connect to address drag.lgbt and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:40] PROBLEM - wiki.casual4casuals.com - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.casual4casuals.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:40] PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.campaign-labour.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:41] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address infectowiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:41] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.worlduniversityandschool.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:42] PROBLEM - holonet.pw - LetsEncrypt on sslhost is CRITICAL: connect to address holonet.pw and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:43] PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ngscott.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:45] PROBLEM - interno.guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: connect to address interno.guiasdobrasil.com.br and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:46] PROBLEM - wiki.svenskabriardklubben.se - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.svenskabriardklubben.se and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:47] PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.lbcomms.co.za and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:49] PROBLEM - crimesciencewiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address crimesciencewiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:53] PROBLEM - prfm.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address prfm.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:04:56] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.whentheycry.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/fj4xx [20:05:06] [02miraheze/puppet] 07paladox 0372de519 - Update nginx.conf.erb [20:05:07] PROBLEM - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.lostminecraftminers.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:08] [02puppet] 07paladox synchronize pull request 03#1003: Update frame, also add X-Frame-Options header to nginx - 13https://git.io/fj4xN [20:05:13] PROBLEM - wiki.casadocarvalho.net - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.casadocarvalho.net and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:27] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.joust.ro and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:31] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.ciptamedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:46] PROBLEM - aman.info.tm - LetsEncrypt on sslhost is CRITICAL: connect to address aman.info.tm and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:05:47] uh [20:05:48] PROBLEM - wiki.dobots.nl - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.dobots.nl and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:13] [02puppet] 07paladox closed pull request 03#1003: Update frame, also add X-Frame-Options header to nginx - 13https://git.io/fj4xN [20:06:13] PROBLEM - pc-architect.tk - LetsEncrypt on sslhost is CRITICAL: connect to address pc-architect.tk and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fj4pT [20:06:14] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address madzebrascience.wiki and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:15] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: connect to address enc.for.uz and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:16] [02miraheze/puppet] 07paladox 032ecd9b9 - Update frame, also add X-Frame-Options header to nginx (#1003) * Update frame, also add X-Frame-Options header to nginx * Update init.pp * Update nginx.conf.erb [20:06:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [20:06:19] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [20:06:20] PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is CRITICAL: connect to address wikipuk.cl and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:23] PROBLEM - disabled.life - LetsEncrypt on sslhost is CRITICAL: connect to address disabled.life and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:31] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: connect to address guiasdobrasil.com.br and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:38] PROBLEM - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: connect to address pa.gyaanipedia.co.in and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:39] PROBLEM - drag.lgbt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'drag.lgbt' expires in 13 day(s) (Wed 05 Jun 2019 02:46:32 PM GMT +0000). [20:06:39] PROBLEM - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is CRITICAL: connect to address fa.gyaanipedia.co.in and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:40] PROBLEM - wiki.casual4casuals.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.casual4casuals.com' expires in 13 day(s) (Wed 05 Jun 2019 03:12:19 PM GMT +0000). [20:06:40] PROBLEM - wiki.campaign-labour.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.campaign-labour.org' expires in 13 day(s) (Wed 05 Jun 2019 01:47:11 PM GMT +0000). [20:06:42] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 13 day(s) (Wed 05 Jun 2019 02:15:30 PM GMT +0000). [20:06:42] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 12 day(s) (Tue 04 Jun 2019 07:49:09 PM GMT +0000). [20:06:42] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: connect to address 81.4.109.133 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:06:42] PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holonet.pw' expires in 13 day(s) (Wed 05 Jun 2019 01:15:19 PM GMT +0000). [20:06:43] PROBLEM - wiki.ngscott.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ngscott.net' expires in 13 day(s) (Wed 05 Jun 2019 01:16:18 PM GMT +0000). [20:06:45] PROBLEM - interno.guiasdobrasil.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'interno.guiasdobrasil.com.br' expires in 13 day(s) (Wed 05 Jun 2019 03:06:09 PM GMT +0000). [20:06:48] PROBLEM - wiki.lbcomms.co.za - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lbcomms.co.za' expires in 12 day(s) (Tue 04 Jun 2019 03:30:06 PM GMT +0000). [20:06:49] PROBLEM - crimesciencewiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crimesciencewiki.org' expires in 13 day(s) (Wed 05 Jun 2019 03:21:03 PM GMT +0000). [20:06:54] PROBLEM - prfm.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'prfm.wiki' expires in 13 day(s) (Wed 05 Jun 2019 02:35:45 PM GMT +0000). [20:06:55] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:06:56] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 13 day(s) (Wed 05 Jun 2019 02:24:49 PM GMT +0000). [20:07:01] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 51 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:07:07] PROBLEM - wiki.lostminecraftminers.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lostminecraftminers.org' expires in 13 day(s) (Wed 05 Jun 2019 12:36:51 PM GMT +0000). [20:07:13] PROBLEM - wiki.casadocarvalho.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.casadocarvalho.net' expires in 13 day(s) (Wed 05 Jun 2019 03:22:54 PM GMT +0000). [20:07:27] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.joust.ro' expires in 13 day(s) (Wed 05 Jun 2019 02:53:42 PM GMT +0000). [20:07:29] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:07:32] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 13 day(s) (Tue 04 Jun 2019 08:37:52 PM GMT +0000). [20:07:42] PROBLEM - netazar.org - LetsEncrypt on sslhost is CRITICAL: connect to address netazar.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:07:47] PROBLEM - aman.info.tm - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.info.tm' expires in 12 day(s) (Mon 03 Jun 2019 10:49:27 PM GMT +0000). [20:07:49] PROBLEM - wiki.dobots.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dobots.nl' expires in 13 day(s) (Wed 05 Jun 2019 01:21:09 PM GMT +0000). [20:08:13] PROBLEM - pc-architect.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pc-architect.tk' expires in 13 day(s) (Wed 05 Jun 2019 02:39:23 PM GMT +0000). [20:08:15] PROBLEM - madzebrascience.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'madzebrascience.wiki' expires in 13 day(s) (Wed 05 Jun 2019 03:47:29 PM GMT +0000). [20:08:16] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:08:20] PROBLEM - wikipuk.cl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipuk.cl' expires in 12 day(s) (Tue 04 Jun 2019 04:30:05 PM GMT +0000). [20:08:24] PROBLEM - disabled.life - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'disabled.life' expires in 12 day(s) (Tue 04 Jun 2019 07:33:19 PM GMT +0000). [20:08:39] PROBLEM - astrapedia.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'astrapedia.ru' expires in 13 day(s) (Wed 05 Jun 2019 01:50:29 PM GMT +0000). [20:08:39] RECOVERY - pa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Sun 18 Aug 2019 04:10:08 PM GMT +0000. [20:08:40] RECOVERY - fa.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'en.gyaanipedia.co.in' will expire on Sun 18 Aug 2019 04:10:08 PM GMT +0000. [20:08:46] PROBLEM - wiki.svenskabriardklubben.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.svenskabriardklubben.se' expires in 13 day(s) (Wed 05 Jun 2019 01:13:21 PM GMT +0000). [20:10:39] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [20:12:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4pZ [20:12:07] [02miraheze/puppet] 07paladox 03bcd9afb - Fix nginx config [20:12:41] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address infectowiki.com and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:12:42] PROBLEM - holonet.pw - LetsEncrypt on sslhost is CRITICAL: connect to address holonet.pw and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:12:49] PROBLEM - crimesciencewiki.org - LetsEncrypt on sslhost is CRITICAL: connect to address crimesciencewiki.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:12:55] PROBLEM - froggy.info - LetsEncrypt on sslhost is CRITICAL: connect to address froggy.info and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:13:00] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:13:37] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is CRITICAL: connect to address nonciclopedia.org and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:13:42] RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Mon 19 Aug 2019 08:36:02 PM GMT +0000. [20:14:10] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:14:12] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[nginx] [20:14:12] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: connect to address 185.52.1.76 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:13] PROBLEM - misc1 grafana.miraheze.org HTTPS on misc1 is CRITICAL: connect to address 185.52.1.76 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:15] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 13 day(s) (Wed 05 Jun 2019 01:27:29 PM GMT +0000). [20:14:23] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: connect to address 185.52.2.113 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:28] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: connect to address 185.52.3.121 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:31] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guiasdobrasil.com.br' expires in 13 day(s) (Wed 05 Jun 2019 03:04:44 PM GMT +0000). [20:14:39] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: connect to address 185.52.3.121 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:41] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 13 day(s) (Wed 05 Jun 2019 02:15:30 PM GMT +0000). [20:14:42] PROBLEM - holonet.pw - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'holonet.pw' expires in 13 day(s) (Wed 05 Jun 2019 01:15:19 PM GMT +0000). [20:14:45] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:14:46] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:14:49] PROBLEM - crimesciencewiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crimesciencewiki.org' expires in 13 day(s) (Wed 05 Jun 2019 03:21:03 PM GMT +0000). [20:14:49] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:14:50] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4037 bytes in 0.043 second response time [20:14:51] PROBLEM - misc1 HTTPS on misc1 is CRITICAL: connect to address 185.52.1.76 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:14:55] PROBLEM - froggy.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'froggy.info' expires in 12 day(s) (Tue 04 Jun 2019 07:53:58 PM GMT +0000). [20:14:55] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:14:57] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:15:06] PROBLEM - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is CRITICAL: connect to address 81.4.122.121 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:15:07] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4037 bytes in 0.027 second response time [20:15:11] PROBLEM - misc3 lizard.miraheze.org HTTPS on misc3 is CRITICAL: connect to address 185.52.1.144 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:15:20] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:15:23] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 92% [20:15:24] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: connect to address 81.4.127.174 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:15:27] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: connect to address 185.52.1.76 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [20:15:29] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[nginx] [20:15:37] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 13 day(s) (Wed 05 Jun 2019 02:49:22 PM GMT +0000). [20:15:40] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4039 bytes in 0.025 second response time [20:15:45] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [20:15:47] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:15:48] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [20:16:23] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.006 second response time [20:16:42] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1426 bytes in 0.160 second response time [20:16:45] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18665 bytes in 0.026 second response time [20:16:51] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:16:56] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:17:05] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 46% [20:17:07] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18664 bytes in 0.026 second response time [20:17:23] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 17% [20:17:40] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18665 bytes in 0.028 second response time [20:17:45] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:17:45] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:17:48] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 12% [20:18:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:18:45] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:18:46] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:18:49] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:19:05] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 21% [20:19:11] RECOVERY - misc3 lizard.miraheze.org HTTPS on misc3 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 381 bytes in 0.011 second response time [20:19:19] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:19:24] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 41137 bytes in 0.168 second response time [20:19:27] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5806 bytes in 0.026 second response time [20:20:10] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:20:12] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:20:13] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.052 second response time [20:20:13] RECOVERY - misc1 grafana.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 200 OK - 23850 bytes in 0.060 second response time [20:20:28] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17724 bytes in 0.060 second response time [20:20:39] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19073 bytes in 0.160 second response time [20:20:51] RECOVERY - misc1 HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 200 OK - 23850 bytes in 0.021 second response time [20:21:28] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:22:39] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:23:06] RECOVERY - elasticsearch1 elasticsearch-lb.miraheze.org HTTPS on elasticsearch1 is OK: HTTP OK: HTTP/1.1 200 OK - 624 bytes in 0.011 second response time [20:23:29] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:33] I can't do anything on any wiki [20:24:50] That's because rights system broke [20:24:57] Here's an example: https://nonsensopedia.org/wiki/Specjalna:Grupy_u%C5%BCytkownik%C3%B3w [20:24:58] [ Uprawnienia grup użytkowników – Nonsensopedia, polska encyklopedia humoru ] - nonsensopedia.org [20:26:18] paladox: ^ [20:26:30] hmm [20:26:41] works for me in europe [20:26:47] not for me, in Europe [20:27:05] oh, rights system [20:28:26] Meta's also affected: https://meta.miraheze.org/wiki/Special:ListGroupRights [20:28:28] [ User group rights - Miraheze Meta ] - meta.miraheze.org [20:28:41] oh [20:29:10] po209 thanks for telling us [20:29:24] it appears all wiki's are affected :( (we will work on fixing it) [20:55:38] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 1.97, 1.42, 0.78 [20:56:50] Hello Binyomin! If you have any questions feel free to ask and someone should answer soon. [20:57:38] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 1.14, 1.29, 0.82 [20:57:43] Hi, is it possible to know what happened to my account (username: BinyominZeev)? I cannot ask a question on Community noticeboard and cannot reopen my own closed wiki (mafteach.miraheze.org). [20:59:42] !log installing mysql-utilities temporarily on db4 [21:00:43] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/fj4hG [21:00:44] [02miraheze/ManageWiki] 07JohnFLewis 032d22de7 - minr changes for code health + change API around [21:01:42] OK, sorry, now I see in the log, question answered [21:01:49] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Diff, 4265 files, 86.67MB, 2019-05-22 21:00:00 (1.8 minutes ago) [21:03:42] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 3.56, 2.53, 1.50 [21:05:35] RECOVERY - bacula1 Bacula Private Git on bacula1 is OK: OK: Full, 3741 files, 7.200MB, 2019-05-22 21:04:00 (1.6 minutes ago) [21:13:38] PROBLEM - bacula1 Current Load on bacula1 is WARNING: WARNING - load average: 0.22, 1.79, 1.82 [21:15:38] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.03, 1.19, 1.59 [22:00:21] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj4jo [22:00:23] [02miraheze/services] 07MirahezeSSLBot 039313deb - BOT: Updating services config for wikis [22:25:15] !log dropping mw_permissions on db4 [22:25:23] i saved a backup in case [22:42:59] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - check plugin (check_jobqueue) or PHP errors [22:44:30] !log running populateGroupPermissionsWithDefaults.php for all wikis [22:50:54] [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fjBel [22:50:55] [02miraheze/IncidentReporting] 07JohnFLewis 0376d9b0d - add logging and other fixes for code health [23:20:39] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [23:40:00] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjBvt [23:40:02] [02miraheze/mediawiki] 07JohnFLewis 031d6e41a - update IR [23:42:45] [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjBvY [23:42:46] [02miraheze/IncidentReporting] 07JohnFLewis 03ef1139a - fix variables [23:43:08] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjBv3 [23:43:09] [02miraheze/mediawiki] 07JohnFLewis 031616c71 - IR [23:47:26] !log rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* [23:47:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:48:01] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]