[00:34:58] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.43, 4.99, 3.93 [00:35:34] Hmmm [00:35:54] * Zppix pats jbr [00:37:05] PROBLEM - jobrunner4 Current Load on jobrunner4 is CRITICAL: CRITICAL - load average: 6.41, 5.20, 3.77 [00:39:00] PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.95, 5.80, 4.49 [00:59:05] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 3.70, 5.23, 5.52 [01:05:04] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 3.06, 4.01, 4.88 [01:10:58] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 4.00, 5.12, 5.97 [01:18:58] RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 2.81, 3.49, 4.84 [06:02:48] RECOVERY - rdb4 APT on rdb4 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:05:22] RECOVERY - services4 APT on services4 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:06:22] RECOVERY - ldap2 APT on ldap2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:08:47] RECOVERY - bacula2 APT on bacula2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:10:37] RECOVERY - db11 APT on db11 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [06:11:31] RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 32 packages available for upgrade (0 critical updates). [06:13:10] RECOVERY - db12 APT on db12 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [06:14:22] RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [06:14:34] RECOVERY - services3 APT on services3 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:14:36] RECOVERY - cp10 APT on cp10 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:16:04] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [06:17:47] RECOVERY - puppet3 APT on puppet3 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [06:18:11] RECOVERY - mw10 APT on mw10 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:19:29] RECOVERY - test3 APT on test3 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:19:48] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [06:22:58] RECOVERY - mail2 APT on mail2 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [06:26:51] RECOVERY - ns2 APT on ns2 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:29:45] RECOVERY - ns1 APT on ns1 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [06:30:10] RECOVERY - cp12 APT on cp12 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:34:44] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [06:40:37] RECOVERY - phab2 APT on phab2 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:41:01] RECOVERY - gluster3 APT on gluster3 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:42:34] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 97 packages available for upgrade (0 critical updates). [06:42:53] RECOVERY - cp3 APT on cp3 is OK: APT OK: 26 packages available for upgrade (0 critical updates). [06:44:12] RECOVERY - rdb3 APT on rdb3 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [06:44:22] RECOVERY - mw11 APT on mw11 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:45:09] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [06:45:25] RECOVERY - mon2 APT on mon2 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:49:26] RECOVERY - db13 APT on db13 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [06:49:32] RECOVERY - cp11 APT on cp11 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [06:52:35] RECOVERY - mw8 APT on mw8 is OK: APT OK: 29 packages available for upgrade (0 critical updates). [06:54:40] RECOVERY - gluster4 APT on gluster4 is OK: APT OK: 24 packages available for upgrade (0 critical updates). [07:32:06] PROBLEM - ping4 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 337.46 ms [07:34:09] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 257.84 ms [07:58:21] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.41, 1.95, 1.30 [08:00:19] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 3.77, 2.40, 1.52 [08:02:17] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.11, 1.78, 1.39 [08:04:15] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.98, 1.50, 1.34 [08:38:02] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 248.83 ms [08:44:16] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 257.97 ms [08:56:49] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.96 ms [09:00:55] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 100.69 ms [09:31:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtKYU [09:31:44] [02miraheze/puppet] 07paladox 0359f83bd - Set higher ping for dbbackup2 [09:37:42] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.97 ms [09:43:52] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 100.26 ms [09:50:32] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 100.55 ms [10:35:57] @RhinosF1 please do set priorities for tasks https://phabricator.miraheze.org/T6837, there's no need to leave them untriaged [10:35:58] [ ⚓ T6837 Deal with rDNS alerts ] - phabricator.miraheze.org [15:32:13] [02landing] 07phtlay opened pull request 03#62: Fix typo in English message - 13https://git.io/JtKgK [15:33:15] miraheze/landing - phtlay the build passed. [15:34:57] [02landing] 07dmehus commented on pull request 03#62: Fix typo in English message - 13https://git.io/JtKgy [15:38:46] [02miraheze/landing] 07Universal-Omega pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JtKgN [15:38:47] [02miraheze/landing] 07phtlay 03593b8d2 - Fix typo in English message [15:38:49] [02miraheze/landing] 07Universal-Omega 03bc76b7b - Merge pull request #62 from phtlay/master [15:38:50] [02landing] 07Universal-Omega closed pull request 03#62: Fix typo in English message - 13https://git.io/JtKgK [15:39:52] miraheze/landing - Universal-Omega the build passed. [16:42:17] SPF|Cloud: fyi https://phabricator.miraheze.org/T4019#134828 [16:42:18] [ ⚓ T4019 Encrypt Redis traffic ] - phabricator.miraheze.org [16:59:23] we can build php-redis JohnLewis [17:05:44] -rwxr-xr-x 1 root root 3.6M Feb 11 17:04 redis.so [17:05:48] very easy to build it [17:05:50] paladox: I'm aware we can, but there's no pressing need to require it to be built. Alternatively, you can pin the bullseye repo [17:06:13] oh yeh, but wouldn't that version be built against a newer php version? [17:06:30] But it goes against our desire to minimise many repos, local software etc. [17:25:11] sure i guess. [17:25:12] well bullseye i guess gets released in the summer [17:56:08] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtK6O [17:56:10] [02miraheze/WikiDiscover] 07translatewiki 03855bdbb - Localisation updates from https://translatewiki.net. [17:56:11] [ Main page - translatewiki.net ] - translatewiki.net [17:56:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtK63 [17:56:13] [02miraheze/MirahezeMagic] 07translatewiki 03f8afb43 - Localisation updates from https://translatewiki.net. [17:56:14] [ Main page - translatewiki.net ] - translatewiki.net [17:56:14] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtK6s [17:56:16] [02miraheze/DataDump] 07translatewiki 03bc73461 - Localisation updates from https://translatewiki.net. [17:56:17] [ Main page - translatewiki.net ] - translatewiki.net [17:57:07] miraheze/MirahezeMagic - translatewiki the build passed. [17:57:08] miraheze/DataDump - translatewiki the build passed. [17:57:15] miraheze/WikiDiscover - translatewiki the build passed. [18:17:51] PROBLEM - wiki.finnsoftware.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.finnsoftware.net reverse DNS resolves to cdn-185-199-108-153.github.com [18:26:04] [02miraheze/ssl] 07Reception123 pushed 035 commits to 03master [+0/-5/±4] 13https://git.io/JtKi7 [18:26:05] [02miraheze/ssl] 07Reception123 0320718b3 - remove data.maantietaja.org No longer pointing [18:26:07] [02miraheze/ssl] 07Reception123 035d3dc45 - remove evenair.tk No longer pointing [18:26:08] [02miraheze/ssl] 07Reception123 03ab2cb67 - remove www.museummiddelland.nl No longer pointing [18:26:10] [02miraheze/ssl] ... and 2 more commits. [18:30:59] JohnLewis: are you able to access https://phabricator.miraheze.org/T6837#134791 these links? [18:31:00] [ ⚓ T6837 Deal with rDNS alerts ] - phabricator.miraheze.org [18:31:09] Rhinos says he can access them but for me it gives the exception I listed below [18:31:19] and yes, I am logged in ;) [18:31:21] * dmehus is looking [18:31:28] dmehus: you won't be, you're not a syasdmin! [18:31:51] Reception123, guest account :) [18:32:34] Reception123: they work for me [18:32:44] they work for me as well [18:33:13] *assuming you meant the icinga links, not the wikis [18:34:07] JohnLewis: really, that's odd, I'm getting that exception [18:34:23] why is my account being discriminated against :'( [18:34:28] Reception123: what's the full exception? Surely it doesn't go straight into the stack trace [18:34:56] this is everything https://usercontent.irccloud-cdn.com/file/djlE8Vwf/image.png [18:35:13] hhm [18:35:59] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:36:01] oh [18:36:01] JohnLewis: I can see other errors though, like test3 puppet not working [18:36:04] let me fix it [18:36:09] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:36:23] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:36:25] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:36:29] wait [18:36:55] ^ fixing that (the cert thing) [18:37:03] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:37:05] hmm [18:37:12] Reception123 did you log out and back in? [18:37:23] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:37:38] paladox: I can try that, but I did check this morning, was logged out in the meantime and logged in again [18:37:40] but I'll try again [18:37:46] oh [18:38:00] trying again wouldn't make a difference only if you were logged in already [18:38:01] paladox: same thing :( [18:38:16] I can see other errors but not the cert ones [18:38:49] https://github.com/miraheze/puppet/commit/e6e9b611e424659c16cbca6e2b2c55fed3377751#diff-4364f2310d3cf97d4c6c401c24d64109b572f5f2bec995f21971ab941b00f926 [18:38:50] [ restructure icinga group roles · miraheze/puppet@e6e9b61 · GitHub ] - github.com [18:39:06] i think sre-mediawiki is conflicting [18:39:07] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl] [18:40:28] yeah I did imagine it was something like that [18:41:52] paladox: I added myself to sre-mediawiki and I can still view them [18:42:04] JohnLewis you logged out and back in? [18:42:25] nope, but now I have, I can reproduce [18:44:06] Though why is Rhinos able to view them, considering memberships are the same, baring Reception123 should have admin on top - technically Rhinos shouldn't be able to view sslhost if only sre-mediawiki is applying and not sre-mediawiki-view [18:44:28] but dmehus can view them as a 'guest' [18:44:44] guest has view for all [18:45:14] Which should also be applying here as well via sre-mediawiki-view [18:46:47] hmm [18:47:06] yeah, I imagine it's sort of like wiki permissions that the lower level view rights should apply to the higher order groups [18:47:41] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl],File[www.museummiddelland.nl_private] [18:48:02] and why is RhinosF1 on Phabricator but not IRC? If one is active on Phabricator, it holds they should be in #miraheze-sre to communicate with colleagues on issues [18:51:17] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtKXc [18:51:18] [02miraheze/ssl] 07Reception123 03d72bc63 - fix [18:54:04] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:55:28] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl],File[www.museummiddelland.nl_private] [18:56:07] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl],File[www.museummiddelland.nl_private] [19:04:04] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:04:11] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:04:11] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:04:14] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [19:04:56] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:05:30] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:06:30] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:08:33] paladox: after you figure out my issue, do you think you could have a look at https://phabricator.miraheze.org/T6843 ? I'm not sure how the file would've "disappeared" like that [19:08:34] [ ⚓ T6843 Uploaded image is missing (returns 404 Not Found) ] - phabricator.miraheze.org [19:09:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtK1C [19:09:02] [02miraheze/puppet] 07paladox 03bcec70a - icingaweb2: Work around an issue with groups [19:09:04] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:09:05] [02puppet] 07paladox opened pull request 03#1643: icingaweb2: Work around an issue with groups - 13https://git.io/JtK1l [19:09:27] [02puppet] 07paladox edited pull request 03#1643: icingaweb2: Work around an issue with groups - 13https://git.io/JtK1l [19:10:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtK12 [19:10:52] [02miraheze/puppet] 07paladox 03a7347a4 - Update roles.ini.erb [19:10:54] [02puppet] 07paladox synchronize pull request 03#1643: icingaweb2: Work around an issue with groups - 13https://git.io/JtK1l [19:12:34] Reception123, did it get missed in the migration possibly? Did we verify the file sizes copied to the new server matched exactly the file sizes from the old server? [19:13:19] that's why I'm asking paladox, I don't know the answer to that question :D [19:15:31] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:35] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [19:23:51] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:27:16] Reception123, yeah... [19:29:52] JohnLewis, if you get a chance, mind taking look at `polskajestwiki`? I had Zppix run an SQL SELECT query to confirm the date of last RC edit/log entry, which was October 11, 2020. Wiki is still showing as 'active'. How come it never went inactive and closed on December 11, 2020? [19:31:59] JohnLewis, Disregard ^. I figured it out. Some edits were bot edits on February 13th [19:32:10] https://polskajest.miraheze.org/wiki/Specjalna:Ostatnie_zmiany?reviewStatus=manual;auto&limit=50&days=180&enhanced=1&urlversion=2 [19:32:12] [ Ostatnie zmiany – PJJLNA ] - polskajest.miraheze.org [19:39:48] [02puppet] 07JohnFLewis closed pull request 03#1643: icingaweb2: Work around an issue with groups - 13https://git.io/JtK1l [19:40:19] paladox: ^ I'll talk to Reception about that as from messages, seems we need to make a decision about level of access [19:40:37] ok [19:40:46] see your dms [19:50:23] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtKD5 [19:50:24] [02miraheze/puppet] 07JohnFLewis 032948dab - give sre-mediawiki basic monitoring permissions globally with icinga [20:53:28] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.27, 1.75, 1.23 [20:55:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.48, 1.20