[00:02:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.19, 19.53, 14.39 [00:06:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.36, 20.39, 15.96 [00:08:21] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.48, 16.87, 15.20 [00:27:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jt6x9 [00:27:47] [02miraheze/puppet] 07paladox 0307285cf - gluster::mount: Try and improve caching and performance [00:27:48] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [00:27:50] [02puppet] 07paladox opened pull request 03#1644: gluster::mount: Try and improve caching and performance - 13https://git.io/Jt6xH [00:28:09] [02puppet] 07paladox closed pull request 03#1644: gluster::mount: Try and improve caching and performance - 13https://git.io/Jt6xH [00:28:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt6xQ [00:28:12] [02miraheze/puppet] 07paladox 036c44275 - gluster::mount: Try and improve caching and performance (#1644) [00:28:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [00:28:15] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [00:36:19] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [00:52:19] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:57:14] night :) [00:59:38] Night, SPF|Cloud! [01:33:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt6hh [01:33:08] [02miraheze/puppet] 07paladox 034626229 - gluster prometheus: Tweak sync-interval to every 30s rather than 5s [03:50:14] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - guia.cineastas.pt All nameservers failed to answer the query. [03:57:09] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp10.miraheze.org [07:00:06] !log changed a user's email address on Phab [07:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:43:38] paladox: https://phabricator.miraheze.org/T6843#134879 hm so is this resolved or is it not great that a file was lost in the migration, maybe there's more? [10:43:39] [ ⚓ T6843 Uploaded image is missing (returns 404 Not Found) ] - phabricator.miraheze.org [10:59:13] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JtiGr [10:59:15] [02miraheze/ssl] 07Reception123 0304da6d7 - add wiki.bikinibottomdiscord.org cert [10:59:16] [02miraheze/ssl] 07Reception123 037476790 - add puardian.xyz cert [11:07:19] [02mw-config] 07R4356th opened pull request 03#3721: Add missing full stop - 13https://git.io/JtiG5 [11:08:24] miraheze/mw-config - R4356th the build passed. [11:17:04] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:24:12] PROBLEM - cp12 Puppet on cp12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:25:11] PROBLEM - cp10 Puppet on cp10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:35:21] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:36:22] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:36:57] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:36:59] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:37:25] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:37:44] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:37:47] PROBLEM - cp11 Puppet on cp11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[puardian.xyz_private] [11:52:56] RECOVERY - cp10 Puppet on cp10 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [11:53:32] RECOVERY - cp12 Puppet on cp12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:03:23] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [12:04:04] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 26 packages available for upgrade (2 critical updates). [12:04:11] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [12:04:43] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:55] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:58] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:05:07] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:05:20] RECOVERY - cp11 Puppet on cp11 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [12:16:19] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:21:13] [02mw-config] 07Reception123 closed pull request 03#3721: Add missing full stop - 13https://git.io/JtiG5 [12:21:15] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jtinp [12:21:16] [02miraheze/mw-config] 07R4356th 03a08e726 - Add missing full stop (#3721) [12:22:15] miraheze/mw-config - Reception123 the build passed. [13:08:17] Reception123: yes [13:09:19] Do we know channel logging ain't working? [13:09:21] http://wm-bot.wmflabs.org/dump/%23miraheze-sre.htm [13:09:21] @info [13:09:23] [ #miraheze-sre ] - wm-bot.wmflabs.org [13:11:03] Looks upstream [13:11:09] I can see the txt files [13:11:15] But the nice browser ain't working [13:12:21] Works now [13:18:03] .op [13:18:04] Attempting to OP... [13:19:12] Encoding on that is messed up [13:19:16] .deop [13:19:23] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 12.08, 4.43, 1.91 [13:19:23] RhinosF1: refreshing the page fixed the issue for me a few hours ago. [13:21:21] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.83, 3.60, 1.92 [13:23:21] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.77, 2.56, 1.74 [13:45:21] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.86, 3.26, 1.87 [13:47:21] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.10, 2.40, 1.73 [15:57:54] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 255.76 ms [15:59:58] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 254.36 ms [16:43:16] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 255.76 ms [16:45:20] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 254.85 ms [16:57:23] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.68, 4.05, 2.43 [16:59:23] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.48, 2.94, 2.21 [17:47:41] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 42.77, 24.58, 14.53 [17:47:42] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 32.00, 16.25, 6.76 [17:49:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.32, 22.41, 15.04 [17:51:40] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 9.58, 17.66, 14.19 [17:59:32] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.28, 2.30, 3.73 [18:01:31] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.31, 1.66, 3.32 [18:18:22] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 256.09 ms [18:22:30] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 255.16 ms [18:25:43] paladox: why do we keep getting so many alerts now since migratioh [18:28:45] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 254.97 ms [18:30:48] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 255.82 ms [18:30:53] we don't? [18:30:59] we've always had this many [18:32:13] ^ if anything, it might be a bit less, but is probably roughly the same [18:33:52] maybe im just paying more attention [18:34:23] heh maybe [18:36:53] Evening Zppix [19:03:21] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.89, 6.28, 3.12 [19:04:15] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 256.20 ms [19:07:22] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.32, 3.28, 2.58 [21:42:48] dmehus: can you undelete https://meta.miraheze.org/wiki/Special:ManageWiki/core/madamebadgerwiki#mw-section-handling per request [21:42:49] [ Manage this wiki's core settings - Miraheze Meta ] - meta.miraheze.org [21:44:05] RhinosF1|NotHere, sure, where'd the request come from [21:44:23] dmehus: sre@ [21:44:39] RhinosF1|NotHere, sure, thanks [21:44:55] Did they say a username? [21:45:40] Can you forward the email to stewards@miraheze.org please? [21:47:16] {{Done}} [21:49:59] Ty [22:12:52] What's going on with the relay bot not responding with such commands like this one? [[Special:CentralAuth/Example]] [22:12:53]

https://meta.miraheze.org/wiki/Special:CentralAuth/Example [22:12:54] [ Global account information for Example - Miraheze Meta ] - meta.miraheze.org [22:13:01] Oh wait, never mind, it works. [22:40:24] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.59, 8.04, 4.11 [22:44:20] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.26, 3.80, 3.25 [22:46:16] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.12, 3.13, 3.08 [23:07:07] PROBLEM - highstreethistory.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - highstreethistory.uk reverse DNS resolves to shared1.uk2.webhostserver.co.uk [23:13:36] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found [23:13:39] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [23:20:25] RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: rDNS OK - wiki.tulpa.info reverse DNS resolves to cp11.miraheze.org [23:20:29] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: rDNS OK - podpedia.org reverse DNS resolves to cp10.miraheze.org [23:38:10] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 2.27, 3.43, 1.96 [23:40:10] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.51, 2.36, 1.75