[00:03:33] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 20.05, 22.66, 18.52
[00:07:20] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 13.15, 19.07, 18.15
[00:35:11] <Not>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTdA0
[00:35:13] <Not>	 [02miraheze/services] 07MirahezeSSLBot 03864ad3d - BOT: Updating services config for wikis
[01:35:12] <icinga-miraheze>	 PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 7.29, 6.25, 4.39
[01:37:11] <icinga-miraheze>	 RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 4.15, 5.44, 4.33
[01:39:54] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3361: Remove non existing custom domains - 13https://git.io/JTdS7
[02:06:49] <Not>	 [02mw-config] 07Universal-Omega opened pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversation to regex - 13https://git.io/JTFed
[02:12:17] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversation to regex - 13https://git.io/JTFed
[02:23:31] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversation to regex - 13https://git.io/JTFed
[02:27:09] <Not>	 [02mw-config] 07Universal-Omega edited pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversion to regex - 13https://git.io/JTFed
[02:40:08] <Not>	 [02mw-config] 07Universal-Omega edited pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversion to regex - 13https://git.io/JTFed
[02:59:25] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversion to regex - 13https://git.io/JTFed
[03:16:40] <Not>	 [02mw-config] 07Universal-Omega edited pull request 03#3362: Make $wgUrlShortnerAllowedDomains more efficient by automatically allowing the current $wgServer through conversion to regex - 13https://git.io/JTFed
[04:46:11] <Not>	 [02mw-config] 07Universal-Omega opened pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTFmu
[04:50:14] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTFmu
[04:55:02] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTFmu
[05:12:16] <icinga-miraheze>	 PROBLEM - wiki.dom64.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dom64.xyz' expires in 15 day(s) (Thu 19 Nov 2020 05:10:14 GMT +0000).
[05:20:26] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTFON
[05:20:27] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot 0341ff86f - Bot: Update SSL cert for wiki.dom64.xyz
[05:22:13] <icinga-miraheze>	 PROBLEM - wiki.yunachannel.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yunachannel.com' expires in 15 day(s) (Thu 19 Nov 2020 05:14:12 GMT +0000).
[05:30:01] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTF3E
[05:30:02] <Not>	 [02miraheze/ssl] 07MirahezeSSLBot 03bbd5edd - Bot: Update SSL cert for wiki.yunachannel.com
[05:33:09] <icinga-miraheze>	 RECOVERY - wiki.dom64.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dom64.xyz' will expire on Mon 01 Feb 2021 04:20:19 GMT +0000.
[05:42:31] <icinga-miraheze>	 RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Mon 01 Feb 2021 04:29:54 GMT +0000.
[07:47:55] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTFmu
[15:11:20] <hispano76>	 My wiki is more active than ever and was marked as inactive haha XD https://usercontent.irccloud-cdn.com/file/CFrjphsd/mywiki.jpg
[15:13:17] <hispano76>	 I have reactivated it. But is that what happened? And is this happening in other wikis?
[15:18:06] <dmehus>	 @hispano76 It's not clear from that screenshot your message which of your wikis you are referring to. Also, since the usernames are blacked out, I can't verify whether those may be change that shows up in on-wiki Special:RecentChanges. Can you clarify this just a bit?
[15:19:53] <hispano76>	 dmehus: https://hispano76.miraheze.org/wiki/Especial:CambiosRecientes?hidebots=1&limit=500&days=180&urlversion=2 
[15:19:58] <MirahezeBot>	 [ Cambios recientes - Hispano76 ] - hispano76.miraheze.org
[15:21:45] <dmehus>	 @hispano76 thx *looking*
[15:23:52] <icinga-miraheze>	 PROBLEM - mw6 Disk Space on mw6 is WARNING: DISK WARNING - free space: / 2083 MB (10% inode=75%);
[15:24:22] <dmehus>	 @hispano76 That's odd as there were definitely enough changes your wiki shouldn't have gone inactive; however, just troubleshooting, what was:
[15:24:22] <dmehus>	 ```14:49 31 oct 2020 Hispano76 discusión contribs. modificó la configuración (nimbus) de «hispano76wiki» (enable)```
[15:26:20] <MH-Discord>	 <Zppix> Thats odd
[15:26:44] <MH-Discord>	 <Zppix> The script i guess didnt like something
[15:27:12] <MH-Discord>	 <Zppix> @Paladox #Biden2020 how often does the dormancy script run?
[15:28:07] <MH-Discord>	 <Paladox #Biden2020> https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/jobrunner.pp#L72
[15:28:08] <MirahezeBot>	 [ puppet/jobrunner.pp at master · miraheze/puppet · GitHub ] - github.com
[15:28:10] <MH-Discord>	 <Paladox #Biden2020> every 12 h
[15:28:38] <MH-Discord>	 <Zppix> Hispano, if it does again in about 12h lmk
[15:28:38] <MH-Discord>	 <Doug> loves how @Paladox #Biden2020 can pull up a relevant blob within seconds 😛
[15:29:08] <hispano76>	 ok
[15:29:41] <hispano76>	 thanks! :)
[16:40:14] <musikanimal>	 same happened to me. How can I make the "your wiki has been closed" notice go away? It's visible to all readers
[16:40:29] <musikanimal>	 I already unchecked "closed" from Special:ManageWiki/core
[16:40:43] <MH-Discord>	 <Universal Omega> @RhinosF1 @Zppix ^
[16:42:30] <MH-Discord>	 <Zppix> Seen
[16:42:34] <MH-Discord>	 <Zppix> Thats 3 wikis now
[16:45:27] <MH-Discord>	 <Zppix> Musikanimal, mind if I try running the script on your wiki to see if it closes it again?
[16:47:50] <musikanimal>	 what are the risks? right now as a logged out user I get the "this wiki has closed" banner even though I re-opened it
[16:48:09] <hispano76>	 cache?
[16:49:25] <hispano76>	 oh, I see the message in your wiki
[16:49:27] <MH-Discord>	 <Zppix> Musikanimal, nevermind seems i cant run on a specific wiki
[16:49:38] <MH-Discord>	 <Zppix> It may just be cache
[16:50:53] <musikanimal>	 how might I clear this cache? it must be server-side. Shows up in a new browser / device, etc.
[16:51:06] <MH-Discord>	 <Zppix> Link to wiki i forget it
[16:51:15] <musikanimal>	 https://spcodex.wiki/wiki/Main_Page
[16:51:15] <MirahezeBot>	 [ SPCodex - The Smashing Pumpkins wiki ] - spcodex.wiki
[16:53:45] <MH-Discord>	 <Zppix> Thats strange i dont see it
[16:55:36] <MH-Discord>	 <Zppix> Musikanimal, you still see it? Cause logged out or in and dont
[16:56:22] <musikanimal>	 yes, I opened it in a new browser I've never used before
[16:56:30] <MH-Discord>	 <Zppix> Hmm
[16:56:41] <MH-Discord>	 <Zppix> I doubt its a cp* thing since hispano also sees it
[16:56:58] <MH-Discord>	 <Zppix> Let me try this one thing real quick
[16:59:24] <MH-Discord>	 <Zppix> Musikanimal, can you clear cookies (doubt its that but to be safe), and cache and see if its still there
[16:59:39] <MH-Discord>	 <Zppix> I just cleared the wiki’s ManageWiki cache
[17:00:27] <musikanimal>	 I tried in a private window (no cookies), with browser cache cleared, still see it
[17:00:48] <MH-Discord>	 <Universal Omega> So do I
[17:00:52] <MH-Discord>	 <Zppix> Does it still allow edits?
[17:01:14] <MH-Discord>	 <Zppix> I find it odd, that i dont see it
[17:01:23] <musikanimal>	 appears to, yes
[17:01:29] <MH-Discord>	 <Universal Omega> Oh wait I did a second ago, now I don't.
[17:01:42] <MH-Discord>	 <Universal Omega> I don't see it anymore on your wiki.
[17:02:03] <MH-Discord>	 <Zppix> Wonder, why i was never able to see it 🤔
[17:05:08] <Not>	 [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JTbnM
[17:05:09] <Not>	 [02miraheze/dns] 07Reception123 03505387b - add petrawiki.org zone
[17:08:18] <musikanimal>	 I still see it :(
[17:08:23] <MH-Discord>	 <Zppix> I created a task about the premature closures, https://phabricator.miraheze.org/T6403#125414
[17:08:24] <MirahezeBot>	 [ ⚓ T6403 Script closing wikis when it shouldn’t ] - phabricator.miraheze.org
[17:10:21] <MH-Discord>	 <Zppix> @Paladox #Biden2020 any idea why musikanimal stills sees the wiki closed warning thing?
[17:10:43] <MH-Discord>	 <Paladox #Biden2020> yes, if i was to guess it's because of managewiki cache.
[17:10:56] <MH-Discord>	 <Zppix> I reset it
[17:11:07] <MH-Discord>	 <Zppix> So that isnt it
[17:11:08] <MH-Discord>	 <Paladox #Biden2020> yes, but appears the cache isn't being cleared
[17:11:15] <MH-Discord>	 <Paladox #Biden2020> i've reported this but John couldn't reproduce
[17:11:36] <MH-Discord>	 <Paladox #Biden2020> i've had to manually remove the wikis .json file in /srv/mediawiki/w/cache/
[17:11:46] <MH-Discord>	 <Paladox #Biden2020> before
[17:20:42] <musikanimal>	 did we want to try that?
[17:21:18] <MH-Discord>	 <Zppix> @Paladox #Biden2020 ^
[17:21:30] <MH-Discord>	 <Paladox #Biden2020> we can
[17:23:10] <paladox>	 !log remove /srv/mediawiki/w/cache/solarawiki.json on mw* and jobrunner*
[17:23:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:23:12] <paladox>	 musikanimal done ^
[17:25:19] <musikanimal>	 that didn't appear to do it for me :(
[17:25:30] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±2] 13https://git.io/JTbWf
[17:25:32] <Not>	 [02miraheze/ssl] 07Reception123 03fae332e - add www.petrawiki.org cert
[17:25:44] <musikanimal>	 how long does it take for this thing to normally disappear?
[17:30:22] <MH-Discord>	 <Zppix> @Paladox #Biden2020 could it be varnish?
[17:31:16] <MH-Discord>	 <Paladox #Biden2020> It should have disappeared?
[17:31:20] <MH-Discord>	 <Paladox #Biden2020> Doesn’t show for me
[17:32:02] <MH-Discord>	 <Zppix> It never showed for me
[17:32:35] <musikanimal>	 I see it in any new session (e.g. private browsing)
[17:34:38] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:01] <icinga-miraheze>	 PROBLEM - jobrunner2 Puppet on jobrunner2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:11] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:14] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:28] <icinga-miraheze>	 PROBLEM - cp9 Puppet on cp9 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:40] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:44] <icinga-miraheze>	 PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:35:56] <icinga-miraheze>	 PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:36:31] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:36:32] <Reception123>	 ^ paladox urgh I don't get why I never get those right :(
[17:36:33] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:36:36] <Reception123>	 the www ones really confuse me
[17:36:53] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[petrawiki.org],File[petrawiki.org_private]
[17:39:33] <MH-Discord>	 <Zppix> What happened?
[17:41:42] <Not>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JTblg
[17:41:44] <Not>	 [02miraheze/ssl] 07Reception123 03e82d9d1 - attempt fix
[17:52:54] <icinga-miraheze>	 RECOVERY - jobrunner2 Puppet on jobrunner2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures
[17:53:36] <icinga-miraheze>	 RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures
[17:53:39] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures
[17:53:40] <icinga-miraheze>	 RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:53:58] <icinga-miraheze>	 RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:53:58] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:54:13] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:54:29] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:54:39] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:55:17] <icinga-miraheze>	 RECOVERY - cp9 Puppet on cp9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:57:05] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:10:06] <MH-Discord>	 <Doug> > Thats 3 wikis now @Zppix I think this also happened to @revi, possibly, when the closure notice didn't disappear, though that may be an unrelated problem? 🤔
[18:16:52] <musikanimal>	 I just hid the notice in Common.css for now
[18:30:25] <revi>	 for me it was just someone not clicking the button
[18:43:41] <Not>	 [02mw-config] 07JohnFLewis commented on pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTbRj
[18:44:03] <dmehus>	 @revi ah
[21:19:12] <icinga-miraheze>	 PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 155s
[21:23:12] <icinga-miraheze>	 RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s
[21:27:39] <MirahezeBot>	 Hello [Bailif]! If you have any questions, feel free to ask and someone should answer soon.
[21:31:08] <Not>	 [02mw-config] 07Universal-Omega synchronize pull request 03#3363: Migrate $wgRestrictionLevels to ManageWikiSettings - 13https://git.io/JTFmu
[21:31:22] <[Bailif]>	 JohnLewis
[21:31:36] <MH-Discord>	 <Zppix> Can we help ypu?
[21:31:51] <[Bailif]>	 Zppix
[21:33:37] <MirahezeBot>	 Hello Guest55993! If you have any questions, feel free to ask and someone should answer soon.
[21:33:56] <[Bailif]>	 flood
[21:37:54] <Zppix>	 .op
[21:38:01] <Zppix>	 [Bailif]: do you need help?
[21:38:32] <[Bailif]>	 no
[21:38:48] <[Bailif]>	 that took you a long time
[21:39:03] <Zppix>	 .kb [Bailif] LTA
[22:04:09] <Not>	 [02miraheze/ssl] 07Pix1234 created branch 03Remove-mashedpotaytoeswiki 13https://git.io/JTbi2
[22:04:10] <Not>	 [02ssl] 07Pix1234 created branch 03Remove-mashedpotaytoeswiki - 13https://git.io/vxP9L
[22:04:45] <Not>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03Remove-mashedpotaytoeswiki [+0/-1/±0] 13https://git.io/JTbiw
[22:04:47] <Not>	 [02miraheze/ssl] 07Pix1234 033a365fb - Delete www.mashedpotaytoes.com.crt
[22:06:05] <Not>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03Remove-mashedpotaytoeswiki [+0/-0/±1] 13https://git.io/JTbio
[22:06:07] <Not>	 [02miraheze/ssl] 07Pix1234 030fcb6fe - Remove mashedpotaytoeswiki
[22:06:28] <Not>	 [02ssl] 07Pix1234 opened pull request 03#371: Remove mashedpotaytoeswiki, no longer points - 13https://git.io/JTbii
[22:07:21] <Not>	 [02ssl] 07Pix1234 closed pull request 03#371: Remove mashedpotaytoeswiki, no longer points - 13https://git.io/JTbii
[22:07:23] <Not>	 [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JTbiM
[22:07:24] <Not>	 [02miraheze/ssl] 07Pix1234 039054318 - Remove mashedpotaytoeswiki, no longer points (#371) * Delete www.mashedpotaytoes.com.crt * Remove mashedpotaytoeswiki
[22:07:25] <Not>	 [02ssl] 07Pix1234 deleted branch 03Remove-mashedpotaytoeswiki - 13https://git.io/vxP9L
[22:07:27] <Not>	 [02miraheze/ssl] 07Pix1234 deleted branch 03Remove-mashedpotaytoeswiki
[23:14:34] <icinga-miraheze>	 PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 21.16, 20.83, 18.51
[23:16:30] <icinga-miraheze>	 RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 16.46, 20.14, 18.62