[00:25:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRsc
[00:25:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 038d72338 - BOT: Updating services config for wikis
[03:03:09] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2114 MB (10% inode=93%);
[05:02:12] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3863 MB (20% inode=93%);
[06:26:51] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3208 MB (13% inode=94%);
[06:33:15] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 3.35, 3.18, 1.57
[06:37:08] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.30, 1.59, 1.27
[09:56:48] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRBB
[09:56:50] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 03c3f8244 - browndustwiki readonly
[10:00:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRBR
[10:00:11] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03a9535ec - BOT: Updating services config for wikis
[10:03:57] <icinga-miraheze>	 PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[browndust.wiki_private]
[10:04:07] <Reception123>	 !log RENAME browndustwiki to braveninewiki (db + script)
[10:04:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[10:04:49] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JvRBa
[10:04:50] <Not-5a0a>	 [02miraheze/ssl] 07Reception123 03fdd4c17 - remove browndust.wiki cert requested T5256
[10:04:56] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[browndust.wiki_private]
[10:05:02] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[browndust.wiki_private]
[10:05:04] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[browndust.wiki_private]
[10:05:07] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRBV
[10:05:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03c799a28 - BOT: Updating services config for wikis
[10:05:13] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRBw
[10:05:14] <Not-5a0a>	 [02miraheze/mw-config] 07Reception123 0380c679e - rm readonly for browndustwiki
[10:06:43] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[browndust.wiki_private]
[10:12:06] <icinga-miraheze>	 RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[10:13:06] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures
[10:13:07] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:13:19] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[10:14:49] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[10:15:48] <RhinosF1>	 Reception123: everything done on the modules! Only one bug filed as https://phabricator.wikimedia.org/T245811
[10:15:49] <ZppixBot>	 [ ⚓ T245811 TranslatePreparation doesn't detect languages tag already in page ] - phabricator.wikimedia.org
[10:16:07] <Reception123>	 RhinosF1: great :)
[10:16:19] <Reception123>	 RhinosF1: next time maybe use a flood tag though :P
[10:16:28] <RhinosF1>	 Reception123: I did today
[10:16:59] <RhinosF1>	 User rights log; 09:21  RhinosF1 talk contribs block changed group membership for RhinosF1 from administrator and wiki creator to administrator, wiki creator and flooder ‎(+flooder - setting up translations)
[10:17:01] <Reception123>	 RhinosF1: uh maybe you forgot to click save?
[10:17:02] <Reception123>	 https://meta.miraheze.org/wiki/Special:RecentChanges?hidebots=1&translations=filter&limit=100&days=7&enhanced=1&urlversion=2
[10:17:08] <ZppixBot>	 [ Recent changes - Miraheze Meta ] - meta.miraheze.org
[10:17:14] <RhinosF1>	 Reception123: I can see, flag was active
[10:17:16] <Reception123>	 well something definitely didn't go right
[10:17:34] <Reception123>	   09:24 	Training modules/Dealing with online harassment/slides/actionable-and-non-actionable-cases‎ diffhist  +58‎  RhinosF1 talk contribs block Prepared the page for translation
[10:17:34] <Reception123>	      09:21 	User rights log  RhinosF1 talk contribs block changed group membership for RhinosF1 from administrator and wiki creator to administrator, wiki creator and flooder ‎(+flooder - setting up translations)
[10:17:35] <RhinosF1>	 hmm
[10:17:46] <Reception123>	 Yeah this is definitely very strange the flood flag just seems to not have worked
[10:17:56] <RhinosF1>	 ye
[10:27:38] <RhinosF1>	 Please can Stewards, CVT, CoCC please check your email for the message with the subject: "A proposed policy change may affect you"
[10:29:05] <Reception123>	 ok
[11:35:10] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins]
[11:43:07] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures
[13:50:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvR2P
[13:50:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03f49def0 - BOT: Updating services config for wikis
[13:50:48] <icinga-miraheze>	 PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Sun 08 Mar 2020 01:48:28 PM GMT +0000).
[13:51:03] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvR2D
[13:51:05] <Not-5a0a>	 [02miraheze/ssl] 07MirahezeSSLBot 038f12e24 - Bot: Update SSL cert for wiki.zymonic.com
[13:52:01] <RhinosF1>	 Voidwalker: can you close the community directors poll on CN ? It’s overdue
[13:53:32] <Voidwalker>	 I'll take a look, working on a few other things rn
[14:01:17] <ZppixBot>	 Hello rocknroll[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:01:17] <rocknroll[m]>	 It was presidents day monday so a bit late but time for tuneess
[14:01:21] <rocknroll[m]>	 I love rock n' roll
[14:01:21] <rocknroll[m]>	 So put another dime in the jukebox, baby!
[14:01:21] <rocknroll[m]>	 I love rock n' roll
[14:01:27] <rocknroll[m]>	 He smiled, so I got up and asked for his name
[14:01:27] <rocknroll[m]>	 “That don't matter”, he said, “'cause it's all the same”
[14:01:32] <rocknroll[m]>	 Said, “Can I take you home where we can be alone?”
[14:01:32] <rocknroll[m]>	 An' next we were movin' on
[14:01:39] <rocknroll[m]>	 He was with me, yeah me
[14:01:39] <rocknroll[m]>	 Next we were movin' on
[14:01:46] <rocknroll[m]>	 I love rock n' roll
[14:01:46] <rocknroll[m]>	 So put another dime in the jukebox, baby!
[14:01:46] <rocknroll[m]>	 I love rock n' roll
[14:01:53] <rocknroll[m]>	 An we'll be movin' on
[14:01:53] <rocknroll[m]>	 An' singin' that same old song
[14:01:53] <rocknroll[m]>	 Yeah with me, singin'
[14:02:17] <rocknroll[m]>	 An we'll be movin' on
[14:02:17] <rocknroll[m]>	 An' singin' that same old song
[14:02:17] <rocknroll[m]>	 Yeah with me, singin'
[14:02:30] <rocknroll[m]>	 .......
[14:02:33] <rocknroll[m]>	 .....
[14:02:35] <rocknroll[m]>	 Rising up, back on the street
[14:02:36] <rocknroll[m]>	 Did my time, took my chances
[14:02:40] <rocknroll[m]>	 Went the distance, now I'm back on my feet
[14:02:40] <rocknroll[m]>	 Just a man and his will to survive
[14:02:45] <rocknroll[m]>	 So many times it happens too fast
[14:02:45] <rocknroll[m]>	 You trade your passion for glory
[14:02:48] <icinga-miraheze>	 RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Thu 21 May 2020 12:50:57 PM GMT +0000.
[14:02:50] <rocknroll[m]>	 Don't lose your grip on the dreams of the past
[14:02:50] <rocknroll[m]>	 You must fight just to keep them alive
[14:03:04] <rocknroll[m]>	 It's the eye of the tiger
[14:03:04] <rocknroll[m]>	 It's the thrill of the fight
[14:03:06] <rocknroll[m]>	 Rising up to the challenge of our rival
[14:03:06] <rocknroll[m]>	 And the last known survivor
[14:03:08] <rocknroll[m]>	 Stalks his prey in the night
[14:03:08] <rocknroll[m]>	 And he's watching us all
[14:03:14] <rocknroll[m]>	 Face to face, out in the heat
[14:03:14] <rocknroll[m]>	 Hanging tough, staying hungry
[14:03:18] <rocknroll[m]>	 They stack the odds 'til we take to the street
[14:03:18] <rocknroll[m]>	 For the kill with the skill to survive
[14:03:26] <rocknroll[m]>	 It's the eye of the tiger
[14:03:26] <rocknroll[m]>	 It's the thrill of the fight
[14:03:31] <rocknroll[m]>	 Rising up to the challenge of our rival
[14:03:31] <rocknroll[m]>	 And the last known survivor
[14:03:35] <rocknroll[m]>	 Stalks his prey in the night
[14:03:35] <rocknroll[m]>	 And he's watching us all
[14:03:44] <rocknroll[m]>	 The eye of the tiger
[14:03:44] <rocknroll[m]>	 The eye of the tiger
[14:03:47] <rocknroll[m]>	 The eye of the tiger
[14:03:47] <rocknroll[m]>	 The eye of the tiger
[14:06:12] <ZppixBot>	 Hello ghostbusters[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:06:16] <ghostbusters[m]>	 It was presidents day monday so a bit late but time for tuneess
[14:07:26] <ZppixBot>	 Hello buster1[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:07:38] <Voidwalker>	 what
[14:07:46] <buster1[m]>	 It was presidents day monday so a bit late but time for tuneess RhinosF1 Reception123 Voidwalker
[14:10:21] <busterghost[m]>	 If there's somethin' strange in your neighborhood
[14:10:21] <busterghost[m]>	 Who ya gonna call?
[14:10:21] <ZppixBot>	 Hello busterghost[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:10:56] <niko>	 the /x- is random at each reconnect...
[14:11:47] <Voidwalker>	 yeah, that's why I set the extban after
[14:12:03] <niko>	 the ident part stays 
[14:12:28] <niko>	 the username is probably the homeserver he's connecting from
[14:17:42] <ZppixBot>	 Hello bustersghosts[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:17:42] <bustersghosts[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <bustersghosts[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <bustersghosts[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <bustersghosts[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <bustersghosts[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <musictrollboi[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:42] <musictrollboi[m]>	 I ain't afraid o' no ghostI ain't afraid o' no ghost (Voidwalker RhinosF1 Reception123 Zppix)
[14:17:43] <ZppixBot>	 Hello musictrollboi[m]! If you have any questions, feel free to ask and someone should answer soon.
[14:46:06] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%);
[14:46:56] <RhinosF1>	 grumble: did matrix ever block him?
[14:50:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRVr
[14:50:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 0397d8fd7 - BOT: Updating services config for wikis
[14:59:49] <Zppix>	 RhinosF1: i would say the obvious answer is no
[15:00:04] <Zppix>	 (Incase he comes back)
[15:00:35] <RhinosF1>	 Zppix: he won't come back until the next holiday and he could have moved since impeachment day
[15:00:54] <Zppix>	 RhinosF1: he may not be done yet
[15:01:12] <RhinosF1>	 Zppix: he never sings more than a few songs
[15:24:23] <RhinosF1>	 Zppix: what are you doing?
[15:25:26] <Zppix>	 RhinosF1: trying to exempt trusted users from bans that use matrix cause something we may have planned
[15:25:38] <RhinosF1>	 ok
[15:25:45] <Zppix>	 RhinosF1: irccloud tries to fix invalid args in /mode and that happens
[15:25:52] <RhinosF1>	 i see
[16:26:32] <grumble>	 RhinosF1, Zppix: I think so. But they'll almost certainly face the same problem as we've been facing when the troll was using webchats: they're switching ip addresses
[16:26:45] <RhinosF1>	 yeah
[16:34:00] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2607:5300:205:200::17f6/cpweb
[16:36:06] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[18:00:09] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvR1Z
[18:00:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 03ca7ee5d - BOT: Updating services config for wikis
[18:02:42] <RhinosF1>	 Reception123, paladox: is https://phabricator.miraheze.org/T5251 possible
[18:02:44] <ZppixBot>	 [ ⚓ T5251 changing path: example.org/wiki/Page_title → example.org/view/Page_title ] - phabricator.miraheze.org
[18:20:54] <paladox>	 RhinosF1 it can be changed, just not without majour efforts to do it, so no, we won't be doing that.
[18:21:05] <RhinosF1>	 paladox: thought so
[18:25:17] <paladox>	 RhinosF1 i prefer not to be assigned to tasks that i haven't done the work for, thanks
[18:27:20] <RhinosF1>	 Ok
[19:33:19] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 2.09, 3.27, 1.96
[19:37:20] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.79, 1.78, 1.63
[19:39:18] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.42, 1.28, 1.46
[20:34:54] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2a00:d880:5:8ea::ebc7/cpweb, 2607:5300:205:200::17f6/cpweb
[20:36:51] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:00:34] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb
[21:02:34] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:06:31] <RhinosF1>	 paladox: https://phabricator.miraheze.org/T5253#99875
[21:06:34] <ZppixBot>	 [ ⚓ T5253 Locking a wiki out of Special pages is permanent ] - phabricator.miraheze.org
[21:09:17] <paladox>	 !log root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=unicodesubsetswiki --overwrite
[21:09:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:10:21] <paladox>	 !log MariaDB [mhglobal]> delete from mw_namespaces where ns_dbname = 'unicodesubsetswiki';
[21:10:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:10:56] <paladox>	 oh
[21:11:00] <paladox>	 damn me
[21:11:04] <paladox>	 i ran the wrong script
[21:11:28] <paladox>	 !log root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateNamespacesWithDefaults.php --wiki=unicodesubsetswiki --overwrite
[21:11:30] <paladox>	 RhinosF1 ^
[21:11:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:13:04] <RhinosF1>	 paladox: Thanks, Can we blacklist it in MW?
[21:13:09] <paladox>	 yes
[21:14:29] <RhinosF1>	 paladox: can you do that?
[21:15:18] <paladox>	 sure... i can try
[21:18:16] <paladox>	 though, Special pages only have one ns
[21:18:20] <paladox>	 they doin't have talk pages
[21:18:31] <paladox>	 so i doin't know how this can work with MW?
[21:20:08] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvR5y
[21:20:10] <Not-5a0a>	 [02miraheze/services] 07MirahezeSSLBot 032303d87 - BOT: Updating services config for wikis
[21:20:34] <RhinosF1>	 paladox: don’t allow anyone to create a namespace with the name Special
[21:20:54] <paladox>	 that would require a change to MW
[21:21:32] <paladox>	 so not a thing for me as i wouldn't have a clue on how to blacklist this :)
[21:22:35] <RhinosF1>	 paladox: I’ll file a task
[21:22:40] <paladox>	 ok
[22:25:32] <MH-Discord>	 <Owen> For anyone in general (but mostly wikicreators), if anyone fancies reviewing the accuracy of previously closed wiki requests with in the past year but comments only - it would be greatly appreciated for accuracy of future work based on them - https://github.com/OwenBaines/TextClassify/blob/master/res/Miraheze-Requests-Comments-Result.csv
[22:25:33] <ZppixBot>	 [ TextClassify/Miraheze-Requests-Comments-Result.csv at master · OwenBaines/TextClassify · GitHub ] - github.com
[22:28:22] <RhinosF1>	 @Owen: How do we review accuracy
[22:29:19] <MH-Discord>	 <Owen> Assess whether the comment alone is a good indicator of whether it's a valid wiki request or whether it should be reviewed by a human (actual results are at the end of each comment)
[22:32:07] <RhinosF1>	 @Owen: Do I just have to edit the last column appropiately? If so, I can do it hopefully tommorow maybe next week
[22:32:19] <MH-Discord>	 <Owen> yes
[22:32:48] <MH-Discord>	 <RhinosF1> I’ll do that yes
[22:35:47] <RhinosF1>	 Can you check there’s no exact dupes Owen
[22:36:28] <MH-Discord>	 <Owen> Exact dupes are relevant
[22:37:27] <RhinosF1>	 They’ll both be the same based on comment only though
[22:37:55] <MH-Discord>	 <Owen> Yes, but when approaching a problem from a statistical side, duplicates are relevant
[22:38:19] <RhinosF1>	 Ok
[22:40:21] <MH-Discord>	 <Owen> As you deal with an evaluation based on a word-based analysis. So if there are 9 occasions are 'test', but one of "the website is a called 'Counting test'", the input "random test", is more likely to match with the former rather than the latter, which is a more accurate evaluation. If there was just one of each, it would be harder to decipher which is a closer match analytically
[22:40:53] <RhinosF1>	 Ah
[23:03:04] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[23:03:09] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb
[23:03:36] <icinga-miraheze>	 PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.402 second response time
[23:03:39] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.365 second response time
[23:04:14] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3
[23:04:56] <icinga-miraheze>	 PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:05:24] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3
[23:05:24] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3
[23:07:52] <icinga-miraheze>	 RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18695 bytes in 1.594 second response time
[23:08:00] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18696 bytes in 0.527 second response time
[23:08:15] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy
[23:08:50] <icinga-miraheze>	 RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18696 bytes in 1.404 second response time
[23:09:11] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:09:15] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[23:09:25] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy
[23:09:26] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 11 backends are healthy
[23:13:10] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 51.161.32.127/cpweb
[23:13:18] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2607:5300:205:200::17f6/cpweb
[23:15:08] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[23:15:18] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[23:28:50] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRA0
[23:28:52] <Not-5a0a>	 [02miraheze/puppet] 07paladox 039313a1f - Add port 80/443 to matomo firewall
[23:30:13] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JvRAz
[23:30:14] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03b70ca86 - Migrate matomo to mon1
[23:30:16] <Not-5a0a>	 [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS
[23:30:18] <Not-5a0a>	 [02puppet] 07paladox opened pull request 03#1256: Migrate matomo to mon1 - 13https://git.io/JvRAg
[23:30:53] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JvRA2
[23:30:54] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03069336a - Update stunnel.conf
[23:30:56] <Not-5a0a>	 [02puppet] 07paladox synchronize pull request 03#1256: Migrate matomo to mon1 - 13https://git.io/JvRAg
[23:31:19] <Not-5a0a>	 [02puppet] 07paladox closed pull request 03#1256: Migrate matomo to mon1 - 13https://git.io/JvRAg
[23:31:21] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvRAa
[23:31:22] <Not-5a0a>	 [02miraheze/puppet] 07paladox 03ff72c81 - Migrate matomo to mon1 (#1256) * Migrate matomo to mon1 * Update stunnel.conf
[23:31:23] <paladox>	 !log migrating matomo to new infra
[23:31:24] <Not-5a0a>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8
[23:31:25] <Not-5a0a>	 [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS
[23:31:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:53:36] <icinga-miraheze>	 PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures
[23:54:16] <icinga-miraheze>	 PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL - No data received from host
[23:54:17] <icinga-miraheze>	 PROBLEM - misc2 HTTPS on misc2 is CRITICAL: connect to address 81.4.127.174 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[23:54:28] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host
[23:55:50] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for misc2 on cp8 is CRITICAL: HTTP CRITICAL - No data received from host
[23:56:33] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRxJ
[23:56:34] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0373b1c72 - Update nrpe.cfg.erb
[23:57:28] <Not-5a0a>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvRxU
[23:57:30] <Not-5a0a>	 [02miraheze/puppet] 07paladox 0394cf9c9 - Update init.pp