[01:22:27] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=93%);
[02:24:02] <icinga-miraheze>	 PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:28:41] <icinga-miraheze>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18700 bytes in 0.810 second response time
[02:31:22] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw5 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[02:32:09] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb
[02:35:05] <icinga-miraheze>	 RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.760 second response time
[02:35:41] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[06:26:36] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3502 MB (14% inode=93%);
[08:12:37] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw5
[08:12:47] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw7 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[08:13:17] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[08:16:34] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy
[08:16:36] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw7 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 1.428 second response time
[08:20:16] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[11:55:07] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDu1
[11:55:09] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot 0308b23ce - BOT: Updating services config for wikis
[13:35:08] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDa5
[13:35:10] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot 03845bf30 - BOT: Updating services config for wikis
[15:28:31] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb
[15:32:19] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[16:08:38] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb
[16:09:26] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:12:40] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[16:13:11] <icinga-miraheze>	 RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.009 second response time
[17:41:40] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvD6K
[17:41:42] <Not-a789>	 [02miraheze/dns] 07paladox 034c37452 - Revert "Remove enc.for.uz" This reverts commit e638d0784378da31b74afe885e94e3db5742ed2d.
[17:41:58] <Not-a789>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvD6i
[17:42:00] <Not-a789>	 [02miraheze/ssl] 07paladox 0363c33d6 - Revert "Remove enc.for.uz (#275)" This reverts commit 189366545b5f1e98c16b7c0520af5a1036862db0.
[17:50:08] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvD6M
[17:50:10] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot 03e48746b - BOT: Updating services config for wikis
[19:12:00] * hispano76 greetings
[20:16:08] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JvD13
[20:16:10] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis 03e47da23 - release 1.0.8, with several fixes and improvements
[20:16:15] <JohnLewis>	 paladox: ^ can you give that a look over please
[20:18:07] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/JvD1n
[20:18:08] <Not-a789>	 [02miraheze/RottenLinks] 07JohnFLewis 03aebc1c6 - release 1.0.8, with several fixes and improvements
[20:20:14] <paladox>	 JohnLewis sure, in a min
[20:25:59] <paladox>	 JohnLewis reviewing now
[20:27:52] <paladox>	 JohnLewis what if the url is not https?
[20:27:55] <paladox>	 (here https://github.com/miraheze/RottenLinks/compare/e47da23f2cb0...aebc1c6806b2#diff-ab2f94fd93012ec523fc6bd074c83c30R46 )
[20:27:56] <ZppixBot>	 [ Comparing e47da23f2cb0...aebc1c6806b2 · miraheze/RottenLinks · GitHub ] - github.com
[20:28:46] <JohnLewis>	 Then there's a lot of questions at hand but a "can't access" error isn't the top of anyone lists
[20:29:08] <JohnLewis>	 A HTTP -> HTTPS redirect would be deemed a pass, even if the HTTPS link 404s
[20:29:21] <JohnLewis>	 So it would make the extension useless in theory
[20:30:03] <paladox>	 ok
[20:31:58] <paladox>	 JohnLewis looks perfect to me
[20:34:50] <RhinosF1>	 paladox: can you update mw.org page
[20:35:28] <RhinosF1>	 If not, ill do later
[20:35:47] <paladox>	 Update what?
[20:37:56] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JvDMk
[20:37:58] <Not-a789>	 [02miraheze/mediawiki] 07JohnFLewis 03450ccc7 - update RL
[20:38:37] <RhinosF1>	 paladox: mediawiki.org/wiki/Extension:RottenLinks
[20:38:49] <RhinosF1>	 Make sure the infobox params are right
[20:38:57] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb
[20:39:10] <RhinosF1>	 And maybe add version + updated params are there
[20:39:55] <JohnLewis>	 10 extension tasks resolved in 24 hours :D
[20:40:22] <RhinosF1>	 JohnLewis: great!
[20:40:49] <RhinosF1>	 JohnLewis: I’ve already updated the extension page for incident reporting
[20:41:00] <RhinosF1>	 I think Reception123 had a list of ones to fo
[20:41:02] <RhinosF1>	 Do
[20:41:39] <paladox>	 done
[20:42:32] <RhinosF1>	 paladox: thx
[20:42:59] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[20:43:40] <RhinosF1>	 paladox: can you do DataDump as well
[20:44:07] <RhinosF1>	 JohnLewis: https://www.mediawiki.org/w/index.php?title=Topic:Vhhh43vdy0txvelp&action=history - thoughts?
[20:44:08] <ZppixBot>	 [ Width of Special:RottenLinks on Extension talk:RottenLinks ] - www.mediawiki.org
[20:44:26] <paladox>	 RhinosF1 don't forget you can do it :)
[20:44:52] <RhinosF1>	 If you don’t, i’ll get round to it
[20:45:18] <RhinosF1>	 You missed the infobox btw
[20:49:45] <JohnLewis>	 https://phabricator.miraheze.org/T3706
[20:49:46] <ZppixBot>	 [ ⚓ T3706 Define a maximum width for the colums ] - phabricator.miraheze.org
[20:55:38] <RhinosF1>	 JohnLewis: replied to them
[21:03:19] <paladox>	 i've done the infobox now
[21:03:50] <MikeV>	 We're seeing frequent errors and delays editing on nenawiki.org and ndg.nenawiki.org.  "Unknown error, HTTP status 0"  and "Error loading data from server: Could not connect to the server...."
[21:08:33] <Zppix>	 MikeV:  Known
[21:08:40] <Zppix>	 MikeV:  not much we can do honestly
[21:10:03] <MikeV>	 Zppix: How about dishonistly?
[21:10:33] <Zppix>	 MikeV:  heh, that would be for paladox to say
[21:12:32] <MikeV>	 Zppix: Is this permanent?
[21:13:39] <Zppix>	 MikeV:  No, in fact i think I may of found a fix, do we (miraheze staff) have permission to migrate your wikis to the new infrastructure?
[21:15:16] <paladox>	 MikeV can you grant me admin rights please? I cannot seem to edit.
[21:16:17] <MikeV>	 Zppix:  Without much background on all of the implications, I would say YES, if our sites will perform better.
[21:16:42] <Zppix>	 paladox:  ^
[21:16:47] <paladox>	 ok, doing
[21:17:12] <MikeV>	 Zippix: When would the sites will be out of service?
[21:17:56] <Zppix>	 MikeV:  paladox would be the one handling it 
[21:17:57] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JvDDP
[21:17:59] <Not-a789>	 [02miraheze/mw-config] 07paladox 03fc31a12 - Migrate nenawikiwiki to db7
[21:18:00] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3
[21:18:02] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2947: Migrate nenawikiwiki to db7 - 13https://git.io/JvDDX
[21:19:05] <Zppix>	 MikeV:  I would think it shouldnt take too long
[21:19:31] <Zppix>	 MikeV:  I would make sure any planned edits are saved offline
[21:19:34] <travis-ci>	 miraheze/mw-config/paladox-patch-4/fc31a12 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/665617602
[21:20:51] <MikeV>	 paladox: You're an admin.
[21:20:58] <paladox>	 thanks
[21:21:45] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2947: Migrate nenawikiwiki to db7 - 13https://git.io/JvDDX
[21:21:47] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDDQ
[21:21:48] <Not-a789>	 [02miraheze/mw-config] 07paladox 0338748fe - Migrate nenawikiwiki to db7 (#2947)
[21:21:52] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3
[21:21:53] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4
[21:22:46] <RhinosF1>	 paladox: thought sysadmin had global edit access. Weird
[21:23:03] <paladox>	 we do, but not if a wiki changes things locally i think
[21:23:34] <Zppix>	 I've never had an issue with editing any wiki, then again I think CVT is allowed to edit regardless of local changes
[21:24:12] <RhinosF1>	 paladox: there’s few changes that block it. Must be one of them.
[21:24:18] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDDx
[21:24:20] <Not-a789>	 [02miraheze/dns] 07paladox 036727cf2 - Migrate nenawiki.org to cp7
[21:24:36] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDDh
[21:24:38] <Not-a789>	 [02miraheze/dns] 07paladox 03b7cef24 - Update nenawiki.org
[21:25:45] <paladox>	 MikeV does it work now?
[21:25:52] <RhinosF1>	 paladox: I see it, you should be able to edit your talk page without though
[21:26:01] <paladox>	 oh
[21:26:18] <paladox>	 Migrating ndg.nenawiki.org now
[21:27:42] <Zppix>	 !log (paladox): migrated ndg and nena wikis
[21:27:43] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDyk
[21:27:44] <Not-a789>	 [02miraheze/mw-config] 07paladox 03c2703d5 - Migrate ndgwiki to db7
[21:28:19] <Not-a789>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDyt
[21:28:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:28:21] <Not-a789>	 [02miraheze/dns] 07paladox 03f34b82e - Migrate ndg.nenawiki.org to cp7
[21:28:32] <Zppix>	 paladox:  should MirahezeLogbot  be taking that long to react?
[21:28:43] <paladox>	 Zppix yes
[21:28:51] <Zppix>	 why paladox?
[21:28:53] <paladox>	 its using meta api, so if meta is slow, so is the bot
[21:29:16] <MikeV>	 So far, so good.
[21:29:20] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[21:29:42] <Zppix>	 MikeV:  if you end up with any errors about not being able to reach a server just clear your cache 
[21:30:14] <MikeV>	 Zppix & paladox: Thanks!
[21:30:53] <Zppix>	 MikeV:  of course, as always if you have any further issues/questions please blame paladox for any issues, and we can answer any questions :)
[21:32:09] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown)
[21:33:17] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures
[21:35:23] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:35:31] <paladox>	 !log drop database ndgwiki and nenawikiwiki on db4
[21:35:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:50:33] <hispano76>	 strange https://prnt.sc/rku13x
[22:50:34] <ZppixBot>	 [ Screenshot by Lightshot ] - prnt.sc
[22:52:18] <hispano76>	 (privadowiki)
[22:55:49] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvD9b
[22:55:50] <Not-a789>	 [02miraheze/puppet] 07paladox 03221f09b - Update default.vcl
[23:08:24] <hispano76>	 MediaWiki\Storage\NameTableAccessException  seems to have a problem with at least one name space (USER)
[23:19:03] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvDHt
[23:19:05] <Not-a789>	 [02miraheze/puppet] 07paladox 03379a948 - Update db6.yaml
[23:19:56] <paladox>	 hispano76 where do you see that? Page?
[23:23:18] <hispano76>	 paladox:  Sometimes when trying to restore edits to a page, or preview the page with other content    https://privado.miraheze.org/w/index.php?title=Especial:Restaurar&target=Usuario%3AHispano76%2Fzona+de+pruebas&timestamp=20200322002005 
[23:23:21] <ZppixBot>	 [ Error de permisos - Privado ] - privado.miraheze.org
[23:25:04] <icinga-miraheze>	 PROBLEM - db6 MySQL on db6 is CRITICAL: Can't connect to MySQL server on '51.89.160.130' (115)
[23:26:13] <icinga-miraheze>	 PROBLEM - db6 Puppet on db6 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file
[23:28:09] <paladox>	 !log MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (2, 'css'); - db7
[23:28:18] <paladox>	 !log MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (5, 'json'); - db7
[23:28:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:28:27] <paladox>	 !log MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (4, 'Scribunto'); - db7
[23:28:34] <paladox>	 !log MariaDB [privadowiki]> INSERT INTO content_models (`model_id`, `model_name`) VALUES (3, 'javascript'); - db7
[23:28:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:28:57] <paladox>	 hispano76 fixed
[23:29:08] <hispano76>	 ah, ok. Thanks, thanks paladox  
[23:29:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:29:13] <paladox>	 yw :)
[23:29:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:33:57] <icinga-miraheze>	 PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:37:42] <icinga-miraheze>	 RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures