[00:47:49] <Tiger->	 Thank you Paladox for https://phabricator.miraheze.org/T4256. But it seems that it is not working as we expected.
[00:47:50] <ZppixBot>	 [ Login ] - phabricator.miraheze.org
[00:48:26] <paladox>	 oh
[00:48:30] <Tiger->	 We want to import from zhw.wikipedia.org but have en.wikipedia.org now. XD
[00:48:42] <Tiger->	 typo: zh.wikipedia.org
[00:49:06] <paladox>	 ah wgImportSources is a interwiki prefix
[00:49:33] <paladox>	 wikipedia being en.w.org in interwiki
[00:56:53] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnZF
[00:56:55] <Not-7290>	 [02miraheze/mw-config] 07paladox 03c0f9ee2 - Add "zhwwikipedia" to zhdelwiki wgImportSources Change-Id: I0020660b07b2e5c14d93fc2a25d98f8977b2a9b3
[00:57:32] <paladox>	 Tiger- zhw does not exist
[00:58:04] <Tiger->	 ah, it's zh.wikipedia.org
[00:58:09] <paladox>	 thanks
[00:58:29] <JohnLewis>	 in fairness, he did correct himself immediately
[00:59:47] <paladox>	 ah i see
[00:59:49] <paladox>	 sorry Tiger- 
[01:00:03] <paladox>	 fixed with https://review.gerrithub.io/c/miraheze/mw-config/+/453041 Tiger- 
[01:00:17] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnZh
[01:00:19] <Not-7290>	 [02miraheze/mw-config] 07paladox 03fad5d03 - zhwwikipedia -> zhwikipedia Change-Id: I571534804b653160e08c7b9a7544429fa5a8b0b2
[01:01:22] <paladox>	 Tiger- could you see if it works please? :) (it's deployed now!)
[01:03:43] <Tiger->	 I'm doing that 
[01:07:02] <Tiger->	 Seen zhwikipedia in "Source wiki" but got "Import failed: Could not open import file" when importing.
[01:07:58] <paladox>	 Tiger- try again please (think i fixed it)
[01:09:13] <Tiger->	 It works!
[01:09:41] <paladox>	 great!
[01:09:59] <Tiger->	 Thanks a lot :-)
[01:10:48] <paladox>	 your welcome :)
[01:10:54] <paladox>	 thanks for being patient!
[03:35:10] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjncY
[03:35:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03d408319 - BOT: Updating services config for wikis
[04:00:38] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjncC
[04:00:39] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis 03240b5bd - migrate final form to OOUIJS
[04:00:48] <JohnLewis>	 paladox: ^ hows that look?
[04:03:20] <paladox>	 JohnLewis: 👍👍
[04:04:34] <paladox>	 JohnLewis: lgtm
[04:07:51] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjnc4
[04:07:53] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis 03a12cbec - minor fixes
[04:09:10] <Not-7290>	 [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjncB
[04:09:12] <Not-7290>	 [02miraheze/mediawiki] 07JohnFLewis 03e1ab9d3 - update MW
[04:09:37] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[04:26:17] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures
[05:05:59] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjnc1
[05:06:00] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis 038ece8fd - add delete checkbox
[05:10:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjncy
[05:10:15] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 0301e14cf - BOT: Updating services config for wikis
[05:11:14] <Reception123>	 !log UPDATE cw_wikis SET wiki_deleted = 0 WHERE wiki_dbname = "test1wiki";
[05:11:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[05:12:39] <Reception123>	 !log running wikibackups (./wikibackups.sh /home/reception/allpublic.dblist /srv/mediawiki/w/maintenance/dumpBackup.php)
[05:12:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[05:15:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjncH
[05:15:13] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03eee1940 - BOT: Updating services config for wikis
[05:19:39] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[05:35:09] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjncN
[05:35:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03b47912f - BOT: Updating services config for wikis
[05:37:55] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjncp
[05:37:56] <Not-7290>	 [02miraheze/ManageWiki] 07JohnFLewis 03b309277 - don't allow global wiki to be deleted
[05:46:53] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:05:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnCO
[06:05:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 039f4e017 - BOT: Updating services config for wikis
[09:15:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnW7
[09:15:12] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03851f609 - BOT: Updating services config for wikis
[12:33:18] <Reception123>	 !log actually undelete test1wiki (forgot '')
[12:33:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[12:35:10] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn85
[12:35:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 038b78385 - BOT: Updating services config for wikis
[13:37:39] <Not-7290>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn4X
[13:37:41] <Not-7290>	 [02miraheze/puppet] 07paladox 0364db1e5 - Whitelist imgur and sims3store Bug: T4333
[13:53:32] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjn4b
[13:53:34] <Not-7290>	 [02miraheze/mw-config] 07paladox 03bbcc3bb - Set wgPasswordSender within "Moderation Email" ManageWiki.php is loaded before wgConf is set thus will not have wgPasswordSender defined. Change-Id: I1fb80555ee461335f4b3966c5a2931ae01321e48
[14:22:38] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnBO
[14:22:39] <Not-7290>	 [02miraheze/mw-config] 07paladox 0329800e3 - Fix configuration of wgAbuseFilterActions Change-Id: Ic1a90b6e3a40e2d35b3701f45975534a9556fa1a
[14:32:41] <Not-7290>	 [02mw-config] 07paladox closed pull request 03#2640: Move wgAbuseFilterActions to ManageWiki - 13https://git.io/fhbxn
[14:32:47] <Not-7290>	 [02mw-config] 07paladox commented on pull request 03#2640: Move wgAbuseFilterActions to ManageWiki - 13https://git.io/fjnBn
[14:32:49] <Not-7290>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[14:32:51] <Not-7290>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[14:34:07] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnBC
[14:34:08] <Not-7290>	 [02miraheze/mw-config] 07paladox 039ea2762 - Set MustBeLoggedIn to false explicitly in ContactPage This is the default already and avoids notices in the log.
[14:40:29] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±2] 13https://git.io/fjnB0
[14:40:31] <Not-7290>	 [02miraheze/mediawiki] 07paladox 0371c8235 - Update DynamicPageList3 clone link and extension too DynamicPageList3 has been moved to gitlab, so let's update the url.
[14:41:03] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjnBu
[14:41:05] <Not-7290>	 [02miraheze/mediawiki] 07paladox 039557821 - Update PageForms
[14:41:32] <Not-7290>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjnBz
[14:41:33] <Not-7290>	 [02miraheze/mediawiki] 07paladox 038b03629 - Update Cargo
[14:43:26] <paladox>	 !log running lc on mw*
[14:43:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:48:02] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[14:49:39] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[14:49:57] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:53:42] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[15:29:11] <Not-7290>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnRT
[15:29:12] <Not-7290>	 [02miraheze/dns] 07paladox 038536089 - Add "m" alias to panadev.ir - T4311
[15:32:26] <Not-7290>	 [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnRI
[15:32:28] <Not-7290>	 [02miraheze/dns] 07paladox 039f9e273 - Update panadev.ir
[15:36:38] <Not-7290>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[15:36:39] <Not-7290>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjnR3
[15:36:41] <Not-7290>	 [02miraheze/ssl] 07paladox 0394eb02b - Add mobile domain for panadev.ir
[15:36:42] <Not-7290>	 [02ssl] 07paladox opened pull request 03#179: Add mobile domain for panadev.ir - 13https://git.io/fjnRs
[15:37:10] <Not-7290>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjnRG
[15:37:11] <Not-7290>	 [02miraheze/ssl] 07paladox 03a45d70f - Update panadev.ir.crt
[15:37:13] <Not-7290>	 [02ssl] 07paladox synchronize pull request 03#179: Add mobile domain for panadev.ir - 13https://git.io/fjnRs
[15:37:32] <Not-7290>	 [02ssl] 07paladox closed pull request 03#179: Add mobile domain for panadev.ir - 13https://git.io/fjnRs
[15:37:34] <Not-7290>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjnRZ
[15:37:35] <Not-7290>	 [02miraheze/ssl] 07paladox 034120918 - Add mobile domain for panadev.ir (#179) * Add mobile domain for panadev.ir * Update panadev.ir.crt
[15:37:37] <Not-7290>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[15:37:38] <Not-7290>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[17:08:41] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±5] 13https://git.io/fjn0V
[17:08:43] <Not-7290>	 [02miraheze/mw-config] 07paladox 032483897 - Add wmgMFAutodetectMobileView to ManageWiki Change-Id: I485337746a41e992e1877d4bb885a211453c8cb6
[17:08:44] <Not-7290>	 [02miraheze/mw-config] 07paladox 03f4b75d4 - Move MsCatSelect, MsLinks and MsUpload related configs to ManageWiki * Removes $wmgUseMsPackage as it was unused, instead allows users to decide which extension they want to enable. * Adds MsCatSelect and MsLinks to ManageWiki Change-Id: Iba95b6961361768acb073889261de11aa6cde7af
[17:15:55] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[17:23:40] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+2/-1/±1] 13https://git.io/fjn0S
[17:23:42] <Not-7290>	 [02miraheze/mw-config] 07paladox 037b923e6 - Split up ManageWiki.php This moves extensions/skins config to ManageWikiExtensions.php, it also moves settings to ManageWikiSettings.php. The reason why i'm splitting this is so that it's easier to edit the file, the file is becoming too large. Change-Id: Ia0f90a66d94dcb17ec363b1255eaf0b93553178e
[17:23:53] <icinga-miraheze>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:24:52] <travis-ci>	 miraheze/mw-config/master/7b923e6 - thomasmulhall410 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/528216282
[17:25:19] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn05
[17:25:21] <Not-7290>	 [02miraheze/mw-config] 07paladox 03f835150 - Fix syntax Change-Id: Id17e861c2a423fd63d38aca08193e043387986a8
[17:26:51] <travis-ci>	 miraheze/mw-config/master/f835150 - thomasmulhall410 The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/528216689
[17:41:35] <ZppixBot>	 Hello hacker! If you have any questions feel free to ask and someone should answer soon.
[17:42:02] <hacker>	 Heh.. been like 2 years since I've been here.
[17:45:08] <paladox>	 heh
[17:59:04] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjnE4
[17:59:06] <Not-7290>	 [02miraheze/mw-config] 07paladox 039c1731f - Move wgMultiBoilerplateDiplaySpecialPage to ManageWiki Change-Id: I4953848bed1d71333ebcffdd4da2864bf5f16bd8
[18:21:10] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[18:21:45] <icinga-miraheze>	 PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb
[18:23:45] <icinga-miraheze>	 RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online
[18:25:10] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[19:55:10] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnuA
[19:55:11] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03dc92a58 - BOT: Updating services config for wikis
[21:13:38] <SPF|Cloud>	 !log killed old RottenLinks processes on mw3 (s aux | grep RottenLinks | grep -E '(Apr|May)' | awk '{print $2}' | sudo xargs kill)
[21:13:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:19:41] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnz7
[21:19:42] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 0354ae349 - Load vpncloud role on cp3/elasticsearch1/test1
[21:20:36] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjnz5
[21:20:38] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 031925eba - cp3: assign vpncloud ip
[21:25:00] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjnzj
[21:25:01] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 0385b95ac - Fix vpncloud module
[21:25:05] <icinga-miraheze>	 PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet]
[21:25:19] <paladox>	 SPF|Cloud that's wrong :)
[21:25:24] <paladox>	 you need Optional for undef
[21:25:26] <paladox>	 so it's like:
[21:25:32] <paladox>	 Optional[String]
[21:25:41] <paladox>	 not Optional[string] (or String)
[21:25:51] <SPF|Cloud>	 well, the parameter is required
[21:26:05] <paladox>	 ah ok, you will need to change the default for it then :)
[21:26:07] <SPF|Cloud>	 so I thought removing Optional seems right for that
[21:26:30] <SPF|Cloud>	 alright, thanks
[21:26:40] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[21:27:29] <paladox>	 someone's got a dependency cycle going :D
[21:28:02] <SPF|Cloud>	 Failed to connect to bus: No such file or directory
[21:28:37] <paladox>	 hmm
[21:28:42] <paladox>	 sudo service dbus status?
[21:29:11] <SPF|Cloud>	 nevermind, yet another case of 
[21:29:25] <SPF|Cloud>	 'command that refuses to work without sudo yet doesn't display nice error message'
[21:29:32] <paladox>	 oh
[21:30:40] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[21:32:01] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjngI
[21:32:03] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 0356c381f - Change default for vpncloud::server_ip
[21:33:04] <icinga-miraheze>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[21:34:28] <icinga-miraheze>	 PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[21:40:59] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjngW
[21:41:00] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 032c06a26 - Sitenotice for ES server maintenance
[21:43:20] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjng4
[21:43:22] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 035574567 - Restrict sitenotice to wikis using ES
[21:44:12] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjngR
[21:44:14] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 03318fd67 - Update Sitenotice.php
[21:44:29] <SPF|Cloud>	 crap.. that was a syntax error
[21:44:51] <travis-ci>	 miraheze/mw-config/master/5574567 - Southparkfan The build was broken. https://travis-ci.org/miraheze/mw-config/builds/528277346
[21:48:47] <paladox>	 not under php 7.3 :D
[21:50:38] <SPF|Cloud>	 !log graceful elasticsearch, attempting to re-enable TUN/TAP
[21:50:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:57:16] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjngX
[21:57:17] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 031f1f831 - Remove sitenotice
[21:57:32] <SPF|Cloud>	 paladox: but that does not prevent other syntax errors
[21:57:41] <SPF|Cloud>	 you see, it can happen to anyone
[21:57:47] <paladox>	 yup for php7.2 or lower
[21:57:56] <SPF|Cloud>	 I mean any kind of syntax error
[21:58:00] <paladox>	 ah yeh
[21:59:05] <Not-7290>	 [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/fjngH
[21:59:07] <Not-7290>	 [02miraheze/mw-config] 07paladox 03ff232e8 - Move wgRCMaxAge and wgRCLinkDays to ManageWiki Change-Id: I0ff5f30079cdf36c39d0592622ee6582c91fa72d
[21:59:08] <Not-7290>	 [02miraheze/mw-config] 07paladox 03115724c - Merge https://github.com/miraheze/mw-config
[21:59:09] <ZppixBot>	 [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com
[22:05:36] <SPF|Cloud>	 !log vpncloud refuses to work on elasticsearch, possible container/node issue, opened ticket #158843
[22:05:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:21:57] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2e
[22:21:59] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 031efb251 - Maintenance window for Elasticsearch
[22:22:27] <icinga-miraheze>	 RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[22:35:16] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2t
[22:35:17] <Not-7290>	 [02miraheze/services] 07MirahezeSSLBot 03fa04a1c - BOT: Updating services config for wikis
[22:45:37] <paladox>	 !log allowing 10.0.1.1 to acccess port 9200 temporarily on elasticsearch1
[22:45:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:45:55] <paladox>	 oh wrong ip
[22:49:54] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2s
[22:49:56] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 03a79e1c8 - Enable vpncloud on db4
[22:51:11] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2B
[22:51:13] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 03593cdf7 - Add db4 to vpncloud peers
[22:52:55] <paladox>	 !log removed temp firewall rule
[22:52:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:53:15] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn20
[22:53:17] <Not-7290>	 [02miraheze/mw-config] 07Southparkfan 0367b4d03 - Remove site notice
[22:54:57] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2u
[22:54:59] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 034b7a6c9 - Add firewall rule for vpncloud db4
[22:59:08] <icinga-miraheze>	 PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[22:59:33] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:00:41] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:03:02] <icinga-miraheze>	 RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:03:32] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures
[23:04:31] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2r
[23:04:33] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 03ada597f - Add comments to miraheze-internal.net file
[23:04:40] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures
[23:10:28] <paladox>	 !log installing cargo on misc1
[23:10:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:12:27] <Not-7290>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjn2i
[23:12:28] <Not-7290>	 [02miraheze/puppet] 07Southparkfan 038734e2a - fix port numbers
[23:14:31] <icinga-miraheze>	 PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:14:42] <icinga-miraheze>	 PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:15:34] <SPF|Cloud>	 !log delete firewall rules on cp4/test1/db4/elasticsearch1 related to vpncloud port
[23:15:35] <icinga-miraheze>	 PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:15:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:15:51] <Hispano76>	 Is there a maintenance sitenotice?
[23:16:41] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Service[vpncloud@miraheze-internal]
[23:17:10] <Hispano76>	 SPF|Cloud: paladox ?
[23:17:22] <paladox>	 hi, there was one.
[23:17:26] <paladox>	 but it's been removed.
[23:17:58] <Hispano76>	 Okay, so you can edit. :)
[23:18:30] <icinga-miraheze>	 RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:18:37] <icinga-miraheze>	 RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:19:38] <icinga-miraheze>	 RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:24:40] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures
[23:37:39] <Hispano76>	 https://prnt.sc/nkhww1 for a few seconds.. XD
[23:37:40] <ZppixBot>	 [ Screenshot by Lightshot ] - prnt.sc
[23:43:10] <paladox>	 !log remove cargo from misc1
[23:43:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:44:26] <paladox>	 !log switch on TUN on puppet1 - T4346
[23:44:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:45:17] <paladox>	 !log switch on TUN on ns1 - T4346
[23:45:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:46:11] <SPF|Cloud>	 paladox: remember this causes a reboot
[23:46:17] <paladox>	 yup :)
[23:46:30] <paladox>	 lf* and misc4 will need a maint window
[23:47:06] <paladox>	 !log switch on TUN on misc4 - T4346
[23:47:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:47:25] <paladox>	 SPF|Cloud do we need a maint window for misc2?
[23:47:55] <SPF|Cloud>	 well, I wouldn't touch ns1 either without one..
[23:48:07] <paladox>	 SPF|Cloud uh, see log above
[23:48:17] <SPF|Cloud>	 yes, I see, that's why I said that
[23:48:30] <paladox>	 ok
[23:48:57] <SPF|Cloud>	 misc1 is also dns + icinga so critical, needs to be handled with care
[23:49:15] <SPF|Cloud>	 misc2 is redis so reboot causes site issues, also needs maintenance window
[23:49:27] <paladox>	 !log depool mw1
[23:49:31] <SPF|Cloud>	 misc3 is lizardfs so needs master failover (a procedure which must be documented really now!)
[23:49:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:49:42] <paladox>	 SPF|Cloud master fall over??
[23:49:49] <paladox>	 we have no other server to fall over to
[23:49:59] <SPF|Cloud>	 yeah then we can't do it
[23:50:46] <paladox>	 Why not? :)
[23:51:04] <SPF|Cloud>	 because images would become inaccessible?
[23:51:40] <paladox>	 No, the whole site would go down SPF|Cloud (if you did misc4)
[23:51:45] <SPF|Cloud>	 remember, this is not an emergency, we are here to keep the site up as much as possible
[23:52:03] <paladox>	 yup
[23:52:19] <SPF|Cloud>	 not worth risking complete site outages for now
[23:52:54] <SPF|Cloud>	 so I suggest we leave mw3/misc*/lizardfs* alone for today
[23:53:04] <paladox>	 ok
[23:53:32] <SPF|Cloud>	 and preferably depool each cache proxy at its lowest usage moment
[23:53:38] <icinga-miraheze>	 PROBLEM - mw1 Disk Space on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:53:38] <icinga-miraheze>	 PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:53:48] <icinga-miraheze>	 PROBLEM - mw1 php-fpm on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:54:05] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1
[23:54:12] <icinga-miraheze>	 PROBLEM - mw1 MirahezeRenewSsl on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[23:54:15] <icinga-miraheze>	 PROBLEM - mw1 Puppet on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:54:26] <SPF|Cloud>	 .. and silence the host in icinga beforehand
[23:54:27] <icinga-miraheze>	 PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[23:54:29] <icinga-miraheze>	 PROBLEM - mw1 Current Load on mw1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:55:14] <icinga-miraheze>	 PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1
[23:55:28] <icinga-miraheze>	 PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1
[23:56:56] <paladox>	 !log repool mw1
[23:57:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:57:21] <paladox>	 !log depool mw2 (and downtime for 15 minutes)
[23:57:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:58:05] <icinga-miraheze>	 RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy
[23:59:14] <icinga-miraheze>	 RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy