[00:05:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [00:07:40] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:40] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:28] https://phabricator.miraheze.org/T4352#99160 [00:08:29] [ ⚓ T4352 Create the Miraheze-feed-es channel ] - phabricator.miraheze.org [00:09:21] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:09:42] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 5.402 second response time [00:09:43] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 5.498 second response time [00:19:06] I just realized that it's been almost a year since that task [00:19:11] XD [00:19:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAf [00:19:15] [02miraheze/puppet] 07paladox 03e3c43d0 - postgresql: Add support for buster [00:19:16] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:19:18] [02puppet] 07paladox opened pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:19:39] [02puppet] 07paladox synchronize pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:19:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAT [00:19:42] [02miraheze/puppet] 07paladox 037a1e27b - Update master.pp [00:20:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAI [00:20:51] [02miraheze/puppet] 07paladox 0336dc852 - Update slave.pp [00:20:53] [02puppet] 07paladox synchronize pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:21:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAt [00:21:32] [02miraheze/puppet] 07paladox 039cdaef6 - Update user.pp [00:21:34] [02puppet] 07paladox synchronize pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:22:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [00:23:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAm [00:23:56] [02miraheze/puppet] 07paladox 0354e0816 - Update postgis.pp [00:23:58] [02puppet] 07paladox synchronize pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:24:19] [02puppet] 07paladox closed pull request 03#1190: postgresql: Add support for buster - 13https://git.io/JvnAJ [00:24:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JvnAY [00:24:22] [02miraheze/puppet] 07paladox 0313be799 - postgresql: Add support for buster (#1190) * postgresql: Add support for buster * Update master.pp * Update slave.pp * Update user.pp * Update postgis.pp [00:24:23] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:24:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:24:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:29:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvnAW [00:29:49] [02miraheze/puppet] 07paladox 034670fd2 - postgresql: Fix pacage for buster [00:30:17] typo... [00:35:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [00:35:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvnAa [00:35:24] [02miraheze/puppet] 07paladox 0367efab4 - Update server.pp [00:38:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:41:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+1/-1/±0] 13https://git.io/JvnA1 [00:41:23] [02miraheze/puppet] 07paladox 03cf9f76c - postgresql: Move ssl.conf to template [00:41:24] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:41:26] [02puppet] 07paladox opened pull request 03#1191: postgresql: Move ssl.conf to template - 13https://git.io/JvnAM [00:42:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnAy [00:42:17] [02miraheze/puppet] 07paladox 039e939c4 - Update server.pp [00:42:18] [02puppet] 07paladox synchronize pull request 03#1191: postgresql: Move ssl.conf to template - 13https://git.io/JvnAM [00:42:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvnA9 [00:42:37] [02miraheze/puppet] 07paladox 03390c45b - Update ssl.conf.erb [00:42:38] [02puppet] 07paladox synchronize pull request 03#1191: postgresql: Move ssl.conf to template - 13https://git.io/JvnAM [00:43:01] [02puppet] 07paladox closed pull request 03#1191: postgresql: Move ssl.conf to template - 13https://git.io/JvnAM [00:43:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JvnAH [00:43:04] [02miraheze/puppet] 07paladox 0305eeab5 - postgresql: Move ssl.conf to template (#1191) * postgresql: Move ssl.conf to template * Update server.pp * Update ssl.conf.erb [00:43:06] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:43:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:04:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvnpL [02:04:08] [02miraheze/puppet] 07paladox 034b59717 - Set puppetserver_hostname to puppet2.miraheze.org (for puppet2 only) [02:20:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:21:11] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [02:23:12] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:24:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:32:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [02:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:42:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [02:44:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:22:50] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:24:50] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:34:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvnhv [03:34:41] [02miraheze/puppet] 07paladox 03fac050e - Add puppet2 [03:36:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvnhf [03:36:28] [02miraheze/puppet] 07paladox 03a3d012b - Update puppet2.yaml [06:24:17] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [06:26:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:27:19] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3029 MB (12% inode=94%); [09:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcUQ [09:00:10] [02miraheze/services] 07MirahezeSSLBot 034ad806e - BOT: Updating services config for wikis [09:02:30] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [09:04:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:22:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [09:24:59] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [09:26:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [09:26:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [10:23:19] Hello arsenalfan! If you have any questions, feel free to ask and someone should answer soon. [10:24:01] hi guys, just setting up my site still. [10:24:39] is it possible to change the /wiki url to something else like /something [10:27:00] so currently, domains follow this pattern: abc.miraheze.org/wiki/myarticle but would like abc.miraheze.org/something/myarticle [10:30:22] 'changing the path' of a wiki [10:30:24] https://www.mediawiki.org/wiki/Manual:Short_URL#Moving_a_wiki_from_/wiki_to_/w [10:30:25] [ Manual:Short URL - MediaWiki ] - www.mediawiki.org [11:21:19] arsenalfan: I don’t think we’d be able to do that [11:22:48] ok thank you [11:23:13] are there any internalisation addons that mangle the path [11:23:39] maybe i would prefer the mangle [11:27:40] (i want to hide the fact its a wiki) [11:28:03] to website viewers, but not editors [11:29:02] arsenalfan: you could you css to hide a lot of the links [11:29:36] cheers. but i mean on the url path for every article [11:30:25] You can open a phab task to ask but there’s no guarntee itll be done. Not my call. [11:31:05] thank you [11:31:30] other potentially risky question [11:32:29] if we donate a certain amount, can you/we remove the fundraiser banner from our page (im guessing i could do with css). guessing this may be against the community etiquette [11:36:43] 😳 [11:48:57] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [11:52:23] arsenalfan: you’ll have to remove it yourself using css [11:53:47] OK, thats great if thats acceptable for you. if/when we launch ill chip in real money, beyond the small amount i have already [11:56:56] arsenalfan: no problem, it’s your wiki but of course we’d like the fundraiser to be well advertised [11:57:44] we will fundraise ourself, then pass on the cash! [11:57:55] thanks you guys are angels [13:57:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [13:59:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:09:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvcOa [15:09:46] [02miraheze/puppet] 07paladox 030c219c2 - db6: Add to manifests [15:09:47] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [15:09:49] [02puppet] 07paladox opened pull request 03#1192: db6: Add to manifests - 13https://git.io/JvcOV [15:10:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvcOr [15:10:52] [02miraheze/puppet] 07paladox 032c0c3e2 - Create db6.yaml [15:11:14] [02puppet] 07paladox closed pull request 03#1192: db6: Add to manifests - 13https://git.io/JvcOV [15:11:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcOo [15:11:17] [02miraheze/puppet] 07paladox 03ccc4028 - db6: Add to manifests (#1192) [15:15:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcO1 [15:15:46] [02miraheze/puppet] 07paladox 03cfe4de7 - puppetdb::database: Add support for buster [15:29:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jvc3m [15:29:38] [02miraheze/puppet] 07paladox 034fce9d5 - mariadb: Fix installing on new installs [15:29:39] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [15:29:41] [02puppet] 07paladox opened pull request 03#1193: mariadb: Fix installing on new installs - 13https://git.io/Jvc3Y [15:31:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jvc3G [15:31:51] [02miraheze/puppet] 07paladox 03d66094c - Update packages.pp [15:31:53] [02puppet] 07paladox synchronize pull request 03#1193: mariadb: Fix installing on new installs - 13https://git.io/Jvc3Y [15:32:27] [02puppet] 07paladox synchronize pull request 03#1193: mariadb: Fix installing on new installs - 13https://git.io/Jvc3Y [15:32:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/Jvc3c [15:32:30] [02miraheze/puppet] 07paladox 03fd41049 - Update packages.pp [15:32:42] [02puppet] 07paladox closed pull request 03#1193: mariadb: Fix installing on new installs - 13https://git.io/Jvc3Y [15:32:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jvc3C [15:32:45] [02miraheze/puppet] 07paladox 039197038 - mariadb: Fix installing on new installs (#1193) * mariadb: Fix installing on new installs * Update packages.pp * Update packages.pp [15:32:46] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [15:32:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:42:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb [15:44:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:53:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:55:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:23:18] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [16:25:19] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcGJ [16:40:08] [02miraheze/services] 07MirahezeSSLBot 0367d04b7 - BOT: Updating services config for wikis [16:43:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [16:45:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:12:06] PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldsofweary.com' expires in 15 day(s) (Tue 25 Feb 2020 06:09:48 PM GMT +0000). [18:12:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvcns [18:12:21] [02miraheze/ssl] 07MirahezeSSLBot 03b247fc7 - Bot: Update SSL cert for wiki.worldsofweary.com [18:14:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [18:18:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:22:01] RECOVERY - wiki.worldsofweary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldsofweary.com' will expire on Sat 09 May 2020 05:12:13 PM GMT +0000. [18:22:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:24:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcnX [18:30:09] [02miraheze/services] 07MirahezeSSLBot 0390a6ca7 - BOT: Updating services config for wikis [18:40:34] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:42:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [18:48:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:49:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:52:54] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:53:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [18:57:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:58:58] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:03:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:06:25] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:58] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:07:47] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:33] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 8.248 second response time [19:09:07] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 8.197 second response time [19:09:53] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 6.384 second response time [19:11:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [19:11:51] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:17:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [19:19:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:21:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:23:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvcch [19:23:13] [02miraheze/puppet] 07paladox 039b82010 - db6: Set puppetmaster to puppet2 [19:23:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:25:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:27:38] paladox|UKInEU: what’s icinga moaning at [19:27:52] I guess mw3? [19:28:51] seems recovered [19:30:10] paladox|UKInEU: why though [19:33:59] i'm not sure [19:37:51] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:38:34] paladox|UKInEU: again ^ [19:38:47] things work for me [19:39:45] paladox|UKInEU: test and meta wont load for me [19:39:48] Taking forever [19:40:11] Yeh, it'll be slow [19:40:31] paladox|UKInEU: can we recover things? [19:40:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [19:40:50] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [19:40:57] I've been trying to do that for the last few months. [19:42:53] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:43:03] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [19:43:09] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:43:29] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:45:09] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18676 bytes in 1.928 second response time [19:48:21] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:49:15] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 1.399 second response time [19:49:53] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.738 second response time [19:50:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvcCX [19:50:08] [02miraheze/puppet] 07paladox 03778e275 - cloud1: Set puppetmaster to puppet2 [19:50:35] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 4.342 second response time [19:58:18] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:58:42] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:02:47] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [20:04:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:12:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [20:12:41] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:34] PROBLEM - cp3 Stunnel Http for mw2 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:14:11] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:14:36] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.240 second response time [20:15:31] RECOVERY - cp3 Stunnel Http for mw2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 0.493 second response time [20:16:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:18:20] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [20:24:26] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:24:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:38:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [20:40:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:46:39] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:58:37] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:03:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:09:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:10:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jvclw [21:10:26] [02miraheze/puppet] 07paladox 03a7f85ce - cloud2: Set puppetmaster to puppet2 [21:20:58] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb [21:29:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:33:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [21:33:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb [21:35:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:39:34] freenode is doing some server rerouting, so netsplits will be widespread. Expect lack of connectivity for a while. [21:39:53] @k6ka: I can’t get on the irc cloud app [21:39:57] IRCCloud in particular struggles during heavy netsplits like this because it needs to handle processing the quit and join messages for everyone. [21:40:05] It started to crash at the same time [21:41:29] Wm-bot shows it ping timed out [21:41:56] Normal, just wait for everything to reconnect and for the dust to settle. [21:43:54] Snoonet is back [21:44:19] Freenode is stuck on the wrong side of a split [21:45:30] ?en Demasiados netsplits [21:50:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:50:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:54:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:54:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:01:00] paladox|UKInEU: would you be able to reply to the email I just copied tech@ in about wikibase [22:01:38] Not at the moment :( [22:02:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [22:02:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [22:02:31] paladox|UKInEU: let’s just do it sooner rather than later. It’s only been an open task since July! [22:02:45] RhinosF1 cloud virts are taking priority at the moment. [22:02:53] unless JohnLewis objects [22:03:20] Priorities remain as they are [22:03:45] JohnLewis: Kk [22:04:05] I only want to be reminded of what exactly didnt work [22:04:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:04:15] ?en Ya hay una solución para Wikibase Client? [22:04:22] RhinosF1: is it not listed in the tasks? [22:04:23] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:04:30] JohnLewis: no [22:05:14] @Hispano76: no but we’re hopefully getting one soon if the technical team can work with WMDE [22:06:04] RhinosF1: I remember now, it's something to do with when enabling Wikibase Repo, you can't add sitelinks and thusly Client won't work [22:06:24] Stuff should be enabled on test1, if not let me know and I'll enable it for you to play around with it [22:06:26] ?es ah. gracias por la aclración, esperemos que se solucione. He pensado en que Wikibase Cliente y Wikibase Repo sería muy útil en mis wikis [22:06:52] JohnLewis: yeah, look now and I’ll have a play over the next few says [22:06:54] Days [22:07:06] ?en ah. gracias por la aclración, esperemos que se solucione. He pensado en que Wikibase Cliente y Wikibase Repo sería muy útil en mis wikis [22:07:38] RhinosF1: okay, liaise with Reception123 though, he's free. Myself and Paladox are focused on trying to sort out cloud [22:07:49] even Zppix as well can help :) [22:08:19] Reception123, Zppix [22:09:50] ?en una de las razones es para solo añadir {{infobox person}} en lugar de añadirlo con todos los parametros necesarios ({{infobox person|name=|image=... ) [22:10:45] No problem [22:11:31] Reception123, Zppix: can we update wikibase stuff as well as social profile prior to tommorow afternoon or be around to deploy when i break things in git [22:12:33] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:13:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [22:19:48] RhinosF1 there's updates for wikibase? [22:22:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:22:38] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:27:01] paladox|UKInEU: check [22:27:42] It’s a make sure I’m not testing on something that wasnt updated before the issues started and has had 50 commits sinfe [22:27:55] Thats to Reception123 and Zppix [22:29:08] Also SP is not an easy update, Zppix make sure if you do SP, to look at the schema changes and apply. [22:30:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jvc40 [22:30:08] [02miraheze/mediawiki] 07paladox 03fc9ef4c - Update Wikibase [22:31:00] paladox|UKInEU: We know [22:33:16] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb [22:40:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvc4h [22:40:13] [02miraheze/dns] 07paladox 0361aa958 - Add gluster1 to dns [22:47:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:47:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:49:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:51:50] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:53:53] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 2.604 second response time [22:57:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:58:41] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:59:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:02:55] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:03:17] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:04:52] RECOVERY - cp4 Stunnel Http for mw2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 3.649 second response time [23:07:38] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:16] PROBLEM - cp4 Stunnel Http for mw3 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:32] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:10:35] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18689 bytes in 0.407 second response time [23:10:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:11:42] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 1.009 second response time [23:16:29] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 81.4.109.133/cpweb [23:17:14] RECOVERY - cp4 Stunnel Http for mw3 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 5.223 second response time [23:27:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:28:41] PROBLEM - cp2 Stunnel Http for mw3 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:30:24] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:48] RECOVERY - cp2 Stunnel Http for mw3 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 8.165 second response time [23:31:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [23:33:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb [23:34:57] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 8.610 second response time [23:35:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:37:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:39:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [23:41:27] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [23:43:28] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcRa [23:45:09] [02miraheze/services] 07MirahezeSSLBot 030eeffb9 - BOT: Updating services config for wikis [23:49:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvcRD [23:55:18] [02miraheze/services] 07MirahezeSSLBot 0350d3b95 - BOT: Updating services config for wikis