[00:31:15] PROBLEM - db5 Disk Space on db5 is CRITICAL: DISK CRITICAL - free space: / 10940 MB (5% inode=99%); [00:35:17] RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 40602 MB (22% inode=99%); [00:51:57] paladox: you could try this again to see if it's corrected, there's no rush if you can't log root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages [00:52:26] https://www.mediawiki.org/wiki/Manual:RecountCategories.php [00:52:27] [ Manual:recountCategories.php - MediaWiki ] - www.mediawiki.org [00:52:34] !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages [00:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:52:51] !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=subcats [00:53:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:53:03] !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=files [00:53:06] hispano76 done :) [00:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:13:19] paladox: Oh, I didn't realize you also used this root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php refreshLinks.php --wiki=ucroniaswiki required? [01:13:43] nope, not required. I just thought i try it per RhinosF1 [01:13:48] curiosity [01:14:02] ah, okay thanks ´paladox [01:14:06] yw [02:23:42] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:23:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:25:38] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15288 bytes in 0.502 second response time [02:25:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:26:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3029 MB (12% inode=94%); [07:08:18] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:09:33] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:10:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:10:08] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:10:42] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:10:46] PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:10:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [07:11:02] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:03] PROBLEM - guiasdobrasil.com.br - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:11:21] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:38] PROBLEM - cp4 Stunnel Http for mw2 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [07:11:59] PROBLEM - netazar.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:12:14] PROBLEM - Host cp4 is DOWN: PING CRITICAL - Packet loss = 100% [07:12:41] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:13:02] paladox, PuppyKun, Reception123, SPF|Cloud: CP4 DOWN [07:13:30] @Site Reliability Engineers [07:15:00] RECOVERY - netazar.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.netazar.org' will expire on Tue 18 Feb 2020 07:52:57 PM GMT +0000. [07:16:04] RhinosF1: server is suspended [07:20:53] SPF|Cloud: :( [07:21:03] I’ll guess bandwidth [07:21:11] This is why we have a fundraiser [07:22:43] SPF|Cloud: they said they wanted us to pay but Paladox got it restarted after a ticket last time [07:27:34] Probably. [07:28:13] With a bit of luck we won’t have to suffer anymore soon. [07:30:12] SPF|Cloud: hopefully, we’re up to £1622 now + whatever’s gone in directly so over halfway [07:30:49] It’ll get us the server and a good few months paid, just need to keep donations up. [08:18:23] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [08:21:46] Uh oh [08:26:19] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:29:24] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [08:37:21] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:43:48] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [09:00:28] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:46] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [09:14:29] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:24:00] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [09:30:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYeP [09:30:12] [02miraheze/services] 07MirahezeSSLBot 031e02f1c - BOT: Updating services config for wikis [09:31:49] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:55] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [09:42:38] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:00] paladox: shall I ask RamNode why the server has been suspended? [10:30:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYJ2 [10:30:11] [02miraheze/services] 07MirahezeSSLBot 038003b66 - BOT: Updating services config for wikis [10:57:12] SPF|Cloud: I mean you could, but I can already be 99.9% certain that it’s bandwidth [10:59:55] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [11:07:40] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:10:45] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [11:18:17] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:40:40] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [11:48:09] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:48:15] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%); [12:26:44] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [12:34:26] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:47:41] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [12:55:32] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:08:34] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [13:08:45] Hello AdrianBennetton! If you have any questions, feel free to ask and someone should answer soon. [13:09:44] Hi We have Miraheze private wiki (pwiki.arkcls.com). However it has been down all morning [13:10:45] hi AdrianBennetton [13:10:50] I will take a look [13:11:05] Thanks much appreciated [13:11:49] AdrianBennetton: What do you mean by "down"? I am able to access it [13:12:51] When we try in our office we get "This site can't be reached". I will try from my mobile [13:13:29] !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Toshinori "Agent Mace" --logwiki=metawiki [13:13:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:13:51] AdrianBennetton: That is weird, but that seems to be a local problem rather than a problem on our side [13:13:59] Please let me know if it works on mobile for you [13:14:52] It works from my mobile using 4G network. I will check that with our office systems admin as to why we can't access wiki from within our office [13:15:16] AdrianBennetton: Okay, please do since it could perhaps be a local block [13:15:25] Yes it does look like it is a local problem [13:15:31] Thanks for checking [13:17:11] RhinosF1: hi. Don't know why you said I needed info from Owen since it looks like all I need to do is add a FUNDING.yaml? [13:17:18] AdrianBennetton: Ok. You're welcome :) [13:24:34] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:04] Reception123: iirc funding.yml is for the custom links, but you need GH approval or whatever for a proper donations button. maybe not though [13:27:34] PuppyKun: ah ok, I didn't see that being mentioned [13:27:40] But I'll do funding.yaml and see what happens next [14:06:38] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [14:14:30] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:24:04] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [14:43:25] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:46:29] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [14:54:03] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:57:09] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [15:04:54] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:11:12] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [15:44:39] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:33] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [16:38:27] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:46:58] PuppyKun: GH Approval is needed for direct transfer to bank [16:51:11] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [16:58:55] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:48] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [17:22:24] Reception123: 16:46:58 PuppyKun: GH Approval is needed for direct transfer to bank [17:31:33] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:22] RhinosF1: oh ok [17:32:35] RhinosF1: since you proposed it, could you maybe make a PR for FUNDS.yaml? Since I don't really know the syntax [17:32:53] Reception123: see the twinkle repo [17:32:56] It’s in their [17:33:12] Then just turn sponsorships on in settings [17:33:22] RhinosF1: it's already turned on though [17:35:08] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JvYcT [17:35:10] [02miraheze/mw-config] 07Reception123 039c8750b - add FUNDING.yaml Per T5114 [17:35:13] Reception123: then just copy the funding file from the twinkle repo to as many MH ones as possible [17:35:31] oh my mad it's YML not YAML [17:36:08] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/JvYcI [17:36:09] [02miraheze/mw-config] 07Reception123 036148781 - YAML->YML typo [17:36:18] RhinosF1: it already seems to work... [17:36:26] RhinosF1: https://github.com/miraheze/mw-config do you see the button? [17:36:27] [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com [17:40:05] Reception123: yeah working [18:00:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYcd [18:00:12] [02miraheze/services] 07MirahezeSSLBot 03eb7d5bc - BOT: Updating services config for wikis [18:00:59] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [18:08:42] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:49] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [18:56:52] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:32:38] !log root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3001 --delete [19:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:35:56] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [19:43:40] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:39] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [20:04:37] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:24:25] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [20:32:07] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:05] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [20:40:54] JohnLewis: You around? [20:43:03] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:49:35] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [20:57:42] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:04:35] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [21:09:07] Examknow: not really, what's up? [21:12:19] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:15:22] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [21:18:05] Has there been an example of integration of local Wikidata and modules into a Miraheze wiki? [21:23:32] hispano76: not that I know of [21:26:06] It would be interesting to do this (because I would like the format to save bytes..) :) [21:27:15] By the way, what was the reason why Wikidata can't be used directly? [21:27:22] curiosity* [21:40:51] hispano76: broke extension I think [21:41:17] ah, ok. thanks RhinosF1 [21:42:07] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:12] paladox: whats up with the ssl alerts [21:48:56] I'm not entirly sure, but the site works. [21:49:01] (hence why i didn't remove it) [21:50:09] paladox: they started when cp4 failed [21:50:21] Sims wiki was on as well [21:50:22] host enc.for.uz [21:50:25] points to cp4 [21:50:27] err [21:50:28] cp2 [21:50:50] paladox: hmm, quite a few crashed before cp4 paged down [21:51:25] quote a few? [21:51:49] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [21:52:05] paladox: whatever is in icinga [21:52:26] sims wiki is due to cp4 [21:52:33] since they have cp4 hardcoded [21:52:56] paladox: can you compress ungamewiki? [21:52:58] guiasdobrasil.com.br [21:52:59] guiasdobrasil.com.br seems to no longer be pointing at us? [21:53:13] Zppix you just add ungamewiki to the config and run a maint script [21:53:42] paladox: im mobile atm [21:53:48] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvY2t [21:53:50] [02miraheze/ssl] 07paladox 030457aba - Remove guiasdobrasil.com.br No longer pointing at us [21:53:51] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [21:53:53] [02ssl] 07paladox opened pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q [21:53:56] oh [21:53:57] paladox: look at logs from here when icinga paged cp4 and you’ll see [21:54:26] well seems recovered anyways for the ones your talking about [21:54:58] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JvY23 [21:54:59] [02miraheze/ssl] 07paladox 0387fc9d3 - Delete guiasdobrasil.com.br.crt [21:55:01] [02ssl] 07paladox synchronize pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q [21:55:15] [02ssl] 07paladox closed pull request 03#263: Remove guiasdobrasil.com.br - 13https://git.io/JvY2q [21:55:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JvY2s [21:55:17] [02miraheze/ssl] 07paladox 033a1ddbc - Remove guiasdobrasil.com.br (#263) * Remove guiasdobrasil.com.br No longer pointing at us * Delete guiasdobrasil.com.br.crt [21:55:19] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [21:55:20] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [21:55:21] paladox: well simswiki we know about and that you’ve deleted [21:55:35] They went about 10s before cp4 [21:56:08] netazar.org went 1 min later [21:56:29] enc.for.uz 2 mins after that [21:58:26] !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/guiasdobrasil.com.br "', null ) WHERE wiki_dbname = 'guiaslocaiswiki'; [21:58:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:59:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY20 [21:59:20] [02miraheze/mw-config] 07paladox 038f029b5 - Compress ungamewiki wiki [21:59:40] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:00:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2u [22:00:11] [02miraheze/services] 07MirahezeSSLBot 035cb4600 - BOT: Updating services config for wikis [22:00:40] !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=ungamewiki --type=gzip [22:00:42] Zppix done ^ [22:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:02:12] i see the issue on misc1... [22:02:34] ok, it's using cp4 itermittently??? [22:02:48] host enc.for.uz [22:02:48] enc.for.uz has address 81.4.109.133 [22:02:48] enc.for.uz has IPv6 address 2604:180:0:33b::2 [22:02:48] enc.for.uz mail is handled by 10 mx.yandex.net. [22:02:53] 81.4.109.133 = cp4 [22:03:23] Ab [22:03:24] Ah [22:03:45] thanks paladox [22:04:48] * paladox depools cp4 [22:05:13] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2P [22:05:15] [02miraheze/dns] 07paladox 03453a193 - Depool cp4 [22:05:34] Thanks [22:06:32] miraheze/dns/master/453a193 - paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/643120732 [22:06:42] well that didn't work... (stop it from using cp4) [22:07:13] 😦 [22:07:30] wtf [22:07:42] it's using a cp4 ipv4 address but cp2 ipv6 address [22:07:52] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:07:56] Strange [22:08:00] Poor icinga-miraheze [22:08:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2D [22:08:46] [02miraheze/ssl] 07paladox 03dea1dca - Remove www.guiasdobrasil.com.br [22:08:48] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:08:53] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:09:01] lol [22:09:26] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:09:33] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:09:44] Zppix: see discord for a TestWiki question in discrete [22:10:16] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:10:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[guiasdobrasil.com.br] [22:10:48] oh! [22:11:20] RhinosF1: im good with whatever void decides :D I'm a consul purely to help out, and i trust his judgement [22:11:51] it's itermittently using yandex, then it's using ours. [22:12:10] paladox: probably need to contact whoever manages the domain then [22:12:20] yup [22:13:10] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:13:13] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:13:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:14:20] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:14:35] @Zppix: kk [22:14:48] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:14:54] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:15:50] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:16:09] https://phabricator.miraheze.org/T5174 [22:16:10] [ ⚓ T5174 enc.for.uz itermittently using both yandex and miraheze dns ] - phabricator.miraheze.org [22:16:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvY2b [22:16:50] [02miraheze/dns] 07paladox 03e71e24f - Revert "Depool cp4" This reverts commit 453a1936afed13ba9750d3080dd52a2643dabd02. [22:17:52] paladox: maybe we should not allow people to hardcode what cp* they are using? [22:18:00] miraheze/dns/master/e71e24f - Paladox The build has errored. https://travis-ci.org/miraheze/dns/builds/643125670 [22:18:09] well some have to unfournitly. [22:18:29] like simswiki (as they want access to the wiki on the main domain rather then a subdomain) [22:18:49] paladox: what do you mean? [22:19:20] Simswiki can only use an ip, they carn't use mw-lib. This is due to the use of the main domain. [22:19:20] they cant do it to point to mw-lb's ip? [22:19:30] mw-lib does not have an ip [22:19:48] a domain has to have a valid ip? [22:20:26] Zppix https://github.com/miraheze/dns/blame/master/zones/miraheze.org#L115 [22:20:27] [ dns/zones/miraheze.org at master · miraheze/dns · GitHub ] - github.com [22:22:28] oh i see paladox [22:22:35] maybe we could assign mw-lb an ip paladox ? [22:22:51] but an ip is pointless? [22:23:03] that means if that server goes down, things will go down which use it [22:23:13] true [23:14:23] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [23:27:41] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:30:44] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [23:36:31] !log upgrade grafana - misc1 [23:36:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:41:28] !log upgrade php - misc1 [23:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:42:31] !log upgrase puppet-agent & salt on misc1 [23:42:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:42:45] paladox: upgrase? [23:43:01] yes, there's updates available in apt for the packages [23:43:15] paladox: but you didnt update it you upgrased it :P [23:43:22] oh [23:43:35] fixed [23:43:41] he [23:44:35] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:45:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYw3 [23:45:33] [02miraheze/puppet] 07paladox 03c4b846a - icingaweb2: upgrade to php 7.3 [23:47:23] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:50:53] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Thu 09 Apr 2020 11:57:25 AM GMT +0000. [23:51:43] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [23:51:46] PROBLEM - misc1 icinga.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.005 second response time [23:52:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYwu [23:52:08] [02miraheze/puppet] 07paladox 03f6be00d - roundcubemail: Upgrade php to 7.3 [23:53:32] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:53:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvYwg [23:53:34] [02miraheze/puppet] 07paladox 0354fe1da - roundcubemail: Update to 1.4.2 [23:54:46] RECOVERY - misc1 icinga.miraheze.org HTTPS on misc1 is OK: HTTP OK: HTTP/1.1 302 Found - 334 bytes in 0.009 second response time