[00:05:29] PROBLEM - www.apcwiki.org - Comodo on sslhost is WARNING: WARNING - Certificate 'www.apcwiki.org' expires in 30 day(s) (Sun 10 Nov 2019 11:59:59 PM GMT +0000). [01:10:22] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf] [01:15:32] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:47:58] RECOVERY - bacula1 Bacula Databases db4 on bacula1 is OK: OK: Full, 840751 files, 46.33GB, 2019-10-11 01:46:00 (1.9 minutes ago) [02:56:30] RECOVERY - bacula1 Bacula Databases db5 on bacula1 is OK: OK: Full, 2150 files, 60.32GB, 2019-10-11 02:56:00 (30.0 seconds ago) [03:04:20] RECOVERY - bacula1 Bacula Phabricator Static on bacula1 is OK: OK: Full, 81004 files, 2.632GB, 2019-10-11 03:03:00 (1.3 minutes ago) [06:25:16] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2652 MB (11% inode=94%); [07:24:35] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 6.95, 5.70 [07:27:15] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.41, 6.17, 5.59 [07:47:38] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown) [07:55:41] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:37:21] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/mnt/mediawiki-static] [08:40:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JelCr [08:40:13] [02miraheze/services] 07MirahezeSSLBot 03b745a7c - BOT: Updating services config for wikis [08:45:06] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JelWs [09:20:13] [02miraheze/services] 07MirahezeSSLBot 037c175ee - BOT: Updating services config for wikis [12:04:05] Hello TransEmo! If you have any questions, feel free to ask and someone should answer soon. [12:38:11] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown) [12:42:53] !log depool and repool mw[123] [12:43:53] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 1 failures [12:44:01] [02miraheze/puppet] 07paladox pushed 035 commits to 03master [+0/-0/±5] 13https://git.io/JelBg [12:44:03] [02miraheze/puppet] 07paladox 03f22c13e - Revert "Update mediawiki.pp" This reverts commit 7846961eb27d91389dc54ff415267c6a8778bf18. [12:44:04] [02miraheze/puppet] 07paladox 034d92e97 - Revert "Update mediawiki.pp" This reverts commit 91069f18cf700cd18e5ec8b426ef17aab08394b0. [12:44:06] [02miraheze/puppet] 07paladox 0336bbd54 - Revert "Update mediawiki.pp" This reverts commit 995e6b906006f4e07a07fcac5655c46bcc4f2d92. [12:44:07] [02miraheze/puppet] ... and 2 more commits. [12:46:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-4/±2] 13https://git.io/JelBi [12:46:07] [02miraheze/puppet] 07paladox 03e7d723f - Revert "Reintroduce nfs (#1095)" This reverts commit 09ffc1ea418382a4bdaceb0ee215a57f9b348590. [12:46:58] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 29 seconds ago with 1 failures. Failed resources (up to 3 shown) [12:48:48] !log reboot mw3 [12:48:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:49:31] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [12:50:25] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 1 failures [12:50:25] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 1 failures [12:54:24] !log reboot mw2 [12:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:55:11] PROBLEM - mw3 JobChron Service on mw3 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [12:55:11] PROBLEM - mw1 Current Load on mw1 is CRITICAL: CRITICAL - load average: 8.26, 7.17, 5.65 [12:56:05] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [12:56:27] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [12:58:01] RECOVERY - mw1 Current Load on mw1 is OK: OK - load average: 2.55, 5.70, 5.39 [12:58:16] !log reboot mw1 [12:58:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:58:46] !log reboot test1 [12:58:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:58:52] paladox: reboot all teh things [12:59:02] :D [12:59:14] paladox: who gonna reboot you? [12:59:25] the big red button next to you :P [12:59:49] paladox: also when you get a chance https://phabricator.miraheze.org/T4778 [12:59:49] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [12:59:50] [ ⚓ T4778 Change database name and URL of animatedfeet.miraheze.org ] - phabricator.miraheze.org [13:00:04] ok [13:02:22] !log disabled puppet on misc3 and experimenting with vm.swappiness = 0 [13:02:36] !log restart lizardfs-master [13:05:34] PROBLEM - cp4 Stunnel Http for test1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:05:50] * Zppix slowly watches paladox burn the world of Miraheze [13:05:51] RECOVERY - mw3 JobChron Service on mw3 is OK: PROCS OK: 1 process with args 'redisJobChronService' [13:06:12] heh [13:06:31] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:58] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:07:52] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:07:55] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [13:10:04] RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24569 bytes in 0.846 second response time [13:10:55] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24569 bytes in 0.494 second response time [13:11:51] RECOVERY - cp4 Stunnel Http for test1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24569 bytes in 0.007 second response time [13:12:35] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.008 second response time [13:13:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [13:13:40] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:13:47] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:35:09] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [13:35:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JelRA [13:35:12] [02miraheze/puppet] 07paladox 0392af1a7 - base: Allow changing swappiness value through hiera Also set base::sysctl::swap_value to 0 for misc3 [13:35:21] [02puppet] 07paladox opened pull request 03#1096: base: Allow changing swappiness value through hiera - 13https://git.io/JelRx [13:35:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JelRj [13:35:47] [02miraheze/puppet] 07paladox 030c114bc - Update misc3.yaml [13:35:49] [02puppet] 07paladox synchronize pull request 03#1096: base: Allow changing swappiness value through hiera - 13https://git.io/JelRx [13:36:12] [02puppet] 07paladox synchronize pull request 03#1096: base: Allow changing swappiness value through hiera - 13https://git.io/JelRx [13:36:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jel0v [13:36:15] [02miraheze/puppet] 07paladox 039fd4ac4 - Update sysctl.pp [13:36:24] [02puppet] 07paladox closed pull request 03#1096: base: Allow changing swappiness value through hiera - 13https://git.io/JelRx [13:36:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jel0f [13:36:27] [02miraheze/puppet] 07paladox 03fac385a - base: Allow changing swappiness value through hiera (#1096) * base: Allow changing swappiness value through hiera Also set base::sysctl::swap_value to 0 for misc3 * Update misc3.yaml * Update sysctl.pp [13:36:29] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbiAS [13:36:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [13:37:12] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:37:40] !log re-enabled puppet on misc3 [13:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:40:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jel0L [13:40:13] [02miraheze/services] 07MirahezeSSLBot 037e1a4b9 - BOT: Updating services config for wikis [14:00:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jel0u [14:00:13] [02miraheze/services] 07MirahezeSSLBot 036f18f99 - BOT: Updating services config for wikis [15:46:55] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [15:49:24] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2665 MB (11% inode=94%); [16:20:44] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%); [18:51:24] 503 again [18:52:31] paladox: ^ [18:52:37] ok [18:53:04] Well, it was about a min ago [19:06:23] That is about 3 503's today. I am getting pissed off now. [19:10:54] BurningPrincess: were trying our hardest [19:20:48] RhinosF1: sorry to be a pain like that, it 503'd at just the wrong time as well [19:23:03] BurningPrincess: the back button should allow you to recover any lost edits [19:25:13] RhinosF1: thank you, I did not know that [19:25:29] BurningPrincess: no problem [19:25:36] * BurningPrincess bakes Miraheze cakes and gives them to RhinosF1 [19:26:04] Miraheze cakes? Sounds good [19:26:45] Why has there been so many 503's [19:26:59] paladox: you'll be better at explaining [19:27:57] It's due to the file storage [19:28:20] Oh okay [19:28:28] the mount seems to freeze for a few minutes causing stats to fail, thus causing severe performance issues for the wikis as they will just sit there waiting [19:28:36] until they hit the timeout or it goes through [19:28:37] I hope that gets fixed [19:29:07] Would upgrading to a bigger server fix that? [19:30:11] BurningPrincess: better specs will help but we need money for that and there's been some issues with PayPal donations [19:32:39] I can't help, sorry [19:33:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:33:24] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [19:34:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [19:34:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:34:32] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [19:34:39] paladox: ^ [19:34:47] yup [19:34:50] i'm aware [19:35:23] Good [19:37:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [19:37:10] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:37:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:37:48] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:37:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:38:47] I hope Miraheze lasts more then a few more months [19:39:15] BurningPrincess: we should do [19:41:51] Moving it to a wiki farm woulda pain as I would have to ask its satff to upload xdml dumps of both pages and categories for both wikis [19:41:59] *another wiki farm [19:42:46] BurningPrincess: we'll stay online [19:43:13] Thank you, its less of a pain then asking my freind to self host or moving it all to shoutwiki [19:43:45] If you need anything, just ask [19:44:03] Thank you. so are such a nice group of staff [19:44:18] :) [19:45:13] * BurningPrincess smiles at RhinosF1 and says "Miraheze is amswome, probly tyhe BEST wiki farm" [19:47:19] Thanks [19:47:25] Lovely to hear! [19:49:08] I like it as it has no ads (I use an adblocker anyway), but I think lots of ads slow loading times - looking at you Wikia :) and allows uploads of xdml files (unlkile shoutwiki) [19:53:12] Good to hear, we’re completely against having ads on our wikis [19:54:58] I did hear that sometimes ads have bad links wityh viruses and the link [19:55:22] In theory it would be possible [19:56:04] Why not ads on miraheze? [19:57:16] BurningPrincess: people don’t like them [19:57:19] Generally [19:57:26] And discourages donations a lot [20:00:02] I don';r like ads either, esply vidio ads that play themselves or ads that get in the way of content, I don't mind small still ads though [20:02:32] BurningPrincess: chrome is getting strict on ads like that [20:06:43] That is good [20:11:05] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [20:11:08] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:11:14] 503 [20:11:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [20:11:53] * BurningPrincess kicks the Miraheze servers [20:11:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:11:55] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [20:12:27] Has Labster been around to accept any of the donations? If no, then it's probably time to assign someone else to handle the finances [20:14:13] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:14:18] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:14:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:14:47] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:14:47] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:35:22] !log ndkilla@misc1 sudo passwd ndkilla [20:35:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:36:35] k6ka: plans are being discussed [20:36:58] PuppyKun: for a minute there i was like why is PuppyKun using ndkilla’s user on misc1, then i was oh wait PuppyKun is ndkilla (i swear im slowly losing IQ) [20:37:54] "Written by: TheSkull31, written for: Kin'toni Games Setting, taken from: Lothrenidiom.freeforums.net " - is that good enough attubretion [20:38:27] The chnages are recorded in the history if that is inpornat [20:38:55] What license does that website use BurningPrincess [20:39:01] cc-by-nc-sa [20:39:24] That should be fine imho [20:39:36] Just make sure you can easily locate that attribution [20:39:58] Zppix: its at the top of the page above all the content. Would I need to add a changes log as well? [20:40:12] BurningPrincess: you mean in the page history? [20:40:18] Yes [20:40:39] I mean like if a page is altered, would I need to log changes in the page or does the histoy count [20:40:44] BurningPrincess: do you have a way i could see the original content from the origin site? [20:40:59] Its been deleted long long ago [20:41:21] BurningPrincess: i say just do the attribution and if anyone complains then yes do that [20:41:37] Does the page hisoty count though [20:41:50] As the logging of changes to the page like say me adding stuff [20:41:58] Yes [20:42:26] Would I need to say "x did so and so" or can I jsut point them at the page history for that [20:42:58] All changes are tracked in page history no need to mention that [20:43:10] Zppix: Thank you [20:43:14] Np [20:43:31] And if im completely wrong just point them to me and ill take the blame [20:43:51] You seem a nice girl/guy/non-binary person [20:44:37] Im a guy, atleast i was last i checked :) [20:44:59] On the inernet its hard to tell [20:45:30] BurningPrincess: meh imho on the internet if someone gets your pronoun wrong politely correct em and move on no need to make it a huge deal [20:46:37] Zppix: If it helps I amd female and use she/her/hers as pronouns [20:46:56] BurningPrincess: cool i tend to just use nicks [20:46:57] Or they [21:10:03] "You seem a nice girl/guy/non-binary person" [21:10:11] i just immediately remembered the on-going SO/SE drama [21:26:53] How do I find logs of find replace [21:30:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [21:32:07] cp2 was depooled?!?! [21:33:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:42:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jelot [21:42:35] [02miraheze/puppet] 07paladox 03e5c5efc - php: Lower request_terminate_timeout to 180 [21:43:03] !log hacked php fpm on mw1 to have 32 childs (experimenting) [21:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:45:32] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [21:47:02] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:47:27] !log lower childs to 22 on mw1 [21:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:47:53] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [21:49:33] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24592 bytes in 0.389 second response time [21:50:15] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:57:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jelog [21:57:20] [02miraheze/puppet] 07paladox 039f5d4d7 - Update mediawiki.conf [22:02:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeloi [22:02:50] [02miraheze/puppet] 07paladox 032d5685e - Update mediawiki.conf [22:04:49] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:05:32] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:07:58] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [22:08:07] known ^ [22:08:18] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [22:08:38] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.390 second response time [22:09:06] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:10:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2604:180:0:33b::2/cpweb [22:11:09] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:13:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:13:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:15:02] !log restarting lizardfs-master [22:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:18:00] PROBLEM - misc3 electron on misc3 is CRITICAL: connect to address 185.52.1.71 and port 3000: Connection refused [22:18:40] Miraheze is so bloody slow [22:20:00] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:09] PROBLEM - misc3 restbase on misc3 is CRITICAL: connect to address 185.52.1.71 and port 7231: Connection refused [22:20:54] PROBLEM - misc3 Lizardfs Master Port 2 on misc3 is CRITICAL: connect to address 185.52.1.71 and port 9420: Connection refused [22:21:14] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [22:21:23] PROBLEM - misc3 Lizardfs Master Port 3 on misc3 is CRITICAL: connect to address 185.52.1.71 and port 9421: Connection refused [22:21:23] PROBLEM - misc3 Lizardfs Master Port 1 on misc3 is CRITICAL: connect to address 185.52.1.71 and port 9419: Connection refused [22:21:25] PROBLEM - misc3 citoid on misc3 is CRITICAL: connect to address 185.52.1.71 and port 6927: Connection refused [22:21:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:22:10] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [22:22:27] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [22:22:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jelop [22:22:59] [02miraheze/puppet] 07paladox 032ebdbaf - Update mediawiki.pp [22:23:14] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24586 bytes in 0.681 second response time [22:23:16] RECOVERY - misc3 restbase on misc3 is OK: TCP OK - 0.005 second response time on 185.52.1.71 port 7231 [22:23:52] RECOVERY - misc3 Lizardfs Master Port 2 on misc3 is OK: TCP OK - 0.009 second response time on 185.52.1.71 port 9420 [22:23:58] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [22:24:12] RECOVERY - misc3 Lizardfs Master Port 3 on misc3 is OK: TCP OK - 0.008 second response time on 185.52.1.71 port 9421 [22:24:14] RECOVERY - misc3 citoid on misc3 is OK: TCP OK - 0.009 second response time on 185.52.1.71 port 6927 [22:24:14] RECOVERY - misc3 Lizardfs Master Port 1 on misc3 is OK: TCP OK - 0.011 second response time on 185.52.1.71 port 9419 [22:24:16] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:24:31] RECOVERY - misc3 electron on misc3 is OK: TCP OK - 0.010 second response time on 185.52.1.71 port 3000 [22:25:10] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [22:25:36] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [22:27:08] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:27:14] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 1 failures [22:28:04] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[/mnt/mediawiki-static] [22:30:53] !log hacked php-fpm on mw[23] [22:30:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:32:36] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [22:33:20] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [22:37:44] I hope all the 503's and slow loading is fixed soon [22:43:31] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [22:51:36] Yay mira does not have the slow loading p[robelm now [22:53:01] \o/ [22:53:04] doin't ginx it :P [23:40:40] Zppix: will I need to add a changes log if I copy paste the page into a new wiki [23:45:54] PROBLEM - lizardfs4 Puppet on lizardfs4 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 4 minutes ago with 11 failures. Failed resources (up to 3 shown) [23:45:55] PROBLEM - lizardfs5 Puppet on lizardfs5 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 4 minutes ago with 9 failures. Failed resources (up to 3 shown) [23:45:55] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 4 minutes ago with 6 failures. Failed resources (up to 3 shown) [23:46:04] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 4 minutes ago with 6 failures. Failed resources (up to 3 shown) [23:46:30] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 48 failures. Last run 4 minutes ago with 48 failures. Failed resources (up to 3 shown) [23:46:33] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 4 minutes ago with 16 failures. Failed resources (up to 3 shown) [23:46:55] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 365 failures. Last run 5 minutes ago with 365 failures. Failed resources (up to 3 shown) [23:46:58] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 5 minutes ago with 23 failures. Failed resources (up to 3 shown) [23:47:04] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown) [23:47:04] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 5 minutes ago with 12 failures. Failed resources (up to 3 shown) [23:47:30] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 5 minutes ago with 10 failures. Failed resources (up to 3 shown) [23:48:07] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 6 minutes ago with 10 failures. Failed resources (up to 3 shown) [23:48:17] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 295 failures. Last run 5 minutes ago with 295 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params] [23:48:27] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 373 failures. Last run 6 minutes ago with 373 failures. Failed resources (up to 3 shown) [23:48:52] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 208 failures. Last run 5 minutes ago with 208 failures. Failed resources (up to 3 shown): File[/etc/sudoers],File[/home/nagiosre],File[/etc/nginx/mime.types],File[/etc/nginx/fastcgi_params] [23:52:55] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [23:53:41] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [23:53:54] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [23:53:55] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:54:06] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:07] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:16] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [23:54:16] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:44] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:54:45] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:55:02] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:03] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:05] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:55:06] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures