[00:04:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4bdU [00:04:06] [02miraheze/mw-config] 07paladox 030a98b1a - Update FileBackend.php [00:12:28] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4bdP [00:12:29] [02miraheze/mw-config] 07JohnFLewis 030aaec2d - fix name of variable [00:14:42] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/f4bd7 [00:14:43] [02miraheze/mediawiki] 07paladox 032bf6656 - Update MirahezeMagic [00:16:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4bFv [00:16:35] [02miraheze/mw-config] 07paladox 03def1bdc - Update LocalSettings.php [02:04:21] PROBLEM - JobQueue on mw2 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2374 [02:04:34] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2373 [02:05:22] ony... [02:05:24] *only [02:05:28] three... [02:05:28] hours... [02:05:30] left... [02:05:31] >_< [02:08:52] Three hours until what NeoBogan ? [02:09:41] FREEDOM! [02:10:02] Lol [02:10:19] NeoBogan: you do know you can leave work at any time? [02:10:25] then I’ll have a whole 60 hours all to myself [02:10:30] yes, I could [02:10:37] but I only get paid for time _at_ work [02:10:49] Yep [02:11:25] and if I want to be vansitting tonight (as I’ve done every night this week), I have to wait until the other office are finished with it [02:11:40] A what? [02:11:51] Why are you van sitting? [02:12:05] weekend been... one of our vehicles was stolen (and, of all the vehicles in our fleet, they got the one van that hadn’t yet been fitted with GPS tracking...) [02:12:12] Also lablaster is here [02:12:15] labster: [02:12:20] so, since then, the remaining vehicles spend nights at employee homes [02:12:33] Oh [02:12:35] upshot: free commuting wagon + feel fuel [02:12:45] *free fuel [02:12:46] >_< [02:12:51] Lol [02:13:09] it was only 5 degrees C outside my place when I woke up this morning [02:13:19] and -1.3 degrees C outside work [02:13:22] NeoBogan: see comment above about blaster [02:13:29] Heh [02:13:32] you honestly think I’m going to _walk_ to work in those sorts of temperatures?! [02:13:38] Yes [02:13:39] Lol [02:13:41] yeah, I saw the “lablaster” typo [02:13:49] yeah, really can’t afford the flu at this point [02:13:56] Heh [02:13:59] though it would give me more tiberian sun time I will acknowledge [02:14:07] It’s worm here! [02:14:17] speaking of... had a 1 v 7 brutal last night [02:14:30] Huh? [02:14:40] I’d elected to go as the GDI faction, the 7 brutal AIs were a mix of Nod & GDI [02:15:02] I’d whittled down 6 of those AIs to nothing using little more than ghost stalker and the mech walker [02:15:10] well, Mk II Mammoth [02:15:28] I was just powering through the 7th AI when the game hit a bug and died [02:16:06] Heh [02:28:23] RECOVERY - JobQueue on mw2 is OK: JOBQUEUE OK - job queue below 300 jobs [02:31:02] User account Przed użyciem zapoznaj się z treścią ulotki dołączonej do opakowania bądź skonsultuj się z lekarzem lub farmaceutą, gdyż każdy lek niewłaściwie stosowany zagraża Twojemu życiu lub zdrowiu (talk | contribs | block) was created automatically [02:31:05] wut [02:31:46] Voidwalker: huh? [02:32:52] that's a user account [02:33:56] Oh [02:34:09] You mean the length of the username Voidwalker ? [02:34:19] it's even weirder when you put it through translate [02:34:32] that too [02:34:50] translates roughly to "Before using, read the package leaflet or consult your doctor or pharmacist, as any drug used improperly jeopardizes your life or health" [02:35:07] Lol [02:35:14] my life *OR* health?! [02:35:35] I’m pretty sure jeapardizing one would surely affect the other [03:15:25] we got de-john’d? [03:15:27] * NeoBogan celebrates [03:15:44] I mean... is it me, or does he come across as the miragrump? [04:17:43] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 48% [04:19:43] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 4% [08:37:51] NeoBogan: it's not life XOR health, it's life OR health. [12:05:37] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:07:37] PROBLEM - Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/apt/apt.conf.d/50unattended-upgrades],File[/etc/apt/apt.conf.d/20auto-upgrades],File[/root/ufw-fix] [12:11:37] RECOVERY - Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [12:13:37] RECOVERY - Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:53:46] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [13:01:47] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:43:48] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [13:51:48] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:22:16] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4NiV [14:22:16] [02miraheze/mw-config] 07MacFan4000 03710cdd8 - Fix incorrect [14:23:01] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4NiP [14:23:02] [02miraheze/mw-config] 07MacFan4000 037f5b8e5 - Fix [14:32:28] The versions didn’t quite agree [14:48:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+5/-0/±0] 13https://git.io/f4N1D [14:48:23] [02miraheze/puppet] 07paladox 033aa49df - Initial creation of swift / swift::proxy puppet class [14:48:24] JohnLewis ^^! [14:48:46] okay [14:54:58] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/f4NMi [14:54:59] [02miraheze/mw-config] 07paladox 037d0039a - Revert "Fix" This reverts commit 7f5b8e549ac6e285d44f4c692c10acaff984e764. [14:55:01] [02miraheze/mw-config] 07paladox 03306a68b - Revert "Fix incorrect" This reverts commit 710cdd82c8b397d646faf9df5f7fecb854c88088. [15:08:19] a few minutes ago, "503" was coming out. No longer appears [15:08:22] oh, revert [15:08:42] That's what happens when I don't notice xd [15:08:46] !log reboot misc1 due to network problems [15:09:34] * Wiki-1776 gives cookies to paladox [15:09:38] :) [15:10:36] nope [15:10:39] reboot didn't help [15:11:15] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:11:28] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [15:11:54] ah [15:11:56] it's back [15:12:03] wait [15:12:09] mw1 and mw2 are down? [15:12:31] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:12:52] i belive everything just had a network problem [15:12:53] then [15:13:32] PROBLEM - Puppet on misc3 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 2 minutes ago with 8 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/electron],File[/etc/logrotate.d/mathoid],File[/etc/logrotate.d/restbase],File[/etc/nginx/sites-available/parsoid] [15:13:45] i guess mw1 and mw2 show as down for cp5 [15:13:55] if network went down in ramnode data center [15:14:22] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [15:14:30] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [15:15:02] still shown as down for cp5 [15:16:54] http://status.ramnode.com [15:18:35] On their Twitter it says Investigating large DDoS in NL [15:18:57] paladox: ^^ [15:20:32] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4NSn [15:20:34] [02miraheze/mw-config] 07MacFan4000 0336fc98f - Fix incorrect URL [15:21:49] RECOVERY - Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:22:03] JohnLewis looks like some in the nl are down [15:22:35] mw2 is down for cp2+5 but not 4 [15:25:17] cp4 May get connections internally fine [15:25:59] paladox: https://mobile.twitter.com/miraheze/status/1012718663777112070 [15:26:00] Title: [ Twitter ] - mobile.twitter.com [15:26:18] https://mobile.twitter.com/NodeStatus/status/1012713053509574657 [15:26:19] Title: [ Twitter ] - mobile.twitter.com [15:26:45] I retweeted as @miraheze with a comment [15:48:58] ok grafana has gone offline [15:49:02] which means misc2 [15:49:03] JohnLewis ^^ [15:49:32] uh [15:49:36] meta not loading [15:53:06] PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.75, 3.09, 1.72 [15:53:50] https://status.miraheze.wiki/components/5b191c7f87d844329ab643c2 updated now [15:53:52] Title: [ Miraheze Meta - Status Page - Miraheze ] - status.miraheze.wiki [15:53:56] https://status.miraheze.wiki [15:53:57] Title: [ Status Page - Miraheze ] - status.miraheze.wiki [15:56:24] PROBLEM - Puppet on mw2 is WARNING: WARNING: Puppet last ran 1 hour ago [15:57:05] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 0.84, 1.87, 1.52 [16:17:23] misc1 is down [16:19:37] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:47:24] RIP adminbot (needs a restart) [16:51:08] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:03:42] PROBLEM - Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:03:52] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:04:05] PROBLEM - Puppet on db4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:04:13] PROBLEM - Puppet on bacula1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:09:49] PROBLEM - Puppet on nfs1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:09:52] PROBLEM - Puppet on cp2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:09:53] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:09:57] PROBLEM - Puppet on ns1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:09:59] PROBLEM - Puppet on cp5 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:11:20] that can't be good [18:11:51] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:11:53] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [18:12:01] RECOVERY - Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:49] RECOVERY - Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [18:13:51] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [18:13:59] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:16:17] Voidwalker that was me [18:28:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/f4AJj [18:28:42] [02miraheze/puppet] 07paladox 037c493f0 - Add role::swift [18:29:24] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [18:29:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/f4AUJ [18:29:26] [02miraheze/puppet] 07paladox 03c473eeb - Enable role::swift for swift1.miraheze.org [18:29:27] [02puppet] 07paladox opened pull request 03#760: Enable role::swift for swift1.miraheze.org - 13https://git.io/f4AUT [18:29:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+1/-0/±0] 13https://git.io/f4AUL [18:30:01] [02miraheze/puppet] 07paladox 03c9adeb8 - Create swift1.yaml [18:30:02] [02puppet] 07paladox synchronize pull request 03#760: Enable role::swift for swift1.miraheze.org - 13https://git.io/f4AUT [18:50:09] paladox: is it me, or is Phabricator down? [18:58:42] Reception123 phab works for me [18:59:10] This site can’t be reached [18:59:10] phabricator.miraheze.org took too long to respond. [18:59:19] Reception123 hmm [18:59:22] reload? [18:59:38] paladox: tried. a ping also returns a "request time out" [18:59:43] *timed [18:59:47] hmm [18:59:53] i can try to restart misc1 [19:00:01] but then not sure if that will resolve it [19:00:18] https://mobile.twitter.com/NodeStatus/status/1012760608989728771 [19:00:18] Title: [ Twitter ] - mobile.twitter.com [19:00:21] Reception123 maybe ^ [19:00:30] Reception123 ramnode had a ddos attack [19:01:36] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:02:06] Reception123 try now [19:04:03] PROBLEM - IRC RC Bot on misc1 is CRITICAL: PROCS CRITICAL: 0 processes with args ircrcbot.py [19:04:31] that [19:04:33] that's new [19:06:03] RECOVERY - IRC RC Bot on misc1 is OK: PROCS OK: 1 process with args ircrcbot.py [19:06:19] !log rebooted misc1 [19:06:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:06:35] !log systemctl enable ircrcbot on misc1 so that ircrcbot service starts on boot [19:06:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:07:45] paladox: still doesn't [19:07:57] Reception123 im guessing it may be due to your isp [19:08:03] or something networking related [19:08:04] paladox: did we really need a misc1 reboot? [19:08:06] hmm possibly [19:08:25] Reception123 not sure, i wanted to rule that possibly out. [19:08:31] but nope, network related it seems [19:08:36] ok [19:09:26] paladox: it works now! :) [19:09:32] :) [19:13:21] Unable to establish a connection to any database host (while trying "phabricator_draft"). All masters and replicas are completely unreachable. AphrontConnectionQueryException: Attempt to connect to phabricator@db4.miraheze.org failed with error #2002: php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution. [19:13:23] ^ paladox hmmm [19:13:29] uh [19:13:30] ah [19:13:47] Reception123 i am thinking the ddos attack is ongoing [19:13:52] i saw that earlier today [19:14:08] it is affecting the netherlands data center [19:14:11] we're getting DDOSed again? [19:14:22] Reception123 not us specifically but ramnode [19:14:23] tsk tsk do people have nothing better to do in their lives [19:14:30] paladox: yeah, it's happened before a few times [19:14:34] oh [19:15:11] well it's better for it to be Phabricator rather than actual wikis [19:15:14] but it's still quite annoying [19:15:39] yeh [19:17:36] urghh can they stop ddossing our bots [19:18:06] Reception123 no one is ddos specifically they are ddos ramnode which hosts our services on [19:18:14] speciffically us [19:18:24] yes, I know. Just a joke :P [19:18:41] heh [19:20:31] [02miraheze/mw-config] 07revi pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4ALQ [19:20:33] [02miraheze/mw-config] 07revi 03906abe8 - no CCL4 As a sole contributor of the wiki, I utterly reject the CCL4, under no circumstances. Signed-off-by: Yongmin Hong [19:20:34] f***cking no. [19:20:45] and all ManageWiki option has CCL4. [19:23:34] whoever made this damned change should've notified of this [19:23:58] revi john [19:28:34] miraheze staff do not own the copyright required to (older version) => 4.0 upgrade imo- https://wiki.creativecommons.org/wiki/4.0_upgrade_guidelines#Upgrading_from_prior_versions_to_4.0 [19:28:34] Title: [ 4.0 upgrade guidelines - Creative Commons ] - wiki.creativecommons.org [19:28:55] I don’t think our tos allows us to blindly relicense to 4.0? [19:35:18] hmm seems the LocalSettings don’t work [19:35:57] revi wmgWikiLicense [19:36:07] damnit [19:36:27] can you fix it for me? I just turned off the laptop after I pushed that fix [19:36:41] ok [19:36:43] i will check [19:36:55] (if you didn’t tell me I was going to pull that wiki out of miraheze) [19:37:00] after I sleep [19:37:02] https://github.com/miraheze/mw-config/blob/master/ManageWiki.php#L889 [19:37:03] Title: [ mw-config/ManageWiki.php at master · miraheze/mw-config · GitHub ] - github.com [19:37:09] i guess i need to add your license there [19:37:14] revi what is it called? [19:37:19] 'Creative Commons BY 4.0' => 'cc-by', [19:37:23] well that doesn’t seem ideal [19:37:38] I am pretty sure I am the only one who wants to use that license [19:37:54] and yet I have to use that one? (then ok) [19:38:01] (CC BY SA 2.0 Korea, that is) [19:38:02] as your requesting the license i can add it [19:38:05] ah [19:38:05] ok [19:38:08] thanks [19:38:45] wait uh that also means all other wikis are also 4.0 [19:39:47] uh fuck phab is down thanks to ramnode [19:40:59] revi 'Creative Commons BY-SA 2.0 Korea' => 'cc-by-sa-kr', [19:41:26] ty [19:41:28] lgtm [19:41:31] ok [19:41:33] pushing then [19:41:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4Aqy [19:41:51] [02miraheze/mw-config] 07paladox 03ce0647e - Update ManageWiki.php [19:42:49] oh [19:42:52] found the switch [19:44:21] I don’t think “service. You agree to let Miraheze personnel … change your Wiki's configuration” in tos is enough to change the license [19:44:35] license is simply not a configuration [19:45:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4AmY [19:45:04] [02miraheze/mw-config] 07paladox 0396dbe3d - Update LocalWiki.php [19:45:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4AmG [19:45:32] [02miraheze/mw-config] 07paladox 037451efd - Update ManageWiki.php [19:45:34] anyway I’m going to bed hoping the network works tomorrow [19:45:34] seeya [19:46:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4AmC [19:46:04] [02miraheze/mw-config] 07paladox 038880661 - Update LocalWiki.php [19:48:14] root@mw1:/srv/mediawiki/config# sudo -u www-data git pull [19:48:15] fatal: unable to access 'https://github.com/miraheze/mw-config.git/': Could not resolve host: github.com [19:48:23] well that's why you doin't see your change revi [19:49:38] BS [19:49:46] Internet sucking today [19:49:50] yeh [19:50:30] i have a feeling the wiki' [19:50:34] wiki's may go down [19:50:38] root@mw2:/srv/mediawiki/config# curl https://meta.miraheze.org is stalled [19:50:39] Title: [ Meta ] - meta.miraheze.org [19:52:54] phab is already down for me [19:53:30] probably my network is failing to make a connection to the cp or cp is failing to connect to misc [19:53:59] phab dosent load for me either [19:54:04] given I can access cp5 as usual, it’s my connection failing connection to misc [19:54:06] misc rip [20:12:03] let's just leave it at that for a while [20:12:17] ok [20:12:31] Voidwalker the topic? [20:12:38] yep [20:12:52] ok [20:17:37] PROBLEM - wiki.consentcraft.uk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:37] PROBLEM - aman.info.tm - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:18:01] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [20:22:31] lol [20:22:37] all traffic going through mw3 mostly [20:22:44] high load [20:22:45] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=mw3.miraheze.org&var-port=9100 [20:22:48] Title: [ Grafana ] - grafana.miraheze.org [20:34:07] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:34:22] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [20:42:52] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [20:43:50] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: connect to address 185.52.2.113 and port 443: Connection refused [20:43:58] mw2 is down (can ssh in) [20:44:01] but nginx is failing [20:44:50] PROBLEM - HTTP on mw2 is CRITICAL: connect to address 185.52.2.113 and port 80: Connection refused [21:18:34] RECOVERY - HTTPS on mw2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 459 bytes in 0.085 second response time [21:18:57] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [21:18:57] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 30097 bytes in 0.119 second response time [21:25:31] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [21:27:21] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:35:22] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:35:23] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:41:22] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [21:47:40] !log test [21:48:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:50:51] It worked paladox [21:51:04] yeh i know [21:51:06] itermittent [21:51:11] !log paladox stopped icinga [21:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:51:31] There [21:53:30] RECOVERY - Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 12 minutes ago with 0 failures [21:53:33] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:53:46] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [21:54:13] PROBLEM - Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [21:54:22] uh [21:58:18] So this means that all traffic is currently going to mw3? [21:59:12] cp4 is going to all mw* [21:59:19] cp2 and cp5 are going to mw3 [22:03:16] PROBLEM - Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 48 seconds ago with 1 failures [22:08:40] [02puppet] 07paladox closed pull request 03#760: Enable role::swift for swift1.miraheze.org - 13https://git.io/f4AUT [22:08:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/f4AlK [22:08:43] [02miraheze/puppet] 07paladox 03bff7749 - Enable role::swift for swift1.miraheze.org (#760) * Enable role::swift for swift1.miraheze.org * Create swift1.yaml [22:09:24] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [22:09:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [22:09:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4Al1 [22:09:41] [02miraheze/puppet] 07paladox 03aaca167 - Update site.pp [22:10:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4AlM [22:10:32] [02miraheze/puppet] 07paladox 03795fe6c - Update proxy.pp [22:12:35] PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 42 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[swift-proxy] [22:13:04] known [22:16:39] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbiAS [22:16:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/f4Alx [22:16:42] [02miraheze/puppet] 07paladox 03c8e7bd0 - Update PrivateSettings.php [22:16:43] [02puppet] 07paladox opened pull request 03#761: Update PrivateSettings.php - 13https://git.io/f4Alp [22:17:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/f4A8J [22:17:46] [02miraheze/puppet] 07paladox 033383ede - Update init.pp [22:17:48] [02puppet] 07paladox synchronize pull request 03#761: Update PrivateSettings.php - 13https://git.io/f4Alp [22:18:36] [02puppet] 07JohnFLewis closed pull request 03#752: Add list.miraheze.org to postfix - 13https://git.io/vphid [22:18:38] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [22:18:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/f4A8k [22:18:42] [02miraheze/mw-config] 07paladox 030b960d6 - use swift admin password variable [22:18:43] [02mw-config] 07paladox opened pull request 03#2274: use swift admin password variable - 13https://git.io/f4A8I [22:19:34] [02puppet] 07paladox synchronize pull request 03#761: Update PrivateSettings.php - 13https://git.io/f4Alp [22:19:36] PROBLEM - Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:19:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/f4A8m [22:19:37] [02miraheze/puppet] 07paladox 03410307d - Update init.pp [22:19:38] miraheze/mw-config/paladox-patch-2/0b960d6 - paladox The build was fixed. https://travis-ci.org/miraheze/mw-config/builds/398444659 [22:24:30] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:27:32] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [22:29:10] PROBLEM - Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[icinga] [22:30:54] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [22:31:42] PROBLEM - www.radviser.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:31:58] RECOVERY - Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [22:32:00] RECOVERY - www.radviser.org - LetsEncrypt on sslhost is OK: OK - Certificate radviser.org will expire on Sun 12 Aug 2018 02:59:36 PM GMT +0000. [22:40:09] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:55] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate wiki.worlduniversityandschool.org will expire on Fri 21 Sep 2018 03:50:32 PM GMT +0000. [22:41:15] PROBLEM - Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:45:13] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:49:00] [02GitHub] Anything added dilutes everything else. [22:51:16] PROBLEM - athenapedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:52:38] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4ABL [22:52:39] [02miraheze/rings] 07paladox 03abd5bc3 - Test [22:53:08] RECOVERY - athenapedia.org - LetsEncrypt on sslhost is OK: OK - Certificate athenapedia.org will expire on Fri 31 Aug 2018 02:19:41 PM GMT +0000. [22:56:48] RECOVERY - Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:05:21] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+6/-0/±1] 13https://git.io/f4ABS [23:05:23] [02miraheze/rings] 07paladox 03ed6f914 - Add ring files [23:05:24] JohnLewis ^^! [23:05:34] k [23:19:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4ARM [23:19:17] [02miraheze/puppet] 07paladox 03551dc91 - swift: symnlink to ring files [23:24:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/f4A0U [23:24:33] [02miraheze/puppet] 07paladox 031d35ecc - Make sure some swift dir exists [23:25:17] finally swift-proxy is running on swift1 JohnLewis! [23:26:48] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:45:32] [02puppet] 07paladox closed pull request 03#761: Update PrivateSettings.php - 13https://git.io/f4Alp [23:45:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/f4AEY [23:45:35] [02miraheze/puppet] 07paladox 03b7d0579 - Update PrivateSettings.php (#761) * Update PrivateSettings.php * Update init.pp * Update init.pp [23:46:20] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+4/-0/±4] 13https://git.io/f4AE3 [23:46:21] [02miraheze/puppet] 07paladox 03053abc8 - Initial import of swift::backend [23:46:23] [02miraheze/puppet] 07paladox 03aac9ef9 - Merge branch 'master' of github.com:miraheze/puppet [23:46:52] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbiAS [23:46:54]