[14:09:59] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1Mw [14:10:00] [02miraheze/CreateWiki] 07translatewiki 03f8a5c90 - Localisation updates from https://translatewiki.net. [14:10:01] [ Main page - translatewiki.net ] - translatewiki.net. [14:10:02] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1Mr [14:10:03] [02miraheze/IncidentReporting] 07translatewiki 03220b84b - Localisation updates from https://translatewiki.net. [14:10:04] [ Main page - translatewiki.net ] - translatewiki.net. [14:10:05] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv1Mo [14:10:06] [02miraheze/MirahezeMagic] 07translatewiki 035e22104 - Localisation updates from https://translatewiki.net. [14:10:07] [ Main page - translatewiki.net ] - translatewiki.net. [14:10:08] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1MK [14:10:09] [02miraheze/ManageWiki] 07translatewiki 038a8fbcc - Localisation updates from https://translatewiki.net. [14:10:10] [ Main page - translatewiki.net ] - translatewiki.net. [14:11:55] miraheze/IncidentReporting/master/220b84b - translatewiki.net The build was fixed. https://travis-ci.com/miraheze/IncidentReporting/builds/154095397 [14:12:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [14:14:02] miraheze/CreateWiki/master/f8a5c90 - translatewiki.net The build was fixed. https://travis-ci.com/miraheze/CreateWiki/builds/154095401 [14:15:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:43:40] Zppix, paladox: can you fix the i18n cache on https://wiki.conworlds.org/w/index.php?title=Main_Page&action=info? [16:43:41] [ No such action - Constructed Worlds ] - wiki.conworlds.org [16:43:44] works on testwiki [16:43:53] ⧼restriction-protect⧽ Allow all users (infinite) [16:44:31] !log running lc on mw* [16:45:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:46:43] done [16:48:03] works on testwiki because it was overridden [16:48:45] paladox: are you sure? [16:48:50] yes [16:49:01] there might be slight chance of a bug then [16:51:02] paladox: that's core right? [16:52:42] I have no idea :) [16:53:04] I'm 99% sure, It's missing from i18n altogether [16:53:21] We can probably hack a fix through Magic repo but I'll file upstream [16:59:50] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 355 failures. Last run 6 minutes ago with 355 failures. Failed resources (up to 3 shown) [17:00:55] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 387 failures. Last run 7 minutes ago with 387 failures. Failed resources (up to 3 shown) [17:00:59] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 375 failures. Last run 7 minutes ago with 375 failures. Failed resources (up to 3 shown) [17:01:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 382 failures. Last run 7 minutes ago with 382 failures. Failed resources (up to 3 shown): File[wiki.meeusen.net],File[wiki.meeusen.net_private],File[wiki.drones4nature.info],File[wiki.drones4nature.info_private] [17:01:23] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 372 failures. Last run 7 minutes ago with 372 failures. Failed resources (up to 3 shown) [17:01:31] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 379 failures. Last run 7 minutes ago with 379 failures. Failed resources (up to 3 shown): File[speleo.wiki],File[speleo.wiki_private],File[disabled.life],File[disabled.life_private] [17:02:07] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 354 failures. Last run 8 minutes ago with 354 failures. Failed resources (up to 3 shown) [17:02:16] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 6 minutes ago with 51 failures. Failed resources (up to 3 shown): File[thelonsdalebattalion.co.uk],File[thelonsdalebattalion.co.uk_private],File[allthetropes.org],File[allthetropes.org_private] [17:02:27] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 376 failures. Last run 8 minutes ago with 376 failures. Failed resources (up to 3 shown) [17:02:58] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 370 failures. Last run 8 minutes ago with 370 failures. Failed resources (up to 3 shown): File[disabled.life_private],File[wiki.teessidehackspace.org.uk],File[wiki.teessidehackspace.org.uk_private],File[nenawiki.org] [17:03:03] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 372 failures. Last run 9 minutes ago with 372 failures. Failed resources (up to 3 shown) [17:03:06] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 371 failures. Last run 9 minutes ago with 371 failures. Failed resources (up to 3 shown) [17:04:06] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 179 failures. Last run 9 minutes ago with 179 failures. Failed resources (up to 3 shown): File[nenawiki.org_private],File[reviwiki.info],File[reviwiki.info_private],File[wiki.tulpa.info] [17:04:18] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [17:04:59] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:00] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:11] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:15] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:19] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:05:55] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:06:03] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [17:06:13] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:06:34] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:06:37] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [17:06:38] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [17:07:18] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [17:22:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:24:22] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2607:5300:205:200::17f6/cpweb [17:26:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:27:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:28:56] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1ds [17:28:58] [02miraheze/mw-config] 07Reception123 036b80ef1 - wgCategoryPagingLimit to 1500 for nenawikiwiki per req [17:34:01] * hispano76 greetings [17:34:40] * quirc waves [17:41:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:205:200::17f6/cpweb [17:44:35] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:02:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb [18:06:05] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:11:27] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:53] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:58] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 5.192 second response time [18:16:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.161.32.127/cpweb [18:17:06] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.865 second response time [18:18:20] paladox: bug fix is being worked on by me and Reedy now. We can either update when it goes to MW1.34 backport or hack Magic for now [18:19:30] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:21:47] ok [18:23:09] paladox: do you want me to hack Magic or are we gonna just wait? [18:23:40] if there's a fix i can just cherry pick it [18:24:49] paladox: Reedy’s reviewing now, spotted something else [18:24:56] ok [18:24:57] https://phabricator.wikimedia.org/T248112 [18:24:58] [ ⚓ T248112 missing from MediaWiki i18n ] - phabricator.wikimedia.org [18:52:48] paladox: insane recursive loop is extreme [19:01:43] paladox: based on the ensuing conversation, I think we might be best using the Magic hack temp [19:01:57] ok [19:03:19] paladox: I’ll do later then [19:03:23] ok [19:43:34] [02MirahezeMagic] 07RhinosF1 opened pull request 03#102: Add overrides to avoid missing i18n - 13https://git.io/Jv1N7 [19:51:09] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#102: Add overrides to avoid missing i18n - 13https://git.io/Jv1N7 [19:51:21] paladox: ^ [19:52:23] [02MirahezeMagic] 07paladox reviewed pull request 03#102 commit - 13https://git.io/Jv1Ae [19:52:33] comment ^ [19:56:27] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1AT [19:56:28] [02miraheze/ManageWiki] 07JohnFLewis 03600fa63 - T5258: Disallow special namespaces being created [19:56:53] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#102: Add overrides to avoid missing i18n - 13https://git.io/Jv1N7 [19:57:06] paladox: yes [19:57:52] [02MirahezeMagic] 07paladox closed pull request 03#102: Add overrides to avoid missing i18n - 13https://git.io/Jv1N7 [19:57:53] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jv1Ak [19:57:55] [02miraheze/MirahezeMagic] 07RhinosF1 0369cc793 - Add overrides to avoid missing i18n (#102) * Add overrides to avoid missing i18n Upstream fix could be a while, deploying farm-wide fix * Add en and qqq for new override * Update en.json [19:58:31] paladox: ping when cache has been rebuilt [19:58:33] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jv1AI [19:58:34] [02miraheze/mediawiki] 07paladox 0314a8cfe - Update MM [20:12:02] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv1A8 [20:12:04] [02miraheze/ssl] 07Reception123 034883b43 - add wiki.animalrebellion.cz cert [20:12:24] paladox: has cache ran [20:12:28] Hi Reception123 [20:12:31] hi RhinosF1 [20:12:38] Thanks for your fix JohnLewis [20:12:47] Nope, i got distracted [20:12:49] running now [20:13:30] RhinosF1: You're welcome [20:18:59] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://git.io/Jv1Az [20:19:01] [02miraheze/dns] 07paladox 03e638d07 - Remove enc.for.uz Not pointing at our nameservers anymore. [20:19:19] paladox: did we have task for that? [20:19:30] Was it not flapping between us and someone else? [20:20:05] RhinosF1: https://phabricator.miraheze.org/T5174 [20:20:06] [ ⚓ T5174 enc.for.uz itermittently using both yandex and miraheze dns ] - phabricator.miraheze.org [20:20:20] Its hard coded cp4, we've shutdown that. [20:20:21] That would be it [20:21:04] paladox: can you [20:21:08] Update task? [20:21:57] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jv1Ar [20:21:59] [02miraheze/ssl] 07paladox 03dbc6ba0 - Remove enc.for.uz [20:22:00] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:22:02] [02ssl] 07paladox opened pull request 03#275: Remove enc.for.uz - 13https://git.io/Jv1Ao [20:22:15] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/Jv1AK [20:22:17] [02miraheze/ssl] 07paladox 038655575 - Delete enc.for.uz.crt [20:22:18] [02ssl] 07paladox synchronize pull request 03#275: Remove enc.for.uz - 13https://git.io/Jv1Ao [20:22:33] [02ssl] 07paladox closed pull request 03#275: Remove enc.for.uz - 13https://git.io/Jv1Ao [20:22:34] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/Jv1A6 [20:22:36] [02miraheze/ssl] 07paladox 031893665 - Remove enc.for.uz (#275) * Remove enc.for.uz * Delete enc.for.uz.crt [20:22:37] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:22:39] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [20:24:25] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv1Ay [20:24:26] [02miraheze/ssl] 07Reception123 03fc9c5cd - add devwiki.anders.moldin.com cert [20:26:21] !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"enc.for.uz"', null ) WHERE wiki_dbname = 'uzencwiki'; - db4 [20:26:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1Ab [20:30:10] [02miraheze/services] 07MirahezeSSLBot 037328ac4 - BOT: Updating services config for wikis [20:43:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb [20:47:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:47:26] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:48:42] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [20:50:49] RECOVERY - cp7 Stunnel Http for mw7 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.014 second response time [20:51:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:55:52] Is it my connection that's slow or is it something from Infra? curiosity [20:56:15] infra [20:57:06] ah, ok [21:10:06] !log rename test1wiki to test2wiki [21:10:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:11:28] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1xz [21:11:29] [02miraheze/ssl] 07paladox 0364514c9 - Redirect test1 to test2 [21:13:44] !log apt-get upgrade - test2 [21:14:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:14:16] paladox: update check sheet pls [21:14:51] i've forgotten where that is [21:15:02] even then, couldn't you update? [21:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1xK [21:15:09] [02miraheze/services] 07MirahezeSSLBot 03d6a1642 - BOT: Updating services config for wikis [21:15:39] paladox: I could but SRE could manage it [21:16:03] We also need to sort out the script for the new server pages [21:17:42] !log reboot test2 [21:17:50] script for the new server pages? [21:18:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:19:03] paladox: auto create the new pages for the servers [21:19:25] paladox: https://meta.miraheze.org/wiki/Tech:Cloud_Server_Comparison - can you do db6/7 notes [21:19:27] [ Tech:Cloud Server Comparison - Miraheze Meta ] - meta.miraheze.org [21:19:33] ok, that's low priority atm for me [21:20:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1x1 [21:20:02] [02miraheze/puppet] 07paladox 038c8e501 - Update test2.yaml [21:20:12] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 36 seconds ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [21:20:31] paladox: I’m going to do it, I just need the spec from you [21:21:08] Test2 has 1gb of ram, 3.5ghz cpu (1) [21:21:20] and 20g disk [21:23:48] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [21:24:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jv1x9 [21:24:42] [02miraheze/mw-config] 07paladox 03cb77f68 - test1wiki -> test2wiki [21:24:43] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:24:45] [02mw-config] 07paladox opened pull request 03#2946: test1wiki -> test2wiki - 13https://git.io/Jv1xH [21:25:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jv1xQ [21:25:08] [02miraheze/mw-config] 07paladox 03e86dd35 - Update Wikibase.php [21:25:10] [02mw-config] 07paladox synchronize pull request 03#2946: test1wiki -> test2wiki - 13https://git.io/Jv1xH [21:25:18] paladox: is that same across lot or? [21:25:25] [02mw-config] 07paladox closed pull request 03#2946: test1wiki -> test2wiki - 13https://git.io/Jv1xH [21:25:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv1x7 [21:25:28] [02miraheze/mw-config] 07paladox 03bee4176 - test1wiki -> test2wiki (#2946) * test1wiki -> test2wiki * Update Wikibase.php [21:25:29] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [21:25:31] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [21:25:36] RhinosF1 no [21:26:10] rdb1 has a 10g disk, 3gb of ram, 10g disk and 2 cpu @ 3.5ghz [21:26:14] same for rdb2 [21:26:42] miraheze/mw-config/paladox-patch-4/e86dd35 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/664601170 [21:27:10] paladox: I’ll try and do some Monday [21:27:33] The page frame is on https://phabricator.miraheze.org/P282 [21:27:34] [ ✎ P282 Server page template ] - phabricator.miraheze.org [21:27:58] Name location usage I can do myself [21:28:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1xd [21:28:06] [02miraheze/mw-config] 07paladox 035db1944 - switch test2wiki to db7 [21:28:16] I don’t have a store link but not sure that applies now [21:28:31] And the something is based on the spec [21:28:40] Rest of parameters I need [21:29:04] Or I can whack the code into ZppixBot and allow you to run anytime as well [21:30:43] [9b835816a74f553fece17b51] 2020-03-19 21:29:57: Excepción grave de tipo "Wikimedia\Rdbms\DBQueryError" Login error :( [21:30:50] (metawiki) [21:31:10] paladox: ^ [21:31:18] hmm [21:31:45] paladox: what’s the stack trace say? [21:32:39] RhinosF1 i think this is related to test1 [21:32:44] fixed in https://git.io/Jv1xd [21:32:45] [ Comparing bee41768e193...5db19444dd0f · miraheze/mw-config · GitHub ] - git.io [21:32:50] hispano76 ^ [21:33:33] paladox: test1. Isn’t redirecting yet as well [21:37:18] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pI [21:37:19] [02miraheze/dns] 07paladox 037974057 - Remove misc2, cp4, misc4, puppet1 and test1 from dns [21:38:14] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pq [21:38:15] [02miraheze/dns] 07paladox 03f04c3f9 - Update miraheze.org [21:38:37] :) [21:40:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb [21:40:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv1p8 [21:40:54] [02miraheze/puppet] 07paladox 035c8ca85 - varnish: Remove test1 [21:40:56] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [21:40:57] [02puppet] 07paladox opened pull request 03#1294: varnish: Remove test1 - 13https://git.io/Jv1pB [21:41:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv1p0 [21:41:33] [02miraheze/puppet] 07paladox 037df531e - Update stunnel.conf [21:41:35] [02puppet] 07paladox synchronize pull request 03#1294: varnish: Remove test1 - 13https://git.io/Jv1pB [21:41:41] [02puppet] 07paladox closed pull request 03#1294: varnish: Remove test1 - 13https://git.io/Jv1pB [21:41:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jv1pE [21:41:44] [02miraheze/puppet] 07paladox 03b416220 - varnish: Remove test1 (#1294) * varnish: Remove test1 * Update stunnel.conf [21:41:50] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:42:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pz [21:42:52] [02miraheze/puppet] 07paladox 039beea14 - gluster: Remove lizardfs6 and test1 [21:43:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pg [21:43:18] [02miraheze/puppet] 07paladox 03e2cb61f - irc: Remove test1 [21:43:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1p2 [21:43:46] [02miraheze/puppet] 07paladox 03e353cb2 - services: Remove test1 [21:43:47] Hello nagger! If you have any questions, feel free to ask and someone should answer soon. [21:44:18] hello people, I don't mean to be someone who annoys you, but the miraheze servers are really slow for me and I haven't read any info about it [21:45:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1p6 [21:45:10] [02miraheze/puppet] 07paladox 0380175fb - varnish: Remove lizardfs6 and test1 from firewall [21:45:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pi [21:45:59] [02miraheze/puppet] 07paladox 03ea0ea53 - db: Remove lizardfs6 and test1 from firewall [21:46:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pP [21:46:32] [02miraheze/puppet] 07paladox 037d2fd17 - Update nrpe.cfg.erb [21:46:38] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 2.116 second response time [21:46:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pX [21:46:52] [02miraheze/puppet] 07paladox 03759b016 - Update nrpe.cfg.erb [21:47:47] RhinosF1 works now [21:47:58] paladox: ok [21:48:00] nagger hi, yup this is known and we apologise :( [21:48:06] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:48:31] okay, as long as it's not something on my end and you're working on it I'm relieved [21:48:33] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw7 [21:48:37] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:48:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pQ [21:48:57] [02miraheze/puppet] 07paladox 0316125df - Update init.pp [21:49:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1p7 [21:49:06] [02miraheze/puppet] 07paladox 0353e904d - Update gluster1.yaml [21:49:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pd [21:49:37] [02miraheze/puppet] 07paladox 032e94d8f - irc: Remove lizardfs6 [21:49:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pF [21:50:00] [02miraheze/puppet] 07paladox 03521786f - Update init.pp [21:50:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv1pb [21:50:26] [02miraheze/puppet] 07paladox 03dbe1b89 - services: Remove lizardfs6 from firewall [21:52:00] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18699 bytes in 0.830 second response time [21:56:19] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [21:58:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:58:18] PROBLEM - cp7 Stunnel Http for test1 on cp7 is CRITICAL: connect to address localhost and port 8084: Connection refusedHTTP CRITICAL - Unable to open TCP socket [21:59:25] PROBLEM - cp6 Stunnel Http for test1 on cp6 is CRITICAL: connect to address localhost and port 8084: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:00:58] PROBLEM - cp8 Stunnel Http for test1 on cp8 is CRITICAL: connect to address localhost and port 8084: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:00:59] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: connect to address localhost and port 8084: Connection refusedHTTP CRITICAL - Unable to open TCP socket [22:02:12] paladox: can you turn them off now? [22:02:25] Turn what off? [22:03:00] paladox: icinga for test1, it's decom now, right? [22:03:15] i have turned it off, at least for varnish [22:03:18] yes [22:04:29] paladox: what about the 4 criticals that just went off? [22:04:46] that is what i just said :) "i have turned it off, at least for varnish" [22:04:53] ok [22:15:12] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:15:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb [22:16:12] PROBLEM - bacula2 Bacula Databases db7 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [22:18:15] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:18:51] paladox: why is test1 pinging me? [22:19:04] What do you mean pinging you? [22:19:11] where? [22:19:28] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:19:49] paladox: right above, the word critical pings me. My point is a decom’d server is still firing alerts [22:20:05] well that's you problem to fix, nothing i can fix :) [22:20:08] it's your side [22:20:21] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:29] The fact a decom’d server is still alerting isn’t [22:20:37] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:20:45] no, but pinging, yes. [22:21:02] I want to be pinged to servers that are critical and in use [22:21:20] My point is, when are notifications actually going for it? [22:21:28] ok, and huh? [22:21:48] If an alert is going off for test1, they’re enabled [22:21:59] When will they go? [22:22:08] Ok, i've disabled varnish. Test1 will go in about a week. [22:22:19] Ok [22:22:20] question is why are you getting pinged for critical alerts [22:22:30] that seems pointless [22:22:36] Zppix: It’s called being nosey mainly. [22:22:44] cause if you know about the alerts then more then likely we know [22:22:47] so then not my problem? [22:23:03] Zppix: I would hope [22:23:18] so like paladox said the issue is your end not ours [22:23:29] ^ [22:23:44] The point seems to missed, a server not serving anything surely doesn’t need to alert [22:24:35] but you telling us about it pinging you is wasting our time. Its your problem not ours. [22:24:36] RhinosF1: i feel like this going in circles... so heres the conclusion of this discussion, don't have it ping you when it says critical or just deal with it [22:24:39] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.315 second response time [22:24:54] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 6.841 second response time [22:24:54] per Zppix [22:24:58] further discussion will be pointless and waste of our time [22:25:45] * RhinosF1 is going to sleep [22:26:04] The monitoring is for us fyi :)