[00:02:24] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3967 MB (20% inode=93%); [00:08:39] PROBLEM - bacula2 Bacula Private Git on bacula2 is WARNING: WARNING: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (1.1 weeks ago) [00:10:50] [02mw-config] 07paladox closed pull request 03#2967: Add 'articles' to wgUserProfileDisplay - 13https://git.io/JvNrI [00:10:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvNrR [00:10:53] [02miraheze/mw-config] 07paladox 03a3bac1a - Add 'articles' to wgUserProfileDisplay (#2967) [00:10:55] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:10:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [00:16:21] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvNru [00:16:22] [02miraheze/mw-config] 07paladox 038a343ee - Remove att from wgRelatedArticlesFooterWhitelistedSkins [00:30:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNrS [00:30:10] [02miraheze/services] 07MirahezeSSLBot 039915817 - BOT: Updating services config for wikis [00:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNrd [00:35:09] [02miraheze/services] 07MirahezeSSLBot 03e9a01d4 - BOT: Updating services config for wikis [00:45:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNrh [00:45:02] [02miraheze/mw-config] 07paladox 03e12bb87 - Fix overridedefault value [00:45:03] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:45:05] [02mw-config] 07paladox opened pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:46:29] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:46:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoJ [00:46:32] [02miraheze/mw-config] 07paladox 03236db51 - Update ManageWikiSettings.php [00:46:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoU [00:46:49] [02miraheze/mw-config] 07paladox 03aa67b0c - Update LocalSettings.php [00:46:50] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:50:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoI [00:50:50] [02miraheze/mw-config] 07paladox 038256b52 - Update ManageWikiSettings.php [00:50:51] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:51:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNot [00:51:40] [02miraheze/mw-config] 07paladox 03a6a1b8a - Update LocalSettings.php [00:51:41] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:53:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvNoO [00:53:35] [02miraheze/mw-config] 07paladox 03cc1678e - Fix typo [00:57:00] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:57:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoG [00:57:03] [02miraheze/mw-config] 07paladox 034bef2f4 - Update ManageWikiSettings.php [00:57:13] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:57:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoZ [00:57:16] [02miraheze/mw-config] 07paladox 037c3aa60 - Update ManageWikiSettings.php [00:58:15] [02mw-config] 07paladox synchronize pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [00:58:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoc [00:58:18] [02miraheze/mw-config] 07paladox 03db03d2e - Update LocalSettings.php [01:00:35] [02mw-config] 07paladox closed pull request 03#2968: Fix overridedefault value - 13https://git.io/JvNrj [01:00:37] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±2] 13https://git.io/JvNoC [01:00:38] [02miraheze/mw-config] 07paladox 037abfb8e - Fix overridedefault value (#2968) * Fix overridedefault value * Update ManageWikiSettings.php * Update LocalSettings.php * Update ManageWikiSettings.php * Update LocalSettings.php * Update ManageWikiSettings.php * Update ManageWikiSettings.php * Update LocalSettings.php [01:00:40] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:00:41] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [01:15:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNoX [01:15:09] [02miraheze/services] 07MirahezeSSLBot 0326a0d80 - BOT: Updating services config for wikis [01:16:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNoD [01:16:42] [02miraheze/mw-config] 07paladox 03ecad484 - Update LocalSettings.php [01:16:43] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:16:45] [02mw-config] 07paladox opened pull request 03#2969: Update LocalSettings.php - 13https://git.io/JvNoy [01:17:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvNo9 [01:17:19] [02miraheze/mw-config] 07paladox 037f0199f - Update ManageWikiSettings.php [01:17:21] [02mw-config] 07paladox synchronize pull request 03#2969: Update LocalSettings.php - 13https://git.io/JvNoy [01:17:29] [02mw-config] 07paladox edited pull request 03#2969: DisqusShortname -> wgPageDisqusShortname - 13https://git.io/JvNoy [01:17:33] [02mw-config] 07paladox closed pull request 03#2969: DisqusShortname -> wgPageDisqusShortname - 13https://git.io/JvNoy [01:17:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03new-cache [+0/-0/±2] 13https://git.io/JvNoH [01:17:36] [02miraheze/mw-config] 07paladox 036c10768 - DisqusShortname -> wgPageDisqusShortname (#2969) * Update LocalSettings.php * Update ManageWikiSettings.php [01:17:38] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:17:39] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [04:34:08] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.07, 1.71, 1.54 [04:36:55] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.34, 1.57, 1.52 [05:56:05] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.85, 1.71, 1.51 [05:58:45] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.46, 1.62, 1.52 [06:06:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvN1w [06:06:39] [02miraheze/ssl] 07Reception123 03956b0c3 - add en.wikireserve.com cert [06:18:16] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:24:05] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:27:28] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2969 MB (12% inode=93%); [06:29:26] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:35:01] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:05:58] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.05, 2.08, 1.71 [07:08:44] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.22, 1.70, 1.62 [08:36:44] [02mw-config] 07RhinosF1 opened pull request 03#2970: Add DT extensions for testing - 13https://git.io/JvNyj [08:38:09] [02mw-config] 07RhinosF1 synchronize pull request 03#2970: Add DT extensions for testing - 13https://git.io/JvNyj [08:39:28] [02mw-config] 07RhinosF1 synchronize pull request 03#2970: Add DT extensions for testing - 13https://git.io/JvNyj [08:40:20] [02mw-config] 07RhinosF1 synchronize pull request 03#2970: Add DT extensions for testing - 13https://git.io/JvNyj [08:41:36] Reception123: ^ can you check while I do the MediaWiki PR [08:42:21] and fix any mistakes [08:49:05] It's 1.35+ :( [08:50:44] https://github.com/wikimedia/mediawiki-extensions-DiscussionTools/commit/13777278ff97c2bbacc8b07273206cc813a7a353#diff-ef32f2a18f1fd51247cbaab39dcde592 [08:50:45] [ Bump MW dependency to 1.35 · wikimedia/mediawiki-extensions-DiscussionTools@1377727 · GitHub ] - github.com [09:16:38] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.74, 1.77, 1.62 [09:16:59] RhinosF1: not merging anything right now because John was working on converting settings so I don't want to mess anything up with that [09:17:21] Reception123: we can't anyway, see my comment straight after [09:19:19] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.38, 1.61, 1.58 [09:25:29] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.72, 1.77, 1.67 [09:28:09] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.37, 1.62, 1.63 [09:35:26] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 9.58, 6.69, 5.55 [09:38:04] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 3.32, 5.36, 5.23 [09:55:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNHC [09:55:09] [02miraheze/services] 07MirahezeSSLBot 0382a878e - BOT: Updating services config for wikis [10:00:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNHw [10:00:10] [02miraheze/services] 07MirahezeSSLBot 03cae3f6a - BOT: Updating services config for wikis [11:26:02] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=93%); [11:33:43] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 8.66, 7.17, 5.74 [11:36:26] RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: OK - load average: 4.84, 6.33, 5.67 [13:25:44] Paladox: 503 on meta [13:26:26] {{fixed}} [13:26:31] resolved [14:24:18] [02dns] 07revi opened pull request 03#141: DNS Repo purge 2020-04 - 13https://git.io/JvNxw [14:25:14] [02dns] 07revi commented on pull request 03#141: DNS Repo purge 2020-04 - 13https://git.io/JvNx6 [14:27:28] revi: thanks for the help :) [15:03:47] JohnLewis, Reception123: [67183d27fe5303fbe0c85475] 2020-04-06 15:03:26: Fatal exception of type "ConfigException" [15:03:57] After an Upload [15:21:40] An issue with prod and testing environments differing if that’s on test2 [15:23:52] JohnLewis: yes [15:24:18] Wasn’t a question, was a statement :) [15:30:47] If that’s on test2 sounded like a question [15:30:55] paladox: we got a response on Phab for ISBN [15:31:03] paladox: we can also use zotero [15:31:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvNjU [15:31:36] [02miraheze/puppet] 07paladox 03b97af24 - citoid Set wskey to false [15:58:32] [02dns] 07paladox closed pull request 03#141: DNS Repo purge 2020-04 - 13https://git.io/JvNxw [15:58:34] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-9/±0] 13https://git.io/JvNjF [15:58:35] [02miraheze/dns] 07revi 032557b4f - DNS Repo purge 2020-04 (#141) * Remove browndust.wiki Not hosted by Miraheze NS ;; QUESTION SECTION: ;browndust.wiki. IN NS ;; ANSWER SECTION: browndust.wiki. 1800 IN NS dns1.registrar-servers.com. browndust.wiki. 1800 IN NS dns2.registrar-servers.com. Signed-off-by: Yongmin Hong * Remove islamkosh.tk NXDOMAIN, A record redirects to dot.tk ;; QUESTION [15:58:35] SECTION: ;islamkosh.tk. IN NS ;; AUTHORITY SECTION: tk. 5 IN SOA a.ns.tk. joost\.zuurbier.dot.tk. 1586181727 10800 3600 604800 5 Signed-off-by: Yongmin Hong * Remove itpedia.sk Pending record deletion EPP Status: pendingDelete, serverHold, redemptionPeriod Signed-off-by: Yongmin Hong * Remove knuxwiki.com DNS no longer pointing to Miraheze NS ;; QUESTION SECTION: ;knuxwiki.com. IN NS [15:58:35] ;; ANSWER SECTION: knuxwiki.com. 172800 IN NS dns2.namecheaphosting.com. knuxwiki.com. 172800 IN NS dns1.namecheaphosting.com. Signed-off-by: Yongmin Hong * Remove nonsensopedia.org DNS no longer pointing to Miraheze NS ;; QUESTION SECTION: ;nonsensopedia.org. IN NS ;; ANSWER SECTION: nonsensopedia.org. 3600 IN NS ns3211.ispapi.net. nonsensopedia.org. 3600 IN NS ns2224.ispapi.net. nonsensopedia.org. [15:58:35] 3600 IN NS ns1188.ispapi.net. Signed-off-by: Yongmin Hong * Remote prfm.wiki DNS no longer pointing to Miraheze NS ;; QUESTION SECTION: ;prfm.wiki. IN NS ;; ANSWER SECTION: prfm.wiki. 3600 IN NS sns-00.interhact.net. prfm.wiki. 3600 IN NS sns-01.interhact.net. Signed-off-by: Yongmin Hong * Remove wikiverte.pl Domain not registered. No information available about domain name [15:58:35] wikiverte.pl in the Registry NASK database. Signed-off-by: Yongmin Hong * Remove wintermoor.net Domain not registered No match for domain "WINTERMOOR.NET". Signed-off-by: Yongmin Hong * Remove yukesh.com.np DNS not using Miraheze NS ;; QUESTION SECTION: ;yukesh.com.np. IN NS ;; ANSWER SECTION: yukesh.com.np. 86400 IN NS dns1.namecheaphosting.com. yukesh.com.np. 86400 IN NS [15:58:36] dns2.namecheaphosting.com. Signed-off-by: Yongmin Hong [15:59:30] paladox: ^ could you please check if we have any SSL certs there? [16:00:01] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JvNjb [16:00:02] [02miraheze/ssl] 07paladox 031b5f6da - Remove two certificates [16:00:04] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:00:05] [02ssl] 07paladox opened pull request 03#282: Remove two certificates - 13https://git.io/JvNjN [16:00:31] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JvNjx [16:00:33] [02miraheze/ssl] 07paladox 035ec1e2d - Delete prfm.wiki.crt [16:00:34] [02ssl] 07paladox synchronize pull request 03#282: Remove two certificates - 13https://git.io/JvNjN [16:00:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/JvNjp [16:00:45] [02miraheze/ssl] 07paladox 0384b98c8 - Delete itpedia.sk.crt [16:00:46] [02ssl] 07paladox synchronize pull request 03#282: Remove two certificates - 13https://git.io/JvNjN [16:00:55] [02ssl] 07paladox closed pull request 03#282: Remove two certificates - 13https://git.io/JvNjN [16:00:56] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-2/±1] 13https://git.io/JvNjh [16:00:58] [02miraheze/ssl] 07paladox 0311d2efd - Remove two certificates (#282) * Remove two certificates * Delete prfm.wiki.crt * Delete itpedia.sk.crt [16:00:59] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [16:01:01] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [16:01:25] ^ paladox and if there's wikis that are still pointed since John hasn't rolled out the new changes yet you need to remove those from cw_wikis (make wgserver="") [16:02:02] RhinosF1: found anything else wrong with test1? [16:04:18] !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"wikiverte.pl"', null ) WHERE wiki_dbname = 'wikivertewiki'; [16:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:04:26] Reception123 test2! [16:04:28] not test1 [16:04:32] It’s not existed for a while! [16:04:40] But none on test2! [16:04:42] my bad hard to get used to calling it test2 [16:05:16] !log MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"yukesh.com.np"', null ) WHERE wiki_dbname = 'hamropediawiki'; [16:05:19] revi: did some cleanup did you [16:05:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:10:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAe4 [16:10:11] [02miraheze/services] 07MirahezeSSLBot 03b981a32 - BOT: Updating services config for wikis [16:17:29] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:17:54] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:18:02] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:18:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAe6 [16:18:15] [02miraheze/puppet] 07paladox 03976b5d9 - Update jobrunner.json.erb [16:18:22] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:18:46] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:18:49] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:18:58] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:19:02] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:19:25] ^ paladox [16:19:34] hmm [16:19:42] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.24, 1.82, 1.52 [16:20:03] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAeS [16:20:04] [02miraheze/ssl] 07paladox 037fd3857 - Remove www.itpedia.sk [16:22:07] * hispano76 greetings [16:22:55] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [16:23:41] hi hispano76 [16:23:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAeA [16:23:54] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[itpedia.sk] [16:23:55] [02miraheze/dns] 07paladox 0351b0bd1 - Depool cp3 [16:24:20] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.40, 1.64, 1.53 [16:24:25] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:32] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:33] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [16:24:50] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:56] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:25:07] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:25:22] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:26:41] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:34:17] PROBLEM - cp3 Stunnel Http for test2 on cp3 is CRITICAL: connect to address localhost and port 8090: Connection refusedHTTP CRITICAL - Unable to open TCP socket [16:37:09] PROBLEM - cp3 Stunnel Http for test2 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.003 second response time [16:38:44] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [16:38:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:39:00] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 17813 MB (73% inode=93%); [16:39:11] !log upgrade cp3 to buster [16:39:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:41:22] PROBLEM - cp3 Puppet on cp3 is CRITICAL: connect to address 128.199.139.216 port 5666: Connection refusedconnect to host 128.199.139.216 port 5666: Connection refused [16:41:42] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: connect to address 128.199.139.216 port 5666: Connection refusedconnect to host 128.199.139.216 port 5666: Connection refused [16:42:35] PROBLEM - cp3 Stunnel Http for test2 on cp3 is CRITICAL: connect to address 128.199.139.216 port 5666: Connection refusedconnect to host 128.199.139.216 port 5666: Connection refused [16:43:59] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.956 second response time [16:44:01] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [16:44:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [16:44:10] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [16:44:25] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [16:44:28] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.48, 2.00, 1.69 [16:45:21] PROBLEM - cp3 Stunnel Http for test2 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.003 second response time [16:46:43] PROBLEM - cp3 Puppet on cp3 is UNKNOWN: NRPE: Unable to read output [16:46:49] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is UNKNOWN: NRPE: Unable to read output [16:47:05] PROBLEM - cp3 Varnish Backends on cp3 is UNKNOWN: NRPE: Unable to read output [16:47:08] PROBLEM - cp3 Current Load on cp3 is UNKNOWN: NRPE: Unable to read output [16:48:00] PROBLEM - cp3 Stunnel Http for test2 on cp3 is UNKNOWN: NRPE: Unable to read output [16:49:54] PROBLEM - cp3 Stunnel Http for mon1 on cp3 is UNKNOWN: NRPE: Unable to read output [16:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAfz [16:50:10] [02miraheze/services] 07MirahezeSSLBot 036367067 - BOT: Updating services config for wikis [16:50:28] PROBLEM - cp3 Stunnel Http for mw5 on cp3 is UNKNOWN: NRPE: Unable to read output [16:50:30] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is UNKNOWN: NRPE: Unable to read output [16:50:31] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is UNKNOWN: NRPE: Unable to read output [16:50:35] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is UNKNOWN: NRPE: Unable to read output [16:52:06] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [16:52:08] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [16:52:25] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [16:52:33] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.87, 2.53, 2.06 [16:52:36] RECOVERY - cp3 Stunnel Http for mon1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 30936 bytes in 0.951 second response time [16:53:14] RECOVERY - cp3 Stunnel Http for mw5 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15324 bytes in 0.721 second response time [16:53:20] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.708 second response time [16:53:21] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.725 second response time [16:53:22] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.708 second response time [16:53:23] PROBLEM - cp3 Stunnel Http for test2 on cp3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.003 second response time [16:54:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:55:02] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1571 bytes in 1.239 second response time [16:55:10] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:55:13] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [16:57:18] !log rebooted cp3 [16:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:57:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:01:17] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [17:01:22] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 12% [17:01:30] [02dns] 07Pix1234 opened pull request 03#142: Create civwiki.org - 13https://git.io/JvAf5 [17:01:36] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1961 MB (8% inode=93%); [17:01:47] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.14, 0.46, 0.17 [17:03:44] [02dns] 07Reception123 closed pull request 03#142: Create civwiki.org - 13https://git.io/JvAf5 [17:03:46] [02miraheze/dns] 07Reception123 pushed 032 commits to 03master [+2/-0/±0] 13https://git.io/JvAfx [17:03:47] [02miraheze/dns] 07Pix1234 0323a5cd4 - Create civwiki.org [17:03:49] [02miraheze/dns] 07Reception123 03690f7e0 - Merge pull request #142 from Pix1234/patch-2 Create civwiki.org [17:04:05] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:09:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAJO [17:09:03] [02miraheze/puppet] 07paladox 03910aab8 - cp3: Lower cache size to 13g [17:09:26] [02miraheze/ssl] 07Pix1234 created branch 03civclassic 13https://git.io/JvAJ3 [17:09:27] [02ssl] 07Pix1234 created branch 03civclassic - 13https://git.io/vxP9L [17:10:32] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03civclassic [+0/-0/±1] 13https://git.io/JvAJn [17:10:34] [02miraheze/ssl] 07Pix1234 0300113c1 - + civclassic setup [17:11:08] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAJc [17:11:09] [02miraheze/dns] 07paladox 0353be341 - Revert "Depool cp3" This reverts commit 51b0bd12a94be6a8b950ddf3f2c9c28a61810097. [17:13:26] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 4011 MB (16% inode=93%); [17:14:55] !log apt-get upgrade - ns1 [17:15:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:16:04] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03civclassic [+1/-0/±0] 13https://git.io/JvAJE [17:16:06] [02miraheze/ssl] 07Pix1234 035da3b20 - Create civclassic.civwiki.org.crt [17:16:26] [02ssl] 07Pix1234 opened pull request 03#283: add civclassic cert - 13https://git.io/JvAJz [17:16:39] [02ssl] 07Pix1234 closed pull request 03#283: add civclassic cert - 13https://git.io/JvAJz [17:16:40] [02miraheze/ssl] 07Pix1234 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JvAJ2 [17:16:42] [02miraheze/ssl] 07Pix1234 03485fa04 - add civclassic cert (#283) * + civclassic setup * Create civclassic.civwiki.org.crt [17:16:45] [02ssl] 07Pix1234 deleted branch 03civclassic - 13https://git.io/vxP9L [17:16:46] [02miraheze/ssl] 07Pix1234 deleted branch 03civclassic [17:28:41] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:29:36] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:29:49] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:30:15] Zppix did you commit the key? [17:30:19] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:30:24] paladox: yes [17:30:32] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:30:44] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:30:48] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:30:56] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:31:03] the key name looks wrong [17:31:28] oops [17:31:31] i forgot the .org [17:31:47] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[civclassic.civwiki.org_private] [17:32:28] fixed [17:32:46] i was just about to fix it [17:33:43] paladox: did you rerun puppet? [17:33:53] no, not yet [17:33:58] paladox: ok [17:35:46] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:36:27] paladox: i am running it on jobrunner [17:36:33] ok [17:37:07] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:16] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:35] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:43] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:39:51] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:39:57] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:40:07] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAUm [17:40:09] [02miraheze/ssl] 07paladox 03e68faec - Fix typo [17:40:46] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:45:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAUl [17:45:09] [02miraheze/services] 07MirahezeSSLBot 03ae343a3 - BOT: Updating services config for wikis [17:47:35] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:13:26] RhinosF1: https://phabricator.miraheze.org/T5209#102263 [18:13:27] [ ⚓ T5209 Please add an interwiki link for simpleelectronics wiki. ] - phabricator.miraheze.org [18:13:59] Reception123: okay... [18:22:38] RhinosF1: just for the future [18:46:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvATj [18:46:33] [02miraheze/puppet] 07paladox 03f9faf14 - Revert "Update jobrunner.json.erb" This reverts commit 976b5d97cfbbb34471dd4daae0449d51b9d7b071. [19:06:26] * hispano76 a strange and fast AWB speed at Miraheze [19:08:32] OVH really improved the servers a lot. :) [19:10:50] * hispano76 gives everyone cookies and soda :D [19:31:56] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki mundanapediawiki /home/reception/Mundanapedia-20160208152735.xml [19:32:12] hispano76: well before we were on RamNode, now we changed to OVH for most of our servers [19:32:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:35:22] ah, yep :) [19:35:33] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAIS [19:35:34] [02miraheze/mw-config] 07Pix1234 03c612ad7 - Set delete rev limit to 1k [19:50:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvALG [19:50:11] [02miraheze/services] 07MirahezeSSLBot 032e94f6b - BOT: Updating services config for wikis [20:58:57] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAth [20:58:58] [02miraheze/puppet] 07JohnFLewis 033260d5a - add new key [22:04:24] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03new-cache [+0/-0/±1] 13https://git.io/JvAm6 [22:04:26] [02miraheze/mw-config] 07JohnFLewis 0324e4a19 - more changes [22:06:41] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03new-cache [+0/-0/±2] 13https://git.io/JvAmi [22:06:42] [02miraheze/mw-config] 07JohnFLewis 036f305de - more changes [22:08:09] [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JvAmX [22:08:11] [02miraheze/CreateWiki] 07JohnFLewis 03b0f54f6 - final fixes to code [22:10:30] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvAmQ [22:10:32] [02miraheze/ManageWiki] 07JohnFLewis 03f0a8341 - push final code fixes [22:14:43] [02miraheze/MirahezeMagic] 07JohnFLewis pushed 032 commits to 03master [+7/-0/±42] 13https://git.io/JvAmx [22:14:45] [02miraheze/MirahezeMagic] 07JohnFLewis 031d3c063 - use new cwPrivate [22:14:46] [02miraheze/MirahezeMagic] 07JohnFLewis 03c88e45b - Merge branch 'master' of www.github.com:miraheze/MirahezeMagic [22:31:17] !log depool mw4 for isolated deployment [22:31:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:35:07] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [22:35:07] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [22:35:45] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [22:36:55] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw4 [22:51:04] !log repool mw4 to get real load conditions [22:51:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:51:20] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [22:51:36] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [22:52:04] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [22:52:36] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [22:53:15] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [23:10:37] PROBLEM - public.revi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'public.revi.wiki' expires in 15 day(s) (Wed 22 Apr 2020 23:05:01 GMT +0000). [23:13:32] PROBLEM - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zw.fontainebleau-avon.fr' expires in 15 day(s) (Wed 22 Apr 2020 23:07:48 GMT +0000). [23:17:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAOl [23:17:35] [02miraheze/ssl] 07MirahezeSSLBot 0398f61f6 - Bot: Update SSL cert for public.revi.wiki [23:23:46] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 1 failures [23:24:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvAOa [23:24:04] [02miraheze/ssl] 07MirahezeSSLBot 039f2fa2c - Bot: Update SSL cert for zw.fontainebleau-avon.fr [23:24:41] RECOVERY - public.revi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'public.revi.wiki' will expire on Sun 05 Jul 2020 22:17:27 GMT +0000. [23:33:02] RECOVERY - zw.fontainebleau-avon.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'zw.fontainebleau-avon.fr' will expire on Sun 05 Jul 2020 22:23:56 GMT +0000.