[00:01:22] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [00:02:02] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:03:15] PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 26.04, 18.28, 14.40 [00:04:19] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:12] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.11, 18.56, 14.98 [00:06:41] PROBLEM - ping4 on cloud5 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:06:44] PROBLEM - cloud5 APT on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:08] PROBLEM - cloud5 SSH on cloud5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:26] PROBLEM - cloud5 NTP time on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:49] PROBLEM - cloud5 Disk Space on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:07:52] PROBLEM - Host cloud5 is DOWN: PING CRITICAL - Packet loss = 100% [00:10:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 1.64, 1.09, 0.48 [00:10:39] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:10:47] RECOVERY - cloud4 NTP time on cloud4 is OK: NTP OK: Offset -0.002007842064 secs [00:12:05] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [00:12:15] RECOVERY - cloud4 Disk Space on cloud4 is OK: DISK OK - free space: / 3561525 MB (99% inode=99%); [00:18:04] PROBLEM - cloud4 Puppet on cloud4 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg],Package[proxmox-ve] [00:19:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBv1 [00:19:54] [02miraheze/puppet] 07paladox 030eb7cd4 - cloud: Purge cloud-init [00:20:20] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBvy [00:20:21] [02miraheze/dns] 07Southparkfan 03ebd3cb2 - Replace cp9 with cp12 [00:21:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBvd [00:21:48] [02miraheze/mw-config] 07Universal-Omega 0374dde4b - Allow DiscordNotifications on private wikis [00:21:49] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [00:21:51] [02mw-config] 07Universal-Omega opened pull request 03#3690: Allow DiscordNotifications on private wikis - 13https://git.io/JtBvF [00:22:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBvx [00:22:38] [02miraheze/mw-config] 07Universal-Omega 03b6de832 - Update GlobalExtensions.php [00:22:39] [02mw-config] 07Universal-Omega synchronize pull request 03#3690: Allow DiscordNotifications on private wikis - 13https://git.io/JtBvF [00:22:55] miraheze/mw-config - Universal-Omega the build passed. [00:23:43] miraheze/mw-config - Universal-Omega the build passed. [00:24:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBfJ [00:24:24] [02miraheze/mw-config] 07Universal-Omega 03fbc0cfe - Update ManageWikiSettings.php [00:24:26] [02mw-config] 07Universal-Omega synchronize pull request 03#3690: Allow DiscordNotifications on private wikis - 13https://git.io/JtBvF [00:24:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBfT [00:24:46] [02miraheze/mw-config] 07Universal-Omega 031cf0b51 - Update ManageWikiExtensions.php [00:24:47] [02mw-config] 07Universal-Omega synchronize pull request 03#3690: Allow DiscordNotifications on private wikis - 13https://git.io/JtBvF [00:25:00] [02mw-config] 07Universal-Omega edited pull request 03#3690: Allow DiscordNotifications and SlackNotifications on private wikis - 13https://git.io/JtBvF [00:25:23] miraheze/mw-config - Universal-Omega the build passed. [00:25:46] miraheze/mw-config - Universal-Omega the build passed. [00:28:25] [02mw-config] 07Universal-Omega closed pull request 03#3690: Allow DiscordNotifications and SlackNotifications on private wikis - 13https://git.io/JtBvF [00:28:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JtBfB [00:28:28] [02miraheze/mw-config] 07Universal-Omega 03caaac1a - Allow DiscordNotifications and SlackNotifications on private wikis (#3690) * Allow DiscordNotifications on private wikis * Update GlobalExtensions.php * Update ManageWikiSettings.php * Update ManageWikiExtensions.php [00:28:29] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [00:28:31] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [00:29:20] miraheze/mw-config - Universal-Omega the build passed. [00:29:33] RECOVERY - Host cloud5 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [00:30:22] RECOVERY - ping4 on cloud5 is OK: PING OK - Packet loss = 0%, RTA = 3.19 ms [00:30:54] RECOVERY - cloud5 SSH on cloud5 is OK: SSH OK - OpenSSH_6.7p1 OVH-rescue (protocol 2.0) [00:33:57] PROBLEM - ping4 on cloud4 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:34:19] PROBLEM - cloud4 NTP time on cloud4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:34:31] PROBLEM - cloud4 SSH on cloud4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:42] PROBLEM - cloud4 Disk Space on cloud4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:23] PROBLEM - cloud4 APT on cloud4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:35:36] PROBLEM - Host cloud4 is DOWN: PING CRITICAL - Packet loss = 100% [00:36:59] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[cloud-init] [00:39:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJm [00:39:19] [02miraheze/puppet] 07paladox 03889ef51 - cloud: Add support for changing the interface [00:39:20] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:39:22] [02puppet] 07paladox opened pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:40:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJ3 [00:40:12] [02miraheze/puppet] 07paladox 03dfa85f6 - Update interfaces.erb [00:40:14] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:40:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJG [00:40:37] [02miraheze/puppet] 07paladox 03d1fdbf0 - Update cloud4.yaml [00:40:39] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:40:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJZ [00:40:49] [02miraheze/puppet] 07paladox 0320d68f6 - Update init.pp [00:40:50] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:41:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJc [00:41:06] [02miraheze/puppet] 07paladox 033933432 - Update cloud5.yaml [00:41:07] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:41:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJW [00:41:27] [02miraheze/puppet] 07paladox 0302b6fbd - Update cloud.pp [00:41:29] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:41:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJ8 [00:41:48] [02miraheze/puppet] 07paladox 037bfe740 - Update init.pp [00:41:50] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:42:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBJ4 [00:42:03] [02miraheze/puppet] 07paladox 032676c34 - Update init.pp [00:42:05] [02puppet] 07paladox synchronize pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:42:37] [02puppet] 07paladox closed pull request 03#1609: cloud: Add support for changing the interface - 13https://git.io/JtBJY [00:42:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JtBJB [00:42:40] [02miraheze/puppet] 07paladox 034614ca2 - cloud: Add support for changing the interface (#1609) * cloud: Add support for changing the interface * Update interfaces.erb * Update cloud4.yaml * Update init.pp * Update cloud5.yaml * Update cloud.pp * Update init.pp * Update init.pp [00:42:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:42:43] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:48:48] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 75.99 ms [00:50:36] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:50:42] RECOVERY - cloud5 NTP time on cloud5 is OK: NTP OK: Offset -0.0009344816208 secs [00:51:01] PROBLEM - cloud5 Puppet on cloud5 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [00:51:19] RECOVERY - cloud5 Disk Space on cloud5 is OK: DISK OK - free space: / 3561522 MB (99% inode=99%); [00:51:21] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:51:40] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 1.41, 1.16, 0.55 [00:52:59] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 10 seconds ago with 2 failures. Failed resources (up to 3 shown): Package[cloud-init],Package[proxmox-ve] [00:53:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBUT [00:53:22] [02miraheze/puppet] 07paladox 03e1ac7c1 - Update init.pp [00:54:59] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:56:03] huh [00:56:08] oh [00:58:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBUn [00:58:33] [02miraheze/puppet] 07paladox 03c271f8d - cloud: Fix default value for role::cloud::main_interface Also convert hiera to lookup. [01:02:11] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:03:23] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [01:03:45] PROBLEM - Host cloud5 is DOWN: PING CRITICAL - Packet loss = 100% [01:07:47] [02mw-config] 07Universal-Omega closed pull request 03#3679: Add Naver site verification key for linkwikiwiki (T6764) - 13https://git.io/Jtc65 [01:07:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBUw [01:07:50] [02miraheze/mw-config] 07R4356th 037751c6c - Add Naver site verification key for linkwikiwiki (T6764) (#3679) * Add Naver site verification key for linkwikiwiki * Remove unused hook This won't break anything. * Update hook for erislywiki [01:08:56] miraheze/mw-config - Universal-Omega the build passed. [01:12:12] RECOVERY - Host cloud4 is UP: PING OK - Packet loss = 0%, RTA = 6.07 ms [01:12:13] RECOVERY - cloud4 SSH on cloud4 is OK: SSH OK - OpenSSH_6.7p1 OVH-rescue (protocol 2.0) [01:12:19] RECOVERY - ping4 on cloud4 is OK: PING OK - Packet loss = 0%, RTA = 7.75 ms [01:14:55] ohh [01:15:45] RECOVERY - Host cloud5 is UP: PING OK - Packet loss = 0%, RTA = 1.77 ms [01:15:46] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 0.60, 0.21, 0.08 [01:15:56] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBTL [01:16:10] [02miraheze/mw-config] 07Universal-Omega 034d801ca - fix meta tag functions [01:17:13] miraheze/mw-config - Universal-Omega the build passed. [01:18:14] RECOVERY - ping6 on cloud5 is OK: PING OK - Packet loss = 0%, RTA = 0.85 ms [01:18:55] cloud5s back :D [01:19:07] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [01:19:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBT4 [01:19:39] [02miraheze/puppet] 07paladox 030a8b492 - cloud4 and 5 fix interface [01:19:40] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:19:42] [02puppet] 07paladox opened pull request 03#1610: cloud4 and 5 fix interface - 13https://git.io/JtBTB [01:19:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBT0 [01:19:53] [02miraheze/puppet] 07paladox 038f6ef69 - Update cloud5.yaml [01:19:55] [02puppet] 07paladox synchronize pull request 03#1610: cloud4 and 5 fix interface - 13https://git.io/JtBTB [01:19:59] [02puppet] 07paladox closed pull request 03#1610: cloud4 and 5 fix interface - 13https://git.io/JtBTB [01:20:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtBTE [01:20:02] [02miraheze/puppet] 07paladox 032156dd5 - cloud4 and 5 fix interface (#1610) * cloud4 and 5 fix interface * Update cloud5.yaml [01:20:04] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:20:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:22:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JtBT6 [01:22:59] [02miraheze/mediawiki] 07Universal-Omega 032a9d6e8 - Install GeoGebra [01:23:01] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:23:24] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:25:40] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+2/-0/±1] 13https://git.io/JtBTP [01:25:41] [02miraheze/mediawiki] 07Universal-Omega 030a8a81f - Install ProtectionIndicator [01:25:43] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [01:26:11] [02mediawiki] 07Universal-Omega opened pull request 03#1096: Install GeoGebra - 13https://git.io/JtBTX [01:27:05] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [01:27:07] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:28:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JtBTD [01:28:23] [02miraheze/mediawiki] 07Universal-Omega 037ea3fd6 - add ignore = dirty [01:28:25] [02mediawiki] 07Universal-Omega synchronize pull request 03#1096: Install GeoGebra - 13https://git.io/JtBTX [01:29:43] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±1] 13https://git.io/JtBTH [01:29:44] [02miraheze/mediawiki] 07Universal-Omega 03d579747 - Install ProtectionIndicator [01:29:46] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [01:30:09] [02mediawiki] 07Universal-Omega opened pull request 03#1097: Install ProtectionIndicator - 13https://git.io/JtBTQ [01:30:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBT7 [01:30:52] [02miraheze/mediawiki] 07Universal-Omega 033fe3581 - add ignore = dirty [01:30:54] [02mediawiki] 07Universal-Omega synchronize pull request 03#1097: Install ProtectionIndicator - 13https://git.io/JtBTQ [01:31:06] [02mediawiki] 07Universal-Omega closed pull request 03#1097: Install ProtectionIndicator - 13https://git.io/JtBTQ [01:31:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JtBT5 [01:31:09] [02miraheze/mediawiki] 07Universal-Omega 03ceef6f0 - Install ProtectionIndicator (#1097) * Install ProtectionIndicator * add ignore = dirty [01:31:11] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbL5b [01:31:12] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:32:26] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03Universal-Omega-patch-1 [+2/-0/±2] 13https://git.io/JtBTx [01:32:28] [02miraheze/mediawiki] 07Universal-Omega 03d90e34c - Merge branch 'REL1_35' into Universal-Omega-patch-1 [01:32:29] [02mediawiki] 07Universal-Omega synchronize pull request 03#1096: Install GeoGebra - 13https://git.io/JtBTX [01:32:45] [02mediawiki] 07Universal-Omega closed pull request 03#1096: Install GeoGebra - 13https://git.io/JtBTX [01:32:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JtBke [01:32:48] [02miraheze/mediawiki] 07Universal-Omega 03e1be946 - Install GeoGebra (#1096) * Install GeoGebra * add ignore = dirty [01:32:50] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:32:51] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [01:37:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkI [01:37:25] [02miraheze/mw-config] 07Universal-Omega 03f8d5df8 - Add Extension:ProtectionIndicator [01:37:27] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:37:32] [02mw-config] 07Universal-Omega opened pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:38:27] miraheze/mw-config - Universal-Omega the build passed. [01:38:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkm [01:38:41] [02miraheze/mw-config] 07Universal-Omega 03b91b2e7 - Update LocalSettings.php [01:38:42] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:39:52] miraheze/mw-config - Universal-Omega the build passed. [01:39:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkZ [01:40:00] [02miraheze/mw-config] 07Universal-Omega 039d8e7aa - Update LocalExtensions.php [01:40:02] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:40:57] miraheze/mw-config - Universal-Omega the build passed. [01:41:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkC [01:41:09] [02miraheze/mw-config] 07Universal-Omega 038335472 - Update extension-list [01:41:11] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:41:17] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:41:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkW [01:41:39] [02miraheze/mw-config] 07Universal-Omega 03d354848 - Update LocalSettings.php [01:41:40] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:42:01] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:42:08] [02miraheze/puppet] 07Southparkfan pushed 032 commits to 03master [+0/-0/±9] 13https://git.io/JtBkB [01:42:08] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:42:10] [02miraheze/puppet] 07Southparkfan 03d28e8ca - Enable blackbox exporter [01:42:11] [02miraheze/puppet] 07Southparkfan 035da4b3b - Merge branch 'master' of github.com:miraheze/puppet [01:42:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkR [01:42:14] [02miraheze/mw-config] 07Universal-Omega 03bccbf06 - Update ManageWikiExtensions.php [01:42:16] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:42:20] miraheze/mw-config - Universal-Omega the build passed. [01:42:38] miraheze/mw-config - Universal-Omega the build passed. [01:42:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBkE [01:42:51] [02miraheze/mw-config] 07Universal-Omega 033e6a97f - Update LocalExtensions.php [01:42:52] [02mw-config] 07Universal-Omega synchronize pull request 03#3691: Add Extension:ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:43:03] [02mw-config] 07Universal-Omega edited pull request 03#3691: Add ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:43:08] miraheze/mw-config - Universal-Omega the build passed. [01:43:50] miraheze/mw-config - Universal-Omega the build passed. [01:44:41] [02mw-config] 07Universal-Omega closed pull request 03#3691: Add ProtectionIndicator (T6753) - 13https://git.io/JtBkt [01:44:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JtBk2 [01:44:44] [02miraheze/mw-config] 07Universal-Omega 0329e808c - Add ProtectionIndicator (T6753) (#3691) * Add Extension:ProtectionIndicator * Update LocalSettings.php * Update LocalExtensions.php * Update extension-list * Update LocalSettings.php * Update ManageWikiExtensions.php * Update LocalExtensions.php [01:44:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:44:48] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:46:08] miraheze/mw-config - Universal-Omega the build passed. [01:46:09] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [01:48:25] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBIY [01:48:27] [02miraheze/puppet] 07Southparkfan 03d645d19 - Fix blackbox exporter config [01:54:12] Hello orbit! If you have any questions, feel free to ask and someone should answer soon. [01:54:21] hello from windows for workgroups 3.11 [01:54:23] don't ask [01:55:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBLT [01:55:25] [02miraheze/mw-config] 07Universal-Omega 0312881ed - Add GeoGebra [01:55:26] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:55:28] [02mw-config] 07Universal-Omega opened pull request 03#3692: Add GeoGebra - 13https://git.io/JtBLk [01:55:39] [02mw-config] 07Universal-Omega edited pull request 03#3692: Add GeoGebra (T6743) - 13https://git.io/JtBLk [01:56:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBLt [01:56:20] [02miraheze/mw-config] 07Universal-Omega 03dd4c70f - Update LocalSettings.php [01:56:22] [02mw-config] 07Universal-Omega synchronize pull request 03#3692: Add GeoGebra (T6743) - 13https://git.io/JtBLk [01:56:26] PROBLEM - cloud4 Puppet on cloud4 is UNKNOWN: NRPE: Unable to read output [01:56:37] RECOVERY - cloud4 Disk Space on cloud4 is OK: DISK OK - free space: / 3561541 MB (99% inode=99%); [01:56:43] miraheze/mw-config - Universal-Omega the build passed. [01:56:45] RECOVERY - cloud4 NTP time on cloud4 is OK: NTP OK: Offset -0.001561582088 secs [01:57:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBLm [01:57:01] [02miraheze/mw-config] 07Universal-Omega 03bf875ac - Update LocalExtensions.php [01:57:03] [02mw-config] 07Universal-Omega synchronize pull request 03#3692: Add GeoGebra (T6743) - 13https://git.io/JtBLk [01:57:15] miraheze/mw-config - Universal-Omega the build passed. [01:57:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBLO [01:57:32] [02miraheze/mw-config] 07Universal-Omega 03a424de3 - Update extension-list [01:57:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 1.27, 1.14, 0.57 [01:57:34] [02mw-config] 07Universal-Omega synchronize pull request 03#3692: Add GeoGebra (T6743) - 13https://git.io/JtBLk [01:58:00] miraheze/mw-config - Universal-Omega the build passed. [01:58:05] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [01:58:30] miraheze/mw-config - Universal-Omega the build passed. [01:59:10] [02mw-config] 07Universal-Omega closed pull request 03#3692: Add GeoGebra (T6743) - 13https://git.io/JtBLk [01:59:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JtBLW [01:59:13] [02miraheze/mw-config] 07Universal-Omega 032a17f4c - Add GeoGebra (T6743) (#3692) * Add GeoGebra * Update LocalSettings.php * Update LocalExtensions.php * Update extension-list [01:59:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:59:18] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:00:05] miraheze/mw-config - Universal-Omega the build passed. [02:03:58] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:04:25] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:04:57] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:07:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBL1 [02:07:11] [02miraheze/puppet] 07paladox 03f24412f - prometheus: Fix which class to use for php-fpm [02:08:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBLS [02:08:41] [02miraheze/mw-config] 07Universal-Omega 03cdc8aca - Disable Cosmos RecentChanges module for thewhiteroomwiki [02:08:43] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:08:44] [02mw-config] 07Universal-Omega opened pull request 03#3693: Disable Cosmos RecentChanges module for thewhiteroomwiki - 13https://git.io/JtBL9 [02:09:13] [02mw-config] 07Universal-Omega synchronize pull request 03#3693: Disable Cosmos RecentChanges module for thewhiteroomwiki - 13https://git.io/JtBL9 [02:09:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JtBL7 [02:09:16] [02miraheze/mw-config] 07Universal-Omega 03f1bdb6e - add prefix [02:09:45] miraheze/mw-config - Universal-Omega the build passed. [02:10:03] [02mw-config] 07Universal-Omega closed pull request 03#3693: Disable Cosmos RecentChanges module for thewhiteroomwiki - 13https://git.io/JtBL9 [02:10:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBL5 [02:10:06] [02miraheze/mw-config] 07Universal-Omega 035240caf - Disable Cosmos RecentChanges module for thewhiteroomwiki (#3693) * Disable Cosmos RecentChanges module for thewhiteroomwiki * add prefix [02:10:08] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:10:09] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:10:15] miraheze/mw-config - Universal-Omega the build passed. [02:11:05] miraheze/mw-config - Universal-Omega the build passed. [02:15:43] [02miraheze/puppet] 07Southparkfan pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/JtBtf [02:15:45] [02miraheze/puppet] 07Southparkfan 035888720 - blackbox: monitor all cache proxies [02:15:46] [02miraheze/puppet] 07Southparkfan 033cd3f28 - Merge branch 'master' of github.com:miraheze/puppet [02:15:48] [02miraheze/puppet] 07Southparkfan 03b891864 - Add host header for blackbox exporter [02:18:17] RECOVERY - ping6 on cloud4 is OK: PING OK - Packet loss = 0%, RTA = 1.85 ms [02:36:18] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"managewiki",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%managewiki%'; [02:36:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:56] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"62":', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki'; [02:37:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:50] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitecss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitecss",%'; [02:40:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:19] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejson",%'; [02:41:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:39] UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejs",%'; [02:41:54] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editsitejs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editsitejs",%'; [02:41:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:15] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjson",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjson",%'; [02:42:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:37] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"edituserjs",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"edituserjs",%'; [02:42:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:47] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"editusercss",', '' ) WHERE perm_group = '*' AND perm_permissions LIKE '%"editusercss",%'; [02:42:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:45:22] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, '"15":"16":"17":"18":"19":"20"', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki'; [02:45:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:01] !log UPDATE mw_permissions SET perm_permissions = REPLACE( perm_permissions, ',"74":"","75":"","76":"","77":""', '' ) WHERE perm_group = '*' AND perm_dbname = 'unicodesubsetswiki'; [02:47:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:34:08] @Universal_Omega See DM on IRC when you get a chance [04:25:13] @dmehus: replied [04:45:57] Universal_Omega: How's the day been going? [07:31:04] Command sent from Discord by R4356th: [07:31:05] .tell Xana You can try asking your question in #mediawiki-visualeditor. [07:31:05] MH-Discord: I'll pass that on when Xana is around. [07:52:00] PROBLEM - news.famepedia.co.in - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for news.famepedia.co.in could not be found [13:00:53] @Universal_Omega thanks—replied to your reply [13:01:58] @Reception123 Did you remove the ssl and cert entries for "news.famepedia.co.in" when custom domain changed to "en.famepedia.org" iirc? [13:20:08] dmehus: did it in the same commit, yes [13:20:09] see https://github.com/miraheze/ssl/commit/8707e44d81bd4d2aea0b009bb442c6e9e731799b [13:20:10] [ en.famepedia.co.in -> en.famepedia.org · miraheze/ssl@8707e44 · GitHub ] - github.com [14:03:06] @Reception123 oh ok, weird that we're getting those "could not be found" errors then [14:03:25] Do we need to cancel the LetsEncrypt certificate? [14:04:03] dmehus: oh sorry, you said news.* not en.* I didn't change news, no, as that wasn't included in the request [14:14:18] @Reception123 Yeah, not sure what our process is, but since the certificate will still exist, it'll still point to landing page if the DNS is hosted remotely, so that's probably what's triggering these errors. We probably should remove it completely, unless they specifically ask for a redirect to be set up, no? [14:14:50] maybe they pointed it away but forgot about their news.* wiki [14:14:58] oh [14:15:02] yeah, good point [14:15:29] where is the news wiki pointing? I didn't realize that I said news.* actually even [14:16:43] ```id 53618 [14:16:43] opcode QUERY [14:16:43] rcode NXDOMAIN [14:16:43] flags QR RD RA [14:16:43] ;QUESTION [14:16:43] news.famepedia.co.in. IN A [14:16:43] ;ANSWER [14:16:44] ;AUTHORITY [14:16:44] famepedia.co.in. 1799 IN SOA mary.ns.cloudflare.com. dns.cloudflare.com. 2036374577 10000 2400 604800 3600 [14:16:45] ;ADDITIONAL``` [14:17:10] I'd say remove it as they've not specified a target IP in their DNS [14:19:02] https://phabricator.miraheze.org/T6712 ah [14:19:03] [ ⚓ T6712 custom domain ] - phabricator.miraheze.org [14:20:50] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JtBw4 [14:20:51] [02miraheze/ssl] 07Reception123 0306e0ea5 - remove news.famepedia.co.in cert No longer used [14:21:24] 👍 ty [14:22:03] Should we delete the cert and also remove the private key as well? [14:22:38] oh you did nevermind [14:22:46] I only saw one commit [14:30:48] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:30:59] PROBLEM - cloud5 NTP time on cloud5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:31:03] PROBLEM - ping4 on cloud5 is CRITICAL: PING CRITICAL - Packet loss = 100% [14:32:43] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:32:54] RECOVERY - cloud5 NTP time on cloud5 is OK: NTP OK: Offset -0.003732174635 secs [14:33:02] RECOVERY - ping4 on cloud5 is OK: PING OK - Packet loss = 0%, RTA = 0.68 ms [14:42:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBr7 [14:42:27] [02miraheze/puppet] 07paladox 0376880e7 - cloud: Do not use syslog_ng We need to log locally because incase of failures, it may not log to graylog. This is to be on the safe side. [14:42:29] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:42:31] [02puppet] 07paladox opened pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:42:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBrd [14:42:49] [02miraheze/puppet] 07paladox 03824bdf5 - Update cloud2.yaml [14:42:50] [02puppet] 07paladox synchronize pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:42:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBrF [14:42:59] [02miraheze/puppet] 07paladox 031d31923 - Update cloud3.yaml [14:43:01] [02puppet] 07paladox synchronize pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:43:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBrb [14:43:10] [02miraheze/puppet] 07paladox 03e2135a4 - Update cloud4.yaml [14:43:12] [02puppet] 07paladox synchronize pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:43:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBrN [14:43:21] [02miraheze/puppet] 07paladox 038b58538 - Update cloud5.yaml [14:43:22] [02puppet] 07paladox synchronize pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:43:29] [02puppet] 07paladox closed pull request 03#1611: cloud: Do not use syslog_ng - 13https://git.io/JtBr5 [14:43:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JtBrA [14:43:32] [02miraheze/puppet] 07paladox 034cb7832 - cloud: Do not use syslog_ng (#1611) * cloud: Do not use syslog_ng We need to log locally because incase of failures, it may not log to graylog. This is to be on the safe side. * Update cloud2.yaml * Update cloud3.yaml * Update cloud4.yaml * Update cloud5.yaml [14:43:34] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [14:43:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [14:59:24] PROBLEM - cloud4 Puppet on cloud4 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 14 minutes ago with 0 failures [14:59:44] PROBLEM - cloud5 Puppet on cloud5 is WARNING: WARNING: Puppet is currently disabled, message: John, last run 5 minutes ago with 0 failures [15:14:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBKK [15:14:56] [02miraheze/dns] 07paladox 03cd748c9 - Add puppet3 to dns [15:16:26] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBK1 [15:16:27] [02miraheze/dns] 07paladox 03bedf18e - Fix ipv6 address [15:17:15] RECOVERY - cloud4 Puppet on cloud4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:43:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtB6D [15:43:08] [02miraheze/puppet] 07paladox 03ab96e33 - Setup puppet3 [15:43:10] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:43:11] [02puppet] 07paladox opened pull request 03#1612: Setup puppet3 - 13https://git.io/JtB6y [15:43:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JtB6S [15:44:01] [02miraheze/puppet] 07paladox 03e4d49ff - Create puppet3.yaml [15:44:02] [02puppet] 07paladox synchronize pull request 03#1612: Setup puppet3 - 13https://git.io/JtB6y [15:44:10] [02puppet] 07paladox closed pull request 03#1612: Setup puppet3 - 13https://git.io/JtB6y [15:44:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JtB6H [15:44:13] [02miraheze/puppet] 07paladox 0368a29c2 - Setup puppet3 (#1612) * Setup puppet3 * Create puppet3.yaml [15:44:14] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:44:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [15:49:36] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): File[/var/lib/puppetdb],File[/etc/puppetdb/conf.d],Exec[pgreload],File[/etc/puppetlabs/puppetdb/jvm_prometheus_puppetdb_jmx_exporter.yaml] [15:50:04] PROBLEM - puppet3 puppetdb on puppet3 is CRITICAL: connect to address 51.195.236.216 and port 8081: Connection refused [16:19:45] RECOVERY - puppet3 puppetdb on puppet3 is OK: TCP OK - 0.001 second response time on 51.195.236.216 port 8081 [16:29:27] [02mediawiki] 07RhinosF1 commented on pull request 03#1088: Bump skins/Medik from `7a0fa00` to `3cb92c1` - 13https://git.io/JtBP8 [16:29:29] [02mediawiki] 07dependabot[bot] commented on pull request 03#1088: Bump skins/Medik from `7a0fa00` to `3cb92c1` - 13https://git.io/JtBPB [16:30:41] I was hoping that would make it check early [16:31:09] Universal_Omega: can you bump to latest version if you're not mobile? [16:39:31] PROBLEM - puppet3 puppetdb on puppet3 is CRITICAL: connect to address 51.195.236.216 and port 8081: Connection refused [16:41:11] RhinosF1: I am mobile right now. [16:41:18] Universal_Omega: ok [16:41:27] Reception123: ^ [16:42:58] sure [16:43:09] hopefully this time won't be like the last thing I bumped to the latest version :) [16:43:28] RECOVERY - puppet3 puppetdb on puppet3 is OK: TCP OK - 0.001 second response time on 51.195.236.216 port 8081 [16:43:36] Reception123: it shouldn't be looking at it [16:43:43] at least this is just a skin, @Reception123 :P [16:44:01] RhinosF1|NotHere: https://github.com/miraheze/mediawiki/pull/1088#issuecomment-770408706 oh so you want me to manually update it then? [16:44:02] [ Bump skins/Medik from `7a0fa00` to `3cb92c1` by dependabot · Pull Request #1088 · miraheze/mediawiki · GitHub ] - github.com [16:44:06] dmehus: it's somehow scaring the new CI [16:44:11] Reception123: yes [16:44:18] ok [16:44:28] @RhinosF1|NotHere who is somehow here, ack [16:44:53] will do in a few [16:45:06] dmehus: I know the moment I change my nick back something will happen so ye [16:45:44] @RhinosF1|NotHere That's why I think you should lurk as a totally different alias [16:45:58] dmehus: probably [16:46:28] like RhinostoneCowboy [16:46:31] get it? [16:46:47] No [16:47:15] Cultural references normally go over my head [16:47:22] well it's like RhinestoneCowboy except Rhine sounds like Rhinos so it's a play on words [16:47:23] PROBLEM - puppet3 puppetdb on puppet3 is CRITICAL: connect to address 51.195.236.216 and port 8081: Connection refused [16:47:40] but a very obscure/bad play on words [16:48:34] Yeah that would never make sense to me [16:48:51] yeah, heh [16:48:55] I'll try and do better [16:49:20] RECOVERY - puppet3 puppetdb on puppet3 is OK: TCP OK - 0.001 second response time on 51.195.236.216 port 8081 [17:02:15] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtB1O [17:02:16] [02miraheze/mediawiki] 07Reception123 037242717 - Update Medik skin [17:02:41] ^ RhinosF1|NotHere there seemed to only have been a .json i18n file change, not sure why you thought that had to be updated? [17:03:11] Reception123: to try and fix a weird issue with git the new ci was giving [17:03:29] oh [17:10:42] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:11:25] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:11:54] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:13:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtB19 [17:13:34] [02miraheze/mw-config] 07Universal-Omega 035ee06d9 - Fix tabbing [17:14:35] miraheze/mw-config - Universal-Omega the build passed. [17:15:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtB1Q [17:15:27] [02miraheze/mw-config] 07Universal-Omega 0310358a3 - Fix tabbing [17:16:18] Reception123: it worked [17:16:24] great! [17:16:25] CI now installs the repos [17:16:30] miraheze/mw-config - Universal-Omega the build passed. [17:16:35] Now we just need to make it run install.php [17:34:40] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:34:49] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 133.15 ms [17:35:25] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:35:38] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:03:14] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 117.92 ms [18:11:35] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:12:55] PROBLEM - gluster2 Puppet on gluster2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:13:33] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:14:30] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:40:58] RECOVERY - gluster2 Puppet on gluster2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:01] !log root@test2:/srv/mediawiki/w/extensions/MirahezeMagic/maintenance# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/fixDumpsPermissions.php [18:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:44:17] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:44:30] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:47:03] hello! We have noticed that __DISAMBIG__ no longer prevets the article to being counted as a normal article. Is there a configuration for that? [18:47:42] maybe we knew about it in the past, but we have forgot [18:49:06] @Jakeukalane Possibly? But I will say that's probably by design, since disambiguation pages are navigational aids, not unlike category pages and redirects [18:51:14] ok, thanks, so maybe is not configurable by ourselves right? [18:51:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtB9F [18:51:38] [02miraheze/mw-config] 07paladox 03773943e - Fix [18:52:43] miraheze/mw-config - paladox the build passed. [18:56:52] changing topic. Is there any future plan to include IPFS in the infrastructure of miraheze? [19:01:15] Jakeukalane: If you're talking about the file sharing system, I'm not sure how that would benefit anyone. Is there a specific feature of it you'd like? [19:02:31] no, I was just thinking about it, maybe could reduce costs of servers, etc. [19:02:59] paladox: any thoughts? [19:03:02] but I don't have much idea. Although I know about IPFS before the brave publicity stunt [19:03:18] i have no thoughts on a thing i don't know about :) [19:03:22] But given our file server is just a single VM, my guess is no. [19:03:37] ok, :) [19:03:43] paladox: fair enough, I found out from a quick Wikipedia search. It's like but torrent from what I can tell. [19:29:12] !log upgrade ca-certificates puppet-agent puppetserver tzdata on puppet2 [19:29:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:38] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CRITICAL - load average: 11.89, 8.26, 5.81 [19:34:05] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [19:34:06] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:34:37] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Puppet has 12 failures. Last run 3 minutes ago with 12 failures. Failed resources (up to 3 shown) [19:35:19] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 10.30, 8.00, 5.74 [19:35:38] RECOVERY - mw5 Current Load on mw5 is OK: OK - load average: 7.58, 8.21, 6.12 [19:36:32] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.43, 1.72, 1.03 [19:37:20] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 5.24, 6.82, 5.59 [19:38:33] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.65, 1.31, 0.96 [19:48:34] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [19:51:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtB53 [19:51:22] [02miraheze/puppet] 07paladox 036ae62aa - Set puppet3 to use its own puppetserver [19:51:24] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:51:25] [02puppet] 07paladox opened pull request 03#1613: Set puppet3 to use its own puppetserver - 13https://git.io/JtB5s [19:52:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtB5G [19:52:08] [02miraheze/puppet] 07paladox 03a3d5036 - Update puppet3.yaml [19:52:09] [02puppet] 07paladox synchronize pull request 03#1613: Set puppet3 to use its own puppetserver - 13https://git.io/JtB5s [19:52:15] [02puppet] 07paladox closed pull request 03#1613: Set puppet3 to use its own puppetserver - 13https://git.io/JtB5s [19:52:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtB5Z [19:52:18] [02miraheze/puppet] 07paladox 03c2ad337 - Set puppet3 to use its own puppetserver (#1613) * Set puppet3 to use its own puppetserver * Update puppet3.yaml [19:52:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:52:21] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:56:36] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:58:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtB5g [19:58:15] [02miraheze/puppet] 07paladox 0301f6ca8 - cp12: Use puppet3 [20:02:03] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:03] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:02:34] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:34] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki eswiki /home/reception/delbackups3/eskwiki.xml [20:16:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:04] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBd8 [20:17:05] [02miraheze/dns] 07paladox 039722ccc - Add gluster[34] to dns [20:21:38] !log icingaui: convert 'Administrator' access to sre-infrastructure and sre-management (total access to Infra + EMs above) [20:21:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:54] !log icingaui: add 'guest' group with only member being guest account, limit password changes on it [20:22:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:45] !log renamed jacksucksatlifewiki to jackmasseywelshwiki [20:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:55] !log renamed projectchaoswiki to cnvnwiki [20:29:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:51:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtBbs [20:51:43] [02miraheze/puppet] 07paladox 031085839 - gluster3: Set to use puppet3 [20:52:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtBbG [20:52:04] [02miraheze/puppet] 07paladox 03eb58d71 - gluster4: Set to use puppet3 [20:53:57] * hispano76 greet [21:04:37] PROBLEM - cloud5 Puppet on cloud5 is CRITICAL: connect to address 51.77.117.189 port 5666: Connection refusedconnect to host 51.77.117.189 port 5666: Connection refused [21:04:52] PROBLEM - cloud5 Disk Space on cloud5 is CRITICAL: connect to address 51.77.117.189 port 5666: Connection refusedconnect to host 51.77.117.189 port 5666: Connection refused [21:04:52] PROBLEM - cloud5 APT on cloud5 is CRITICAL: connect to address 51.77.117.189 port 5666: Connection refusedconnect to host 51.77.117.189 port 5666: Connection refused [21:05:17] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: connect to address 51.77.117.189 port 5666: Connection refusedconnect to host 51.77.117.189 port 5666: Connection refused [21:05:43] PROBLEM - cloud5 NTP time on cloud5 is CRITICAL: connect to address 51.77.117.189 port 5666: Connection refusedconnect to host 51.77.117.189 port 5666: Connection refused [21:08:15] PROBLEM - cloud5 SSH on cloud5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:48] RECOVERY - cloud5 Puppet on cloud5 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [21:09:09] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:09:09] RECOVERY - cloud5 Disk Space on cloud5 is OK: DISK OK - free space: / 3559894 MB (99% inode=99%); [21:09:35] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 1.26, 0.54, 0.20 [21:09:57] RECOVERY - cloud5 NTP time on cloud5 is OK: NTP OK: Offset -0.07088908553 secs [21:10:12] RECOVERY - cloud5 SSH on cloud5 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [21:37:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBxI [21:37:51] [02miraheze/puppet] 07paladox 0379dfe42 - Setup gluster on gluster[34] [21:37:52] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:37:54] [02puppet] 07paladox opened pull request 03#1614: Setup gluster on gluster[34] - 13https://git.io/JtBxL [21:38:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtBxm [21:38:25] [02miraheze/puppet] 07paladox 0377bcdf1 - Update gluster3.yaml [21:38:26] [02puppet] 07paladox synchronize pull request 03#1614: Setup gluster on gluster[34] - 13https://git.io/JtBxL [21:38:33] [02puppet] 07paladox closed pull request 03#1614: Setup gluster on gluster[34] - 13https://git.io/JtBxL [21:38:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtBxO [21:38:36] [02miraheze/puppet] 07paladox 03603c93c - Setup gluster on gluster[34] (#1614) * Setup gluster on gluster[34] * Update gluster3.yaml [21:38:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [21:38:39] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [21:45:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBxP [21:45:03] [02miraheze/puppet] 07paladox 03d13f10e - gluster: Add gluster3 to firewall [21:49:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBx5 [21:49:38] [02miraheze/puppet] 07paladox 0397720c3 - gluster: Add gluster4 to firewall [22:07:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBp2 [22:07:56] [02miraheze/puppet] 07paladox 0371713e6 - mediawiki: Connect to gluster3 using /mnt/mediawiki-static-new [22:08:46] [02mw-config] 07paladox synchronize pull request 03#3686: Support migration to new volume - 13https://git.io/Jt81B [22:08:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JtBpK [22:08:49] [02miraheze/mw-config] 07paladox 03cc837ac - Update LocalWiki.php [22:09:13] [02mw-config] 07paladox closed pull request 03#3686: Support migration to new volume - 13https://git.io/Jt81B [22:09:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBp6 [22:09:16] [02miraheze/mw-config] 07paladox 035aeb316 - Support migration to new volume (#3686) * Support migration to new volume * Update LocalWiki.php [22:09:17] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vbvb3 [22:09:19] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [22:09:48] miraheze/mw-config - paladox the build passed. [22:10:21] miraheze/mw-config - paladox the build passed. [22:10:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBpS [22:10:29] [02miraheze/puppet] 07paladox 03c877b64 - gluster: Setup /mnt/mediawiki-static-new to transfer data [22:11:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBpd [22:11:52] [02miraheze/puppet] 07paladox 0357a5aea - Fix [22:12:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBpb [22:12:36] [02miraheze/mw-config] 07Universal-Omega 0366a685a - add dbbackup to CreateWiki blacklist [22:12:42] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:13:33] miraheze/mw-config - Universal-Omega the build passed. [22:14:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBhv [22:14:39] [02miraheze/puppet] 07paladox 0381a7f4e - Fix [22:16:44] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:21:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBhs [22:21:50] [02miraheze/mw-config] 07Universal-Omega 034ef73a2 - fix tabbing [22:22:40] !log installed rsync on gluster3 [22:22:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:52] miraheze/mw-config - Universal-Omega the build passed. [22:22:53] !log starting rsync for every directory starting with a [22:22:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:51] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBhz [22:25:52] [02miraheze/mw-config] 07Universal-Omega 0316242d9 - fix tabbing [22:27:54] miraheze/mw-config - Universal-Omega the build passed. [22:28:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBhK [22:28:07] [02miraheze/mw-config] 07Universal-Omega 0313a8288 - fix tabbing [22:29:09] miraheze/mw-config - Universal-Omega the build passed. [22:36:10] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtBjf [22:36:11] [02miraheze/dns] 07paladox 03a337fbd - Update ns2 ip [22:37:44] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+2/-0/±6] 13https://git.io/JtBjt [22:37:45] [02miraheze/puppet] 07JohnFLewis 03e6e9b61 - restructure icinga group roles [22:37:47] [02miraheze/puppet] 07JohnFLewis 0339678a6 - Merge branch 'master' of github.com:/miraheze/puppet [22:49:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtReU [22:49:36] [02miraheze/puppet] 07paladox 0328ce2f3 - Set ns2 to use puppet3 [22:49:53] John can probably tell me if I need to change the glue records for dns [22:50:52] JohnLewis ^ [22:52:09] SPF|Cloud: if we define them with the registrar, it's a good idea :) [22:57:09] ns3 is set to 128.199.196.94 [22:57:44] that server must have been decommissioned about four years ago [22:58:14] nah thats ns1 ip [22:58:27] oh [22:59:39] ns3 was set to ns1's IP because people were still using ns3 from 2016 [23:00:05] yes, but the glue record was still set to an IP we haven't used in years [23:00:14] whereas ns3 in our zone file points to ns1 [23:00:20] oh [23:00:27] the glue for ns3 can go [23:00:50] ERROR: The nameserver is in use. [23:01:14] and hund.io tells me ns2 is down [23:01:15] err? [23:01:18] hm? [23:02:55] glue records ns1-3 are now equal to the zone records for ns1-3 [23:03:04] https://status.miraheze.wiki/components/5b1ec78f87d844329ab6478c says it was only down for 15mins [23:03:05] [ ns2 - Status Page - Miraheze ] - status.miraheze.wiki [23:03:38] and you did not shutdown the older ns2? [23:04:02] i haven't yet, i was migrating it to the newer one [23:04:12] i can do that now [23:04:41] !log shutdown ns2 (old) on cloud1 (it was migrated to cloud4) [23:04:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:05:17] I don't know what happened at hund [23:06:07] I'm quite proud of https://status.miraheze.wiki/components/5b191c7f87d844329ab643c2/history, by the way [23:06:09] [ History - Miraheze Login - Status Page - Miraheze ] - status.miraheze.wiki [23:06:39] that's about 99.995% uptime ;) [23:07:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:12] PROBLEM - ns2 Disk Space on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:51] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:59] is this the old ns2? [23:09:08] PROBLEM - Host ns2 is DOWN: PING CRITICAL - Packet loss = 100% [23:10:49] yeh [23:10:58] the new server uses puppet3 [23:11:07] but monitoring uses the old puppet [23:11:43] I must say the network latency from db13 to mw4 is dubious, ranging from 0.166 - 16.573 ms for 34 packets [23:12:40] 5/34 packets are 1+ ms [23:15:37] pinging cloud1 from cloud3 doesn't show the problem, for 37 packets they're all between 0.105 - 0.213 ms [23:16:42] PROBLEM - jobrunner2 Current Load on jobrunner2 is CRITICAL: CRITICAL - load average: 6.78, 4.84, 3.18 [23:17:10] Uh oh my precious jobrunners arent happy :( [23:18:24] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtRvR [23:18:26] [02miraheze/dns] 07paladox 036a57530 - Add mon2 to dns [23:19:48] PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: CRITICAL - load average: 6.72, 4.83, 3.26 [23:21:07] ffmpeg process on jobrunner is using lots of cpu [23:21:57] converting an 113M webm file [23:22:20] Oof [23:22:56] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 5.56, 5.61, 4.14 [23:25:11] paladox: minor suggestion, how about to reduce the number of DNS changes+updates, you go through all the IPs, allocate them now and do one big DNS update? :) [23:25:23] oh, sure! [23:25:45] and then migrating is easier because them you know the IPs already, just need to create the VMs :) [23:26:23] with needing to waiting for DNS updates ;) since we already have, a few blackholing entries isn't a problem for the purpose of a migration [23:27:06] PROBLEM - jobrunner2 Current Load on jobrunner2 is CRITICAL: CRITICAL - load average: 6.13, 5.86, 4.59 [23:29:03] PROBLEM - jobrunner2 Current Load on jobrunner2 is WARNING: WARNING - load average: 5.91, 5.90, 4.76 [23:32:43] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtRvh [23:32:45] [02miraheze/dns] 07paladox 03e34f749 - Add new service names [23:32:46] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl [23:32:48] [02dns] 07paladox opened pull request 03#184: Add new service names - 13https://git.io/JtRvj [23:33:27] paladox: cp13/cp14? [23:33:41] SPF|Cloud replacements for cp6/7 [23:33:51] the plan says cp10/cp11 [23:33:59] oh [23:36:04] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtRfm [23:36:06] [02miraheze/dns] 07paladox 0329fb2db - Update miraheze.org [23:36:07] [02dns] 07paladox synchronize pull request 03#184: Add new service names - 13https://git.io/JtRvj [23:38:19] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtRfZ [23:38:20] [02miraheze/dns] 07paladox 037972634 - Update miraheze.org [23:38:22] [02dns] 07paladox synchronize pull request 03#184: Add new service names - 13https://git.io/JtRvj [23:39:00] phab1 twice? [23:41:07] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtRfR [23:41:09] [02miraheze/dns] 07paladox 039e16f3c - Update miraheze.org [23:41:10] [02dns] 07paladox synchronize pull request 03#184: Add new service names - 13https://git.io/JtRvj [23:47:38] RECOVERY - jobrunner2 Current Load on jobrunner2 is OK: OK - load average: 4.32, 4.95, 5.02 [23:48:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JtRfd [23:48:46] [02miraheze/puppet] 07paladox 032cd4f41 - mon2: Set to use puppet3 [23:51:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtRJk [23:51:05] [02miraheze/puppet] 07JohnFLewis 035c51980 - grafana: spit admin into two sre ldap groups [23:51:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtRJI [23:51:16] [02miraheze/puppet] 07paladox 0318c65f4 - Setup mon2 [23:51:17] !log delete mediawiki-admins LDAP group [23:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:59:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtRJE [23:59:29] [02miraheze/puppet] 07paladox 0353f9fb6 - Fix