[00:03:10] PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 23.47, 16.88, 13.31 [00:03:44] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLpQo [00:03:46] [02miraheze/landing] 07Universal-Omega 038d53ee1 - remove extra new line [00:04:42] miraheze/landing - Universal-Omega the build passed. [00:05:07] RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.89, 18.48, 14.37 [00:08:39] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:09:34] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:10:12] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:07:03] [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLp5w [01:07:04] [02miraheze/YouTube] 07Universal-Omega 037156c16 - drop flash support [01:07:57] miraheze/YouTube - Universal-Omega the build passed. [01:08:42] [02mediawiki] 07Universal-Omega commented on pull request 03#954: Bump extensions/YouTube from `b059390` to `ec8c966` - 13https://git.io/JLp56 [01:08:44] [02mediawiki] 07dependabot[bot] edited pull request 03#954: Bump extensions/YouTube from `b059390` to `ec8c966` - 13https://git.io/JLA9m [01:11:05] [02mediawiki] 07dependabot[bot] edited pull request 03#954: Bump extensions/YouTube from `b059390` to `ec8c966` - 13https://git.io/JLA9m [01:11:11] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/YouTube-7156c16 [+0/-0/±1] 13https://git.io/JLp5X [01:11:12] [02miraheze/mediawiki] 07dependabot[bot] 03efb84d2 - Bump extensions/YouTube from `b059390` to `7156c16` Bumps [extensions/YouTube](https://github.com/miraheze/YouTube) from `b059390` to `7156c16`. - [Release notes](https://github.com/miraheze/YouTube/releases) - [Commits](https://github.com/miraheze/YouTube/compare/b059390ba6dad65347ccd9755c333005f8e4f5f1...7156c1645d44449333fc49027f5afd67aeaeb82d) Signed- [01:11:12] off-by: dependabot[bot] [01:11:13] [ GitHub - miraheze/YouTube: A Miraheze-patched version of the YouTube extension. ] - github.com [01:11:13] [ Releases · miraheze/YouTube · GitHub ] - github.com [01:11:14] [02mediawiki] 07dependabot[bot] opened pull request 03#973: Bump extensions/YouTube from `b059390` to `7156c16` - 13https://git.io/JLp51 [01:11:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/YouTube-7156c16 - 13https://git.io/vbL5b [01:11:15] [ Comparing b059390ba6dad65347ccd9755c333005f8e4f5f1...7156c1645d44449333fc49027f5afd67aeaeb82d · miraheze/YouTube · GitHub ] - github.com [01:11:17] [02mediawiki] 07dependabot[bot] labeled pull request 03#973: Bump extensions/YouTube from `b059390` to `7156c16` - 13https://git.io/JLp51 [01:11:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#973: Bump extensions/YouTube from `b059390` to `7156c16` - 13https://git.io/JLp51 [01:11:20] [02mediawiki] 07dependabot[bot] commented on pull request 03#954: Bump extensions/YouTube from `b059390` to `ec8c966` - 13https://git.io/JLp5M [01:11:21] [02mediawiki] 07dependabot[bot] closed pull request 03#954: Bump extensions/YouTube from `b059390` to `ec8c966` - 13https://git.io/JLA9m [01:11:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/YouTube-ec8c966 - 13https://git.io/vbL5b [01:11:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/YouTube-ec8c966 [02:10:31] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JLpFg [02:10:33] [02miraheze/mediawiki] 07Universal-Omega 03d6f58ab - Install Commentbox [02:10:34] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [02:11:11] [02mediawiki] 07Universal-Omega opened pull request 03#974: Install Commentbox - 13https://git.io/JLpFr [02:12:09] [02mediawiki] 07Universal-Omega closed pull request 03#974: Install Commentbox - 13https://git.io/JLpFr [02:12:12] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:12:13] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [02:18:51] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JLpbT [02:18:52] [02miraheze/mediawiki] 07Universal-Omega 033c89cf0 - Install Commentbox [02:18:54] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [02:19:24] [02mediawiki] 07Universal-Omega opened pull request 03#975: Install Commentbox - 13https://git.io/JLpbt [02:19:33] [02mediawiki] 07Universal-Omega closed pull request 03#975: Install Commentbox - 13https://git.io/JLpbt [02:19:35] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+2/-0/±2] 13https://git.io/JLpbm [02:19:36] [02miraheze/mediawiki] 07Universal-Omega 0314aaf65 - Merge pull request #975 from miraheze/Universal-Omega-patch-1 Install Commentbox [02:19:38] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [02:19:39] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [02:22:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLpbG [02:22:30] [02miraheze/mw-config] 07Universal-Omega 0366e894f - Add Commentbox extension (T6607) [02:22:32] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [02:22:34] [02mw-config] 07Universal-Omega opened pull request 03#3650: Add Commentbox extension (T6607) - 13https://git.io/JLpbZ [02:23:30] miraheze/mw-config - Universal-Omega the build passed. [02:23:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLpbc [02:23:36] [02miraheze/mw-config] 07Universal-Omega 0362f575a - add Commentbox extension [02:23:37] [02mw-config] 07Universal-Omega synchronize pull request 03#3650: Add Commentbox extension (T6607) - 13https://git.io/JLpbZ [02:24:32] miraheze/mw-config - Universal-Omega the build passed. [02:24:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLpbC [02:24:47] [02miraheze/mw-config] 07Universal-Omega 0309899cf - add commentbox ext [02:24:48] [02mw-config] 07Universal-Omega synchronize pull request 03#3650: Add Commentbox extension (T6607) - 13https://git.io/JLpbZ [02:25:40] miraheze/mw-config - Universal-Omega the build passed. [02:26:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLpbB [02:26:48] [02miraheze/mw-config] 07Universal-Omega 03f2f1974 - add commentbox extension [02:26:50] [02mw-config] 07Universal-Omega synchronize pull request 03#3650: Add Commentbox extension (T6607) - 13https://git.io/JLpbZ [02:27:45] miraheze/mw-config - Universal-Omega the build passed. [02:46:13] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:59:49] [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL [02:59:51] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3 [03:08:12] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [05:35:56] [02mw-config] 07Universal-Omega closed pull request 03#3650: Add Commentbox extension (T6607) - 13https://git.io/JLpbZ [05:35:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JLphr [05:35:59] [02miraheze/mw-config] 07Universal-Omega 030e28692 - Add Commentbox extension (T6607) (#3650) * Add Commentbox extension (T6607) * add Commentbox extension * add commentbox ext * add commentbox extension [05:36:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [05:36:02] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [05:36:55] miraheze/mw-config - Universal-Omega the build passed. [05:56:51] [02mediawiki] 07Universal-Omega closed pull request 03#973: Bump extensions/YouTube from `b059390` to `7156c16` - 13https://git.io/JLp51 [05:56:53] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLpjs [05:56:55] [02miraheze/mediawiki] 07Universal-Omega 035cd1622 - Merge pull request #973 from miraheze/dependabot/submodules/REL1_35/extensions/YouTube-7156c16 Bump extensions/YouTube from `b059390` to `7156c16` [05:56:56] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/YouTube-7156c16 - 13https://git.io/vbL5b [05:56:58] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/YouTube-7156c16 [06:41:44] !log sudo -u www-data php /srv/mediawiki/w/maint*/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on mw* and jbr* [06:41:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:42:39] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and jbr* [06:42:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:59:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhvb [07:59:13] [02miraheze/mw-config] 07Universal-Omega 037f545e0 - Remove conflict for hawelcome [07:59:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhvA [07:59:58] [02miraheze/mw-config] 07Universal-Omega 038ed793f - remove wiki override for csydeswiki [08:00:05] miraheze/mw-config - Universal-Omega the build passed. [08:00:52] miraheze/mw-config - Universal-Omega the build passed. [08:01:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhvx [08:01:10] [02miraheze/mw-config] 07Universal-Omega 036ffe704 - add per-wiki settings comment [08:02:12] miraheze/mw-config - Universal-Omega the build passed. [08:03:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhvj [08:03:10] [02miraheze/mw-config] 07Universal-Omega 03512d498 - re-order [08:03:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfe [08:03:43] [02miraheze/mw-config] 07Universal-Omega 033d9d723 - re-order [08:04:07] miraheze/mw-config - Universal-Omega the build passed. [08:04:32] miraheze/mw-config - Universal-Omega the build passed. [08:06:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfT [08:06:32] [02miraheze/mw-config] 07Universal-Omega 03f941a60 - Move Miraheze Commons code to LocalWiki.php [08:07:20] miraheze/mw-config - Universal-Omega the build passed. [08:07:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfI [08:07:32] [02miraheze/mw-config] 07Universal-Omega 03e663197 - add Miraheze Commons code [08:08:27] miraheze/mw-config - Universal-Omega the build passed. [08:08:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfq [08:09:00] [02miraheze/mw-config] 07Universal-Omega 03bb95774 - re-order [08:09:50] miraheze/mw-config - Universal-Omega the build passed. [08:09:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfm [08:09:55] [02miraheze/mw-config] 07Universal-Omega 034775dc0 - fix spacing [08:10:47] miraheze/mw-config - Universal-Omega the build passed. [08:20:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfW [08:20:56] [02miraheze/mw-config] 07Universal-Omega 037a826af - Enable wgNativeImageLazyLoading for idolish7wiki (T6702) [08:21:46] miraheze/mw-config - Universal-Omega the build passed. [08:38:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfo [08:38:34] [02miraheze/mw-config] 07Universal-Omega 03dc6842e - remove duplicative wgAllowExternalImagesFrom [08:39:27] miraheze/mw-config - Universal-Omega the build passed. [08:45:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhfX [08:45:54] [02miraheze/mw-config] 07Universal-Omega 0316f3eb5 - add wgAllowExternalImagesFrom back for nonbinarywiki [08:46:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhf1 [08:46:35] [02miraheze/mw-config] 07Universal-Omega 03660e489 - fix typo in database name [08:46:47] miraheze/mw-config - Universal-Omega the build passed. [08:47:30] miraheze/mw-config - Universal-Omega the build passed. [09:45:45] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhUo [09:45:47] [02miraheze/landing] 07Universal-Omega 03e979f10 - update translate url [09:46:23] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhUK [09:46:25] [02miraheze/landing] 07Universal-Omega 03486217c - update translate url [09:46:55] miraheze/landing - Universal-Omega the build passed. [09:47:20] miraheze/landing - Universal-Omega the build passed. [09:48:22] [02mw-config] 07R4356th commented on pull request 03#3648: Add Translate extension-related custom configuration for snapwikiwiki - 13https://git.io/JLhUi [09:50:54] [02mw-config] 07Universal-Omega commented on pull request 03#3648: Add Translate extension-related custom configuration for snapwikiwiki - 13https://git.io/JLhU1 [11:10:05] PROBLEM - cp9 Current Load on cp9 is WARNING: WARNING - load average: 1.69, 1.78, 1.06 [11:12:04] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 1.04, 1.45, 1.02 [11:54:33] !log purchase dbbackup1.miraheze.org; 4GB MKVM in Atlanta [11:54:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:44] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JLhqD [12:23:45] [02miraheze/puppet] 07Southparkfan 034354fde - Add automation to MariaDB firewall rules [12:36:04] [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhml [12:36:05] [02miraheze/dns] 07Southparkfan 034b0dc94 - Add dbbackup1 No IPv6 (yet), not enabled on the server [12:43:25] PROBLEM - dbbackup1 Puppet on dbbackup1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [12:43:49] PROBLEM - ping6 on dbbackup1 is CRITICAL: CRITICAL - Destination Unreachable (fe80::f816:3eff:fe06:814f) [12:45:22] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [12:48:42] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhmj [12:48:44] [02miraheze/puppet] 07Southparkfan 032036a14 - Assign role::dbreplication to dbbackup1 [12:53:14] PROBLEM - dbbackup1 Puppet on dbbackup1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/private],Package[mariadb-server-10.2] [12:54:47] ack [12:57:46] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-5/±25] 13https://git.io/JLhYW [12:57:47] [02miraheze/puppet] 07Southparkfan 03932da4c - Fix default MariaDB servers and remove obsolete hiera variables [13:13:08] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JLhYX [13:13:10] [02miraheze/puppet] 07Southparkfan 0317e17e9 - Adjust InnoDB buffer pool and table caches for dbbackup1 [13:14:42] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:19:09] PROBLEM - dbbackup1 MariaDB on dbbackup1 is CRITICAL: Host '51.89.160.138' is not allowed to connect to this MariaDB server [13:20:04] PROBLEM - dbbackup1 Check MariaDB Replication on dbbackup1 is UNKNOWN: NRPE: Unable to read output [13:27:28] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/JLhOO [13:27:29] [02miraheze/puppet] 07Southparkfan 03a96bcbc - Split MariaDB grants [13:29:02] RECOVERY - dbbackup1 MariaDB on dbbackup1 is OK: Uptime: 1550 Threads: 8 Questions: 475 Slow queries: 29 Opens: 172 Flush tables: 1 Open tables: 30 Queries per second avg: 0.306 [13:30:05] PROBLEM - dbbackup1 Check MariaDB Replication on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [13:33:45] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 104.59 ms [13:37:46] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 99.05 ms [13:43:56] PROBLEM - db13 Puppet on db13 is WARNING: WARNING: Puppet is currently disabled, message: SPF, last run 21 minutes ago with 0 failures [14:24:52] !log set 4755 manually on /usr/lib/mysql/plugin/auth_pam_tool_dir/auth_pam_tool [14:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:27:57] !log dbbackup1: mysql_install_db --user=mysql --datadir=/srv/mariadb.c3 [14:28:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:16:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhcU [15:16:38] [02miraheze/mw-config] 07Universal-Omega 036ec3dfe - Remove unneeded $wgAllowExternalImagesFrom value [15:17:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhct [15:17:30] [02miraheze/mw-config] 07Universal-Omega 035a4b3d4 - remove another unneeded wiki from wgAllowExternalImagesFrom [15:17:37] miraheze/mw-config - Universal-Omega the build passed. [15:18:25] miraheze/mw-config - Universal-Omega the build passed. [15:18:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhcs [15:18:53] [02miraheze/mw-config] 07Universal-Omega 03b5a3c86 - Remove final wiki from wgAllowExternalImagesFrom since it's unneeded [15:19:50] miraheze/mw-config - Universal-Omega the build passed. [15:20:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhcW [15:20:03] [02miraheze/mw-config] 07Universal-Omega 03a574dac - remove full variable for wgAllowExternalImagesFrom Since $wgAllowExternalImages overwrites that anyways [15:20:55] miraheze/mw-config - Universal-Omega the build passed. [15:31:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhcD [15:31:32] [02miraheze/mw-config] 07Universal-Omega 037e05e8d - remove non-existent variable [15:32:25] miraheze/mw-config - Universal-Omega the build passed. [15:34:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhcH [15:34:48] [02miraheze/mw-config] 07Universal-Omega 0343fdf38 - remove configs for a deleted wiki [15:35:45] miraheze/mw-config - Universal-Omega the build passed. [16:02:28] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 101.40 ms [16:04:28] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 99.28 ms [16:08:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhCp [16:08:21] [02miraheze/mw-config] 07Universal-Omega 0381e67b4 - fix some configs/cleanup [16:09:20] miraheze/mw-config - Universal-Omega the build passed. [16:22:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhW1 [16:22:12] [02miraheze/mw-config] 07Universal-Omega 037c69ef6 - remove deleted wikis' configs [16:23:09] miraheze/mw-config - Universal-Omega the build passed. [16:38:26] [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL [16:38:28] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1 [16:43:08] [02mediawiki] 07paladox closed pull request 03#956: Bump extensions/WikiDiscover from `b46f4a3` to `fc0bd11` - 13https://git.io/JLxhd [16:43:09] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh86 [16:43:11] [02miraheze/mediawiki] 07paladox 03207923d - Merge pull request #956 from miraheze/dependabot/submodules/REL1_35/extensions/WikiDiscover-fc0bd11 Bump extensions/WikiDiscover from `b46f4a3` to `fc0bd11` [16:43:12] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/WikiDiscover-fc0bd11 - 13https://git.io/vbL5b [16:43:14] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/WikiDiscover-fc0bd11 [16:44:50] !log depool and repool mw[4567] and rebuild lc [16:44:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:25] [02mediawiki] 07paladox closed pull request 03#970: Bump extensions/DataDump from `f6c3727` to `f9dc4f6` - 13https://git.io/JLxjW [16:45:26] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh85 [16:45:28] [02miraheze/mediawiki] 07paladox 034d33a68 - Merge pull request #970 from miraheze/dependabot/submodules/REL1_35/extensions/DataDump-f9dc4f6 Bump extensions/DataDump from `f6c3727` to `f9dc4f6` [16:45:29] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/DataDump-f9dc4f6 [16:45:31] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/DataDump-f9dc4f6 - 13https://git.io/vbL5b [16:45:37] [02mediawiki] 07paladox closed pull request 03#969: Bump extensions/RottenLinks from `febf121` to `874ab34` - 13https://git.io/JLxjc [16:45:39] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh8d [16:45:40] [02miraheze/mediawiki] 07paladox 036684d00 - Merge pull request #969 from miraheze/dependabot/submodules/REL1_35/extensions/RottenLinks-874ab34 Bump extensions/RottenLinks from `febf121` to `874ab34` [16:45:42] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/RottenLinks-874ab34 - 13https://git.io/vbL5b [16:45:43] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/RottenLinks-874ab34 [16:45:48] [02mediawiki] 07paladox closed pull request 03#968: Bump extensions/ManageWiki from `8ec622b` to `2114794` - 13https://git.io/JLxjZ [16:45:50] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh8F [16:45:51] [02miraheze/mediawiki] 07paladox 0302602dd - Merge pull request #968 from miraheze/dependabot/submodules/REL1_35/extensions/ManageWiki-2114794 Bump extensions/ManageWiki from `8ec622b` to `2114794` [16:45:53] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/ManageWiki-2114794 [16:45:54] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/ManageWiki-2114794 - 13https://git.io/vbL5b [16:45:59] [02mediawiki] 07paladox closed pull request 03#961: Bump extensions/IncidentReporting from `a493b01` to `0b1e26c` - 13https://git.io/JLxjf [16:46:00] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh8b [16:46:02] [02miraheze/mediawiki] 07paladox 03fb7821c - Merge pull request #961 from miraheze/dependabot/submodules/REL1_35/extensions/IncidentReporting-0b1e26c Bump extensions/IncidentReporting from `a493b01` to `0b1e26c` [16:46:03] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/IncidentReporting-0b1e26c [16:46:05] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/IncidentReporting-0b1e26c - 13https://git.io/vbL5b [16:46:12] [02mediawiki] 07paladox closed pull request 03#960: Bump extensions/MirahezeMagic from `2572dbf` to `6be775c` - 13https://git.io/JLxje [16:46:13] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh8N [16:46:15] [02miraheze/mediawiki] 07paladox 03d1edc47 - Merge pull request #960 from miraheze/dependabot/submodules/REL1_35/extensions/MirahezeMagic-6be775c Bump extensions/MirahezeMagic from `2572dbf` to `6be775c` [16:46:16] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/MirahezeMagic-6be775c - 13https://git.io/vbL5b [16:46:18] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/MirahezeMagic-6be775c [16:46:22] [02mediawiki] 07paladox closed pull request 03#959: Bump extensions/MatomoAnalytics from `de9a445` to `b99f7d7` - 13https://git.io/JLxhh [16:46:24] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh8x [16:46:25] [02miraheze/mediawiki] 07paladox 0350198de - Merge pull request #959 from miraheze/dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b99f7d7 Bump extensions/MatomoAnalytics from `de9a445` to `b99f7d7` [16:46:27] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b99f7d7 - 13https://git.io/vbL5b [16:46:28] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/MatomoAnalytics-b99f7d7 [16:46:35] PROBLEM - test2 MediaWiki Rendering on test2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.122 second response time [16:46:52] PROBLEM - mw6 Puppet on mw6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [16:46:55] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw4 [16:46:55] PROBLEM - mw5 Puppet on mw5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [16:47:15] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+2/-0/±2] 13https://git.io/JLh4s [16:47:15] PROBLEM - mw4 Puppet on mw4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [16:47:16] [02miraheze/puppet] 07Southparkfan 03b3dbcdf - Base configuration for multi-instance MariaDB setup [16:47:33] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw4 [16:47:59] PROBLEM - mw7 Puppet on mw7 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [16:48:16] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw4 [16:48:24] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw4 [16:48:38] !log rebuild lc on test2 and jobrunner[12] [16:48:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:48] what's wrong with mw4? [16:49:28] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.078 second response time [16:50:25] SPF|Cloud i'm rebuilding lc, it's caused by https://github.com/miraheze/mediawiki/commit/207923d90d3f0307539559f5ee93c0cbf719d570 [16:50:26] [ Merge pull request #956 from miraheze/dependabot/submodules/REL1_35/e… · miraheze/mediawiki@207923d · GitHub ] - github.com [16:50:30] requires rebuilding of lc [16:50:35] hence why i'm depooling/repooling [16:51:05] you have depooled the mediawiki servers manually? [16:51:18] yeh [16:51:18] in varnish [16:51:21] yeh [16:51:24] ah okay :") [16:51:27] :)* [16:51:27] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 9.51, 7.18, 4.99 [16:51:29] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 20532 bytes in 0.270 second response time [16:51:42] btw, we have dbbackup1.miraheze.org now [16:52:14] awesome [16:52:29] RECOVERY - test2 MediaWiki Rendering on test2 is OK: HTTP OK: HTTP/1.1 200 OK - 20569 bytes in 0.392 second response time [16:53:28] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CRITICAL - load average: 11.75, 8.57, 5.75 [16:54:05] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.08, 7.69, 5.41 [16:56:05] PROBLEM - mw7 Current Load on mw7 is CRITICAL: CRITICAL - load average: 11.61, 8.91, 6.13 [16:58:05] PROBLEM - mw7 Current Load on mw7 is WARNING: WARNING - load average: 9.60, 9.20, 6.58 [16:59:27] PROBLEM - mw6 Current Load on mw6 is WARNING: WARNING - load average: 9.76, 9.79, 7.26 [17:01:28] RECOVERY - mw6 Current Load on mw6 is OK: OK - load average: 5.26, 8.11, 6.95 [17:02:05] RECOVERY - mw7 Current Load on mw7 is OK: OK - load average: 6.06, 7.88, 6.70 [17:03:06] PROBLEM - mw5 MediaWiki Rendering on mw5 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.093 second response time [17:03:52] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JLhBU [17:03:53] [02miraheze/puppet] 07Southparkfan 038912ec0 - Fix order of stanzas in role::dbreplication + fix wrong data type in mariadb::instance [17:04:39] I needed a few hours to get multi-instance mariadb working... [17:04:54] but it works now :) [17:05:06] RECOVERY - mw5 MediaWiki Rendering on mw5 is OK: HTTP OK: HTTP/1.1 200 OK - 20532 bytes in 0.198 second response time [17:12:15] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [17:12:25] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:14:55] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhBc [17:14:56] [02miraheze/mediawiki] 07Universal-Omega 03c82208a - Updating WikiForum [17:14:58] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [17:15:28] [02mediawiki] 07Universal-Omega opened pull request 03#976: Updating WikiForum - 13https://git.io/JLhBC [17:16:16] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [17:16:29] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [17:17:15] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:18:52] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:18:55] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:24:15] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [17:24:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:24:53] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhBV [17:24:54] [02miraheze/WikiDiscover] 07Universal-Omega 0358af65a - use DB_REPLICA instead of DB_MASTER [17:24:55] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [17:24:56] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [17:24:57] [02WikiDiscover] 07Universal-Omega opened pull request 03#33: use DB_REPLICA instead of DB_MASTER - 13https://git.io/JLhBw [17:25:24] !log rpeool mw6 [17:25:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:25:32] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [17:25:54] miraheze/WikiDiscover - Universal-Omega the build passed. [17:26:00] Universal_Omega: dbw -> dbr [17:26:18] SPF|Cloud: oh, right. Thanks [17:27:50] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhBi [17:27:51] [02miraheze/WikiDiscover] 07Universal-Omega 0338e60a0 - $dbw -> $dbr [17:27:53] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#33: use DB_REPLICA instead of DB_MASTER - 13https://git.io/JLhBw [17:28:54] miraheze/WikiDiscover - Universal-Omega the build passed. [17:30:05] [02WikiDiscover] 07Universal-Omega closed pull request 03#33: use DB_REPLICA instead of DB_MASTER - 13https://git.io/JLhBw [17:30:06] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhBX [17:30:08] [02miraheze/WikiDiscover] 07Universal-Omega 03c71c196 - use DB_REPLICA instead of DB_MASTER (#33) * use DB_REPLICA instead of DB_MASTER * $dbw -> $dbr [17:30:09] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:30:11] [02WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [17:31:02] miraheze/WikiDiscover - Universal-Omega the build passed. [17:33:30] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 110.53 ms [17:43:28] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JLhRk [17:43:29] [02miraheze/puppet] 07Southparkfan 035d89e5d - Fix option suffix for multi-instance, wrong ownership of debian.cnf and update documentation [17:44:22] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhRI [17:44:23] [02miraheze/puppet] 07Southparkfan 0360cbd90 - Use dynamic pid file in mariadb::instance [17:46:03] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhRq [17:46:04] [02miraheze/puppet] 07Southparkfan 03e9cd6c1 - fiel -> file [17:47:32] brb [17:56:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhRl [17:56:14] [02miraheze/mw-config] 07Universal-Omega 03f4e374c - temporarily restrict WikiForum [17:57:10] miraheze/mw-config - Universal-Omega the build passed. [17:58:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhRR [17:58:30] [02miraheze/mw-config] 07Universal-Omega 03484f016 - fix typo [17:59:27] miraheze/mw-config - Universal-Omega the build passed. [18:05:35] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 99.26 ms [18:06:51] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/add-wft_closed_timestamp-to-wikiforum_threads.sql [18:06:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:19] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /home/universalomega/migrateOldWikiForumClosedTimestampColumnToNew.php [18:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:06] [02mediawiki] 07Universal-Omega closed pull request 03#976: Updating WikiForum - 13https://git.io/JLhBC [18:16:08] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLh0f [18:16:09] [02miraheze/mediawiki] 07Universal-Omega 031aa1917 - Merge pull request #976 from miraheze/Universal-Omega-patch-1 Updating WikiForum [18:16:12] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [18:16:13] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [18:22:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLh0R [18:22:13] [02miraheze/mw-config] 07Universal-Omega 032553359 - un-restrict WikiForum [18:23:05] miraheze/mw-config - Universal-Omega the build passed. [18:30:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLh0b [18:30:50] [02miraheze/mw-config] 07Universal-Omega 0333ae56c - remove .travis.yml [18:31:42] miraheze/mw-config - Universal-Omega the build passed. [18:37:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhEk [18:37:33] [02miraheze/mw-config] 07Universal-Omega 03a7cc4c4 - use github actions badge instead of travisci [18:38:23] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:38:30] miraheze/mw-config - Universal-Omega the build passed. [18:38:34] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:38:56] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 130.43 ms [18:41:16] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [18:43:57] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [18:44:15] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw7 [18:44:36] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw7 [18:44:56] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw7 [18:45:05] PROBLEM - mw7 MediaWiki Rendering on mw7 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.099 second response time [18:45:33] PROBLEM - cp9 Varnish Backends on cp9 is CRITICAL: 1 backends are down. mw7 [18:46:50] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw7 [18:46:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:05] RECOVERY - mw7 MediaWiki Rendering on mw7 is OK: HTTP OK: HTTP/1.1 200 OK - 20532 bytes in 0.323 second response time [18:49:53] Universal_Omega i'm running that [18:50:19] paladox: Oh, I was getting errors and checked graylog and that was why. [18:54:16] !log running mariabackup on db12 & db13 to dump all databases to dbbackup1 [18:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:41] paladox: if the backup is impacting production severely, you can stop the backup [18:55:48] ok [18:56:05] increased cpu usage is normal though [18:56:05] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CRITICAL - load average: 11.41, 8.86, 6.27 [18:57:32] RECOVERY - cp9 Varnish Backends on cp9 is OK: All 7 backends are healthy [18:58:05] PROBLEM - mw4 Current Load on mw4 is WARNING: WARNING - load average: 8.28, 8.70, 6.53 [18:58:19] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [18:58:33] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:58:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [18:58:55] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [19:00:05] RECOVERY - mw4 Current Load on mw4 is OK: OK - load average: 4.07, 7.08, 6.21 [19:05:15] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:26:51] !log reception@jobrunner1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki opencircuitswiki /home/reception/opencircuitscom-20200218-history.xml [19:26:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:23] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03revert-976-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhzv [19:29:24] [02miraheze/mediawiki] 07Universal-Omega 03674dfd4 - Revert "Updating WikiForum" [19:29:26] [02mediawiki] 07Universal-Omega created branch 03revert-976-Universal-Omega-patch-1 - 13https://git.io/vbL5b [19:29:27] [02mediawiki] 07Universal-Omega opened pull request 03#977: Revert "Updating WikiForum" - 13https://git.io/JLhzf [19:29:35] [02mediawiki] 07Universal-Omega closed pull request 03#977: Revert "Updating WikiForum" - 13https://git.io/JLhzf [19:29:37] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLhzJ [19:29:38] [02miraheze/mediawiki] 07Universal-Omega 03cb86fe6 - Merge pull request #977 from miraheze/revert-976-Universal-Omega-patch-1 Revert "Updating WikiForum" [19:29:40] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-976-Universal-Omega-patch-1 [19:29:41] [02mediawiki] 07Universal-Omega deleted branch 03revert-976-Universal-Omega-patch-1 - 13https://git.io/vbL5b [19:31:31] !log root@dbbackup1:/home/paladox# chmod 1777 /tmp [19:31:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:31:43] fyi SPF|Cloud ^ (since apt was failing) [20:06:49] !log restart db13 dump (crashed due to: xtrabackup_copy_logfile() failed: redo log block is overwritten, please increase redo log size with innodb_log_file_size parameter.) [20:06:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:47] afk now [20:08:35] Something odd with manage wiki permissions logs, see testwiki recent changes [20:08:36] https://cdn.discordapp.com/attachments/435711390544560128/797557452769591306/unknown.png [20:11:56] @Universal Omega ^^ [20:13:52] Might be related to the WikiForum upgrade which caused threads to be automatically closed. That did get reverted nevertheless. [20:14:04] It's not related. [20:14:17] I may know why though one second. [20:56:48] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhar [20:56:50] [02miraheze/ManageWiki] 07Universal-Omega 03b03f56c - fix logging when user group is deleted [20:56:52] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [20:56:53] [02ManageWiki] 07Universal-Omega opened pull request 03#225: fix logging when user group is deleted - 13https://git.io/JLhao [20:58:00] miraheze/ManageWiki - Universal-Omega the build passed. [20:58:04] @R4356th, dmehus, and Voidwalker: https://github.com/miraheze/ManageWiki/pull/225/ should fix that logging issue [20:58:05] [ fix logging when user group is deleted by Universal-Omega · Pull Request #225 · miraheze/ManageWiki · GitHub ] - github.com [20:58:54] Great! [21:05:22] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhad [21:05:23] [02miraheze/ManageWiki] 07Universal-Omega 0330c1e1f - fix logging when namespaces are deleted [21:05:25] [02ManageWiki] 07Universal-Omega synchronize pull request 03#225: fix logging when user group is deleted - 13https://git.io/JLhao [21:05:38] [02ManageWiki] 07Universal-Omega edited pull request 03#225: fix logging when user groups or namespaces are deleted - 13https://git.io/JLhao [21:06:24] miraheze/ManageWiki - Universal-Omega the build passed. [21:10:03] [02ManageWiki] 07Universal-Omega closed pull request 03#225: fix logging when user groups or namespaces are deleted - 13https://git.io/JLhao [21:10:05] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JLhVL [21:10:06] [02miraheze/ManageWiki] 07Universal-Omega 03f16370f - fix logging when user groups or namespaces are deleted (#225) * fix logging when user group is deleted * fix logging when namespaces are deleted [21:10:08] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [21:10:09] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:10:56] PROBLEM - cp9 Current Load on cp9 is CRITICAL: CRITICAL - load average: 1.29, 2.19, 1.28 [21:11:00] miraheze/ManageWiki - Universal-Omega the build passed. [21:12:56] RECOVERY - cp9 Current Load on cp9 is OK: OK - load average: 0.33, 1.53, 1.14 [21:15:35] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±2] 13https://git.io/JLhVW [21:15:36] [02miraheze/mediawiki] 07Universal-Omega 034056e9b - Updating ManageWiki & WikiForum [21:15:38] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:16:51] [02mediawiki] 07Universal-Omega opened pull request 03#978: Updating ManageWiki & WikiForum - 13https://git.io/JLhV4 [21:17:13] [02mediawiki] 07Universal-Omega closed pull request 03#978: Updating ManageWiki & WikiForum - 13https://git.io/JLhV4 [21:17:14] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±4] 13https://git.io/JLhVB [21:17:16] [02miraheze/mediawiki] 07Universal-Omega 035049ceb - Merge pull request #978 from miraheze/Universal-Omega-patch-1 Updating ManageWiki & WikiForum [21:17:17] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:17:19] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:40:57] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:41:17] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:49:32] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhwM [21:49:34] [02miraheze/ManageWiki] 07Universal-Omega 0375fa02e - fix logging when deleting namespaces [21:50:30] miraheze/ManageWiki - Universal-Omega the build passed. [21:51:36] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhwd [21:51:37] [02miraheze/mediawiki] 07Universal-Omega 032907595 - Updating ManageWiki [21:51:39] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:52:12] [02mediawiki] 07Universal-Omega opened pull request 03#979: Updating ManageWiki - 13https://git.io/JLhwF [21:52:41] [02mediawiki] 07Universal-Omega closed pull request 03#979: Updating ManageWiki - 13https://git.io/JLhwF [21:52:43] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/JLhwb [21:52:44] [02miraheze/mediawiki] 07Universal-Omega 0384e4763 - Merge pull request #979 from miraheze/Universal-Omega-patch-1 Updating ManageWiki [21:52:46] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [21:52:47] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:55:49] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 99.99 ms [22:06:56] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:09:16] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:16:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JLhry [22:16:40] [02miraheze/mw-config] 07Universal-Omega 03bda201b - re-add wgUrlShortenerDomainsWhitelist [22:17:44] miraheze/mw-config - Universal-Omega the build passed. [22:19:08] paladox: https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=dbbackup1.miraheze.org&var-port=9100 do you see a mixture of dbbackup metrics and the metrics of an alleged cache proxy? [22:19:09] [ Grafana ] - grafana.miraheze.org [22:19:26] yeh [22:19:33] i restarted prometheus earlier to fix that [22:19:35] what the heck? [22:19:59] also why is the dump process so awful slow [22:20:06] Did you add the dbbackup1 to dns before doing puppet? [22:20:38] around the same tiem [22:20:39] time [22:21:24] but it's possible I haven't waited long enough for it th deploy, therefore mon1 had the wrong ip address for dbbackup1.miraheze.org... [22:21:52] yeh [22:21:58] well [22:22:15] i think when it looked it up (prometheus) it could only find the cps [22:22:51] if you use a non-existent subdomain, it points to the cps [22:22:59] wildcard = fallbacl [22:23:15] fallback* and that's normal.. [22:26:06] I presume the heavy write load on the database servers makes mariabackup slow too [22:26:24] since it does a point-in-time backup [22:26:30] anyways, afk again [22:46:55] !log sudo -u www-data /usr/local/bin/foreachwikiindblist /home/universalomega/wikiforum.json /srv/mediawiki/w/maintenance/sql.php /home/universalomega/drop-wft_closed-from-wikiforum_threads.sql [22:47:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:49] * hispano76 saluda [23:17:38] PROBLEM - ping4 on dbbackup1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 107.11 ms [23:23:38] RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 95.96 ms [23:39:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±2] 13https://git.io/JLhi1 [23:39:06] [02miraheze/mediawiki] 07Universal-Omega 032847d95 - Change UrlShortener to use master branch [23:39:07] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [23:42:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JLhiQ [23:42:49] [02miraheze/mediawiki] 07Universal-Omega 030e7e69c - Update .gitmodules [23:43:17] [02mediawiki] 07Universal-Omega opened pull request 03#980: Change UrlShortener to use master branch - 13https://git.io/JLhiF [23:55:10] [02mediawiki] 07Universal-Omega closed pull request 03#980: Change UrlShortener to use master branch - 13https://git.io/JLhiF [23:55:14] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [23:55:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1