[00:26:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJtdN [00:26:03] [02miraheze/puppet] 07paladox 030da9a1b - Switch db7 to replica mode [00:26:10] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.51, 2.05, 1.54 [00:28:46] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.98, 1.53, 1.41 [00:30:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJtFL [00:30:07] [02miraheze/puppet] 07paladox 03718d1a1 - Add db7 to db firewall [00:34:34] PROBLEM - db7 Check MariaDB Replication on db7 is UNKNOWN: NRPE: Unable to read output [01:05:33] RECOVERY - bacula2 Bacula Databases db13 on bacula2 is OK: OK: Full, 22577 files, 63.50GB, 2020-07-08 01:03:00 (2.5 minutes ago) [01:23:50] !log apt-get upgrade - db7 (mariadb/kernel and some other packages) [01:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:59] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave IO State not correct, slave stopped or replication broken! [01:29:09] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [01:29:22] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 14.32, 20.82, 15.44 [01:29:29] RECOVERY - db7 MySQL on db7 is OK: Uptime: 96 Threads: 12 Questions: 16372 Slow queries: 29 Opens: 5955 Flush tables: 1 Open tables: 5813 Queries per second avg: 170.541 [01:32:06] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 27.52, 24.38, 17.70 [01:37:16] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:46:49] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [01:51:26] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 11.41, 21.65, 23.18 [01:56:44] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.40, 15.56, 20.07 [02:29:36] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.92, 1.86, 1.55 [02:34:48] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.77, 1.42, 1.49 [05:10:00] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.12, 1.97, 1.49 [05:12:39] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.42, 1.33, 1.32 [06:03:28] PROBLEM - wiki.earthmcbe.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.earthmcbe.space' expires in 15 day(s) (Fri 24 Jul 2020 05:59:23 GMT +0000). [06:04:29] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqUY [06:04:31] [02miraheze/ssl] 07MirahezeSSLBot 031e8d2d2 - Bot: Update SSL cert for wiki.earthmcbe.space [06:09:57] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JJqUl [06:09:58] [02miraheze/ssl] 07Reception123 03006ab0a - add www.museummiddelland.nl cert [06:13:55] RECOVERY - wiki.earthmcbe.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.earthmcbe.space' will expire on Tue 06 Oct 2020 05:04:22 GMT +0000. [06:25:44] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:26:00] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:26:34] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:26:49] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:26:57] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:27:00] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:27:23] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:27:43] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:29:14] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:29:48] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[www.museummiddelland.nl_private] [06:45:13] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqkT [06:45:15] [02miraheze/ssl] 07Reception123 03e78cc53 - add www redirect for museuminddelland.nl [06:46:27] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[postgresql-contrib-11] [06:52:58] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:35:49] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/pbr-5.4.5 [+0/-0/±1] 13https://git.io/JJqmQ [08:35:51] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03cd1fa49 - Bump pbr from 4.2.0 to 5.4.5 Bumps [pbr](https://docs.openstack.org/pbr/latest/) from 4.2.0 to 5.4.5. Signed-off-by: dependabot-preview[bot] [08:35:51] [ OpenStack Docs: pbr - Python Build Reasonableness ] - docs.openstack.org [08:35:52] [02bots-pip-auto] 07dependabot-preview[bot] created branch 03dependabot/pip/master/pbr-5.4.5 - 13https://git.io/JJLum [08:35:54] [02bots-pip-auto] 07dependabot-preview[bot] opened pull request 03#20: Bump pbr from 4.2.0 to 5.4.5 - 13https://git.io/JJqm7 [08:35:55] [02bots-pip-auto] 07dependabot-preview[bot] labeled pull request 03#20: Bump pbr from 4.2.0 to 5.4.5 - 13https://git.io/JJqm7 [08:36:14] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqmd [08:36:15] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03229317c - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [08:36:15] [08:36:16] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [08:36:17] [ Releases · praw-dev/praw · GitHub ] - github.com [08:36:17] [02bots-pip-auto] 07dependabot-preview[bot] created branch 03dependabot/pip/master/praw-7.1.0 - 13https://git.io/JJLum [08:36:18] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [08:36:18] [02bots-pip-auto] 07dependabot-preview[bot] opened pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:36:20] [02bots-pip-auto] 07dependabot-preview[bot] labeled pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:36:28] [ Comparing v5.4.0...v7.1.0 · praw-dev/praw · GitHub ] - github.com [08:36:35] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/keyrings-alt-3.4.0 [+0/-0/±1] 13https://git.io/JJqmb [08:36:36] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03a15dcb1 - Bump keyrings-alt from 3.1.1 to 3.4.0 Bumps [keyrings-alt](https://github.com/jaraco/keyrings.alt) from 3.1.1 to 3.4.0. - [Release notes](https://github.com/jaraco/keyrings.alt/releases) - [Changelog](https://github.com/jaraco/keyrings.alt/blob/master/CHANGES.rst) - [Commits](https://github.com/jaraco/keyrings.alt/compare/3.1.1...v3.4.0) Signed- [08:36:37] off-by: dependabot-preview[bot] [08:36:37] [ GitHub - jaraco/keyrings.alt ] - github.com [08:36:38] [ Releases · jaraco/keyrings.alt · GitHub ] - github.com [08:36:38] [02bots-pip-auto] 07dependabot-preview[bot] created branch 03dependabot/pip/master/keyrings-alt-3.4.0 - 13https://git.io/JJLum [08:36:38] [ keyrings.alt/CHANGES.rst at master · jaraco/keyrings.alt · GitHub ] - github.com [08:36:39] [02bots-pip-auto] 07dependabot-preview[bot] opened pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:36:40] [ Comparing 3.1.1...v3.4.0 · jaraco/keyrings.alt · GitHub ] - github.com [08:36:41] [02bots-pip-auto] 07dependabot-preview[bot] labeled pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:36:52] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/pycurl-7.43.0.5 [+0/-0/±1] 13https://git.io/JJqmA [08:36:53] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 034cb2b5b - Bump pycurl from 7.43.0.2 to 7.43.0.5 Bumps [pycurl](https://github.com/pycurl/pycurl) from 7.43.0.2 to 7.43.0.5. - [Release notes](https://github.com/pycurl/pycurl/releases) - [Changelog](https://github.com/pycurl/pycurl/blob/master/ChangeLog) - [Commits](https://github.com/pycurl/pycurl/commits) Signed-off-by: dependabot-preview[bot] [08:36:53] [08:36:54] [ GitHub - pycurl/pycurl: PycURL - Python interface to libcurl ] - github.com [08:36:54] [ Releases · pycurl/pycurl · GitHub ] - github.com [08:36:55] [02bots-pip-auto] 07dependabot-preview[bot] created branch 03dependabot/pip/master/pycurl-7.43.0.5 - 13https://git.io/JJLum [08:36:55] [ pycurl/ChangeLog at master · pycurl/pycurl · GitHub ] - github.com [08:36:56] [ Commits · pycurl/pycurl · GitHub ] - github.com [08:36:56] [02bots-pip-auto] 07dependabot-preview[bot] opened pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:36:58] [02bots-pip-auto] 07dependabot-preview[bot] labeled pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:37:34] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/sopel-modules-youtube-0.3.1 [+0/-0/±1] 13https://git.io/JJqmp [08:37:35] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0373423a0 - Bump sopel-modules-youtube from 0.3.0 to 0.3.1 Bumps [sopel-modules-youtube](https://github.com/sopel-irc/sopel-youtube) from 0.3.0 to 0.3.1. - [Release notes](https://github.com/sopel-irc/sopel-youtube/releases) - [Changelog](https://github.com/sopel-irc/sopel-youtube/blob/master/NEWS) - [Commits](https://github.com/sopel-irc/sopel- [08:37:35] youtube/compare/0.3.0...0.3.1) Signed-off-by: dependabot-preview[bot] [08:37:36] [ GitHub - sopel-irc/sopel-youtube: YouTube plugin for Sopel ] - github.com [08:37:37] [ Releases · sopel-irc/sopel-youtube · GitHub ] - github.com [08:37:37] [02bots-pip-auto] 07dependabot-preview[bot] created branch 03dependabot/pip/master/sopel-modules-youtube-0.3.1 - 13https://git.io/JJLum [08:37:37] [ sopel-youtube/NEWS at master · sopel-irc/sopel-youtube · GitHub ] - github.com [08:37:38] [ Page not found · GitHub · GitHub ] - github.com [08:37:38] [02bots-pip-auto] 07dependabot-preview[bot] opened pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:37:40] [02bots-pip-auto] 07dependabot-preview[bot] labeled pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:38:30] [02bots-pip-auto] 07dependabot-preview[bot] closed pull request 03#20: Bump pbr from 4.2.0 to 5.4.5 - 13https://git.io/JJqm7 [08:38:31] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJqYf [08:38:33] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03cd1fa49 - Bump pbr from 4.2.0 to 5.4.5 Bumps [pbr](https://docs.openstack.org/pbr/latest/) from 4.2.0 to 5.4.5. Signed-off-by: dependabot-preview[bot] [08:38:33] [ OpenStack Docs: pbr - Python Build Reasonableness ] - docs.openstack.org [08:38:35] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03a22467e - Merge pull request #20 from miraheze/dependabot/pip/master/pbr-5.4.5 [08:38:36] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/pbr-5.4.5 [08:38:38] [02bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/pbr-5.4.5 - 13https://git.io/JJLum [08:38:41] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:38:43] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:38:44] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:38:46] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:39:36] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:39:38] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:39:39] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:39:41] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:39:42] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/sopel-modules-youtube-0.3.1 [+0/-0/±1] 13https://git.io/JJqYL [08:39:44] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 032952ef7 - Bump sopel-modules-youtube from 0.3.0 to 0.3.1 Bumps [sopel-modules-youtube](https://github.com/sopel-irc/sopel-youtube) from 0.3.0 to 0.3.1. - [Release notes](https://github.com/sopel-irc/sopel-youtube/releases) - [Changelog](https://github.com/sopel-irc/sopel-youtube/blob/master/NEWS) - [Commits](https://github.com/sopel-irc/sopel- [08:39:44] youtube/compare/0.3.0...0.3.1) Signed-off-by: dependabot-preview[bot] [08:39:45] [ GitHub - sopel-irc/sopel-youtube: YouTube plugin for Sopel ] - github.com [08:39:45] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/keyrings-alt-3.4.0 [+0/-0/±1] 13https://git.io/JJqYt [08:39:45] [ Releases · sopel-irc/sopel-youtube · GitHub ] - github.com [08:39:46] [ sopel-youtube/NEWS at master · sopel-irc/sopel-youtube · GitHub ] - github.com [08:39:47] [ Page not found · GitHub · GitHub ] - github.com [08:39:47] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 036052332 - Bump keyrings-alt from 3.1.1 to 3.4.0 Bumps [keyrings-alt](https://github.com/jaraco/keyrings.alt) from 3.1.1 to 3.4.0. - [Release notes](https://github.com/jaraco/keyrings.alt/releases) - [Changelog](https://github.com/jaraco/keyrings.alt/blob/master/CHANGES.rst) - [Commits](https://github.com/jaraco/keyrings.alt/compare/3.1.1...v3.4.0) Signed- [08:39:47] off-by: dependabot-preview[bot] [08:39:48] [ GitHub - jaraco/keyrings.alt ] - github.com [08:39:48] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:39:48] [ Releases · jaraco/keyrings.alt · GitHub ] - github.com [08:39:50] [ keyrings.alt/CHANGES.rst at master · jaraco/keyrings.alt · GitHub ] - github.com [08:39:50] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:39:51] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqYq [08:39:51] [ Comparing 3.1.1...v3.4.0 · jaraco/keyrings.alt · GitHub ] - github.com [08:39:53] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 031bd9d90 - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [08:39:53] [08:39:54] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [08:39:54] [ Releases · praw-dev/praw · GitHub ] - github.com [08:39:54] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/pycurl-7.43.0.5 [+0/-0/±1] 13https://git.io/JJqYm [08:39:55] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [08:39:56] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0326fd17c - Bump pycurl from 7.43.0.2 to 7.43.0.5 Bumps [pycurl](https://github.com/pycurl/pycurl) from 7.43.0.2 to 7.43.0.5. - [Release notes](https://github.com/pycurl/pycurl/releases) - [Changelog](https://github.com/pycurl/pycurl/blob/master/ChangeLog) - [Commits](https://github.com/pycurl/pycurl/commits) Signed-off-by: dependabot-preview[bot] [08:39:56] [08:39:57] [ GitHub - pycurl/pycurl: PycURL - Python interface to libcurl ] - github.com [08:39:57] [ Releases · pycurl/pycurl · GitHub ] - github.com [08:39:57] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:39:58] [ pycurl/ChangeLog at master · pycurl/pycurl · GitHub ] - github.com [08:39:59] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:39:59] [ Commits · pycurl/pycurl · GitHub ] - github.com [08:40:06] [ Unicorn! · GitHub ] - github.com [08:47:02] [02bots-pip-auto] 07dependabot-preview[bot] closed pull request 03#24: Bump sopel-modules-youtube from 0.3.0 to 0.3.1 - 13https://git.io/JJqmh [08:47:03] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJqY7 [08:47:05] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 038cebc69 - Merge pull request #24 from miraheze/dependabot/pip/master/sopel-modules-youtube-0.3.1 [08:47:06] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/sopel-modules-youtube-0.3.1 [08:47:08] [02bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/sopel-modules-youtube-0.3.1 - 13https://git.io/JJLum [08:47:16] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:47:18] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:47:19] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:48:10] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:48:11] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:48:13] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:48:14] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/keyrings-alt-3.4.0 [+0/-0/±1] 13https://git.io/JJqYF [08:48:16] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0316a4e3f - Bump keyrings-alt from 3.1.1 to 3.4.0 Bumps [keyrings-alt](https://github.com/jaraco/keyrings.alt) from 3.1.1 to 3.4.0. - [Release notes](https://github.com/jaraco/keyrings.alt/releases) - [Changelog](https://github.com/jaraco/keyrings.alt/blob/master/CHANGES.rst) - [Commits](https://github.com/jaraco/keyrings.alt/compare/3.1.1...v3.4.0) Signed- [08:48:16] off-by: dependabot-preview[bot] [08:48:16] [ GitHub - jaraco/keyrings.alt ] - github.com [08:48:17] [ Releases · jaraco/keyrings.alt · GitHub ] - github.com [08:48:17] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/pycurl-7.43.0.5 [+0/-0/±1] 13https://git.io/JJqYb [08:48:17] [ keyrings.alt/CHANGES.rst at master · jaraco/keyrings.alt · GitHub ] - github.com [08:48:19] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 037374df0 - Bump pycurl from 7.43.0.2 to 7.43.0.5 Bumps [pycurl](https://github.com/pycurl/pycurl) from 7.43.0.2 to 7.43.0.5. - [Release notes](https://github.com/pycurl/pycurl/releases) - [Changelog](https://github.com/pycurl/pycurl/blob/master/ChangeLog) - [Commits](https://github.com/pycurl/pycurl/commits) Signed-off-by: dependabot-preview[bot] [08:48:19] [08:48:19] [ Comparing 3.1.1...v3.4.0 · jaraco/keyrings.alt · GitHub ] - github.com [08:48:20] [ GitHub - pycurl/pycurl: PycURL - Python interface to libcurl ] - github.com [08:48:20] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:48:21] [ Releases · pycurl/pycurl · GitHub ] - github.com [08:48:22] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:48:22] [ pycurl/ChangeLog at master · pycurl/pycurl · GitHub ] - github.com [08:48:23] [ Commits · pycurl/pycurl · GitHub ] - github.com [08:48:23] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqYN [08:48:25] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0306b79c8 - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [08:48:25] [08:48:25] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [08:48:26] [ Releases · praw-dev/praw · GitHub ] - github.com [08:48:26] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:48:27] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [08:48:37] [ Unicorn! · GitHub ] - github.com [08:55:32] [02bots-pip-auto] 07dependabot-preview[bot] closed pull request 03#22: Bump keyrings-alt from 3.1.1 to 3.4.0 - 13https://git.io/JJqmN [08:55:34] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJqOW [08:55:35] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03b60bba1 - Merge pull request #22 from miraheze/dependabot/pip/master/keyrings-alt-3.4.0 [08:55:37] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/keyrings-alt-3.4.0 [08:55:38] [02bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/keyrings-alt-3.4.0 - 13https://git.io/JJLum [08:55:49] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:55:50] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:56:30] [02bots-pip-auto] 07dependabot-preview[bot] commented on pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqOR [08:56:43] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:56:44] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:56:46] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqOE [08:56:47] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03e0326ce - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [08:56:47] [08:56:48] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [08:56:49] [ Releases · praw-dev/praw · GitHub ] - github.com [08:56:49] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [08:56:49] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [08:56:50] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/pycurl-7.43.0.5 [+0/-0/±1] 13https://git.io/JJqOu [08:56:52] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0377af132 - Bump pycurl from 7.43.0.2 to 7.43.0.5 Bumps [pycurl](https://github.com/pycurl/pycurl) from 7.43.0.2 to 7.43.0.5. - [Release notes](https://github.com/pycurl/pycurl/releases) - [Changelog](https://github.com/pycurl/pycurl/blob/master/ChangeLog) - [Commits](https://github.com/pycurl/pycurl/commits) Signed-off-by: dependabot-preview[bot] [08:56:52] [08:56:53] [ GitHub - pycurl/pycurl: PycURL - Python interface to libcurl ] - github.com [08:56:53] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [08:56:53] [ Releases · pycurl/pycurl · GitHub ] - github.com [08:56:54] [ pycurl/ChangeLog at master · pycurl/pycurl · GitHub ] - github.com [08:56:55] [ Commits · pycurl/pycurl · GitHub ] - github.com [08:56:58] [ Comparing v5.4.0...v7.1.0 · praw-dev/praw · GitHub ] - github.com [09:02:03] [02bots-pip-auto] 07dependabot-preview[bot] closed pull request 03#23: Bump pycurl from 7.43.0.2 to 7.43.0.5 - 13https://git.io/JJqmx [09:02:04] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JJqOd [09:02:06] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 0377af132 - Bump pycurl from 7.43.0.2 to 7.43.0.5 Bumps [pycurl](https://github.com/pycurl/pycurl) from 7.43.0.2 to 7.43.0.5. - [Release notes](https://github.com/pycurl/pycurl/releases) - [Changelog](https://github.com/pycurl/pycurl/blob/master/ChangeLog) - [Commits](https://github.com/pycurl/pycurl/commits) Signed-off-by: dependabot-preview[bot] [09:02:06] [09:02:07] [ GitHub - pycurl/pycurl: PycURL - Python interface to libcurl ] - github.com [09:02:07] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 031c65247 - Merge pull request #23 from miraheze/dependabot/pip/master/pycurl-7.43.0.5 [09:02:07] [ Releases · pycurl/pycurl · GitHub ] - github.com [09:02:08] [ pycurl/ChangeLog at master · pycurl/pycurl · GitHub ] - github.com [09:02:09] [02bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/pycurl-7.43.0.5 - 13https://git.io/JJLum [09:02:09] [ Commits · pycurl/pycurl · GitHub ] - github.com [09:02:10] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/pycurl-7.43.0.5 [09:02:12] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:03:07] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:03:09] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqOx [09:03:11] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 038d5f0d9 - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [09:03:11] [09:03:11] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [09:03:12] [ Releases · praw-dev/praw · GitHub ] - github.com [09:03:12] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:03:12] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [09:03:19] [ Comparing v5.4.0...v7.1.0 · praw-dev/praw · GitHub ] - github.com [09:03:39] [02bots-pip-auto] 07RhinosF1 commented on pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqOj [09:34:10] [02miraheze/bots-pip-auto] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqGe [09:34:12] [02miraheze/bots-pip-auto] 07RhinosF1 0353be82a - Update requirements.txt [09:34:22] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:35:18] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:35:21] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqGT [09:35:23] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03cdff92e - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [09:35:23] [09:35:23] [ GitHub - praw-dev/praw: PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API. ] - github.com [09:35:24] [ Releases · praw-dev/praw · GitHub ] - github.com [09:35:24] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:35:24] [ praw/CHANGES.rst at master · praw-dev/praw · GitHub ] - github.com [09:35:33] [ Comparing v5.4.0...v7.1.0 · praw-dev/praw · GitHub ] - github.com [09:38:55] [02miraheze/bots-pip-auto] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqGW [09:38:56] [02miraheze/bots-pip-auto] 07RhinosF1 0306628c1 - Update requirements.txt [09:39:04] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:40:00] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [09:40:03] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqGB [09:40:04] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03e7ba389 - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [09:40:04] [09:40:06] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [10:02:02] [02miraheze/bots-pip-auto] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqZg [10:02:04] [02miraheze/bots-pip-auto] 07RhinosF1 0353381a0 - Update requirements.txt [10:02:14] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [10:03:09] [02bots-pip-auto] 07dependabot-preview[bot] edited pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [10:03:11] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] pushed 031 commit to 03dependabot/pip/master/praw-7.1.0 [+0/-0/±1] 13https://git.io/JJqZr [10:03:12] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] 03982c2a4 - Bump praw from 5.4.0 to 7.1.0 Bumps [praw](https://github.com/praw-dev/praw) from 5.4.0 to 7.1.0. - [Release notes](https://github.com/praw-dev/praw/releases) - [Changelog](https://github.com/praw-dev/praw/blob/master/CHANGES.rst) - [Commits](https://github.com/praw-dev/praw/compare/v5.4.0...v7.1.0) Signed-off-by: dependabot-preview[bot] [10:03:13] [10:03:14] [02bots-pip-auto] 07dependabot-preview[bot] synchronize pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [10:03:55] [02bots-pip-auto] 07RhinosF1 commented on pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqZK [10:03:56] [02bots-pip-auto] 07dependabot-preview[bot] closed pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqmF [10:03:58] [02miraheze/bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/praw-7.1.0 [10:03:59] [02bots-pip-auto] 07dependabot-preview[bot] deleted branch 03dependabot/pip/master/praw-7.1.0 - 13https://git.io/JJLum [10:05:31] [02bots-pip-auto] 07RhinosF1 commented on pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqZ1 [10:05:56] [02bots-pip-auto] 07RhinosF1 commented on pull request 03#21: Bump praw from 5.4.0 to 7.1.0 - 13https://git.io/JJqZM [10:53:20] PROBLEM - db7 Check MariaDB Replication on db7 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 284s [10:58:43] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJqCG [10:58:44] [02miraheze/mw-config] 07Reception123 03ee542ac - dcwiki settings T5883 [10:58:46] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [10:58:47] [02mw-config] 07Reception123 opened pull request 03#3153: dcwiki settings T5883 - 13https://git.io/JJqCZ [11:02:52] [02mw-config] 07Reception123 closed pull request 03#3153: dcwiki settings T5883 - 13https://git.io/JJqCZ [11:02:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqCV [11:02:55] [02miraheze/mw-config] 07Reception123 0373006b8 - dcwiki settings T5883 (#3153) [11:02:57] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [11:02:58] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [11:44:19] PROBLEM - cloud2 Current Load on cloud2 is CRITICAL: CRITICAL - load average: 22.09, 32.34, 24.72 [11:47:05] PROBLEM - cloud2 Current Load on cloud2 is WARNING: WARNING - load average: 10.62, 23.25, 22.51 [11:52:35] RECOVERY - cloud2 Current Load on cloud2 is OK: OK - load average: 12.38, 17.23, 20.16 [11:56:05] RECOVERY - db7 Check MariaDB Replication on db7 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [13:24:11] [02mw-config] 07Amanda-Catherine commented on pull request 03#3133: Userrights config for simcitywiki - 13https://git.io/JJq0w [13:40:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJqEX [13:40:23] [02miraheze/mw-config] 07Reception123 03ade5247 - add wgVisualEditorEnableDiffPage to managewiki [13:40:25] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [13:40:34] [02mw-config] 07Reception123 opened pull request 03#3154: add wgVisualEditorEnableDiffPage to managewiki - 13https://git.io/JJqE1 [13:41:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJqEM [13:41:27] [02miraheze/mw-config] 07Reception123 039a68edf - Update LocalSettings.php [13:41:28] [02mw-config] 07Reception123 synchronize pull request 03#3154: add wgVisualEditorEnableDiffPage to managewiki - 13https://git.io/JJqE1 [13:45:03] [02mw-config] 07Reception123 closed pull request 03#3154: add wgVisualEditorEnableDiffPage to managewiki - 13https://git.io/JJqE1 [13:45:05] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JJqEA [13:45:06] [02miraheze/mw-config] 07Reception123 03f6f554b - add wgVisualEditorEnableDiffPage to managewiki (#3154) * add wgVisualEditorEnableDiffPage to managewiki * Update LocalSettings.php [13:45:08] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [13:45:09] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [13:51:08] PROBLEM - wiki.tallguysfree.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tallguysfree.com' expires in 15 day(s) (Fri 24 Jul 2020 13:45:16 GMT +0000). [13:51:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJquP [13:51:56] [02miraheze/ssl] 07MirahezeSSLBot 038183473 - Bot: Update SSL cert for wiki.tallguysfree.com [14:10:12] PROBLEM - cp6 Stunnel Http for mw4 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:12] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw4 mw6 [14:11:23] PROBLEM - mw4 HTTPS on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:11:23] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 2 backends are down. mw4 mw6 [14:11:46] PROBLEM - cp8 Stunnel Http for mw4 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:11:56] PROBLEM - mw6 HTTPS on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:02] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.54, 1.79, 1.44 [14:12:10] PROBLEM - cp3 Stunnel Http for mw4 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:15] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 2 backends are down. mw4 mw6 [14:12:41] PROBLEM - cp6 Stunnel Http for mw6 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:12:59] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:27] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:35] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:13:37] PROBLEM - cp8 Stunnel Http for mw6 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:14:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw4 mw6 [14:15:14] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.39, 1.15, 1.26 [14:29:17] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15539 bytes in 0.989 second response time [14:29:27] RECOVERY - cp7 Stunnel Http for mw6 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.003 second response time [14:29:38] RECOVERY - cp8 Stunnel Http for mw6 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.322 second response time [14:30:50] RECOVERY - mw4 HTTPS on mw4 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.005 second response time [14:31:06] RECOVERY - cp8 Stunnel Http for mw4 on cp8 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.337 second response time [14:31:09] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [14:31:29] RECOVERY - mw6 HTTPS on mw6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 557 bytes in 0.005 second response time [14:31:56] RECOVERY - cp3 Stunnel Http for mw4 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 1.011 second response time [14:31:57] RECOVERY - cp6 Stunnel Http for mw6 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.004 second response time [14:32:29] RECOVERY - cp7 Stunnel Http for mw4 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.041 second response time [14:32:40] RECOVERY - cp6 Stunnel Http for mw4 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15525 bytes in 0.010 second response time [14:32:46] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [14:32:46] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [14:33:15] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [14:34:52] !log root@gluster1:/home/paladox# gluster volume set mvol network.inode-lru-limit 1000 [14:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:38] !log root@gluster1:/home/paladox# gluster volume set mvol performance.cache-size 112MB [14:35:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:25] !log root@gluster1:/home/paladox# gluster volume set mvol performance.low-prio-threads 24 [14:36:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:30] !log root@gluster1:/home/paladox# gluster volume set mvol performance.normal-prio-threads 24 [14:36:36] !log root@gluster1:/home/paladox# gluster volume set mvol performance.high-prio-threads 24 [14:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:39:41] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:42:37] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [14:46:54] [02mw-config] 07RhinosF1 commented on pull request 03#3133: Userrights config for simcitywiki - 13https://git.io/JJqaA [14:50:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqVL [14:50:18] [02miraheze/puppet] 07paladox 030939f48 - glusterfs: Modify the options slightly Changes: - entry-timeout to 0 - attribute-timeout to 0 - unsets negative-timeout [14:52:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqVY [14:52:24] [02miraheze/puppet] 07paladox 0374c633d - glusterfs: Set lru-limit to 2000 in the mount Defaults to 131072 [14:52:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqVO [14:52:52] [02miraheze/puppet] 07paladox 03527cc44 - glusterfs: Set kernel-writeback-cache to on [14:57:37] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:59:35] * Hispano76 greetings [15:00:04] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [15:02:52] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:03:55] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:00] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:41] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:04:45] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:58] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:05:00] RECOVERY - wiki.tallguysfree.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tallguysfree.com' will expire on Tue 06 Oct 2020 12:51:47 GMT +0000. [15:05:28] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [15:05:48] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [15:07:39] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [15:10:23] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JJqwe [15:10:25] [02miraheze/mw-config] 07Reception123 03f7465b4 - wgVisualEditorAvailableContentModels for dcwiki per req T5883 [15:10:26] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:10:31] [02mw-config] 07Reception123 opened pull request 03#3155: wgVisualEditorAvailableContentModels for dcwiki per req T5883 - 13https://git.io/JJqwv [15:13:01] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:13:05] [02mw-config] 07Reception123 closed pull request 03#3155: wgVisualEditorAvailableContentModels for dcwiki per req T5883 - 13https://git.io/JJqwv [15:13:07] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqwL [15:13:08] [02miraheze/mw-config] 07Reception123 03254136a - wgVisualEditorAvailableContentModels for dcwiki per req T5883 (#3155) [15:13:10] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [15:13:11] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [15:14:32] !log root@gluster1:/mnt# gluster volume set mvol network.inode-lru-limit 9000 [15:14:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:20:02] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqwR [15:20:04] [02miraheze/mw-config] 07Reception123 031d1dab3 - rm usebetatoolbar-cgd on dcwiki per req T5883 [15:20:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqwz [15:20:37] [02miraheze/mw-config] 07Reception123 032ded9dd - Update LocalSettings.php [15:36:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqrI [15:36:20] [02miraheze/puppet] 07paladox 03997d023 - db9: Lower buffer pool by 1g Won't apply this, but if MariaDB OOM's, it'll pick this up. [16:12:15] [02mw-config] 07Amanda-Catherine commented on pull request 03#3133: Userrights config for simcitywiki - 13https://git.io/JJqo1 [16:13:29] hi [16:14:12] i want to ask something about audio in wiki articles [16:14:56] tell us [16:16:33] Hi Hispano [16:17:40] I want to make the audio play in the same tab where it is in the link and not to open a new tab [16:18:09] in this wikipedia page the audio plays in the same article page: https://en.wikipedia.org/wiki/The_Story_of_Us_(song)#Composition [16:19:06] What do I have to do so that my wiki can play the audio in a similar way? [16:20:39] Eso está usando la plantilla Listen, en español es https://es.wikipedia.org/wiki/Plantilla:Multimedia [16:20:53] bien. Gracias. [16:21:08] tendría que añadir una plantilla [16:22:14] ¿puedo copiar esa plantilla tal cual e introducirla en mi wiki? [16:22:50] claro [16:23:12] bien, pues muchas gracias [16:23:29] si tengo alguna duda ya contactaré de nuevo [16:23:33] En general, todas las plantillas de los proyectos Wikimedia pueden ser usadas siempre que atribuyas el origen en el doc o en el resumen de edición [16:26:39] PROBLEM - cp8 APT on cp8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [16:50:41] PROBLEM - phab1 APT on phab1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:33:08] PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [17:54:40] paladox On metawiki, fallback is Skin or is bug? (and apioutput is exist?) in prefereces user [17:55:35] [02puppet] 07Elaeagnifolia opened pull request 03#1436: Add NicoVideo to CSP per T5535 - 13https://git.io/JJqPG [17:56:16] yeh, it's a skin. [17:56:24] oh [17:56:44] i believe that's an internal skin for mediawiki [17:57:10] it's supposed to be used for the webinstaller i think? [18:20:45] Paladox: That's a bug. It shouldn't be available in preferences. We need to put it in $wgSkipSkins [18:21:05] hispano76: ^ [18:22:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqXf [18:22:40] [02miraheze/mw-config] 07paladox 034e757af - Fix wgSkipSkins [18:23:52] @Reception123 FYI [18:23:54] Fixed [18:25:07] Ty [18:42:16] Hello SouthparkfanTest! If you have any questions, feel free to ask and someone should answer soon. [18:44:11] [02mw-config] 07paladox closed pull request 03#3133: Userrights config for simcitywiki - 13https://git.io/JJJmA [18:44:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqXr [18:44:14] [02miraheze/mw-config] 07Amanda-Catherine 032ddb181 - Userrights config for simcitywiki (#3133) * Userrights config for simcitywiki Re-adding a few steward rights that were lost while overriding CU and OS groups Grant all 2FA-related rights to stewards, just in case Create banned group to remove read access from a registered user * remove orphaned "user" permission the logged in user only protection level that [18:44:14] defines this right has been removed * Update LocalSettings.php * re-add a permission per Voidwalker * drop restricted OATHAuth rights for now disputed as to whether or not they are necessary and holding up this PR. Will likely revisit at a later time [19:03:32] Is there a tidy way to add a page-wide line, as a break? Similar to what you'd see under a Header 2? [19:07:28] Pretty sure you can put four dashes on a line like ---- to get that effect. If not, try
[19:09:35] Four dashes worked, thank you kindly Void! [19:09:47] 👍 [20:08:09] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqDO [20:08:10] [02miraheze/puppet] 07Southparkfan 0359c75f0 - mysql prometheus exporter: collect more data [20:13:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJqD4 [20:13:18] [02miraheze/mw-config] 07paladox 03d2ffe1c - Update Sitenotice.php [20:13:19] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:13:21] [02mw-config] 07paladox opened pull request 03#3156: Update Sitenotice.php - 13https://git.io/JJqDB [20:14:24] [02mw-config] 07paladox synchronize pull request 03#3156: Update Sitenotice.php - 13https://git.io/JJqDB [20:14:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JJqDz [20:14:27] [02miraheze/mw-config] 07paladox 037e81c22 - Update Sitenotice.php [20:14:52] [02mw-config] 07paladox closed pull request 03#3156: Update Sitenotice.php - 13https://git.io/JJqDB [20:14:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqDa [20:14:55] [02miraheze/mw-config] 07paladox 036b2c328 - Update Sitenotice.php (#3156) * Update Sitenotice.php * Update Sitenotice.php [20:14:56] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:14:58] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:15:10] Miraheze needs to perform emergency maintenance on one of its database clusters. To do that, one of the servers must be restarted. Maintenance is scheduled between July 8, 20:45 UTC and 21:15 UTC. Please save your edits before 20:45 UTC! [20:16:13] miraheze/mw-config/paladox-patch-1/7e81c22 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/706298949 [20:16:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqDo [20:16:14] [02miraheze/mw-config] 07paladox 03e77894e - Update Sitenotice.php [20:18:22] !log added 2GB swapfile on db9 to avoid mariadb OOM - emergency restart planned to reduce mem usage [20:18:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:05] PROBLEM - antiguabarbudacalypso.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/JJqDD [20:21:55] [02miraheze/puppet] 07paladox 03ef78272 - db9: Decrease open cache table to 7500 [20:21:56] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [20:21:58] [02puppet] 07paladox opened pull request 03#1437: db9: Decrease open cache table to 7500 - 13https://git.io/JJqDy [20:22:44] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Thu 20 Aug 2020 14:54:45 GMT +0000. [20:29:08] !log db9: set global innodb_max_dirty_pages_pct = 1; [20:29:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:39] i must be an idiot because i can't follow this https://meta.miraheze.org/wiki/ManageWiki#How_do_I_change_my_logo.2Ffavicon.3F [20:36:06] once i have the URL of the raw image of the favicon i want, what do i do with it to make it the favicon for my wiki? [20:38:44] ah i figured it out [20:38:54] can't edit the wiki though [20:42:54] but that section should say something about going to the styling tab in additional settings [20:43:30] !log set pct to 0 [20:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:44:36] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqy3 [20:44:38] [02miraheze/mw-config] 07Southparkfan 03794463c - All wikis in RO mode [20:45:59] !log stop jobrunner/jobchron [20:46:12] !log db9 set global read_only=1; [20:46:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqyc [20:46:20] [02miraheze/puppet] 07paladox 03a25433f - mariadb: Set table-open-cache-instances to 2 [20:46:41] [02puppet] 07Southparkfan closed pull request 03#1437: db9: Decrease open cache table to 7500 - 13https://git.io/JJqDy [20:46:42] [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqyC [20:46:44] [02miraheze/puppet] 07paladox 03a8d5196 - db9: Decrease open cache table to 7500 (#1437) [20:47:51] !log db13 set global read_only=1; [20:48:26] !log restarting db9 mysql [20:48:42] !log restarting db13 mysql [20:49:17] bleb: maint in progress [20:49:25] PROBLEM - jobrunner1 JobChron Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:50:19] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:51:03] Good evening [20:51:16] Hi AmandaCath [20:51:43] I'm assuming the 503s are intentional/expected? [20:51:53] AmandaCath: see topic [20:51:54] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [20:52:06] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:52:13] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 57% [20:52:22] I saw the global sitenotice, which is why I assumed it was expected [20:52:40] PROBLEM - db9 MySQL on db9 is CRITICAL: Can't connect to MySQL server on '51.77.109.151' (115) [20:52:48] !log starting mysql now (shutdown took 4 minutes) [20:52:55] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [20:52:59] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [20:53:15] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:53:18] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 51.89.160.142/cpweb, 2001:41d0:800:1056::2/cpweb, 2001:41d0:800:105a::10/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [20:53:24] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:53:29] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [20:53:29] !log killed jobrunner/jobchron again and disabled puppet on jobrunner1 [20:53:36] AmandaCath: yep, emergency maintenance to avoid database oom'ing [20:53:37] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:53:37] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 58% [20:53:43] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [20:54:22] SpF|Cloud: I don't think the logbot is working [20:54:30] we know that [20:54:34] but it is for IRC logging purposes [20:54:43] Ok [20:54:46] and '!log' so you can look for that string [20:54:51] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [20:55:04] AmandaCath: it's transparency, it very obviously doesn't work. Also we can back fill. [20:55:25] "Very obviously doesn't work"? [20:56:13] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [20:56:39] PROBLEM - jobrunner1 Puppet on jobrunner1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [20:57:10] AmandaCath: all wikis are down, it can log to a wiki page [20:57:57] s/can/can't [20:57:57] RhinosF1 meant to say: AmandaCath: all wikis are down, it can't log to a wiki page [20:58:05] PROBLEM - jobrunner1 JobChron Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:58:07] Oh, I thought the bot would always be able to access the one page that it needs to [20:58:14] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 57% [20:58:22] Even if the wiki is down [20:58:36] AmandaCath: no, I don't even know how that would work. [20:58:44] PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [20:59:31] paladox: https://dba.stackexchange.com/questions/60165/what-variables-to-look-to-maintain-mysql-speed-of-scanning-through-all-databas [21:00:02] since the dirty pages trick doesn't seem to make that much difference... [21:00:10] I don't know if it would work with MediaWiki, but in some cases it can be done where a website can be offline to users in a web browser but can be accessed by bots via a backend [21:00:10] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is WARNING: WARNING - NGINX Error Rate is 53% [21:00:20] RhinosF1 ^ [21:00:32] oh [21:00:52] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [21:01:11] AmandaCath: the backend is what's down. [21:01:34] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is WARNING: WARNING - NGINX Error Rate is 44% [21:01:40] And yes that could in theory work with huge redundancy [21:01:44] Which we don't have [21:02:52] Well, in practice the bot(s) would be on their own servers, so they're not dependent on the primary server that runs the website [21:03:10] (I'm talking about a single website here, not a farm) [21:03:27] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 21% [21:04:07] Essentially, you can do some fiddling with bots and servers and create a way to be able to access something that you can't access in a web browser [21:04:16] AmandaCath: the bot's are on another server. It's the databases that are all being restarted so we'd have no backend unless we had fully read/write redundancy. [21:04:49] You could have a directly modified database but not if it's the thing down [21:05:20] these disks are hella slow [21:05:26] I can't really explain the process in words anymore than I already have... it's a complicated mess and I've thankfully only had to do it a few times [21:05:36] SPF|Cloud the block storage? [21:05:40] yes [21:06:00] ovh should be ashamed of calling this high performance storage [21:06:10] ehh, you could probably more easily have the bot stash all logs until such point as it can save them (eg if logging fails due to a 5xx error, the bot will store the attempted log in memory and then save it to the page once the error is gone) [21:06:41] AmandaCath: In theory, yes we could but the infra doesn't exist. I get your point, it's just not our reality. Voidwalker's suggestion might work more realistically though. [21:06:45] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [21:07:22] and iotop suggests i/o performance is the bottleneck for the startup time [21:07:36] I don't think we are talking about the exact same thing RhinosF1 [21:07:36] * RhinosF1 was about to say I'll work on search console now but d'oh that'll fail [21:08:36] AmandaCath: We are, trust me. I understand you. It's just nowhere near how anythings set up and would require quite a change [21:08:38] https://dba.stackexchange.com/a/60544 interesting [21:08:39] [ innodb - What variable(s) to look to maintain MySQL speed of scanning through all database tables while it starts? - Database Administrators Stack Exchange ] - dba.stackexchange.com [21:08:56] whereas Voidwalker's idea sounds somewhat feasible [21:09:23] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is WARNING: WARNING - NGINX Error Rate is 46% [21:09:48] very unfortunate we don't have a mariadb replica for this cluster [21:10:42] SPF|Cloud: could we ever have a situation where we could simply switch the replica/primary around and to maintain uptime? [21:11:16] switching around replicas is a hassle [21:11:34] but the lack of a replica is a major issue [21:11:54] SPF|Cloud: so yes but performing it in an emergency isn't ideal? [21:11:59] if we had one [21:12:16] we had tried to setup a replica a few days ago, but the backup process caused mariadb to crash [21:12:24] :( [21:12:52] RhinosF1: the one example I can give of what I'm describing in action would be to repair/revert an "encryption by destruction" attack [21:13:00] A virus can attack a system and lock the user out of all their files and demand they do something to regain access (usually pay ransom)... but even once they do whatever they were demanded to do, they don't get their files/data back [21:13:11] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [21:13:50] So the only way to recover the data is to access the data through a bot and maintenance wizard [21:14:24] great, mysql started [21:14:44] keeping it in RO for quick testing [21:14:48] AmandaCath: I mean I'd never build that anyway. Just have the !log data saved somewhere by the bot offsite and have it populate when we start up. Maybe have a way to manually mark things unsaved [21:15:17] SPF|Cloud: still 503s Error 503 Backend fetch failed, forwarded for 2a00:23c6:9200:9700:8dcd:c4c8:1084:673, 127.0.0.1 [21:15:17] (Varnish XID 20360659) via cp7 at Wed, 08 Jul 2020 21:14:56 GMT. [21:15:23] give it some time [21:15:33] ok [21:15:53] RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 11% [21:15:57] warming up the buffer pool first [21:16:08] RECOVERY - db9 MySQL on db9 is OK: Uptime: 1413 Threads: 35 Questions: 28427 Slow queries: 904 Opens: 2369 Flush tables: 1 Open tables: 2363 Queries per second avg: 20.118 [21:16:22] RhinosF1: I'm more talking about a case where a single website was taken offline maliciously [21:16:30] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:16:36] !log disabled RO on db9 [21:16:37] Although it can work in any situation [21:16:44] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [21:16:45] disabling RO on mw cluster [21:16:45] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 23% [21:16:54] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [21:16:57] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy [21:16:59] AmandaCath: we're not a single website though. You're completely ignoring how we actually work. [21:17:09] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqSa [21:17:11] [02miraheze/mw-config] 07Southparkfan 03f47fd2e - Disable RO on all clusters [21:17:11] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 4% [21:17:14] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 1% [21:17:29] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [21:17:37] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:17:37] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:17:47] !log enabled puppet on jobrunner1 and restarted job services [21:17:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:03] I'm not "ignoring" anything... I'm just providing explanation behind my previous comment [21:18:22] Oh, I thought the bot would always be able to access the one page that it needs to [21:18:31] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqSw [21:18:32] [02miraheze/mw-config] 07Southparkfan 03fe0d307 - Comment site notice [21:18:46] AmandaCath: which I'm saying your explanation doesn't apply to us [21:19:27] RECOVERY - jobrunner1 JobChron Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [21:19:29] RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:20:02] And that's fine... I'm never said I expected it to [21:21:12] okay, conversation over then [21:24:07] Now, the reason why I signed on is because the PR doesn't seem to have worked [21:24:46] AmandaCath: what's the issue? [21:24:54] Despite my adding the banned group into the AdditionalAddGroups and AdditionalRemoveGroups arrays, it is uneditable [21:25:01] And doesn't appear in ManageWikiPermissions [21:25:04] !log removed or replaced all broken sitemaps on SearchConsole [21:25:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:29:48] !log blacklisted a page in search console & submitted rechecks for a number of errors [21:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:44] !log restart mysqld exporter on db9 [21:49:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:44] !log restarted mysqld exporter on all servers [21:50:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:38] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.29, 1.95, 1.45 [22:33:26] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.96, 1.48, 1.35 [22:49:51] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JJqHD [22:49:52] [02miraheze/MirahezeMagic] 07RhinosF1 038b954b7 - Create generateSitemapIndex.py [22:49:59] paladox: ^ [22:50:22] .gh miraheze/mediawiki [22:50:22] https://github.com/miraheze/mediawiki [22:50:51] ok [22:50:56] #irc-relay is back 🙂 [22:51:09] then you can add the cron to jobrunner.pp [22:51:16] #irc-relay never died, you just don't hear icinga-miraheze [22:51:24] you can use https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/jobrunner.pp#L84 as a guide [22:51:24] [ puppet/jobrunner.pp at master · miraheze/puppet · GitHub ] - github.com [22:51:27] paladox: updating the extension first. [22:51:28] changing the script ofc [22:51:31] and I get that [22:51:34] RhinosF1 your updating? [22:51:56] paladox: yes. How's the file going to be there if MM is outdated? [22:52:30] ok, no need to be snarky. I didn't know if you wanted me to update it. [22:52:34] ok [22:55:56] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JJqHd [22:55:57] [02miraheze/mediawiki] 07RhinosF1 03d0aa5bf - update MM [22:56:22] paladox: can you rebuild LC as there's some i18n changes in there via salt? [22:56:31] ok [22:57:32] !log rebuild lc on mw* and jobrunner1 [22:57:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:40] paladox: test2? [22:58:00] test2 not the priority, it was prod [22:58:12] I know [22:58:28] I just always find it strange how test2 has the worst cache [22:58:32] * RhinosF1 shrugs [22:58:35] !log rebuild lc test2 [22:58:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:59:13] [02puppet] 07RhinosF1 opened pull request 03#1438: Add generateSitemapIndex script - 13https://git.io/JJqHx [22:59:20] paladox: ^ [22:59:39] We should probably do a manual run first so we know it works [23:00:33] [02puppet] 07paladox closed pull request 03#1438: Add generateSitemapIndex script - 13https://git.io/JJqHx [23:00:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqHh [23:00:36] [02miraheze/puppet] 07RhinosF1 03e06c027 - Add generateSitemapIndex script (#1438) This populates static.miraheze.org/sitemaps/sitemap.xml with an index of all sitemaps. It runs 24 hours after the sitemap script for each wiki to ensure that has finished. [23:03:10] paladox: are we going to do a manual run now to ensure it works? ping me when done and I'll add it to GSC [23:04:36] RhinosF1 try running it [23:04:42] *now [23:07:20] paladox: https://static.miraheze.org/sitemaps/sitemap.xml [23:07:25] PROBLEM - mw5 APT on mw5 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:07:27] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:07:28] Ah, there we go [23:07:38] * AmandaCath is back online after a power outage [23:07:42] it doesn't work? [23:07:44] PROBLEM - mw6 APT on mw6 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (12 critical updates). [23:08:08] [02puppet] 07RhinosF1 opened pull request 03#1439: Update jobrunner.pp - 13https://git.io/JJqQI [23:08:08] !log upgrade some packages on mw* and jobrunner1 [23:08:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:08:19] [02puppet] 07paladox closed pull request 03#1439: Update jobrunner.pp - 13https://git.io/JJqQI [23:08:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JJqQt [23:08:22] paladox: no, xml error, looking [23:08:22] [02miraheze/puppet] 07RhinosF1 03ac39bc9 - Update jobrunner.pp (#1439) [23:09:03] ok [23:09:07] well i have to go for now [23:09:19] i will look back in a bit [23:10:08] RECOVERY - mw5 APT on mw5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:11] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:16] paladox: i think it just hiccuped. trying again [23:10:29] RECOVERY - mw6 APT on mw6 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:10:48] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [23:11:33] paladox: yeah looks good now, I think it was just weird [23:12:08] paladox: oh shit! That's not deploying to all appservers [23:12:25] so it's inconsistent in what's showing [23:13:31] The crown is only ran on one server [23:13:53] I assume you mean cron and how do we get it on all servers? [23:14:17] We don’t put it on all since it’s saving the file on a mount? [23:14:35] it's not consistently showing though [23:14:36] So all servers have access? [23:14:46] refresh the page and it's different [23:15:16] paladox@ [23:15:33] Ok, can you see why [23:15:57] I guess that each appserver is serving differently but no [23:16:17] This is the bit I wanted your help with. Getting it to static [23:16:19] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:16:34] It should be on static [23:16:58] You legit are doing https://github.com/miraheze/MirahezeMagic/blob/master/py/generateSitemapIndex.py#L18 [23:16:58] [ MirahezeMagic/generateSitemapIndex.py at master · miraheze/MirahezeMagic · GitHub ] - github.com [23:17:03] Which is static [23:17:06] keep refreshing https://static.miraheze.org/sitemaps/sitemap.xml and you'll see it change [23:17:51] Hmm [23:18:34] Can you do me a favour and see if all mw* are connected to gluster please? [23:18:51] how? [23:18:56] Simply doing ls /mnt/mediawiki-static [23:19:10] You may have to run that as www-data [23:20:05] looking [23:20:58] mw[45] and jobrunner1 are good [23:22:15] mw6 fine [23:22:42] Works for me now (reloading) [23:22:44] RhinosF1: were we able to figure out why https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L1802 and https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L224 is still producing this: [23:22:45] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [23:22:46] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [23:22:47] Wasn’t before [23:22:59] https://i.imgur.com/dggv3eZ.jpg [23:23:16] @paladox: all servers display fine [23:23:41] I can see one wrong [23:24:13] One wrong? [23:24:39] yeah one showing wrong version [23:24:42] AmandaCath: no [23:24:45] Oh? [23:25:50] Reception123 Voidwalker any ideas? [23:25:59] @paladox: Is there anyway from console to tell which backend is serving? [23:26:07] AmandaCath: one of them is asleep [23:26:24] You can use curl -I [23:26:37] Meh, they've both been idle for an hour, so I can't tell [23:26:56] @paladox: from a browser console? [23:27:12] AmandaCath: Reception.123 [23:27:58] Oh, I think it’s possible to view headers [23:28:04] .tell JohnLewis can you please investigate why https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L1802 and https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L224 is still resulting in https://i.imgur.com/dggv3eZ.jpg [23:28:05] AmandaCath: I'll pass that on when JohnLewis is around. [23:28:05] how? [23:28:05] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [23:28:05] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [23:28:24] https://mkyong.com/computer-tips/how-to-view-http-headers-in-google-chrome/ [23:28:24] [ How to view HTTP headers in Google Chrome? - Mkyong.com ] - mkyong.com [23:29:14] .tell JohnLewis sorry, second link should be [23:29:14] AmandaCath: I'll pass that on when JohnLewis is around. [23:29:18] Agh [23:29:42] Anyway to invalidate a tell message? [23:30:10] AmandaCath: set a reminder for me to hack the db at 9:30am UK time tommorow [23:30:59] .tell JohnLewis sorry, my phone is screwing up. The second link in my first message should be https://github.com/miraheze/mw-config/blob/master/LocalSettings.php#L2224 [23:30:59] AmandaCath: I'll pass that on when JohnLewis is around. [23:31:00] [ mw-config/LocalSettings.php at master · miraheze/mw-config · GitHub ] - github.com [23:31:17] > https://mkyong.com/computer-tips/how-to-view-http-headers-in-google-chrome/ @paladox @RhinosF1 just had me look up the http headers. I found this web tool (https://reqbin.com/curl) super easy to use. [23:31:18] [ How to view HTTP headers in Google Chrome? - Mkyong.com ] - mkyong.com [23:31:18] [ Post Curl Requests Online ] - reqbin.com [23:31:26] UK time = what in Canada time? [23:31:40] And which time do I use in the command? [23:31:42] AmandaCath: what timezone are you? [23:31:46] AmandaCath, UK is 7 hours ahead of me, so 4 hours ahead of you [23:31:56] RhinosF1: Ottawa [23:32:10] The place I call home :) [23:32:28] GMT-4.00, I believe, @RhinosF1 [23:32:29] AmandaCath: you are -4, we are +1 now so 5 hours difference 4:30am [23:33:04] @paladox: I can't reproduce now so I'm blaming slow purges [23:33:29] So, do I set a reminder for 9:30 AM which would be using my phone's time, or 4:30 AM so that it would be 9:30 for you? [23:33:56] .at 09:30 hack AmandaCath's tell [23:33:57] RhinosF1: Okay, will remind at 2020-07-09 - 09:29:59UTC [23:34:01] AmandaCath: done [23:34:18] Well, that works [23:34:27] !log add https://static.miraheze.org/sitemaps/sitemap.xml to GSC, will purge old one's in the morning [23:34:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:34:33] .time RhinosF1 [23:34:33] Could not find a timezone for you. You can set your timezone with `.settz ` [23:34:37] eh? [23:34:44] .settz Europe/London [23:34:44] I don't know that timezone. Try one from https://sopel.chat/tz [23:34:47] * AmandaCath is clueless as to how to set reminders across timezones [23:35:03] .settz Europe/London [23:35:04] I don't know that timezone. Try one from https://sopel.chat/tz [23:35:15] MirahezeBot: what are you on? [23:35:25] .at 10:30 fix MirahezeBot [23:35:25] RhinosF1: Okay, will remind at 2020-07-09 - 10:29:59UTC [23:35:45] Huh, it's logging in UTC time [23:36:42] I know [23:36:48] it's being a weirdo [23:36:54] .settz Europe/London [23:36:54] I don't know that timezone. Try one from https://sopel.chat/tz [23:37:01] yes you do [23:37:23] hi [23:37:40] Hi hispano76 [23:38:35] and google don't support nested indexing [23:38:45] anyway, I'll fix that in the morning I guess [23:50:36] * AmandaCath thinks we need to recruit more extension reviewers, perhaps from upstream [23:50:56] The "blocked by security review" workboard is getting quite long [23:51:06] Uh oh Notifco, come back