[00:02:28] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 2 minutes ago with 9 failures. Failed resources (up to 3 shown) [00:03:11] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 214s [00:09:55] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 37.52, 25.96, 17.80 [00:11:49] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.62, 23.07, 17.72 [00:13:15] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 114s [00:13:44] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.89, 19.85, 17.17 [00:15:13] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 8s [00:25:10] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 301s [00:28:30] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.88, 3.67, 2.88 [00:30:28] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [00:30:31] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.36, 3.22, 2.81 [00:35:11] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 45s [01:09:12] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 225s [01:11:56] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 5.65, 3.68, 2.51 [01:13:53] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.19, 3.42, 2.55 [01:15:51] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.09, 2.98, 2.50 [01:17:12] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 184s [01:21:11] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [02:33:26] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:40:06] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Mon 07 Jun 2021 23:30:57 GMT +0000. [05:40:55] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/J3wNd [05:40:56] [02miraheze/ssl] 07Reception123 038715ec2 - add wiki.consid.vn cert [05:40:58] [02miraheze/ssl] 07Reception123 03bf6c823 - add thepolicyhub.org.uk cert [05:40:59] [02miraheze/ssl] 07Reception123 035854b06 - add mabi.myomi.net cert [06:05:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3whH [06:05:13] [02miraheze/services] 07MirahezeSSLBot 035545a77 - BOT: Updating services config for wikis [06:08:39] !log DELETED and DROPPED wintergatancommunitywiki (reset) [06:08:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:10:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3wjg [06:10:14] [02miraheze/services] 07MirahezeSSLBot 03518b807 - BOT: Updating services config for wikis [07:50:39] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 2.33, 4.15, 2.22 [07:52:39] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.47, 2.84, 1.97 [09:59:37] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [10:13:22] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: rDNS OK - wiki.zymonic.com reverse DNS resolves to cp10.miraheze.org [11:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3r5K [11:10:12] [02miraheze/services] 07MirahezeSSLBot 0337d02a8 - BOT: Updating services config for wikis [11:50:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3rjG [11:50:12] [02miraheze/services] 07MirahezeSSLBot 03f389d40 - BOT: Updating services config for wikis [14:10:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3oa7 [14:10:15] [02miraheze/services] 07MirahezeSSLBot 031ee6a86 - BOT: Updating services config for wikis [14:26:50] Was MW updated recently? [14:28:25] R4356th: no, still running 1.35 [14:29:44] Oh, I was wondering if there was a patch version update. [14:30:12] Anyway, I got that sorted. Thanks. 🙂 [14:37:30] [02miraheze/mediawiki] 07paladox pushed 034 commits to 03REL1_35 [+0/-0/±4] 13https://git.io/J3oiH [14:37:31] [02miraheze/mediawiki] 07paladox 036cd0f79 - Update CommentBox [14:37:33] [02miraheze/mediawiki] 07paladox 030228249 - Update AbuseFilter [14:37:34] [02miraheze/mediawiki] 07paladox 03a704add - Update PageForms [14:37:36] [02miraheze/mediawiki] 07paladox 03efc29b3 - update OAuth [14:46:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/J3oXH [14:46:19] [02miraheze/mediawiki] 07paladox 03d05e032 - Revert "Dependabot: Switch to REL1_36 (#1444)" [14:47:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/DataDump-85d1107 [+0/-0/±1] 13https://git.io/J3oXj [14:47:06] [02miraheze/mediawiki] 07dependabot[bot] 0332062d6 - Bump extensions/DataDump from `6c78be2` to `85d1107` [14:47:07] [02mediawiki] 07dependabot[bot] opened pull request 03#1552: Bump extensions/DataDump from `6c78be2` to `85d1107` - 13https://git.io/J3o1e [14:47:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/DataDump-85d1107 - 13https://git.io/vbL5b [14:47:10] [02mediawiki] 07dependabot[bot] labeled pull request 03#1552: Bump extensions/DataDump from `6c78be2` to `85d1107` - 13https://git.io/J3o1e [14:47:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CommentStreams-daf5e8f [+0/-0/±1] 13https://git.io/J3o1v [14:47:13] [02miraheze/mediawiki] 07dependabot[bot] 03fa25950 - Bump extensions/CommentStreams from `ed3fc73` to `daf5e8f` [14:47:15] [02mediawiki] 07dependabot[bot] opened pull request 03#1553: Bump extensions/CommentStreams from `ed3fc73` to `daf5e8f` - 13https://git.io/J3o1f [14:47:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CommentStreams-daf5e8f - 13https://git.io/vbL5b [14:47:18] [02mediawiki] 07dependabot[bot] labeled pull request 03#1553: Bump extensions/CommentStreams from `ed3fc73` to `daf5e8f` - 13https://git.io/J3o1f [14:47:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Citizen-38f674e [+0/-0/±1] 13https://git.io/J3o1J [14:47:21] [02miraheze/mediawiki] 07dependabot[bot] 03402d63f - Bump skins/Citizen from `2653336` to `38f674e` [14:47:22] [02mediawiki] 07dependabot[bot] opened pull request 03#1554: Bump skins/Citizen from `2653336` to `38f674e` - 13https://git.io/J3o1U [14:47:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Citizen-38f674e - 13https://git.io/vbL5b [14:47:25] [02mediawiki] 07dependabot[bot] labeled pull request 03#1554: Bump skins/Citizen from `2653336` to `38f674e` - 13https://git.io/J3o1U [14:47:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#1539: Bump skins/Citizen from `8d82e7c` to `293d748` - 13https://git.io/J3o1k [14:47:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-9fa7338 [+0/-0/±1] 13https://git.io/J3o1I [14:47:30] [02miraheze/mediawiki] 07dependabot[bot] 03707ed49 - Bump extensions/ArticleToCategory2 from `09ffed6` to `9fa7338` [14:47:31] [02mediawiki] 07dependabot[bot] opened pull request 03#1555: Bump extensions/ArticleToCategory2 from `09ffed6` to `9fa7338` - 13https://git.io/J3o1L [14:47:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ArticleToCategory2-9fa7338 - 13https://git.io/vbL5b [14:47:34] [02mediawiki] 07dependabot[bot] closed pull request 03#1539: Bump skins/Citizen from `8d82e7c` to `293d748` - 13https://git.io/J3RmZ [14:47:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#1555: Bump extensions/ArticleToCategory2 from `09ffed6` to `9fa7338` - 13https://git.io/J3o1L [14:47:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-293d748 [14:47:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ProtectSite-0c217bd [+0/-0/±1] 13https://git.io/J3o1q [14:47:40] [02miraheze/mediawiki] 07dependabot[bot] 0315ccfe6 - Bump extensions/ProtectSite from `0608d19` to `0c217bd` [14:47:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/skins/Citizen-293d748 - 13https://git.io/vbL5b [14:47:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/ProtectSite-0c217bd - 13https://git.io/vbL5b [14:47:45] [02mediawiki] 07dependabot[bot] opened pull request 03#1556: Bump extensions/ProtectSite from `0608d19` to `0c217bd` - 13https://git.io/J3o1m [14:47:46] [02mediawiki] 07dependabot[bot] labeled pull request 03#1556: Bump extensions/ProtectSite from `0608d19` to `0c217bd` - 13https://git.io/J3o1m [14:47:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/GeoCrumbs-4143df1 [+0/-0/±1] 13https://git.io/J3o1O [14:47:49] [02miraheze/mediawiki] 07dependabot[bot] 0388d1f74 - Bump extensions/GeoCrumbs from `ff63e3b` to `4143df1` [14:47:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/GeoCrumbs-4143df1 - 13https://git.io/vbL5b [14:47:52] [02mediawiki] 07dependabot[bot] opened pull request 03#1557: Bump extensions/GeoCrumbs from `ff63e3b` to `4143df1` - 13https://git.io/J3o13 [14:47:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1557: Bump extensions/GeoCrumbs from `ff63e3b` to `4143df1` - 13https://git.io/J3o13 [14:47:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/VisualEditor-c16410f [+0/-0/±1] 13https://git.io/J3o1c [14:47:57] [02miraheze/mediawiki] 07dependabot[bot] 03a99f728 - Bump extensions/VisualEditor from `07ed253` to `c16410f` [14:47:58] [02mediawiki] 07dependabot[bot] opened pull request 03#1558: Bump extensions/VisualEditor from `07ed253` to `c16410f` - 13https://git.io/J3o1C [14:48:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-c16410f - 13https://git.io/vbL5b [14:48:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#1558: Bump extensions/VisualEditor from `07ed253` to `c16410f` - 13https://git.io/J3o1C [14:48:03] [02mediawiki] 07dependabot[bot] commented on pull request 03#1548: Bump extensions/VisualEditor from `eaa1d00` to `09aed3b` - 13https://git.io/J3o1l [14:48:04] [02mediawiki] 07dependabot[bot] closed pull request 03#1548: Bump extensions/VisualEditor from `eaa1d00` to `09aed3b` - 13https://git.io/J3RmA [14:48:06] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VisualEditor-09aed3b [14:48:07] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/VisualEditor-09aed3b - 13https://git.io/vbL5b [14:48:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Maps-11d8882 [+0/-0/±1] 13https://git.io/J3o18 [14:48:10] [02miraheze/mediawiki] 07dependabot[bot] 0316181f7 - Bump extensions/Maps from `56bc21f` to `11d8882` [14:48:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Maps-11d8882 - 13https://git.io/vbL5b [14:48:13] [02mediawiki] 07dependabot[bot] opened pull request 03#1559: Bump extensions/Maps from `56bc21f` to `11d8882` - 13https://git.io/J3o14 [14:48:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1559: Bump extensions/Maps from `56bc21f` to `11d8882` - 13https://git.io/J3o14 [14:48:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/DarkMode-a4b2ada [+0/-0/±1] 13https://git.io/J3o1B [14:48:18] [02miraheze/mediawiki] 07dependabot[bot] 03f52bff5 - Bump extensions/DarkMode from `73d8847` to `a4b2ada` [14:48:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/DarkMode-a4b2ada - 13https://git.io/vbL5b [14:48:21] [02mediawiki] 07dependabot[bot] opened pull request 03#1560: Bump extensions/DarkMode from `73d8847` to `a4b2ada` - 13https://git.io/J3o1R [14:48:22] [02mediawiki] 07dependabot[bot] labeled pull request 03#1560: Bump extensions/DarkMode from `73d8847` to `a4b2ada` - 13https://git.io/J3o1R [14:48:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SocialProfile-1e4e3de [+0/-0/±1] 13https://git.io/J3o1u [14:48:25] [02miraheze/mediawiki] 07dependabot[bot] 033a8ae57 - Bump extensions/SocialProfile from `fbb875b` to `1e4e3de` [14:48:27] [02mediawiki] 07dependabot[bot] opened pull request 03#1561: Bump extensions/SocialProfile from `fbb875b` to `1e4e3de` - 13https://git.io/J3o1z [14:48:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SocialProfile-1e4e3de - 13https://git.io/vbL5b [14:48:30] [02mediawiki] 07dependabot[bot] commented on pull request 03#1529: Bump extensions/DarkMode from `9db53ec` to `e56ef5a` - 13https://git.io/J3o1g [14:48:31] [02mediawiki] 07dependabot[bot] closed pull request 03#1529: Bump extensions/DarkMode from `9db53ec` to `e56ef5a` - 13https://git.io/J3RqK [14:48:33] [02mediawiki] 07dependabot[bot] labeled pull request 03#1561: Bump extensions/SocialProfile from `fbb875b` to `1e4e3de` - 13https://git.io/J3o1z [14:48:34] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DarkMode-e56ef5a - 13https://git.io/vbL5b [14:48:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/DarkMode-e56ef5a [14:48:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#1538: Bump extensions/SocialProfile from `cdf8e8b` to `97bd251` - 13https://git.io/J3o12 [14:48:39] [02mediawiki] 07dependabot[bot] closed pull request 03#1538: Bump extensions/SocialProfile from `cdf8e8b` to `97bd251` - 13https://git.io/J3Rmq [14:48:40] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-97bd251 [14:48:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/SocialProfile-97bd251 - 13https://git.io/vbL5b [14:48:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/LocalisationUpdate-3181c7a [+0/-0/±1] 13https://git.io/J3o1w [14:48:45] [02miraheze/mediawiki] 07dependabot[bot] 037dcb413 - Bump extensions/LocalisationUpdate from `7fc5f07` to `3181c7a` [14:48:46] [02mediawiki] 07dependabot[bot] opened pull request 03#1562: Bump extensions/LocalisationUpdate from `7fc5f07` to `3181c7a` - 13https://git.io/J3o1r [14:48:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/LocalisationUpdate-3181c7a - 13https://git.io/vbL5b [14:48:49] [02mediawiki] 07dependabot[bot] labeled pull request 03#1562: Bump extensions/LocalisationUpdate from `7fc5f07` to `3181c7a` - 13https://git.io/J3o1r [14:48:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Comments-ba02f49 [+0/-0/±1] 13https://git.io/J3o1o [14:48:52] [02miraheze/mediawiki] 07dependabot[bot] 03092d234 - Bump extensions/Comments from `d75fa77` to `ba02f49` [14:48:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Comments-ba02f49 - 13https://git.io/vbL5b [14:48:55] [02mediawiki] 07dependabot[bot] opened pull request 03#1563: Bump extensions/Comments from `d75fa77` to `ba02f49` - 13https://git.io/J3o16 [14:48:57] [02mediawiki] 07dependabot[bot] labeled pull request 03#1563: Bump extensions/Comments from `d75fa77` to `ba02f49` - 13https://git.io/J3o16 [14:48:58] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/CreateWiki-4c29852 [+0/-0/±1] 13https://git.io/J3o1P [14:49:00] [02miraheze/mediawiki] 07dependabot[bot] 03312d4fa - Bump extensions/CreateWiki from `f424044` to `4c29852` [14:49:01] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/CreateWiki-4c29852 - 13https://git.io/vbL5b [14:49:03] [02mediawiki] 07dependabot[bot] opened pull request 03#1564: Bump extensions/CreateWiki from `f424044` to `4c29852` - 13https://git.io/J3o1X [14:49:04] [02mediawiki] 07dependabot[bot] labeled pull request 03#1564: Bump extensions/CreateWiki from `f424044` to `4c29852` - 13https://git.io/J3o1X [14:49:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-cf18475 [+0/-0/±1] 13https://git.io/J3o11 [14:49:07] [02miraheze/mediawiki] 07dependabot[bot] 030452abb - Bump extensions/SnapProjectEmbed from `53ffd48` to `cf18475` [14:49:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#1540: Bump extensions/CreateWiki from `70bfc9c` to `f424044` - 13https://git.io/J3o1M [14:49:10] [02mediawiki] 07dependabot[bot] opened pull request 03#1565: Bump extensions/SnapProjectEmbed from `53ffd48` to `cf18475` - 13https://git.io/J3o1D [14:49:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-cf18475 - 13https://git.io/vbL5b [14:49:13] [02mediawiki] 07dependabot[bot] closed pull request 03#1540: Bump extensions/CreateWiki from `70bfc9c` to `f424044` - 13https://git.io/J3Rm4 [14:49:15] [02mediawiki] 07dependabot[bot] labeled pull request 03#1565: Bump extensions/SnapProjectEmbed from `53ffd48` to `cf18475` - 13https://git.io/J3o1D [14:49:16] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-f424044 - 13https://git.io/vbL5b [14:49:18] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/extensions/CreateWiki-f424044 [14:49:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/vendor-6070660 [+0/-0/±1] 13https://git.io/J3o1y [14:49:21] [02miraheze/mediawiki] 07dependabot[bot] 03448d68a - Bump vendor from `327fc31` to `6070660` [14:49:22] [02mediawiki] 07dependabot[bot] opened pull request 03#1566: Bump vendor from `327fc31` to `6070660` - 13https://git.io/J3o19 [14:49:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/vendor-6070660 - 13https://git.io/vbL5b [14:49:25] [02mediawiki] 07dependabot[bot] labeled pull request 03#1566: Bump vendor from `327fc31` to `6070660` - 13https://git.io/J3o19 [14:49:27] [02mediawiki] 07dependabot[bot] commented on pull request 03#1550: Bump vendor from `3daaee4` to `8705df6` - 13https://git.io/J3o1Q [14:49:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/TwitterTag-e4c0e39 [+0/-0/±1] 13https://git.io/J3o17 [14:49:30] [02miraheze/mediawiki] 07dependabot[bot] 0338e8bb8 - Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` [14:49:31] [02mediawiki] 07dependabot[bot] opened pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/J3o15 [14:49:33] [02mediawiki] 07dependabot[bot] closed pull request 03#1550: Bump vendor from `3daaee4` to `8705df6` - 13https://git.io/J3RYJ [14:49:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/TwitterTag-e4c0e39 - 13https://git.io/vbL5b [14:49:36] [02mediawiki] 07dependabot[bot] labeled pull request 03#1567: Bump extensions/TwitterTag from `0c2cee6` to `e4c0e39` - 13https://git.io/J3o15 [14:49:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-8705df6 [14:49:39] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_36/vendor-8705df6 - 13https://git.io/vbL5b [14:49:40] [02mediawiki] 07dependabot[bot] opened pull request 03#1568: Bump extensions/Purge from `47b21bc` to `3fadfbd` - 13https://git.io/J3o1F [14:49:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Purge-3fadfbd [+0/-0/±1] 13https://git.io/J3o1b [14:49:43] [02miraheze/mediawiki] 07dependabot[bot] 0356f8af1 - Bump extensions/Purge from `47b21bc` to `3fadfbd` [14:49:45] [02mediawiki] 07dependabot[bot] labeled pull request 03#1568: Bump extensions/Purge from `47b21bc` to `3fadfbd` - 13https://git.io/J3o1F [14:49:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Purge-3fadfbd - 13https://git.io/vbL5b [14:49:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/GettingStarted-0a055cf [+0/-0/±1] 13https://git.io/J3o1N [14:49:49] [02miraheze/mediawiki] 07dependabot[bot] 03ba1df93 - Bump extensions/GettingStarted from `5ec1c75` to `0a055cf` [14:49:51] [02mediawiki] 07dependabot[bot] opened pull request 03#1569: Bump extensions/GettingStarted from `5ec1c75` to `0a055cf` - 13https://git.io/J3o1A [14:49:51] can we monitor multiple branches with dependabot? [14:49:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/GettingStarted-0a055cf - 13https://git.io/vbL5b [14:49:54] [02mediawiki] 07dependabot[bot] labeled pull request 03#1569: Bump extensions/GettingStarted from `5ec1c75` to `0a055cf` - 13https://git.io/J3o1A [14:49:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/skins/Gamepress-d8079f3 [+0/-0/±1] 13https://git.io/J3o1p [14:49:57] [02miraheze/mediawiki] 07dependabot[bot] 0303e93fb - Bump skins/Gamepress from `e01aa9a` to `d8079f3` [14:49:58] [02mediawiki] 07dependabot[bot] opened pull request 03#1570: Bump skins/Gamepress from `e01aa9a` to `d8079f3` - 13https://git.io/J3o1h [14:50:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/skins/Gamepress-d8079f3 - 13https://git.io/vbL5b [14:50:01] [02mediawiki] 07dependabot[bot] labeled pull request 03#1570: Bump skins/Gamepress from `e01aa9a` to `d8079f3` - 13https://git.io/J3o1h [14:50:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/VEForAll-3fd8b9b [+0/-0/±1] 13https://git.io/J3o1j [14:50:04] [02miraheze/mediawiki] 07dependabot[bot] 0392e671c - Bump extensions/VEForAll from `8c90b8d` to `3fd8b9b` [14:50:06] [02mediawiki] 07dependabot[bot] opened pull request 03#1571: Bump extensions/VEForAll from `8c90b8d` to `3fd8b9b` - 13https://git.io/J3oMe [14:50:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/VEForAll-3fd8b9b - 13https://git.io/vbL5b [14:50:09] [02mediawiki] 07dependabot[bot] labeled pull request 03#1571: Bump extensions/VEForAll from `8c90b8d` to `3fd8b9b` - 13https://git.io/J3oMe [14:50:10] [02mediawiki] 07dependabot[bot] commented on pull request 03#1382: Bump skins/Gamepress from `e01aa9a` to `3a323f0` - 13https://git.io/J3oMv [14:50:12] [02mediawiki] 07dependabot[bot] closed pull request 03#1382: Bump skins/Gamepress from `e01aa9a` to `3a323f0` - 13https://git.io/JYcN9 [14:50:13] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Gamepress-3a323f0 - 13https://git.io/vbL5b [14:50:15] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/skins/Gamepress-3a323f0 [14:50:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/LinkTitles-c1dca11 [+0/-0/±1] 13https://git.io/J3oMJ [14:50:18] [02miraheze/mediawiki] 07dependabot[bot] 03aad36e6 - Bump extensions/LinkTitles from `749211f` to `c1dca11` [14:50:19] [02mediawiki] 07dependabot[bot] opened pull request 03#1572: Bump extensions/LinkTitles from `749211f` to `c1dca11` - 13https://git.io/J3oMU [14:50:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/LinkTitles-c1dca11 - 13https://git.io/vbL5b [14:50:22] [02mediawiki] 07dependabot[bot] labeled pull request 03#1572: Bump extensions/LinkTitles from `749211f` to `c1dca11` - 13https://git.io/J3oMU [14:50:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Cargo-ef423f2 [+0/-0/±1] 13https://git.io/J3oMT [14:50:25] [02miraheze/mediawiki] 07dependabot[bot] 035b9e1b0 - Bump extensions/Cargo from `8fc3204` to `ef423f2` [14:50:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Cargo-ef423f2 - 13https://git.io/vbL5b [14:50:28] [02mediawiki] 07dependabot[bot] opened pull request 03#1573: Bump extensions/Cargo from `8fc3204` to `ef423f2` - 13https://git.io/J3oMI [14:50:30] [02mediawiki] 07dependabot[bot] commented on pull request 03#1406: Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` - 13https://git.io/J3oML [14:50:31] [02mediawiki] 07dependabot[bot] labeled pull request 03#1573: Bump extensions/Cargo from `8fc3204` to `ef423f2` - 13https://git.io/J3oMI [14:50:33] [02mediawiki] 07dependabot[bot] closed pull request 03#1406: Bump extensions/LinkTitles from `a77bf03` to `4a2d7c9` - 13https://git.io/JYSFx [14:50:34] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/LinkTitles-4a2d7c9 - 13https://git.io/vbL5b [14:50:36] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/LinkTitles-4a2d7c9 [14:50:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-4c0b45e [+0/-0/±1] 13https://git.io/J3oMq [14:50:39] [02miraheze/mediawiki] 07dependabot[bot] 032f22e48 - Bump extensions/ApprovedRevs from `5668e4e` to `4c0b45e` [14:50:40] [02mediawiki] 07dependabot[bot] commented on pull request 03#1410: Bump extensions/Cargo from `8fc3204` to `d31ffcc` - 13https://git.io/J3oMm [14:50:49] 16:49:52 <+SPF|Cloud> can we monitor multiple branches with dependabot? [14:52:28] paladox: https://github.com/miraheze/mediawiki/commit/d05e032515d5a8f08beda2073fe794ea7efe7c4c [14:52:28] [ Revert "Dependabot: Switch to REL1_36 (#1444)" · miraheze/mediawiki@d05e032 · GitHub ] - github.com [14:52:35] SPF|Cloud: not to my knowledge [14:52:47] But we chose to stop them when prepping 1.36 [14:52:59] ah [14:53:05] i mean you should really create a script that foreaches over all and pulls [14:53:25] paladox: we have a plan to replace it in the coming months [14:53:34] ok [14:53:52] can we keep dependabot on REL1_35 till a few days before REL1_36 release? [14:54:16] SPF|Cloud: 1.36 will be on test3 hopefully within a week or so [14:54:29] when REL1_36 is in production, we don't need to monitor REL1_35 obviously, but missing patches on REL1_35 bad is bad [14:54:58] paladox: people are getting MWExceptions apparently. I don't have access currently [14:55:07] The merge rate for Dependabot is low anyway [14:55:14] oh [14:55:21] Reception123: any exception id? [14:55:22] I asked for a copy/paste so we can have the ID [14:55:28] Original exception: [612859f8d88df22e92a58450] 2021-05-05 14:53:46: Fatal exception of type "MWException" [14:55:47] * SPF|Cloud looks in graylog [14:56:20] Because as I'm sure we're about to remember [14:56:24] It often goes wrong [14:56:26] SPF|Cloud: also exceptions haven't seemed to be logged since we upgraded to mw 1.35.2 [14:56:30] in the channel [14:56:39] https://graylog.miraheze.org/streams/5fc826549b05260901034cc2/search?q=&rangetype=relative&streams=5fc826549b05260901034cc2&relative=300 is empty [14:56:41] all exceptions are related to PageForms [14:56:55] please revert the PageForms change [14:56:56] oh [14:56:59] found it [14:57:02] i'll run lc [14:57:18] that'll take a while, though? [14:57:45] or can it be sped up? (for instance, using the multiple threads option in the rebuild script) [14:58:11] --threads 8? [14:58:11] Oh that's an interesting idea, the last time there would have been donwtime I had to depool mw* servers one at a time, which wasn't great for load [14:58:28] --threads 2 is enough [14:59:00] !log Run this on all mediawiki application servers : sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --threads 2 --wiki loginwiki [14:59:04] with our core overselling, more threads will cause higher load rather than making the rebuild faster [15:00:07] hmm https://graylog.miraheze.org/messages/graylog_171/1c468661-adb2-11eb-b741-0200001a24a4 [15:00:36] i don't see which wikis that for [15:01:16] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?viewPanel=281&orgId=1&var-job=node&var-node=mw8.miraheze.org&var-port=9100 [15:01:17] [ Grafana ] - grafana.miraheze.org [15:01:22] reverting back to not using --threads [15:01:27] using that caused downtime [15:01:38] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 46.64, 33.13, 19.42 [15:01:39] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 19.43, 11.10, 6.58 [15:02:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 21.80, 13.09, 7.26 [15:02:31] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 27.22, 15.36, 8.19 [15:02:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb [15:03:08] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 107.84, 69.13, 36.01 [15:03:09] 27.22? [15:03:22] paladox: why are you running the script with 8 threads? [15:03:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:178a::5/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb [15:03:36] SPF|Cloud: can you help me to cancel all salts? [15:03:38] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.70, 11.23, 7.42 [15:03:39] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.65, 2.79, 1.55 [15:03:42] ctrl-c doesn't stop salt [15:03:50] PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.82, 7.54, 3.62 [15:04:09] ps aux | grep salt-user [15:04:28] PROBLEM - mw10 Check Gluster Clients on mw10 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [15:04:37] PROBLEM - mw8 Check Gluster Clients on mw8 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [15:04:52] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:04:58] PROBLEM - mw9 Check Gluster Clients on mw9 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [15:05:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:05:26] I think you've already fixed it? [15:05:40] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.50, 2.38, 1.56 [15:05:49] PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 1.65, 5.34, 3.29 [15:06:26] 16:59:05 <@SPF|Cloud> with our core overselling, more threads will cause higher load rather than making the rebuild faster [15:07:16] PROBLEM - wiki.campaign-labour.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.campaign-labour.org could not be found [15:07:36] !log run this on all mediawiki application servers: ps aux | grep salt-user | awk '{print $2}' | xargs kill -9 [15:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:07:40] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.38, 7.58, 6.80 [15:07:44] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.09, 22.87, 19.74 [15:07:47] RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 0.95, 3.89, 3.00 [15:08:00] paladox: should I do the LC rebuild? [15:08:17] I tried and it says: [15:08:19] https://www.irccloud.com/pastebin/8qbKRSBT/ [15:08:19] [ Snippet | IRCCloud ] - www.irccloud.com [15:08:50] well, it does look like the exceptions have stopped [15:09:00] since :03 [15:09:39] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 12.81, 19.43, 18.84 [15:09:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.44, 7.11, 6.72 [15:11:01] SPF|Cloud: do you know how to fix https://graylog.miraheze.org/streams/5fc826549b05260901034cc2/search?q=&rangetype=relative&streams=5fc826549b05260901034cc2&relative=300 ? [15:11:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.33, 6.63, 6.59 [15:11:39] I see no messages [15:11:53] yeh [15:11:53] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:11:56] and you should restart glusterfs on mw8, mw9 and mw10 [15:12:04] hasn't since 2021-04-08 23:36:23.000 +01:00 [15:12:26] PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [15:12:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.82, 7.20, 7.81 [15:12:45] RECOVERY - mw9 Check Gluster Clients on mw9 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [15:12:52] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:13:14] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:13:16] Done [15:13:19] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],File[/mnt/mediawiki-static] [15:13:30] !log remounted gluster on mw 8,9,10,11 & jobrunner[12] [15:13:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:13:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.88, 6.50, 6.59 [15:13:51] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],File[/mnt/mediawiki-static] [15:14:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.78, 7.07, 7.78 [15:14:13] RECOVERY - mw8 Check Gluster Clients on mw8 is OK: PROCS OK: 2 processes with args '/usr/sbin/glusterfs' [15:14:19] RECOVERY - mw10 Check Gluster Clients on mw10 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [15:14:22] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core],File[/mnt/mediawiki-static] [15:17:27] paladox: 'MediaWiki Exception Logs' is empty due to the rule "mediawiki_exception_class must match regular expression .+" [15:17:54] yeh but we've had that before and it worked since we upgraded to 1.35.2 [15:18:00] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.56, 4.96, 6.74 [15:18:31] *hasn't work [15:18:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.61, 4.76, 6.50 [15:19:29] then you have to change it :) [15:20:46] I don't understand the rules 'mediawiki_channel must match ' and 'mediawiki_exception_class must match '. Why not just 'mediawiki_channel is equal to "exception"'? [15:21:36] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.04, 2.98, 1.94 [15:21:48] i don't think a exception channel exists? At least i have the rule like "mediawiki_channel must match regular expression ^(?!.*(exec|thumbnail))" [15:22:00] mediawiki_channel:"exception" works fine [15:22:05] ohhh [15:23:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.06, 19.62, 23.91 [15:23:21] fixed, thanks! [15:23:32] you're welcome [15:23:35] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.05, 2.24, 1.79 [15:24:16] hmm [15:24:24] SPF|Cloud: it doesn't tell us which wiki it is anymore [15:24:34] is the 'wiki' processor enabled? [15:25:33] yeh https://github.com/miraheze/mw-config/blob/master/GlobalLogging.php#L15 [15:25:34] [ mw-config/GlobalLogging.php at master · miraheze/mw-config · GitHub ] - github.com [15:28:24] https://graylog.miraheze.org/streams/5f8c6fd446640840f104b0ba/search?q=mediawiki_wiki%3Ametawiki&rangetype=relative&streams=5f8c6fd446640840f104b0ba&relative=2592000 [15:28:34] shows mediawiki_wiki stopped working when we upgraded to 1.35.2 [15:29:56] * Upgrade monolog/monolog from 1.25.3 to 2.2.0 for PHP 8.0 support. [15:30:18] https://github.com/wikimedia/mediawiki/commit/d039e8158356b43908393869eee09782a6eefc02#diff-5504ab39112a02473258a603edc2836c2ec04b4d3df15fc8e2090194d41d410a [15:30:19] [ Updating monolog/monolog (1.25.3 => 2.0.2) · wikimedia/mediawiki@d039e81 · GitHub ] - github.com [15:31:29] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/logging.php#L87 class names are still the same [15:31:30] [ operations-mediawiki-config/logging.php at master · wikimedia/operations-mediawiki-config · GitHub ] - github.com [15:32:49] let's test on test3 [15:32:57] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.94, 17.16, 20.37 [15:33:49] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/logging.php#L145 would be helpful [15:33:50] [ operations-mediawiki-config/logging.php at master · wikimedia/operations-mediawiki-config · GitHub ] - github.com [15:33:53] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [15:34:26] maybe, create a task for it [15:34:51] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [15:35:05] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:35:20] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:35:49] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:20] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:40:57] mediawiki on test3 doesn't log to syslog? [15:41:10] nope [15:42:45] noticed that mediawiki_channel:error hasn't worked since the upgrade either [15:42:47] where has that been configured? wmgLogToDisk is false [15:48:32] i was checking when mediawiki_wiki last worked and i saw "error" for "mediawiki_class" so i thought i'll try it now [15:48:34] didn't work [15:50:30] why doesn't test3 log to syslog? [15:50:35] where is that configured? [15:54:08] because we log to syslog-ng? [15:54:27] SPF|Cloud: ^ [15:54:29] logging to syslog-ng = logging to syslog [15:54:48] let me rephrase the question: why do nginx access log end up in graylog, whereas mediawiki logs don't? [15:58:53] the log configuration looks the same to me, therefore I don't know why mediawiki logs for test3 cannot be found in graylog [16:00:53] nevermind... the logs are in graylog [16:02:13] SPF|Cloud: i can see test3 in graylog [16:02:16] yeh [16:11:14] dmehus why did you add a horribletvshowepisodes interwiki to terribletvshowswiki? It will be redundant anyway since horribletvshowepisodeswiki will be merged there anyway [16:19:45] I had to go [16:20:03] Brb [16:26:20] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J3Kqe [16:26:22] [02miraheze/mw-config] 07Southparkfan 031f99e83 - Fix configuration for Logstash formatter [16:26:24] [02mw-config] 07Southparkfan synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [16:27:23] miraheze/mw-config - Southparkfan the build passed. [16:28:45] paladox ^ [16:35:20] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.42, 6.23, 5.11 [16:37:19] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.22, 5.79, 5.09 [16:50:56] Oh nice! [16:51:05] Thanks SPF|Cloud [17:16:45] [02mediawiki] 07R4356th commented on pull request 03#1527: Update DarkMode - 13https://git.io/J3KWw [17:18:01] [02mediawiki] 07R4356th commented on pull request 03#1527: Update DarkMode - 13https://git.io/J3KWH [17:18:02] [02mediawiki] 07R4356th closed pull request 03#1527: Update DarkMode - 13https://git.io/J3UoM [17:35:36] [02mediawiki] 07R4356th opened pull request 03#1603: Install CreatedPagesList (T7236) - 13https://git.io/J3KRu [17:44:37] [02mw-config] 07R4356th opened pull request 03#3872: Add CreatedPagesList (T7236) - 13https://git.io/J3KEl [17:45:44] miraheze/mw-config - R4356th the build passed. [17:49:11] [02mediawiki] 07R4356th edited pull request 03#1603: Install CreatedPagesList (T7236) - 13https://git.io/J3KRu [17:50:13] [02mw-config] 07thelevking opened pull request 03#3873: Test change in FlaggedRevs config for isvwiki - 13https://git.io/J3KuV [17:51:14] miraheze/mw-config - thelevking the build passed. [17:54:37] [02mw-config] 07R4356th reviewed pull request 03#3873 commit - 13https://git.io/J3KzM [17:55:03] [02mw-config] 07R4356th reviewed pull request 03#3873 commit - 13https://git.io/J3KzM [17:59:46] [02mw-config] 07thelevking reviewed pull request 03#3873 commit - 13https://git.io/J3Kgb [18:02:54] @R4356th: see 1.36 branch for replacement config [18:03:43] RhinosF1: 1.36 branch of ? [18:03:55] Config repo [18:04:30] And I should be looking at that for? [18:06:53] RhinosF1: ^ [18:07:08] #3873 [18:07:33] Oh, got it. [18:08:04] @Lev ^ [18:10:34] Do I need to do anything now? [18:12:05] [02mw-config] 07R4356th synchronize pull request 03#3872: Add CreatedPagesList (T7236) - 13https://git.io/J3KEl [18:12:19] It probably works without those values on MediaWiki 1.36, but does not on 1.35, right? [18:13:06] miraheze/mw-config - R4356th the build passed. [18:13:13] I am not too fussed about them, I just trying to fix autoreview process breaking specifically on isvwiki and nowhere else. [18:13:26] I'd just wait until 1.36 [18:13:41] It's at most probably 8 weeks [18:14:12] And debugging flagged revs is painful [18:14:26] [02mw-config] 07R4356th edited pull request 03#3872: Add CreatedPagesList (T7236) - 13https://git.io/J3KEl [18:14:32] But if it would fix it, then all the better now, no? [18:14:57] [02mw-config] 07R4356th edited pull request 03#3872: Add CreatedPagesList (T7236) - 13https://git.io/J3KEl [18:20:49] FYI, .gitmodules is missing ignore = dirty for some extensions. [18:22:36] [02mediawiki] 07R4356th opened pull request 03#1604: Add "ignore = dirty" where missing - 13https://git.io/J3KoU [18:26:39] [02mw-config] 07R4356th synchronize pull request 03#3858: Add wgScratchBlocks4BlockVersion to ManageWiki - 13https://git.io/J3UPK [18:27:20] [02mw-config] 07R4356th reviewed pull request 03#3858 commit - 13https://git.io/J3KKe [18:27:36] miraheze/mw-config - R4356th the build passed. [18:29:44] [02mw-config] 07R4356th commented on pull request 03#3858: Add wgScratchBlocks4BlockVersion to ManageWiki - 13https://git.io/J3KKB [20:20:50] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [20:21:34] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 4.71, 3.74 [20:23:30] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.57, 4.64, 3.83 [20:27:42] RECOVERY - wiki.etn.link - reverse DNS on sslhost is OK: rDNS OK - wiki.etn.link reverse DNS resolves to cp10.miraheze.org