[00:02:30] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown) [00:32:26] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:15:27] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 207s [02:21:28] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 193s [02:27:26] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 247s [02:31:27] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 177s [02:33:28] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [05:15:16] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason field - 13https://git.io/J36M3 [05:15:44] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason field - 13https://git.io/J36M3 [05:16:09] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason field - 13https://git.io/J36M3 [05:16:17] miraheze/CreateWiki - Universal-Omega the build passed. [05:16:44] miraheze/CreateWiki - Universal-Omega the build passed. [05:17:09] miraheze/CreateWiki - Universal-Omega the build passed. [05:21:27] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason field - 13https://git.io/J36M3 [05:21:46] [02CreateWiki] 07Universal-Omega edited pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [05:22:24] miraheze/CreateWiki - Universal-Omega the build passed. [05:26:06] [02CreateWiki] 07Universal-Omega synchronize pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [05:27:04] miraheze/CreateWiki - Universal-Omega the build passed. [05:29:43] [02CreateWiki] 07Universal-Omega edited pull request 03#215: Use textarea type for reason and comment fields - 13https://git.io/J36M3 [05:31:27] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki contraaowiki /home/reception/delbackups2/contraaowiki.xml --username-prefix=contraao [05:31:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:12:37] !log rename trocwiki to cnvnwiki [06:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:37] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/J3HgW [06:15:38] [02miraheze/dns] 07Reception123 03a273f83 - add wlhmn.tk zone [06:24:11] [02puppet] 07Reception123 commented on pull request 03#1752: make requestwiki AI job memory 1750M - 13https://git.io/J3Ha2 [06:45:06] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/J3HrR [06:45:07] [02miraheze/ssl] 07Reception123 0348d7891 - add wlhmn.tk cert [08:15:33] [02puppet] 07JohnFLewis commented on pull request 03#1752: make requestwiki AI job memory 1750M - 13https://git.io/J3H78 [08:24:55] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 109.03 ms [08:30:54] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 74.04 ms [08:36:14] [02puppet] 07Reception123 commented on pull request 03#1752: make requestwiki AI job memory 1750M - 13https://git.io/J3HNJ [08:38:55] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.86 ms [08:40:54] PROBLEM - ping6 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.37 ms [08:42:54] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 202.41 ms [08:43:54] !log [reception@jobrunner3] sudo service jobrunner disable [08:43:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:44:53] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 73.84 ms [08:46:16] !log started jobrunner on jobrunner3 [08:46:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:46:19] [02puppet] 07Reception123 commented on pull request 03#1752: make requestwiki AI job memory 1750M - 13https://git.io/J3HAQ [08:46:22] [02puppet] 07Reception123 closed pull request 03#1752: make requestwiki AI job memory 1750M - 13https://git.io/J3y70 [08:46:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J3HA7 [08:46:25] [02miraheze/puppet] 07Reception123 03e20ae55 - make requestwiki AI job memory 1750M (#1752) [08:46:27] [02puppet] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbiAS [08:46:28] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-2 [08:48:59] PROBLEM - ping6 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 193.22 ms [08:50:59] RECOVERY - ping6 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 78.10 ms [13:18:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 202s [13:20:23] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 45s [13:30:25] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 241s [13:34:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [14:10:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 218s [14:12:26] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.16, 3.25, 2.12 [14:14:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [14:14:23] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.78, 2.73, 2.07 [14:34:03] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:35:59] PROBLEM - dbbackup1 Check MariaDB Replication c4 on dbbackup1 is UNKNOWN: NRPE: Unable to read output [14:36:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.57, 2.97, 2.18 [14:36:29] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 250s [14:38:08] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.87, 2.90, 2.25 [14:42:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 199s [14:44:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [15:35:55] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.91, 6.57, 3.73 [15:39:49] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.39, 3.31, 3.03 [15:45:47] [02mw-config] 07R4356th commented on pull request 03#3877: Add Wikidata as an import source for snapdatawiki - 13https://git.io/J37Ca [15:47:33] [02mediawiki] 07R4356th commented on pull request 03#1544: Bump extensions/mw-ScratchBlocks4 from `74ca1cd` to `b71d5cd` - 13https://git.io/J37CA [15:48:25] [02mediawiki] 07R4356th commented on pull request 03#1560: Bump extensions/DarkMode from `73d8847` to `a4b2ada` - 13https://git.io/J37WJ [16:33:03] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 12.49, 11.10, 5.28 [16:34:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.38, 20.50, 15.40 [16:36:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 8.70, 15.85, 14.31 [16:40:51] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.45, 3.53, 3.99 [16:44:47] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.31, 3.64, 3.86 [16:48:42] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.12, 3.34, 3.82 [16:52:39] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.23, 1.97, 3.13 [17:01:24] SPF|Cloud: Domain names [17:49:21] [02mw-config] 07Njsmooth opened pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J379h [17:52:42] miraheze/mw-config - Reception123 the build passed. [17:53:14] [02mw-config] 07Njsmooth closed pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J379h [17:54:35] [02mw-config] 07Reception123 commented on pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J37QJ [17:54:45] [02mw-config] 07Reception123 edited a comment on pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J37QJ [18:02:09] !log renewed miraheze & orain domain names [18:02:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 189s [18:12:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 280s [18:13:11] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.22, 3.13, 1.89 [18:14:23] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 61s [18:15:10] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 1.87, 2.65, 1.87 [18:30:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 206s [18:32:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 66s [18:52:05] [02mw-config] 07Njsmooth reopened pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J379h [18:52:23] [02mw-config] 07Njsmooth commented on pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J37jL [19:21:48] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.99, 1.60, 1.10 [19:23:42] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.89, 1.29, 1.04 [19:34:42] [02mw-config] 07Reception123 closed pull request 03#3879: Configured "HighlightLinksInCategory" Extension for my wiki - 13https://git.io/J379h [19:34:43] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J35Lm [19:34:45] [02miraheze/mw-config] 07Njsmooth 03d1cf29c - Configured "HighlightLinksInCategory" Extension for my wiki (#3879) [19:34:46] [02mw-config] 07Reception123 synchronize pull request 03#3845: Merge 'master' into REL1_36 - 13https://git.io/JOKA0 [19:35:43] miraheze/mw-config - Reception123 the build passed. [20:18:08] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.76, 6.43, 3.35 [20:18:49] PROBLEM - wiki.worldsofweary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.worldsofweary.com could not be found [20:18:55] PROBLEM - en.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for en.gyaanipedia.com could not be found [20:22:02] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.77, 3.28, 2.74 [20:25:42] RECOVERY - en.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - en.gyaanipedia.com reverse DNS resolves to cp10.miraheze.org [20:25:49] RECOVERY - wiki.worldsofweary.com - reverse DNS on sslhost is OK: rDNS OK - wiki.worldsofweary.com reverse DNS resolves to cp10.miraheze.org [21:04:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 280s [21:06:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 189s [21:08:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 2s [21:36:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 232s [21:37:33] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.78, 3.01, 1.98 [21:38:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 90s [21:39:31] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.23, 2.75, 2.02 [22:02:24] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 267s [22:03:16] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.60, 3.66, 2.41 [22:05:16] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 3.19, 3.40, 2.47 [22:10:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 115s [22:12:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 1s [22:20:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 189s [22:22:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 259s [22:23:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.73, 3.28, 2.70 [22:24:21] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 53s [22:25:10] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.95, 3.05, 2.68 [22:28:23] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 247s [22:41:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 3.30, 3.62, 3.29 [22:45:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is CRITICAL: CRITICAL - load average: 4.09, 3.62, 3.34 [22:47:10] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.78, 3.45, 3.32 [22:49:10] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.27, 3.00, 3.17 [22:57:12] PROBLEM - dbbackup2 Current Load on dbbackup2 is WARNING: WARNING - load average: 2.87, 3.47, 3.32 [22:59:10] RECOVERY - dbbackup2 Current Load on dbbackup2 is OK: OK - load average: 2.05, 2.98, 3.16 [23:00:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [23:22:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 231s [23:24:21] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 141s [23:26:22] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 230s [23:40:22] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s