[00:05:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.82, 5.88, 5.03 [00:07:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.15, 5.99, 5.17 [00:11:56] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.03, 1.98, 1.44 [00:13:53] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.84, 1.56, 1.34 [00:24:31] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYvLu [00:24:32] [02miraheze/IncidentReporting] 07Universal-Omega 031097a0b - Comment statistics until it works [00:24:34] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [00:24:35] [02IncidentReporting] 07Universal-Omega opened pull request 03#18: Comment statistics until it works - 13https://git.io/JYvLg [00:25:39] miraheze/IncidentReporting - Universal-Omega the build passed. [00:25:55] [02IncidentReporting] 07Universal-Omega closed pull request 03#18: Comment statistics until it works - 13https://git.io/JYvLg [00:25:56] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYvLD [00:25:58] [02miraheze/IncidentReporting] 07Universal-Omega 030edd105 - Comment statistics until it works (#18) [00:26:00] [02IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fh5YJ [00:26:01] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:26:49] miraheze/IncidentReporting - Universal-Omega the build passed. [01:06:02] oh, test3 is down [01:21:35] yeah [01:23:32] RECOVERY - cp10 Stunnel Http for test3 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 16884 bytes in 0.207 second response time [01:23:34] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 16875 bytes in 0.998 second response time [01:23:50] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 22402 bytes in 0.032 second response time [01:23:55] RECOVERY - cp12 Stunnel Http for test3 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 16876 bytes in 0.316 second response time [01:23:59] RECOVERY - test3 HTTPS on test3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2221 bytes in 0.183 second response time [01:24:14] RECOVERY - cp11 Stunnel Http for test3 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 16884 bytes in 0.046 second response time [01:32:05] PROBLEM - cp11 Stunnel Http for test3 on cp11 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Cannot find server. - 595 bytes in 0.202 second response time [01:33:26] PROBLEM - cp10 Stunnel Http for test3 on cp10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Cannot find server. - 595 bytes in 0.140 second response time [01:33:29] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Cannot find server. - 595 bytes in 0.962 second response time [01:33:44] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Cannot find server. - 595 bytes in 0.050 second response time [01:34:02] PROBLEM - cp12 Stunnel Http for test3 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Cannot find server. - 595 bytes in 0.447 second response time [01:51:27] RECOVERY - cp10 Stunnel Http for test3 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15214 bytes in 0.009 second response time [01:51:29] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15228 bytes in 1.044 second response time [01:51:41] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20757 bytes in 0.253 second response time [01:52:03] RECOVERY - cp12 Stunnel Http for test3 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15214 bytes in 0.311 second response time [01:52:03] RECOVERY - cp11 Stunnel Http for test3 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15214 bytes in 0.012 second response time [02:02:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 318s [02:16:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 170s [02:18:37] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 60s [02:35:38] PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [02:37:41] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: connect to address 51.195.236.247 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:37:47] PROBLEM - test3 HTTPS on test3 is CRITICAL: connect to address 51.195.236.247 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [02:38:12] PROBLEM - cp11 Stunnel Http for test3 on cp11 is CRITICAL: HTTP CRITICAL - No data received from host [02:38:14] PROBLEM - cp12 Stunnel Http for test3 on cp12 is CRITICAL: HTTP CRITICAL - No data received from host [02:39:26] PROBLEM - cp10 Stunnel Http for test3 on cp10 is CRITICAL: HTTP CRITICAL - No data received from host [02:39:27] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [03:03:27] RECOVERY - cp10 Stunnel Http for test3 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15210 bytes in 0.008 second response time [03:03:29] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15202 bytes in 1.004 second response time [03:03:41] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20745 bytes in 0.395 second response time [03:03:47] RECOVERY - test3 HTTPS on test3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 561 bytes in 0.009 second response time [03:04:02] RECOVERY - cp11 Stunnel Http for test3 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15216 bytes in 0.008 second response time [03:04:22] RECOVERY - cp12 Stunnel Http for test3 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15202 bytes in 0.315 second response time [03:34:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 259s [03:36:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 183s [03:38:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 225s [03:40:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 127s [03:46:37] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [03:58:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 244s [04:16:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 132s [04:18:37] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [05:00:00] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.78, 1.47, 1.19 [05:01:56] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.75, 1.20, 1.13 [05:54:37] PROBLEM - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 236s [06:22:37] RECOVERY - dbbackup2 Check MariaDB Replication c3 on dbbackup2 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 60s [06:30:52] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYfTH [06:30:54] [02miraheze/dns] 07Reception123 03f41af5b - add "incubator" to nocyclo.tk zone [06:34:11] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JYfk3 [06:34:12] [02miraheze/ssl] 07Reception123 03fde3884 - add bn.gyaanipedia.com cert [06:34:14] [02miraheze/ssl] 07Reception123 03bf46655 - add nocyclo.tk cert [06:34:15] [02miraheze/ssl] 07Reception123 031e8e5de - add incubator.nocyclo.tk cert [06:37:26] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=trollpastawiki --r "[[phab:T7019|Requested]]" /home/reception/trollpastadel.txt [06:37:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.91, 7.31, 5.97 [06:45:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.91, 6.85, 5.98 [06:47:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.31, 6.50, 5.97 [07:14:39] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.037 second response time [07:14:59] Reception123: ^ [07:15:11] RhinosF1: I know, doing the reclone [07:20:34] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 1260 bytes in 0.465 second response time [07:21:44] RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:24:27] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2502 bytes in 0.009 second response time [07:26:13] ^ RhinosF1 urgh, what's up with that? [07:26:26] no puppet failures either [07:26:50] Eh [07:27:14] .dns test3.miraheze.org [07:27:15] RhinosF1: 51.195.236.247 [07:27:27] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.43 ms [07:27:27] .dns 51.195.236.247 PTR [07:27:28] RhinosF1: test3.miraheze.org. [07:27:33] Why the fuck [07:27:42] Reception123: I'm very confused [07:27:52] me too, don't know where a 404 could come from [07:28:04] I just deleted w and ran puppet [07:31:31] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 156.36 ms [07:33:34] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 162.52 ms [07:37:37] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.73 ms [07:38:38] Reception123: ManageWiki cache? [07:38:59] definitely seems to think it doesn't exist: The wiki database 'test3wiki' was not found. [07:39:26] and because of that no maint script could be ran either, even eval.php [07:39:41] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 148.20 ms [07:39:53] /srv/mediawiki/w/cache is empty, probably not a great sign [07:39:54] Get databases.json from another server [07:40:52] Reception123: ^ [07:40:54] yeah guess we can try the hack-y way [07:41:42] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 146.89 ms [07:44:02] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 324 bytes in 0.096 second response time [07:46:35] Reception123: eh eh eh [07:47:35] will it never be happy [07:48:04] paladox: ? [07:49:47] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 147.93 ms [07:50:18] PHP Notice: Undefined index: test3wiki in /srv/mediawiki/w/extensions/CreateWiki/includes/WikiInitialise.php on line 108 [07:51:27] Reception123: https://github.com/miraheze/CreateWiki/blob/master/maintenance/DBListGenerator.php [07:51:28] [ CreateWiki/DBListGenerator.php at master · miraheze/CreateWiki · GitHub ] - github.com [07:52:18] RhinosF1: that's great and all but we can't run maint scripts [07:52:33] https://www.irccloud.com/pastebin/mc0gR2MR/ [07:52:34] [ Snippet | IRCCloud ] - www.irccloud.com [07:52:57] This is messed up [07:53:30] It's test3 so let's wait for help [07:53:52] PROBLEM - ping4 on bacula2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 144.87 ms [07:53:59] And maybe in future we make mw cache in its own folder [07:54:07] yeah, though I can't really fix puppet on the other servers either as if this happens we'll be lost [07:54:12] Like /srv/MediaWiki/cache [07:54:43] Reception123: wait for help, I suggest when we can though we move MW cache [07:55:01] Because this a catch 22 [07:55:29] yeah, though one shouldn't be deleting /srv/mediawiki/w on a regular basis either [07:55:58] True [07:56:05] But things happen [07:56:20] And I'm super confused here why cache isn't auto populating [07:57:05] yeah [08:00:44] RhinosF1: what's even weirder is that stuff is missing from vendor [08:00:47] like monolog [08:00:57] https://www.irccloud.com/pastebin/TyDcIMoB/ [08:00:57] [ Snippet | IRCCloud ] - www.irccloud.com [08:01:03] How the hell did you manage that [08:01:07] https://www.irccloud.com/pastebin/Fagt1MLu/ [08:01:07] [ Snippet | IRCCloud ] - www.irccloud.com [08:01:17] RhinosF1: I really don't know, all I did is remove w and run puppet (twice to make sure) [08:01:19] Puppet should do that stuff [08:03:59] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 76.82 ms [08:10:18] PROBLEM - ping4 on bacula2 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 192.62 ms [08:29:08] RECOVERY - ping4 on bacula2 is OK: PING OK - Packet loss = 0%, RTA = 78.81 ms [09:14:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.16, 4.73, 2.79 [09:15:15] PROBLEM - test3 MediaWiki Rendering on test3 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 2502 bytes in 0.010 second response time [09:16:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.49, 3.24, 2.48 [09:17:16] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.153 second response time [09:18:36] fixed [09:19:17] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20745 bytes in 0.482 second response time [09:23:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.46, 6.49, 5.42 [09:25:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.13, 6.32, 5.48 [09:30:47] paladox: how? [09:31:01] running composer install [09:31:08] in w [09:53:54] That's how it's done Reception123 [09:55:56] If only we had docs for the MW stack ;) [09:56:54] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.11, 3.98, 2.43 [09:58:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.78, 3.03, 2.26 [09:59:48] paladox: should puppet do that though? [09:59:54] and heh, the docs will come eventually :) [10:00:21] paladox: also if you're around I can't seem to be able to fix puppet not running on mw*/jbr*, there's a merge conflict, git reset --hard doesn't work [10:00:33] and it would save time if I didn't have to remove w on all servers to fix it [10:00:52] sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update [10:01:49] paladox: thanks, let me try that :) [10:04:06] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:04:23] yay, it worked, thanks :) [10:04:31] !log running sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update on mw*/jbr* [10:04:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:06:34] RECOVERY - jobrunner4 Puppet on jobrunner4 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [10:07:05] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [10:07:14] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [10:08:20] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:08:28] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYfxL [10:10:12] [02miraheze/services] 07MirahezeSSLBot 036b6205d - BOT: Updating services config for wikis [10:11:32] !log renamed basebuilderwiki to vassalswiki [10:11:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:19:22] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 5.18, 7.60, 4.31 [10:23:11] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.56, 3.82, 3.47 [10:25:07] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.57, 2.76, 3.12 [10:52:27] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki gyaanipediawiki /home/reception/GBU_XML/GBU[1-5].xml --username-prefix="mh:bluepages" [10:52:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:55:17] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JYJk3 [10:55:19] [02miraheze/mediawiki] 07Reception123 03184f947 - Install PdfBook ext. per T6996 [10:55:59] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYJkl [10:56:01] [02miraheze/mediawiki] 07Reception123 03afd1b13 - ignore dirty PdfBook [11:00:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±4] 13https://git.io/JYJIq [11:00:01] [02miraheze/mw-config] 07Reception123 034e6d812 - add PdfBook per T6996 [11:00:03] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [11:00:07] [02mw-config] 07Reception123 opened pull request 03#3803: add PdfBook per T6996 - 13https://git.io/JYJIm [11:01:17] miraheze/mw-config - Reception123 the build passed. [11:02:54] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 3.06, 3.78, 2.51 [11:04:13] paladox: still around? [11:04:19] yes [11:04:33] paladox: not sure why https://graylog.miraheze.org/messages/graylog_129/4a6340d0-8c90-11eb-b6dc-0200001a24a4 is happening [11:04:39] seems to be affecting all wikis with Comments on [11:04:56] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.13, 3.32, 2.50 [11:05:15] oh, must be because puppet wasn't working for a few days so it must've been one of those mediawiki updates [11:05:26] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03revert-1311-dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 [+0/-0/±1] 13https://git.io/JYJLO [11:05:28] [02miraheze/mediawiki] 07Reception123 035f1fb0e - Revert "Bump extensions/Comments from `d75fa77` to `c70f7d5` (#1311)" [11:05:29] [02mediawiki] 07Reception123 created branch 03revert-1311-dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 - 13https://git.io/vbL5b [11:05:31] [02mediawiki] 07Reception123 opened pull request 03#1376: Revert "Bump extensions/Comments from `d75fa77` to `c70f7d5`" - 13https://git.io/JYJL3 [11:05:38] [02mediawiki] 07Reception123 closed pull request 03#1376: Revert "Bump extensions/Comments from `d75fa77` to `c70f7d5`" - 13https://git.io/JYJL3 [11:05:39] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYJLs [11:05:41] [02miraheze/mediawiki] 07Reception123 03a742643 - Revert "Bump extensions/Comments from `d75fa77` to `c70f7d5` (#1311)" (#1376) [11:06:21] That'll be because of https://github.com/wikimedia/mediawiki-extensions-Comments/commit/a1a0f61757a58a0a6dd22d9347128a2754eb56ba [11:06:22] [ Avoid using LogEntry::getPerformer in favor of getPerformerIdentity · wikimedia/mediawiki-extensions-Comments@a1a0f61 · GitHub ] - github.com [11:06:55] yeah, sorry for bothering, I didn't think about the fact that puppet was not working on the 21st either so that change did go through [11:07:13] [02mediawiki] 07Reception123 commented on pull request 03#1376: Revert "Bump extensions/Comments from `d75fa77` to `c70f7d5`" - 13https://git.io/JYJLg [11:07:17] [02miraheze/mediawiki] 07Reception123 deleted branch 03revert-1311-dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 [11:07:19] [02mediawiki] 07Reception123 deleted branch 03revert-1311-dependabot/submodules/REL1_35/extensions/Comments-c70f7d5 - 13https://git.io/vbL5b [11:08:33] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 7.45, 6.25, 2.84 [11:10:02] Dependabot strikes again [11:12:34] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 1.00, 3.42, 2.46 [11:14:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.00, 5.84 [11:14:34] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.69, 2.51, 2.24 [11:16:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.19, 7.30, 6.08 [11:18:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.28, 6.88, 6.08 [11:20:21] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.22, 5.93 [11:21:28] [02mw-config] 07Reception123 closed pull request 03#3803: add PdfBook per T6996 - 13https://git.io/JYJIm [11:21:29] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JYJmt [11:21:31] [02miraheze/mw-config] 07Reception123 0356bb012 - add PdfBook per T6996 (#3803) [11:21:32] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [11:21:34] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [11:22:47] miraheze/mw-config - Reception123 the build passed. [11:24:34] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 3.45, 4.99, 3.29 [11:26:57] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki' on mw*/jbr* [11:27:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:14] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr* [11:27:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:28:33] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 2.01, 3.12, 2.90 [11:29:30] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+1/-0/±1] 13https://git.io/JYJY9 [11:29:32] [02miraheze/mediawiki] 07Reception123 03ff31b90 - Install ShortDescription ext. T7004 [11:30:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.88, 5.98, 5.08 [11:32:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.79, 6.57, 5.96 [11:32:23] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYJO0 [11:32:25] [02miraheze/mediawiki] 07Reception123 0386da49e - ignore dirty ShortDescription why do I keep forgetting... [11:32:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.12, 6.53, 5.37 [11:33:23] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.01, 7.57, 5.89 [11:33:28] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.16, 7.04, 5.89 [11:34:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.02, 8.17, 6.62 [11:36:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.67, 21.17, 17.03 [11:36:41] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.14, 18.95, 15.99 [11:38:09] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.79, 24.07, 18.61 [11:38:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.54, 6.92, 6.07 [11:38:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.33, 19.14, 16.43 [11:38:37] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±4] 13https://git.io/JYJ3h [11:38:38] [02miraheze/mw-config] 07Reception123 0315889eb - add ShortDescription per T7004 [11:38:40] [02mw-config] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [11:39:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.91, 7.21, 6.38 [11:39:26] [02mw-config] 07Reception123 opened pull request 03#3804: add ShortDescription per T7004 - 13https://git.io/JYJsm [11:40:06] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.42, 22.41, 18.62 [11:40:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.55, 7.78, 6.48 [11:40:39] miraheze/mw-config - Reception123 the build passed. [11:41:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.81, 7.87, 6.73 [11:42:49] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [11:43:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.81, 7.44, 6.90 [11:44:20] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.28, 7.97, 7.50 [11:44:25] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [11:44:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.73, 7.70, 6.80 [11:45:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.90, 7.83, 6.97 [11:45:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.19, 20.21, 18.79 [11:46:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.15, 8.18, 7.64 [11:47:23] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.83, 7.93, 7.09 [11:48:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.81, 7.58 [11:48:26] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [11:48:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.86, 6.34, 6.45 [11:48:47] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:49:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.74, 6.26, 6.63 [11:49:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.15, 7.14, 6.90 [11:50:12] [02mw-config] 07Reception123 closed pull request 03#3804: add ShortDescription per T7004 - 13https://git.io/JYJsm [11:50:14] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JYJZG [11:50:15] [02miraheze/mw-config] 07Reception123 03b5fa5b4 - add ShortDescription per T7004 (#3804) [11:50:17] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-2 [11:50:18] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbvb3 [11:51:19] miraheze/mw-config - Reception123 the build passed. [11:51:22] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.93, 6.51, 6.68 [11:52:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki' on mw*/jbr* [11:52:23] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw*/jbr* [11:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.59, 6.81, 6.63 [11:58:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.60, 6.74, 6.81 [11:59:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.56, 7.67, 6.96 [12:00:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.66, 7.68, 7.14 [12:00:22] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.98, 7.58, 7.24 [12:00:32] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 7.49, 6.84 [12:02:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.51, 7.57, 7.28 [12:02:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.90, 7.63, 6.98 [12:04:33] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.77, 7.97, 7.18 [12:06:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.04, 7.57, 7.12 [12:09:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.84, 7.33, 7.27 [12:10:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.92, 7.30, 7.53 [12:12:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.34, 5.72, 6.61 [12:12:32] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.62, 6.45, 6.73 [12:12:33] PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 8.59, 5.45, 2.74 [12:13:08] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.80, 5.73, 6.64 [12:14:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 2.87, 5.38, 6.72 [12:16:10] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 20.50, 26.71, 20.67 [12:18:06] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 11.47, 21.20, 19.36 [12:20:01] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 11.18, 18.00, 18.41 [12:32:33] PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.74, 1.64, 3.71 [12:36:34] RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.77, 1.29, 3.10 [13:55:59] PROBLEM - db1.miraheze.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for db1.miraheze.wiki could not be found [13:56:01] PROBLEM - madgenderscience.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for madgenderscience.wiki could not be found [13:56:03] PROBLEM - pt.graalmilitary.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pt.graalmilitary.com could not be found [13:56:04] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tulpa.info could not be found [13:56:06] PROBLEM - rangpurpedia.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rangpurpedia.xyz could not be found [13:56:07] PROBLEM - wiki.mastodon.kr - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mastodon.kr could not be found [13:56:55] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.25, 5.47, 3.26 [13:58:54] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.79, 3.81, 2.92 [14:00:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.33, 2.63, 2.59 [14:02:49] RECOVERY - db1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - db1.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [14:02:54] RECOVERY - madgenderscience.wiki - reverse DNS on sslhost is OK: rDNS OK - madgenderscience.wiki reverse DNS resolves to cp11.miraheze.org [14:02:58] RECOVERY - pt.graalmilitary.com - reverse DNS on sslhost is OK: rDNS OK - pt.graalmilitary.com reverse DNS resolves to cp10.miraheze.org [14:03:00] RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: rDNS OK - wiki.tulpa.info reverse DNS resolves to cp10.miraheze.org [14:03:05] RECOVERY - wiki.mastodon.kr - reverse DNS on sslhost is OK: rDNS OK - wiki.mastodon.kr reverse DNS resolves to cp11.miraheze.org [14:03:06] RECOVERY - rangpurpedia.xyz - reverse DNS on sslhost is OK: rDNS OK - rangpurpedia.xyz reverse DNS resolves to cp11.miraheze.org [14:23:32] paladox: do you know how to resolve https://phabricator.miraheze.org/T7024#139060 ? [14:23:33] [ ⚓ T7024 Change domain for wiki: basebuilder.miraheze.org -> vassals.miraheze.org ] - phabricator.miraheze.org [14:24:00] not sure what more can be suggested other than what I suggested with cookies [14:24:03] could try logging out of login.miraheze.org and that domain and then try logging in [14:24:17] or could try deleting all cookies. Otherwise no :( [14:24:50] paladox: I did suggest that to them above but apparently it didn't work [14:25:34] oh i'm not sure then apart from deleting all cookies not just specific cookies [14:27:02] JohnLewis: would you perhaps have any idea what else we could do if the user says that deleting cookies didn't fix it? [14:31:44] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 11.38, 6.04, 3.37 [14:32:52] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.25, 21.63, 17.69 [14:34:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.33, 19.75, 17.50 [14:35:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 6.50, 5.49 [14:35:33] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.61, 4.00, 3.14 [14:37:27] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.67, 2.88, 2.82 [14:37:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.46, 5.82, 5.36 [15:05:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 6.30, 5.59 [15:07:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.87, 5.62, 5.41 [15:07:45] PROBLEM - www.wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikiescola.com.br could not be found [15:10:21] PROBLEM - wikiescola.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikiescola.com.br could not be found [15:36:25] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.63, 6.67, 5.66 [15:38:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.98, 5.85, 5.47 [15:44:46] PROBLEM - test3 Puppet on test3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 11 minutes ago with 0 failures [15:48:23] Reception123: is that the no active session issue? [15:51:17] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.13, 4.48, 2.88 [15:53:11] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.80, 3.20, 2.60 [15:53:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.85, 6.02, 5.73 [15:55:04] RhinosF1, yes [15:55:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.59, 5.65, 5.65 [15:57:40] dmehus: ack [16:00:05] RhinosF1: do you perhaps have any thoughts then / agree with what dmehus suggested on the task? [16:00:25] Reception123: best bet [16:00:51] so wouldn't be harm in trying it? worse case, it doesn't work [16:06:06] there's no harm in clearing a token for a single user [16:06:13] you have to use eval though Reception123 [16:06:48] RhinosF1: oh, why not the script? [16:07:06] > there's no harm in clearing a token for a single user [16:07:06] RhinosF1, ack [16:07:12] Reception123: script for local accounts isn't user specfic [16:07:48] ah [16:08:07] If it's `eval.php`, doesn't that mean RhinosF1 can also reset tokens for specific users too? [16:08:32] any sysadmin can use eval [16:09:01] yeah, that's good and makes sense [16:38:45] !log running delbackups [16:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:42:01] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.48, 1.08 [16:44:01] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.12, 1.36, 1.08 [16:44:03] RhinosF1, I just noticed on SAL you forgot to log your usage of `migrateAccount.php` yesterday [16:45:39] dmehus: I used $ [16:45:50] oh yeah [16:45:54] so it never got logged [16:46:16] `rhinos@jobrunner3:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php migrateAccount.php --wiki=besttvshowswiki --username=Brandond360 --auto` [16:46:17] RhinosF1, for you to copy and paste if you want [16:47:03] dmehus: https://meta.miraheze.org/w/index.php?title=Tech:Server_admin_log&diff=168500&oldid=168499 [16:47:05] [ Difference between revisions of "Tech:Server admin log" - Miraheze Meta ] - meta.miraheze.org [16:47:45] RhinosF1, ack that works too, ty [16:55:50] got an error when running the deletion script and it exists [16:55:54] https://www.irccloud.com/pastebin/j3xv6Eph/ [16:55:54] [ Snippet | IRCCloud ] - www.irccloud.com [16:57:03] oh I wonder if that's related to our switch to memcache Reception123 [16:57:11] did we update the wiki deletion script code? [16:57:23] yes, Universal Omega has said so when I asked him about this error when I was running another script [16:57:28] so I'll have to ping paladox ^ for this one [16:57:34] yeah [16:58:00] hmm? [16:58:03] oh [16:58:05] let me fix [16:58:08] paladox: thanks! [16:58:12] thanks, paladox :) [16:58:18] also got the error when I ran the rename script [16:58:25] but that still worked [16:59:18] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYUKh [16:59:19] [02miraheze/MirahezeMagic] 07paladox 039f796e4 - Fix config in removeRedisKey [16:59:52] thanks [16:59:58] oh, wow, good timing too since Puppet should run in a couple minutes [17:00:16] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYU68 [17:00:18] [02miraheze/mediawiki] 07paladox 0312ba556 - Update MirahezeMagic [17:00:23] miraheze/MirahezeMagic - paladox the build passed. [17:00:41] 👍 [17:01:08] [02mediawiki] 07Universal-Omega closed pull request 03#1375: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `6ac655f` - 13https://git.io/JmFoJ [17:01:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#1375: Bump extensions/UniversalLanguageSelector from `f31b0c9` to `6ac655f` - 13https://git.io/JYU6i [17:01:11] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-6ac655f [17:01:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/UniversalLanguageSelector-6ac655f - 13https://git.io/vbL5b [17:02:22] [02mediawiki] 07Universal-Omega commented on pull request 03#1356: Bump extensions/Comments from `c70f7d5` to `0c05cc8` - 13https://git.io/JYU6p [17:02:24] [02mediawiki] 07dependabot[bot] edited pull request 03#1356: Bump extensions/Comments from `c70f7d5` to `0c05cc8` - 13https://git.io/JmFrv [17:02:55] [02mediawiki] 07Universal-Omega closed pull request 03#1372: Bump extensions/Wikibase from `14df0d2` to `292153f` - 13https://git.io/JmFrN [17:02:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYUiL [17:02:58] [02miraheze/mediawiki] 07dependabot[bot] 034fedd03 - Bump extensions/Wikibase from `14df0d2` to `292153f` (#1372) [17:02:59] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-292153f - 13https://git.io/vbL5b [17:03:01] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/Wikibase-292153f [17:03:26] [02mediawiki] 07dependabot[bot] edited pull request 03#1356: Bump extensions/Comments from `c70f7d5` to `0c05cc8` - 13https://git.io/JmFrv [17:03:28] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/Comments-73c87ef [+0/-0/±1] 13https://git.io/JYUiZ [17:03:30] [02miraheze/mediawiki] 07dependabot[bot] 03a234850 - Bump extensions/Comments from `d75fa77` to `73c87ef` [17:03:31] [02mediawiki] 07dependabot[bot] opened pull request 03#1377: Bump extensions/Comments from `d75fa77` to `73c87ef` - 13https://git.io/JYUin [17:03:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/Comments-73c87ef - 13https://git.io/vbL5b [17:03:34] [02mediawiki] 07dependabot[bot] labeled pull request 03#1377: Bump extensions/Comments from `d75fa77` to `73c87ef` - 13https://git.io/JYUin [17:03:35] [02mediawiki] 07dependabot[bot] labeled pull request 03#1377: Bump extensions/Comments from `d75fa77` to `73c87ef` - 13https://git.io/JYUin [17:03:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#1356: Bump extensions/Comments from `c70f7d5` to `0c05cc8` - 13https://git.io/JYUic [17:03:38] [02mediawiki] 07dependabot[bot] closed pull request 03#1356: Bump extensions/Comments from `c70f7d5` to `0c05cc8` - 13https://git.io/JmFrv [17:03:40] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-0c05cc8 [17:03:42] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-0c05cc8 - 13https://git.io/vbL5b [17:08:10] [02mediawiki] 07Universal-Omega closed pull request 03#1377: Bump extensions/Comments from `d75fa77` to `73c87ef` - 13https://git.io/JYUin [17:08:12] [02mediawiki] 07dependabot[bot] commented on pull request 03#1377: Bump extensions/Comments from `d75fa77` to `73c87ef` - 13https://git.io/JYUP5 [17:08:21] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-73c87ef - 13https://git.io/vbL5b [17:08:22] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_35/extensions/Comments-73c87ef [17:08:49] [02mediawiki] 07Universal-Omega closed pull request 03#1371: Bump extensions/OAuth from `5a56193` to `70588c6` - 13https://git.io/JmFrF [17:08:50] [02mediawiki] 07dependabot[bot] commented on pull request 03#1371: Bump extensions/OAuth from `5a56193` to `70588c6` - 13https://git.io/JYUXf [17:08:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/OAuth-70588c6 [17:08:53] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/OAuth-70588c6 - 13https://git.io/vbL5b [17:09:32] [02mediawiki] 07Universal-Omega closed pull request 03#1368: Bump extensions/LocalisationUpdate from `f1cc54c` to `7fc5f07` - 13https://git.io/JmFrK [17:09:34] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYUXG [17:09:35] [02miraheze/mediawiki] 07dependabot[bot] 03af8f148 - Bump extensions/LocalisationUpdate from `f1cc54c` to `7fc5f07` (#1368) [17:09:37] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/LocalisationUpdate-7fc5f07 [17:09:38] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/LocalisationUpdate-7fc5f07 - 13https://git.io/vbL5b [17:09:58] [02mediawiki] 07Universal-Omega closed pull request 03#1354: Bump skins/Truglass from `e01b332` to `cc8fe11` - 13https://git.io/JmFwp [17:09:59] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYUXR [17:10:01] [02miraheze/mediawiki] 07dependabot[bot] 03e102d1f - Bump skins/Truglass from `e01b332` to `cc8fe11` (#1354) [17:10:05] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Truglass-cc8fe11 - 13https://git.io/vbL5b [17:10:06] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Truglass-cc8fe11 [17:10:26] [02mediawiki] 07Universal-Omega closed pull request 03#1345: Bump skins/Medik from `921c3b8` to `3538960` - 13https://git.io/JmFwo [17:10:28] [02mediawiki] 07dependabot[bot] commented on pull request 03#1345: Bump skins/Medik from `921c3b8` to `3538960` - 13https://git.io/JYUXP [17:10:29] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Medik-3538960 - 13https://git.io/vbL5b [17:10:31] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Medik-3538960 [17:11:03] [02mediawiki] 07Universal-Omega closed pull request 03#1357: Bump extensions/cldr from `c03d2bb` to `8eeb9d8` - 13https://git.io/JmFrU [17:11:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYUX9 [17:11:06] [02miraheze/mediawiki] 07dependabot[bot] 03e92cb0c - Bump extensions/cldr from `c03d2bb` to `8eeb9d8` (#1357) [17:11:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/cldr-8eeb9d8 - 13https://git.io/vbL5b [17:11:09] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/cldr-8eeb9d8 [17:11:33] [02mediawiki] 07Universal-Omega closed pull request 03#1360: Bump extensions/ContributionScores from `6fac009` to `4dc3f4f` - 13https://git.io/JmFrm [17:11:35] [02mediawiki] 07dependabot[bot] commented on pull request 03#1360: Bump extensions/ContributionScores from `6fac009` to `4dc3f4f` - 13https://git.io/JYUXA [17:11:36] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ContributionScores-4dc3f4f - 13https://git.io/vbL5b [17:11:38] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ContributionScores-4dc3f4f [17:12:02] [02mediawiki] 07Universal-Omega closed pull request 03#1353: Bump extensions/ApprovedRevs from `801399b` to `f66b77f` - 13https://git.io/JmFwA [17:12:04] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYU1v [17:12:05] [02miraheze/mediawiki] 07dependabot[bot] 03a3fc41b - Bump extensions/ApprovedRevs from `801399b` to `f66b77f` (#1353) [17:12:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-f66b77f - 13https://git.io/vbL5b [17:12:08] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/ApprovedRevs-f66b77f [17:12:16] [02mediawiki] 07Universal-Omega closed pull request 03#1359: Bump extensions/VisualEditor from `be34a2d` to `f089b74` - 13https://git.io/JmFrt [17:12:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#1359: Bump extensions/VisualEditor from `be34a2d` to `f089b74` - 13https://git.io/JYU1m [17:12:19] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-f089b74 [17:12:21] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VisualEditor-f089b74 - 13https://git.io/vbL5b [17:12:47] [02mediawiki] 07Universal-Omega closed pull request 03#1351: Bump extensions/VoteNY from `c7a829a` to `31e1b12` - 13https://git.io/JmFw7 [17:12:49] [02mediawiki] 07dependabot[bot] commented on pull request 03#1351: Bump extensions/VoteNY from `c7a829a` to `31e1b12` - 13https://git.io/JYU1l [17:12:50] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VoteNY-31e1b12 - 13https://git.io/vbL5b [17:12:52] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/VoteNY-31e1b12 [17:13:05] [02mediawiki] 07Universal-Omega closed pull request 03#1349: Bump skins/Citizen from `b68a2bd` to `2653336` - 13https://git.io/JmFwS [17:13:06] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYU1u [17:13:08] [02miraheze/mediawiki] 07dependabot[bot] 039b6c196 - Bump skins/Citizen from `b68a2bd` to `2653336` (#1349) [17:13:09] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-2653336 [17:13:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/skins/Citizen-2653336 - 13https://git.io/vbL5b [17:13:35] [02mediawiki] 07Universal-Omega closed pull request 03#1355: Bump extensions/WikiSEO from `c0579da` to `c6ce83b` - 13https://git.io/JmFwj [17:13:37] [02mediawiki] 07dependabot[bot] commented on pull request 03#1355: Bump extensions/WikiSEO from `c0579da` to `c6ce83b` - 13https://git.io/JYU1y [17:13:39] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-c6ce83b [17:13:40] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/WikiSEO-c6ce83b - 13https://git.io/vbL5b [17:18:08] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [17:19:00] Universal_Omega: looks like that annoying error is back [17:19:06] (the one that caused puppet to not work for 3 days) [17:19:26] Reception123: same fix [17:19:32] Maybe we need to up the depth [17:19:37] Or merge slower [17:19:40] yeah, I wonder why it's happening all of a sudden [17:20:32] PROBLEM - cp11 Stunnel Http for test3 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:34] !log running sudo -u www-data git reset --hard origin/REL1_35 && sudo -u www-data git pull ; sudo -u www-data git submodule update on mw*/jbr* [17:20:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:44] PROBLEM - cp10 Stunnel Http for test3 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:50] PROBLEM - test3 MediaWiki Rendering on test3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:53] PROBLEM - test3 HTTPS on test3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:11] PROBLEM - cp12 Stunnel Http for test3 on cp12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:21:47] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:22:06] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [17:22:28] paladox: even after running puppet still getting the error [17:22:45] what error? [17:22:56] https://www.irccloud.com/pastebin/PKHZgivu/ [17:22:56] [ Snippet | IRCCloud ] - www.irccloud.com [17:23:00] ^ paladox same one as before [17:23:17] doesn't look like its the same error to me [17:23:37] oh [17:23:41] i guess it needs a job [17:23:47] actually [17:23:49] no [17:23:58] port needs opening to mw* [17:24:11] i see the error [17:24:39] oh, that's good [17:24:44] yeah sorry I realise it's not exactly the same, I thought so because they both had the call to getAllKeys() error [17:25:02] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYUSf [17:25:03] [02miraheze/MirahezeMagic] 07paladox 03df7324e - Fix variable name in removeMemcachedKey [17:25:51] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±7] 13https://git.io/JYUSt [17:25:52] [02miraheze/mediawiki] 07paladox 038a2c276 - Update MirahezeMagic [17:25:53] [02miraheze/mediawiki] 07paladox 0322a04cc - Merge branch 'REL1_35' of github.com:miraheze/mediawiki into REL1_35 [17:26:05] miraheze/MirahezeMagic - paladox the build passed. [17:26:30] Reception123 fixed [17:26:44] paladox: thanks again :) [17:29:37] can confirm it works now [17:30:19] paladox: nevermind, it dropped two and then came up with a new error [17:30:21] https://www.irccloud.com/pastebin/ZSSnN10o/ [17:30:21] [ Snippet | IRCCloud ] - www.irccloud.com [17:30:38] *deleted two [17:31:04] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.39, 6.83, 5.56 [17:32:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.47, 8.44, 6.07 [17:32:46] i'll brb [17:32:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.16, 6.76, 5.91 [17:33:01] Reception123 you just have to add a check to $keys = $memcached->getAllKeys(); [17:33:02] ok [17:33:06] making sure it is an array [17:33:18] hmm, let me see if I can figure that out [17:33:21] PROBLEM - wiki.warframestat.us - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.warframestat.us could not be found [17:33:23] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for antiguabarbudacalypso.com could not be found [17:33:24] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [17:33:27] PROBLEM - petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for petrawiki.org could not be found [17:34:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.67, 7.22, 5.90 [17:35:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.70, 6.69, 5.91 [17:36:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 6.60, 5.83 [17:36:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.21, 6.17, 5.88 [17:40:08] RECOVERY - wiki.warframestat.us - reverse DNS on sslhost is OK: rDNS OK - wiki.warframestat.us reverse DNS resolves to cp11.miraheze.org [17:40:11] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: rDNS OK - antiguabarbudacalypso.com reverse DNS resolves to cp11.miraheze.org [17:40:11] RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: rDNS OK - wiki.mikrodev.com reverse DNS resolves to cp10.miraheze.org [17:40:26] RECOVERY - petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - petrawiki.org reverse DNS resolves to cp10.miraheze.org [17:58:26] i don't understand how it's failing. [17:58:29] the port is open [17:58:32] and it works for me in eval [18:00:51] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/JYUFT [18:00:53] [02miraheze/MirahezeMagic] 07Reception123 037a4cab5 - check if $keys is array [18:00:54] [02MirahezeMagic] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/fQRGX [18:00:56] [02MirahezeMagic] 07Reception123 opened pull request 03#237: check if $keys is array - 13https://git.io/JYUFI [18:01:17] ^ paladox is that ok [18:01:55] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYUFc [18:01:57] [02miraheze/MirahezeMagic] 07paladox 03518d704 - Fix "Argument 1 passed to Memcached::getDelayed() must be of the type array, bool given" [18:01:58] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [18:02:07] [02MirahezeMagic] 07paladox opened pull request 03#238: Fix "Argument 1 passed to Memcached::getDelayed() must be of the type array, bool given" - 13https://git.io/JYUFu [18:02:10] Reception123 that ^ [18:02:10] miraheze/MirahezeMagic - Reception123 the build passed. [18:02:21] [02MirahezeMagic] 07Reception123 closed pull request 03#237: check if $keys is array - 13https://git.io/JYUFI [18:02:27] paladox: ah ok, so I was missing the return part [18:02:51] [02MirahezeMagic] 07paladox closed pull request 03#238: Fix "Argument 1 passed to Memcached::getDelayed() must be of the type array, bool given" - 13https://git.io/JYUFu [18:02:53] [02miraheze/MirahezeMagic] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JYUFD [18:02:54] [02miraheze/MirahezeMagic] 07paladox 033975198 - Merge pull request #238 from miraheze/paladox-patch-1 [18:02:56] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/fQRGX [18:02:57] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-1 [18:03:20] miraheze/MirahezeMagic - paladox the build passed. [18:03:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYUF5 [18:03:26] [02miraheze/mediawiki] 07paladox 0311a7ff5 - Update MirahezeMagic [18:04:00] miraheze/MirahezeMagic - paladox the build passed. [18:04:38] dmehus: https://phabricator.miraheze.org/T7007#139087 [18:04:39] [ ⚓ T7007 Hoole History Domain Connection ] - phabricator.miraheze.org [18:04:45] That's one good thing about cloudflare [18:04:57] We expand all CNAMEs at DNS level [18:04:59] RhinosF1, 💯 [18:05:18] So you'll never get a CNAME for MirahezeBots domains [18:05:22] Just a cached A record [18:05:30] I migrated all my domains to Cloudflare from Gandi [18:05:44] except for two that Cloudflare doesn't support, so went with Google Domains [18:06:03] paladox: seems to work now but with a warning [18:06:04] Warning: Invalid argument supplied for foreach() in /srv/mediawiki/w/extensions/MirahezeMagic/includes/MirahezeMagicHooks.php on line 306 [18:06:09] RhinosF1, oh true, yeah [18:06:49] Reception123, heh this was a bit of a rabbit hole you and paladox had to go down [18:07:08] * dmehus did not intend the bunny pun [18:07:24] yeah, memcached did change quite a few things [18:07:33] yeah [18:07:50] dmehus: cf is good for speed [18:07:58] RhinosF1, yeah [18:08:14] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 51.195.236.219/cpweb, 2001:41d0:800:1bbd::4/cpweb, 2607:5300:205:200::1c30/cpweb [18:08:36] Huh [18:08:36] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.75, 8.36, 6.79 [18:10:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.44, 6.76, 5.93 [18:10:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:10:58] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 198s [18:12:02] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.73, 7.57, 6.34 [18:14:00] dmehus: re phab though, A records will work fine as long as you assume responsibility in an outage or if server gets removed [18:15:07] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 33s [18:15:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:15:41] PROBLEM - cp10 Stunnel Http for mw11 on cp10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.017 second response time [18:15:48] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.763 second response time [18:15:58] paladox: ? [18:16:06] uh [18:16:11] RhinosF1, ack [18:16:15] paladox: something shit happened [18:16:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.219/cpweb, 51.195.236.250/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [18:16:28] We're super slow and icinga shows 502s [18:16:32] I'm getting extreme load times on Meta wiki and I also can't get bolded page names to show up in the search bar [18:16:43] Yeah [18:16:48] Look at Icinga [18:16:49] PROBLEM - cp10 Stunnel Http for mw10 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:56] PROBLEM - cp3 Stunnel Http for mw11 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:07] RhinosF1, yeah [18:17:16] welp, everything broke now, lol [18:17:18] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 10.13, 5.81, 3.13 [18:17:21] i don't see php eception [18:17:28] [ef76507414532e61d6a55981] 2021-03-24 18:16:39: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [18:17:40] paladox: somethings gone [18:17:45] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.65, 7.02, 6.01 [18:17:48] RECOVERY - cp10 Stunnel Http for mw11 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 0.915 second response time [18:17:54] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 2.776 second response time [18:18:08] Lake that's just search [18:18:10] paladox: cp12 completely down [18:18:28] HTTPS critical [18:18:35] oh right [18:19:02] RECOVERY - cp10 Stunnel Http for mw10 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 7.305 second response time [18:19:10] RECOVERY - cp3 Stunnel Http for mw11 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 1.523 second response time [18:19:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.20, 7.85, 6.12 [18:19:42] paladox: there was a huge load spike on cp11 but that was about an hour ago [18:19:43] looks like none of them went down according to syslog [18:19:49] PROBLEM - wiki.minkyu.kim - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minkyu.kim could not be found [18:19:50] PROBLEM - stablestate.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for stablestate.org could not be found [18:19:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.90, 6.69, 6.03 [18:19:54] PROBLEM - cp3 Stunnel Http for mw9 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:19:55] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for guia.cineastas.pt could not be found [18:19:56] PROBLEM - meta.orain.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for meta.orain.org could not be found [18:19:57] Something caused them to get very alert [18:20:04] PROBLEM - marthaspeaks.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for marthaspeaks.miraheze.org could not be found [18:20:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYUxc [18:20:13] [02miraheze/services] 07MirahezeSSLBot 0366be0bd - BOT: Updating services config for wikis [18:20:22] PROBLEM - cp11 Stunnel Http for mw9 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:37] paladox: still slow [18:20:37] PROBLEM - cp10 Stunnel Http for mw9 on cp10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:20:53] I am unable to do anything because of CSRF errors. [18:20:59] Not even able to log out. [18:21:18] https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=cp10.miraheze.org&var-port=9100 guess it was the high loadoh. [18:21:19] [ Grafana ] - grafana.miraheze.org [18:21:21] Whole of Miraheze seems to be having issues [18:21:21] oh [18:21:25] let me check the db [18:21:38] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:38] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.68, 3.30, 2.81 [18:21:39] Ping JohnLewis / Reception123 [18:22:10] none of the dbs went down [18:22:11] * Reception123 is here now [18:22:13] RECOVERY - cp3 Stunnel Http for mw9 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15200 bytes in 2.320 second response time [18:22:13] what's going on [18:22:20] db is definitely fine as I was just running drop queries on it [18:22:30] RECOVERY - cp11 Stunnel Http for mw9 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15214 bytes in 0.009 second response time [18:22:39] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:45] RECOVERY - cp10 Stunnel Http for mw9 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15200 bytes in 0.013 second response time [18:22:55] PROBLEM - cp3 Stunnel Http for mw10 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:23:10] https://graylog.miraheze.org/messages/graylog_129/fa7a4961-8ccb-11eb-b6dc-0200001a24a4 [18:23:13] dns went down [18:23:27] that'll explain why [18:23:28] dmehus: I think you should send a message on announcements (on discord) before people go crazy [18:23:34] Oh, what caused that paladox? [18:23:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20708 bytes in 0.919 second response time [18:23:57] my guess is a network blip between here and ns1. I emphasise this is a guess. [18:24:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 7.77, 6.66 [18:24:19] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:32] PROBLEM - cp11 Stunnel Http for mw10 on cp11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:26] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.gyaanipedia.com could not be found [18:25:28] PROBLEM - www.wikilukas.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.wikilukas.tk could not be found [18:25:30] PROBLEM - wiki.redeemer.live - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.redeemer.live could not be found [18:25:31] PROBLEM - podpedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for podpedia.org could not be found [18:25:36] paladox: ack, is ns1 RN or? [18:25:36] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for housing.wiki could not be found [18:25:37] PROBLEM - wiki.jacksonheights.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jacksonheights.nyc could not be found [18:25:37] PROBLEM - wiki.ciptamedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ciptamedia.org could not be found [18:25:37] PROBLEM - pastport.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pastport.org could not be found [18:25:38] PROBLEM - wiki.vinesh.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.vinesh.eu.org could not be found [18:25:38] PROBLEM - vise.dayid.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vise.dayid.org could not be found [18:25:38] PROBLEM - www.opendatascot.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.opendatascot.org could not be found [18:25:39] PROBLEM - wiki.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.fraterniterebellion.com could not be found [18:25:39] PROBLEM - www.mcpk.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.mcpk.wiki could not be found [18:25:42] PROBLEM - wiki.minecraftathome.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.minecraftathome.com could not be found [18:25:42] PROBLEM - wiki.shaazzz.ir - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.shaazzz.ir could not be found [18:25:44] PROBLEM - wiki.kourouklides.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.kourouklides.com could not be found [18:25:46] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for fc.songcontests.eu could not be found [18:25:46] ns1 is ramnode [18:25:47] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for tl.awiki.org could not be found [18:25:49] PROBLEM - wiki.dom64.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.dom64.xyz could not be found [18:25:54] PROBLEM - wiki.macc.nyc - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.macc.nyc could not be found [18:26:01] PROBLEM - biblestrength.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for biblestrength.org could not be found [18:26:02] PROBLEM - wiki.etn.link - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.etn.link could not be found [18:26:07] PROBLEM - hi.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for hi.gyaanipedia.com could not be found [18:26:07] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [18:26:27] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20710 bytes in 6.136 second response time [18:26:32] RECOVERY - stablestate.org - reverse DNS on sslhost is OK: rDNS OK - stablestate.org reverse DNS resolves to cp11.miraheze.org [18:26:39] RECOVERY - cp11 Stunnel Http for mw10 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 8.659 second response time [18:26:40] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp11.miraheze.org [18:26:50] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20710 bytes in 0.617 second response time [18:26:50] RECOVERY - wiki.minkyu.kim - reverse DNS on sslhost is OK: rDNS OK - wiki.minkyu.kim reverse DNS resolves to cp11.miraheze.org [18:27:00] RECOVERY - marthaspeaks.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - marthaspeaks.miraheze.org reverse DNS resolves to cp11.miraheze.org [18:27:00] RECOVERY - meta.orain.org - reverse DNS on sslhost is OK: rDNS OK - meta.orain.org reverse DNS resolves to cp12.miraheze.org [18:27:07] RECOVERY - cp3 Stunnel Http for mw10 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15201 bytes in 4.427 second response time [18:27:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:28:08] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.36, 6.13, 6.25 [18:28:44] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:29:28] ```Error contacting the Parsoid/RESTBase server: (curl error: 28) Timeout was reached``` [18:29:54] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 270s [18:30:27] dmehus: everything went all over the place [18:30:37] ```Invalid CSRF token``` I hope I didn't get logged out again [18:30:53] RhinosF1, yeah, will we have another IR for this? [18:30:59] Ask infra [18:31:03] NS in their issue [18:31:08] yeah [18:31:31] dmehus: I'm still seeing alerts flapping so just be patient [18:31:40] okay [18:31:42] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.195.236.250/cpweb, 2607:5300:205:200::1c30/cpweb [18:31:52] paladox: can you raise a ticket? [18:31:54] .op [18:31:55] Attempting to OP... [18:32:23] i cannot without knowing what happened. Network isn't my expetese rather its JohnLewis [18:32:28] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - de.gyaanipedia.com reverse DNS resolves to cp12.miraheze.org [18:32:30] RECOVERY - www.wikilukas.tk - reverse DNS on sslhost is OK: rDNS OK - www.wikilukas.tk reverse DNS resolves to cp12.miraheze.org [18:32:32] RECOVERY - wiki.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - wiki.fraterniterebellion.com reverse DNS resolves to cp12.miraheze.org [18:32:34] RECOVERY - podpedia.org - reverse DNS on sslhost is OK: rDNS OK - podpedia.org reverse DNS resolves to cp11.miraheze.org [18:32:34] RECOVERY - www.opendatascot.org - reverse DNS on sslhost is OK: rDNS OK - www.opendatascot.org reverse DNS resolves to cp11.miraheze.org [18:32:35] RECOVERY - www.mcpk.wiki - reverse DNS on sslhost is OK: rDNS OK - www.mcpk.wiki reverse DNS resolves to cp10.miraheze.org [18:32:36] otherwise it would just be closed [18:32:36] RECOVERY - wiki.minecraftathome.com - reverse DNS on sslhost is OK: rDNS OK - wiki.minecraftathome.com reverse DNS resolves to cp11.miraheze.org [18:32:37] RECOVERY - wiki.redeemer.live - reverse DNS on sslhost is OK: rDNS OK - wiki.redeemer.live reverse DNS resolves to cp10.miraheze.org [18:32:38] RECOVERY - wiki.shaazzz.ir - reverse DNS on sslhost is OK: rDNS OK - wiki.shaazzz.ir reverse DNS resolves to cp11.miraheze.org [18:32:38] as its working now [18:32:39] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: rDNS OK - housing.wiki reverse DNS resolves to cp11.miraheze.org [18:32:41] RECOVERY - wiki.jacksonheights.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.jacksonheights.nyc reverse DNS resolves to cp11.miraheze.org [18:32:42] RECOVERY - wiki.ciptamedia.org - reverse DNS on sslhost is OK: rDNS OK - wiki.ciptamedia.org reverse DNS resolves to cp11.miraheze.org [18:32:42] RECOVERY - pastport.org - reverse DNS on sslhost is OK: rDNS OK - pastport.org reverse DNS resolves to cp11.miraheze.org [18:32:46] RECOVERY - wiki.kourouklides.com - reverse DNS on sslhost is OK: rDNS OK - wiki.kourouklides.com reverse DNS resolves to cp11.miraheze.org [18:32:48] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: rDNS OK - vise.dayid.org reverse DNS resolves to cp11.miraheze.org [18:32:50] RECOVERY - wiki.dom64.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.dom64.xyz reverse DNS resolves to cp11.miraheze.org [18:32:57] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.236.219/cpweb, 51.195.236.250/cpweb, 2001:41d0:800:1bbd::4/cpweb, 51.222.25.132/cpweb [18:33:00] paladox: it's not working, we're still seeing flapping alerts [18:33:01] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.589 second response time [18:33:04] PROBLEM - wikien.wildterra2.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikien.wildterra2.com could not be found [18:33:05] RECOVERY - wiki.macc.nyc - reverse DNS on sslhost is OK: rDNS OK - wiki.macc.nyc reverse DNS resolves to cp11.miraheze.org [18:33:12] JohnLewis: help please [18:33:12] yeah, definitely a task for John [18:33:24] PROBLEM - cyberlaw.ccdcoe.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cyberlaw.ccdcoe.org could not be found [18:33:24] or SPF|Cloud if he's around [18:33:27] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [18:33:29] PROBLEM - www.pornwiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.pornwiki.org could not be found [18:34:36] PROBLEM - www.petrawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.petrawiki.org could not be found [18:34:36] PROBLEM - ipv6bolivia.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ipv6bolivia.tk could not be found [18:34:57] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:35:02] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 20709 bytes in 0.116 second response time [18:35:03] PROBLEM - wiki.twilightsignal.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.twilightsignal.com could not be found [18:35:05] oh [18:35:06] hold on [18:35:09] PROBLEM - erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for erikapedia.com could not be found [18:35:10] PROBLEM - miraheze.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.ml could not be found [18:35:12] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for archive.stellurgists.wiki could not be found [18:35:13] PROBLEM - egoish.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for egoish.org could not be found [18:35:15] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [18:35:15] PROBLEM - www.zenbuddhism.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.zenbuddhism.info could not be found [18:35:20] PROBLEM - wiki.astralprojections.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.astralprojections.org could not be found [18:35:22] PROBLEM - mrlove.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mrlove.wiki could not be found [18:35:32] PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.gq could not be found [18:35:33] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dc-multiverse.dcwikis.com could not be found [18:35:43] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for miraheze.com could not be found [18:35:49] https://graylog.miraheze.org/messages/graylog_129/74f7d510-8ccf-11eb-b6dc-0200001a24a4 [18:36:13] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.58, 20.96, 17.98 [18:36:24] Mobile so you'll have to put in -security paladox [18:36:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.02, 6.84, 6.45 [18:37:22] !log restarted memcached on mem2 [18:38:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.79, 7.54, 6.73 [18:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:39:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.236.219/cpweb, 51.195.236.250/cpweb [18:39:36] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: rDNS OK - fc.songcontests.eu reverse DNS resolves to cp10.miraheze.org [18:39:39] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: rDNS OK - tl.awiki.org reverse DNS resolves to cp10.miraheze.org [18:39:43] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:39:45] RECOVERY - biblestrength.org - reverse DNS on sslhost is OK: rDNS OK - biblestrength.org reverse DNS resolves to cp10.miraheze.org [18:39:46] RECOVERY - wiki.vinesh.eu.org - reverse DNS on sslhost is OK: rDNS OK - wiki.vinesh.eu.org reverse DNS resolves to cp10.miraheze.org [18:39:47] RECOVERY - wikien.wildterra2.com - reverse DNS on sslhost is OK: rDNS OK - wikien.wildterra2.com reverse DNS resolves to cp11.miraheze.org [18:39:55] RECOVERY - hi.gyaanipedia.com - reverse DNS on sslhost is OK: rDNS OK - hi.gyaanipedia.com reverse DNS resolves to cp11.miraheze.org [18:39:56] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: rDNS OK - www.sdiy.info reverse DNS resolves to cp10.miraheze.org [18:40:10] RECOVERY - cyberlaw.ccdcoe.org - reverse DNS on sslhost is OK: rDNS OK - cyberlaw.ccdcoe.org reverse DNS resolves to cp10.miraheze.org [18:40:10] RECOVERY - wiki.etn.link - reverse DNS on sslhost is OK: rDNS OK - wiki.etn.link reverse DNS resolves to cp11.miraheze.org [18:40:14] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: rDNS OK - storytime.jdstroy.cf reverse DNS resolves to cp10.miraheze.org [18:40:26] RECOVERY - www.pornwiki.org - reverse DNS on sslhost is OK: rDNS OK - www.pornwiki.org reverse DNS resolves to cp11.miraheze.org [18:40:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.42, 7.44, 6.80 [18:40:37] !log 18:37:21: restarted memcached on mem2 [18:40:39] PROBLEM - iesd.mobagenie.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for iesd.mobagenie.my.id could not be found [18:40:40] PROBLEM - www.fraterniterebellion.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.fraterniterebellion.com could not be found [18:40:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:40:48] PROBLEM - cnt.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for cnt.miraheze.org could not be found [18:40:52] PROBLEM - www.erikapedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.erikapedia.com could not be found [18:40:53] PROBLEM - wiki.jafaron.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.jafaron.net could not be found [18:41:02] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:41:28] RECOVERY - www.petrawiki.org - reverse DNS on sslhost is OK: rDNS OK - www.petrawiki.org reverse DNS resolves to cp11.miraheze.org [18:41:29] RECOVERY - ipv6bolivia.tk - reverse DNS on sslhost is OK: rDNS OK - ipv6bolivia.tk reverse DNS resolves to cp11.miraheze.org [18:41:38] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.67, 4.78, 3.60 [18:41:53] RECOVERY - wiki.twilightsignal.com - reverse DNS on sslhost is OK: rDNS OK - wiki.twilightsignal.com reverse DNS resolves to cp11.miraheze.org [18:41:54] RECOVERY - erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - erikapedia.com reverse DNS resolves to cp11.miraheze.org [18:41:55] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: rDNS OK - archive.stellurgists.wiki reverse DNS resolves to cp10.miraheze.org [18:42:01] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: rDNS OK - munwiki.info reverse DNS resolves to cp10.miraheze.org [18:42:04] RECOVERY - miraheze.ml - reverse DNS on sslhost is OK: rDNS OK - miraheze.ml reverse DNS resolves to cp11.miraheze.org [18:42:11] RECOVERY - egoish.org - reverse DNS on sslhost is OK: rDNS OK - egoish.org reverse DNS resolves to cp11.miraheze.org [18:42:12] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: rDNS OK - wiki.astralprojections.org reverse DNS resolves to cp11.miraheze.org [18:42:12] RECOVERY - www.zenbuddhism.info - reverse DNS on sslhost is OK: rDNS OK - www.zenbuddhism.info reverse DNS resolves to cp11.miraheze.org [18:42:15] RECOVERY - mrlove.wiki - reverse DNS on sslhost is OK: rDNS OK - mrlove.wiki reverse DNS resolves to cp10.miraheze.org [18:42:21] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.48, 23.13, 20.07 [18:42:22] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: rDNS OK - miraheze.gq reverse DNS resolves to cp11.miraheze.org [18:42:24] RECOVERY - dc-multiverse.dcwikis.com - reverse DNS on sslhost is OK: rDNS OK - dc-multiverse.dcwikis.com reverse DNS resolves to cp11.miraheze.org [18:42:42] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: rDNS OK - miraheze.com reverse DNS resolves to cp11.miraheze.org [18:42:49] paladox: probably not the best time but after working for quite a bit the script spammed Warning: Memcached::fetchAll(): could not unserialize value in /srv/mediawiki/w/extensions/MirahezeMagic/includes/MirahezeMagicHooks.php on line 303 [18:42:58] and was killed with a Segmentation fault [18:43:19] I'll try to restart it [18:43:35] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.31, 3.58, 3.30 [18:43:57] nope, failed again with segmentation fault [18:44:19] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.17, 19.83, 19.23 [18:44:45] I’ll be back later [18:44:46] Things seem to be recovering [18:45:10] ok [18:45:29] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.41, 2.50, 2.93 [18:46:08] please ping me when you're back / fix the issue [18:47:13] !log DELETED and DROPPED https://phabricator.miraheze.org/P399 (draft 1, further deletions to follow after error is fixed which will be updated on the paste) [18:47:14] [ ✎ P399 Deleted wikis 24/03/2021 (draft 1) ] - phabricator.miraheze.org [18:47:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:47:30] RECOVERY - www.fraterniterebellion.com - reverse DNS on sslhost is OK: rDNS OK - www.fraterniterebellion.com reverse DNS resolves to cp11.miraheze.org [18:47:31] RECOVERY - cnt.miraheze.org - reverse DNS on sslhost is OK: rDNS OK - cnt.miraheze.org reverse DNS resolves to cp10.miraheze.org [18:47:38] RECOVERY - iesd.mobagenie.my.id - reverse DNS on sslhost is OK: rDNS OK - iesd.mobagenie.my.id reverse DNS resolves to cp11.miraheze.org [18:47:38] RECOVERY - www.erikapedia.com - reverse DNS on sslhost is OK: rDNS OK - www.erikapedia.com reverse DNS resolves to cp10.miraheze.org [18:47:42] RECOVERY - wiki.jafaron.net - reverse DNS on sslhost is OK: rDNS OK - wiki.jafaron.net reverse DNS resolves to cp10.miraheze.org [18:48:01] Reception123: please be careful [18:48:16] RhinosF1, why? He didn't do anything wrong [18:48:17] Let's let icinga calm first [18:48:32] RhinosF1: it's just what I did so far [18:48:39] I can't proceed further anyway as it's broken again [18:48:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.47, 6.60, 6.69 [18:48:48] dmehus: as Reception123 said, not the best time to run the delete script when we're struggling with load. [18:49:05] RhinosF1, in fairness, he started this before the load issues [18:49:12] He started it like 2-3 hours ago [18:49:12] dmehus: I know [18:49:21] I'm saying be careful at the moment [18:49:44] Load is still high on cp10/mw10/mw11 [18:51:13] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.20, 3.45, 3.24 [18:51:51] that too but what I meant is that even if I wanted to run it, I can't as there's a segmentation fault [18:52:21] dmehus: less than 2 hours but yeah I started when I had no clue about this [18:53:09] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.22, 2.76, 3.01 [18:53:16] That's good [18:53:27] Hopefully mw* will drop in load too soon then [18:54:04] mw10 looks okay [18:54:08] Mw11 maybe not [18:54:22] Replication is screwed [18:56:11] Reception123: I wouldn't drop until https://grafana.miraheze.org/d/_9zrwMHmk/mariadb-replication?viewPanel=16&orgId=1&refresh=1m&var-interval=$__auto_interval_interval&var-host=dbbackup1.miraheze.org:9104&from=now-1h&to=now calms [18:56:12] [ Grafana ] - grafana.miraheze.org [18:56:18] Replication is a mess [18:56:41] RhinosF1: yeah, I don't want another segmentation fault so I'll wait [18:57:25] Load is still a mess everywhere [18:57:35] Not sure what's going on with cp10/mw10/mw11 [18:58:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.17, 4.45, 3.84 [19:00:54] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.11, 6.61, 7.88 [19:00:55] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.85, 3.64, 3.62 [19:03:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.62, 6.59, 7.79 [19:03:54] Oh now we gonna get apt spam too [19:04:52] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:04:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 6.08, 5.61, 4.41 [19:05:04] dmehus: how are things looking? [19:05:06] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:05:32] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:05:44] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:06:25] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (2 critical updates). [19:06:28] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 27 packages available for upgrade (2 critical updates). [19:06:31] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (2 critical updates). [19:06:56] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.28, 3.97, 3.96 [19:07:44] RhinosF1, seems a bit better; still a bit sluggish I think [19:07:48] Let me check again [19:08:20] RhinosF1, actually, great now. `Queued Jobs: 0244ms (PHP7 via metawiki@mw11 / cp12)` [19:08:58] dmehus: good, guess it's just db replication then [19:09:31] RhinosF1, ack, okay. oh, db replication, that might be something to let SPF|Cloud know about as that's his area [19:09:48] Or just give it time [19:09:55] It'll catch up eventually [19:10:03] ah [19:10:07] Just got slow because everything else was [19:10:17] oh 244ms is fine for me. I've never had anything below 0200ms [19:10:31] I consider anything under 1000ms great [19:10:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.45, 2.07, 3.17 [19:11:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.49, 5.32, 6.74 [19:11:37] Yeah but when things were slow the databases backed up [19:11:46] icinga-miraheze: \o/ [19:12:38] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.61, 5.84, 6.76 [19:14:36] Reception123 oh [19:15:09] guess your script was the cause of downtime which i guess is removeMemcachedKey is the real cause [19:16:25] paladox: wasn't it NS though? [19:16:31] How was it the deletion script? [19:16:40] high load on the memcached server [19:17:20] Oh [19:17:31] So what can we do then? [19:20:33] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.43, 7.01, 6.32 [19:21:46] paladox: I guess I shouldn't run the script until we figure it out then [19:22:07] how did you run the script? [19:24:34] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 6.25, 6.17 [19:25:05] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 6.63, 6.38 [19:25:30] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.00, 6.79 [19:25:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.16, 2.66 [19:27:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 2.91, 2.63 [19:28:33] paladox: what do you mean? [19:28:41] As it says in Tech:Delete a wiki [19:29:02] ok, thanks [19:29:11] > I guess I shouldn't run the script until we figure it out then [19:29:11] ergh, that's not great, though. Hopefully we can figure out and correct the issue soon, or we may have to migrate back to redis? [19:29:23] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.25, 6.73, 6.76 [19:29:39] There was a warning though even after you fixed the other errors which I pasted above [19:30:01] yeah [19:31:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.22, 5.98, 6.19 [19:37:10] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 149s [19:39:12] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [19:42:51] That's good [19:42:55] .deop [19:42:55] Attempting to OP... [19:54:30] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JYTYb [19:54:32] [02miraheze/MirahezeMagic] 07paladox 03146bf0b - Use qmegas/memcache-search for fetching all keys [19:54:34] [02MirahezeMagic] 07paladox created branch 03paladox-patch-1 - 13https://git.io/fQRGX [19:54:35] [02MirahezeMagic] 07paladox opened pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JYTYN [19:55:37] miraheze/MirahezeMagic - paladox the build passed. [19:55:38] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYTOT [19:55:39] [02miraheze/MirahezeMagic] 07paladox 03f71c5ac - Update extension.json [19:55:41] [02MirahezeMagic] 07paladox synchronize pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JYTYN [19:56:52] miraheze/MirahezeMagic - paladox the build passed. [19:59:39] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYTOH [19:59:40] [02miraheze/MirahezeMagic] 07paladox 039867e6d - Update MirahezeMagicHooks.php [19:59:42] [02MirahezeMagic] 07paladox synchronize pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JYTYN [20:00:48] miraheze/MirahezeMagic - paladox the build passed. [20:02:07] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYT3n [20:02:09] [02miraheze/MirahezeMagic] 07paladox 03708b4b2 - Update MirahezeMagicHooks.php [20:02:10] [02MirahezeMagic] 07paladox synchronize pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JYTYN [20:03:12] miraheze/MirahezeMagic - paladox the build passed. [20:03:38] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JYT3a [20:03:39] [02miraheze/MirahezeMagic] 07paladox 0337fdd97 - Update MirahezeMagicHooks.php [20:03:41] [02MirahezeMagic] 07paladox synchronize pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JYTYN [20:04:43] miraheze/MirahezeMagic - paladox the build passed. [20:05:30] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 3.14, 4.02, 2.12 [20:07:28] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.98, 2.93, 1.95 [20:09:11] Reception123 i think ^ may fix it. [20:09:41] paladox: okay, so should I be safe to resume it then? [20:09:53] not yet, i haven't merged and deployed :) [20:10:38] i can disable that part for now [20:10:48] would like JohnLewis opinion on the pull [20:12:05] ok, then I'll wait for John and the merge [20:12:11] (I didn't realise you didn't merge heh) [20:12:12] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYTsh [20:12:14] [02miraheze/MirahezeMagic] 07paladox 0309460e5 - Stop using removeMemcachedKey temporarily [20:12:20] Reception123 ^ [20:12:52] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JYTGq [20:12:53] [02miraheze/mediawiki] 07paladox 0354a9ed5 - Update MirahezeMagic [20:13:13] Reception123 ^ [20:13:20] miraheze/MirahezeMagic - paladox the build passed. [20:13:31] deployed [20:13:36] you can run your script now [20:14:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.37, 7.38, 6.21 [20:16:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.62, 6.70, 6.11 [20:17:23] paladox: ok, let's see then [20:18:47] seems to be working fine for now [20:23:07] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 194s [20:24:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.53, 18.58, 15.57 [20:25:05] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 273s [20:26:00] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 7.69, 6.35, 3.51 [20:26:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.83, 16.59, 15.22 [20:29:53] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 1.68, 3.74, 3.05 [20:30:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYTnx [20:30:14] [02miraheze/services] 07MirahezeSSLBot 038e0a402 - BOT: Updating services config for wikis [20:31:51] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.82, 2.75, 2.76 [20:35:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYTcr [20:35:13] [02miraheze/services] 07MirahezeSSLBot 03b8b9474 - BOT: Updating services config for wikis [20:35:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.00, 9.19, 6.73 [20:35:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.12, 7.86, 6.26 [20:37:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.71, 7.90, 6.57 [20:37:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.72, 6.41 [20:38:24] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.30, 6.59, 6.00 [20:39:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.39, 7.17, 6.03 [20:40:20] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.37, 6.14, 5.91 [20:41:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.39, 6.16, 5.80 [20:41:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.26, 7.68, 6.73 [20:41:44] and it's done :) [20:41:48] !log finished running deleted script [20:41:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.27, 6.59, 6.24 [20:41:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:21] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.90, 7.75, 6.87 [20:43:29] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15281 bytes in 0.994 second response time [20:43:56] RECOVERY - cp10 Stunnel Http for test3 on cp10 is OK: HTTP OK: HTTP/1.1 200 OK - 15289 bytes in 0.009 second response time [20:44:07] RECOVERY - cp11 Stunnel Http for test3 on cp11 is OK: HTTP OK: HTTP/1.1 200 OK - 15281 bytes in 0.012 second response time [20:44:19] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20824 bytes in 0.213 second response time [20:44:47] RECOVERY - cp12 Stunnel Http for test3 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15281 bytes in 0.327 second response time [20:45:18] RECOVERY - test3 HTTPS on test3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 670 bytes in 0.020 second response time [20:46:13] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 2.63, 3.74, 3.00 [20:47:20] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.84, 6.34, 6.50 [20:48:07] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.97, 2.81, 2.75 [21:10:36] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 342.09 ms [21:31:37] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.65, 4.57, 2.77 [21:39:13] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.73, 3.90, 3.40 [21:41:08] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.84, 2.96, 3.11 [21:44:24] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 270.72 ms [21:56:37] [02dns] 07RhinosF1 opened pull request 03#200: Miraheze-org: remove record for lists.mh.o - 13https://git.io/JYTzC [21:56:49] paladox: ^ [21:57:18] [02dns] 07paladox closed pull request 03#200: Miraheze-org: remove record for lists.mh.o - 13https://git.io/JYTzC [21:57:19] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYTz0 [21:57:21] [02miraheze/dns] 07RhinosF1 030af357b - Miraheze-org: remove record for lists.mh.o (#200) [22:02:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.64, 6.86, 4.09 [22:06:56] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.60, 3.32, 3.26 [22:26:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 11.99, 7.02, 3.86 [22:28:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.99, 22.58, 16.83 [22:32:24] [02puppet] 07Universal-Omega opened pull request 03#1721: T4005: Install firejail package on mediawiki servers - 13https://git.io/JYTVV [22:32:36] [02puppet] 07Universal-Omega edited pull request 03#1721: T4005: install firejail package on mediawiki servers - 13https://git.io/JYTVV [22:32:52] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.41, 23.69, 18.89 [22:36:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 11.46, 17.76, 17.61 [22:38:54] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.40, 2.59, 3.73 [22:40:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYTw2 [22:40:02] [02miraheze/mw-config] 07Universal-Omega 030198e28 - Fix ImageMagick variable name [22:40:03] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [22:40:08] [02mw-config] 07Universal-Omega opened pull request 03#3805: Fix ImageMagick variable name - 13https://git.io/JYTwV [22:40:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.16, 1.78, 3.29 [22:41:15] miraheze/mw-config - Universal-Omega the build passed. [22:46:29] [02puppet] 07Universal-Omega synchronize pull request 03#1721: T4005: install firejail package on mediawiki servers - 13https://git.io/JYTVV [22:47:37] [02puppet] 07Universal-Omega synchronize pull request 03#1721: T4005: install firejail package on mediawiki servers - 13https://git.io/JYTVV [22:50:54] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 15.23, 8.49, 5.04 [23:00:58] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.27, 2.87, 3.82 [23:01:31] [02puppet] 07Universal-Omega synchronize pull request 03#1721: T4005: install firejail package on mediawiki servers - 13https://git.io/JYTVV [23:02:07] PROBLEM - dbbackup1 APT on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:54] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 2.27, 2.08, 3.24 [23:06:24] [02puppet] 07Universal-Omega synchronize pull request 03#1721: T4005: install firejail package on mediawiki servers - 13https://git.io/JYTVV [23:07:57] [02puppet] 07Universal-Omega edited pull request 03#1721: T4005: install firejail on mediawiki servers - 13https://git.io/JYTVV [23:08:27] [02puppet] 07paladox closed pull request 03#1721: T4005: install firejail on mediawiki servers - 13https://git.io/JYTVV [23:08:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+3/-0/±1] 13https://git.io/JYTKi [23:08:30] [02miraheze/puppet] 07Universal-Omega 030c56476 - T4005: install firejail on mediawiki servers (#1721) [23:10:46] RECOVERY - dbbackup1 APT on dbbackup1 is OK: APT OK: 11 packages available for upgrade (0 critical updates). [23:11:09] [02mw-config] 07Universal-Omega closed pull request 03#3805: Fix ImageMagick variable name - 13https://git.io/JYTwV [23:11:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JYTKj [23:11:12] [02miraheze/mw-config] 07Universal-Omega 03d4fd13b - Fix ImageMagick variable name (#3805) [23:11:14] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [23:11:15] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:12:08] miraheze/mw-config - Universal-Omega the build passed. [23:15:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYT6K [23:15:37] [02miraheze/mw-config] 07Universal-Omega 03ed85ef7 - Use firejail [23:15:39] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:15:55] [02mw-config] 07Universal-Omega opened pull request 03#3806: T4005: use firejail - 13https://git.io/JYT6i [23:17:03] miraheze/mw-config - Universal-Omega the build passed. [23:29:59] PROBLEM - dbbackup1 APT on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:35:28] PROBLEM - dbbackup1 Puppet on dbbackup1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:37:32] RECOVERY - dbbackup1 Puppet on dbbackup1 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [23:40:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYTXw [23:40:21] [02miraheze/mw-config] 07Universal-Omega 0396b1de6 - Update LocalSettings.php [23:40:23] [02mw-config] 07Universal-Omega synchronize pull request 03#3806: T4005: use firejail - 13https://git.io/JYT6i [23:40:44] RECOVERY - dbbackup1 APT on dbbackup1 is OK: APT OK: 11 packages available for upgrade (0 critical updates). [23:41:24] miraheze/mw-config - Universal-Omega the build passed. [23:42:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JYTXd [23:42:26] [02miraheze/mw-config] 07Universal-Omega 038ae76e2 - Update LocalSettings.php [23:42:28] [02mw-config] 07Universal-Omega synchronize pull request 03#3806: T4005: use firejail - 13https://git.io/JYT6i [23:43:29] miraheze/mw-config - Universal-Omega the build passed. [23:49:14] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s