[00:12:51] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:16:40] PROBLEM - cp3 Stunnel Http for test3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:18:37] RECOVERY - cp3 Stunnel Http for test3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15133 bytes in 0.992 second response time [00:18:58] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [00:40:41] PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 103.38 ms [00:42:44] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 104.24 ms [01:40:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtwVT [01:40:13] [02miraheze/services] 07MirahezeSSLBot 031523e19 - BOT: Updating services config for wikis [01:45:18] PROBLEM - wiki.finnsoftware.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.finnsoftware.net reverse DNS resolves to cdn-185-199-111-153.github.com [01:59:17] PROBLEM - wiki.finnsoftware.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.finnsoftware.net could not be found [02:06:18] PROBLEM - wiki.finnsoftware.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.finnsoftware.net reverse DNS resolves to cdn-185-199-108-153.github.com [02:49:11] Universal_Omega, look at the page response time on `metawiki`, `57992ms (PHP7 via metawiki@mw8 / 0cp12)`. In fairness, that was after I globally locked 189 spambot accounts [02:49:30] Oh, Omega's gone :P [02:49:40] .tell Universal_Omega, look at the page response time on `metawiki`, `57992ms (PHP7 via metawiki@mw8 / 0cp12)`. In fairness, that was after I globally locked 189 spambot accounts [02:49:41] dmehus: I'll pass that on when Universal_Omega is around. [03:41:53] @Doug, oh, okay. [03:42:03] Thanks for clarifying. [03:43:48] np [04:01:06] PROBLEM - wiki.mnzp.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mnzp.xyz All nameservers failed to answer the query. [04:07:55] RECOVERY - wiki.mnzp.xyz - reverse DNS on sslhost is OK: rDNS OK - wiki.mnzp.xyz reverse DNS resolves to cp10.miraheze.org [04:12:17] PROBLEM - wiki.finnsoftware.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.finnsoftware.net could not be found [05:28:10] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 256.33 ms [05:30:16] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 254.62 ms [06:01:00] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.08 ms [06:05:07] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 101.75 ms [06:17:20] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.65 ms [06:22:54] PROBLEM - bacula2 Disk Space on bacula2 is WARNING: DISK WARNING - free space: / 100755 MB (10% inode=99%); [06:23:29] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 102.27 ms [06:45:34] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - guia.cineastas.pt All nameservers failed to answer the query. [06:52:29] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp10.miraheze.org [08:46:05] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 254.16 ms [08:48:08] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 254.13 ms [09:00:39] PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 100% [09:02:41] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 101.47 ms [09:30:31] PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 2.77, 5.36, 2.85 [09:31:15] RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.55 ms [09:32:32] PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.68, 3.67, 2.53 [09:34:32] RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.20, 2.52, 2.24 [09:38:54] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 56853 MB (5% inode=99%); [10:35:39] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 101.82 ms [13:10:42] PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:12:45] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 103.72 ms [13:36:27] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±9] 13https://git.io/JtrLS [13:36:28] [02miraheze/CreateWiki] 07translatewiki 03de7eb07 - Localisation updates from https://translatewiki.net. [13:36:29] [ Main page - translatewiki.net ] - translatewiki.net [13:36:30] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtrL9 [13:36:31] [02miraheze/DataDump] 07translatewiki 03fe675ca - Localisation updates from https://translatewiki.net. [13:36:32] [ Main page - translatewiki.net ] - translatewiki.net [13:36:33] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JtrLH [13:36:34] [02miraheze/landing] 07translatewiki 033516d12 - Localisation updates from https://translatewiki.net. [13:36:35] [ Main page - translatewiki.net ] - translatewiki.net [13:36:36] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtrLQ [13:36:37] [02miraheze/IncidentReporting] 07translatewiki 034c85e3f - Localisation updates from https://translatewiki.net. [13:36:38] [ Main page - translatewiki.net ] - translatewiki.net [13:36:39] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±26] 13https://git.io/JtrL7 [13:36:40] [02miraheze/ManageWiki] 07translatewiki 03a061250 - Localisation updates from https://translatewiki.net. [13:36:41] [ Main page - translatewiki.net ] - translatewiki.net [13:36:42] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtrL5 [13:36:43] [02miraheze/MatomoAnalytics] 07translatewiki 03df872db - Localisation updates from https://translatewiki.net. [13:36:44] ... [13:36:45] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtrLd [13:36:46] [02miraheze/WikiDiscover] 07translatewiki 034b617e0 - Localisation updates from https://translatewiki.net. [13:36:47] ... [13:36:48] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/JtrLF [13:36:49] [02miraheze/MirahezeMagic] 07translatewiki 03d7fc796 - Localisation updates from https://translatewiki.net. [13:36:50] ... [13:37:30] miraheze/CreateWiki - translatewiki the build passed. [13:37:37] miraheze/DataDump - translatewiki the build passed. [13:37:40] miraheze/IncidentReporting - translatewiki the build passed. [13:37:47] miraheze/landing - translatewiki the build passed. [13:37:55] miraheze/MatomoAnalytics - translatewiki the build passed. [13:37:56] miraheze/WikiDiscover - translatewiki the build passed. [13:38:05] miraheze/MirahezeMagic - translatewiki the build passed. [13:38:14] miraheze/ManageWiki - translatewiki the build passed. [15:55:00] paladox, around for me to ask you a quick question? [15:55:13] yes [15:55:16] I ask you because you're basically one of three puppetmasters heh [15:56:13] If we wanted to set up a cron to retrieve updated spam blacklist file(s) from https://www.mediawiki.org/wiki/Extension:StopForumSpam#IP_blocking and run the `updateBlacklist.php` maintenance script, say, on a weekly schedule, would it be best to use a curl command to download the file(s)? [15:56:14] [ Extension:StopForumSpam - MediaWiki ] - www.mediawiki.org [15:59:58] yes [16:03:17] okay, thanks, Reception123's given me a challenge to see if I can figure out how to do that heh [16:28:00] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 100.08 ms [16:29:59] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.78 ms [16:49:01] PROBLEM - dreamsit.com.br - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for dreamsit.com.br could not be found [18:04:25] !log reception@jobrunner3:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki voidlinuxwiki /home/reception/Void+Linux+Wiki-20201015234107.xml [18:04:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:39] PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:27:41] PROBLEM - ping6 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 103.32 ms [19:45:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtrrB [19:45:47] [02miraheze/puppet] 07paladox 03d82d861 - mediawiki::servicessetup: Fix group for mathoid [19:47:59] !log cd /srv ; chwon -R www-data:www-data mathoid on mw, test3 and jobrunner[34] [19:48:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:18] fixed command [20:20:50] paladox: https://phabricator.miraheze.org/T6832 forgetting something with this task? [20:20:51] [ Login ] - phabricator.miraheze.org [20:21:07] oh opening it [20:29:33] PROBLEM - ping4 on dbbackup2 is WARNING: PING WARNING - Packet loss = 0%, RTA = 101.67 ms [20:35:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.03, 5.24, 3.44 [20:35:37] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.72, 6.18, 3.94 [20:37:28] RECOVERY - ping4 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 99.25 ms [20:37:31] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.15, 4.32, 3.31 [20:37:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.80, 4.90, 3.75 [23:11:22] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtrSp [23:11:23] [02miraheze/dns] 07paladox 03c7ea98a - Remove old infra [23:11:25] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbQXl [23:11:26] [02dns] 07paladox opened pull request 03#191: Remove old infra - 13https://git.io/JtrSh [23:12:47] paladox: wait a minute [23:12:54] SPF|Cloud oh sure [23:13:11] there's one little thing you should backup for me [23:13:22] and that is my home directory on cp7 [23:13:54] sure [23:13:57] if I recall correctly there are only three files, no private data [23:13:59] where should i sync [23:14:14] can you run 'ls /home/southparkfan' and PM that to me? [23:14:16] bacula2 is out of storage [23:14:18] sure [23:20:08] (we have backed up the data) [23:20:39] !log shutting down cp6 and deleting the vm [23:20:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:29] !log shutting down graylog1 and deleting the vm [23:37:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:26] !log shutting down services1 and rdb1 and deleting the vm [23:41:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:14] !log shutting down jobrunner1 and mw[45] and deleting the vm [23:42:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:04] !log shutting down mail1 and deleting the vm [23:43:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:31] !log shutting down puppet2 and deleting the vm [23:44:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:08] !log shutting down jobrunner2, test2 and mw[67] and deleting the vm [23:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:41] !log shutting down phab1 and mon1 and deleting the vm [23:54:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:58] !log shutting down gluster1 and ldap1 and deleting the vm [23:55:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:37] !log shutting down rdb1 and services1 and deleting the vm [23:55:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:41] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:17] PROBLEM - wikiverte.pl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:53] PROBLEM - pol.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds