[01:25:23] !log increase gluster1 disk by 50g [01:25:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:56:42] PROBLEM - gluster1 GlusterFS port 24007 on gluster1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:47] PROBLEM - ping4 on gluster1 is CRITICAL: PING CRITICAL - Packet loss = 100% [01:57:35] PROBLEM - gluster1 SSH on gluster1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:01] PROBLEM - gluster1 Current Load on gluster1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:59:05] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:59:48] PROBLEM - Host gluster1 is DOWN: PING CRITICAL - Packet loss = 100% [02:12:29] pala [02:12:31] paladox [02:17:54] hispano76 [02:17:55] Err [02:17:56] Hi [02:17:58] * paladox looks at JL [02:19:36] Don't look at me, I didn't technically cause this [02:20:07] JohnLewis: did you reboot it? [02:20:14] Well it can be reinstalled anyways [02:20:14] yes [02:21:01] and because something can be reinstalled doesn't mean a routine oepration going wrong is permissible [02:22:02] Indeed [06:27:38] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3407 MB (14% inode=94%); [06:50:55] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 9 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:54:45] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvK7K [12:10:09] [02miraheze/services] 07MirahezeSSLBot 037377233 - BOT: Updating services config for wikis [13:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKFT [13:35:09] [02miraheze/services] 07MirahezeSSLBot 030eab0c6 - BOT: Updating services config for wikis [13:50:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKFV [13:50:09] [02miraheze/services] 07MirahezeSSLBot 03f48a525 - BOT: Updating services config for wikis [14:08:44] !log increase gluster1 disk to 720G [14:08:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:11:28] RECOVERY - Host gluster1 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [14:11:32] PROBLEM - gluster1 GlusterFS port 49152 on gluster1 is CRITICAL: connect to address 51.77.107.209 and port 49152: Connection refused [14:12:49] RECOVERY - ping4 on gluster1 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms [14:13:51] PROBLEM - gluster1 GlusterFS port 49152 on gluster1 is CRITICAL: connect to address 51.77.107.209 and port 49152: Connection refused [14:17:55] RECOVERY - gluster1 SSH on gluster1 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [14:23:58] RECOVERY - gluster1 Disk Space on gluster1 is OK: DISK OK - free space: / 686001 MB (99% inode=99%); [14:25:07] PROBLEM - gluster1 Puppet on gluster1 is UNKNOWN: NRPE: Unable to read output [14:26:01] RECOVERY - gluster1 Current Load on gluster1 is OK: OK - load average: 0.54, 0.43, 0.20 [14:31:10] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:32:48] RECOVERY - gluster1 GlusterFS port 24007 on gluster1 is OK: TCP OK - 0.000 second response time on 51.77.107.209 port 24007 [14:35:23] RECOVERY - gluster1 GlusterFS port 49152 on gluster1 is OK: TCP OK - 0.000 second response time on 51.77.107.209 port 49152 [14:45:40] PROBLEM - gluster1 GlusterFS port 49152 on gluster1 is CRITICAL: connect to address 51.77.107.209 and port 49152: Connection refused [14:49:49] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:49:52] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:50:19] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 5 minutes ago with 18 failures. Failed resources (up to 3 shown) [14:50:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:50:46] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:50:46] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:50:47] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 5 minutes ago with 18 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [14:50:52] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 5 minutes ago with 16 failures. Failed resources (up to 3 shown) [14:51:02] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 5 minutes ago with 40 failures. Failed resources (up to 3 shown) [14:51:04] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 240 failures. Last run 5 minutes ago with 240 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [14:51:05] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Puppet has 26 failures. Last run 5 minutes ago with 26 failures. Failed resources (up to 3 shown) [14:51:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:51:15] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 250 failures. Last run 5 minutes ago with 250 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [14:51:15] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:51:40] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:51:52] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:51:55] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:51:55] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:52:05] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:09] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:52:12] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:52:13] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:52:30] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 242 failures. Last run 7 minutes ago with 242 failures. Failed resources (up to 3 shown) [14:53:14] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [14:53:16] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [14:53:45] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:11] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:12] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:14] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:18] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:22] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:25] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:25] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:29] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:54:35] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:54:35] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:03] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:17] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:18] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:19] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:19] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:55:19] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:20] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:55:25] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:55:42] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [14:56:24] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKAc [14:56:25] [02miraheze/mw-config] 07paladox 03dd7a89d - Update Redis.php [14:57:23] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:05:20] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [15:08:30] PROBLEM - puppet2 Disk Space on puppet2 is WARNING: DISK WARNING - free space: / 1464 MB (10% inode=87%); [15:11:00] !log root@puppet2:/opt/puppetlabs/server/data/puppetserver/reports# rm -rf * [15:11:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:13:00] [02mediawiki] 07paladox closed pull request 03#133: Update Genealogy extension to 2.0.1 - 13https://git.io/JvaeW [15:13:02] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_34 [+0/-0/±2] 13https://git.io/JvKAF [15:13:03] [02miraheze/mediawiki] 07samwilson 031169921 - Update Genealogy extension to 2.0.1 [15:13:05] [02miraheze/mediawiki] 07paladox 03fe8e315 - Merge pull request #133 from samwilson/update-genealogy Update Genealogy extension to 2.0.1 [15:14:00] RECOVERY - puppet2 Disk Space on puppet2 is OK: DISK OK - free space: / 7439 MB (55% inode=87%); [15:14:53] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet last ran 16 hours ago [15:25:00] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:26:25] !log reboot test1 [15:26:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:28:37] !log apt-get dist-upgrade - test1 [15:28:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:32:15] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:35:36] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [15:36:07] !log rm -rf /srv/mediawiki/config and let it reclone, merge conflicts so recloning fresh [15:36:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:39:50] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 7 minutes ago with 8 failures. Failed resources (up to 3 shown): Package[postfix],Package[php7.3-cli],Package[php7.3-fpm],Package[php7.3-dev] [15:43:17] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:47:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKxb [15:47:48] [02miraheze/puppet] 07paladox 032ad4f68 - redis: Fix enabling saving RDB [15:47:50] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [15:47:51] [02puppet] 07paladox opened pull request 03#1286: redis: Fix enabling saving RDB - 13https://git.io/JvKxN [15:49:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvKxx [15:49:35] [02miraheze/puppet] 07paladox 0366d2e18 - Update redis.conf.erb [15:49:37] [02puppet] 07paladox synchronize pull request 03#1286: redis: Fix enabling saving RDB - 13https://git.io/JvKxN [15:49:45] [02puppet] 07paladox closed pull request 03#1286: redis: Fix enabling saving RDB - 13https://git.io/JvKxN [15:49:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvKxp [15:49:48] [02miraheze/puppet] 07paladox 031298fd1 - redis: Fix enabling saving RDB (#1286) * redis: Fix enabling saving RDB * Update redis.conf.erb [15:49:49] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [15:49:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [15:55:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvKp8 [15:55:09] [02miraheze/services] 07MirahezeSSLBot 03a202bc5 - BOT: Updating services config for wikis [19:43:35] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%); [20:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv6UT [20:10:09] [02miraheze/services] 07MirahezeSSLBot 0361dbe86 - BOT: Updating services config for wikis [20:41:54] * hispano76 greettings [22:02:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv6km [22:02:38] [02miraheze/puppet] 07paladox 03c5dfa61 - letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge [22:02:39] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:02:48] [02puppet] 07paladox opened pull request 03#1287: letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge - 13https://git.io/Jv6kY [22:03:04] [02puppet] 07paladox synchronize pull request 03#1287: letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge - 13https://git.io/Jv6kY [22:03:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jv6kO [22:03:07] [02miraheze/puppet] 07paladox 03c02b5d4 - Update init.pp [22:03:44] [02puppet] 07paladox closed pull request 03#1287: letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge - 13https://git.io/Jv6kY [22:03:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv6k3 [22:03:47] [02miraheze/puppet] 07paladox 034a3cfb2 - letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge (#1287) * letsencrypt: Symnlink /var/www/challenges to /var/www/.well-known/acme-challenge * Update init.pp [22:03:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [22:03:50] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [22:05:42] [02ssl] 07Pix1234 opened pull request 03#271: Add cert for wiki.teamrelectric.ca - 13https://git.io/Jv6kZ [22:07:44] [02ssl] 07paladox closed pull request 03#271: Add cert for wiki.teamrelectric.ca - 13https://git.io/Jv6kZ [22:07:45] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jv6kc [22:07:47] [02miraheze/ssl] 07Pix1234 03f86d225 - Add cert for wiki.teamrelectric.ca (#271) * Create wiki.teamrelectric.ca.crt * add teamrelectric cert [22:20:18] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:45] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [22:27:47] @paladox wel irccloud decided to die [22:27:57] heh [22:28:17] https://twitter.com/irccloud/status/1237864962657112064?s=21 [22:28:18] [ IRCCloud op Twitter: "We're experiencing some issues with the service at the moment. We're looking into it." ] - twitter.com [22:28:43] lol [22:35:07] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2069 MB (10% inode=93%); [23:30:29] Hello [23:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv6Ip [23:35:09] [02miraheze/services] 07MirahezeSSLBot 0337fc10e - BOT: Updating services config for wikis [23:38:41] Hi! Here is the list of currently open high priority tasks on Phabricator [23:38:48] No updates for 17 days - https://phabricator.miraheze.org/T5258 - Prevent creating Special: Namespace in ManageWiki - authored by RhinosF1, assigned to John [23:39:00] No updates for 12 days - https://phabricator.miraheze.org/T5244 - Perform/write postmortem for the RamNode->OVH migration - authored by RobLa, assigned to None [23:39:07] No updates for 30 days - https://phabricator.miraheze.org/T5222 - MediaWiki response time can fluctuate due to messages - authored by Southparkfan, assigned to None [23:39:14] No updates for 43 days - https://phabricator.miraheze.org/T5174 - enc.for.uz itermittently using both yandex and miraheze dns - authored by Paladox, assigned to Sf7_uz [23:39:21] No updates for 34 days - https://phabricator.miraheze.org/T5130 - Internal error - authored by Kadmin, assigned to None