[00:01:46] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 4055 MB (21% inode=93%); [00:05:25] hum, version 2.4 of the Cargo extension has had minor corrections since January. Perhaps one of those corrections will fix a minor detail that affects a Table. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/Cargo/+log/HEAD?s=ee597a33ba3fbd5b95d142678915eb151292ec35 [00:05:25] [ Log - HEAD - mediawiki/extensions/Cargo - Gitiles ] - gerrit.wikimedia.org [00:26:03] [02puppet] 07paladox closed pull request 03#1289: nutcracker: Switch to rdb1 - 13https://git.io/Jvi9l [00:26:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvihx [00:26:06] [02miraheze/puppet] 07paladox 036f262cc - nutcracker: Switch to rdb1 (#1289) [00:26:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:26:09] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:32:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvijJ [00:32:11] [02miraheze/puppet] 07paladox 03e28041f - varnish: Remove lizardfs6 and us mw[4567] for old infra [00:32:12] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:32:14] [02puppet] 07paladox opened pull request 03#1290: varnish: Remove lizardfs6 and us mw[4567] for old infra - 13https://git.io/JvijU [00:32:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvijL [00:32:41] [02miraheze/puppet] 07paladox 03be8bbee - Update stunnel.conf [00:32:43] [02puppet] 07paladox synchronize pull request 03#1290: varnish: Remove lizardfs6 and us mw[4567] for old infra - 13https://git.io/JvijU [00:33:08] [02puppet] 07paladox closed pull request 03#1290: varnish: Remove lizardfs6 and us mw[4567] for old infra - 13https://git.io/JvijU [00:33:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jvijt [00:33:11] [02miraheze/puppet] 07paladox 031fa10f6 - varnish: Remove lizardfs6 and us mw[4567] for old infra (#1290) * varnish: Remove lizardfs6 and us mw[4567] for old infra * Update stunnel.conf [00:33:13] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:33:14] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:54:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jvijo [00:54:20] [02miraheze/puppet] 07paladox 0352b2df3 - bacula: Backup from gluster1 rather then lizardfs6 [00:54:22] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:54:23] [02puppet] 07paladox opened pull request 03#1291: bacula: Backup from gluster1 rather then lizardfs6 - 13https://git.io/JvijK [00:55:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvijP [00:55:19] [02miraheze/puppet] 07paladox 035bd319b - Update nrpe.cfg.erb [00:55:20] [02puppet] 07paladox synchronize pull request 03#1291: bacula: Backup from gluster1 rather then lizardfs6 - 13https://git.io/JvijK [00:55:34] [02puppet] 07paladox closed pull request 03#1291: bacula: Backup from gluster1 rather then lizardfs6 - 13https://git.io/JvijK [00:55:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jvij1 [00:55:37] [02miraheze/puppet] 07paladox 03da705cb - bacula: Backup from gluster1 rather then lizardfs6 (#1291) * bacula: Backup from gluster1 rather then lizardfs6 * Update nrpe.cfg.erb [00:55:38] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:55:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:56:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvijM [00:56:29] [02miraheze/puppet] 07paladox 033eeb292 - bacula: Switch from backing up db6 to backing up db7 [00:56:30] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:56:32] [02puppet] 07paladox opened pull request 03#1292: bacula: Switch from backing up db6 to backing up db7 - 13https://git.io/JvijD [00:56:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jvijy [00:56:56] [02miraheze/puppet] 07paladox 03440fb7e - Update director.pp [00:56:57] [02puppet] 07paladox synchronize pull request 03#1292: bacula: Switch from backing up db6 to backing up db7 - 13https://git.io/JvijD [00:57:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvijS [00:57:12] [02miraheze/puppet] 07paladox 03046690a - Update nrpe.cfg.erb [00:57:13] [02puppet] 07paladox synchronize pull request 03#1292: bacula: Switch from backing up db6 to backing up db7 - 13https://git.io/JvijD [00:57:24] [02puppet] 07paladox closed pull request 03#1292: bacula: Switch from backing up db6 to backing up db7 - 13https://git.io/JvijD [00:57:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jvij9 [00:57:27] [02miraheze/puppet] 07paladox 03cf1a16e - bacula: Switch from backing up db6 to backing up db7 (#1292) * bacula: Switch from backing up db6 to backing up db7 * Update director.pp * Update nrpe.cfg.erb [00:57:28] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:57:30] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [01:00:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvijN [01:00:49] [02miraheze/puppet] 07paladox 0378e8170 - redis: Increase by 200MB [01:02:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb [01:05:23] PROBLEM - lizardfs6 GlusterFS port 49152 on lizardfs6 is CRITICAL: connect to address 54.36.165.161 and port 49152: Connection refused [01:06:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:08:19] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:08:34] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 231 failures. Last run 5 minutes ago with 231 failures. Failed resources (up to 3 shown) [01:08:47] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 6 minutes ago with 17 failures. Failed resources (up to 3 shown) [01:08:49] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [01:08:56] PROBLEM - bacula2 Bacula Databases db6 on bacula2 is UNKNOWN: NRPE: Command 'check_bacula_databasesdb6' not defined [01:09:03] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 6 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:09:22] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:09:26] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 6 minutes ago with 18 failures. Failed resources (up to 3 shown) [01:09:27] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:09:41] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 7 minutes ago with 40 failures. Failed resources (up to 3 shown) [01:09:53] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 7 minutes ago with 17 failures. Failed resources (up to 3 shown) [01:10:10] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:10:17] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 7 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:10:24] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 7 minutes ago with 17 failures. Failed resources (up to 3 shown) [01:10:28] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 7 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:10:34] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 8 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:10:56] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:10:57] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 232 failures. Last run 8 minutes ago with 232 failures. Failed resources (up to 3 shown) [01:10:58] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 20 failures. Last run 8 minutes ago with 20 failures. Failed resources (up to 3 shown) [01:10:58] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.conf] [01:11:00] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:11:13] PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 8 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:11:24] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 9 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:11:39] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:11:50] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 246 failures. Last run 9 minutes ago with 246 failures. Failed resources (up to 3 shown) [01:12:14] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [01:12:22] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 247 failures. Last run 9 minutes ago with 247 failures. Failed resources (up to 3 shown) [01:12:29] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [01:12:40] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:12:54] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:13:11] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:13:13] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:13:30] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:13:41] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:13:54] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:04] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:06] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:07] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:19] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:33] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:34] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:35] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:35] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:14:46] RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:56] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:59] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:15:09] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:15:32] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:16:20] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:50:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPfT [02:50:09] [02miraheze/services] 07MirahezeSSLBot 03fc0a50e - BOT: Updating services config for wikis [05:09:39] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 100.03 ms [05:12:38] RECOVERY - ping4 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 99.87 ms [06:26:20] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3325 MB (13% inode=94%); [08:10:57] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-51.89.160.136-to-any-port-443] [08:13:55] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:42:01] Hello BurningPrincess1! If you have any questions, feel free to ask and someone should answer soon. [10:42:18] I wanted to say that Miraheze is the best wiki host I have ever been too [10:42:58] And I noted a lack of 503 errors that is good [12:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPOy [12:15:09] [02miraheze/services] 07MirahezeSSLBot 03fb52c82 - BOT: Updating services config for wikis [13:16:27] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:19:33] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15356 bytes in 0.552 second response time [13:38:42] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw7 [13:42:14] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [14:06:11] PROBLEM - cp6 Stunnel Http for mw5 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:06:23] PROBLEM - cp7 Stunnel Http for mw5 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:09:30] RECOVERY - cp6 Stunnel Http for mw5 on cp6 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.005 second response time [14:09:39] RECOVERY - cp7 Stunnel Http for mw5 on cp7 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.004 second response time [14:49:02] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPZy [14:49:04] [02miraheze/ManageWiki] 07translatewiki 03ee9eb57 - Localisation updates from https://translatewiki.net. [14:49:05] [ Main page - translatewiki.net ] - translatewiki.net. [15:07:27] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb [15:08:29] PROBLEM - cp3 Stunnel Http for mw7 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:11:26] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw4 mw5 [15:12:21] !log restart php7.3-fpm on mw* [15:12:47] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw6 [15:12:47] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw6 [15:12:51] RECOVERY - cp3 Stunnel Http for mw7 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 0.774 second response time [15:13:09] PROBLEM - cp3 Stunnel Http for mw6 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPnW [15:13:30] [02miraheze/puppet] 07paladox 035a720a7 - Update default.vcl [15:15:23] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw6 [15:16:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPnB [15:16:11] [02miraheze/puppet] 07paladox 03f63cf30 - Revert "Update default.vcl" This reverts commit 5a720a7f09c7d7bf906dbc6b6611d4fd64349b30. [15:16:36] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [15:16:37] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 11 backends are healthy [15:17:01] RECOVERY - cp3 Stunnel Http for mw6 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15330 bytes in 4.339 second response time [15:22:18] PROBLEM - mw4 MediaWiki Rendering on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:19] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [15:23:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPna [15:24:00] [02miraheze/puppet] 07paladox 03428636f - Update php.pp [15:25:57] RECOVERY - mw4 MediaWiki Rendering on mw4 is OK: HTTP OK: HTTP/1.1 200 OK - 18699 bytes in 0.655 second response time [15:27:23] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [15:31:08] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:58:45] [02mw-config] 07cmgi201 opened pull request 03#2944: Add MIME extensions - 13https://git.io/JvPcE [17:03:44] [02mw-config] 07Pix1234 commented on pull request 03#2944: Add MIME extensions - 13https://git.io/JvP8Y [17:12:05] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw5 mw7 [17:15:14] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [17:39:04] [02mw-config] 07cmgi201 commented on pull request 03#2944: Add MIME extensions - 13https://git.io/JvP4k [17:46:40] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --no-local-users --username-prefix="리버티게임" --wiki libertygamewiki 22115932221_w-20200309-history.xml on jobrunner1 [17:47:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [17:48:17] should of done --report=1 but meh [18:25:07] * hispano76 greetings [18:40:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPBj [18:40:09] [02miraheze/services] 07MirahezeSSLBot 03fd24560 - BOT: Updating services config for wikis [18:50:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvPRg [18:50:10] [02miraheze/services] 07MirahezeSSLBot 039ad6891 - BOT: Updating services config for wikis [19:50:40] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [21:33:00] uh [21:35:45] Hi hispano76 [22:41:30] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw5 mw7 [22:44:38] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy