[00:00:33] PROBLEM - cp6 Disk Space on cp6 is WARNING: DISK WARNING - free space: / 2054 MB (10% inode=93%); [00:03:50] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 3514 MB (18% inode=93%); [00:06:10] PROBLEM - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:08:02] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 3987 files, 11.59MB, 2020-03-29 00:05:00 (3.0 minutes ago) [00:09:20] RECOVERY - cp7 HTTP 4xx/5xx ERROR Rate on cp7 is OK: OK - NGINX Error Rate is 5% [00:11:57] paladox: I've just seen https://phabricator.miraheze.org/rPUPC9554ff440bb3a3eda5010d3b282367b3c97910b3 [00:11:59] [ rPUPC9554ff440bb3 ] - phabricator.miraheze.org [00:12:02] can't this be done with hiera? [00:12:13] Hard coding hostnames in modules is bad and we shouldn't be doing it [00:12:21] oh, bugger, yess [00:12:31] i'll do that in a bit after i'm finished [00:12:42] third batch finished now! [00:14:01] now restoring mhglobal/centralauth [00:18:11] [02mw-config] 07paladox closed pull request 03#2957: Switch default db connection to dbt1 - 13https://git.io/JvQRC [00:18:13] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQwC [00:18:14] [02miraheze/mw-config] 07paladox 03de3202e - Switch default db connection to dbt1 (#2957) [00:18:16] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbvb3 [00:18:17] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [00:19:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQwl [00:19:38] [02miraheze/puppet] 07paladox 03ac5dea1 - Varnish: Remove mw[123] as a backend [00:22:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQwE [00:22:28] [02miraheze/mw-config] 07paladox 037c0a280 - Take dbt1 wikis out of read only [00:22:30] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:22:31] [02mw-config] 07paladox opened pull request 03#2958: Take dbt1 wikis out of read only - 13https://git.io/JvQwu [00:26:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQwr [00:26:44] [02miraheze/puppet] 07paladox 039525f88 - Update default.vcl [00:30:42] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 8 backends are healthy [00:31:04] [02mw-config] 07paladox closed pull request 03#2958: Take dbt1 wikis out of read only - 13https://git.io/JvQwu [00:31:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQwM [00:31:07] [02miraheze/mw-config] 07paladox 03d57268d - Take dbt1 wikis out of read only (#2958) [00:31:23] RECOVERY - cp7 Varnish Backends on cp7 is OK: All 8 backends are healthy [00:31:38] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Puppet has 356 failures. Last run 2 minutes ago with 356 failures. Failed resources (up to 3 shown): Exec[varnish-server-syntax] [00:31:42] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:32:09] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQwS [00:32:11] [02miraheze/mw-config] 07paladox 03a8a8099 - Remove sitenotice [00:33:05] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 192 failures. Last run 26 seconds ago with 192 failures. Failed resources (up to 3 shown): File[/etc/apt/apt.conf.d/20auto-upgrades],File[/root/ufw-fix],File[/usr/local/bin/gen_fingerprints],File[/etc/vim/vimrc.local] [00:33:19] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:33:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQw7 [00:33:55] [02miraheze/puppet] 07paladox 03d8e9710 - varnish: Properly remove mw[123] [00:33:57] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:33:59] [02puppet] 07paladox opened pull request 03#1300: varnish: Properly remove mw[123] - 13https://git.io/JvQw5 [00:34:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQwF [00:34:26] [02miraheze/puppet] 07paladox 03cafbeea - Update stunnel.conf [00:34:27] [02puppet] 07paladox synchronize pull request 03#1300: varnish: Properly remove mw[123] - 13https://git.io/JvQw5 [00:34:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQwN [00:34:47] [02miraheze/puppet] 07paladox 03170b10f - Update init.pp [00:34:48] [02puppet] 07paladox synchronize pull request 03#1300: varnish: Properly remove mw[123] - 13https://git.io/JvQw5 [00:34:59] [02puppet] 07paladox closed pull request 03#1300: varnish: Properly remove mw[123] - 13https://git.io/JvQw5 [00:35:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvQwp [00:35:02] [02miraheze/puppet] 07paladox 037ad9b32 - varnish: Properly remove mw[123] (#1300) * varnish: Properly remove mw[123] * Update stunnel.conf * Update init.pp [00:35:22] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:35:23] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:35:47] PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:35:50] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:00] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 4 minutes ago with 16 failures. Failed resources (up to 3 shown) [00:36:03] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQwj [00:36:05] [02miraheze/dns] 07paladox 031e60aa6 - Remove cnams for cp7 [00:36:06] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [00:36:06] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:08] [02dns] 07paladox opened pull request 03#140: Remove cnames for cp7 - 13https://git.io/JvQre [00:36:08] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:08] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:09] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:27] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 251 failures. Last run 4 minutes ago with 251 failures. Failed resources (up to 3 shown) [00:36:30] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:36:58] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 5 minutes ago with 15 failures. Failed resources (up to 3 shown) [00:37:01] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:02] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:03] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQrv [00:37:05] [02miraheze/dns] 07paladox 033373d79 - Update nonbinary.wiki [00:37:06] [02dns] 07paladox synchronize pull request 03#140: Remove cnames for cp7 - 13https://git.io/JvQre [00:37:15] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:21] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:24] PROBLEM - dbt1 Puppet on dbt1 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 5 minutes ago with 17 failures. Failed resources (up to 3 shown) [00:37:24] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 18 failures. Last run 5 minutes ago with 18 failures. Failed resources (up to 3 shown) [00:37:24] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:25] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:34] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:37:42] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQrJ [00:37:44] [02miraheze/dns] 07paladox 03a30174b - Update nenawiki.org [00:37:45] [02dns] 07paladox synchronize pull request 03#140: Remove cnames for cp7 - 13https://git.io/JvQre [00:37:59] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:06] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:38:08] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:38:13] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:38:14] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:38:17] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:38:19] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [00:38:21] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQrU [00:38:22] [02miraheze/dns] 07paladox 0310eae0d - Update allthetropes.org [00:38:24] [02dns] 07paladox synchronize pull request 03#140: Remove cnames for cp7 - 13https://git.io/JvQre [00:38:24] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 251 failures. Last run 6 minutes ago with 251 failures. Failed resources (up to 3 shown) [00:38:24] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:38:36] [02dns] 07paladox closed pull request 03#140: Remove cnames for cp7 - 13https://git.io/JvQre [00:38:38] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JvQrT [00:38:39] [02miraheze/dns] 07paladox 03783d527 - Remove cnames for cp7 (#140) * Remove cnams for cp7 * Update nonbinary.wiki * Update nenawiki.org * Update allthetropes.org [00:38:41] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [00:38:42] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [00:38:47] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy [00:38:56] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:39:34] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:40:07] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [00:40:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQrt [00:41:00] [02miraheze/puppet] 07paladox 03b200eda - mediawiki: Increase childs to 12 [00:41:50] PROBLEM - cp7 Stunnel Http for mw1 on cp7 is CRITICAL: connect to address localhost and port 8080: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:42:06] PROBLEM - cp8 Stunnel Http for mw1 on cp8 is CRITICAL: connect to address localhost and port 8080: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:42:12] PROBLEM - cp8 Stunnel Http for mw2 on cp8 is CRITICAL: connect to address localhost and port 8081: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:42:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [00:42:22] PROBLEM - cp6 Stunnel Http for mw2 on cp6 is CRITICAL: connect to address localhost and port 8081: Connection refusedHTTP CRITICAL - Unable to open TCP socket [00:42:34] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [00:42:37] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [00:42:57] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [00:43:01] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [00:43:38] !log restart php-fpm on mw* [00:43:40] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:43:43] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:43:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:43:59] RECOVERY - dbt1 Puppet on dbt1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:43:59] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:09] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:10] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:22] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:27] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:34] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:34] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:36] RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:47] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:47] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:44:53] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:57] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:57] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:44:58] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:45:11] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:45:19] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:45:59] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:54:58] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:01:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQrB [01:01:53] [02miraheze/puppet] 07paladox 035c7a7e9 - nginx: Remove buffering of logs [01:14:51] !log delete all wikis on db4 that have been migrated to dbt1 [01:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:17:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQry [01:17:42] [02miraheze/mw-config] 07paladox 037d1298d - Revert "Disable createwiki (#2954)" This reverts commit e02d58406e6128cd4743fcf786c83e56bfe38fa9. [01:18:24] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown) [01:18:45] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown) [01:18:54] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown) [01:19:00] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown) [01:21:33] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 39 failures. Last run 4 minutes ago with 39 failures. Failed resources (up to 3 shown): File[dariawiki.org],File[dariawiki.org_private],File[pwiki.arkcls.com],File[pwiki.arkcls.com_private] [01:22:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQrA [01:22:32] [02miraheze/puppet] 07paladox 03243c58e - puppet2: Increase heap by 500M [01:22:33] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:22:36] [02puppet] 07paladox opened pull request 03#1301: puppet2: Increase heap by 500M - 13https://git.io/JvQrx [01:24:48] !log add 1g swap to puppet2 [01:24:52] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [01:24:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:25:14] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:25:20] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:25:25] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [01:27:47] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:31:11] !log increased puppet2 ram by 1gb and removed swap [01:31:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:32:10] [02puppet] 07paladox closed pull request 03#1301: puppet2: Increase heap by 500M - 13https://git.io/JvQrx [01:32:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQon [01:32:13] [02miraheze/puppet] 07paladox 038e71ebd - puppet2: Increase heap by 500M (#1301) [01:32:15] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:32:16] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [01:35:35] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:35:49] PROBLEM - phab1 Puppet on phab1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:18] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:18] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:37] PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 16 failures. Last run 5 minutes ago with 16 failures. Failed resources (up to 3 shown) [01:36:40] PROBLEM - rdb1 Puppet on rdb1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:40] PROBLEM - db7 Puppet on db7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:46] PROBLEM - services2 Puppet on services2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:47] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:48] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 5 minutes ago with 15 failures. Failed resources (up to 3 shown) [01:36:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQoa [01:36:56] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:56] [02miraheze/puppet] 07paladox 03bc1872d - Remove mw[123] [01:36:57] PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:36:57] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [01:36:59] [02puppet] 07paladox opened pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:37:00] PROBLEM - rdb2 Puppet on rdb2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:02] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 236 failures. Last run 5 minutes ago with 236 failures. Failed resources (up to 3 shown) [01:37:03] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:03] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 251 failures. Last run 4 minutes ago with 251 failures. Failed resources (up to 3 shown) [01:37:09] PROBLEM - db6 Puppet on db6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:09] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 5 minutes ago with 40 failures. Failed resources (up to 3 shown) [01:37:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQow [01:37:32] [02miraheze/puppet] 07paladox 038ded9c1 - Update firewall.yaml [01:37:34] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:37:40] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:41] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:54] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:55] PROBLEM - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [01:37:55] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:37:56] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:38:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQoi [01:38:09] PROBLEM - dbt1 Puppet on dbt1 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 6 minutes ago with 17 failures. Failed resources (up to 3 shown) [01:38:09] [02miraheze/puppet] 07paladox 03ebacc6d - Update varnish.pp [01:38:11] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:38:12] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:38:12] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:38:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 7 minutes ago with 15 failures. Failed resources (up to 3 shown) [01:38:22] PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:38:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQoX [01:38:31] [02miraheze/puppet] 07paladox 03c63599d - Update irc.pp [01:38:32] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:38:35] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 7 minutes ago with 23 failures. Failed resources (up to 3 shown) [01:38:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQo1 [01:38:56] [02miraheze/puppet] 07paladox 03a3ce586 - Update db.pp [01:38:57] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:39:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQoy [01:39:23] [02miraheze/puppet] 07paladox 0373fc1d5 - Update services.pp [01:39:24] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:39:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQoS [01:39:46] [02miraheze/puppet] 07paladox 03541fbdf - Update nrpe.cfg.erb [01:39:47] [02puppet] 07paladox synchronize pull request 03#1302: Remove mw[123] - 13https://git.io/JvQoV [01:40:04] [02puppet] 07paladox closed pull request 03#1261: switch cp[248] to use mw[4567] - 13https://git.io/Jv0vp [01:40:06] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [01:40:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [01:40:12] [02puppet] 07paladox closed pull request 03#1247: varnish: Use mw[4567] only on cp7 - 13https://git.io/Jv4L5 [01:40:15] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://git.io/vbiAS [01:40:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [01:40:27] [02puppet] 07paladox closed pull request 03#1230: gluster: Remove mw[123] and lizardfs6 from firewall - 13https://git.io/JvlED [01:40:30] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [01:40:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [01:42:48] RECOVERY - rdb1 Puppet on rdb1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [01:42:52] RECOVERY - bacula2 Puppet on bacula2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:42:55] RECOVERY - cloud2 Puppet on cloud2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:42:58] RECOVERY - db7 Puppet on db7 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [01:43:05] RECOVERY - services2 Puppet on services2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [01:43:09] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [01:43:18] RECOVERY - db6 Puppet on db6 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [01:43:23] RECOVERY - cp7 Puppet on cp7 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:43:27] RECOVERY - rdb2 Puppet on rdb2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [01:43:38] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [01:44:16] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:21] RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [01:44:21] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:23] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:38] RECOVERY - dbt1 Puppet on dbt1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:44:38] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:44:39] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [01:44:43] PROBLEM - dbt1 Current Load on dbt1 is CRITICAL: CRITICAL - load average: 23.15, 15.46, 8.12 [01:44:56] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:44:56] RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [01:45:06] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:45:06] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:45:18] RECOVERY - phab1 Puppet on phab1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:45:34] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:45:40] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:46:11] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:46:17] RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:46:30] RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:46:36] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [01:46:58] RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:47:23] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [02:05:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKK [02:05:03] [02miraheze/puppet] 07paladox 038858221 - Remove db4 [02:05:04] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [02:05:06] [02puppet] 07paladox opened pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:05:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKi [02:05:38] [02miraheze/puppet] 07paladox 03d2690bd - Update main.pp [02:05:40] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:06:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQK1 [02:06:19] [02miraheze/puppet] 07paladox 03efe13d1 - Update dbreplication.pp [02:06:20] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:06:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKM [02:06:40] [02miraheze/puppet] 07paladox 03793ddd5 - Update main.pp [02:06:41] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:06:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKy [02:06:56] [02miraheze/puppet] 07paladox 031576859 - Update init.pp [02:06:58] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:07:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQK9 [02:07:19] [02miraheze/puppet] 07paladox 0321fd343 - Update groups.conf [02:07:20] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:07:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKH [02:07:38] [02miraheze/puppet] 07paladox 03f1cd848 - Update db.pp [02:07:39] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:07:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQK5 [02:07:55] [02miraheze/puppet] 07paladox 03f9b4385 - Update nrpe.cfg.erb [02:07:56] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:08:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKb [02:08:15] [02miraheze/puppet] 07paladox 034b3a1e0 - Update init.pp [02:08:17] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:08:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKA [02:08:48] [02miraheze/puppet] 07paladox 0331e9f52 - Update director.pp [02:08:50] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:09:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JvQKp [02:09:04] [02miraheze/puppet] 07paladox 039c4e852 - Update init.pp [02:09:06] [02puppet] 07paladox synchronize pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [02:10:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JvQKj [02:10:46] [02miraheze/puppet] 07paladox 0399c2ebb - Switch roundcubemail to db7 [02:10:47] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [02:10:49] [02puppet] 07paladox opened pull request 03#1304: Switch roundcubemail to db7 - 13https://git.io/JvQ6e [02:11:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/JvQ6v [02:11:10] [02miraheze/puppet] 07paladox 03227affc - Update main.pp [02:11:11] [02puppet] 07paladox synchronize pull request 03#1304: Switch roundcubemail to db7 - 13https://git.io/JvQ6e [02:11:16] [02puppet] 07paladox closed pull request 03#1304: Switch roundcubemail to db7 - 13https://git.io/JvQ6e [02:11:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JvQ6J [02:11:19] [02miraheze/puppet] 07paladox 0315e7d79 - Switch roundcubemail to db7 (#1304) * Switch roundcubemail to db7 * Update main.pp [02:11:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [02:11:22] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [02:14:11] Hello drnslyng! If you have any questions, feel free to ask and someone should answer soon. [02:24:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ6w [02:25:00] [02miraheze/puppet] 07paladox 03b53ef5a - Update default-grants.sql.erb [02:38:17] PROBLEM - dbt1 Current Load on dbt1 is WARNING: WARNING - load average: 4.19, 5.03, 7.34 [02:38:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQiu [02:38:36] [02miraheze/puppet] 07paladox 03c359454 - Update site.pp [02:39:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQiw [02:39:24] [02miraheze/dns] 07paladox 03a4cb16b - Remove lizardfs6 [02:40:27] !log puppetserver ca clean --certname lizardfs6.m.org/mw3.m.org [02:40:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:41:10] RECOVERY - dbt1 Current Load on dbt1 is OK: OK - load average: 3.26, 4.27, 6.65 [02:44:34] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:45:59] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:35:06] Is this an acceptable place to report a technical problem? [03:59:49] drnslyng: yes, it might not get fixed while the morning though [04:24:59] I get the following error — [04:25:00] for both of two Miraheze accounts I have, Drnslyng and Kualeyi522912. Both have at least the Administrator role on nilamwikiubzx217c40wiki and Drnslyng is at least an Administrator on marwiki. Unfortunately, I can't use Phabricator for this bug as I hadn't already linked an account to Phabricator and when I try to do so, the same error occurs. [04:25:00] there a way I can regain access? Thanks. [04:25:07] ^ also on https://meta.miraheze.org/wiki/Stewards%27_noticeboard [04:25:08] [ Stewards' noticeboard - Miraheze Meta ] - meta.miraheze.org [04:28:45] ^^^ Update: resolved (not solved). Thanks for reading. It'd be great if someone could explain why the problem happened, to maybe give me a better understanding of what to do and not to do in future (if in fact it's something wrong with me). [04:35:32] PROBLEM - ping4 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 106.58 ms [04:38:18] PROBLEM - ping4 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 105.91 ms [04:42:43] [02puppet] 07paladox closed pull request 03#1303: Remove db4 - 13https://git.io/JvQK6 [04:42:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±9] 13https://git.io/JvQ1o [04:42:46] [02miraheze/puppet] 07paladox 03b793075 - Remove db4 (#1303) [04:42:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [04:42:49] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [04:45:33] RECOVERY - dbt1 MySQL on dbt1 is OK: Uptime: 132728 Threads: 25 Questions: 26039762 Slow queries: 389799 Opens: 680458 Flush tables: 1 Open tables: 50000 Queries per second avg: 196.188 [04:47:47] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:52:58] !log stop mysql on db4 [04:53:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [04:53:57] !log restart logbot on mon1 [04:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:08:16] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [05:14:13] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:28:31] PROBLEM - gluster1 Puppet on gluster1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [05:34:08] RECOVERY - gluster1 Puppet on gluster1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:11:10] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.18, 1.78, 1.52 [06:14:03] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.41, 1.61, 1.50 [06:28:47] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3384 MB (14% inode=93%); [07:58:13] !log reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=sagan4alphawiki --search-recursively --skip-dupes /home/reception/images [07:58:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [08:00:15] morning Reception123 [08:00:35] morning [08:29:09] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.16, 1.71, 1.49 [08:31:59] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.72, 1.67, 1.51 [08:34:49] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.51, 1.66, 1.54 [09:13:00] Hello kiwi_32! If you have any questions, feel free to ask and someone should answer soon. [09:13:43] Reception123 Miraheze Toolforge when is here??? [09:13:53] RhinosF1 Miraheze Toolforge when is here??? [09:17:50] Hello MirahezeToolforg! If you have any questions, feel free to ask and someone should answer soon. [09:18:02] Reception123 Why you ban me??? [10:19:31] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±2] 13https://git.io/JvQ5e [10:19:33] [02miraheze/ssl] 07Reception123 033805c7c - add wiki.velaan.org cert [10:19:34] [02miraheze/ssl] 07Reception123 03146b76a - add wiki.counterculturelabs.org cert [10:25:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ5l [10:25:09] [02miraheze/services] 07MirahezeSSLBot 03d315dea - BOT: Updating services config for wikis [10:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ5E [10:30:09] [02miraheze/services] 07MirahezeSSLBot 037adcd9d - BOT: Updating services config for wikis [10:40:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ56 [10:40:10] [02miraheze/services] 07MirahezeSSLBot 034a6c79c - BOT: Updating services config for wikis [11:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQdn [11:10:09] [02miraheze/services] 07MirahezeSSLBot 03689fb07 - BOT: Updating services config for wikis [11:56:33] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.06, 1.69, 1.49 [11:59:18] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.60, 1.61, 1.49 [14:25:00] !log reception@mw4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt [14:25:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:30:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQxZ [14:30:09] [02miraheze/services] 07MirahezeSSLBot 0332b24e1 - BOT: Updating services config for wikis [14:48:29] !log decrease puppet2 vcpu by one [14:48:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:50:10] !log increased mw[45] vpcu by 1 [14:50:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:50:42] !log depool mw4 [14:50:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:51:15] !log repool mw4 [14:51:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:51:25] paladox: oh, I was running a script there :( [14:51:31] oh [14:51:35] guess I'll go to mw6 [14:51:38] Reception123 what was the script? [14:51:44] Reception123 no, this is my fault [14:51:47] paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -r "Requested - T5324" /home/reception/cgwdel.txt [14:51:49] you can start it on mw4 now :) [14:51:52] *again [14:51:54] paladox: ok, thanks :) [14:52:03] it's a really long one, many pages to delete [14:52:13] Reception123 are you running anything on mw5? [14:53:30] !log depool mw5 [14:53:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:54:04] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:54:13] PROBLEM - cloud2 Puppet on cloud2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:54:33] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:54:35] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:54:38] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:54:42] Reception123: jobrunner1 is the only place to run scripts unless needed please [14:54:50]