[00:02:26] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3796 MB (19% inode=93%); [00:11:09] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvupp [00:11:11] [02miraheze/dns] 07paladox 0377f0280 - cache: Switch to cp6 from cp4 cp6 still uses mw[123] so this is a safe switch in so far as no majour performance degradation. [00:11:12] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [00:11:14] [02dns] 07paladox opened pull request 03#137: cache: Switch to cp6 from cp4 - 13https://git.io/Jvuph [00:26:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvuhS [00:26:48] [02miraheze/puppet] 07paladox 032044154 - Fix puppet on mon1 [00:26:49] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:26:51] [02puppet] 07paladox opened pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:27:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvuhF [00:27:47] [02miraheze/puppet] 07paladox 035e80395 - Update init.pp [00:27:48] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:29:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvuhj [00:29:01] [02miraheze/puppet] 07paladox 03b1b0693 - Update init.pp [00:29:01] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2645 MB (10% inode=94%); [00:29:03] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:32:18] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:32:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvujm [00:32:21] [02miraheze/puppet] 07paladox 037e5352f - Update init.pp [00:32:41] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:32:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvujO [00:32:44] [02miraheze/puppet] 07paladox 03e8ff9ca - Update init.pp [00:33:02] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:33:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvuj3 [00:33:05] [02miraheze/puppet] 07paladox 03788d5b1 - Update init.pp [00:36:18] PROBLEM - bacula2 Bacula Databases db4 on bacula2 is WARNING: WARNING: Diff, 75994 files, 41.57GB, 2020-01-30 19:53:00 (3.9 weeks ago) [00:36:24] PROBLEM - bacula2 Bacula Databases db5 on bacula2 is WARNING: WARNING: Diff, 478 files, 63.37GB, 2020-01-30 21:00:00 (3.9 weeks ago) [00:36:26] PROBLEM - bacula2 Bacula Static on bacula2 is CRITICAL: CRITICAL: no terminated jobs [00:37:54] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:43:00] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:43:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvujW [00:43:03] [02miraheze/puppet] 07paladox 0300f7fd2 - Update init.pp [00:43:16] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:43:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvuj8 [00:43:19] [02miraheze/puppet] 07paladox 03b081863 - Update init.pp [00:43:35] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:43:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvuj4 [00:43:38] [02miraheze/puppet] 07paladox 039d2c384 - Update init.pp [00:43:41] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:43:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/JvujB [00:44:00] [02miraheze/puppet] 07paladox 03da54c10 - Update mon1.yaml [00:44:02] [02puppet] 07paladox synchronize pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:44:24] [02puppet] 07paladox closed pull request 03#1265: Fix puppet on mon1 - 13https://git.io/JvuhQ [00:44:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jvuj0 [00:44:27] [02miraheze/puppet] 07paladox 0320d3212 - Fix puppet on mon1 (#1265) [00:45:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvujz [00:45:20] [02miraheze/puppet] 07paladox 03c382662 - Update init.pp [00:48:35] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:27] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:52:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [00:52:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvujw [00:52:47] [02miraheze/puppet] 07paladox 034ce6e3c - Update init.pp [00:54:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvujr [00:54:34] [02miraheze/puppet] 07paladox 03e37fcc4 - Update init.pp [00:58:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvuji [00:58:31] [02miraheze/puppet] 07paladox 033c5440f - Fix [01:25:59] [02dns] 07paladox closed pull request 03#137: cache: Switch to cp6 from cp4 - 13https://git.io/Jvuph [01:26:01] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvze2 [01:26:02] [02miraheze/dns] 07paladox 03c6b8b73 - cache: Switch to cp6 from cp4 (#137) cp6 still uses mw[123] so this is a safe switch in so far as no majour performance degradation. [01:26:04] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbQXl [01:26:05] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [01:30:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvzew [01:30:24] [02miraheze/puppet] 07paladox 03b5ae258 - misc1: Remove dns From https://grafana.miraheze.org/d/ldrByI1iz/dns?orgId=1 i see no hits hitting it anymore [01:30:25] [ Grafana ] - grafana.miraheze.org [02:27:54] RECOVERY - bacula2 Bacula Databases db4 on bacula2 is OK: OK: Full, 1072605 files, 46.70GB, 2020-02-27 02:25:00 (2.9 minutes ago) [03:03:20] RECOVERY - bacula2 Bacula Databases db5 on bacula2 is OK: OK: Full, 3069 files, 47.08GB, 2020-02-27 03:03:00 (20.0 seconds ago) [06:25:11] PROBLEM - services1 Puppet on services1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[nagios-plugins] [06:26:28] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3265 MB (13% inode=94%); [06:31:04] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:35:15] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:43:50] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:46:30] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown) [09:52:08] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [11:25:35] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz3R [11:25:36] [02miraheze/mw-config] 07Reception123 03ffc6fe8 - simswiki readonly for db migration [11:30:04] [02miraheze/ManageWiki] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz3a [11:30:05] [02miraheze/ManageWiki] 07Pix1234 0361bd84a - Add security vulnerablity section to readme [11:31:23] [02miraheze/CreateWiki] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz3V [11:31:25] [02miraheze/CreateWiki] 07Pix1234 03d2fed37 - Add security vulnerable section to README [11:31:47] [02CreateWiki] 07Pix1234 commented on commit 03d2fed376a703f4a647a65068d29e9e641d68d91a - 13https://git.io/Jvz3w [11:38:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz31 [11:38:20] [02miraheze/mw-config] 07Reception123 036f57209 - simswiki to db5 (c2) [11:48:13] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz3N [11:48:14] [02miraheze/mw-config] 07Reception123 03d0706d5 - rm readonly [11:48:49] !log moved simswiki from db4 to db5 [11:48:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:50:24] paladox: well that's 14GB on db4, should last us for a bit more [12:05:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvzsl [12:05:09] [02miraheze/services] 07MirahezeSSLBot 03bbaee7b - BOT: Updating services config for wikis [12:46:24] Heh [14:09:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [14:09:43] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:09:58] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:10:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [14:10:28] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw1 [14:11:26] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw1 [14:11:54] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [14:12:36] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18700 bytes in 0.880 second response time [14:13:12] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [14:14:23] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 11 backends are healthy [14:14:51] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [14:15:11] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:15:41] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18701 bytes in 0.883 second response time [14:16:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:06:28] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw2 [15:09:18] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 11 backends are healthy [16:04:04] !log reception@mw2:~/graalmilitary$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=gmcwiki --search-recursively --skip-dupes /home/reception/graalmilitary/wikipics [16:04:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:22:56] PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.56, 2.79, 1.72 [16:25:47] PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.62, 1.79, 1.51 [16:28:20] RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.48, 1.27, 1.35 [16:31:34] PROBLEM - wiki.archman.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.archman.de' expires in 15 day(s) (Sat 14 Mar 2020 04:27:41 PM GMT +0000). [16:31:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvz87 [16:31:51] [02miraheze/ssl] 07MirahezeSSLBot 0317bb9a0 - Bot: Update SSL cert for wiki.archman.de [16:42:36] RECOVERY - wiki.archman.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.archman.de' will expire on Wed 27 May 2020 03:31:43 PM GMT +0000. [16:59:23] Hello QuiQuiQui! If you have any questions, feel free to ask and someone should answer soon. [16:59:44] I want admin! [16:59:49] Give me admin [16:59:55] NOW! [16:59:57] Right now [16:59:59] Admin [17:00:01] Admin [17:00:03] Admin [17:00:06] Admin [17:06:35] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=94%); [17:07:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.77.107.210/cpweb, 2001:41d0:800:1056::2/cpweb, 51.161.32.127/cpweb, 2607:5300:205:200::17f6/cpweb [17:08:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [17:08:12] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. lizardfs6 [17:08:16] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. lizardfs6 [17:10:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:10:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:11:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 11 backends are healthy [17:11:04] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [17:15:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvzBz [17:15:17] [02miraheze/puppet] 07paladox 03769635a - Update mon1.yaml [17:24:37] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:31:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvzBx [17:31:09] [02miraheze/puppet] 07paladox 0351f6293 - Fix [17:33:17] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:34:26] PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2114 MB (10% inode=93%); [22:15:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvzwg [22:15:09] [02miraheze/services] 07MirahezeSSLBot 0352830f7 - BOT: Updating services config for wikis [22:42:26] !log root@db4:/home/paladox# sysctl net.ipv4.tcp_tw_reuse=1 [22:42:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:44:01] !log restart php.7-3-fpm on mw* & lizardfs* [22:44:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:47:44] !log root@db4:/home/paladox# sysctl net.ipv4.tcp_fin_timeout=3 [22:47:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:52:18] !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=1 [22:52:25] !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=3 [22:52:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:58:48] !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=1 [22:58:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:01:01] !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_recycle=0 [23:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:25:33] !log !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_fin_timeout=60 [23:25:41] !log root@db4:/home/paladox# sysctl -w net.ipv4.tcp_tw_reuse=0 [23:25:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:40:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvzo7 [23:40:09] [02miraheze/services] 07MirahezeSSLBot 03570d612 - BOT: Updating services config for wikis