[00:28:41] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:28:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjYVf [00:28:44] [02miraheze/puppet] 07paladox 03674dd4b - Reduce nginx timeout to 220 This reduces it to below php-fpm in the hopes that php-fpm won't kill the process. [00:28:45] [02puppet] 07paladox opened pull request 03#995: Reduce nginx timeout to 220 - 13https://git.io/fjYVJ [00:29:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjYVU [00:29:38] [02miraheze/puppet] 07paladox 0375aca28 - Update mediawiki.conf [00:29:40] [02puppet] 07paladox synchronize pull request 03#995: Reduce nginx timeout to 220 - 13https://git.io/fjYVJ [00:32:48] [02puppet] 07paladox closed pull request 03#995: Reduce nginx timeout to 220 - 13https://git.io/fjYVJ [00:32:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjYVk [00:32:51] [02miraheze/puppet] 07paladox 0331820b4 - Reduce nginx timeout to 220 (#995) * Reduce nginx timeout to 220 This reduces it to below php-fpm in the hopes that php-fpm won't kill the process. * Update mediawiki.conf [00:32:52] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:32:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:38:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [00:38:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [00:38:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [00:38:48] paladox :P [00:38:53] uh [00:39:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [00:40:03] it's nothing php related at least [00:40:15] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:40:26] just greped php7.2-fpm.log for cp4 and ns1 ip on mw3 [00:41:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYVO [00:41:49] [02miraheze/puppet] 07paladox 033a39fd6 - php: reduce emergency_restart_interval to 30s [00:44:04] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb [00:44:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [00:45:40] !log restarting php-fpm on mw* (depooling each one as i restart) [00:45:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:45:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:46:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [00:46:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [00:46:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [00:47:10] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [00:47:15] !log removing cp5 from salt-master [00:47:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:57:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYVl [00:57:01] [02miraheze/puppet] 07paladox 03c1731aa - Set max_execution_time to 220 [02:11:43] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYwB [02:11:44] [02miraheze/ManageWiki] 07JohnFLewis 03eeb9c73 - Assume if $current, we don't need to hande installs again [02:12:43] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjYw0 [02:12:44] [02miraheze/mediawiki] 07paladox 03098e677 - Update MW [02:24:05] !log killed a long running puppet process causing higher cpu then normal on db4 [02:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:26:23] !log What rolls down stairs alone or in pairs, and over your neighbor's dog? What's great for a snack, And fits on your back? It's log, log, log [02:26:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:26:34] yes! [02:27:06] Mikeee|: Please don't do that. Thanks [02:27:49] It's big, it's heavy, it's wood. [02:38:44] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40287 MB (10% inode=95%); [02:45:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYwb [02:45:12] [02miraheze/services] 07MirahezeSSLBot 03ac6d0d2 - BOT: Updating services config for wikis [02:55:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [02:55:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:56:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [02:56:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [02:57:10] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [02:57:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:57:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:02:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [03:02:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [03:03:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [04:28:08] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5061 MB (20% inode=95%); [06:25:14] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5238 MB (21% inode=95%); [07:42:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [07:42:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw3 [07:43:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [07:44:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [07:44:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [07:49:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [08:08:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [08:09:11] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [08:09:19] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw2 [08:09:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [08:09:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [08:11:20] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [08:11:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:14:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [08:15:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [08:15:15] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [13:02:51] !log disabling puppet on mw1 to test tweeking php-fpm using live traffic. [13:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:17:29] it would be a good idea to set permissions for that log bot [13:19:11] Mikeee|: why? we can easily remove any improper use [13:19:17] and then take care of the issue as needed [13:20:17] professionalism, bot attacks [13:20:47] Mikeee|: like i said we can easily revert any improper use, and take care of the issue on here as needed [13:21:15] but it'd be easier to add permissions [13:21:48] i get that you don't want to, and may have your real reasons, but chose not to share them [13:21:56] so i'm going to drop the matter [13:22:02] g'day [13:22:14] That is our real reasons? [13:22:29] If its not broke don't fix it? [13:27:05] it's more trust and respect, if people chose to abuse it, we don't want them in this channel [13:35:44] !log disabling puppet on mw2 to test tweeking php-fpm using live traffic. [13:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:36:18] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: php-fpm tweeking - paladox, last run 3 minutes ago with 0 failures [13:39:34] !log disabling puppet on mw3 to test tweeking php-fpm using live traffic. [13:39:36] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: php-fpm tweeking - paladox, last run 7 minutes ago with 0 failures [13:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:40:46] JohnLewis: thats what i was saying really, I've never seen this user on IRC before i suspect hes probably jut trying to get a rise [13:40:51] s/jut/just [13:40:51] Zppix meant to say: JohnLewis: thats what i was saying really, I've never seen this user on IRC before i suspect hes probably just trying to get a rise [13:41:10] maybe [13:43:45] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: php-fpm tweeking - paladox, last run 1 minute ago with 0 failures [13:48:52] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [14:00:14] !log live hacking puppetmaster [14:00:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:01:55] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [14:03:45] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:13:36] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [14:16:18] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:54:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±11] 13https://git.io/fjYS0 [14:54:15] [02miraheze/puppet] 07paladox 03e207b45 - Tweek php-fpm to try and improve uptime * This renames fpm_max_child to fpm_min_child. This better reflect what this *param* does. * this increases post_max_size to 60M for matomo. * Increases max_execution_time to 230 from 220 for mediawiki. * Increases fpm_min_child to 12 for mediawiki seeing as mediawiki get's most of the traffic. * Moves fastcgi_send_timeout inside [14:54:15] the location {} block and also adds fastcgi_read_timeout. * Increases process_control_timeout to 230 from 180. * Increases request_slowlog_timeout from 20 to 30. * Increases the default for max_execution_time from 180 to 230. * Increases default_socket_timeout to 2 from 1. Change-Id: I9ac7569fb8cc95278f2f6a6e794f0ff5aaaa887c [14:54:17] !log rolling out php-fpm tweeks [14:54:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:55:55] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:56:52] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:20:43] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [16:21:15] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [16:21:31] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [16:23:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:24:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [16:28:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [16:29:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:30:43] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [17:54:39] !log chrown sotuhparkfan:mail for southparkfan in /var/mail/ on misc1 (matching the other files) [17:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:18:39] !log renaming imperiuswiki to addawiki - T4296 [20:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:20:44] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 40598 MB (11% inode=95%); [20:24:49] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:24:51] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjYdz [20:24:52] [02miraheze/ssl] 07paladox 0335057cf - Add wiki.tinutulpadurenilor.eu ssl certificate [20:24:54] [02ssl] 07paladox opened pull request 03#172: Add wiki.tinutulpadurenilor.eu ssl certificate - 13https://git.io/fjYdg [20:25:16] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjYd2 [20:25:18] [02miraheze/ssl] 07paladox 03076f759 - Create wiki.tinutulpadurenilor.eu.crt [20:25:19] [02ssl] 07paladox synchronize pull request 03#172: Add wiki.tinutulpadurenilor.eu ssl certificate - 13https://git.io/fjYdg [20:26:04] [02ssl] 07paladox closed pull request 03#172: Add wiki.tinutulpadurenilor.eu ssl certificate - 13https://git.io/fjYdg [20:26:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjYda [20:26:08] [02miraheze/ssl] 07paladox 03bec2009 - Add wiki.tinutulpadurenilor.eu ssl certificate (#172) * Add wiki.tinutulpadurenilor.eu ssl certificate * Create wiki.tinutulpadurenilor.eu.crt [20:26:09] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [20:26:11] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:28:30] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fjYdw [20:28:31] [02miraheze/dns] 07paladox 0353cdac7 - Add browndust.wiki to dns - T4293 [20:33:37] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:33:39] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fjYdP [20:33:40] [02miraheze/ssl] 07paladox 03d1a5111 - Add browndust.wiki ssl certificate [20:33:42] [02ssl] 07paladox opened pull request 03#173: Add browndust.wiki ssl certificate - 13https://git.io/fjYdX [20:34:06] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fjYd1 [20:34:08] [02miraheze/ssl] 07paladox 03a2eeff5 - Create browndust.wiki.crt [20:34:10] [02ssl] 07paladox synchronize pull request 03#173: Add browndust.wiki ssl certificate - 13https://git.io/fjYdX [20:34:41] [02ssl] 07paladox closed pull request 03#173: Add browndust.wiki ssl certificate - 13https://git.io/fjYdX [20:34:43] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fjYdM [20:34:44] [02miraheze/ssl] 07paladox 03f87fd63 - Add browndust.wiki ssl certificate (#173) * Add browndust.wiki ssl certificate * Create browndust.wiki.crt [20:34:46] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L [20:34:47] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [20:40:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjYdH [20:40:14] [02miraheze/services] 07MirahezeSSLBot 0373a734c - BOT: Updating services config for wikis [21:17:24] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 40295 MB (10% inode=95%); [21:31:46] !log set columns to be [] where null in the value in mw_permissions [21:31:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master