[00:02:55] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:07] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:13] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:19] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:35] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:37] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:04:15] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:11:07] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:11:11] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:11:35] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [00:11:37] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [00:12:15] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:12:55] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:17] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:46:44] Hello joshhhhhhh! If you have any questions feel free to ask and someone should answer soon. [01:13:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+19/-0/±3] 13https://git.io/fp5qD [01:13:26] [02miraheze/puppet] 07paladox 03168ab49 - Initial import of the php module from upstream imports https://github.com/wikimedia/puppet/tree/production/modules/php but modifys it. [01:13:27] Title: [ puppet/modules/php at production · wikimedia/puppet · GitHub ] - github.com [01:15:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp5qd [01:15:37] [02miraheze/puppet] 07paladox 03d535c77 - Supply a default to hiera [01:16:53] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:18:53] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [01:22:15] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:32:14] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:04] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [04:28:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [04:31:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [04:34:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:36:53] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [05:08:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [05:10:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:36:25] Hello RobertSterbal! If you have any questions feel free to ask and someone should answer soon. [06:24:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [06:24:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [06:25:17] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:25:55] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [06:27:19] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 41% [06:28:11] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:29:00] Hello shreyasminocha! If you have any questions feel free to ask and someone should answer soon. [06:29:22] :o [06:30:10] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/1.1 200 OK - 23729 bytes in 1.341 second response time [06:31:18] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 36% [06:32:00] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [06:32:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [06:32:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:33:08] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-gettext] [06:41:09] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [10:10:49] !log enabling new php module on test1 [10:10:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [10:15:29] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:37:28] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:41:28] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 29 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[$package_name] [10:49:29] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:57:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fp5ru [10:57:44] [02miraheze/puppet] 07paladox 0381af729 - Fix mediawiki use of the new php module [11:05:12] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [11:15:22] PROBLEM - test1 php7.2-fpm on test1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [11:19:02] !log testing php7.3 on test1 [11:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [11:21:29] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:23:29] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:31:28] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:41:28] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:08:25] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [13:08:55] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [13:10:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:12:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:16:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [13:16:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [13:18:26] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:20:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:22:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [13:24:24] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:28:24] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [13:29:38] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [13:34:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [13:35:21] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:17:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fp57a [14:17:17] [02miraheze/puppet] 07paladox 03f4125a9 - Tweek mediawiki::php for new php module [14:20:44] RECOVERY - test1 php7.2-fpm on test1 is OK: PROCS OK: 2 processes with command name 'php-fpm7.2' [14:21:12] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [14:30:48] !log depooling mw1 [14:30:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:32:58] !log switching on new php module on mw1 [14:33:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:35:52] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [14:37:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp5dV [14:37:25] [02miraheze/puppet] 07paladox 03c723364 - Switch on new php module on test1 [14:37:52] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [14:41:48] hey :) [14:41:53] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [14:42:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [14:42:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [14:42:19] hi [14:43:11] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [14:43:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp5FG [14:43:59] [02miraheze/puppet] 07paladox 03fe02331 - Fix php::version var [14:44:41] PROBLEM - mw1 php7.2-fpm on mw1 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm7.2' [14:45:53] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [14:46:05] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:46:07] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [14:48:05] hey paladox, how are you doing :) [14:48:49] i am doing fine :) (im working on getting php 7.3 supported and also working on tweeking things behind the scenes) [14:48:54] how are you? [14:49:30] doing fine [14:50:20] nice, you guys use Debian Sid? [14:50:25] :) and nope [14:50:30] we use debian stretch [14:51:20] ah, but with custom repos [14:51:21] ok [14:51:36] yup [14:52:06] nice, wikimedia also uses Debian/Ubuntu [14:53:27] i'm planning to use FreeBSD :) [14:54:45] yup [14:54:53] the wmf are moving away from ubuntu [14:55:14] or rather most of prod uses debian now, it's just labs (wmcloud) that is migrating. [14:55:58] are you familiar with NixOS? [14:57:00] https://nixos.wiki/wiki/NixOS [14:57:01] Title: [ NixOS - NixOS Wiki ] - nixos.wiki [15:04:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fp5Ns [15:04:21] [02miraheze/puppet] 07paladox 03f451a41 - More php tweeks [15:05:34] 1.32 will be releses this month [15:05:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp5N8 [15:05:37] [02miraheze/puppet] 07paladox 03a1a5558 - Fix syntax error [15:05:44] if I recall correctly [15:05:51] *released [15:07:15] yes, 1.32 is set to be released in December [15:09:30] Reception123, hey [15:09:36] hi psiconauta [15:11:12] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:13:01] paladox, wmf still uses hhvm, right? [15:13:13] currently yes, but they are getting rid of it [15:18:41] RECOVERY - mw1 php7.2-fpm on mw1 is OK: PROCS OK: 4 processes with command name 'php-fpm7.2' [15:22:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fp5xE [15:22:46] [02miraheze/puppet] 07paladox 032586188 - Switch the new php module on by default for mediawiki [15:22:53] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [15:24:24] !log repool mw1 [15:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:24:36] !log depool mw2 [15:24:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:26:15] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [15:28:08] !log repool mw2 [15:28:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:28:39] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [15:28:59] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [15:29:03] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [15:49:46] !log depool mw3 [15:49:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:51:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdeW [15:51:54] [02miraheze/puppet] 07paladox 036610699 - Install "luasandbox" in the new php module [15:53:05] PROBLEM - mw3 JobQueue on mw3 is UNKNOWN: NRPE: Unable to read output [15:54:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdea [15:54:11] [02miraheze/puppet] 07paladox 035a4d552 - Fix installing php-luasandbox [15:54:15] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 52 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-luasandbox] [15:55:05] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:55:29] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 17 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[php7.2-luasandbox] [15:56:15] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:56:25] !log repool mw3 [15:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:57:29] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:08:36] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [16:09:00] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [16:11:04] we should create a custom skin for miraheze :P [16:11:49] I would help if needed [16:12:24] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:12:46] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [16:22:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdT5 [16:22:16] [02miraheze/puppet] 07paladox 03f6a1c70 - Do not create ini files for some php extensions [16:24:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdkt [16:24:53] [02miraheze/puppet] 07paladox 03d508b5d - Fix installing php-mailparse [16:26:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdk3 [16:26:04] [02miraheze/puppet] 07paladox 03746d2ed - Fix syntax error [16:29:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdkR [16:29:40] [02miraheze/puppet] 07paladox 03d82834a - Update init.pp [16:29:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdkg [16:29:56] [02miraheze/puppet] 07paladox 032eb2a15 - Update nginx.pp [16:31:53] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 5 backends are healthy [16:34:25] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [16:36:53] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:39:04] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:20:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fpdcW [21:20:16] [02miraheze/services] 07MirahezeSSLBot 039e271c8 - BOT: Updating services config for wikis [22:03:05] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 3234 [23:13:05] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs