[01:53:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuVN [01:53:29] [02miraheze/puppet] 07paladox 031083546 - php: Set request_terminate_timeout_track_finished to yes [01:54:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuVp [01:54:28] [02miraheze/puppet] 07paladox 037bcdabd - Fix syntax [01:57:36] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JeuVj [01:57:37] [02miraheze/puppet] 07paladox 03ee14782 - Revert "Fix syntax" This reverts commit 7bcdabd3f8e9e390e11d3425342615469c17fbdd. [01:57:39] [02miraheze/puppet] 07paladox 035923aca - Revert "php: Set request_terminate_timeout_track_finished to yes" This reverts commit 1083546edf98df2189398b9b80b46db0ff1469b6. [01:58:29] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [01:59:26] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:00:25] !log depooled and repooled mw1 [02:01:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:01:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb [02:02:43] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:03:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:03:30] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:24:52] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 11055 MB (14% inode=98%); [04:15:07] PROBLEM - wiki.opendominion.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.opendominion.net' expires in 15 day(s) (Thu 14 Nov 2019 04:11:40 AM GMT +0000). [04:15:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeurf [04:15:25] [02miraheze/ssl] 07MirahezeSSLBot 0386b02c8 - Bot: Update SSL cert for wiki.opendominion.net [04:23:06] RECOVERY - wiki.opendominion.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.opendominion.net' will expire on Mon 27 Jan 2020 03:15:17 AM GMT +0000. [06:26:07] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2838 MB (11% inode=94%); [06:26:11] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[php7.3-apcu],Package[php7.3-redis] [06:34:20] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:56:32] RhinosF1: poke Reception123 or Zppix [07:58:24] One of you say hello [07:59:05] I'm here but mobile [10:05:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuP8 [10:05:09] [02miraheze/services] 07MirahezeSSLBot 034b169fb - BOT: Updating services config for wikis [10:26:55] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2641 MB (10% inode=94%); [10:28:55] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2656 MB (11% inode=94%); [10:39:00] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2647 MB (10% inode=94%); [12:01:31] revi: any updates on the CoCC? [12:01:50] Reception123: you are late to the party [12:01:52] see the CN [12:02:02] (that thread) [12:02:17] revi: ah okay, my bad [12:02:18] revi: thanks [12:17:20] revi, Reception123: added some CoC election questions,m [12:17:25] s/,m/ [12:17:25] RhinosF1 meant to say: revi, Reception123: added some CoC election questions [12:17:47] ok [12:17:51] will look [12:18:15] * RhinosF1 gives Reception123 about a hundred things to do [12:24:13] * revi is not running anyway [12:29:40] revi: you’re fine [14:21:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuSn [14:21:42] [02miraheze/puppet] 07paladox 039e4bf51 - matomo: Update matomo to 3.12.0 [14:26:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuS0 [14:26:32] [02miraheze/puppet] 07paladox 03680b41f - matomo: Install DeviceDetectorCache [14:31:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuSa [14:31:08] [02miraheze/puppet] 07paladox 0307f19e7 - mediawiki: Increase mfswriteworkers to 100 [14:32:24] !log depool and repool mw[123] [14:32:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:33:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuS6 [14:33:42] [02miraheze/puppet] 07paladox 03ffd0c4a - Update mediawiki.pp [14:36:36] !log install iotop on mw1 [14:37:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [14:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:38:10] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [14:39:25] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:39:33] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [14:40:12] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [14:41:31] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [14:42:44] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.82, 6.41, 5.84 [14:44:43] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.62, 7.26, 6.22 [14:48:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.96, 7.46, 6.56 [14:50:19] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [14:52:48] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.31, 6.71, 6.45 [14:53:05] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [14:54:16] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:55:03] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:56:45] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.93, 7.37, 6.78 [15:12:10] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [15:12:43] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.61, 6.61, 6.80 [15:14:08] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [15:16:43] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.57, 7.10, 6.95 [15:17:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeu9x [15:17:44] [02miraheze/puppet] 07paladox 03b7b4963 - Update mediawiki.pp [15:24:45] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.09, 6.38, 6.68 [15:32:02] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [15:34:02] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:45:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuHo [15:45:10] [02miraheze/services] 07MirahezeSSLBot 03b3f994d - BOT: Updating services config for wikis [16:04:31] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JeuQg [16:04:32] [02miraheze/ssl] 07Reception123 038c90d51 - add vitriol.spoon.army cert [16:35:24] revi: what’s the deadline for CoC noms? [16:36:41] RhinosF1: 10 days from yesterday [16:36:56] so around Nov 7 [16:37:47] revi: thanks, I’m considering [16:37:55] hi [16:38:06] Oh hi apap [18:02:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:02:57] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [18:03:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw1 mw3 [18:04:58] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [18:05:12] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [18:06:54] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:07:12] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:07:37] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:12:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:13:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:15:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeudR [18:15:10] [02miraheze/services] 07MirahezeSSLBot 034d9d2d4 - BOT: Updating services config for wikis [18:18:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 2a00:d880:5:8ea::ebc7/cpweb [18:19:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [18:20:06] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [18:21:56] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [18:23:14] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [18:33:42] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:33:55] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:34:03] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [18:35:17] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [18:35:26] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [18:35:48] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [18:39:55] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 6.91, 6.39, 5.84 [18:41:59] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.20, 6.38, 5.91 [18:43:29] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:47:07] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Mount[/mnt/mediawiki-static] [18:53:18] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:33:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeubJ [19:33:57] [02miraheze/mw-config] 07paladox 0371481b5 - Enable wgGenerateThumbnailOnParse for ciptamediawiki and staffwiki [19:33:59] Zppix ^ [19:34:23] paladox: should that be default? [19:34:27] no [19:34:30] why not? [19:35:01] it is the default, we switched it off as we are a large wiki farm so cannot afford for requests to be blocked generating thumberneils on parse. [19:35:07] It's what wikimedia do [19:43:38] [02ssl] 07Pix1234 opened pull request 03#232: Create wiki.serwerwanilia.pl.cert - 13https://git.io/JeubO [19:43:44] Reception123: ^ [19:44:11] doin't forget to add it to the yaml file [19:44:14] i did [19:44:14] Zppix: don't forget certs.yaml [19:44:16] Zppix: certs.yaml [19:44:21] Zppix: check again [19:44:21] ^ [19:44:36] Reception123: ah i did it on my fork but on the wrong branch :P [19:45:47] [02ssl] 07Pix1234 synchronize pull request 03#232: Create wiki.serwerwanilia.pl.cert - 13https://git.io/JeubO [19:45:54] Reception123: {{done}} [19:48:26] ok, will do the private key [19:48:29] ok [19:48:32] Looks fine [19:51:25] [02ssl] 07Reception123 commented on pull request 03#232: Create wiki.serwerwanilia.pl.cert - 13https://git.io/Jeubl [19:51:29] [02ssl] 07Reception123 closed pull request 03#232: Create wiki.serwerwanilia.pl.cert - 13https://git.io/JeubO [19:51:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jeub8 [19:51:32] [02miraheze/ssl] 07Pix1234 037b63ac0 - Create wiki.serwerwanilia.pl.cert (#232) * Create wiki.serwerwanilia.pl.cert * wiki.serwerwanilia.pl add [19:52:04] .in 10m update wanilla's url settings on ManageWiki [19:52:05] Zppix: Okay, I will set the reminder for: 2019-10-29 - 15:02:04CDT [19:52:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jeub4 [19:52:39] [02miraheze/puppet] 07paladox 030c04b4e - Update php.pp [19:53:21] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:53:54] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [19:55:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:55:54] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [19:57:20] !log restarting php7.2-fpm accross mw[123] [19:57:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [20:00:23] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [20:00:25] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [20:00:26] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [20:01:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:01:56] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:02:05] Zppix: update wanilla's url settings on ManageWiki [20:02:09] doing it now [20:05:11] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:05:24] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:05:37] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:05:38] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:06:01] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:06:06] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:06:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:06:28] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:06:28] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:06:48] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:07:07] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:08:27] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[wiki.serwerwanilia.pl] [20:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeubS [20:10:12] [02miraheze/services] 07MirahezeSSLBot 036c53026 - BOT: Updating services config for wikis [20:15:14] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-1/±0] 13https://git.io/Jeubb [20:15:16] [02miraheze/ssl] 07Reception123 03ef2f93a - typo [20:18:12] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 107.191.126.23/cpweb [20:18:28] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 107.191.126.23/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:19:52] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.62, 6.58, 5.70 [20:21:47] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.85, 6.36, 5.72 [20:22:19] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:22:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:23:12] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:23:52] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:07] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:15] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:52] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:25:02] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:25:27] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:40:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuNV [20:40:12] [02miraheze/services] 07MirahezeSSLBot 03884cc8a - BOT: Updating services config for wikis [20:46:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuNX [20:46:07] [02miraheze/puppet] 07paladox 03024d570 - Revert "Update php.pp" This reverts commit 0c04b4ebb13e1e82d2564bdd9ec91ccc565353c3. [20:47:23] !log depool and repool mw[123] [20:47:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:03:32] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [21:03:41] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [21:04:22] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:05:30] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:05:40] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:06:22] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:08:52] Hello hispano7640! If you have any questions, feel free to ask and someone should answer soon. [21:20:15] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03Pix1234-patch-1 [+0/-0/±1] 13https://git.io/JeuA3 [21:20:16] [02miraheze/mw-config] 07Pix1234 03b5298d9 - add new restriction levels per T4819 [21:20:18] [02mw-config] 07Pix1234 created branch 03Pix1234-patch-1 - 13https://git.io/vbvb3 [21:20:22] [02mw-config] 07Pix1234 opened pull request 03#2782: add new restriction levels per T4819 - 13https://git.io/JeuAs [21:22:00] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03Pix1234-patch-1 [+0/-0/±1] 13https://git.io/JeuAn [21:22:02] [02miraheze/mw-config] 07Pix1234 035784be6 - + owner [21:22:03] [02mw-config] 07Pix1234 synchronize pull request 03#2782: add new restriction levels per T4819 - 13https://git.io/JeuAs [21:24:49] [02mw-config] 07Pix1234 synchronize pull request 03#2782: add new restriction levels per T4819 - 13https://git.io/JeuAs [21:24:51] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03Pix1234-patch-1 [+0/-0/±1] 13https://git.io/JeuAl [21:24:52] [02miraheze/mw-config] 07Pix1234 0319c1f60 - + owner restriction level [21:26:44] [02mw-config] 07Pix1234 closed pull request 03#2782: add new restriction levels per T4819 - 13https://git.io/JeuAs [21:26:46] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuAR [21:26:47] [02miraheze/mw-config] 07Pix1234 034d2abe8 - add new restriction levels per T4819 (#2782) * add new restriction levels per T4819 * + owner * + owner restriction level [21:31:34] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuAa [21:31:35] [02miraheze/mw-config] 07Pix1234 03b38f11a - typo [21:37:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuAX [21:37:12] [02miraheze/puppet] 07paladox 0377bd930 - php: use new tideway php extension [21:39:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuAD [21:39:40] [02miraheze/puppet] 07paladox 03851c6c5 - Revert "php: use new tideway php extension" This reverts commit 77bd93077f0cde5af5459b189c134db0afbe3110. [21:40:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuA9 [21:40:18] [02miraheze/puppet] 07paladox 037704c14 - Update php.pp [21:43:04] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 30 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[php-tideways-xhprof] [21:45:21] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php-tideways-xhprof] [21:45:42] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php-tideways-xhprof] [21:53:17] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:53:23] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [21:53:47] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:53:58] Zppix: ^ [21:55:35] [02miraheze/mw-config] 07Pix1234 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeuxT [21:55:37] [02miraheze/mw-config] 07Pix1234 037995a02 - capitallization [22:14:06] PROBLEM - mw3 Disk Space on mw3 is WARNING: DISK WARNING - free space: / 8409 MB (10% inode=98%); [22:25:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.01, 6.33, 5.65 [22:29:14] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.41, 6.66, 5.95 [22:32:02] PROBLEM - mw3 Disk Space on mw3 is CRITICAL: DISK CRITICAL - free space: / 4408 MB (5% inode=98%); [22:33:19] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.53, 7.11, 6.28 [22:35:15] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.54, 6.48, 6.15 [22:44:02] RECOVERY - mw3 Disk Space on mw3 is OK: DISK OK - free space: / 22346 MB (29% inode=98%); [23:04:06] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:04:06] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [23:04:15] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:04:42] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:54] PROBLEM - misc3 Current Load on misc3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:04:56] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [23:07:35] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [23:08:33] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [23:08:46] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:08:50] PROBLEM - misc3 Current Load on misc3 is WARNING: WARNING - load average: 1.09, 3.78, 2.56 [23:09:00] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [23:10:08] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [23:10:59] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:11:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [23:11:29] RECOVERY - misc3 Current Load on misc3 is OK: OK - load average: 0.37, 2.34, 2.20 [23:35:14] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.49, 6.58, 5.93 [23:37:12] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.12, 6.38, 5.94