[00:18:07] @Owen hi [00:18:20] the irc relay works 😛 [00:18:21] Hi 😛 [00:18:23] @JohnLewis hi [00:18:53] haha pinging yourself as well aren't you though [00:19:01] lol yeh [00:44:57] @Paladox Trick or Treat [00:45:08] mutante happy halloween! :D [00:45:23] next up fireworks night :P [00:45:27] paladox: thanks, what type of candy is it? [00:45:40] habows [00:45:48] though i had habows and malteser :P [00:45:55] what's that [00:47:04] *haribo [00:47:19] https://www.hancocks.co.uk/haribo-tangfastics-1-pmp [00:47:34] mutante https://www.tesco.com/groceries/en-GB/products/296070434 [00:48:39] paladox: lol, i see. that's German [00:48:45] i take the Haribo please [00:49:07] lol [00:49:22] mutante i do love my crisps so i had that too :P [00:50:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JezbR [00:51:00] [02miraheze/mw-config] 07paladox 03ada4b3c - Update Sitenotice.php [00:51:01] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:51:03] [02mw-config] 07paladox opened pull request 03#2787: Update Sitenotice.php - 13https://git.io/Jezb0 [00:52:54] [02mw-config] 07paladox closed pull request 03#2787: Update Sitenotice.php - 13https://git.io/Jezb0 [00:52:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jezbu [00:52:57] [02miraheze/mw-config] 07paladox 033bd0d71 - Update Sitenotice.php (#2787) [00:52:59] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:53:00] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:53:56] paladox: do you know what it means? [00:54:05] Haribo [00:54:33] Sweets :P [00:54:38] but the name, nope [00:55:01] Ha = Hans (first name) Ri = Riegel (last name) Bo = Bonn (home town) :p [00:55:34] it's like if your brand was Pamuno [00:58:55] heh [01:25:48] PROBLEM - db4 Puppet on db4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:25:54] PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:25:59] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:25:59] PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:26:20] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:26:22] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [01:26:23] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [01:26:35] PROBLEM - misc3 Puppet on misc3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:26:39] PROBLEM - misc1 Puppet on misc1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:26:44] PROBLEM - bacula1 Puppet on bacula1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:26:55] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [01:27:03] PROBLEM - lizardfs4 Puppet on lizardfs4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:27:04] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:27:08] PROBLEM - misc2 Puppet on misc2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:27:17] PROBLEM - db5 Puppet on db5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [01:27:25] PROBLEM - lizardfs5 Puppet on lizardfs5 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [01:27:26] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 4 minutes ago with 0 failures [01:27:38] PROBLEM - misc4 Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [01:27:41] ACKNOWLEDGEMENT - paladox is fixing all the things on Halloween night [01:29:39] lolol [01:30:06] [02puppet] 07paladox closed pull request 03#1120: Change wildcard.miraheze.org to use Comodo - 13https://git.io/JeEUA [01:30:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±19] 13https://git.io/Jezbj [01:30:09] [02miraheze/puppet] 07paladox 03f393baf - Change wildcard.miraheze.org to use Comodo (#1120) * Change wildcard.miraheze.org to use Comodo * Update nginx.conf * Update ssl.conf * Update parsoid * Update grafana.conf * Update icinga2.conf * Update mw.cnf.erb * Update nginx.conf * Update mediawiki.conf * Update init.pp * Update phabricator.miraheze.org.conf * Update mail.miraheze.org.conf * Update [01:30:09] roundcubemail.conf * Update mathoid * Update restbase * Update mediawiki.conf * Update wildcard.pp * Update phab.miraheze.wiki.conf * Update config.ini.php.erb [01:30:11] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [01:30:12] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [01:30:31] [02ssl] 07paladox synchronize pull request 03#230: Change wildcard.miraheze.org to Comodo certificate - 13https://git.io/JeEUQ [01:30:32] [02miraheze/ssl] 07paladox pushed 0381 commits to 03paladox-patch-2 [+9/-3/±154] 13https://git.io/JezNv [01:30:34] [02miraheze/ssl] 07paladox 035a757a3 - Merge branch 'master' into paladox-patch-2 [01:31:06] [02ssl] 07paladox closed pull request 03#230: Change wildcard.miraheze.org to Comodo certificate - 13https://git.io/JeEUQ [01:31:08] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JezNf [01:31:10] [02miraheze/ssl] 07paladox 0355654a8 - Change wildcard.miraheze.org to Comodo certificate (#230) * Change wildcard.miraheze.org to Comodo certificate * Update redirects.yaml * Update wildcard.miraheze.org.crt [01:31:11] [02ssl] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vxP9L [01:31:13] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-2 [01:32:10] !log update wildcard.miraheze.org private cert [01:32:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:34:36] !log deploying updated wildcard.miraheze.org to test1 [01:34:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:36:02] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [01:37:33] !log deploy updated wildcard.miraheze.org to db[45] [01:37:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:38:38] !log restart mysql on db[45] to pick up new cert, outage will happen [01:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:39:16] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:38] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:39:58] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:40:49] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [01:41:05] PROBLEM - cp3 Stunnel Http for misc2 on cp3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2335 bytes in 0.607 second response time [01:41:07] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4222 bytes in 0.037 second response time [01:41:07] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:41:22] PROBLEM - cp2 Stunnel Http for misc2 on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2335 bytes in 0.399 second response time [01:41:45] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [01:41:57] PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [01:41:58] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:42:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [01:42:05] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:42:13] PROBLEM - cp4 Stunnel Http for misc2 on cp4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2335 bytes in 0.087 second response time [01:42:20] PROBLEM - misc2 HTTPS on misc2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2335 bytes in 0.131 second response time [01:42:22] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (115) [01:42:25] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 6.16, 3.77, 2.18 [01:42:46] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [01:43:45] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17725 bytes in 0.077 second response time [01:44:20] RECOVERY - db4 MySQL on db4 is OK: Uptime: 255 Threads: 57 Questions: 1471 Slow queries: 142 Opens: 115 Flush tables: 1 Open tables: 109 Queries per second avg: 5.768 [01:44:24] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.75, 3.85, 2.42 [01:44:52] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5799 bytes in 0.033 second response time [01:45:06] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19074 bytes in 0.231 second response time [01:46:24] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.60, 2.63, 2.14 [01:49:01] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JezNR [01:49:02] [02miraheze/ssl] 07paladox 030dc2c7e - Create Sectigo.crt [01:49:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JezNE [01:49:54] [02miraheze/puppet] 07paladox 03d92d55f - Use Sectigo instead of comodo [01:49:55] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [01:49:57] [02puppet] 07paladox opened pull request 03#1131: Use Sectigo instead of comodo - 13https://git.io/JezNu [01:50:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JezNz [01:50:32] [02miraheze/puppet] 07paladox 03035418a - Update grafana.conf [01:50:33] [02puppet] 07paladox synchronize pull request 03#1131: Use Sectigo instead of comodo - 13https://git.io/JezNu [01:50:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JezNg [01:50:48] [02miraheze/puppet] 07paladox 03224c160 - Update mw.cnf.erb [01:50:49] [02puppet] 07paladox synchronize pull request 03#1131: Use Sectigo instead of comodo - 13https://git.io/JezNu [01:50:59] [02puppet] 07paladox edited pull request 03#1131: Use Sectigo instead of Comodo part 1 - 13https://git.io/JezNu [01:51:06] [02puppet] 07paladox closed pull request 03#1131: Use Sectigo instead of Comodo part 1 - 13https://git.io/JezNu [01:51:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JezN2 [01:51:09] [02miraheze/puppet] 07paladox 03c799f5b - Use Sectigo instead of Comodo part 1 (#1131) * Use Sectigo instead of comodo * Update grafana.conf * Update mw.cnf.erb [01:52:13] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [01:52:15] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [01:52:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNa [01:52:47] [02miraheze/puppet] 07paladox 03b292c94 - Use Sectigo instead of Comodo part 2 [01:53:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNw [01:53:05] [02miraheze/puppet] 07paladox 03b71d431 - Update icinga2.conf [01:53:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNr [01:53:20] [02miraheze/puppet] 07paladox 0354d8f12 - Update nginx.conf [01:53:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNo [01:53:37] [02miraheze/puppet] 07paladox 03c9c8330 - Update nginx.conf [01:53:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNK [01:53:54] [02miraheze/puppet] 07paladox 03d7e5d69 - Update config.ini.php.erb [01:54:20] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (115) [01:54:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezN6 [01:54:28] [02miraheze/puppet] 07paladox 0336bb493 - Update mediawiki.conf [01:54:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNX [01:54:49] PROBLEM - misc1 webmail.miraheze.org HTTPS on misc1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 200 OK [01:54:50] [02miraheze/puppet] 07paladox 0350bc249 - Update init.pp [01:55:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezN1 [01:55:05] [02miraheze/puppet] 07paladox 039ee77dd - Update phab.miraheze.wiki.conf [01:55:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNM [01:55:16] [02miraheze/puppet] 07paladox 031afb09d - Update phabricator.miraheze.org.conf [01:55:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezND [01:55:29] [02miraheze/puppet] 07paladox 0348a269e - Update ssl.conf [01:55:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNS [01:55:44] [02miraheze/puppet] 07paladox 03b0b4468 - Update mail.miraheze.org.conf [01:55:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezN9 [01:56:00] [02miraheze/puppet] 07paladox 03735305e - Update roundcubemail.conf [01:56:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNH [01:56:14] [02miraheze/puppet] 07paladox 030139be7 - Update mathoid [01:56:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezNQ [01:56:22] RECOVERY - db4 MySQL on db4 is OK: Uptime: 254 Threads: 63 Questions: 6505 Slow queries: 187 Opens: 140 Flush tables: 1 Open tables: 134 Queries per second avg: 25.610 [01:56:22] [02miraheze/puppet] 07paladox 03ed6c699 - Update parsoid [01:56:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezN7 [01:56:32] [02miraheze/puppet] 07paladox 03c65cae6 - Update restbase [01:56:50] RECOVERY - misc1 webmail.miraheze.org HTTPS on misc1 is OK: HTTP OK: Status line output matched "HTTP/1.1 401 Unauthorized" - 5799 bytes in 0.032 second response time [01:56:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezN5 [01:56:53] [02miraheze/puppet] 07paladox 03ae056e9 - Update mediawiki.conf [01:57:58] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [01:59:01] !log deploy updated wildcard.miraheze.org to mw[123] and puppet1 [01:59:03] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:00:18] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:00:41] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:00:55] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [02:01:18] !log deploy updated wildcard.miraheze.org to misc[1234] [02:01:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:02:10] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:02:13] RECOVERY - misc2 HTTPS on misc2 is OK: HTTP OK: HTTP/1.1 200 OK - 43695 bytes in 0.166 second response time [02:02:31] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:02:44] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:02:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:02:48] RECOVERY - cp4 Stunnel Http for misc2 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.074 second response time [02:02:52] !log deploy updated wildcard.miraheze.org to cp[234] [02:02:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:03:06] RECOVERY - cp3 Stunnel Http for misc2 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 4.624 second response time [02:03:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:03:15] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:03:26] RECOVERY - cp2 Stunnel Http for misc2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 43687 bytes in 0.591 second response time [02:03:26] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:03:49] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:04:07] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [02:04:22] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:05:27] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:06:14] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezAv [02:06:16] [02miraheze/ssl] 07paladox 03e36c396 - Switch to Sectigo for wildcard.miraheze.org [02:07:13] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:07:42] !log [01:59:01] <+paladox> !log deploy updated wildcard.miraheze.org to mw[123] and puppet1 [02:08:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:09:18] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezAf [02:09:20] [02miraheze/ssl] 07paladox 031e598ef - Switch private.revi.wiki to use Sectigo CA [02:10:00] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezAT [02:10:01] [02miraheze/mw-config] 07paladox 039786145 - Revert "Update Sitenotice.php (#2787)" This reverts commit 3bd0d7193bb60cad42dab1dbacb2c1432b4c33e9. [02:10:23] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:10:45] PROBLEM - db5 Current Load on db5 is CRITICAL: CRITICAL - load average: 15.72, 8.51, 3.74 [02:10:50] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [02:10:51] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:10:53] RECOVERY - lizardfs5 Puppet on lizardfs5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:03] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:11:12] RECOVERY - lizardfs4 Puppet on lizardfs4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:48] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:11:50] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:55] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [02:12:48] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:13:15] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:13:52] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:14:24] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:14:45] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:14:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:15:11] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [02:15:20] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezAq [02:15:22] [02miraheze/ssl] 07paladox 037edb0df - Update LetsEncrypt.crt [02:15:31] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [02:15:33] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [02:16:36] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JezAO [02:16:37] [02miraheze/ssl] 07paladox 03ac961f7 - Fix [02:17:06] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:31] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:33] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:54] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [02:18:38] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nginx-syntax] [02:19:59] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.63, 3.49, 2.04 [02:21:57] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:22:33] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:23:50] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 2.85, 3.77, 2.51 [02:24:28] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [02:24:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb [02:24:44] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw1 mw2 [02:25:00] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw2 [02:25:53] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.44, 2.50, 2.20 [02:26:11] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [02:28:12] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:29:03] PROBLEM - cp4 Stunnel Http for mw1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:30:33] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [02:30:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:31:03] RECOVERY - cp4 Stunnel Http for mw1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24661 bytes in 0.546 second response time [02:32:19] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24639 bytes in 0.642 second response time [02:32:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [02:32:53] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [02:33:06] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [02:40:45] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.45, 3.63, 2.32 [02:42:44] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 2.34, 3.21, 2.33 [02:47:51] PROBLEM - db5 Current Load on db5 is WARNING: WARNING - load average: 0.17, 1.02, 3.58 [02:49:48] RECOVERY - db5 Current Load on db5 is OK: OK - load average: 0.38, 0.87, 3.23 [03:12:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.18, 3.43, 2.42 [03:14:55] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.96, 2.57, 2.23 [03:20:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.44, 3.50, 2.69 [03:22:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.04, 3.07, 2.64 [03:32:57] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 6.61, 4.03, 2.99 [03:36:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.14, 3.43, 3.07 [03:38:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.40, 2.42, 2.74 [03:39:40] [02mw-config] 07paladox synchronize pull request 03#2781: Support renamed squid configs - 13https://git.io/JeReW [03:39:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JezxW [03:39:43] [02miraheze/mw-config] 07paladox 038d95e37 - Update LocalSettings.php [03:39:50] [02mw-config] 07paladox closed pull request 03#2781: Support renamed squid configs - 13https://git.io/JeReW [03:39:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jezx8 [03:39:53] [02miraheze/mw-config] 07paladox 03443e40c - Support renamed squid configs (#2781) * Support renamed squid configs * Update LocalSettings.php [03:39:54] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [03:39:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [03:40:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jezx4 [03:40:52] [02miraheze/mw-config] 07paladox 03df9a34a - Update LocalSettings.php [03:40:53] miraheze/mw-config/paladox-patch-2/8d95e37 - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/605840141 [03:52:28] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 15.28, 7.13, 3.81 [03:58:39] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.08, 3.50, 3.35 [04:00:37] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.60, 2.52, 3.01 [04:33:43] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.63, 4.13, 2.77 [04:39:26] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 2.71, 3.76, 3.08 [04:43:14] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.94, 3.94, 3.27 [04:48:57] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.51, 3.68, 3.45 [04:50:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.41, 2.88, 3.18 [05:03:15] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.82, 3.62, 2.67 [05:05:14] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.84, 2.57, 2.39 [05:34:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.66, 3.85, 2.57 [05:36:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.10, 3.69, 2.68 [05:38:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.74, 2.59, 2.39 [05:54:53] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 4.28, 3.28, 2.13 [05:56:55] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.00, 2.44, 1.97 [06:26:01] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2862 MB (11% inode=94%); [06:26:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.79, 4.99, 2.99 [06:28:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.58, 3.69, 2.76 [06:32:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 2.31, 3.29, 2.84 [06:39:47] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.50, 4.42, 3.45 [06:43:42] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 1.52, 3.45, 3.34 [06:45:36] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.92, 3.02, 3.19 [06:55:58] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.68, 4.33, 3.20 [07:01:54] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 1.46, 3.66, 3.43 [07:03:53] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.69, 2.71, 3.11 [07:37:37] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.66, 3.84, 2.71 [07:39:31] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.38, 2.93, 2.51 [09:41:13] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.71, 2.40, 1.77 [09:45:10] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 2.49, 3.05, 2.21 [10:30:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.02, 3.95, 2.65 [10:32:55] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.63, 3.72, 2.72 [10:34:55] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.95, 2.73, 2.48 [10:54:30] PROBLEM - vmcodex.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vmcodex.net' expires in 15 day(s) (Sun 17 Nov 2019 10:50:35 AM GMT +0000). [10:54:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegJE [10:54:44] [02miraheze/ssl] 07MirahezeSSLBot 03a71cc4e - Bot: Update SSL cert for vmcodex.net [10:56:13] PROBLEM - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bh.gyaanipedia.co.in' expires in 15 day(s) (Sun 17 Nov 2019 10:52:44 AM GMT +0000). [10:56:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegJ2 [10:56:27] [02miraheze/ssl] 07MirahezeSSLBot 039395a8e - Bot: Update SSL cert for bh.gyaanipedia.co.in [11:04:16] RECOVERY - bh.gyaanipedia.co.in - LetsEncrypt on sslhost is OK: OK - Certificate 'bh.gyaanipedia.co.in' will expire on Thu 30 Jan 2020 09:56:20 AM GMT +0000. [11:04:30] RECOVERY - vmcodex.net - LetsEncrypt on sslhost is OK: OK - Certificate 'vmcodex.net' will expire on Thu 30 Jan 2020 09:54:37 AM GMT +0000. [11:18:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 6.78, 4.20, 2.74 [11:20:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 2.36, 3.56, 2.69 [11:22:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.00, 2.65, 2.46 [11:32:55] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [12:20:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 3.51, 3.08, 2.05 [12:22:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.30, 3.49, 2.32 [12:26:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.40, 2.93, 2.42 [13:17:12] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.35, 3.47, 2.28 [13:21:09] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 2.03, 3.85, 2.81 [13:23:07] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.90, 2.80, 2.54 [13:41:32] hey irc is back [13:42:35] yay! [13:42:38] Zppix: ^ [13:43:54] Reception123: that just means the quota reset [14:05:44] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.24, 3.87, 2.48 [14:07:41] PROBLEM - lizardfs4 Current Load on lizardfs4 is WARNING: WARNING - load average: 2.34, 3.50, 2.53 [14:09:37] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.26, 2.64, 2.33 [14:25:01] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.89, 3.17, 2.24 [14:25:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegI5 [14:25:10] [02miraheze/services] 07MirahezeSSLBot 0346d6fb5 - BOT: Updating services config for wikis [14:28:58] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.72, 2.52, 2.26 [15:16:51] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/Jegtm [15:16:53] [02miraheze/mediawiki] 07RhinosF1 0391d62a9 - Adding DarkMode (1.34) [15:16:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 5.51, 3.38, 2.15 [15:17:34] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/Jegts [15:17:36] [02miraheze/mediawiki] 07RhinosF1 03b1b12a7 - Adding DarkMode (1.33) [15:17:58] wrong name [15:17:59] RhinosF1: that’s wrong :P [15:18:00] Needs to be extensions/ [15:18:01] So git submodule add [15:18:02] Then change the branch in the git modules file and also checkout the branch in the extension submodule [15:18:06] yeah sene [15:18:11] And also add dirty like the rest of the modules :) [15:18:27] paladox: for both branches? [15:18:31] and what command [15:18:41] Yup [15:18:51] So git submodule add [15:18:54] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.83, 2.60, 2.00 [15:18:59] Then go and checkout the branch in that path [15:19:10] Then change .gitmodules and also add dirty [15:19:38] ok [15:20:52] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+0/-1/±1] 13https://git.io/Jegt4 [15:20:53] [02miraheze/mediawiki] 07RhinosF1 03727bdd0 - revert DM [15:22:22] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+0/-1/±1] 13https://git.io/JegtR [15:22:23] [02miraheze/mediawiki] 07RhinosF1 0361340a5 - revert DM [15:27:55] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+1/-0/±1] 13https://git.io/Jegto [15:27:57] [02miraheze/mediawiki] 07RhinosF1 03901de65 - Add DarkMode [15:27:59] paladox: ^ [15:28:26] Looks better [15:30:15] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+1/-0/±1] 13https://git.io/JegtP [15:30:16] [02miraheze/mediawiki] 07RhinosF1 03acc72d2 - Add DarkMode [15:32:21] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jegty [15:32:23] [02miraheze/mw-config] 07RhinosF1 0356fecc6 - install DarkMode to test1 [15:32:24] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [15:32:28] [02mw-config] 07RhinosF1 opened pull request 03#2788: install DarkMode to test1 - 13https://git.io/JegtS [15:34:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jegt9 [15:34:07] [02miraheze/mw-config] 07RhinosF1 03b46e8a0 - Update LocalExtensions.php [15:34:09] [02mw-config] 07RhinosF1 synchronize pull request 03#2788: install DarkMode to test1 - 13https://git.io/JegtS [15:35:55] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/Jegt5 [15:35:56] [02miraheze/mw-config] 07RhinosF1 031c7ab9e - Update ManageWikiExtensions.php [15:35:58] [02mw-config] 07RhinosF1 synchronize pull request 03#2788: install DarkMode to test1 - 13https://git.io/JegtS [15:36:30] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JegtN [15:36:32] [02miraheze/mw-config] 07RhinosF1 035acc8ee - Update extension-list [15:36:33] [02mw-config] 07RhinosF1 synchronize pull request 03#2788: install DarkMode to test1 - 13https://git.io/JegtS [15:37:38] * RhinosF1 waits for travis [15:38:42] [02mw-config] 07RhinosF1 closed pull request 03#2788: install DarkMode to test1 - 13https://git.io/JegtS [15:38:43] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jegth [15:38:45] [02miraheze/mw-config] 07RhinosF1 037f632fb - install DarkMode to test1 (#2788) DarkMode ext - test1 [15:40:21] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki on test1 [15:40:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:40:37] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki [15:40:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:40:43] !log on test1 that is [15:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:42:49] * RhinosF1 knows test1 is down [15:44:03] paladox: why is it erroring about OATHauth [15:44:09] I ain't touched it [15:44:52] It’s not down? [15:44:53] Oh [15:44:54] Can you file a bug report upstream? [15:45:14] paladox: why's it only just failed though and I can [15:45:29] BecUse test1 is on 1.34.... [15:45:42] *because [15:45:46] paladox: test1wiki was up this morning though [15:45:55] It is still up [15:46:02] It works for me, I see the main page [15:46:07] paladox: it's error 500 for me [15:46:15] Ok [15:46:23] We’ll file the bug report :P [15:46:29] Since I’m mobile [15:46:31] paladox: not ok, i've just deployed an extension [15:46:40] i'll file for OATHAuth [15:46:48] Ok [15:47:01] This is exactly why we test this in advance [15:47:24] JohnLewis: can you work out why test1 is 500'ing all I can see is unrelated error [15:47:33] Check the logs [15:47:38] The error is in the logs [15:48:04] paladox: all I see is OATHAuth] [15:48:12] What log are you checking? [15:48:14] Then it’s that [15:48:37] Ofc check the timestamp too [15:50:11] because I don't see a fatal for OATHAuth [15:52:08] JohnLewis: what about php-error? [15:52:08] JohnLewis: exception.log and php-error.log [15:52:28] Can't be checking php-error.log too well then [15:52:34] the error is fairly clear [15:52:35] paladox, JohnLewis: error is in https://phabricator.wikimedia.org/T237114 [15:52:38] [ ⚓ T237114 oathauth-module-invalid on Mediawiki 1.34 ] - phabricator.wikimedia.org [15:53:47] JohnLewis: I also still don't get why i'm getting error 500 [15:53:51] RhinosF1: the last 7 are the issue but from what I can see, are absolutely unrelated to OATHAuth [15:55:04] JohnLewis: okay I see the last 7, there is still a lot of mentions of an OATHAuth exception whether related to this or not [15:55:25] I know [15:55:36] Not saying they don't exist [15:55:44] I'm saying they're not relevant to the 500 [15:56:02] JohnLewis: okay, well i've reported them [16:00:30] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jegql [16:00:32] [02miraheze/mediawiki] 07RhinosF1 03f885edf - Update LogEventsList.php copying from https://raw.githubusercontent.com/wikimedia/mediawiki/master/includes/logging/LogEventsList.php [16:01:14] paladox, JohnLewis: ^ [16:02:43] Thanks! [16:02:44] RhinosF1: you can either git pull or wait for puppet to run on test1 [16:02:54] paladox: puppet is already running [16:04:38] Ok [16:05:11] * RhinosF1 wonders when puppet will finish [16:15:07] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/Jegqx [16:15:09] [02miraheze/mw-config] 07RhinosF1 0318b8488 - uninstalling RCLF [16:15:11] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [16:15:17] [02mw-config] 07RhinosF1 opened pull request 03#2789: uninstalling RCLF - 13https://git.io/Jegqh [16:15:54] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/Jegmv [16:15:56] [02miraheze/mw-config] 07RhinosF1 03cc3ac8a - Update extension-list [16:15:57] [02mw-config] 07RhinosF1 synchronize pull request 03#2789: uninstalling RCLF - 13https://git.io/Jegqh [16:16:40] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/Jegmf [16:16:42] [02miraheze/mw-config] 07RhinosF1 0314cc27d - Update LocalSettings.php [16:16:43] [02mw-config] 07RhinosF1 synchronize pull request 03#2789: uninstalling RCLF - 13https://git.io/Jegqh [16:17:24] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JegmT [16:17:25] [02miraheze/mw-config] 07RhinosF1 03192b777 - Update ManageWikiExtensions.php [16:17:27] [02mw-config] 07RhinosF1 synchronize pull request 03#2789: uninstalling RCLF - 13https://git.io/Jegqh [16:19:25] [02mw-config] 07RhinosF1 closed pull request 03#2789: uninstalling RCLF - 13https://git.io/Jegqh [16:19:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/Jegmm [16:19:28] [02miraheze/mw-config] 07RhinosF1 036f306f7 - uninstalling RCLF (#2789) [16:27:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/Jegm0 [16:28:00] [02miraheze/mw-config] 07RhinosF1 0330370a8 - Update LocalSettings.php [16:28:02] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [16:28:17] [02mw-config] 07RhinosF1 opened pull request 03#2790: Ext DarkMode to production - 13https://git.io/Jegmu [16:28:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-3 [+0/-0/±1] 13https://git.io/Jegmg [16:28:53] [02miraheze/mw-config] 07RhinosF1 032939300 - Update ManageWikiExtensions.php [16:28:54] [02mw-config] 07RhinosF1 synchronize pull request 03#2790: Ext DarkMode to production - 13https://git.io/Jegmu [16:30:40] grumble: welcome back to the non-spooky side [16:31:01] [02mw-config] 07RhinosF1 closed pull request 03#2790: Ext DarkMode to production - 13https://git.io/Jegmu [16:31:02] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JegmK [16:31:04] [02miraheze/mw-config] 07RhinosF1 03d48195d - Ext DarkMode to production (#2790) [16:32:06] * RhinosF1 waits for puppet [16:33:56] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki [16:34:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:34:27] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki (both this and above on mw[123] [16:34:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:35:11] JohnLewis: as a crat on meta can you enable DarkMode [16:37:21] busy [16:37:46] but any steward can [16:37:55] in fact, only stewards can [16:42:20] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegYL [16:42:21] [02miraheze/mw-config] 07JohnFLewis 0387cbd8f - fix overriding of default value incorrectly [16:44:07] JohnLewis: why can't crats? [16:44:35] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 6.39, 4.25, 2.58 [16:45:01] because crats have never been able to on meta [16:45:33] JohnLewis: ok, well as the only active steward can you please? when time appears [16:45:59] tbf, what does DarkMode actually do? [16:46:10] I assume it's opt-in on a user basis? [16:46:33] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.82, 3.32, 2.44 [16:47:18] JohnLewis: it adds a link to the personal links bar to turn it on and off [16:47:38] By default it’s off [16:47:49] enable it globally, opt out on a wiki but on by default [16:48:42] JohnLewis: I’ve just signed out but ok [16:52:26] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/JegY3 [16:52:28] [02miraheze/mw-config] 07RhinosF1 0363f0d9d - Make DarkMode on by default [16:52:29] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [16:52:31] [02mw-config] 07RhinosF1 opened pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYG [16:52:48] JohnLewis: ^ [16:53:41] miraheze/mw-config/RhinosF1-patch-4/63f0d9d - RhinosF1 The build passed. https://travis-ci.org/miraheze/mw-config/builds/606096137 [17:07:02] JohnLewis: DarkMode isn’t showing [17:12:38] Did you enable it on all existing wikis? [17:13:55] [02mw-config] 07JohnFLewis commented on pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYN [17:14:07] JohnLewis: did I merge it? [17:14:47] No [17:14:53] I’ll do it tonight [17:15:02] JohnLewis: you realise it’s my first GE [17:15:34] I’m off again now [17:17:54] But I said opt out, not forced global [17:20:01] JohnLewis: I’ll fix later [17:26:13] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 4.85, 3.62, 2.29 [17:28:12] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.00, 2.56, 2.05 [18:14:54] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 6.40, 4.07, 2.46 [18:18:56] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.51, 3.03, 2.45 [18:32:15] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.24, 3.20, 2.00 [18:34:14] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 1.20, 2.38, 1.85 [18:42:44] .mhca Iron_Patriot [18:42:44] https://meta.miraheze.org/wiki/Special:CentralAuth/Iron_Patriot [18:44:12] .wmca Iron_Patriot [18:44:12] https://meta.wikimedia.org/wiki/Special:CentralAuth/Iron_Patriot [19:25:56] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 6.73, 4.39, 2.75 [19:27:50] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.15, 3.05, 2.45 [19:55:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegG7 [19:55:11] [02miraheze/services] 07MirahezeSSLBot 03337326d - BOT: Updating services config for wikis [20:00:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegZf [20:00:13] [02miraheze/services] 07MirahezeSSLBot 0352a537a - BOT: Updating services config for wikis [20:35:01] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.40, 3.51, 2.37 [20:37:01] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 0.65, 2.39, 2.09 [21:03:55] Hello Smaug! If you have any questions, feel free to ask and someone should answer soon. [22:13:20] PROBLEM - lizardfs5 Current Load on lizardfs5 is WARNING: WARNING - load average: 3.53, 2.90, 1.77 [22:15:19] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.71, 2.02, 1.58 [22:24:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegCs [22:24:10] [02miraheze/puppet] 07paladox 03495793f - Install role::db on lizardfs6 Testing wether the performance is good at this difference. [22:25:51] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/JegCG [22:25:53] [02miraheze/mw-config] 07RhinosF1 038422021 - Update LocalSettings.php [22:25:54] [02mw-config] 07RhinosF1 synchronize pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYG [22:26:27] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/JegCn [22:26:28] [02miraheze/mw-config] 07RhinosF1 0377da9ff - Update LocalExtensions.php [22:26:30] [02mw-config] 07RhinosF1 synchronize pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYG [22:26:59] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/JegCC [22:27:01] [02miraheze/mw-config] 07RhinosF1 03099d506 - Update GlobalExtensions.php [22:27:02] [02mw-config] 07RhinosF1 synchronize pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYG [22:27:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegCW [22:27:09] [02miraheze/puppet] 07paladox 0317210c9 - Update lizardfs6.yaml [22:27:34] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Package[mariadb-client-10.0],Package[mariadb-server-10.0],Package[mariadb-server-core-10.0],File[/srv/mariadb] [22:28:23] [02mw-config] 07RhinosF1 commented on pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegCB [22:28:46] JohnLewis: ^ [22:29:12] on by default is no longer an accurate description [22:29:29] as it's on forever, it's global, there's no "default" about it, it's the sole option [22:29:54] JohnLewis: well how to do you want it doing? [22:30:14] well are you having it on by default, or global? [22:30:48] JohnLewis: you said do it on by default [22:30:54] The code lgtm so i approved them (just an fyi) [22:31:04] didnt merge just approve [22:31:29] Zppix: Still trying to master on by default or global [22:31:31] yes, so look at other examples of extensions "on by default" and not "global" [22:31:44] there's a setting in LocalSettings for it, Cite is an example [22:32:55] JohnLewis: are you wanting it just be in LS as default true? [22:33:04] Zppix: No [22:33:16] JohnLewis: then i guess i dont understand what you want from it [22:33:17] wgManageWikiExtensionsDefault [22:33:36] Zppix: search for cite and you'll see [22:33:37] PROBLEM - lizardfs6 MySQL on lizardfs6 is CRITICAL: Can't connect to MySQL server on '54.36.165.161' (115) [22:33:54] ah i see now [22:33:56] [02mw-config] 07RhinosF1 closed pull request 03#2791: Make DarkMode on by default - 13https://git.io/JegYG [22:34:01] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [22:34:03] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-4 [22:36:42] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-4 [+0/-0/±1] 13https://git.io/JegCo [22:36:43] [02miraheze/mw-config] 07RhinosF1 032c66c69 - Update LocalSettings.php [22:36:45] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [22:36:47] JohnLewis: ^ [22:37:02] [02mw-config] 07RhinosF1 opened pull request 03#2792: DarkMode to global ext - 13https://git.io/JegCK [22:38:46] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 - 13https://git.io/vbvb3 [22:38:47] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-3 [22:38:49] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [22:38:50] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-2 [22:38:52] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [22:38:53] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [22:40:14] JohnLewis: Does that look better? [22:44:10] Yes, but needs to be enabled on all wikis once merged [22:44:50] JohnLewis: using? [22:45:06] toggleExtension.php [22:45:32] JohnLewis: is there a --help or? [22:45:50] [02mw-config] 07RhinosF1 closed pull request 03#2792: DarkMode to global ext - 13https://git.io/JegCK [22:45:52] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JegCy [22:45:53] [02miraheze/mw-config] 07RhinosF1 0370a84e0 - Dark Mode extension to global [22:46:39] yes [22:46:53] every MediaWiki script has a --help [22:47:14] @Rhinosf1 if you want i can run the command if you tell me what to do [22:47:53] @Zppix I've got to find the script first to work out syntax [22:47:59] ok [22:50:01] JohnLewis: Could not open input file: srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php [22:50:15] that's you misplet it. [22:50:18] /srv [22:50:25] thx [22:50:36] this is why i shouldn't do things ill [22:51:38] Zppix: sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php DarkMode --wiki=loginwiki [22:51:41] I think [22:51:47] JohnLewis: ^ [22:52:09] yes, for loginwiki [22:52:22] JohnLewis: is there a easy way to run that cmd for all wikidbs? [22:52:52] there was a foreachwikiindb somewhere [22:53:06] Voidwalker: "somewhere" [22:53:23] yeah, I know, I'm very helpful [22:54:34] @Zppix try reading tools.wmflabs.oprg/bash - there's many more examples of how helpful devs really are [22:54:41] Im aware [22:54:56] .mh SAL [22:55:13] has to type something [22:57:03] @Zppix i would say look on SAL but there's no record of it being ran in 201[89] [23:03:30] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.64, 3.15, 1.99 [23:05:01] !log run /usr/bin/nice -10 sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php DarkMode on mw1 [23:05:32] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.08, 2.33, 1.83 [23:05:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [23:05:55] @RhinosF1 ^ [23:11:30] JohnLewis: so what exactly is this maint script that needed ran doing? [23:13:30] ah yes run it and then ask what it's doing o/ [23:13:45] PuppyKun: i mean JohnLewis said it needed to be ran so :P [23:21:48] PROBLEM - lizardfs5 Current Load on lizardfs5 is CRITICAL: CRITICAL - load average: 5.19, 3.73, 2.20 [23:23:51] RECOVERY - lizardfs5 Current Load on lizardfs5 is OK: OK - load average: 0.83, 2.61, 1.98 [23:41:59] PROBLEM - lizardfs4 Current Load on lizardfs4 is CRITICAL: CRITICAL - load average: 4.79, 3.04, 1.95 [23:43:58] RECOVERY - lizardfs4 Current Load on lizardfs4 is OK: OK - load average: 1.02, 2.27, 1.81 [23:45:38] [02mw-config] 07paladox deleted branch 03RhinosF1-patch-4 - 13https://git.io/vbvb3 [23:45:39] [02miraheze/mw-config] 07paladox deleted branch 03RhinosF1-patch-4 [23:50:23] huh [23:51:30] oh yup [23:52:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-6 [+0/-0/±1] 13https://git.io/JegWV [23:52:35] [02miraheze/puppet] 07paladox 0383c29c7 - Add support for mariadb 10.3 [23:52:36] [02puppet] 07paladox created branch 03paladox-patch-6 - 13https://git.io/vbiAS [23:52:38] [02puppet] 07paladox opened pull request 03#1132: Add support for mariadb 10.3 - 13https://git.io/JegWw