[00:00:12] JohnLewis im going to remove nginx from swift proxy [00:00:22] otherwise it's going to conflict with apache [00:00:35] ooh, conflict there [00:00:45] that's why we need to standardise what we use [00:01:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjcw [00:01:10] [02miraheze/puppet] 07paladox 0395fa702 - Remove nginx from swift proxy [00:01:39] JohnLewis it was unused anyways [00:01:45] oh okay, good [00:03:10] PROBLEM - Puppet on cp4 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:03:40] PROBLEM - Puppet on mw3 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:09:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjcA [00:09:32] [02miraheze/puppet] 07paladox 03ca76e43 - Update misc4.yaml [00:09:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjcp [00:09:49] [02miraheze/puppet] 07paladox 037aac5e1 - Update misc4.yaml [00:11:08] RECOVERY - Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [00:11:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjCf [00:11:30] [02miraheze/puppet] 07paladox 030a04280 - Update db.pp [00:13:26] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:13:36] RECOVERY - Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:38] RECOVERY - Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:14:26] PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.04, 1.86, 1.36 [00:24:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjCy [00:24:50] [02miraheze/puppet] 07paladox 0302ab53c - Update dumps.pp [00:25:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjCQ [00:25:34] [02miraheze/puppet] 07paladox 038dea5c9 - Update dumps.pp [00:28:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjCh [00:29:00] [02miraheze/puppet] 07paladox 037971c52 - Update dumps.pp [00:30:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjWv [00:30:16] [02miraheze/puppet] 07paladox 03ed3c08e - Update dumps.pp [00:33:27] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2692 [00:40:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjWz [00:40:56] [02miraheze/puppet] 07paladox 0316bd3e1 - Convert rest of dumps to use swift for uploads [00:41:00] JohnLewis ^^ :) [00:42:14] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjWw [00:42:15] [02miraheze/mw-config] 07paladox 03a9739bb - Stop using nfs [00:42:16] and JohnLewis ^^ [00:42:21] now im going to remove the nfs client [00:42:30] woo [00:46:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjWH [00:46:32] [02miraheze/puppet] 07paladox 034a5dcb4 - Update dumps.pp [00:47:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjW5 [00:47:15] [02miraheze/puppet] 07paladox 0318c08d0 - Update dumps.pp [00:48:26] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.23, 1.70, 1.98 [00:48:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjWx [00:48:31] [02miraheze/puppet] 07paladox 03b0116cd - Remove nfs client from mw* [00:48:31] JohnLewis ^^! [00:54:27] RECOVERY - Current Load on misc1 is OK: OK - load average: 0.80, 1.23, 1.68 [00:54:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjlt [00:54:44] [02miraheze/mw-config] 07paladox 0301ad8be - Update LocalWiki.php [00:58:48] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±7] 13https://git.io/fbjlC [00:58:50] [02miraheze/puppet] 07JohnFLewis 03312da75 - remove phab from misc1 [00:58:51] [02miraheze/puppet] 07JohnFLewis 030aae510 - Merge branch 'master' of github.com:miraheze/puppet [00:59:05] JohnLewis woooo! [00:59:10] JohnLewis should i switch the dns? [01:00:09] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjl8 [01:00:11] [02miraheze/dns] 07JohnFLewis 038d59b85 - change DNS [01:00:15] no because I'm doing the change [01:00:35] ok [01:04:20] paladox: https://phabricator.miraheze.org/T3313 you can have the grafana one because I really cba to handle the grafana::modules key in hiera [01:04:21] Title: [ ⚓ T3313 Move grafana to misc1 ] - phabricator.miraheze.org [01:04:37] ok [01:04:41] i will do that later [01:05:51] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:05:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fbjlX [01:05:54] [02miraheze/puppet] 07paladox 03a75c1a2 - Add swiftExport which contains stuff for swift command [01:06:00] [02puppet] 07paladox opened pull request 03#765: Add swiftExport which contains stuff for swift command - 13https://git.io/fbjl1 [01:06:31] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjlS [01:06:33] [02miraheze/puppet] 07JohnFLewis 0365771b2 - move phabstatic backup to misc4 [01:06:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/fbjlQ [01:06:54] [02miraheze/puppet] 07paladox 035802dc5 - Create swiftExport [01:06:56] [02puppet] 07paladox synchronize pull request 03#765: Add swiftExport which contains stuff for swift command - 13https://git.io/fbjl1 [01:06:59] [02puppet] 07paladox closed pull request 03#765: Add swiftExport which contains stuff for swift command - 13https://git.io/fbjl1 [01:07:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fbjl7 [01:07:02] [02miraheze/puppet] 07paladox 038f7f7ff - Add swiftExport which contains stuff for swift command (#765) * Add swiftExport which contains stuff for swift command * Create swiftExport [01:07:12] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [01:07:13] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [01:11:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj8f [01:11:43] [02miraheze/puppet] 07paladox 03192fce2 - Update swift.pp [01:11:56] PROBLEM - Bacula - Phabricator Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:13:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj8k [01:13:29] [02miraheze/puppet] 07paladox 03b8083d6 - Update swift.pp [01:13:48] PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:15:46] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:16:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj83 [01:16:28] [02miraheze/puppet] 07paladox 03b92d21d - Switch to misc4 for swift proxy [01:16:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj8s [01:16:54] [02miraheze/mw-config] 07paladox 033fbe072 - Switch to misc4 for swift uploads [01:17:02] !log removed phabricator from misc1 [01:17:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:17:39] why use IP and not hostname? [01:18:52] JohnLewis it had problems in static.miraheze.org nginx [01:18:58] it coulden't find the dns for that domain [01:19:28] okay... strange [01:23:55] !log purged binlogs on db4 [01:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:24:26] !log reinstalling nfs1 as a swift cluster [01:24:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:25:07] RECOVERY - Disk Space on db4 is OK: DISK OK - free space: / 92157 MB (25% inode=95%): [01:29:57] PROBLEM - Bacula - Static on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:30:57] PROBLEM - Puppet on nfs1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:37:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj8N [01:37:37] [02miraheze/puppet] 07paladox 03eb157bb - Update prometheus.yml [01:38:45] RECOVERY - Disk Space on nfs1 is OK: DISK OK - free space: / 152700 MB (99% inode=99%): [01:38:47] PROBLEM - Puppet on nfs1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [01:39:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj4e [01:39:14] [02miraheze/dns] 07paladox 0312093a3 - Add swift2 to dns [01:40:54] RECOVERY - Puppet on nfs1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [01:44:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj4I [01:44:22] [02miraheze/puppet] 07paladox 03ecbab50 - Update site.pp [01:44:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/fbj4L [01:44:41] [02miraheze/puppet] 07paladox 039135307 - Create swift2.yaml [01:51:36] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fbj48 [01:51:37] [02miraheze/rings] 07paladox 03fad90e3 - Add swift2 to rings [01:54:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj4a [01:54:30] [02miraheze/puppet] 07paladox 031b518ea - Update swift.pp [01:55:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbj4V [01:55:20] [02miraheze/puppet] 07paladox 0370fdcaf - Update swift.pp [02:29:55] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fbjBQ [02:29:57] [02miraheze/rings] 07paladox 0334c0fd8 - Update ring files [02:36:10] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fbjRf [02:36:12] [02miraheze/rings] 07paladox 03e12a3d4 - Update ring files [02:42:06] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbjRq [02:42:08] [02miraheze/rings] 07paladox 031a1daab - Update ring files [02:42:24] PROBLEM - Puppet on misc4 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [02:44:24] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:45:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjRC [02:45:45] [02miraheze/puppet] 07paladox 030305843 - Update swiftExport [02:49:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjRB [02:49:23] [02miraheze/puppet] 07paladox 03f7830c8 - Update object-server.conf.erb [02:49:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjRR [02:50:01] [02miraheze/puppet] 07paladox 03f728bf4 - Update container-server.conf.erb [02:54:14] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbjRg [02:54:15] [02miraheze/rings] 07paladox 03f30f1c4 - Update rings [02:55:44] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 45% [02:57:44] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 31% [03:00:34] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbjRo [03:00:36] [02miraheze/rings] 07paladox 03daac694 - Update ring files [03:04:51] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbjRD [03:04:53] [02miraheze/rings] 07paladox 03a0612fb - Update ring files [03:11:03] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbjRA [03:11:05] [02miraheze/rings] 07paladox 036d1dcd9 - Update ring files [03:14:16] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbj0q [03:14:18] [02miraheze/rings] 07paladox 0382a253e - Update ring files [03:17:39] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbj0Z [03:17:40] [02miraheze/rings] 07paladox 03d610885 - Update ring files [03:22:38] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fbj0B [03:22:40] [02miraheze/rings] 07paladox 03b7eed17 - Update ring files [03:26:41] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fbj0o [03:26:43] [02miraheze/rings] 07paladox 03b890368 - Update rings [03:29:05] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [05:19:54] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbjV2 [05:19:55] [02miraheze/mw-config] 07Reception123 0328b1354 - add docs.websmart.media custom domain per T3291 resolves T3291 [07:40:14] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw3 [07:40:24] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [07:42:14] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [07:42:24] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [08:17:05] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5478 [08:41:53] Hello Luc_or_Lubaf! If you have any questions feel free to ask and someone should answer soon. [08:44:15] Hello LucOfAllTheTrope! If you have any questions feel free to ask and someone should answer soon. [08:44:58] I can't log into All The Tropes. It's giving me a "session hijacking" error. [08:45:20] I can log into Miraheze main, though. [08:45:24] Wat Do? [08:46:30] ... [09:07:50] Anybody here? [09:11:31] yeah? [09:13:08] JohnLewis: You know anything about a "session hijacking" error? [09:13:35] yes, it's been an error since we began [09:13:51] I can't log into All The Tropes because of it. I can get into Miraheze, though. [09:14:05] JohnLewis: How to fix it, then? [09:14:13] *I can log into Miraheze [09:14:25] renavigate to login on allthetropes [09:14:51] login is unified so logging into any wiki should log you in everywhere vanway [09:16:37] It's not working on ATT, is the thing. [09:17:34] I suspect the Login mechanism on ATT is busted, because I'm logged into Miraheze, but not ATT. [09:17:43] Which is confusing. [09:17:56] it's the exact same thing [09:19:10] Not on my browser. [09:25:07] JohnLewis: um didn't we disable that so now you need to login to custom domains again? [09:25:22] no [09:25:31] we fixed the behaviour [09:25:46] login is truly global [09:26:59] Not right now, it isn't. [09:27:16] it is for me [09:27:57] Could you try logging out of one and checking the other? [09:31:03] login on allthetropes works for me [09:31:28] ...okay, so it's local to me. [09:46:37] The exact error I'm getting, just FYI: "There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Please resubmit the form. " [12:16:04] PROBLEM - froggy.info - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:06] PROBLEM - phab.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:16:20] RECOVERY - phab.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate phab.miraheze.wiki will expire on Sun 12 Aug 2018 02:32:16 PM GMT +0000. [12:53:18] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [12:53:48] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [12:53:58] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [12:55:18] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [12:55:48] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [12:55:58] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [13:05:59] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [13:34:59] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [13:40:59] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:49:40] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fNeT7 [13:49:41] [02miraheze/rings] 07paladox 03457df72 - Pool swift2 with weight of 25 [13:49:47] !log pooling swift2 with weight of 25 [13:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:09:59] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [14:10:26] yep, known, swift1 high load [14:10:32] trying to get swift2 into the pool [14:10:38] without users seeing images disapear [14:11:59] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [14:22:58] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [15:50:22] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fNenb [15:50:23] [02miraheze/rings] 07paladox 031892170 - Update ring files [15:51:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+28/-0/±0] 13https://git.io/fNenh [15:51:02] [02miraheze/puppet] 07paladox 036f56db5 - Add rsync module from wmf repo [15:52:59] PROBLEM - Puppet on swift1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_swift] [15:54:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNecC [15:54:30] [02miraheze/puppet] 07paladox 036513cac - Add firewall for rsync [15:54:44] known about swift1, i am currently trying to add rsync [15:54:48] to sync containers / accounts [15:54:57] PROBLEM - Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_swift] [15:57:18] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:57:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNecM [15:57:21] [02miraheze/puppet] 07paladox 03f76a915 - Add rsync to swift backend [15:57:22] [02puppet] 07paladox opened pull request 03#766: Add rsync to swift backend - 13https://git.io/fNecy [15:58:40] [02puppet] 07paladox synchronize pull request 03#766: Add rsync to swift backend - 13https://git.io/fNecy [15:58:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNecN [15:58:43] [02miraheze/puppet] 07paladox 03dc543ae - Update backend.pp [15:59:02] [02puppet] 07paladox closed pull request 03#766: Add rsync to swift backend - 13https://git.io/fNecy [15:59:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNecp [15:59:05] [02miraheze/puppet] 07paladox 03581c6a6 - Add rsync to swift backend (#766) * Add rsync to swift backend * Update backend.pp [15:59:35] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:59:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [16:00:58] RECOVERY - Puppet on nfs1 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [16:01:00] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [16:02:58] RECOVERY - Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:11:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeWT [16:11:08] [02miraheze/puppet] 07paladox 030725369 - Update container-server.conf.erb [16:25:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNelc [16:25:08] [02miraheze/puppet] 07paladox 03788805a - Update swift.pp [16:32:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe8C [16:32:33] [02miraheze/puppet] 07paladox 03302b346 - Update container-server.conf.erb [16:32:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe88 [16:32:50] [02miraheze/puppet] 07paladox 03a584080 - Update account-server.conf.erb [16:34:48] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [16:36:48] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 55% [16:38:48] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 22% [16:38:58] PROBLEM - Puppet on swift2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [16:43:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe4n [16:43:05] [02miraheze/puppet] 07paladox 03e7e24cc - Update backend.pp [16:43:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe4C [16:43:24] [02miraheze/puppet] 07paladox 0350da68c - Update swift2.yaml [16:45:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe4r [16:45:13] [02miraheze/puppet] 07paladox 030d65d8c - Update backend.pp [16:46:58] RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:14:58] PROBLEM - Puppet on swift2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [17:21:58] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.45, 3.15, 2.44 [17:23:59] RECOVERY - Current Load on swift1 is OK: OK - load average: 2.34, 2.92, 2.44 [17:24:49] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [17:26:49] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 57% [17:26:59] RECOVERY - Puppet on swift2 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [17:28:59] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 36% [17:39:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNezy [17:39:43] [02miraheze/mw-config] 07paladox 03f0dc0c9 - Update Sitenotice.php [17:44:07] [02puppet] 07MacFan4000 opened pull request 03#767: Bacula phabstatic - phab is on misc4 - 13https://git.io/fNegJ [17:44:15] [02puppet] 07MacFan4000 edited pull request 03#767: Bacula phabstatic - phab is on misc4 - 13https://git.io/fNegJ [17:46:59] [02puppet] 07paladox closed pull request 03#767: Bacula phabstatic - phab is on misc4 - 13https://git.io/fNegJ [17:47:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNegl [17:47:02] [02miraheze/puppet] 07MacFan4000 0318e322f - Bacula phabstatic - phab is on misc4 (#767) I think @JohnFLewis may have missed this earlier [17:57:50] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe2w [17:57:52] [02miraheze/mw-config] 07MacFan4000 03dfc9f1e - Increase ID [19:02:37] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fNe6s [19:02:39] [02miraheze/rings] 07paladox 03fd72e35 - Update ring files [19:14:18] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:14:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeiW [19:14:21] [02miraheze/puppet] 07paladox 0315ebf38 - swift: Set fallocate_reserve [19:14:22] [02puppet] 07paladox opened pull request 03#768: swift: Set fallocate_reserve - 13https://git.io/fNeil [19:15:12] [02puppet] 07paladox synchronize pull request 03#768: swift: Set fallocate_reserve - 13https://git.io/fNeil [19:15:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNei0 [19:15:15] [02miraheze/puppet] 07paladox 033ffd8b5 - Update object-server.conf.erb [19:15:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeiz [19:15:30] [02miraheze/puppet] 07paladox 033ca71e8 - Update container-server.conf.erb [19:15:32] [02puppet] 07paladox synchronize pull request 03#768: swift: Set fallocate_reserve - 13https://git.io/fNeil [19:15:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeig [19:15:44] [02miraheze/puppet] 07paladox 036b182e1 - Update account-server.conf.erb [19:15:46] [02puppet] 07paladox synchronize pull request 03#768: swift: Set fallocate_reserve - 13https://git.io/fNeil [19:16:43] [02puppet] 07paladox closed pull request 03#768: swift: Set fallocate_reserve - 13https://git.io/fNeil [19:16:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/fNeio [19:16:46] [02miraheze/puppet] 07paladox 032d1cef2 - swift: Set fallocate_reserve (#768) * swift: Set fallocate_reserve * Update object-server.conf.erb * Update container-server.conf.erb * Update account-server.conf.erb [19:16:48] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:16:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [20:37:36] [02miraheze/services] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeSQ [20:37:38] [02miraheze/services] 07JohnFLewis 03cb4494e - add oryza [20:38:53] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fNeSF [20:38:54] [02miraheze/mediawiki] 07paladox 038a9c9ae - Update JSBreadCrumbs to upstream [20:48:14] [02miraheze/rings] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/fNe9i [20:48:15] [02miraheze/rings] 07paladox 0328e3dcb - Update ring files [21:17:47] [02CreateWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpJTL [21:17:49] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNe7Z [21:17:50] [02miraheze/CreateWiki] 07paladox 03e2f3a6f - Use setZoneAccess.php script if MirahezeMagic is installed [21:17:52] [02CreateWiki] 07paladox opened pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [21:27:59] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNe7j [21:28:01] [02miraheze/MirahezeMagic] 07paladox 03e7c18a1 - Add more logging to setZoneAccess.php [21:31:15] JohnLewis ^^ [21:31:19] does that look good? [21:31:58] yeah [21:54:43] [02CreateWiki] 07paladox synchronize pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [21:54:45] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeFU [21:54:46] [02miraheze/CreateWiki] 07paladox 0320decbb - Update SpecialCreateWiki.php [21:54:55] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeFk [21:54:56] [02miraheze/CreateWiki] 07paladox 03cbfda9e - Update SpecialCreateWiki.php [21:54:58] [02CreateWiki] 07paladox synchronize pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [21:56:51] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeFl [21:56:52] [02miraheze/CreateWiki] 07paladox 0326350e1 - Update en.json [21:56:54] [02CreateWiki] 07paladox synchronize pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [21:58:30] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeFg [21:58:31] [02miraheze/CreateWiki] 07paladox 03b1a4647 - Update SpecialCreateWiki.php [21:58:33] [02CreateWiki] 07paladox synchronize pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [21:59:35] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/fNeFr [21:59:37] [02miraheze/CreateWiki] 07paladox 034cb4aeb - Update qqq.json [21:59:38] [02CreateWiki] 07paladox synchronize pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [22:01:37] [02CreateWiki] 07JohnFLewis closed pull request 03#82: Use setZoneAccess.php script if MirahezeMagic is installed - 13https://git.io/fNe7n [22:01:39] [02miraheze/CreateWiki] 07JohnFLewis pushed 037 commits to 03master [+0/-0/±9] 13https://git.io/fNeFS [22:01:40] [02miraheze/CreateWiki] 07JohnFLewis 03d33369e - Merge pull request #82 from miraheze/paladox-patch-1 Use setZoneAccess.php script if MirahezeMagic is installed [22:01:49] wooo thanks JohnLewis! [22:02:54] wow Ive made more changes than SPF|Cloud to CW [22:03:03] cheh [22:03:06] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_31 [+0/-0/±2] 13https://git.io/fNeFd [22:03:07] [02miraheze/mediawiki] 07paladox 036c3d7a2 - Update MM and CW [22:03:44] [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL [22:03:45] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1 [22:04:48] !log sudo -u www-data php /srv/mediawiki/w/maint*/rebuildLocalisationCache.php --wiki test1wiki on mw* [22:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:20:41] .mangle [22:20:53] MacFan4000: Mirage Logbot said, "You are logged in at https://meta.miraheze.org/wiki/Tech: server_admin_log, sir" [22:22:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeNt [22:22:33] [02miraheze/mw-config] 07paladox 030e23b78 - Update FileBackend.php [22:41:58] PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 4.39, 3.77, 3.20 [22:43:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeAm [22:43:28] [02miraheze/puppet] 07paladox 03cc74395 - Reduce workers to 1 [22:43:58] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.78, 3.76, 3.27 [22:44:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeAs [22:44:01] [02miraheze/puppet] 07paladox 03a6f0119 - Update object-server.conf.erb [22:53:58] RECOVERY - Current Load on swift1 is OK: OK - load average: 2.72, 3.20, 3.23 [22:59:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNexI [22:59:58] [02miraheze/puppet] 07paladox 0327cc360 - Update backend.pp [23:00:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNext [23:00:22] [02miraheze/puppet] 07paladox 037e7efdf - Update swift2.yaml [23:12:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNexi [23:12:11] [02miraheze/puppet] 07paladox 037bcd89b - Update object-server.conf.erb [23:36:57] [02puppet] 07MacFan4000 opened pull request 03#769: Add mw-admins swift rules - 13https://git.io/fNehW [23:42:39] [02puppet] 07paladox closed pull request 03#769: Add mw-admins swift rules - 13https://git.io/fNehW [23:42:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNeh6 [23:42:42] [02miraheze/puppet] 07MacFan4000 039110996 - Add mw-admins swift rules (#769) [23:50:58] PROBLEM - Current Load on nfs1 is CRITICAL: CRITICAL - load average: 20.23, 21.02, 9.70 [23:51:00] PROBLEM - Current Load on swift2 is CRITICAL: CRITICAL - load average: 18.61, 20.67, 9.64 [23:54:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNejJ [23:54:19] [02miraheze/puppet] 07paladox 03b53fc6e - Update object-server.conf.erb [23:54:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNejT [23:54:28] [02miraheze/puppet] 07paladox 031fbe640 - Update container-server.conf.erb [23:54:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fNejL [23:54:37] [02miraheze/puppet] 07paladox 03678366c - Update account-server.conf.erb