[00:00:37] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [00:02:37] PROBLEM - Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [00:03:16] 503 :( [00:03:16] Wiki-1776: The servers should return back to normal shortly. [00:03:47] PROBLEM - Disk Space on swift1 is CRITICAL: DISK CRITICAL - free space: / 16540 MB (10% inode=96%): [00:08:27] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [00:08:50] Wiki-1776 hello [00:08:58] we are aware and trying to resolve the problem [00:10:27] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 14% [00:20:05] what is swift paladox ? [00:20:27] Wiki-1776 openstack swift, it allows us to clusterise our file uploader allowing us to add more storage. [00:20:35] we are running low on nfs1 [00:20:40] so we had to buy a extra server [00:20:56] which required us switching to something that allowed us to use both servers to store data [00:21:18] ah, ok. thanks paladox [00:21:25] your welcome :) [00:21:35] We are being careful with swift in case we need to revert [00:34:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fF9lT [00:34:05] [02miraheze/puppet] 07paladox 03d8aa7f3 - swift: Set workers to 1 [00:36:28] wouldn't it be good to put some links in the sitenotice? [00:37:21] Voidwalker im deciding weather to roll back [00:37:25] swift1 has high load [00:40:42] so does taylor [00:41:07] lol [00:42:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fF5sa [00:42:20] [02miraheze/puppet] 07paladox 036889dab - include role::swift on nfs1 [00:42:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fF52o [00:42:42] [02miraheze/puppet] 07paladox 036031133 - Update nfs1.yaml [00:43:12] i presume highload due to memcached [00:52:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fFhup [00:52:47] [02miraheze/puppet] 07paladox 03e08e9cb - Update container-server.conf.erb [00:53:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fFh6m [00:53:06] [02miraheze/puppet] 07paladox 038c71711 - Update account-server.conf.erb [00:53:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fFhNy [00:53:20] [02miraheze/puppet] 07paladox 03f7c4ccc - Update object-server.conf.erb [00:55:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbedz [00:55:48] [02miraheze/puppet] 07paladox 03fc651eb - Update container-server.conf.erb [00:56:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbve4 [00:56:01] [02miraheze/puppet] 07paladox 030e3c61e - Update account-server.conf.erb [00:56:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbvs0 [00:56:13] [02miraheze/puppet] 07paladox 033657bf2 - Update object-server.conf.erb [00:56:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbvKl [00:56:43] [02miraheze/puppet] 07paladox 031fe401b - Increase client_max_body_size to 100mb [00:57:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbvFX [00:57:09] [02miraheze/puppet] 07paladox 03a288f85 - Increase client_max_body_size to 100M [01:03:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbURp [01:03:19] [02miraheze/puppet] 07paladox 0337a625d - Update account-server.conf.erb [01:04:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbU0e [01:04:27] [02miraheze/puppet] 07paladox 037c597ea - Update container-server.conf.erb [01:04:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbU0v [01:04:50] [02miraheze/puppet] 07paladox 03c73dae7 - Update object-server.conf.erb [01:18:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbUSb [01:18:16] [02miraheze/puppet] 07paladox 038cf2191 - Update rewrite.py [01:26:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbqzA [01:26:03] [02miraheze/puppet] 07paladox 0328308be - Update proxy-server.conf.erb [01:26:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbq2I [01:26:15] [02miraheze/puppet] 07paladox 039761b95 - Update container-server.conf.erb [01:26:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbqXJ [01:26:25] [02miraheze/puppet] 07paladox 033c83e21 - Update account-server.conf.erb [01:26:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbqFj [01:26:35] [02miraheze/puppet] 07paladox 03e89339b - Update object-server.conf.erb [01:34:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbGi3 [01:34:18] [02miraheze/puppet] 07paladox 037878222 - Update proxy-server.conf.erb [01:38:39] Hello Jordan! If you have any questions feel free to ask and someone should answer soon. [01:40:41] Hello rajavlitra! If you have any questions feel free to ask and someone should answer soon. [01:41:34] uh hi, since the farm moved to swift my images got lost, anyone help me get them back? (my wiki is adiaproject.miraheze.org) [01:43:08] ???? [01:43:12] paladox, ^ [01:43:25] hi [01:43:27] checking [01:43:40] I seem to be getting an authorization error if I visit the static file [01:43:50] Seems it is a new wiki? [01:44:03] nope [01:44:20] though i see it [01:44:29] paladox, I'm seeing the same thing on meta [01:44:36] Voidwalker oh, where? [01:44:49] https://meta.miraheze.org/wiki/File:Response_made_by_sysadmin_of_myht.png [01:44:51] Title: [ File:Response made by sysadmin of myht.png - Miraheze Meta ] - meta.miraheze.org [01:45:08] actually, all of Special:ListFiles [01:45:23] ah [01:45:29] it may be trying to save thumberniles [01:45:34] i've been looking into this [01:45:37] Error creating thumbnail: Unable to save thumbnail to destination [01:45:44] as POST, put are failing [01:45:46] https://static.miraheze.org/metawiki/d/d5/Response_made_by_sysadmin_of_myht.png works for [01:46:42] when I visit that, I get "Unauthorized This server could not verify that you are authorized to access the document you requested." [01:46:55] yeh [01:46:57] hmm [01:47:07] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [01:47:17] Voidwalker visit https://static.miraheze.org/metawiki/d/d5/Response_made_by_sysadmin_of_myht.png ? [01:47:23] it returns unautherised? [01:47:45] yup [01:47:54] hmm [01:48:02] ok, i will revert back to nfs1 then [01:48:17] as post is failing [01:48:21] so i cannot change anything [01:49:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbl4S [01:49:03] [02miraheze/puppet] 07paladox 03b3226e4 - Revert "Update common.yaml" This reverts commit 80dd5a2c14f9320a70117417417050c46f65489c. [01:52:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbl45 [01:52:05] [02miraheze/mw-config] 07paladox 038db2128 - Update site notice [01:55:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbl4A [01:55:57] [02miraheze/mw-config] 07paladox 03b106fb4 - Update LocalSettings.php [02:12:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbaJQ [02:12:05] [02miraheze/mw-config] 07paladox 0328ece14 - Update LocalWiki.php [03:51:09] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 5427 [06:33:27] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [06:39:27] PROBLEM - wiki.itspugle.ga - LetsEncrypt on sslhost is CRITICAL: Name or service not known [06:41:27] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:39:07] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [07:59:06] PROBLEM - JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 2644 [08:03:26] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [08:11:27] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [09:03:31] .tell JohnLewis https://meta.miraheze.org/w/index.php?title=Special:WikiDiscover&sort=wiki_closed_timestamp&limit=50&asc=&desc=1 - looks like WikiDiscover is saying that no wikis are deleteable per the DP, is that normal? [09:03:32] Reception123: I'll pass that on when JohnLewis is around. [09:03:33] Title: [ WikiDiscover - Miraheze Meta ] - meta.miraheze.org [09:36:38] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fbxKj [09:36:39] [02miraheze/ssl] 07Reception123 035a3925a - add wiki.taotac.org cert [09:41:50] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/fbx6n [09:41:51] [02miraheze/ssl] 07Reception123 039f094f6 - remove wiki.itspugle.ga cert Not pointing to us [09:43:29] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbx6l [09:43:30] [02miraheze/mw-config] 07Reception123 037346c6f - add wiki.taotac.org custom domain per T3304 [09:44:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbx60 [09:44:24] [02miraheze/mw-config] 07Reception123 0312626f6 - remove wiki.itspugle.ga custom domain no longer pointing to us [09:47:05] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.91, 1.66, 1.17 [09:53:01] RECOVERY - Current Load on misc1 is OK: OK - load average: 0.90, 1.53, 1.29 [10:04:02] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/fbxix [10:04:04] [02miraheze/ssl] 07Reception123 03cdacec8 - add docs.websmart.media cert [10:25:27] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [10:31:26] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [10:45:26] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [10:49:17] RECOVERY - JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [10:55:34] Reception123: yes because unless I misread the dormancy policy a wiki must be closed for 120 days before it can be deleted. [10:55:35] JohnLewis: 2018-07-04 - 09:03:32 tell JohnLewis https://meta.miraheze.org/w/index.php?title=Special:WikiDiscover&sort=wiki_closed_timestamp&limit=50&asc=&desc=1 - looks like WikiDiscover is saying that no wikis are deleteable per the DP, is that normal? [10:57:48] [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxMV [10:57:50] [02miraheze/WikiDiscover] 07JohnFLewis 032aceee8 - 129 days not 180 [11:01:27] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [11:02:00] [02miraheze/mediawiki] 07JohnFLewis pushed 031 commit to 03REL1_31 [+0/-0/±1] 13https://git.io/fbxMy [11:02:01] [02miraheze/mediawiki] 07JohnFLewis 03aa42769 - update WD [11:02:47] paladox: fyi, your site notice is... too complicated :) [11:04:45] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxMx [11:04:46] [02miraheze/mw-config] 07JohnFLewis 03f8b82cd - make sitenotice more user friendly [11:11:51] JohnLewis: heh, were uploads re enabled? [11:12:10] [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxD8 [11:12:12] [02miraheze/mw-config] 07JohnFLewis 03a581ac6 - change notice [11:12:19] paladox: your notice failed to mention they were still disabled [11:12:37] JohnLewis: you can enable it [11:12:41] (Uploads) [11:13:33] [02miraheze/mw-config] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fbxDV [11:13:35] [02miraheze/mw-config] 07JohnFLewis 03fbc5cbf - enable uploads [11:13:36] [02miraheze/mw-config] 07JohnFLewis 03433d9a4 - Revert "change notice" This reverts commit a581ac690be4690993e9da28a980ff45c4e1fc81. [13:18:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxFZ [13:18:22] [02miraheze/puppet] 07paladox 036d1cce1 - Update mediawiki.conf [13:31:47] PROBLEM - MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:37] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [13:33:37] RECOVERY - MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 30960 bytes in 0.142 second response time [13:34:37] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [13:38:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxbA [13:38:29] [02miraheze/mw-config] 07paladox 0371a9fee - Switch to swift again [13:39:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxbj [13:39:41] [02miraheze/puppet] 07paladox 03edaf305 - Switch on swift again [13:43:27] PROBLEM - Current Load on swift1 is CRITICAL: CRITICAL - load average: 5.78, 2.95, 1.34 [13:44:36] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [13:44:46] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:8902::f03c:91ff:fe07:444e/cpweb [13:46:46] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:49:26] PROBLEM - Current Load on swift1 is WARNING: WARNING - load average: 3.24, 3.85, 2.29 [13:53:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxAl [13:53:07] [02miraheze/puppet] 07paladox 03d703542 - Update proxy-server.conf.erb [13:53:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxA0 [13:53:53] [02miraheze/puppet] 07paladox 03f69594f - Update object-server.conf.erb [13:54:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxAE [13:54:01] [02miraheze/puppet] 07paladox 03335fc01 - Update container-server.conf.erb [13:54:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxAu [13:54:10] [02miraheze/puppet] 07paladox 0327eff44 - Update account-server.conf.erb [13:55:26] RECOVERY - Current Load on swift1 is OK: OK - load average: 1.62, 2.95, 2.43 [14:05:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxx1 [14:05:25] [02miraheze/puppet] 07paladox 0339a63fd - Update account-server.conf.erb [14:06:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxx5 [14:06:36] [02miraheze/puppet] 07paladox 03d15a95d - Update container-server.conf.erb [14:07:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbxxx [14:07:39] [02miraheze/puppet] 07paladox 037bad6d5 - Update object-server.conf.erb [14:36:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpeT [14:36:50] [02miraheze/puppet] 07paladox 0348b47a6 - Update proxy-server.conf.erb [14:37:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpet [14:37:34] [02miraheze/puppet] 07paladox 036fe0e9b - Update proxy-server.conf.erb [14:38:37] PROBLEM - Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [14:39:17] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:39:47] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [14:39:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpec [14:39:54] [02miraheze/puppet] 07paladox 03412bffd - Update proxy-server.conf.erb [14:41:25] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [14:43:27] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 27% [14:44:35] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [14:44:37] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [14:45:45] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:48:35] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [14:49:16] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [15:03:36] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw2 [15:03:46] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [15:04:16] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 107.191.126.23/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [15:16:16] [02services] 07The-Voidwalker opened pull request 03#143: services on ckc for T3307 - 13https://git.io/fbpJd [15:17:29] [02services] 07paladox closed pull request 03#143: services on ckc for T3307 - 13https://git.io/fbpJd [15:17:31] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpUf [15:17:32] [02miraheze/services] 07The-Voidwalker 03819eb39 - services on ckc for T3307 (#143) [15:19:42] when it's Miraheze's anniversary ? [15:20:17] [02services] 07The-Voidwalker opened pull request 03#144: services on valentinawiki for T3223 - 13https://git.io/fbpUY [15:21:09] [02services] 07paladox closed pull request 03#144: services on valentinawiki for T3223 - 13https://git.io/fbpUY [15:21:11] [02miraheze/services] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpUc [15:21:12] [02miraheze/services] 07The-Voidwalker 033439350 - services on valentinawiki for T3223 (#144) [15:22:36] PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.02, 1.82, 1.34 [15:24:36] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.16, 1.56, 1.30 [16:02:36] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [16:03:06] meh [16:03:11] known [16:03:46] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [16:04:36] PROBLEM - Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [16:05:46] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:06:36] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [16:06:38] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [16:08:36] RECOVERY - Varnish Backends on cp4 is OK: All 3 backends are healthy [16:14:07] PROBLEM - HTTP on test1 is CRITICAL: connect to address 185.52.2.243 and port 80: Connection refused [16:14:27] PROBLEM - MediaWiki Rendering on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refused [16:15:17] PROBLEM - HTTPS on test1 is CRITICAL: connect to address 185.52.2.243 and port 443: Connection refused [16:15:19] PROBLEM - Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 24 seconds ago with 0 failures [16:19:15] RECOVERY - HTTPS on test1 is OK: HTTP OK: HTTP/1.1 302 Moved Temporarily - 405 bytes in 0.026 second response time [16:20:07] RECOVERY - HTTP on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 409 bytes in 0.002 second response time [16:20:27] RECOVERY - MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 30993 bytes in 0.347 second response time [16:31:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpt0 [16:31:58] [02miraheze/puppet] 07paladox 03b6b629e - add nginx static cache for static.miraheze.org [16:33:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbptw [16:33:25] [02miraheze/puppet] 07paladox 03158ed04 - Update mediawiki.conf [16:35:16] RECOVERY - Puppet on test1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:40:36] PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.21, 1.67, 1.24 [16:42:37] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.83, 1.71, 1.31 [16:46:37] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.18, 1.39, 1.26 [17:01:36] PROBLEM - Varnish Backends on cp2 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [17:03:36] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [18:23:27] PROBLEM - Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Apt_key[Add key: F86AA916A2195E121AEDB11437BBEE3F7AD95B3F from Apt::Source grafana_apt] [18:24:37] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw2 mw3 [18:24:47] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [18:26:17] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 172.104.111.8/cpweb [18:26:37] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [18:26:47] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:28:17] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [18:31:25] RECOVERY - Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:24:37] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.77, 1.40, 0.92 [19:26:37] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.28, 1.33, 0.95 [20:23:33] hi [21:27:19] [ANNOUNCEMENT] Happy 4th of July, to users in the US! [21:32:37] PROBLEM - Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw1 mw3 [21:32:47] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [21:32:47] Yeah July 4, 1776 XD [21:33:17] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [21:34:37] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [21:34:47] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:35:17] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:41:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpR0 [21:41:55] [02miraheze/puppet] 07paladox 03043ad58 - Update swift.pp [21:42:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpRg [21:42:39] [02miraheze/puppet] 07paladox 03f5c49a5 - Update mediawiki.conf [21:47:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpR5 [21:47:44] [02miraheze/puppet] 07paladox 03eda0674 - Update mediawiki.conf [21:52:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpRj [21:52:15] [02miraheze/puppet] 07paladox 03e2344dc - set swift_proxy: true for nfs1 [21:55:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbp0L [21:55:56] [02miraheze/puppet] 07paladox 031526f49 - Update mediawiki.conf [22:00:49] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbp0R [22:00:51] [02miraheze/dns] 07paladox 03400b92d - Update config [22:01:56] PROBLEM - Puppet on swift1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [22:02:36] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.72, 1.19, 0.74 [22:03:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbp0o [22:03:32] [02miraheze/puppet] 07paladox 03b099c81 - Update swift1.yaml [22:04:24] paladox: the amount of changes is annoying, you can live hack in this instance [22:04:37] RECOVERY - Current Load on misc1 is OK: OK - load average: 1.15, 1.14, 0.77 [22:04:52] JohnLewis ok [22:05:13] most because it isn't prod [22:05:29] JohnLewis what isen't prod? [22:05:34] swift1 is now prod [22:05:41] swift isnt a production service [22:05:44] i switched to swift today [22:05:55] so we're actually on swift? [22:05:59] yep [22:06:01] uploads work [22:06:08] hm [22:06:19] JohnLewis load seems to be down which is good [22:12:37] PROBLEM - Varnish Backends on cp5 is CRITICAL: 2 backends are down. mw1 mw2 [22:12:47] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:8902::f03c:91ff:fe07:444e/cpweb [22:14:14] huh? [22:14:35] RECOVERY - Varnish Backends on cp2 is OK: All 3 backends are healthy [22:14:37] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [22:14:45] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:20:48] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb [22:21:35] that's getting annoying [22:27:36] PROBLEM - Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [22:28:16] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 81.4.109.133/cpweb, 172.104.111.8/cpweb [22:28:26] And it depooled mw1 [22:31:36] PROBLEM - Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [22:32:36] PROBLEM - Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [22:37:47] RECOVERY - GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:39:57] RECOVERY - Puppet on swift1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [22:43:47] PROBLEM - GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 172.104.111.8/cpweb, 2400:8902::f03c:91ff:fe07:444e/cpweb [22:55:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpu4 [22:55:35] [02miraheze/mw-config] 07paladox 031ecc63b - Update FileBackend.php [23:08:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpuF [23:08:06] [02miraheze/mw-config] 07paladox 0385fdb71 - Update FileBackend.php [23:21:55] PROBLEM - Puppet on swift1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [23:23:12] Hello ClassifiedCat! If you have any questions feel free to ask and someone should answer soon. [23:23:25] On our wiki there is an error [23:23:28] "Error creating thumbnail: File missing" [23:23:37] You can see it on here f.e.x. [23:23:37] https://maiasongcontest.miraheze.org/wiki/Maia_Song_Contest_19#Grand_Final [23:23:39] Title: [ Maia Song Contest 19 - Maia Song Contest ] - maiasongcontest.miraheze.org [23:23:41] Any ways to fix this? [23:24:14] the error is that all the image files are missing [23:24:32] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fbpzu [23:24:34] [02miraheze/mw-config] 07paladox 03dd35515 - Revert "Update FileBackend.php" This reverts commit 85fdb71952946338c9204037990da3bc2567d363. [23:24:36] [02miraheze/mw-config] 07paladox 032af8565 - Revert "Update FileBackend.php" This reverts commit 1ecc63b2a6c060d804231e8bfabaa89528c6647b. [23:24:55] ClassifiedCat hello, we are looking into this. [23:25:14] Any idea of when there can be a fix? [23:26:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpzg [23:26:46] [02miraheze/mw-config] 07paladox 0335ee7cb - Update FileBackend.php [23:27:16] ClassifiedCat we are trying to fix this now. We just switched to swift for our file system management [23:28:12] ClassifiedCat did that start suddenly [23:28:18] or was that like that all day? [23:28:57] It started an hour ago but not on all pages [23:28:58] Only on a few [23:29:41] As you may be able to see, it's fine on this page https://maiasongcontest.miraheze.org/wiki/Maia_Song_Contest_18 [23:29:42] Title: [ Maia Song Contest 18 - Maia Song Contest ] - maiasongcontest.miraheze.org [23:29:54] But not here https://maiasongcontest.miraheze.org/wiki/Izola_in_the_Maia_Song_Contest [23:29:54] Title: [ Izola in the Maia Song Contest - Maia Song Contest ] - maiasongcontest.miraheze.org [23:30:32] ok [23:32:36] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.89, 1.56, 1.10 [23:36:46] PROBLEM - GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 107.191.126.23/cpweb, 2a00:d880:5:8ea::ebc7/cpweb, 172.104.111.8/cpweb [23:37:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpzy [23:37:53] [02miraheze/mw-config] 07paladox 03dc6dd35 - Update FileBackend.php [23:38:26] PROBLEM - MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:36] PROBLEM - Current Load on misc1 is CRITICAL: CRITICAL - load average: 2.04, 1.73, 1.30 [23:39:36] PROBLEM - MediaWiki Rendering on mw3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:40:16] RECOVERY - MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 30990 bytes in 0.622 second response time [23:40:26] PROBLEM - HTTP 4xx/5xx ERROR Rate on cp4 is WARNING: WARNING - NGINX Error Rate is 50% [23:40:46] RECOVERY - GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:42:24] ClassifiedCat works now [23:42:27] RECOVERY - HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 13% [23:42:28] fixed the issue [23:43:27] RECOVERY - MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 30990 bytes in 0.112 second response time [23:44:47] RECOVERY - Varnish Backends on cp5 is OK: All 3 backends are healthy [23:46:37] PROBLEM - Current Load on misc1 is WARNING: WARNING - load average: 1.66, 1.84, 1.54 [23:56:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fbpgC [23:56:28] [02miraheze/puppet] 07paladox 039227c40 - Update swift.pp [23:58:48] Thank you :D