[01:09:21] Hello robla[m]! If you have any questions, feel free to ask and someone should answer soon. [03:21:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3wE [03:21:37] [02miraheze/mw-config] 07paladox 03304de02 - Fix typo Per finding on discord [06:26:46] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3036 MB (12% inode=94%); [07:26:09] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-redis] [07:32:07] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [11:26:31] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-80],Exec[ufw-allow-tcp-from-any-to-any-port-443] [11:34:26] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:26:07] !log reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Tundra "Just Amelia" --logwiki=metawiki [12:26:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:32:47] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=94%); [17:24:43] PROBLEM - wiki.mxlinuxusers.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.org' expires in 15 day(s) (Mon 17 Feb 2020 05:20:42 PM GMT +0000). [17:25:00] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3db [17:25:01] [02miraheze/ssl] 07MirahezeSSLBot 03dd57580 - Bot: Update SSL cert for wiki.mxlinuxusers.org [17:27:21] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Mon 17 Feb 2020 05:24:26 PM GMT +0000). [17:27:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3Fe [17:27:36] [02miraheze/ssl] 07MirahezeSSLBot 0344b59f6 - Bot: Update SSL cert for storytime.jdstroy.cf [17:32:42] RECOVERY - wiki.mxlinuxusers.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.org' will expire on Fri 01 May 2020 04:24:53 PM GMT +0000. [17:33:18] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Fri 01 May 2020 04:27:28 PM GMT +0000. [17:46:37] Hello reschultzed! If you have any questions, feel free to ask and someone should answer soon. [17:50:24] Hi there, I'm trying to log in to Phabricator with my new meta.miraheze.org account, and it's telling me that my password is incorrect after I've checked it repeatedly. Then, when I try to reset my password, it tells me that there is no account associated with my email address. Do I need to create a separate account specifically for Phabricator, [17:50:25] like how each wiki has its own separate accounts? Because if so, I don't see how to do that. Thanks! [17:54:33] reschultzed: there should be a login with MediaWiki option [17:54:36] Use that [17:55:06] Each wiki account is linked to a global one though [18:01:47] RhinosF1: Thank you! [18:36:59] @RhinosF1 I cannot get my wiki account to auth [18:38:46] PuppyKun, Reception123: ^ [18:40:45] Examknow: bots slow [18:41:26] k [22:05:07] PROBLEM - cp2 Stunnel Http for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:05:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:05:29] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.014 second response time [22:05:37] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:05:39] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4140 bytes in 0.396 second response time [22:05:48] PROBLEM - cp2 Stunnel Http for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:05:49] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [22:05:59] PROBLEM - cp3 Stunnel Http for mw1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:06:04] PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:06:15] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 2055 bytes in 3.914 second response time [22:06:25] !log stopping mysql, removing bin logs and starting mysql on db4 [22:06:28] oops [22:06:34] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.014 second response time [22:06:36] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 96% [22:06:37] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.022 second response time [22:06:38] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [22:06:46] DB ran out of space :/ [22:06:51] yikes [22:06:53] hence why i need to delete bin logs [22:07:05] that explains the 503's [22:07:05] RECOVERY - cp2 Stunnel Http for mw2 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.391 second response time [22:07:08] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [22:07:23] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [22:07:31] yup [22:07:44] RECOVERY - cp2 Stunnel Http for mw1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 15301 bytes in 0.331 second response time [22:07:55] RECOVERY - cp3 Stunnel Http for mw1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15295 bytes in 0.505 second response time [22:08:02] RECOVERY - cp3 Stunnel Http for mw3 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 15302 bytes in 0.493 second response time [22:08:49] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (115) [22:09:17] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [22:09:38] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4215 bytes in 0.059 second response time [22:09:51] paladox|UKInEU: db4 or 5? [22:10:07] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 70% [22:10:09] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.121 second response time [22:10:33] Ah 4 [22:10:45] Whats space on both now? [22:11:16] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.066 second response time [22:11:29] db4 [22:11:37] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19067 bytes in 0.316 second response time [22:12:01] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18691 bytes in 1.101 second response time [22:12:07] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 2% [22:12:09] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18689 bytes in 0.897 second response time [22:12:09] RECOVERY - db5 Disk Space on db5 is OK: DISK OK - free space: / 39088 MB (21% inode=99%); [22:12:28] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 4% [22:12:35] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [22:12:37] paladox|UKInEU: time to put more on db5? [22:12:38] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18690 bytes in 0.246 second response time [22:12:38] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18690 bytes in 0.259 second response time [22:12:47] RECOVERY - db4 MySQL on db4 is OK: Uptime: 321 Threads: 64 Questions: 78609 Slow queries: 2742 Opens: 9439 Flush tables: 1 Open tables: 1000 Queries per second avg: 244.887 [22:13:02] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [22:13:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:13:12] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [22:13:27] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18690 bytes in 0.380 second response time [22:13:27] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1533 bytes in 0.527 second response time [22:13:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:13:29] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:13:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3jn [22:13:54] [02miraheze/mw-config] 07paladox 03ae45852 - Compress altversewiki [22:14:46] !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=altversewiki --type=gzip [22:14:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:15:03] RhinosF1 maybe [22:15:54] paladox|UKInEU: We need something soon [22:16:27] Db5 has 12G compared to 1 on db4 if i can read grafana so it’ll have an impact [22:17:04] db4 does not have 1 [22:17:09] where did you get 1 from? [22:17:50] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3ju [22:17:51] [02miraheze/mw-config] 07paladox 03247f09a - Fix wiki in wgCompressRevisions [22:18:36] paladox|UKInEU: https://grafana.miraheze.org/d/W9MIkA7iz/miraheze-cluster?orgId=1&var-job=node&var-node=db4.miraheze.org&var-port=9100&fullscreen&panelId=43 [22:18:36] !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=uncyclomirrorwiki --type=gzip [22:18:37] [ Grafana ] - grafana.miraheze.org [22:18:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:18:42] Says 6 now [22:18:51] I'm guessing most of Miraheze is based in the UK, but I hope one day that you consider expanding infrastructure and operations to the US one day [22:18:53] shows ~6 [22:19:06] jdstroy wondering why? :) [22:19:07] paladox|UKInEU: 38 on db5 [22:19:43] yup [22:19:47] jdstroy: we have an american cache proxy but most servers are in the netherlands. We’re a UK company though. [22:19:49] paladox|UKInEU: It's better not to ask... You'll find out one day if you consider doing so. [22:20:02] paladox|UKInEU: thats a lot of room to put something on [22:20:28] well not alot of room. [22:20:34] 38gb is not alot. [22:20:43] but some, so ability to move some wikis over. [22:21:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3jg [22:21:49] (Weird US law stuff that works a bit like the silly Fight Club rules) [22:21:50] [02miraheze/mw-config] 07paladox 03d765636 - Compress toxicfandomsandhatedomswiki [22:22:01] jdstroy oh? [22:22:47] yeah; there's good reason to expand to the US, but if the "reason" for doing so is "wrong", it negates the benefit [22:23:21] so it's better for me to say "it's good to do so for the purposes of colocating the data closer to users" [22:23:24] !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=toxicfandomsandhatedomswiki --type=gzip [22:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:23:43] in the future, when it makes sense [22:24:54] jdstroy: we can only effectively have one data centre at the moment or its too latent [22:25:16] you mean for replication? [22:25:17] Thats why were sometimes slow (you hit the london temp server) [22:25:23] jdstroy: yeah [22:25:31] paladox|UKInEU: its more than db4 has [22:25:46] idea -- maybe for the future, expansions [22:26:35] Maybe but it wont be close [22:26:45] * sharding (different masters for different groups of wikis) [22:27:25] * off-site backup/replica in other geo [22:27:30] jdstroy: itd be something to look up [22:27:43] We have off site backups of some stuff with backupsy [22:27:59] But i have no clue how recent or how much is on there [22:28:04] haha [22:28:05] ok [22:29:10] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jv3jP [22:29:11] [02miraheze/mw-config] 07paladox 037ceb054 - Compress metawiki [22:29:51] !log root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=metawiki --type=gzip [22:29:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [22:31:06] anyway, if you ever decide to have some operations in the US, I'd like to know [22:31:52] jdstroy i mean we have a cache proxy in the US. [22:31:57] we also have ns1 in the us. [22:32:13] not *quite* enough for incorporation in the US ;) [22:32:13] though that may be something to look at once we have enough funds. [22:32:23] jdstroy we incorporated in the UK though :) [22:32:33] yep. I know. [22:32:44] oh, i see what you mean [22:32:46] servers. [22:32:56] yeah [22:33:34] there's a lot of stuff that I can't really say, but you can *probably* figure it out on your own [22:33:40] but it's definitely NOT a good idea to ask about it. [22:34:56] intent should always be for growing Miraheze, largely under its current charter [22:35:54] Just an fyi, we've got an approved budget (which we've come up with a plan to improve the infra to improve reliability) [22:36:10] yay :) [22:36:12] paladox|UKInEU: is it public? [22:36:23] No, it was an emergency budget [22:36:34] well not emergency per see [22:36:39] but to approve it was an urgent [22:36:42] no real announcement, but I think we can talk about it [22:37:04] Voidwalker: It'd be nice to see a public copy of as much as can be [22:37:05] We've purchased 2 new servers and will be moving our infra onto it. [22:37:10] ok [22:37:37] and we've lowered our costs whilsts increasing capacity :) [22:37:44] good [22:38:39] do y'all have stats on registered users by approximate demographics? [22:38:54] in particular country [22:39:23] Nope, i doin't think we track that info. [22:39:45] figured not [22:39:47] oh well. [22:40:04] jdstroy: i mean which demographic? If you want just country accessed from then Special:Analytics will show it [22:40:15] * jdstroy checks [22:40:17] RhinosF1 he wants registered users. [22:40:22] that shows all users. [22:40:54] it might be close enough. [22:41:11] paladox|UKInEU: loginwiki or metawiki might be close but its a rolling monthly count [22:41:22] enough to estimate [22:41:38] Anyone who registers on a wiki will automatically register to login, i think [22:41:54] yep [22:43:37] lol, it doesn't work for login [22:43:40] but it does work for meta [22:43:43] close enough I guess [22:44:37] okay, so I'm guessing this is "unique" "users" or an approximation [22:45:16] so meta has about 3:1 users from US vs UK, and about 2:1 for UK vs Canada [22:46:37] and these would be fairly "active" users to stop by on meta, who would be interested in the well-being of Miraheze [22:47:09] jdstroy: Something like 0.5% of miraheze users are 'active' on meta [22:47:46] although, that means "miraheze" accounts so that can be inflated be alternate accounts etc [22:48:00] By "active" do you mean, actively participate in governance/policy? [22:48:26] jdstroy: have made an edit or logged action in the last 30 days [22:48:37] on meta-only, or on all of Miraheze? [22:48:49] jdstroy: on meta only [22:49:09] I have data for login,meta,template,commons and testwikis [22:49:46] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.024 second response time [22:49:51] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [22:49:55] I'm mostly interested in [users who would be interested in Miraheze's well-being on the whole], so that'd be aggregated users across the whole of Miraheze [22:49:57] e.g. login [22:50:05] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.049 second response time [22:50:07] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [22:50:10] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:50:26] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.039 second response time [22:50:37] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [22:50:38] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4210 bytes in 0.031 second response time [22:50:54] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 4 backends are down. lizardfs6 mw1 mw2 mw3 [22:50:55] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4212 bytes in 0.023 second response time [22:51:01] I'm guessing that since Miraheze hosts things like All The Tropes, it probably has a pretty big user base [22:51:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [22:51:12] PROBLEM - misc4 phab.miraheze.wiki HTTPS on misc4 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [22:51:15] PROBLEM - db4 MySQL on db4 is CRITICAL: Can't connect to MySQL server on '81.4.109.166' (115) [22:51:29] PROBLEM - misc4 phabricator.miraheze.org HTTPS on misc4 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4215 bytes in 0.054 second response time [22:51:34] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [22:51:45] jdstroy: ATT was our biggest wiki. I'm meaning to reget that data [22:52:27] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [22:52:36] thought so. [22:53:10] RECOVERY - misc4 phab.miraheze.wiki HTTPS on misc4 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.068 second response time [22:53:15] RECOVERY - db4 MySQL on db4 is OK: Uptime: 285 Threads: 19 Questions: 1212 Slow queries: 64 Opens: 127 Flush tables: 1 Open tables: 121 Queries per second avg: 4.252 [22:53:22] would be interesting to see user stats by country for All The Tropes and other large wikis on Miraheze [22:53:31] RECOVERY - misc4 phabricator.miraheze.org HTTPS on misc4 is OK: HTTP OK: HTTP/1.1 200 OK - 19067 bytes in 0.148 second response time [22:53:47] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [22:53:49] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.311 second response time [22:54:07] RECOVERY - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is OK: OK - NGINX Error Rate is 2% [22:54:07] jdstroy: Special:Analytics will have whatever is public and wiki stats on toolforge has a list of our largest wikis [22:54:08] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 18676 bytes in 0.344 second response time [22:54:09] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [22:54:22] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 18675 bytes in 0.270 second response time [22:54:24] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 8% [22:54:29] Digging right now ;) [22:54:29] paladox|UKInEU: [22:54:36] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 18676 bytes in 0.868 second response time [22:54:37] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:54:37] Voidwalker: check sre discord [22:54:40] yup, aware, recovered. [22:54:50] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [22:54:56] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 18676 bytes in 0.460 second response time [22:55:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:55:05] yeah holy moly that's a lot [22:55:20] 10:1 for US vs UK; 20816 to 2528 [22:55:28] almost 10:1 [22:55:34] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 11% [22:55:56] 8:1 [23:10:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvsvk [23:10:50] [02miraheze/mw-config] 07paladox 03df943aa - Move uncyclomirrorwiki to db5 [23:10:52] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:10:54] [02mw-config] 07paladox opened pull request 03#2869: Move uncyclomirrorwiki to db5 - 13https://git.io/JvsvL [23:17:28] [02mw-config] 07paladox closed pull request 03#2869: Move uncyclomirrorwiki to db5 - 13https://git.io/JvsvL [23:17:30] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvsvE [23:17:31] [02miraheze/mw-config] 07paladox 03d3e4493 - Move uncyclomirrorwiki to db5 (#2869) [23:17:33] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [23:17:34] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [23:19:20] !log drop database uncyclomirrorwiki on db4 as migrated to db5 [23:19:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master