[00:00:04] PROBLEM - cp7 Stunnel Http for mw7 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw7' not defined [00:00:24] PROBLEM - cp7 Puppet on cp7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[varnish] [00:00:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2R9 [00:00:29] [02miraheze/puppet] 07paladox 03a7b716e - jobrunner1 -> jobrunner3 [00:00:31] PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 4 backends are down. mw8 mw9 mw10 mw11 [00:00:31] PROBLEM - cp7 Stunnel Http for mon1 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mon1' not defined [00:00:44] [02mw-config] 07Universal-Omega closed pull request 03#3696: Move - 13https://git.io/JtRFS [00:00:47] [02miraheze/mw-config] 07Universal-Omega deleted branch 03paladox-patch-2 [00:00:48] [02mw-config] 07Universal-Omega deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:00:49] PROBLEM - cp7 Stunnel Http for mw4 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw4' not defined [00:01:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2RN [00:01:13] [02miraheze/dns] 07paladox 03300e336 - Move backends for grafana and icinga to mon2 [00:01:28] PROBLEM - cp7 Stunnel Http for mw6 on cp7 is UNKNOWN: NRPE: Command 'check_stunnel_mw6' not defined [00:03:57] [02puppet] 07Universal-Omega opened pull request 03#1631: mw6 -> mw10 - 13https://git.io/Jt20I [00:04:13] RECOVERY - test3 MediaWiki Rendering on test3 is OK: HTTP OK: HTTP/1.1 200 OK - 20679 bytes in 0.382 second response time [00:04:47] [02puppet] 07paladox closed pull request 03#1631: mw6 -> mw10 - 13https://git.io/Jt20I [00:04:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20Y [00:04:50] [02miraheze/puppet] 07Universal-Omega 03832c005 - mw6 -> mw10 (#1631) [00:05:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20C [00:05:39] [02miraheze/puppet] 07paladox 03e5f081c - Update groups.conf [00:05:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20B [00:05:58] [02miraheze/puppet] 07paladox 0315b9337 - mon1 -> mon2 [00:06:23] @paladox MacFan4000 reports in #miraheze-bots that MirahezeBots-run RC feeds (the one run by SRE) is down, and thus the wiki RC feeds are not updating [00:06:30] any idea as to the cause? [00:07:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20z [00:07:11] [02miraheze/puppet] 07paladox 03ef2d1a4 - Switch cp6/7 ip to cp10/11 in nginx [00:07:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20a [00:07:36] [02miraheze/puppet] 07paladox 033bb3f71 - cp[67] -> cp(10|11) [00:08:50] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20i [00:08:52] [02miraheze/mw-config] 07Universal-Omega 037daf0de - add graylog to createwiki blacklist [00:10:03] miraheze/mw-config - Universal-Omega the build passed. [00:14:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt20H [00:14:45] [02miraheze/puppet] 07paladox 03f07c1e9 - graylog1 -> graylog2 [00:21:15] is there a history of this channel? because I think someone alredy reply me a lot of time before. I can't upload a ogg [00:21:40] is a file that I recorder myself. When I upload the message says that is a ogx but is an ogg [00:21:48] *recorded [00:22:05] I am exporting it from audacity [00:23:09] @Jakeukalane Yep, see http://wm-bot.wmflabs.org/browser/index.php?display=%23miraheze [00:23:11] [ Wikimedia IRC logs browser ] - wm-bot.wmflabs.org [00:23:18] This channel is publicly logged there [00:24:09] Hi dmehus [00:27:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2EY [00:27:36] [02miraheze/puppet] 07paladox 0399d2d69 - Update nrpe.cfg.erb [00:31:03] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2ER [00:31:04] [02miraheze/dns] 07paladox 03b61abca - Update miraheze.org [00:32:15] the error is still there. I don't know. but I think this happened us before. how can I do to obtainan a ogg-compatible file? I tried audacity and online conversos but it keeps saying that the file is wrong [00:32:42] specifically it says: El tipo de archivo ".ogx" no está permitido. Los tipos de archivos permitidos son apng, djvu, gif, ico, jpeg, jpg, ogg, pdf, png, svg, webp, ogv, oga, flac, opus, wav, webm, mp3, midi, mid, mpg, mpeg. [00:33:16] (The type of files: "ogx" is not allowed. The allowed types are: apng..... ogg ...) [00:33:33] ```Error 503 Backend fetch failed, forwarded for , 127.0.0.1 [00:33:33] (Varnish XID 7669969) via cp12.miraheze.org at Sat, 06 Feb 2021 00:33:08 GMT.``` [00:34:06] I am sory if I am being annoying [00:34:29] @BurningPrincess1 No problem. If regarding your DMs, I'll get to them shortly. [00:34:37] Thank you [00:34:54] @Jakeukalane, I'm not sure...Audacity is what I'd probably use [00:36:30] I wonder if he needs to convert it to an ogg [00:56:36] O, wait, hr is trying [01:08:56] !log rename test2 to test3 [01:09:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:30] Hey would it be best to have a namespace for each era or just each setting? [01:28:48] @BurningPrincess1 Hrm, it's hard to say. I would probably go by era, tbh, and then categorize pages by setting [01:29:02] but setting could also work as well [01:29:36] Thnaks for the input [01:31:27] PROBLEM - jobrunner3 Puppet on jobrunner3 is WARNING: WARNING: Puppet is currently disabled, message: paldox, last run 27 minutes ago with 0 failures [01:35:27] RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:44:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt2z8 [01:44:55] [02miraheze/mw-config] 07Universal-Omega 037debd72 - change ip addresses for cp[67] to cp[1011] [01:44:56] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:45:03] [02mw-config] 07Universal-Omega opened pull request 03#3715: change ip addresses for cp[67] to cp[1011] - 13https://git.io/Jt2zB [01:46:10] miraheze/mw-config - Universal-Omega the build passed. [01:47:29] [02mw-config] 07paladox closed pull request 03#3715: change ip addresses for cp[67] to cp[1011] - 13https://git.io/Jt2zB [01:47:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2zz [01:47:32] [02miraheze/mw-config] 07Universal-Omega 03b4115b0 - change ip addresses for cp[67] to cp[1011] (#3715) [01:47:34] [02mw-config] 07paladox deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:47:35] [02miraheze/mw-config] 07paladox deleted branch 03Universal-Omega-patch-2 [01:48:30] miraheze/mw-config - paladox the build passed. [01:52:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt2zy [01:52:42] [02miraheze/mw-config] 07Universal-Omega 03c5986a8 - ldap1 -> ldap2 [01:52:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [01:52:45] [02mw-config] 07Universal-Omega opened pull request 03#3716: ldap1 -> ldap2 - 13https://git.io/Jt2zS [01:53:44] miraheze/mw-config - Universal-Omega the build passed. [02:02:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2zj [02:02:21] [02miraheze/mw-config] 07Universal-Omega 033be6228 - test2 -> test3 [02:03:15] miraheze/mw-config - Universal-Omega the build passed. [02:03:36] [02ssl] 07Universal-Omega opened pull request 03#388: change test1 redirect to test3; add test2 redirect to test3 - 13https://git.io/Jt2gk [02:05:45] PROBLEM - mon2 Puppet on mon2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 15 minutes ago with 0 failures [02:05:50] [02mw-config] 07paladox closed pull request 03#3716: ldap1 -> ldap2 - 13https://git.io/Jt2zS [02:05:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2gC [02:05:53] [02miraheze/mw-config] 07Universal-Omega 03653e854 - ldap1 -> ldap2 (#3716) [02:06:49] miraheze/mw-config - paladox the build passed. [02:07:40] [02ssl] 07paladox closed pull request 03#388: change test1 redirect to test3; add test2 redirect to test3 - 13https://git.io/Jt2gk [02:07:41] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2g4 [02:07:43] [02miraheze/ssl] 07Universal-Omega 034722265 - change test1 redirect to test3; add test2 redirect to test3 (#388) [02:09:45] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:10:55] PROBLEM - test2.miraheze.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - test2.miraheze.org reverse DNS resolves to test2.miraheze.org [02:16:25] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2go [02:16:27] [02miraheze/dns] 07paladox 036692910 - services update [02:16:46] [02puppet] 07Universal-Omega opened pull request 03#1632: bacula: use new infrastructure - 13https://git.io/Jt2gK [02:17:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/Jt2g1 [02:17:41] [02miraheze/puppet] 07paladox 0358f1113 - Switch over bacula2 to puppet3 [02:18:16] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [02:18:17] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [02:19:16] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:20:58] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.003318041563 secs [02:21:32] PROBLEM - bacula2 Disk Space on bacula2 is CRITICAL: DISK CRITICAL - free space: / 367 MB (0% inode=99%); [02:21:52] PROBLEM - bacula2 Bacula Databases db12 on bacula2 is WARNING: WARNING: Diff, 14584 files, 33.92GB, 2021-01-17 01:56:00 (2.9 weeks ago) [02:21:55] PROBLEM - bacula2 Bacula Static 2 on bacula2 is CRITICAL: CRITICAL: no terminated jobs [02:21:58] PROBLEM - bacula2 Bacula Databases db11 on bacula2 is WARNING: WARNING: Diff, 39329 files, 32.19GB, 2021-01-17 01:00:00 (2.9 weeks ago) [02:22:00] PROBLEM - bacula2 Bacula Static 1 on bacula2 is WARNING: WARNING: Full, 12804977 files, 596.3GB, 2021-01-19 11:44:00 (2.5 weeks ago) [02:22:29] PROBLEM - bacula2 Bacula Databases db13 on bacula2 is WARNING: WARNING: Diff, 41418 files, 68.68GB, 2021-01-17 03:12:00 (2.9 weeks ago) [02:23:47] PROBLEM - bacula2 Bacula Phabricator Static on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:28:51] [02puppet] 07Universal-Omega opened pull request 03#1633: puppet2 -> puppet3 - 13https://git.io/Jt22m [02:32:35] [02puppet] 07paladox closed pull request 03#1633: puppet2 -> puppet3 - 13https://git.io/Jt22m [02:32:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/Jt22n [02:32:38] [02miraheze/puppet] 07Universal-Omega 0344d364f - puppet2 -> puppet3 (#1633) * puppet2 -> puppet3 * Update puppetserver.pp * Update client.pp * Update puppet.pp * Update data.yaml [02:36:02] [02puppet] 07Universal-Omega synchronize pull request 03#1632: bacula: use new infrastructure - 13https://git.io/Jt2gK [02:38:38] [02puppet] 07paladox closed pull request 03#1632: bacula: use new infrastructure - 13https://git.io/Jt2gK [02:38:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jt22R [02:38:41] [02miraheze/puppet] 07Universal-Omega 03d192a18 - bacula: use new infrastructure (#1632) * bacula: use new infrastructure * Update nrpe.cfg.erb [02:40:44] PROBLEM - bacula2 Bacula Static 1 on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:41:23] PROBLEM - bacula2 Bacula Private Git on bacula2 is CRITICAL: CRITICAL: no terminated jobs [02:43:07] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:43:44] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:04] RECOVERY - cp3 APT on cp3 is OK: APT OK: 3 packages available for upgrade (0 critical updates). [02:45:24] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [02:45:40] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [02:46:20] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 89% [02:47:13] !log restarted varnish on cp3 as it appears to have gone down [02:47:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:22] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [02:47:40] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:48:17] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 6% [02:51:32] PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 0.46, 1.89, 1.81 [02:53:28] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.54, 1.43, 1.65 [02:55:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt22M [02:55:22] [02miraheze/puppet] 07paladox 033dd1b67 - Update site.pp [03:02:17] Still 503 on `testwiki` [03:02:17] ```Error 503 Backend fetch failed, forwarded for , 127.0.0.1 [03:02:17] (Varnish XID 67693) via cp7 at Sat, 06 Feb 2021 03:01:36 GMT.``` [03:03:32] hyg [03:03:33] huh [03:05:03] proxying through test3 works [03:05:09] but using prod doesn't.. [03:05:41] hrm, not sure, this is publictestwiki.com though [03:06:26] paladox, created as a task...https://phabricator.miraheze.org/T6825 [03:06:27] [ ⚓ T6825 Persistent 503 Backend Fetch errors on `testwiki` ] - phabricator.miraheze.org [03:10:40] paladox, same probably on https://www.thesimswiki.com (@k6ka's wiki) as on publictestwiki.com (co-founded by @Reception123 and @NDKilla). Seems to be a custom domain issue. Have we updated the zone files for all our hosted domains? [03:10:41] [ 503 Backend fetch failed ] - www.thesimswiki.com [03:10:53] oh [03:10:56] is this a MW bug [03:11:04] well couldn't possibly be [03:11:05] well no [03:11:19] it's a Varnish/database/backend issue, I think [03:11:39] wondering if the custom domain zone files A records are pointing to the old cp7 [03:11:45] and need to be updated [03:12:59] found it i think [03:13:04] its definitely not a varnish issue [03:13:11] it works proxying via test3 [03:13:12] ah, ok [03:13:17] oh [03:13:18] !log rebuild lc on mw* [03:13:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:25] what do you mean by proxying test3? [03:14:09] i proxy a domain via test3 [03:14:40] rather than it randomly give me one of the mw [03:14:41] *mw's [03:15:02] paladox, I think I know the problem maybe [03:15:16] hmm? [03:15:34] publictestwiki.com and thesimswiki.com are hosting their DNS not with us, so they likely need to update the A records for their domains to our new IP address [03:15:49] so if this is the case, we'll definitely need to do a mass email [03:16:34] ```id 64602 [03:16:34] opcode QUERY [03:16:34] rcode NOERROR [03:16:34] flags QR RD RA [03:16:34] ;QUESTION [03:16:34] publictestwiki.com. IN A [03:16:34] ;ANSWER [03:16:35] publictestwiki.com. 59 IN CNAME cp7.miraheze.org. [03:16:35] cp7.miraheze.org. 299 IN A 51.89.160.142 [03:16:36] ;AUTHORITY [03:16:36] ;ADDITIONAL``` [03:16:45] ugh this is going to be ugly [03:16:56] it's pointed to cp7.miraheze.org, not mw-lb.miraheze.org [03:17:16] ```id 28034 [03:17:16] opcode QUERY [03:17:16] rcode NOERROR [03:17:16] flags QR RD RA [03:17:16] ;QUESTION [03:17:17] thesimswiki.com. IN A [03:17:17] ;ANSWER [03:17:18] thesimswiki.com. 119 IN A 51.89.160.142 [03:17:18] ;AUTHORITY [03:17:19] ;ADDITIONAL``` [03:17:23] paladox ^ [03:17:55] oh [03:18:03] so thats why [03:18:10] I think so [03:18:17] Voidwalker Reception123 @NDKilla ^ [03:18:25] ```id 56491 [03:18:25] opcode QUERY [03:18:25] rcode NOERROR [03:18:25] flags QR RD RA [03:18:25] ;QUESTION [03:18:25] revi.wiki. IN A [03:18:25] ;ANSWER [03:18:26] revi.wiki. 149 IN A 51.222.25.132 [03:18:26] ;AUTHORITY [03:18:27] ;ADDITIONAL``` [03:18:27] revi.wiki seems to be correct, if that's the correct IP [03:18:42] waiting for @Doug to get anti-spam'd [03:18:52] lol @NDKilla :P [03:19:07] * Voidwalker recommends a pastebin [03:19:08] Voidwalker: 2021-02-04 - 16:38:38UTC tell Voidwalker Can you see http://wm-bot.wmflabs.org/browser/index.php?display=%23miraheze at 11:38 AM your time for BurningPrincess1's question? [03:19:08] Voidwalker: 2021-02-04 - 16:45:29UTC tell Voidwalker Disregard previous tell. I think I solved it. :D [03:19:08] Voidwalker: 2021-02-05 - 19:18:41UTC tell Voidwalker RhinosF1 said to say to check #miraheze-bots and that you need 3 SSH apps + Icinga [03:19:31] aaaaaaaaaa [03:19:36] * dmehus good point Voidwalker [03:19:38] If you need The Sims Wiki to do something, please alert MrC as he's the one in charge of our DNS stuff. Best. [03:19:50] ah, thanks, k6ka [03:19:50] ahh [03:20:01] MrC, ^ [03:23:57] paladox, going through my CA for all wikis with custom domains to see if there's any other misconfigured ones [03:24:18] works for me [03:31:41] paladox, yeah, I know, most work for me so far but I want to see if there's any other misconfigured custom domains pointing to the old servers whose bureaucrats we'll need to nodify [03:31:47] s/nodify/notify [03:31:47] dmehus meant to say: paladox, yeah, I know, most work for me so far but I want to see if there's any other misconfigured custom domains pointing to the old servers whose bureaucrats we'll need to notify [03:34:29] https://www.thesimswiki.com/wiki/ is back [03:34:31] [ The Sims Wiki ] - www.thesimswiki.com [03:35:46] Is there a way to make a note somewhere that if it's about DNS in particlar, to skip k6ka and ask me directly? [03:44:54] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:46:49] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.002552181482 secs [06:26:20] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on Jobrunner[34], mw[891011], and test3 [06:26:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:11] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt26o [06:59:13] [02miraheze/puppet] 07Reception123 03243895a - jobrunner1->jobrunner3 for ssl renew [08:21:29] [02puppet] 07RhinosF1 opened pull request 03#1634: users: replace my key - 13https://git.io/Jt2P3 [08:30:01] [02mediawiki] 07Universal-Omega closed pull request 03#1120: Bump extensions/SnapProjectEmbed from `c26b3fc` to `bde7c6c` - 13https://git.io/JtRUD [08:30:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/Jt2PK [08:30:04] [02miraheze/mediawiki] 07dependabot[bot] 031505a3e - Bump extensions/SnapProjectEmbed from `c26b3fc` to `bde7c6c` (#1120) Bumps [extensions/SnapProjectEmbed](https://github.com/snapwiki/SnapProjectEmbed) from `c26b3fc` to `bde7c6c`. - [Release notes](https://github.com/snapwiki/SnapProjectEmbed/releases) - [08:30:04] [Commits](https://github.com/snapwiki/SnapProjectEmbed/compare/c26b3fc3e0ed08e624551284184a5468d4146e98...bde7c6c3b50d759a8dcaeff513c22e474ecd382e) Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> [08:30:05] [ GitHub - snapwiki/SnapProjectEmbed: MediaWiki extension for embedding Snap! projects ] - github.com [08:30:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-bde7c6c [08:30:05] [ Releases · snapwiki/SnapProjectEmbed · GitHub ] - github.com [08:30:06] [ Comparing c26b3fc3e0ed08e624551284184a5468d4146e98...bde7c6c3b50d759a8dcaeff513c22e474ecd382e · snapwiki/SnapProjectEmbed · GitHub ] - github.com [08:30:07] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_35/extensions/SnapProjectEmbed-bde7c6c - 13https://git.io/vbL5b [10:09:31] hello again. my problem is that I tried to upload an ogg and the system says it is an ogx [11:21:54] Okay, so I updated my paste to have a shorter name for the namespace [11:22:50] https://pastebin.com/LLB5cThm - so, is this corret or do I need to change it [11:22:51] [ Pages to move - Pastebin.com ] - pastebin.com [12:40:05] Jakeukalane: can you give me the full error? [12:41:29] the system thinks that the archive is an opus file ogx when I exported into a ogg [12:41:36] will give you the literal error in a moment [12:43:14] worked now (!!?) [12:44:03] it works now [12:44:24] did you did something? because yesterday I tried the same file and say to me that error. [12:45:15] The error it was "El tipo de archivo ".ogx" no está permitido. Los tipos de archivos permitidos son apng, djvu, gif, ico, jpeg, jpg, ogg, pdf, png, svg, webp, ogv, oga, flac, opus, wav, webm, mp3, midi, mid, mpg, mpeg." [12:45:39] The type of file ".ogx" it is not allowed. The allowed types of files are "..." [12:47:13] Jakeukalane: no I didn't do anything. [12:47:15] I [12:47:35] Didn't know whether you meant Mime type or actual extension [12:47:47] Actual extension litterally checks the end of the file [12:47:53] Name [12:48:05] @Site Reliability Engineers Meta is unusually slow and the jobrunner is probably sick. [12:48:07] The file was ogg and it mimetype it should be ogg too [12:48:21] @R4356th: looking [12:48:30] Loads fine for me [12:48:41] Hmm... [12:48:46] @R4356th: can you check with cp you are going through? [12:48:48] Discord is working fine? [12:48:54] Okay. [12:49:01] But Jobrunner can't be the CP. [12:49:11] Wikis are taking forever to be created. [12:49:15] checking jobrunner [12:49:28] Meta just got slow for me [12:50:06] CP3 and MW8 [12:50:30] The relay-bot for the cvt-feed isn't functioning properly since it got fixed. [12:52:14] Yeah we know about the relay [12:52:55] Oh, I apologize for that unneeded reminder then. [12:53:52] 0 jobs on meta [12:54:14] Because the wiki that was waiting to be created got created three minutes ago. [14:16:58] * hispano76 greet [14:23:35] [02puppet] 07paladox closed pull request 03#1634: users: replace my key - 13https://git.io/Jt2P3 [14:23:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2N4 [14:23:38] [02miraheze/puppet] 07RhinosF1 0335733fa - users: replace my key (#1634) [14:27:44] !log depool mw8 [14:27:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:31] !log rebuild lc on mw8 [14:28:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:30:41] PROBLEM - cp11 Varnish Backends on cp11 is CRITICAL: 1 backends are down. mw8 [14:30:47] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw8 [14:31:19] PROBLEM - cp10 Varnish Backends on cp10 is CRITICAL: 1 backends are down. mw8 [14:31:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw8 [14:35:08] !log repool mw8 and depool mw9 [14:35:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:12] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.05, 6.07, 4.05 [14:38:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.86, 6.24, 4.36 [14:38:44] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.86, 6.20, 4.32 [14:42:31] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki configuration Error - 1297 bytes in 0.006 second response time [14:44:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.79, 6.57, 5.19 [14:44:32] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 1258 bytes in 0.008 second response time [14:44:42] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.26, 6.32, 5.04 [14:50:32] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 771 bytes in 0.075 second response time [14:52:32] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20681 bytes in 0.252 second response time [14:57:10] !log repool mw9 [14:57:11] !log depool mw10 [14:57:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:57:19] RECOVERY - cp10 Varnish Backends on cp10 is OK: All 7 backends are healthy [14:57:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [14:58:43] RECOVERY - cp11 Varnish Backends on cp11 is OK: All 7 backends are healthy [14:58:47] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 7 backends are healthy [15:00:05] !log depool 11 and repool 10 [15:00:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:13] err mw11 and 10 [15:00:33] !log repool mw11 [15:00:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:38] !log reboot jobrunner[12] [15:00:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:00:49] !log actually rebooted jobrunner[34] [15:00:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:05:46] PROBLEM - jobrunner4 Check Gluster Clients on jobrunner4 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [15:07:46] RECOVERY - jobrunner4 Check Gluster Clients on jobrunner4 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [15:08:59] Hi [15:09:33] I have my file ready for pages to move: https://pastebin.com/LLB5cThm [15:09:34] [ Pages to move - Pastebin.com ] - pastebin.com [15:19:28] [02mediawiki] 07paladox closed pull request 03#1099: Bump extensions/PageForms from `cae2fb1` to `cf0fcd9` - 13https://git.io/JtRUv [15:19:29] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_35 [+0/-0/±2] 13https://git.io/Jt2Aa [15:19:31] [02miraheze/mediawiki] 07paladox 0342cb50c - Merge pull request #1099 from miraheze/dependabot/submodules/REL1_35/extensions/PageForms-cf0fcd9 Bump extensions/PageForms from `cae2fb1` to `cf0fcd9` [15:19:32] [02mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-cf0fcd9 - 13https://git.io/vbL5b [15:19:34] [02miraheze/mediawiki] 07paladox deleted branch 03dependabot/submodules/REL1_35/extensions/PageForms-cf0fcd9 [15:50:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/Jt2xL [15:50:53] [02miraheze/mw-config] 07Universal-Omega 039712dd5 - Change IP addresses for services[12] to services[34] [15:50:55] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [15:50:56] [02mw-config] 07Universal-Omega opened pull request 03#3717: Change IP addresses for services[12] to services[34] - 13https://git.io/Jt2xt [15:51:46] [02mw-config] 07paladox closed pull request 03#3717: Change IP addresses for services[12] to services[34] - 13https://git.io/Jt2xt [15:51:47] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2xm [15:51:49] [02miraheze/mw-config] 07Universal-Omega 039ad75d2 - Change IP addresses for services[12] to services[34] (#3717) [15:51:58] miraheze/mw-config - Universal-Omega the build passed. [15:52:19] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/vbvb3 [15:52:21] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [15:52:43] miraheze/mw-config - paladox the build passed. [16:17:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jt2xH [16:17:46] [02miraheze/puppet] 07paladox 03ad5d924 - Update init.pp [16:40:34] [02puppet] 07RhinosF1 opened pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2pu [16:41:43] JohnLewis, SPF|Cloud: please see ^ as paladox points out automating it would be hard [16:43:45] cp[3|10|11|12] is not valid [16:43:52] I presume you mean cp(3|10|11|12) [16:44:16] SPF|Cloud: changing [16:44:34] remember this is a reverse dns check though, could it point to a non-existent cp? [16:44:59] [02puppet] 07RhinosF1 synchronize pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2pu [16:45:46] SPF|Cloud: that's a fair point. I then massively don't understand why the check didn't go off for last night [16:45:54] what happened? [16:46:34] the purpose of this check is to find domain names that have incorrect DNS settings, ie cloudflare [16:46:38] SPF|Cloud: https://phabricator.miraheze.org/T6825#134450 [16:46:39] [ ⚓ T6825 Persistent 503 Backend Fetch errors on `testwiki` ] - phabricator.miraheze.org [16:46:51] [02puppet] 07Universal-Omega commented on pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2po [16:46:54] I want to try and make sure no other wikis are pointing at CPs that don't exist [16:47:21] as long as the reverse dns check results in anything that starts with 'cp' and ends with '.miraheze.org', the check is OK [16:47:45] The issue was the CNAME from publictestwiki was pointing to cp7 instead of mw-lb [16:47:46] [02puppet] 07RhinosF1 commented on pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2pi [16:48:04] SPF|Cloud: is there anyway we can catch that? [16:48:37] it is possible, but do we want to check it? [16:49:10] RECOVERY - gluster3 Gluster Disk Space on gluster3 is OK: DISK OK - free space: /mnt/mediawiki-static 834465 MB (43% inode=86%); [16:49:15] SPF|Cloud: if it stops an outage of some wikis then it's a benefit to me [16:50:11] even if the DNS entry was incorrect, the moment miraheze decoms cp7 and removes 1) the DNS entry for cp7 in miraheze.org 2) the reverse DNS for the vailover IP, a CNAME to cp7 would match the wildcard record in the miraheze.org file [16:51:16] SPF|Cloud: so it was just too early to go off as the dns hadn't been removed for cp7 then? [16:51:21] I made a ticket for my page move: https://phabricator.miraheze.org/T6827 [16:51:22] [ ⚓ T6827 Move pages using moveBatch.php maintenance script ] - phabricator.miraheze.org [16:51:40] the only situation your proposed change will catch is a situation where we have shifted production traffic to new cache proxies, but still have the old cache proxy in the DNS config (and reverse DNS) [16:52:00] SPF|Cloud: that's a fair point [16:52:13] but your change adds one more action for us to execute [16:52:23] Yeah that's probably rare [16:52:39] I assume we could just wait until the dns goes for the old CPs [16:52:48] the question is whether the risk (likelihood of being in this situation + the impact of a wrong DNS record) is worth the extra debt [16:53:19] Probably not because I'm gonna assume them old dns records will be gone in less than 24 hours [16:53:35] correct [16:53:51] and setting the right DNS records is the responsibility of the domain owner, not us [16:54:02] dmehus: there's your answer to having to check for others like testwiki ^ [16:54:07] Icinga will start doing so soon [16:54:13] SPF|Cloud: I'll close [16:54:45] [02puppet] 07RhinosF1 commented on pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2pd [16:54:47] [02puppet] 07RhinosF1 closed pull request 03#1635: rdns-check: change regex to not match nonexistent cp* - 13https://git.io/Jt2pu [16:55:08] we are responsible for depooling cache proxies (as returned by the mw-lb.miraheze.org returned) and waiting up to five minutes (TTL) so caches can be flushed [16:56:06] Yeah [16:56:21] People shouldn't be pointing directly at a single cp [16:56:38] And that's our sysadmin's fault in testwiki's case for doing it [16:57:08] I wonder why that doesn't use our dns tbh [16:58:41] Have I done something wrong? [17:01:22]