[00:05:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.83, 6.04, 4.64 [00:07:25] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.87, 5.51, 4.61 [00:09:22] Yeah, bacula does need some proper love but I’m leaving it until db backups are off it [00:21:05] PROBLEM - ping4 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 334.02 ms [00:27:22] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+9/-0/±3] 13https://git.io/JtDJp [00:27:23] [02miraheze/puppet] 07paladox 03756d6c9 - Introduce memcached role [00:27:25] [02miraheze/puppet] 07paladox 0313aff51 - Merge branch 'master' of github.com:miraheze/puppet [00:28:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JtDUe [00:28:59] [02miraheze/puppet] 07paladox 036893812 - Install memcached onto mem2 [00:29:01] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:29:02] [02puppet] 07paladox opened pull request 03#1648: Install memcached onto mem2 - 13https://git.io/JtDUv [00:30:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://git.io/JtDUI [00:30:23] [02miraheze/puppet] 07paladox 03b6e8f2b - Create mem2.yaml [00:30:23] [02puppet] 07paladox synchronize pull request 03#1648: Install memcached onto mem2 - 13https://git.io/JtDUv [00:31:06] [02puppet] 07paladox commented on commit 03756d6c9be27999299f933dbe1c653111383d3302 - 13https://git.io/JtDUL [00:31:09] RECOVERY - ping4 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 248.32 ms [00:31:45] [02puppet] 07paladox closed pull request 03#1648: Install memcached onto mem2 - 13https://git.io/JtDUv [00:31:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/JtDUm [00:31:48] [02miraheze/puppet] 07paladox 03f69c565 - Install memcached onto mem2 (#1648) [00:31:49] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:31:51] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:34:54] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:41:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDUa [00:41:56] [02miraheze/puppet] 07paladox 03929a83d - Update memcached.pp [00:48:57] PROBLEM - mem2 Disk Space on mem2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:31] PROBLEM - mem2 SSH on mem2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:40] PROBLEM - mem2 Current Load on mem2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:43] PROBLEM - ping6 on mem2 is CRITICAL: CRITICAL - Destination Unreachable (2001:41d0:800:1bbd::12) [00:49:54] PROBLEM - mem2 NTP time on mem2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:49:57] PROBLEM - ping4 on mem2 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:50:07] PROBLEM - Host mem2 is DOWN: PING CRITICAL - Packet loss = 100% [01:02:12] me ^ [01:22:10] PROBLEM - mem2 Puppet on mem2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [01:24:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDk4 [01:24:55] [02miraheze/puppet] 07paladox 03d945e71 - Fix [01:25:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDkR [01:25:48] [02miraheze/puppet] 07paladox 03c335171 - Fix [01:27:55] RECOVERY - mem2 Puppet on mem2 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [01:29:02] JohnLewis, not sure if I told you or not, but a few global renames ago, I noticed a bug whereby Flow (StructuredDiscussions) user talk pages weren't moved when renaming a user. Did some digging, and apparently there's is/was an upstream task about it. It was an easy fix; the `steward` global group just needed `flow-create-board` added to it. Just checked a few wikis where I knew Flow was the default content model in user talk namespace and [01:29:03] noticed that the Flow user talk page was correctly moved to the new user talk page title (https://abysmalrobloxgames.miraheze.org/wiki/Special:Log?type=&user=&page=User+talk:FireBarrier101&wpdate=&tagfilter=&subtype=) [01:29:04] [ Permission error - Abysmal Roblox Games Wiki ] - abysmalrobloxgames.miraheze.org [01:29:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDk6 [01:29:45] [02miraheze/puppet] 07paladox 033d2cead - mem2: Set role::memcached::size to 7000 (7gb) [01:42:32] If I can remember correctly, wasn't that wiki already closed and moved to the garbagerobloxgameswiki? [01:43:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDIT [01:43:30] [02miraheze/puppet] 07paladox 03daf6b5f - Fix [01:43:56] @DarkMatterMan4500, no clue. I thought it showed as open, but it may have been closed. [01:44:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDIt [01:44:31] [02miraheze/puppet] 07paladox 03bcb9786 - Update memcached.pp [01:47:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDIO [01:47:25] [02miraheze/mw-config] 07paladox 03c103133 - Configure memcache but don't set cache to use it yet [01:47:26] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:47:28] [02mw-config] 07paladox opened pull request 03#3726: Configure memcache but don't set cache to use it yet - 13https://git.io/JtDI3 [01:47:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDIZ [01:47:43] [02miraheze/mw-config] 07paladox 032f15212 - Update Redis.php [01:47:44] [02mw-config] 07paladox synchronize pull request 03#3726: Configure memcache but don't set cache to use it yet - 13https://git.io/JtDI3 [01:47:52] [02mw-config] 07paladox closed pull request 03#3726: Configure memcache but don't set cache to use it yet - 13https://git.io/JtDI3 [01:47:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDIc [01:47:55] [02miraheze/mw-config] 07paladox 0389b7a9e - Configure memcache but don't set cache to use it yet (#3726) [01:47:56] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [01:47:58] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [01:48:30] miraheze/mw-config - paladox the build passed. [01:48:45] miraheze/mw-config - paladox the build passed. [01:48:51] miraheze/mw-config - paladox the build passed. [01:49:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDI8 [01:49:35] [02miraheze/mw-config] 07paladox 0330bd7a9 - Set main and session cache to use memcache [01:50:35] miraheze/mw-config - paladox the build passed. [01:53:55] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 8.80, 4.72, 2.10 [01:55:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDIo [01:55:18] [02miraheze/puppet] 07paladox 03721431d - Update memcached_exporter.pp [02:05:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.84, 7.04, 5.34 [02:05:34] @paladox, I'm wondering if this could be related to your memcached changes...I'm having problems deleting pages on Loginwiki. It keeps asking me to re-confirm the deletion, over and over, like it's not remembering what I asked it to process [02:06:19] Also, it migrates the deletion drop-down reason to the freeform deletion reason text field, and it never used to do that [02:06:44] If you can save, it doesn't sound like it would be related to the memcache issue [02:07:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.87, 6.22, 5.24 [02:07:08] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [02:07:14] oh [02:07:15] actually [02:07:58] Oh? It takes about five attempts before it finally deletes [02:08:03] dmehus does it work now? [02:08:20] I'd have to try deleting another page. Let me find one [02:09:02] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [02:09:29] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 4.27, 2.18, 1.38 [02:10:38] paladox, yep [02:10:47] First try and no issue with the dropdown box [02:10:50] great! [02:10:57] ok, so i guess you were hitting m10 [02:10:59] *mw10 [02:11:02] which i've fixed [02:11:19] anyways, things feel extreamly fast. [02:11:29] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.19, 1.62, 1.27 [02:11:37] oh, maybe that was it then. Had you been rebooting mw10 when I was trying the deletion? [02:11:52] and yes, mw10 is usually the mw server I hit...mw10 and cp12 [02:12:08] No, the change never made it to it (conflicts) [02:12:14] Yeah, I've noticed a performance boost already it. [02:12:35] ah, makes sense, it takes a few minutes for Puppet to cascade the updates to all the servers I guess? [02:12:39] PROBLEM - graylog2 APT on graylog2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:13:52] and will we still use redis for some things, or will it fully decommissioned soon? [02:14:06] No, there was a git conflict, appears i accidentally changed something in Redis.php and forgot about it [02:14:17] We'll likely decomission unless we need redis [02:14:29] paladox, ah, yeah, that makes sense, thanks :) [02:16:54] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [02:17:24] The abysmalrobloxgameswiki was closed and was moved to the garbagerobloxgameswiki instead. [02:20:07] @DarkMatterMan4500, Oh, the wiki is still open, and because there's still edits / logged actions occurring, it won't be closed and deleted per Dormancy Policy. You can, however, organize a discussion on `abysmalrobloxgameswiki` requesting that the wiki be deleted. Or, just ask a bureaucrat on `abysmalrobloxgameswiki` to close and make private the wiki [02:33:19] PROBLEM - graylog2 Puppet on graylog2 is WARNING: WARNING: Puppet last ran 1 hour ago [02:41:23] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [02:43:02] PROBLEM - guia.cineastas.pt - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - guia.cineastas.pt All nameservers failed to answer the query. [02:45:25] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDqJ [02:45:27] [02miraheze/mw-config] 07paladox 039ac8d05 - Update LocalSettings.php [02:46:23] miraheze/mw-config - paladox the build passed. [02:50:00] RECOVERY - guia.cineastas.pt - reverse DNS on sslhost is OK: rDNS OK - guia.cineastas.pt reverse DNS resolves to cp10.miraheze.org [03:18:00] Reception123, we should try reaching out to the bureaucrats on the wiki ^ and see if they can update their DNS to `mw-lb.miraheze.org` instead of `cp10.miraheze.org` [03:19:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDYG [03:19:38] [02miraheze/puppet] 07paladox 03f929774 - Graylog: Set processbuffer_processors and outputbuffer_processors [03:27:18] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:31:17] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [03:47:03] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 0.13, 0.69, 3.99 [03:51:02] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 0.62, 0.59, 3.18 [04:26:43] dmehus: that likely is not actually cname as cp10, just what cp that wiki is connecting through, through mw-lb. [04:27:23] All of them say resolves to a cp server. [05:02:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±1] 13https://git.io/JtDsU [05:02:43] [02miraheze/puppet] 07paladox 0346b0c45 - php: Add wikidiff2 and remove luasandbox for php7.2 [05:06:27] !log restart php7.3-fpm on mw* and jobrunner* to pick up wikidiff2 [05:06:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:28] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDsD [05:29:30] [02miraheze/mw-config] 07paladox 035b42d34 - Set wgDiff to false if using wikidiff2 [05:29:31] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [05:29:33] [02mw-config] 07paladox opened pull request 03#3727: Set wgDiff to false if using wikidiff2 - 13https://git.io/JtDsy [05:30:36] miraheze/mw-config - paladox the build passed. [05:30:57] [02mw-config] 07paladox closed pull request 03#3727: Set wgDiff to false if using wikidiff2 - 13https://git.io/JtDsy [05:30:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDsH [05:31:00] [02miraheze/mw-config] 07paladox 03849f8f6 - Set wgDiff to false if using wikidiff2 (#3727) [05:31:02] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [05:31:03] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [05:31:56] miraheze/mw-config - paladox the build passed. [05:39:53] Universal_Omega, possibly yes, but possibly no as well. Just something we should look into, I think, as we do want to make sure all wikis eventually are resolving to `mw-lb.miraheze.org` to avoid problems with server migrations in future. [05:40:49] dmehus: Most are. Just that resolves to a cp server in the end, I think. [05:41:31] @Universal_Omega, okay, we can set this aside for the time being. I have something else to report [05:42:26] paladox, can you check if we have the database name `trollpastawiki` on the database servers / in cw_wikis? Reason being is https://meta.miraheze.org/wiki/Special:RequestWikiQueue/16703#mw-section-comments. It says wiki already exists, but if that were true, decline would've been the only option. This happened one time before when we couldn't create a wiki, and you and Reception123 look into it and it turned out the database name wasn't [05:42:26] properly dropped by the wiki deletion script [05:42:27] [ Wiki requests queue - Miraheze Meta ] - meta.miraheze.org [06:53:49] !log re-run wikibackups [06:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:54:41] Reception123, see ^ above (that's what I DMed you about) [06:56:50] The trolllpastawiki db does still exist on db13 [07:02:07] oh [07:02:53] I don't think it's in Special:DeletedWikis as when I checked as I checked Special:ManageWiki/core/trollpastawiki, it didn't exist. It's the usual blank entry with an Arabic language code. If it was in DeletedWikis, I'd be able to undelete it, no? [07:06:53] !log MariaDB [(none)]> DROP DATABASE trollpastawiki; [07:06:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:25] thanks for ^, Reception123 [07:14:31] now I can re-approved that request [07:16:08] no problem :) [08:22:09] !log sudo apt-get upgrade on mon2 [08:22:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:33:55] [02mw-config] 07R4356th opened pull request 03#3728: Add "miraheze" to CW blacklist - 13https://git.io/JtDl4 [08:34:57] miraheze/mw-config - R4356th the build passed. [08:41:26] [02mw-config] 07Reception123 closed pull request 03#3728: Add "miraheze" to CW blacklist - 13https://git.io/JtDl4 [08:41:27] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDlF [08:41:29] [02miraheze/mw-config] 07R4356th 0305c4f38 - Add "miraheze" to CW blacklist (#3728) [08:42:38] miraheze/mw-config - Reception123 the build passed. [11:52:55] Reception123: https://phabricator.miraheze.org/T6864 [11:52:56] [ ⚓ T6864 Request email forward for packages-bots ] - phabricator.miraheze.org [11:52:58] or JohnLewis maybe [14:46:03] SPF|Cloud i switched mw* to memcache last night, and things seems fast. [14:46:17] https://grafana.miraheze.org/d/m3OY7rEGz/prometheus-memcached?orgId=1 [14:46:17] [ Grafana ] - grafana.miraheze.org [14:46:39] do you have data to back it up? [14:47:06] ^ was just about to ask that :P [14:47:53] Statements that drive an exploratory software change need to be data driven more than anything [14:48:08] I mean doug said the same :) [14:48:18] [02:12:13] Yeah, I've noticed a performance boost already it. [14:48:23] if we can prove it by looking at https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?orgId=1&from=now-3d&to=now ... [14:48:23] [ Grafana ] - grafana.miraheze.org [14:48:24] Otherwise, I could say 'let's go back to Redis because I prefer it' and without data, you couldn't argue against my making that decision [14:49:13] i deployed it just at around 2am [14:49:23] utc? [14:49:56] yup [14:50:05] https://git.io/JtDI8 [14:50:07] [ Comparing 89b7a9e0557d...30bd7a918fbe · miraheze/mw-config · GitHub ] - git.io [14:51:34] well [14:51:46] we don't measure response times using an active session ;) [14:52:27] on the other hand, the main cache type has been changed as well, so the blackbox exporter should show changes.. [14:52:44] yeh [14:52:47] which it does [14:52:54] link? [14:52:59] http durations seems stable [14:53:16] https://grafana.miraheze.org/d/xtkCtBkiz/prometheus-blackbox-exporter-test-ferran-tufan?viewPanel=25&orgId=1&from=now-3d&to=now [14:53:17] [ Grafana ] - grafana.miraheze.org [14:53:28] oh wait, 2 AM [14:53:32] i don't see a spike since i've switched to memcache [14:53:34] I thought you said PM [14:53:49] oh [14:54:54] So perhaps, "more stable" is better than "things seem fast" [14:55:07] yeh [14:55:40] the average may not have been decreased by a huge margin, but the higher percentiles have seen changes [14:55:56] is that what you meant to say? [14:58:15] well no, i meant the big spikes in http duration. Though i guess by higher percentiles you mean p95. [14:58:58] p95 or p99, or even p99.99 [15:06:42] paladox: so I can start changing the docs to the new infra, just to make sure, are the specs the same as they were for mw4567 and jbr12 for the new ones? [15:07:05] yes, apart from 4 cores and not 5 (for mw*). Jobrunner still have 3 [15:07:51] paladox: ok, thanks :) [15:08:03] and what about all the other servers? any other changes? [15:09:36] cp10/11 use 2 cores rather than 3, gluster uses 3gb of ram [15:09:59] ok [15:21:57] paladox: is the promox password supposed to be the same for cloud45 as it was before? mine doesn't seem to work [15:22:08] Yes [15:22:10] let me check [15:23:05] i can reset your password [15:24:26] Reception123 [15:24:42] paladox: please do then and you can PM it to me :) [15:25:42] ok [15:29:11] paladox: oh and is there any reason for why cloud1 and 2 are still in DNS? [15:29:32] No, let me remove them! SPF|Cloud you can cancel cloud1/2 [15:29:42] ok [15:30:14] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDXA [15:30:16] [02miraheze/dns] 07paladox 035120acb - Remove cloud[12] from dns [15:52:46] !log reboot test3 so it has 2G of RAM not 1.5G [15:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:56:06] PROBLEM - test3 Check Gluster Clients on test3 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:02:39] Universal_Omega: hmm there's this very strange bug that in RC it shows up when I remove flooder from myself but not when I actually added it [16:03:02] JohnLewis: heh, right in time for me to edit the docs I just created for test3 :P [16:04:51] test2 has 1.5G on the wiki? [16:05:10] I really wonder how no one realised and raised that, because that is extremely odd as to why we would allocate 1/2 a GB [16:07:12] Reception123: what happens when you do that? [16:07:24] JohnLewis: yeah, I copied over the text from test2->test3 and it was 1.5 [16:07:36] https://phabricator.miraheze.org/P386 [16:07:37] [ ✎ P386 Resources Table ] - phabricator.miraheze.org [16:07:50] Universal_Omega: oh nothing but it's weird that the act of adding flood doesn't show in RC but removing it does [16:07:52] it should be the opposite [16:08:02] and I believe @MrJaroslavik raised this with me a few months ago as well [16:08:48] Yup [16:10:02] Reception123: I don't think that is wrong though. I'd think that'd be how it is actually, because when you add it you already have it so it doesn't appear, removing it, you no longer have it so it does not. [16:10:38] hmm, I do remember it being the opposite at some point though [16:18:06] RECOVERY - test3 Check Gluster Clients on test3 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [16:21:17] Reception123: oh. Hmmm... [16:33:09] Reception123 Can i get https://www.mediawiki.org/wiki/Extension:Auto_Create_Category_Pages for my wiki? [16:33:11] [ Extension:Auto Create Category Pages - MediaWiki ] - www.mediawiki.org [16:33:45] stewards do that now, so JohnLewis ^ could you please? (and @MrJaroslavik please give us the URL) [16:34:49] https://mrjaroslavik.miraheze.org/wiki/Special:ManageWiki/extensions [16:34:50] [ Extensions - MrJaroslavik ] - mrjaroslavik.miraheze.org [16:34:51] @MrJaroslavik: make sure you've setup the messages you want in the MediaWiki namespace [16:35:27] Yup, i readed documentation, but not created yet 🙂 [16:36:00] Done [16:36:37] Thank You! [17:09:25] Can you please delete https://mrjaroslavik.miraheze.org/wiki/MediaWiki:Autocreatecategorypages-editor?action=delete yet? It was bad idea to create with own name, i cannot login now 🤔 😄 But on Extension page is not warning about it. (sorry for removal of elements, i cannot remember what URL parameter is for disabling of JS/CSS customizations) [17:09:26] [ Permission error - MrJaroslavik ] - mrjaroslavik.miraheze.org [17:14:08] @MrJaroslavik: safemode=1 to disable js/css [17:14:12] And JohnLewis ^ [17:16:51] Thank you, it is what i wanted And sorry for that nonsencial creation of page [17:17:12] so it is https://mrjaroslavik.miraheze.org/wiki/MediaWiki:Autocreatecategorypages-editor?action=delete&safemode=1 [17:17:13] [ Permission error - MrJaroslavik ] - mrjaroslavik.miraheze.org [17:18:42] Deleted for you [17:20:42] Thank you! I can login now. Will note on mw extension page. [18:14:02] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtD5F [18:14:04] [02miraheze/mw-config] 07paladox 03d4a5f43 - Fix default value for wgUsersNotifiedOnAllChanges [18:14:05] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:14:07] [02mw-config] 07paladox opened pull request 03#3729: Fix default value for wgUsersNotifiedOnAllChanges - 13https://git.io/JtD5N [18:14:16] Universal_Omega ^ [18:14:59] [02mw-config] 07paladox closed pull request 03#3729: Fix default value for wgUsersNotifiedOnAllChanges - 13https://git.io/JtD5N [18:15:01] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtD5j [18:15:02] [02miraheze/mw-config] 07paladox 03c4624a3 - Fix default value for wgUsersNotifiedOnAllChanges (#3729) [18:15:10] miraheze/mw-config - paladox the build passed. [18:16:00] miraheze/mw-config - paladox the build passed. [18:16:02] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:16:03] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [18:17:13] hmm "wgUsersNotifiedOnAllChanges":"MrJaroslavik" [18:17:26] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JtDdW [18:17:27] that doesn't appear to be an array [18:17:27] [02miraheze/ssl] 07Reception123 03ac93c30 - remove wiki.yumeka.team (domain expired/on hold) [18:17:37] [02mw-config] 07Universal-Omega commented on pull request 03#3729: Fix default value for wgUsersNotifiedOnAllChanges - 13https://git.io/JtDd4 [18:17:59] paladox ^ [18:18:16] oh, we could change it so we explode it [18:18:19] Universal_Omega, yeah I noticed that as well (re: the minor RC issue). I think it's probably intended functionality, tbh. [18:19:02] Universal_Omega you wouldn't happen to remember the command you used to rename variables? [18:19:13] paladox: yeah it's the only way. But then the default also must be a string `''` or false as it was before. [18:19:30] ok [18:19:51] paladox: it'll be in SAL/2020 for sure. [18:20:00] ok, thanks! [18:20:07] No problem. [18:20:20] dmehus: yeah, probably. [18:21:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtDdS [18:21:28] [02miraheze/mw-config] 07paladox 03a03172b - Revert "Fix default value for wgUsersNotifiedOnAllChanges (#3729)" [18:21:49] found it [18:21:51] update mw_settings set s_settings = REPLACE(s_settings,'wmgFlaggedRevsLowProfile', 'wgFlaggedRevsLowProfile'); [18:22:25] miraheze/mw-config - paladox the build passed. [18:22:28] paladox, what was the intent behind https://git.io/JtD5j, and did you consider Universal_Omega's comment on that PR? [18:22:29] [ Comparing 05c4f382faf2...c4624a39516c · miraheze/mw-config · GitHub ] - git.io [18:22:45] paladox: any idea how I can figure out why this wiki is timing out? https://icinga.miraheze.org/monitoring/service/show?host=sslhost&service=pol.wiki%20-%20LetsEncrypt [18:22:46] [ Icinga Web 2 Login ] - icinga.miraheze.org [18:22:53] I merged it before UO commented on it :) [18:23:08] i can have a look in a bit [18:23:14] thanks [18:23:18] paladox: yep, great! However, maybe we should fix it to explode it by default through managewiki, as we don't and never will really need those users, wikipages, namespaces to output as strings and should probably output as arrays by default. [18:23:35] paladox, oh ok, np and thanks :) [18:23:44] yeh, if you know the fix in ManageWiki, then i guess we can do it that way Universal_Omega [18:23:51] PROBLEM - cp12 APT on cp12 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (11 critical updates). [18:24:03] PROBLEM - puppet3 APT on puppet3 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [18:24:09] guess introduce a new users-array type [18:24:16] PROBLEM - db13 APT on db13 is CRITICAL: APT CRITICAL: 38 packages available for upgrade (11 critical updates). [18:24:28] PROBLEM - cp10 APT on cp10 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (11 critical updates). [18:24:30] I skimmed the discussion from the wee hours in the morning on the switch to memcache, did anything result from that? [18:24:48] paladox: the same thing happens with wikipages, and namespaces types at least. So not just users. [18:25:00] ok [18:25:15] I had to do the explode hack for the wgWhitelistRead. Which uses wikipages type. [18:25:52] https://github.com/miraheze/mw-config/blob/c4624a39516cffb27e6c5c3aeed8dcfefa95276a/LocalWiki.php#L81 paladox [18:25:52] [ mw-config/LocalWiki.php at c4624a39516cffb27e6c5c3aeed8dcfefa95276a · miraheze/mw-config · GitHub ] - github.com [18:25:54] PROBLEM - gluster4 APT on gluster4 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (11 critical updates). [18:26:06] ah [18:26:08] ok, thanks! [18:26:11] i'll do that for now [18:26:13] Gluster? [18:26:28] paladox: nevermind, it's because the wiki was deleted, no need to look :) [18:26:38] Ah ok [18:27:02] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-1/±1] 13https://git.io/JtDFJ [18:27:03] [02miraheze/ssl] 07Reception123 034b2e8aa - remove pol.wiki Wiki deleted [18:27:05] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (13 critical updates). [18:27:49] PROBLEM - cloud5 APT on cloud5 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (11 critical updates). [18:27:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDFL [18:27:53] [02miraheze/mw-config] 07paladox 031fae318 - Fix setting wgUsersNotifiedOnAllChanges [18:27:55] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:27:56] [02mw-config] 07paladox opened pull request 03#3730: Fix setting wgUsersNotifiedOnAllChanges - 13https://git.io/JtDFt [18:28:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDFO [18:28:28] [02miraheze/mw-config] 07paladox 03f78618f - Update ManageWikiSettings.php [18:28:29] [02mw-config] 07paladox synchronize pull request 03#3730: Fix setting wgUsersNotifiedOnAllChanges - 13https://git.io/JtDFt [18:28:56] Universal_Omega ^ [18:29:00] miraheze/mw-config - paladox the build passed. [18:29:24] paladox: there is wikiverte.pl though. the wiki does exist, and it works with the subdomain but the custom domain keeps loading forever and doesn't work [18:29:30] miraheze/mw-config - paladox the build passed. [18:29:47] root@puppet3:/home/paladox# host wikiverte.pl [18:29:47] wikiverte.pl has address 51.89.160.142 [18:29:47] root@puppet3:/home/paladox# host 51.89.160.142 [18:29:47] 142.160.89.51.in-addr.arpa domain name pointer cp7.miraheze.org. [18:29:54] uses cp7 which is no longer [18:30:12] PROBLEM - mail2 APT on mail2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [18:31:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JtDFu [18:31:35] [02miraheze/mw-config] 07paladox 031b6ac55 - Update LocalSettings.php [18:31:36] paladox: urgh I really don't understand how that happens, because I don't see users explicitly not using mw-lb, so I don't see how that happened for so many [18:31:37] [02mw-config] 07paladox synchronize pull request 03#3730: Fix setting wgUsersNotifiedOnAllChanges - 13https://git.io/JtDFt [18:31:38] but thanks [18:32:25] PROBLEM - graylog2 APT on graylog2 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (11 critical updates). [18:32:35] miraheze/mw-config - paladox the build passed. [18:32:37] paladox: Looks good [18:32:58] [02mw-config] 07paladox closed pull request 03#3730: Fix setting wgUsersNotifiedOnAllChanges - 13https://git.io/JtDFt [18:32:59] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtDFo [18:33:00] Thanks! [18:33:01] [02miraheze/mw-config] 07paladox 03c29576d - Fix setting wgUsersNotifiedOnAllChanges (#3730) [18:33:06] !log db11: update mw_settings set s_settings = REPLACE(s_settings,'wgUsersNotifiedOnAllChanges', 'wmgUsersNotifiedOnAllChanges'); [18:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:00] miraheze/mw-config - paladox the build passed. [18:34:01] Reception123, yeah did you see the one I reported ^ last night that we should probably reach out and have them point their domain to `mw-lb.miraheze.org` instead of `cp10.miraheze.org`? It likely happens because users ping our miraheze.org domain to get what it resolves to and then they use that instead of following [[custom domains]] [18:34:05] !log root@jobrunner3:/home/paladox# /usr/local/bin/foreachwikiindblist /srv/mediawiki/w/cache/databases.json /home/paladox/resetWikiCaches.php [18:34:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:11] In short, they don't read [[custom domains]] [18:34:12] PROBLEM - jobrunner4 APT on jobrunner4 is CRITICAL: APT CRITICAL: 41 packages available for upgrade (11 critical updates). [18:34:21] PROBLEM - phab2 APT on phab2 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (11 critical updates). [18:34:25] [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3 [18:34:26] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2 [18:34:28] PROBLEM - rdb3 APT on rdb3 is CRITICAL: APT CRITICAL: 35 packages available for upgrade (11 critical updates). [18:34:45] dmehus: I've emailed them right now [18:34:56] and yes, that's a good theory, but they should really just read the page! [18:35:01] PROBLEM - mw11 APT on mw11 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [18:35:10] and actually, that doesn't explain why testwiki had it! [18:35:21] @Reception123, who is "them"? The one I reported? And yes they should :) [18:35:22] PROBLEM - test3 APT on test3 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [18:35:29] dmehus: wikiverte.pl [18:35:31] I don't know where they even find to use the cp servers. [18:35:36] PROBLEM - mw9 APT on mw9 is CRITICAL: APT CRITICAL: 12 packages available for upgrade (11 critical updates). [18:35:39] oh, I think that was it, yeah [18:35:54] PROBLEM - jobrunner3 APT on jobrunner3 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (11 critical updates). [18:36:04] Reception123, who configured the publictestwiki.com custom domain? Could've been a sysadmin error in that case :D [18:36:17] dmehus: it was definitely not me, I wouldn't have pointed to something non mw-lb! [18:36:21] so if it was not me, you know who it was :P [18:36:21] PROBLEM - mw10 APT on mw10 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [18:36:25] PROBLEM - cp11 APT on cp11 is CRITICAL: APT CRITICAL: 36 packages available for upgrade (11 critical updates). [18:36:42] but that doesn't make sense, because when publictestwiki.com was first used, the cp it was pointed to didn't even exist! [18:36:44] PROBLEM - mw8 APT on mw8 is CRITICAL: APT CRITICAL: 40 packages available for upgrade (11 critical updates). [18:36:52] Reception123, no I don't actually. Maybe NDKilla did it. :D [18:37:00] oh, weird [18:37:04] dmehus: yes, but there's still something odd yeah [18:37:11] and I don't think the entry could update itself heh [18:37:40] Reception123, were they using that geoip thing? That would explain how it got automatically updated [18:38:03] JohnLewis: minor question, but would graylog be considered as "monitoring" or not really? [18:38:07] RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 30 packages available for upgrade (0 critical updates). [18:38:07] !log install security updates on all servers [18:38:11] RECOVERY - db13 APT on db13 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [18:38:14] RECOVERY - cp10 APT on cp10 is OK: APT OK: 25 packages available for upgrade (0 critical updates). [18:38:23] RECOVERY - graylog2 APT on graylog2 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [18:38:23] dmehus: not sure [18:38:25]