[00:00:09] and the log level should use what's before MSGID [00:01:34] and also it's not using the timestamp from the log, rather when it was generated in syslog-ng/recieved [00:01:52] that's a pipeline thing [00:01:53] though one of the funny logs does Received cli list req" repeated 11 times between [2021-03-07 00:54:51.589713] and [2021-03-07 00:56:22.541427] [00:02:45] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.82, 1.52, 1.03 [00:03:11] !log disable puppet on gluster4 for syslog testing [00:03:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:04:45] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.96, 1.40, 1.05 [00:05:10] PROBLEM - gluster4 Puppet on gluster4 is WARNING: WARNING: Puppet is currently disabled, message: SPF, last run 11 minutes ago with 0 failures [00:07:29] [02mw-config] 07Universal-Omega closed pull request 03#3753: Convert wgCreateWikiBlacklistedSubdomains to array - 13https://git.io/JqUgp [00:07:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JqkNu [00:07:31] https://graylog.miraheze.org/messages/graylog_113/298b8111-7fa2-11eb-b6dc-0200001a24a4 [00:07:32] [02miraheze/mw-config] 07Universal-Omega 03b43e32f - Convert wgCreateWikiBlacklistedSubdomains to array (#3753) [00:07:33] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [00:07:35] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [00:07:43] at least the application name is fixed [00:08:17] nice [00:08:34] but these logs are not in syslog format [00:08:41] miraheze/mw-config - Universal-Omega the build passed. [00:08:42] I guess that is causing the issues [00:09:35] let's see if this 'fix' works [00:10:18] yeh [00:10:30] [02mediawiki] 07dependabot[bot] opened pull request 03#1216: Bump extensions/PollNY from `3b9eee7` to `668f186` - 13https://git.io/JqkNM [00:10:32] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/PollNY-668f186 [+0/-0/±1] 13https://git.io/JqkND [00:10:33] [02miraheze/mediawiki] 07dependabot[bot] 03f43b9ec - Bump extensions/PollNY from `3b9eee7` to `668f186` [00:10:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/MirahezeMagic-75330f6 [+0/-0/±1] 13https://git.io/JqkNy [00:10:35] is there something to trigger an update to glusterd.log? [00:10:36] [02miraheze/mediawiki] 07dependabot[bot] 035e73ce8 - Bump extensions/MirahezeMagic from `7e28e8f` to `75330f6` [00:10:38] [02mediawiki] 07dependabot[bot] opened pull request 03#1217: Bump extensions/MirahezeMagic from `7e28e8f` to `75330f6` - 13https://git.io/JqkNS [00:10:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/PollNY-668f186 - 13https://git.io/vbL5b [00:10:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/AJAXPoll-8f2cb6a [+0/-0/±1] 13https://git.io/JqkNH [00:10:42] [02miraheze/mediawiki] 07dependabot[bot] 03a2bd52c - Bump extensions/AJAXPoll from `ee75101` to `8f2cb6a` [00:10:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/MirahezeMagic-75330f6 - 13https://git.io/vbL5b [00:10:45] [02mediawiki] 07dependabot[bot] opened pull request 03#1218: Bump extensions/AJAXPoll from `ee75101` to `8f2cb6a` - 13https://git.io/JqkNQ [00:10:47] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_35/extensions/AJAXPoll-8f2cb6a - 13https://git.io/vbL5b [00:10:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_35/extensions/MediaWikiChat-eadada7 [+0/-0/±1] 13https://git.io/JqkN7 [00:10:50] [02miraheze/mediawiki] 07dependabot[bot] 03eac2561 - Bump extensions/MediaWikiChat from `ecf4031` to `eadada7` [00:10:53] err [00:10:55] whoops [00:10:55] https://graylog.miraheze.org/messages/graylog_113/b957a351-7fa2-11eb-b6dc-0200001a24a4 [00:11:11] what about that? :) [00:11:25] Poor Not-71a9 :( [00:11:31] nice! [00:11:42] SPF|Cloud does graylog parse the time? [00:11:55] no [00:12:13] i think we're going to have to for Received cli list req" repeated 11 times between [2021-03-07 00:54:51.589713] and [2021-03-07 00:56:22.541427] parse the first time [00:12:14] you need a pipeline for that, just like mediawiki logs [00:12:19] ah [00:15:49] I have reviewed your PR [00:16:01] !log enable puppet on gluster4 [00:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:17:09] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:17:23] SPF|Cloud what other changes are needed for my pr? [00:17:54] have you seen the comment regarding the options array? [00:18:12] oh no [00:18:23] + https://github.com/miraheze/puppet/pull/1672/files#diff-8dee9a1f54a342ce8d75548f3679aa85d8f42d778a3e0146ae50e2a104079792R18 shouldn't this be set to 'glusterd'? [00:18:24] [ Introduce gluster::logging by paladox · Pull Request #1672 · miraheze/puppet · GitHub ] - github.com [00:18:34] miraheze/mw-config - Universal-Omega the build passed. [00:18:35] otherwise you'll get very strange program names [00:19:01] well it's using $title thus we can do gluster::logging { 'glusterd': .....} [00:19:46] I don't expect people to know that $title will be the value for such an important value as the program names [00:20:12] if you really need to change the program name, there should be a parameter for the program name [00:20:48] https://github.com/miraheze/puppet/pull/1672/files#diff-8dee9a1f54a342ce8d75548f3679aa85d8f42d778a3e0146ae50e2a104079792R34-R36 and this indent is still incorrect [00:20:49] [ Introduce gluster::logging by paladox · Pull Request #1672 · miraheze/puppet · GitHub ] - github.com [00:21:00] yeh [00:21:21] SPF|Cloud i mean, the logs are named differentl, thus we want to have it named differently per log file. [00:21:33] Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Gluster::Logging[glusterd]: parameter 'file_source_options' index 1 expects a String value, got Struct (file: /etc/puppetlabs/puppet/environments/production/modules/gluster/manifests/init.pp, line: 109) on node gluster3.miraheze.org [00:21:45] that's alright, but give it a program_name parameter [00:21:55] oh [00:21:57] actually [00:22:02] ahh [00:22:04] ok [00:22:25] PROBLEM - gluster3 Puppet on gluster3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:22:41] i guess i can do {} for if program_name is not set in log [00:23:08] program must have a valid value, even if not set [00:23:40] if gluster consists of multiple components, you can go for gluster- and have 'gluster' as a fallback [00:23:49] or make a program name mandatory and fail hard if the name isn't set :) [00:26:41] SPF|Cloud does it look better? :) [00:27:10] PROBLEM - gluster4 Puppet on gluster4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:27:35] I would wonder what happens if program_name is null [00:27:49] sending a syslog message without a program name... very interesting concept [00:28:23] RECOVERY - gluster3 Puppet on gluster3 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [00:28:44] puppet applied [00:29:09] RECOVERY - gluster4 Puppet on gluster4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [00:29:17] my advice is to make 'gluster' or 'glusterd' as the default program name [00:29:27] or make program_name a required(!) parameter [00:29:48] other than that no comments [00:30:20] can I go to sleep now? ;) [00:30:21] ok, making it required [00:30:27] yes! :) [00:30:48] night [00:30:53] good luck with the PR [00:31:49] thanks! [00:41:01] SPF|Cloud oh one other thing if you are still around? [00:41:06] guess you've gone [00:51:12] PROBLEM - cloud3 Puppet on cloud3 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown) [00:57:10] RECOVERY - cloud3 Puppet on cloud3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:14:49] Try /invite? [01:15:46] i have, doesn't seem to work [01:17:38] i'll leave things for the night [01:32:09] miraheze/mw-config - Universal-Omega the build passed. [01:40:59] miraheze/mw-config - Universal-Omega the build passed. [01:41:59] miraheze/mw-config - Universal-Omega the build passed. [01:43:14] miraheze/mw-config - Universal-Omega the build passed. [02:22:45] miraheze/mw-config - Universal-Omega the build passed. [04:10:33] miraheze/mw-config - Universal-Omega the build passed. [04:25:23] miraheze/mw-config - Universal-Omega the build passed. [04:26:53] miraheze/mw-config - Universal-Omega the build passed. [04:34:08] miraheze/mw-config - Universal-Omega the build passed. [06:37:26] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 345.42 ms [06:39:24] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 268.93 ms [06:47:27] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 344.79 ms [06:51:26] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 273.80 ms [06:57:17] JohnLewis: not sure if you're still around but regarding the canned responses is it possible to 1) be able to default to the "Perfect request" response rather than the first one so that one doesn't have to do that for each approval? and 2) would it be possible to also have an "Other" option that opens a text form instead? [06:57:18] PROBLEM - ping6 on cp3 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:57:54] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2400:6180:0:d0::403:f001/cpweb [06:59:22] PROBLEM - ping6 on cp3 is WARNING: PING WARNING - Packet loss = 0%, RTA = 345.19 ms [06:59:54] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:23:22] RECOVERY - ping6 on cp3 is OK: PING OK - Packet loss = 0%, RTA = 272.33 ms [08:32:00] Please deal with [[phab:T6935]] ASAP. [08:32:01]

https://phabricator.miraheze.org/T6935 [08:32:02] [ ⚓ T6935 Let Wiki Creators Manually Respond to Requests ] - phabricator.miraheze.org [08:32:18] I think the quickest solution is to roll the changes back. [08:35:19] miraheze/mw-config - RhinosF1 the build passed. [08:36:41] Thank you! [08:44:03] miraheze/mw-config - Reception123 the build passed. [08:45:48] Reception123: 1) that’s the order of how UO did it. 2) We can, but then we might as well remove the selection list because the whole point was standardised responses that mean we can make AI more effective [08:47:38] Responses should be entirely concise, generic and appropriate. If the ones added aren’t, they need to be changed rather than making the feature entirely useless [08:48:06] well for 2 it seems that at least for the time being the current options for canned responses seems unsatisfactory, so perhaps we could work on adding a larger variety of responses before implementing the canned responses as a general rule [08:48:49] the current ones do seem not to be very generic [08:54:23] They were added by UO entirely at the request of Dmehus, so the issue is perhaps they are based entirely on one persons idea of what they should be not what they appropriately need to be [08:54:30] but that’s a configuration issue [08:55:19] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 73s [08:55:36] Yeah, so perhaps then there should be more discussion surrounding the actual canned responses before they're implemented [08:56:37] https://phabricator.miraheze.org/T6935#136946 [08:56:38] [ ⚓ T6935 Let Wiki Creators Manually Respond to Requests ] - phabricator.miraheze.org [08:57:19] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 9s [08:57:51] It is evident that the current options were added without reading them. They are not appropriate for use everywhere. [09:00:06] I do agree they’re not appropriate for every day use, they seem only specific with generic expectations [09:11:06] miraheze/mw-config - R4356th the build passed. [10:06:08] Hi Reception123 [10:06:38] hi RhinosF1, is it ready to be merged then? [10:06:52] Reception123: half past [10:06:58] 25 minutes [10:06:59] RhinosF1: ok, so 25 minutes to go then :) [10:30:14] RhinosF1: doing then [10:30:53] Merged, not sure why Not-* isn't working though [10:31:13] Reception123: ty [10:31:32] np [10:32:30] Reception123: now we wait 11 minutes so the TTL has expired on DNS and test it. [10:32:40] k [10:32:40] check puppet has ran on ns* [10:32:52] it has, done manually [10:42:37] Reception123: works [10:42:44] great [10:54:11] Reception123: all complete [10:54:38] :) [11:41:51] miraheze/mw-config - R4356th the build passed. [11:48:27] MirahezeBot> Not-71a9 was kicked from #miraheze-sre by paladox (Your behavior is not conducive to the desired environment.) [11:48:29] ^ why was this done? [11:48:34] it seems that Not-* doesn't work anymore now [11:50:01] RECOVERY - db1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - db1.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [11:50:01] RECOVERY - db1.miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Fri 26 Mar 2021 02:32:28 GMT +0000. [11:50:31] RECOVERY - phab.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phab.bots.miraheze.wiki reverse DNS resolves to cp10.miraheze.org [11:50:36] RECOVERY - phabdigests.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phabdigests.bots.miraheze.wiki reverse DNS resolves to cp10.miraheze.org [11:50:51] RECOVERY - tools1.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - tools1.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [11:51:46] RECOVERY - phab-storage.bots.miraheze.wiki - reverse DNS on sslhost is OK: rDNS OK - phab-storage.bots.miraheze.wiki reverse DNS resolves to cp11.miraheze.org [12:06:15] miraheze/mw-config - R4356th the build passed. [12:17:59] miraheze/mw-config - R4356th the build has errored. [12:20:43] miraheze/mw-config - R4356th the build passed. [12:31:59] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is WARNING: MariaDB replication - both - WARNING - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 29s [12:33:32] Reception123: icinga likes us too, I guess my downtime expired [12:33:43] Backups are 30s behind [12:33:46] That's meh [12:33:54] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [12:34:05] Oh good you caught up [12:47:10] Reception123: it was an accident when I was trying to quiet it [12:47:39] paladox: hmm, so why do you think it's not coming back? [12:47:54] do we need to do something to get it back? [12:48:46] I’m not sure, I tried to get it back but failed [12:50:00] RhinosF1: ^ any ideas? [12:50:05] .op [12:50:05] Attempting to OP... [12:50:30] .deop [12:50:30] Attempting to OP... [12:50:43] I used /invite but that's my only idea [12:53:44] I think NDK was the one who set up this system iirc [12:55:52] @NDKilla let me know when you're around, we need some help with Notifico [12:59:02] RhinosF1: yeah the GitHub webhook page makes it appear like everything is fine so nothing there :( [13:00:21] maybe I can try emailing the dev and see if they can help [13:12:58] Ive changed notifico to me [14:11:53] PROBLEM - privacy-wiki.0x.no - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacy-wiki.0x.no could not be found [14:56:12] miraheze/CreateWiki - translatewiki the build passed. [14:56:18] miraheze/ManageWiki - translatewiki the build passed. [14:56:27] miraheze/MirahezeMagic - translatewiki the build passed. [15:34:09] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 71s [15:35:15] Huh [15:49:28] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [16:07:28] JohnLewis: mind seeing my reply on https://phabricator.miraheze.org/T6935#136962? [16:07:29] [ ⚓ T6935 Let Wiki Creators Manually Respond to Requests ] - phabricator.miraheze.org [16:08:54] Because I also think to make it easier on wikicreators. That there should be a freeform option. [16:09:21] That's why I think we should use `combobox` instead of `select` type for that. [16:12:12] Universal_Omega: See my previous comment on the task which equates to "canned or nothing" [16:12:49] As the original purpose was to minimise differences in reasons so AI could understand and interpret them more clearly, which is the main reason why I agreed to add them [16:17:44] JohnLewis: Why though? CannedResponses are good for options, but I still think there should be an option to add a free-form option. I don't understand why we can't have them be free-form as an option. But it's up to you I guess. Also, I fixed the bio field we discussed yesterday, but purposes still doesn't seem to save. Also Reception123 and JohnLewis: I just added the ones Dmehus gave me to add (in the order they were [16:17:44] given), so if they weren't good, apologies, I should've discussed them with more wikicreators first. [16:19:33] Until [16:20:17] Misclicked [16:21:31] I originally envisioned having an "other" option in the drop down box that would make available a freeform text entry approval/decline reason box. So, I do agree with having both canned responses and an other box. If that's too difficult to do, why not just have an "Other reason (see "comments" tab)" option and allow wiki creators to add a decline reason as a comment if one of the canned responses does not fit? Also, the canned responses can [16:21:32] be edited and added to as needed [16:22:06] JohnLewis: I think half the canned responses were poor anyway but we should allow people to edit the response a bit or expand in a single response. Could that be made possible? [16:22:10] dmehus: see your PN [16:22:12] PM [16:36:10] PROBLEM - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is CRITICAL: MariaDB replication - both - CRITICAL - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 64s [16:43:16] It’s possible but defeats the point, so if people want to do it I guess but I’ll drop the potential support for more accurate AI as a result because it becomes impossible [16:47:36] JohnLewis, true. That's a good point. What if we did it temporarily (say for 30 days) until we build out the canned responses more? This would also allow for a wiki creator retraining period to go into the "comments" tab and add their customized messaging following the decline [16:47:39] RECOVERY - dbbackup1 Check MariaDB Replication c2 on dbbackup1 is OK: MariaDB replication - both - OK - Slave_IO_Running state : Yes, Slave_SQL_Running state : Yes, Seconds_Behind_Master : 0s [16:48:12] s/build out/build out and refine [16:48:12] dmehus meant to say: JohnLewis, true. That's a good point. What if we did it temporarily (say for 30 days) until we build out and refine the canned responses more? This would also allow for a wiki creator retraining period to go into the "comments" tab and add their customized messaging following the decline [16:48:52] I think the real solution is just not to rush into making it require and seek feedback over it [16:49:37] miraheze/mw-config - Universal-Omega the build passed. [16:55:53] JohnLewis, yeah, I think that's fair. Originally, though, I did envision having a freeform decline reason field that would be greyed out unless the "Other" option was selected. I also envisioned having separate approval and decline reason drop-down fields, which would only be displayed when either "approve" or "decline" was selected or, failing that, one of which would be greyed out when the other was selected [17:07:15] miraheze/mw-config - Universal-Omega the build passed. [17:07:56] Freeform decline would be fine [17:09:15] miraheze/mw-config - Universal-Omega the build passed. [17:10:10] miraheze/mw-config - Universal-Omega the build passed. [17:10:35] miraheze/mw-config - Universal-Omega the build passed. [17:10:45] miraheze/mw-config - Universal-Omega the build passed. [17:16:40] JohnLewis: How do I use the populateWikiSettings script for migrating settings to ManageWikiNamespaces? Is that possible, or do I have to do it some other way? [17:17:31] It’s only for extensions and settings [17:18:09] It has to be manually for namespace settings [17:21:52] JohnLewis: OK, thanks, and can I do that manually then from ManageWiki on the wikis? Also, do you think this will work for overridedefault? https://github.com/miraheze/mw-config/blob/d3064a994be173a56271c0c479eb719e8644d300/ManageWikiNamespaces.php#L221-L224 (do you think using the $wgDefaultRobotPolicy will work I mean as that's set via MWS and not sure if it can be used as default from MWN. [17:21:53] [ mw-config/ManageWikiNamespaces.php at d3064a994be173a56271c0c479eb719e8644d300 · miraheze/mw-config · GitHub ] - github.com [17:30:14] I have no idea if that would work as I don’t know the setting off my memory. And yea, you can set them if your migrating it from LS [17:30:48] JohnLewis: Thanks! [18:11:48] miraheze/CreateWiki - Universal-Omega the build passed. [18:32:40] miraheze/mw-config - Universal-Omega the build passed. [18:46:27] !log reception@jobrunner4:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=shekhwiki --r "[[phab:T6933|Requested]]" /home/reception/shekhdel.txt [18:46:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:50:30] miraheze/mw-config - Universal-Omega the build passed. [18:51:02] paladox: hmm, it still doesn't seem to work :( [18:51:07] (Not-* that is) [18:51:13] yeh [18:52:04] miraheze/mw-config - Universal-Omega the build passed. [18:54:37] Reception123: if don't work we can look at https://github.com/sopel-irc/sopel-github [18:54:38] [ GitHub - sopel-irc/sopel-github: GitHub plugin for Sopel ] - github.com [18:54:41] On MirahezeBot [18:54:54] miraheze/mw-config - Universal-Omega the build passed. [18:55:25] hmm, ok, we might have to [18:56:31] Ask MacFan to look at the reverse proxy stuff it talks about [18:56:43] If he can set the web hook thing up then I can do [19:00:09] Based on my own troubles with Notifico in the past, I'd prefer we use sopel-github. [19:00:44] miraheze/mw-config - Universal-Omega the build passed. [19:01:04] Sario: I just need someone to look at the reverse proxy stuff [19:01:16] If we can get a webhook working then I don't see why not [19:01:36] Note we'll have to use IP not hostname as 3333 won't be open via hostname proxy [19:11:40] miraheze/mw-config - Universal-Omega the build passed. [19:15:30] miraheze/mw-config - Universal-Omega the build passed. [19:20:38] PROBLEM - mail2 Current Load on mail2 is CRITICAL: CRITICAL - load average: 2.62, 2.28, 1.07 [19:20:58] JohnLewis: what I was saying would happen with the talk namespace change logging does happen, now when the talk namespace is changed it just logs with $5 (no namespace name) [19:21:36] Perhaps, but not for the reason for which you thought it would [19:22:38] PROBLEM - mail2 Current Load on mail2 is WARNING: WARNING - load average: 1.35, 1.73, 1.01 [19:24:39] PROBLEM - mail2 Current Load on mail2 is CRITICAL: CRITICAL - load average: 5.52, 3.27, 1.67 [19:24:50] Universal_Omega: fix committed [19:25:40] miraheze/ManageWiki - JohnFLewis the build passed. [19:28:01] JohnLewis: Isn't it the reason I said? I said it wouldn't log correctly. Adding the check to only add 5::namespace means it won't read it if it's a talk namespace? And thanks for fixing! Wait won't the fix make it like it was never changed at all? If log params is empty it'll log regardless, talk namespace is always empty, then the other check will counteract that, or am I mistaken? [19:28:51] if talk is changed and there's no main namespace entry, it'll log talk [19:28:58] If talk and namespace change, it'll log main [19:29:09] Oh makes sense. Thanks! [19:30:39] RECOVERY - mail2 Current Load on mail2 is OK: OK - load average: 0.05, 1.39, 1.40 [19:56:58] miraheze/ManageWiki - Universal-Omega the build passed. [20:07:34] PROBLEM - ldap2 Puppet on ldap2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:10:22] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:10:24] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 2 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:10:29] PROBLEM - jobrunner4 Puppet on jobrunner4 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:10:33] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:10:50] PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:11:11] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown) [20:12:22] me [20:25:32] miraheze/MirahezeMagic - Universal-Omega the build passed. [20:37:34] RECOVERY - ldap2 Puppet on ldap2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:45:07] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 4.19, 2.52 [21:45:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.65, 6.32, 4.28 [21:46:36] PROBLEM - encyclopediaofastrobiology.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for encyclopediaofastrobiology.org could not be found [21:47:04] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.85, 3.27, 2.39 [21:47:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.99, 5.40, 4.18 [21:53:28] RECOVERY - encyclopediaofastrobiology.org - reverse DNS on sslhost is OK: rDNS OK - encyclopediaofastrobiology.org reverse DNS resolves to cp10.miraheze.org [23:45:49] miraheze/mw-config - Universal-Omega the build passed. [23:49:15] miraheze/mw-config - Universal-Omega the build passed. [23:52:10] miraheze/mw-config - Universal-Omega the build passed. [23:55:25] miraheze/mw-config - Universal-Omega the build passed. [23:56:42] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown)