[00:23:50] <paladox>	 SPF|Cloud that seems like a lot of stuff just to enable logging locally and also remotely. Whereas rsyslog simplifies all that. Less code, simpler, logs get rotated (where as with syslog-ng i don't think they do?)
[00:25:18] <SPF|Cloud>	 it's not a lot, it's a good example of allowing config stanzas to come from any puppet module
[00:25:43] <SPF|Cloud>	 and for log rotation we have logrotated
[00:27:21] <paladox>	 I mean compared to https://github.com/miraheze/puppet/blob/master/modules/base/files/rsyslog/rsyslog.conf#L56 that is alot.
[00:27:21] <MirahezeBot>	 [ puppet/rsyslog.conf at master · miraheze/puppet · GitHub ] - github.com
[00:27:37] <SPF|Cloud>	 of course, you're comparing puppet code to a static config file
[00:28:30] <paladox>	 well i mean even if that config was generated, it would still be more?
[00:28:32] <SPF|Cloud>	 and the file you have linked associates syslog facilities with file names, which isn't quite the same
[00:29:12] <paladox>	 has to generate f_file_puppet-agent, then d_file_puppet-agent and finally get it to write to the file
[00:29:23] <SPF|Cloud>	 yes, associating an filter with a destination
[00:29:30] <SPF|Cloud>	 reusable parts
[00:29:38] <SPF|Cloud>	 or objects, actually
[00:30:22] <SPF|Cloud>	 s_src_system in the code is a reference to the source input that receives messages from the system (ie when you use 'logger')
[00:31:02] <SPF|Cloud>	 and the size of the config file depends on the tabs and newlines, of course
[00:35:04] <SPF|Cloud>	 rsyslog seems to be more procedural (although the new syntax looks much better), whereas syslog-ng goes for an object oriented approach, both options have their advantages and disadvantages
[00:36:31] <paladox>	 yup
[00:39:29] <paladox>	 I just see rsyslog as beneficial to us, as we're not wasting our time which could be better spent on other things. Since i've already created a rsyslog config for remote logging whilst keeping it locally logged as well. I just need to add to that so the syslog port is open so mediawiki can log to it.
[00:39:57] <SPF|Cloud>	 moving to another syslog service is not wasting your time?
[00:40:14] <paladox>	 No, configuring it is though if you can already use the one you have?
[00:40:23] <paladox>	 oh, i misread
[00:40:51] <SPF|Cloud>	 to my knowledge I have provided you with an example of puppet code to log locally on systems with existing configuration for remote logging
[00:41:13] <paladox>	 How do i get it so it logs to a syslog file too ?
[00:41:33] <SPF|Cloud>	 that's what https://phabricator.miraheze.org/P387 does
[00:41:34] <MirahezeBot>	 [ ✎ P387 syslog-ng log to local file ] - phabricator.miraheze.org
[00:41:50] <paladox>	 that logs puppet-agent
[00:41:57] <paladox>	 it doesn't log everything from syslog
[00:42:02] <SPF|Cloud>	 if the syslog identifier matches the regex ^puppet-agent$, log to /var/log/puppet/puppet-agent.log
[00:42:16] <paladox>	 https://github.com/miraheze/puppet/blob/master/modules/base/files/rsyslog/rsyslog.conf#L62
[00:42:16] <MirahezeBot>	 [ puppet/rsyslog.conf at master · miraheze/puppet · GitHub ] - github.com
[00:42:23] <SPF|Cloud>	 logging all messages to remote services and locally?
[00:42:33] <paladox>	 yes
[00:42:51] <SPF|Cloud>	 that's what the buffer and journalctl are for
[00:43:12] <SPF|Cloud>	 in case of unavailability of other solutions
[00:43:27] <SPF|Cloud>	 + the second one is always available, regardless of log configuration
[00:43:47] <paladox>	 its not just about unavailability, its easier sometimes to just read the logs directly
[00:44:01] <paladox>	 There's also the time where it gets backlogged.
[00:44:49] <SPF|Cloud>	 easier -> if I can get examples of situations where documentation may be lacking example queries, you should let me know
[00:45:01] <SPF|Cloud>	 and backlogged -> is a bug, but even then journalctl works
[00:45:38] <SPF|Cloud>	 logs can be viewed locally
[00:47:36] <paladox>	 you could move to loki (the logs)
[00:47:51] <SPF|Cloud>	 yes, one of the options
[00:47:55] <paladox>	 i was looking at that earlier
[00:50:49] <MH-Discord>	 <DarkMatterMan4500> https://phabricator.miraheze.org/T6891 I have approved this action @Doug. You may proceed with the maintenance script bots.
[00:50:50] <MirahezeBot>	 [ ⚓ T6891 More Wikia junk to get deleted ] - phabricator.miraheze.org
[00:51:59] <paladox>	 but then logs wouldn't be behind the ssh proxy
[00:52:15] <paladox>	 so i guess we couldn't do loki unless we setup another grafana instance.
[00:52:26] <dmehus>	 DarkMatterMan4500, ok
[01:03:46] <paladox>	 SPF|Cloud do you know how we can do https://phabricator.miraheze.org/T6870 ?
[01:03:47] <MirahezeBot>	 [ ⚓ T6870 Graylog should process the timestamp based on when the log is generated ] - phabricator.miraheze.org
[01:06:07] <SPF|Cloud>	 https://community.graylog.org/t/searching-imported-logs-by-log-timestamp-not-time-graylog-received-the-log/233/3 is a good solution
[01:06:07] <MirahezeBot>	 [ Searching imported logs by log timestamp, not time Graylog received the log - #3 by lennart - Graylog - Graylog Community ] - community.graylog.org
[01:12:14] <paladox>	 oh, the mediawiki logs are not totally json
[01:12:59] <paladox>	 <11>1 2021-02-16T16:52:56+00:00 mw9.miraheze.org mediawiki - - - {"@timestamp".....
[01:13:04] <paladox>	 like its like ^ SPF|Cloud 
[01:13:22] <SPF|Cloud>	 that's the syslog header
[01:13:26] <paladox>	 yeh
[01:13:37] <paladox>	 oh wait
[01:13:39] <paladox>	 mediawiki_@timestamp
[01:13:39] <paladox>	 2021-02-16T07:52:56.304539+00:00
[01:13:42] <paladox>	 i can use that
[01:14:03] <SPF|Cloud>	 07:52?
[01:14:35] <paladox>	 its an example
[01:14:37] <paladox>	 i mean the field
[01:15:07] <SPF|Cloud>	 you have mentioned two timestamps with the same minute and second, but different hours
[01:15:18] <SPF|Cloud>	 are you sure the time isn't off by nine hours?
[01:15:53] <paladox>	 Oh, yes. That's my point.
[01:16:03] <paladox>	 the timestamp from mw json isn't being appended to the log
[01:16:17] <paladox>	 2021-02-16T16:52:56+00:00 is being generated by syslog-ng not mediawiki
[01:16:41] <SPF|Cloud>	 could be
[01:16:45] <paladox>	 we want to replace the timestamp
[01:16:51] <paladox>	 with the json version
[01:17:00] <SPF|Cloud>	 yes, using a pipeline rule
[01:17:22] <paladox>	 yup, i'm just figuring out how to do that with the example
[01:17:41] <paladox>	 i need to parse the json
[01:17:52] <paladox>	 but some of the text is plain but the rest is json
[01:18:05] <SPF|Cloud>	 what are you looking at?
[01:18:30] <SPF|Cloud>	 your job should be writing a pipeline rule replacing the timestamp field with the parsed contents of mediawiki_@timestamp
[01:19:06] <paladox>	 i'm looking at this:
[01:19:07] <paladox>	 rule "syslog-ng"
[01:19:07] <paladox>	 when
[01:19:08] <paladox>	   has_field("mediawiki_@timestamp")
[01:19:08] <paladox>	 then
[01:19:08] <paladox>	   let new_date = parse_date( to_string($message."mediawiki_@timestamp"),"yyyy-MM-dd HH:mm:ss.SSSX","UTC");
[01:19:09] <paladox>	   set_field("timestamp", new_date);
[01:19:10] <paladox>	 end
[01:19:40] <SPF|Cloud>	 that's a good starting point
[01:21:56] <paladox>	 https://graylog.miraheze.org/system/pipelines/60342f29f48b8341539dcb19
[01:22:05] <paladox>	 hmm
[01:24:39] <paladox>	 https://graylog.miraheze.org/system/pipelines/60342f29f48b8341539dcb19
[01:24:39] <paladox>	 hmm
[01:35:38] <paladox>	 so doesn't seem to work SPF|Cloud 
[01:49:23] <paladox>	 figured it out
[02:26:31] <Not-71a9>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtdQt
[02:26:33] <Not-71a9>	 [02miraheze/mw-config] 07paladox 03a2c4573 - Set memcached logging channel to info
[02:27:33] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[02:27:52] <dmehus>	 paladox?
[02:28:02] <paladox>	 ?
[02:30:01] <dmehus>	 I tried to recreate `greenleftwiki` for GreenLeft following an SN request, but it said that database name still exists. It was supposedly dropped in May 2018 in https://phabricator.miraheze.org/P60, but I think what happened is the database didn't get dropped. So wondering if you can backup that database, delete the database, then extra an XML dump from that database, so we can recreate the wiki and import its pages
[02:30:02] <MirahezeBot>	 [ ✎ P60 (An Untitled Masterwork) ] - phabricator.miraheze.org
[02:31:37] <dmehus>	 Or just recreate the wiki with that database, if that's possible and easy to do
[02:32:12] <paladox>	 I was just heading off for the night (was doing that final commit)
[02:32:31] <paladox>	 as it was late here and i have to get up early
[02:33:15] <paladox>	 dmehus ^
[02:33:19] <dmehus>	 Oh, no one else is around and Reception123 won't be around for another 4-5 hours. But okay, what would you recommend Reception123 do?
[02:33:32] <dmehus>	 Extra the XML from the database or recreate the wiki with that database?
[02:34:19] <paladox>	 do we need to create the xml? If its been deleted, all we need to do is drop the db.
[02:35:12] <dmehus>	 right, except they want their wiki restored, and I thought it was completely lost, but I guess yeah if you can just drop the database, we could use the WikiTeam XML file
[02:35:26] <dmehus>	 would it take long to drop the database?
[02:35:41] <paladox>	 Ok, i can backup the db, drop it and then you can create the wiki
[02:35:44] <paladox>	 no
[02:35:49] <dmehus>	 ok, ty
[02:35:52] <paladox>	 dropping a db is almost instantantious
[02:35:55] <dmehus>	 that'd be awesome, ty :)
[02:36:24] <dmehus>	 and just leave a note for Reception123 where he can grab the db backup from
[02:36:35] <paladox>	 !log backed up and dropped greenleftwiki on db12
[02:36:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:36:43] <dmehus>	 paladox ty :)
[02:36:55] <paladox>	 The backup is stored in my home dir
[02:36:57] <paladox>	 on db12
[02:37:03] <dmehus>	 ok, perfect, 💯
[02:37:04] <dmehus>	 night
[02:37:17] <paladox>	 thanks!
[02:46:11] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 2.14, 6.35, 3.31
[02:50:11] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.28, 3.12, 2.68
[03:12:11] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 6.56, 6.97, 3.64
[03:16:11] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.19, 3.19, 2.84
[06:03:34] <icinga-miraheze>	 RECOVERY - mw10 APT on mw10 is OK: APT OK: 29 packages available for upgrade (0 critical updates).
[06:05:51] <icinga-miraheze>	 RECOVERY - mail2 APT on mail2 is OK: APT OK: 28 packages available for upgrade (0 critical updates).
[06:06:35] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[php7.3-apcu]
[06:06:51] <icinga-miraheze>	 RECOVERY - mw11 APT on mw11 is OK: APT OK: 29 packages available for upgrade (0 critical updates).
[06:18:07] <icinga-miraheze>	 RECOVERY - mw8 APT on mw8 is OK: APT OK: 29 packages available for upgrade (0 critical updates).
[06:22:10] <icinga-miraheze>	 RECOVERY - jobrunner3 APT on jobrunner3 is OK: APT OK: 32 packages available for upgrade (0 critical updates).
[06:24:38] <icinga-miraheze>	 RECOVERY - test3 APT on test3 is OK: APT OK: 31 packages available for upgrade (0 critical updates).
[06:26:53] <icinga-miraheze>	 RECOVERY - services4 APT on services4 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[06:34:33] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:35:14] <icinga-miraheze>	 RECOVERY - services3 APT on services3 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[06:49:49] <icinga-miraheze>	 RECOVERY - jobrunner4 APT on jobrunner4 is OK: APT OK: 30 packages available for upgrade (0 critical updates).
[06:55:12] <icinga-miraheze>	 RECOVERY - mw9 APT on mw9 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[10:35:18] <Reception123>	 !log sudo service logbot start
[10:35:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:22:44] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 36.56, 28.83, 16.86
[12:23:12] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 11.02, 13.89, 6.70
[12:24:41] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.36, 22.31, 15.88
[12:26:38] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 10.76, 18.24, 15.15
[12:33:11] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is WARNING: WARNING - load average: 0.31, 2.30, 3.77
[12:35:11] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.37, 1.71, 3.38
[13:09:09] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 3.98, 4.80, 2.87
[13:11:09] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 0.77, 3.31, 2.56
[15:01:27] <icinga-miraheze>	 PROBLEM - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.teessidehackspace.org.uk could not be found
[15:01:27] <icinga-miraheze>	 PROBLEM - ndg.nenawiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ndg.nenawiki.org could not be found
[15:07:34] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+2/-0/±1] 13https://git.io/JtFzW
[15:07:36] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 0383b5c28 - add pdns to all servers
[15:08:22] <icinga-miraheze>	 RECOVERY - ndg.nenawiki.org - reverse DNS on sslhost is OK: rDNS OK - ndg.nenawiki.org reverse DNS resolves to cp10.miraheze.org
[15:08:23] <icinga-miraheze>	 RECOVERY - wiki.teessidehackspace.org.uk - reverse DNS on sslhost is OK: rDNS OK - wiki.teessidehackspace.org.uk reverse DNS resolves to cp11.miraheze.org
[15:14:34] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JtFzd
[15:14:36] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 03e3d0eb8 - add service stanza for pdns
[15:21:07] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is CRITICAL: CRITICAL - load average: 9.08, 8.65, 4.38
[15:26:41] <icinga-miraheze>	 PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 44 seconds ago with 1 failures. Failed resources (up to 3 shown): Service[pdns-recursor]
[15:28:39] <icinga-miraheze>	 RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures
[15:30:52] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JtF2R
[15:30:53] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 038a6acf0 - use hiera for pdns check
[15:32:09] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JtF26
[15:32:11] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 03423f684 - use hiera for pdns check
[15:33:00] <icinga-miraheze>	 PROBLEM - cp11 Current Load on cp11 is WARNING: WARNING - load average: 0.57, 2.80, 3.66
[15:34:01] <icinga-miraheze>	 PROBLEM - phab2 Puppet on phab2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:34:06] <icinga-miraheze>	 PROBLEM - rdb3 Puppet on rdb3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:34:58] <icinga-miraheze>	 RECOVERY - cp11 Current Load on cp11 is OK: OK - load average: 0.49, 1.99, 3.25
[15:35:35] <icinga-miraheze>	 PROBLEM - jobrunner3 Puppet on jobrunner3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[15:36:03] <icinga-miraheze>	 PROBLEM - test3 Puppet on test3 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[pdns-recursor]
[15:36:06] <icinga-miraheze>	 RECOVERY - rdb3 Puppet on rdb3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:37:58] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.70, 2.86, 1.86
[15:39:55] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.66, 2.40, 1.81
[15:49:31] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.46, 3.23
[15:50:43] <icinga-miraheze>	 PROBLEM - bacula2 Puppet on bacula2 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[pdns-recursor],File[/etc/powerdns/recursor.conf]
[15:53:30] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.17, 2.93, 3.09
[15:58:01] <icinga-miraheze>	 RECOVERY - phab2 Puppet on phab2 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[16:00:28] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+1/-0/±3] 13https://git.io/JtFwr
[16:00:30] <Not-71a9>	 [02miraheze/puppet] 07JohnFLewis 03c2fb09a - add monitoring + resolvconf
[16:04:01] <icinga-miraheze>	 RECOVERY - test3 Puppet on test3 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures
[16:06:31] <icinga-miraheze>	 RECOVERY - jobrunner3 Puppet on jobrunner3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:22:28] <icinga-miraheze>	 PROBLEM - bacula2 PowerDNS Recursor on bacula2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[16:23:55] <JohnLewis>	 https://phabricator.miraheze.org/T6740#135980
[16:23:56] <MirahezeBot>	 [ ⚓ T6740 Implement local nameserver cache daemons on servers ] - phabricator.miraheze.org
[16:24:06] <JohnLewis>	 Good read for all ^ :P
[16:28:37] <dmehus>	 JohnLewis, are we using 8.8.8.8 on our servers to perform DNS resolution lookups?
[16:29:48] <JohnLewis>	 until about 30 minutes ago, yes
[16:30:39] <dmehus>	 ah, okay, yeah, and now we're doing DNS resolution locally?
[16:31:00] <dmehus>	 The rough numbers you posted certainly seem to indicate this change makes a lot of sense
[16:31:49] <JohnLewis>	 Well, we're caching results locally for 5 minutes, rather than keep making external DNS requests. And miraheze.org is now requested directly from us and not from a public recursor
[16:32:27] <dmehus>	 Ah, that clarifies things bit more then. Thanks. :)
[16:34:05] <JohnLewis>	 paladox: status on https://phabricator.miraheze.org/T6862?
[16:34:06] <MirahezeBot>	 [ ⚓ T6862 Use memcache for the cache ] - phabricator.miraheze.org
[16:34:57] <paladox>	 Closed as resolved.
[16:36:46] <JohnLewis>	 paladox: could you prioritise https://phabricator.miraheze.org/T5537on your todo list please? As I found out today, not having such a tool really makes life extremely difficult
[16:36:47] <MirahezeBot>	 [ Login ] - phabricator.miraheze.org
[16:37:01] <paladox>	 Sure
[16:38:00] <icinga-miraheze>	 PROBLEM - cp10 Current Load on cp10 is CRITICAL: CRITICAL - load average: 4.65, 2.81, 1.73
[16:38:00] <paladox>	 I'll have a look now
[16:38:17] <dmehus>	 s/T5537on/T5537
[16:38:31] <dmehus>	 JohnLewis: s/T5537on/T5537
[16:38:31] <MirahezeBot>	 dmehus thinks JohnLewis meant to say: paladox: could you prioritise https://phabricator.miraheze.org/T5537 your todo list please? As I found out today, not having such a tool really makes life extremely difficult
[16:39:58] <icinga-miraheze>	 RECOVERY - cp10 Current Load on cp10 is OK: OK - load average: 1.72, 2.39, 1.70
[16:52:29] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.40, 3.04
[16:54:29] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.07, 2.96
[17:06:22] <SPF|Cloud>	 JohnLewis: awesome work on pdns :)
[17:20:51] <JohnLewis>	 SPF|Cloud: I was actually quite surprised with how effective it is, I knew it would be an improvement but I didn't think we'd be going for microseconds regularly :P
[17:21:45] <SPF|Cloud>	 let's say that 'just' 25ms less can be over 10% of the page load on certain page
[17:21:47] <SPF|Cloud>	 pages
[17:22:10] <SPF|Cloud>	 and https://phabricator.miraheze.org/T6765 will be the next task with the highest performance impact, I suppose
[17:22:11] <MirahezeBot>	 [ ⚓ T6765 Cache frequently accessed files on MediaWiki servers ] - phabricator.miraheze.org
[17:24:04] <SPF|Cloud>	 brb
[18:20:57] <SPF|Cloud>	 hm, wikimedia metawiki's response times are 380+ ms 
[18:20:57] <SPF|Cloud>	 our ~200 may not be that bad then ;-)
[18:21:23] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.62, 3.13
[18:25:21] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.17, 3.54, 3.22
[18:27:20] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.31, 3.17
[18:35:06] <Reception123>	 !log replaced greenleftwiki SQL with backup SQL
[18:35:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:40:59] <dmehus>	 Reception123, I just checked https://greenleft.miraheze.org/wiki/Special:CentralAuth?target=Chaleroge and that user isn't showing up as linked to CA, so seems there may be more steps to do.
[18:41:00] <MirahezeBot>	 [ Global account information for Chaleroge - GreenLeft ] - greenleft.miraheze.org
[18:41:12] <dmehus>	 I think importing XML would be better
[18:41:37] <Reception123>	 dmehus: well XML won't bring back CA..
[18:42:05] <Reception123>	 dmehus: accessing the wiki should fix it. It shows up in my CA
[18:42:26] <dmehus>	 Reception123, it will if you revert your SQL replacement, extract XML, and import it to new database
[18:42:50] <Reception123>	 dmehus: XML is only for revisions though
[18:43:01] <dmehus>	 yeah, that's what I'm saying
[18:43:07] <dmehus>	 I think that's the better approach
[18:43:12] <dmehus>	 Mind reverting and trying that?
[18:43:42] <dmehus>	 Too many unknowns here. It was a nice idea in theory, but just too many unknowns / potential problems.
[18:44:34] <Reception123>	 dmehus: well if I did it, I'd like to see if the requestor's CA shows the old edits. if it does, I'd say there's no potential problems
[18:45:00] <dmehus>	 Reception123, the requestor doesn't show as having an account in CA
[18:45:05] <dmehus>	 No one does except you
[18:45:11] <Reception123>	 dmehus: well yes, because they haven't accessed the wiki
[18:45:24] <Reception123>	 that's what I'm saying, they should access the wiki and we see if it shows them back on CA
[18:46:01] <dmehus>	 Reception123, right, but what if none of those accounts never access the wiki again? I really do think we should reset it and just import the XML
[18:46:28] <dmehus>	 otherwise we'll never easily know which user accounts are true user accounts on that wiki or phantom ones, essentially
[18:46:48] <Reception123>	 dmehus: hm? I'm not quite sure what you mean by that?
[18:49:12] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.69, 3.38, 3.15
[18:51:12] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.04, 3.05
[18:51:13] * dmehus has moved conversation to direct message with Reception123
[19:20:59] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-9/±0] 13https://git.io/JtF7v
[19:21:01] <Not-71a9>	 [02miraheze/puppet] 07paladox 03e694825 - Remove old puppetmaster module
[19:46:41] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.74, 3.19
[19:48:40] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.32, 3.09
[19:51:08] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JtFdv
[19:51:09] <Not-71a9>	 [02miraheze/puppet] 07paladox 03e5f4168 - varnish: Centralise Access-Control-Allow-Origin header
[19:51:11] <Not-71a9>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[19:51:12] <Not-71a9>	 [02puppet] 07paladox opened pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:52:04] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JtFdI
[19:52:06] <Not-71a9>	 [02miraheze/puppet] 07paladox 03ba6b517 - Update mediawiki-includes.conf.erb
[19:52:07] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:52:49] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JtFdt
[19:52:50] <Not-71a9>	 [02miraheze/puppet] 07paladox 030cf5c14 - Update mediawiki.conf.erb
[19:52:52] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:53:42] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JtFdO
[19:53:44] <Not-71a9>	 [02miraheze/puppet] 07paladox 03f29e0b3 - Update default.vcl
[19:53:45] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:54:58] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JtFdG
[19:54:59] <Not-71a9>	 [02miraheze/puppet] 07paladox 038ab8acd - Update default.vcl
[19:55:01] <Not-71a9>	 [02puppet] 07paladox synchronize pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:55:42] <Not-71a9>	 [02puppet] 07paladox edited pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:57:21] <Not-71a9>	 [02puppet] 07paladox closed pull request 03#1663: varnish: Centralise Access-Control-Allow-Origin header - 13https://git.io/JtFdf
[19:57:23] <Not-71a9>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JtFdR
[19:57:24] <Not-71a9>	 [02miraheze/puppet] 07paladox 0319ecf0c - varnish: Centralise Access-Control-Allow-Origin header (#1663)
[19:57:26] <Not-71a9>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[19:57:27] <Not-71a9>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[20:04:52] <icinga-miraheze>	 PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.80, 17.18, 12.89
[20:06:52] <icinga-miraheze>	 RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.18, 15.14, 12.66
[20:13:47] <JohnLewis>	 !log changed devnull to double-bounce (hardcoded postfix discard email)
[20:13:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:16:41] <Not-71a9>	 [02CreateWiki] 07JohnFLewis closed pull request 03#190: Further grammatical and clarity fixes - 13https://git.io/Jtdus
[20:16:43] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFFC
[20:16:44] <Not-71a9>	 [02miraheze/CreateWiki] 07dmehus 035aec7a8 - Further grammatical and clarity fixes (#190)
[20:17:36] <Not-71a9>	 [02CreateWiki] 07JohnFLewis closed pull request 03#188: Remove 'wiki_settings' and 'wiki_extensions' from cw_wikis - 13https://git.io/Jt4P4
[20:18:09] <miraheze-github>	 miraheze/CreateWiki - JohnFLewis the build passed.
[20:19:32] <Not-71a9>	 [02ManageWiki] 07JohnFLewis closed pull request 03#251: make talk pages go to namespace 1 on removal, not namespace 0 - 13https://git.io/JtHYe
[20:19:34] <Not-71a9>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFFi
[20:19:35] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 037f9b8fa - make talk pages go to namespace 1 on removal, not namespace 0 (#251)
[20:20:15] <Not-71a9>	 [02ManageWiki] 07JohnFLewis closed pull request 03#250: ManageWikiNamespacesAdditional: add help message support and improve label (T6878) - 13https://git.io/Jt9bU
[20:20:17] <Not-71a9>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFFP
[20:20:18] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 032119abe - Add help message support and improve label to ManageWikiNamespacesAdditional (#250)
[20:20:54] <miraheze-github>	 miraheze/ManageWiki - JohnFLewis the build passed.
[20:21:29] <miraheze-github>	 miraheze/ManageWiki - JohnFLewis the build passed.
[20:21:57] <Not-71a9>	 [02mw-config] 07JohnFLewis closed pull request 03#3736: Add wgNoFollowLinks and wgNoFollowNsExceptions to ManageWiki - 13https://git.io/Jt9Jp
[20:21:59] <Not-71a9>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JtFF5
[20:22:00] <Not-71a9>	 [02miraheze/mw-config] 07Kelvs599 03e501a83 - Add wgNoFollowLinks and wgNoFollowNsExceptions to ManageWiki (#3736)
[20:22:20] <Not-71a9>	 [02ManageWiki] 07JohnFLewis closed pull request 03#249: fix maintaining prefix - 13https://git.io/JtSB2
[20:22:22] <Not-71a9>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFFx
[20:22:23] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega 03bc973a4 - fix maintaining prefix (#249)
[20:22:50] <Not-71a9>	 [02mw-config] 07JohnFLewis closed pull request 03#3711: Remove namespaces that add content models, with extensions - 13https://git.io/Jt2en
[20:22:51] <Not-71a9>	 [02miraheze/mw-config] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFbv
[20:22:53] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 03d76def5 - Remove namespaces that add content models, with extensions (#3711)
[20:23:09] <miraheze-github>	 miraheze/mw-config - JohnFLewis the build passed.
[20:23:10] <Not-71a9>	 [02CreateWiki] 07JohnFLewis closed pull request 03#186: WM->create: Set time and walltime to 0 for scripts - 13https://git.io/JtsiC
[20:23:11] <Not-71a9>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtFbJ
[20:23:13] <Not-71a9>	 [02miraheze/CreateWiki] 07paladox 03cb05ea4 - WM->create: Set time and walltime to 0 for scripts (#186)
[20:23:39] <miraheze-github>	 miraheze/ManageWiki - JohnFLewis the build passed.
[20:24:00] <miraheze-github>	 miraheze/mw-config - JohnFLewis the build passed.
[20:24:19] <miraheze-github>	 miraheze/CreateWiki - JohnFLewis the build passed.
[20:30:48] <Not-71a9>	 [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL
[20:30:50] <Not-71a9>	 [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1
[20:47:16] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.96, 4.87, 3.38
[20:48:02] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.22, 7.16, 6.24
[20:49:14] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is CRITICAL: CRITICAL - load average: 6.34, 5.35, 3.74
[20:49:58] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.22, 7.25, 6.36
[20:51:53] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.77, 6.68, 6.26
[20:53:14] <icinga-miraheze>	 PROBLEM - jobrunner3 Current Load on jobrunner3 is WARNING: WARNING - load average: 5.98, 5.77, 4.29
[20:55:03] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.55, 3.25
[20:55:08] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.77, 7.66, 6.28
[20:55:47] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.61, 7.66, 6.70
[20:57:10] <icinga-miraheze>	 PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.41, 7.36, 6.37
[20:57:42] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.65, 7.93, 6.93
[21:03:04] <icinga-miraheze>	 RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.16, 6.67, 6.36
[21:03:33] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.57, 7.42, 6.95
[21:05:14] <icinga-miraheze>	 RECOVERY - jobrunner3 Current Load on jobrunner3 is OK: OK - load average: 4.71, 4.97, 4.65
[21:05:41] <icinga-miraheze>	 PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.44, 6.97, 6.06
[21:07:03] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.82, 3.56
[21:07:30] <icinga-miraheze>	 PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 7.59, 7.16
[21:07:41] <icinga-miraheze>	 RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.06, 5.85, 5.76
[21:09:02] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.89, 3.63
[21:17:20] <icinga-miraheze>	 RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.16, 6.08, 6.67
[21:22:53] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.09, 3.32
[21:39:44] <icinga-miraheze>	 PROBLEM - ping6 on dbbackup2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 100.04 ms
[21:45:44] <icinga-miraheze>	 RECOVERY - ping6 on dbbackup2 is OK: PING OK - Packet loss = 0%, RTA = 100.21 ms
[21:51:40] <icinga-miraheze>	 PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.18, 2.83
[21:53:40] <icinga-miraheze>	 RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 2.91, 2.78
[21:58:34] <icinga-miraheze>	 PROBLEM - ping4 on dbbackup1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 103.35 ms
[22:05:11] <icinga-miraheze>	 RECOVERY - ping4 on dbbackup1 is OK: PING OK - Packet loss = 0%, RTA = 104.32 ms
[22:06:53] <icinga-miraheze>	 PROBLEM - test3 HTTPS on test3 is CRITICAL: connect to address 51.195.236.247 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket
[22:07:24] <icinga-miraheze>	 RECOVERY - test3 HTTPS on test3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 561 bytes in 0.009 second response time
[22:11:56] <Not-71a9>	 [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns
[22:11:57] <Not-71a9>	 [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[22:12:13] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5
[22:12:15] <Not-71a9>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3
[22:31:32] <Not-71a9>	 [02ssl] 07Universal-Omega opened pull request 03#391: Remove wikiverte.pl (No response per T6837) - 13https://git.io/JtbvM
[22:31:54] <Not-71a9>	 [02ssl] 07Universal-Omega synchronize pull request 03#391: Remove wikiverte.pl (No response per T6837) - 13https://git.io/JtbvM
[22:36:44] <Universal_Omega>	 @JohnLewis: any ideas on https://phabricator.miraheze.org/T6835 ? Also any thoughts on https://phabricator.miraheze.org/T6771
[22:36:45] <MirahezeBot>	 [ ⚓ T6835 Wikidiscover is listing wikis that are archived, deleted or no longer accesible ] - phabricator.miraheze.org
[22:36:46] <MirahezeBot>	 [ ⚓ T6771 Request to allow archive.org embedding for tuscriaturas wiki ] - phabricator.miraheze.org
[22:40:09] <RhinosF1>	 Universal_Omega: can you update CheckUser
[22:40:27] <RhinosF1>	 Should be 2 commits from today per my comments earlier in private channel
[22:47:46] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_35 [+0/-0/±1] 13https://git.io/JtbJI
[22:47:48] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 0327f9960 - Update CheckUser
[22:49:08] <RhinosF1>	 Universal_Omega: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/667026 is coming out to fix bad past data but I'm meh
[22:56:34] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±1] 13https://git.io/JtbJF
[22:56:35] <Not-71a9>	 [02miraheze/mediawiki] 07Universal-Omega 030c95ad4 - Install GlobalNewFiles
[22:57:53] <Not-71a9>	 [02mediawiki] 07Universal-Omega opened pull request 03#1146: Install GlobalNewFiles - 13https://git.io/JtbUe
[22:58:33] <Not-71a9>	 [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b
[23:02:05] <Universal_Omega>	 RhinosF1: updated. 
[23:05:58] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-3711-Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JtbUH
[23:06:00] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 03e073bce - Revert "Remove namespaces that add content models, with extensions (#3711)"
[23:06:01] <Not-71a9>	 [02mw-config] 07Universal-Omega created branch 03revert-3711-Universal-Omega-patch-5 - 13https://git.io/vbvb3
[23:06:03] <Not-71a9>	 [02mw-config] 07Universal-Omega opened pull request 03#3739: Revert "Remove namespaces that add content models, with extensions" - 13https://git.io/JtbUQ
[23:06:08] <Not-71a9>	 [02mw-config] 07Universal-Omega closed pull request 03#3739: Revert "Remove namespaces that add content models, with extensions" - 13https://git.io/JtbUQ
[23:06:10] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JtbU5
[23:06:11] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega 0361a336b - Revert "Remove namespaces that add content models, with extensions (#3711)" (#3739)
[23:06:13] <Not-71a9>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-3711-Universal-Omega-patch-5
[23:06:14] <Not-71a9>	 [02mw-config] 07Universal-Omega deleted branch 03revert-3711-Universal-Omega-patch-5 - 13https://git.io/vbvb3
[23:07:11] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[23:07:17] <miraheze-github_>	 miraheze/mw-config - Universal-Omega the build passed.
[23:08:51] <Universal_Omega>	 paladox: mind running puppet on all servers, if your available (for the mw-config revert above)
[23:10:10] <paladox>	 done
[23:11:27] <dmehus>	 oh, right, you can only run Puppet on the mw/jbr servers right, Omega?
[23:11:34] <icinga-miraheze>	 PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:12:31] <Universal_Omega>	 paladox: Thanks!
[23:35:34] <icinga-miraheze>	 RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures