[00:02:42] <icinga-miraheze>	 RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 3798 MB (19% inode=93%);
[02:25:11] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf11Z
[02:25:12] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 0398daf3f - BOT: Updating services config for wikis
[03:00:51] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.93, 2.26, 1.58
[03:02:51] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 1.00, 1.79, 1.49
[03:04:49] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.61, 1.37, 1.37
[03:10:20] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf1Md
[03:10:22] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 03e1d21fe - BOT: Updating services config for wikis
[05:43:36] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is CRITICAL: CRITICAL - load average: 1.24, 2.76, 2.15
[05:44:41] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 3.82, 3.45, 2.22
[05:54:46] <icinga-miraheze>	 PROBLEM - cp8 Current Load on cp8 is WARNING: WARNING - load average: 0.44, 1.43, 1.97
[05:55:17] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 0.20, 1.37, 1.95
[05:59:15] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.04, 0.67, 1.52
[06:00:45] <icinga-miraheze>	 RECOVERY - cp8 Current Load on cp8 is OK: OK - load average: 0.57, 0.93, 1.59
[06:40:19] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf19h
[06:40:20] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 031fd1967 - BOT: Updating services config for wikis
[07:22:09] <Naleksuh>	 is anyone from miraheze wiki creators online?
[07:27:09] <MH-Discord>	 <MrJaroslavik> Yeah
[07:27:39] <MH-Discord>	 <MrJaroslavik> Done by RhinosF1
[07:27:49] <Naleksuh>	 how did you even know what i wanted
[07:27:52] <Naleksuh>	 before you read #degeneral?
[07:37:48] <RhinosF1>	 :)
[07:38:32] <Naleksuh>	 i prefer irc over discord to begin with
[07:38:38] <Naleksuh>	 but felt like using both for maximum impatience
[07:38:49] <Naleksuh>	 but now i realize i cant even start editing the wiki for another 8 hours
[10:55:02] <Not-1447>	 [02miraheze/IncidentReporting] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf1d5
[10:55:03] <Not-1447>	 [02miraheze/IncidentReporting] 07JohnFLewis 039e56707 - Technical Debt: Fix function casing
[10:55:07] <Not-1447>	 [02miraheze/MatomoAnalytics] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jf1dd
[10:55:09] <Not-1447>	 [02miraheze/MatomoAnalytics] 07JohnFLewis 03efb4617 - Technical Debt: Convert to methods when function is non-static
[10:55:13] <Not-1447>	 [02miraheze/MirahezeMagic] 07JohnFLewis pushed 037 commits to 03master [+0/-0/±8] 13https://git.io/Jf1dF
[10:55:14] <Not-1447>	 [02miraheze/MirahezeMagic] 07JohnFLewis 0344d9eae - Fix JSON
[10:55:16] <Not-1447>	 [02miraheze/MirahezeMagic] 07JohnFLewis 03fec120c - Technical Debt: Don't override parent visibility
[10:55:17] <Not-1447>	 [02miraheze/MirahezeMagic] 07JohnFLewis 03ffed177 - Technical Debt: Don't assign variables to immediately return them
[10:55:19] <Not-1447>	 [02miraheze/MirahezeMagic] ... and 4 more commits.
[10:55:20] <Not-1447>	 [02miraheze/WikiDiscover] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf1db
[10:55:22] <Not-1447>	 [02miraheze/WikiDiscover] 07JohnFLewis 03f317636 - Technical Debt: Don't override parent visibility
[10:55:25] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 035 commits to 03master [+0/-0/±6] 13https://git.io/Jf1dN
[10:55:27] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis 03f1af43a - Technical Debt: Fix function casing
[10:55:28] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis 033fab7bf - Technical Debt: Don't override parent visibility
[10:55:30] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis 030530295 - Remove eval usage for safer, more common sense approach for limits
[10:55:31] <Not-1447>	 [02miraheze/ManageWiki] ... and 2 more commits.
[10:55:33] <Not-1447>	 [02miraheze/CreateWiki] 07JohnFLewis pushed 034 commits to 03master [+0/-0/±8] 13https://git.io/Jf1dA
[10:55:34] <Not-1447>	 [02miraheze/CreateWiki] 07JohnFLewis 032fe5cd8 - Technical Debt: Fix function casing
[10:55:36] <Not-1447>	 [02miraheze/CreateWiki] 07JohnFLewis 036688260 - Technical Debt: Adding missing useful returns/move out of if
[10:55:37] <Not-1447>	 [02miraheze/CreateWiki] 07JohnFLewis 0349ee0c0 - Technical Debt: Don't override parent visibility
[10:55:39] <Not-1447>	 [02miraheze/CreateWiki] 07JohnFLewis 03377ace8 - Removal of potential minor bugs + non fatal error
[13:44:10] <icinga-miraheze>	 PROBLEM - mail1 APT on mail1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates).
[13:50:57] <paladox>	 !log upgrade nodejs & puppet-agent on mail1
[13:51:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:52:10] <paladox>	 !log upgrade nodejs & puppet-agent on services[12]
[13:52:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[13:52:19] <icinga-miraheze>	 RECOVERY - mail1 APT on mail1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[14:09:44] <icinga-miraheze>	 PROBLEM - cp7 Current Load on cp7 is CRITICAL: CRITICAL - load average: 8.01, 3.88, 2.05
[14:11:50] <icinga-miraheze>	 RECOVERY - cp7 Current Load on cp7 is OK: OK - load average: 2.68, 3.20, 2.04
[14:35:14] <paladox>	 !log upgrade nodejs on mw[4567] & test2 & jobrunner1
[14:35:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:37:55] <Not-1447>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jf1hm
[14:37:57] <Not-1447>	 [02miraheze/puppet] 07paladox 03a85f3dd - nodejs: Also install npm
[14:43:06] <paladox>	 !log npm install on mw* & jobrunner1 & test2 in /srv/mathoid
[14:43:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:49:42] <paladox>	 !log running runJobs in foreach on jobrunner1
[14:49:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[15:06:24] <Not-1447>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/Jf1jl
[15:06:25] <Not-1447>	 [02miraheze/CreateWiki] 07paladox 03730e6a3 - Make sure file write is succeeded before renaming
[15:06:27] <Not-1447>	 [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL
[15:06:28] <Not-1447>	 [02CreateWiki] 07paladox opened pull request 03#146: Make sure file write is succeeded before renaming - 13https://git.io/Jf1j8
[15:11:51] <Not-1447>	 [02CreateWiki] 07paladox edited pull request 03#146: Make sure file write is succeeded before renaming - 13https://git.io/Jf1j8
[17:07:08] <RhinosF1>	 paladox: 502
[17:10:13] <musikanimal>	 hello! at 5:02 UTC someone tried to create an account on spcodex.wiki and they said they got a database error. I see User:Eye was successfully created, but they're saying they can't login, and they never got a confirmation email as they should
[17:10:35] <musikanimal>	 the exception code is 6197c73fb294e88478262ec9
[17:11:05] <musikanimal>	 maybe you could search the logs to see what happened?
[17:14:40] <paladox>	 Error: 1406 Data too long for column 'cuc_agent' at row 1 (dbt1.miraheze.org)
[17:14:43] <paladox>	 musikanimal ^
[17:15:23] <musikanimal>	 oh wow. Well, that sounds like a problem! a problem I can't solve, though :(
[17:15:33] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfMJX
[17:15:34] <Not-1447>	 [02miraheze/mediawiki] 07paladox 03272be8e - Update SP
[17:17:17] <musikanimal>	 paladox: can you confirm that happened around 5:02 UTC? I ask because I had to type the error code from a screenshot, so I want to make sure it's the right one
[17:17:43] <paladox>	 musikanimal 05:32
[17:18:03] <musikanimal>	 they said it just happened again when trying to reset their password
[17:18:35] <musikanimal>	 59890a562f153a107decf569
[17:18:38] <paladox>	 they seem to have a large UA
[17:18:39] <paladox>	 heh
[17:19:00] <paladox>	 musikanimal https://github.com/wikimedia/mediawiki-extensions-CheckUser/blob/c111347e82d1e044131e21e5350b2f3c34dd44e5/cu_changes.sql#L51
[17:19:01] <ZppixBot>	 [ mediawiki-extensions-CheckUser/cu_changes.sql at c111347e82d1e044131e21e5350b2f3c34dd44e5 · wikimedia/mediawiki-extensions-CheckUser · GitHub ] - github.com
[17:19:06] <paladox>	 i think this should be increased
[17:19:19] <paladox>	 to at least a mediumtext
[17:20:01] <musikanimal>	 it's a VARCHAR(255) on WMF production too
[17:20:53] <paladox>	 musikanimal does the WMF have strict sql mode switched off?
[17:21:24] <musikanimal>	 hmm I dunno. But surely if UAs being too long were a thing it would have come up by now
[17:21:53] <paladox>	 well it won't be an issue if strict sql mode is disabled
[17:22:09] <paladox>	 mysql will silently truncate under that, with strict sql mode it becomes an issue.
[17:23:38] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.35, 21.51, 17.84
[17:23:48] <musikanimal>	 is that something we could turn on for Miraheze?
[17:24:38] <RhinosF1>	 That'd be something the DBAs would know for sure if it's on at wmf
[17:25:12] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMJh
[17:25:14] <paladox>	 we have strict mode enabled, i think you mean disable?
[17:25:14] <Not-1447>	 [02miraheze/services] 07MirahezeSSLBot 0341a4535 - BOT: Updating services config for wikis
[17:25:21] <musikanimal>	 yeah sorry, disable
[17:25:47] <musikanimal>	 this person was using the Facebook Messenger browser which is used by default when clicking on links, so this probably will happen again
[17:26:49] <musikanimal>	 seems like CheckUser should just truncate for you, though
[17:27:19] <musikanimal>	 https://phabricator.wikimedia.org/T199323
[17:27:20] <ZppixBot>	 [ ⚓ T199323 Checkuser does not truncate data (cuc_comment, cuc_agent, cuc_actiontext) and that causes exceptions with strict mode ] - phabricator.wikimedia.org
[17:27:51] <musikanimal>	 so yeah, WMF must have strict mode off
[17:28:14] <paladox>	 oh
[17:29:28] <musikanimal>	 so this could happen for edit summaries too
[17:29:44] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 18.18, 19.44, 18.17
[17:29:47] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:30:24] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:30:33] <icinga-miraheze>	 PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:30:36] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:30:56] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:30:58] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[17:32:46] <icinga-miraheze>	 RECOVERY - mw6 Puppet on mw6 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures
[17:33:05] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[17:34:06] <icinga-miraheze>	 RECOVERY - test2 Puppet on test2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:34:16] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is CRITICAL: CRITICAL - load average: 24.66, 25.04, 21.01
[17:36:41] <musikanimal>	 I'm going to test that patch tomorrow and +2 it
[17:36:48] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures
[17:37:09] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:38:08] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 14.49, 21.17, 20.47
[17:40:07] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.06, 18.35, 19.49
[17:42:37] <paladox>	 awesome!
[17:47:00] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:56:18] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfMUS
[17:56:20] <Not-1447>	 [02miraheze/mediawiki] 07paladox 03153466e - Update DynamicPageList3
[18:12:45] <ZppixBot>	 Hello frog_on_a_log! If you have any questions, feel free to ask and someone should answer soon.
[18:25:47] <ZppixBot>	 Hello log_on_a_bog! If you have any questions, feel free to ask and someone should answer soon.
[18:32:14] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.64, 19.28, 18.07
[18:34:13] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.46, 16.29, 17.10
[18:42:24] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.50, 20.86, 18.94
[18:44:21] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 14.02, 18.26, 18.21
[18:50:18] <MH-Discord>	 <AB Louis> Hi 👋
[18:50:25] <RhinosF1>	 Hi
[18:52:29] <MH-Discord>	 <AB Louis> Nice to read you. Do you have some news about the creative and off-topic that some people here have created?
[19:07:37] <Not-1447>	 [02miraheze/dns] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMk9
[19:07:38] <Not-1447>	 [02miraheze/dns] 07Southparkfan 0378fdfd6 - Remove obsolete TXT records
[19:25:41] <Not-1447>	 [02miraheze/puppet] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMIT
[19:25:42] <Not-1447>	 [02miraheze/puppet] 07Southparkfan 0325e03e0 - Add new ssh key
[20:30:40] <ZppixBot>	 Hello Jexa! If you have any questions, feel free to ask and someone should answer soon.
[20:31:33] <paladox>	 Texas RhinosF1 could you fix ^ please?
[20:31:43] <paladox>	 (it does not ban)
[20:32:09] <Texas>	 paladox: It is a bug with eggdrop. It thinks this channel's banlist is full
[20:32:17] <Texas>	 if you clear it out it will ban
[20:32:18] <paladox>	 oh
[20:34:10] <paladox>	 Texas is there anyway to fix that?
[20:34:12] <Not-1447>	 [02puppet] 07Southparkfan commented on pull request 03#1388: Add Reddit and Dropbox to CSP whitelist T5685 - 13https://git.io/JfMtq
[20:34:53] <Texas>	 paladox: According to #eggdrop wait for it to be patched in the next version
[20:34:58] <paladox>	 ok
[20:35:08] <Texas>	 and paladox: You do know there is a ChanServ command for that
[20:35:24] <paladox>	 what command?
[20:35:51] <Texas>	  /msg ChanServ CLEAR #miraheze bans
[20:35:55] <Texas>	 paladox: ^
[20:36:02] <paladox>	 oh!
[20:36:12] <Texas>	 paladox: AmandaNP did the same thing lol
[20:36:36] <Texas>	 much faster huh
[20:40:15] <Texas>	 good girl SigmaBot
[20:40:21] <Texas>	 see paladox now it bans :)
[20:40:40] <paladox>	 we had over 100 bans? heh
[20:40:43] <paladox>	 https://github.com/eggheads/eggdrop/commit/1b348edfec672118a18c8d8b0941b367c7293c24#diff-6adb617016fe9df81660103f49ffd851R1067
[20:40:44] <ZppixBot>	 [ Correctly set use-354 (WHOX) to 1 for net-type Freenode · eggheads/eggdrop@1b348ed · GitHub ] - github.com
[20:40:55] <paladox>	 would be the issue, should be configurable
[20:41:47] <Voidwalker>	 didn't we also use to have +L too?
[20:42:12] <Texas>	 oh hmm
[20:42:20] <Texas>	 I may have to rebuild the eggdrop then
[20:42:38] <Voidwalker>	 we don't now, but I seem to remember that from way back whenever (back when we first got Sigyn and eir)
[20:43:00] <Texas>	 Voidwalker: How did you get eir?
[20:44:35] <Voidwalker>	 we asked nicely :P
[20:44:45] <Texas>	 ah to FN staff?
[20:45:02] <Voidwalker>	 I don't remember, but it was either NDK or John (iirc) who did that
[20:45:28] <Texas>	 oh
[20:45:43] <Texas>	 only reason I ask is I am trying to get it for -bots and -overflow
[20:45:47] <Texas>	 -overflow mainly
[20:49:35] <paladox>	 !log set cpu to host on test2 & reboot
[20:49:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[20:50:36] <paladox>	 !log upgrade libgnutls30 on test2
[20:50:37] <icinga-miraheze>	 PROBLEM - cp8 Disk Space on cp8 is WARNING: DISK WARNING - free space: / 2113 MB (10% inode=93%);
[20:50:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:06:22] * hispano76 greetings
[21:10:27] <Texas>	 hi
[21:12:49] <ZppixBot>	 Hello Luchs57! If you have any questions, feel free to ask and someone should answer soon.
[21:13:38] <Texas>	 conducive is a big word
[21:14:37] <Texas>	 .ip 51.68.141.178
[21:15:18] <ZppixBot>	 [IP/Host Lookup] Hostname: 178.ip-51-68-141.eu | Location: France | Region: Aisne | City: Chauny | ISP: AS16276 OVH SAS
[21:27:42] <Not-1447>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JfMq4
[21:27:43] <Not-1447>	 [02miraheze/ssl] 07paladox 032b58113 - Switch kkutu.wiki to Sectigo
[21:27:45] <Not-1447>	 [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:27:46] <Not-1447>	 [02ssl] 07paladox opened pull request 03#317: Switch kkutu.wiki to Sectigo - 13https://git.io/JfMqB
[21:28:37] <Not-1447>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JfMqu
[21:28:38] <Not-1447>	 [02miraheze/ssl] 07paladox 031c83e1c - Update kkutu.wiki.crt
[21:28:40] <Not-1447>	 [02ssl] 07paladox synchronize pull request 03#317: Switch kkutu.wiki to Sectigo - 13https://git.io/JfMqB
[21:28:49] <Not-1447>	 [02ssl] 07paladox closed pull request 03#317: Switch kkutu.wiki to Sectigo - 13https://git.io/JfMqB
[21:28:51] <Not-1447>	 [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JfMqg
[21:28:52] <Not-1447>	 [02miraheze/ssl] 07paladox 03409c870 - Switch kkutu.wiki to Sectigo (#317) * Switch kkutu.wiki to Sectigo * Update kkutu.wiki.crt
[21:28:54] <Not-1447>	 [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1
[21:28:55] <Not-1447>	 [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vxP9L
[21:43:06] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.96, 19.83, 16.29
[21:45:02] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 12.90, 16.96, 15.65
[21:55:38] <paladox>	 !log depool mw4, repool and also change cpu type to host
[21:55:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[21:56:17] <SPF|Cloud>	 aes-ni :D
[21:56:58] <paladox>	 SPF|Cloud done
[21:57:03] <paladox>	 (mw4 rebooted)
[21:57:13] <paladox>	 and mw4 repooled
[22:00:07] <Not-1447>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JfMmX
[22:00:09] <Not-1447>	 [02miraheze/puppet] 07paladox 037b1ae85 - mediawiki: Increase php childs to 26
[22:00:10] <Not-1447>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[22:00:12] <Not-1447>	 [02puppet] 07paladox opened pull request 03#1389: mediawiki: Increase php childs to 26 - 13https://git.io/JfMm1
[22:13:50] <paladox>	 !log depool mw5, repool and also change cpu type to host
[22:13:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:42:18] <Not-1447>	 [02puppet] 07paladox closed pull request 03#1389: mediawiki: Increase php childs to 26 - 13https://git.io/JfMm1
[22:42:20] <Not-1447>	 [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/JfMY7
[22:42:21] <Not-1447>	 [02miraheze/puppet] 07paladox 032039824 - Merge pull request #1389 from miraheze/paladox-patch-5 mediawiki: Increase php childs to 26
[22:42:23] <Not-1447>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[22:42:24] <Not-1447>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[22:43:46] <paladox>	 !log depool and repool mw[45]
[22:43:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:45:39] <paladox>	 !log depool and repool mw[67]
[22:45:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:51:09] <Not-1447>	 [02CreateWiki] 07paladox closed pull request 03#146: Make sure file write is succeeded before renaming - 13https://git.io/Jf1j8
[22:51:10] <Not-1447>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOU
[22:51:12] <Not-1447>	 [02miraheze/CreateWiki] 07paladox 035e00a8d - Make sure file write is succeeded before renaming (#146)
[22:51:13] <Not-1447>	 [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3
[22:51:15] <Not-1447>	 [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL
[22:51:39] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfMOT
[22:51:40] <Not-1447>	 [02miraheze/mediawiki] 07paladox 030b7dff5 - Update CW
[22:55:37] <paladox>	 !log rebuild lc on mw* & jobrunner1
[22:55:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[22:59:31] <icinga-miraheze>	 PROBLEM - mw7 Puppet on mw7 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[22:59:47] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOs
[22:59:48] <Not-1447>	 [02miraheze/ManageWiki] 07JohnFLewis 03ecb5489 - Convert to ternary operators for newMatrix
[22:59:54] <icinga-miraheze>	 PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[22:59:58] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:01:00] <icinga-miraheze>	 PROBLEM - test2 Puppet on test2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:03:12] <icinga-miraheze>	 PROBLEM - cloud1 Current Load on cloud1 is WARNING: WARNING - load average: 22.67, 20.91, 16.81
[23:03:15] <icinga-miraheze>	 PROBLEM - services1 APT on services1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates).
[23:03:39] <icinga-miraheze>	 RECOVERY - mw7 Puppet on mw7 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:04:01] <icinga-miraheze>	 RECOVERY - mw5 Puppet on mw5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:04:06] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[23:04:13] <Not-1447>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JfMOl
[23:04:14] <Not-1447>	 [02miraheze/CreateWiki] 07paladox 030567f40 - Add missing output to SpecialRequestWikiQueue
[23:04:16] <Not-1447>	 [02CreateWiki] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vpJTL
[23:04:17] <Not-1447>	 [02CreateWiki] 07paladox opened pull request 03#147: Add missing output to SpecialRequestWikiQueue - 13https://git.io/JfMO4
[23:05:10] <icinga-miraheze>	 RECOVERY - cloud1 Current Load on cloud1 is OK: OK - load average: 19.13, 20.08, 16.99
[23:05:18] <Not-1447>	 [02CreateWiki] 07paladox closed pull request 03#147: Add missing output to SpecialRequestWikiQueue - 13https://git.io/JfMO4
[23:05:20] <Not-1447>	 [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOu
[23:05:21] <Not-1447>	 [02miraheze/CreateWiki] 07paladox 03e832652 - Add missing output to SpecialRequestWikiQueue (#147)
[23:05:23] <Not-1447>	 [02CreateWiki] 07paladox deleted branch 03paladox-patch-3 - 13https://git.io/vpJTL
[23:05:24] <Not-1447>	 [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-3
[23:05:53] <travis-ci>	 miraheze/CreateWiki/paladox-patch-3/0567f40 - paladox The build has errored. https://travis-ci.com/miraheze/CreateWiki/builds/170118177
[23:06:08] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfMO2
[23:06:10] <Not-1447>	 [02miraheze/mediawiki] 07paladox 0399b28dd - Update CreateWiki
[23:10:32] <Not-1447>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JfMOP
[23:10:34] <Not-1447>	 [02miraheze/ManageWiki] 07paladox 035fd119d - Add missing output
[23:10:35] <Not-1447>	 [02ManageWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpSns
[23:10:37] <Not-1447>	 [02ManageWiki] 07paladox opened pull request 03#168: Add missing output - 13https://git.io/JfMOX
[23:11:18] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JfMOM
[23:11:19] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox 037f7f705 - Add missing output
[23:11:21] <Not-1447>	 [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX
[23:11:22] <Not-1447>	 [02MirahezeMagic] 07paladox opened pull request 03#126: Add missing output - 13https://git.io/JfMOD
[23:11:45] <Not-1447>	 [02ManageWiki] 07paladox closed pull request 03#168: Add missing output - 13https://git.io/JfMOX
[23:11:47] <Not-1447>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOS
[23:11:48] <Not-1447>	 [02miraheze/ManageWiki] 07paladox 03361f851 - Add missing output (#168)
[23:11:50] <Not-1447>	 [02ManageWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpSns
[23:11:51] <Not-1447>	 [02miraheze/ManageWiki] 07paladox deleted branch 03paladox-patch-1
[23:11:56] <Not-1447>	 [02MirahezeMagic] 07paladox closed pull request 03#126: Add missing output - 13https://git.io/JfMOD
[23:11:57] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMO9
[23:11:59] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox 037fbc30b - Add missing output (#126)
[23:12:00] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2
[23:12:02] <Not-1447>	 [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX
[23:12:24] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JfMOQ
[23:12:26] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox 03db5c00e - Add missing output
[23:12:27] <Not-1447>	 [02WikiDiscover] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vhUAp
[23:12:28] <travis-ci>	 miraheze/MirahezeMagic/paladox-patch-2/7f7f705 - paladox The build has errored. https://travis-ci.com/miraheze/MirahezeMagic/builds/170118482
[23:12:29] <Not-1447>	 [02WikiDiscover] 07paladox opened pull request 03#12: Add missing output - 13https://git.io/JfMO7
[23:12:46] <Not-1447>	 [02WikiDiscover] 07paladox closed pull request 03#12: Add missing output - 13https://git.io/JfMO7
[23:12:48] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOb
[23:12:49] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox 031cd4b05 - Add missing output (#12)
[23:12:53] <travis-ci>	 miraheze/MirahezeMagic/paladox-patch-2/7f7f705 - paladox The build has errored. https://travis-ci.org/miraheze/MirahezeMagic/builds/695533212
[23:13:53] <Not-1447>	 [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMON
[23:13:55] <Not-1447>	 [02miraheze/ManageWiki] 07paladox 030e6fe9a - Fix fatal
[23:14:33] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOx
[23:14:35] <Not-1447>	 [02miraheze/WikiDiscover] 07paladox 03026c758 - Fix fatal
[23:15:00] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfMOp
[23:15:01] <Not-1447>	 [02miraheze/MirahezeMagic] 07paladox 03fa233d1 - Fix fatal
[23:16:01] <Not-1447>	 [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfM3v
[23:16:02] <Not-1447>	 [02miraheze/RottenLinks] 07paladox 030efb610 - Add output
[23:17:00] <Not-1447>	 [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JfM3f
[23:17:02] <Not-1447>	 [02miraheze/IncidentReporting] 07paladox 039c66e9e - Add output
[23:18:30] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfM3U
[23:18:32] <Not-1447>	 [02miraheze/mediawiki] 07paladox 030f9f0bb - Update ManageWiki
[23:20:38] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JfM3I
[23:20:39] <Not-1447>	 [02miraheze/mediawiki] 07paladox 0348dcc6f - Update MM
[23:21:12] <Not-1447>	 [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±2] 13https://git.io/JfM3Y
[23:21:14] <Not-1447>	 [02miraheze/mediawiki] 07paladox 031078f2d - Update IR and WD
[23:21:31] <icinga-miraheze>	 PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:22:54] <paladox>	 !log rebuild lc on mw* and jobrunner1
[23:23:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:25:36] <icinga-miraheze>	 RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures
[23:27:23] <paladox>	 !log depool, repool and change cpu to host on mw7
[23:27:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:27:27] <paladox>	 !log depool, repool and change cpu to host on mw6
[23:27:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:30:01] <icinga-miraheze>	 PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw7
[23:30:12] <icinga-miraheze>	 PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core]
[23:30:45] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw7
[23:31:00] <paladox>	 !log depool, repool and change cpu to host on jobrunner1
[23:31:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[23:32:20] <icinga-miraheze>	 RECOVERY - cp7 Varnish Backends on cp7 is OK: All 7 backends are healthy
[23:32:32] <icinga-miraheze>	 RECOVERY - mw4 Puppet on mw4 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures
[23:33:13] <icinga-miraheze>	 RECOVERY - cp8 Varnish Backends on cp8 is OK: All 7 backends are healthy