[00:33:44] PROBLEM - elasticsearch2 Puppet on elasticsearch2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:35:08] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.93, 1.78, 1.06 [00:56:46] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.81, 1.98, 1.81 [01:00:46] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.91, 2.21, 1.91 [01:08:46] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.42, 1.90, 1.90 [01:10:46] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.02, 1.96, 1.92 [01:14:46] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.98, 2.00, 1.94 [01:16:46] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.63, 2.27, 2.04 [04:03:43] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.72, 2.39, 1.36 [04:05:43] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.52, 1.68, 1.22 [05:18:23] !log purge binary logs before '2019-07-19 03:00:00'; [05:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [05:22:23] PROBLEM - elasticsearch2 Current Load on elasticsearch2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:37] PROBLEM - elasticsearch2 SSH on elasticsearch2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:42] PROBLEM - Host elasticsearch2 is DOWN: PING CRITICAL - Packet loss = 100% [06:12:00] PROBLEM - elasticsearch1 Current Load on elasticsearch1 is WARNING: WARNING - load average: 3.52, 3.70, 2.66 [06:14:00] RECOVERY - elasticsearch1 Current Load on elasticsearch1 is OK: OK - load average: 2.22, 3.14, 2.58 [06:28:36] Hi o/ [06:29:04] @Dark_Dragoon hi [06:29:44] Someone knows how i can add my github account to my Phabricator profile? 👀 [06:29:45] @Dark Dragoon [06:30:02] Yes, it's in the profile settings [06:30:12] Linked Accounts [06:31:29] Sorry Settings --> External Accounts [06:34:53] Its strange but i can't find that option or maybe ins't available for me, because i not use my account 🤔 [06:39:28] Oh, I found it now. Sorry, I was a bit lost since I never used Phabricator [08:23:25] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.52, 2.70, 1.68 [08:25:25] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.65, 1.88, 1.50 [09:45:12] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj1j1 [09:45:14] [02miraheze/services] 07MirahezeSSLBot 0399be71d - BOT: Updating services config for wikis [10:03:43] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CRITICAL - load average: 2.88, 1.85, 1.20 [10:07:43] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 0.88, 1.51, 1.22 [12:12:15] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMfk [12:12:17] [02miraheze/mw-config] 07paladox 03b000397 - Switch from elasticsearch-lb to the new es-lb [12:51:47] !log forking https://github.com/wikimedia/search-extra [12:51:47] [ GitHub - wikimedia/search-extra: Github mirror of "search/extra" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing ] - github.com [12:51:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:54:56] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.59, 3.70, 2.54 [12:56:55] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.99, 3.08, 2.46 [12:57:25] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 3.56, 3.25, 2.10 [12:59:16] !log install iotop on es1 [12:59:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:59:26] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.61, 2.24, 1.87 [13:31:31] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 5.11, 3.48, 2.33 [13:37:25] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 1.40, 3.44, 2.90 [13:37:45] PROBLEM - elasticsearch1 Disk Space on elasticsearch1 is WARNING: DISK WARNING - free space: / 7674 MB (9% inode=98%); [13:39:28] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 8.44, 4.54, 3.32 [13:41:56] PROBLEM - es3 Puppet on es3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:00] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 9 minutes ago with 0 failures [14:08:25] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.38, 3.26, 2.36 [14:10:21] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 3.80, 3.25, 2.46 [14:12:16] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.64, 3.67, 2.70 [14:13:04] !log stopping es on es3 to adjust swap [14:13:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:14:13] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.42, 2.83, 2.51 [14:20:04] !log (started es on es3 a few minutes ag) [14:20:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:24:10] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 9.49, 5.70, 3.59 [14:25:25] PROBLEM - es3 Current Load on es3 is WARNING: WARNING - load average: 2.07, 2.71, 3.82 [14:26:46] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.20, 0.58, 1.86 [14:27:25] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 0.37, 1.85, 3.37 [14:28:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMTG [14:28:04] [02miraheze/puppet] 07paladox 03c71bdec - es1: Lower heap to 2gb [14:28:46] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.14, 0.42, 1.64 [14:29:06] !log stopping es on es1 [14:29:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:29:56] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 1.74, 3.44, 3.32 [14:30:18] [02ssl] 07RhinosF1 opened pull request 03#198: Change redirect - 13https://git.io/fjMTc [14:30:36] [02miraheze/puppet] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/fjMT8 [14:30:37] [02miraheze/puppet] 07paladox 03863cdcf - Revert "fix" This reverts commit 3896d9ec26eb6a3106f76b04886e203fcc4889a5. [14:30:39] [02miraheze/puppet] 07paladox 0367afe7a - Revert "elasticsearch: Add a temp config (#1041)" This reverts commit deec9eb7172c1258db635296a60f504157853f42. [14:31:26] [02ssl] 07paladox edited pull request 03#198: Change redirect - 13https://git.io/fjMTc [14:31:29] [02ssl] 07paladox closed pull request 03#198: Change redirect - 13https://git.io/fjMTc [14:31:31] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMTR [14:31:32] [02miraheze/ssl] 07RhinosF1 0336bfef7 - Change redirect (#198) Per T4533 [14:31:51] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 2.38, 3.18, 3.24 [14:32:10] Paladox: Check my PR [14:32:32] i did [14:33:02] Paladox: Thx [14:38:36] Hello [14:38:42] hi AmandaCath [14:45:13] PROBLEM - cp4 Current Load on cp4 is WARNING: WARNING - load average: 1.86, 1.62, 1.27 [14:47:07] RECOVERY - cp4 Current Load on cp4 is OK: OK - load average: 1.05, 1.35, 1.21 [14:47:54] I don't think that The Pioneer is in the channel, but thanks for +WC [14:48:59] @The Pioneer ^ [14:49:11] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:49:31] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:49:38] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:50:00] PROBLEM - Host cp3 is DOWN: PING CRITICAL - Packet loss = 100% [14:50:04] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [14:50:12] My pleasure :) Technically, I do have an account on IRC, but I usually use discord. [14:51:15] um [14:51:39] cp3 is not down to me [14:51:44] PROBLEM - es1 Current Load on es1 is CRITICAL: CRITICAL - load average: 4.56, 4.25, 3.36 [14:52:03] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [14:52:03] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 168.85 ms [14:52:07] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:07] PROBLEM - cp3 SSH on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:09] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 3% [14:52:11] RECOVERY - cp3 SSH on cp3 is OK: SSH OK - OpenSSH_7.4p1 Debian-10+deb9u6 (protocol 2.0) [14:53:24] !log adjusting swap on es1 [14:53:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:53:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMTH [14:53:56] [02miraheze/puppet] 07paladox 0304ae1fe - es1: raise heap to 4g [14:59:42] PROBLEM - es1 Current Load on es1 is WARNING: WARNING - load average: 1.95, 3.50, 3.69 [15:00:10] Reception123: I thought that the reason why CU and OS weren't given locally is because it violates the privacy policy? [15:00:22] ^ JohnLewis [15:00:47] AmandaCath: that has never been the case [15:00:48] (Just based on your reply at SN) [15:01:31] I thought that we don't assign them because they could reveal private data which, by definition, would violate the privacy policy [15:01:58] PP covers their usage as well as global policies but we've never said we blanket refuse any request. They're advanced rights where our stance is they can be requested and managed by the community on the basis of minimum appointment standards in line with Steward access [15:02:21] Sysadmins don't cover or govern their usage, Stewards and the community do [15:03:05] So wait, the global community has say over the assignment of rights on a local community? [15:03:21] Even if that local community is large enough to speak for itself? [15:03:41] RECOVERY - es1 Current Load on es1 is OK: OK - load average: 1.04, 2.32, 3.17 [15:03:52] no, local community [15:04:17] but global community have their say over the global policy [15:04:27] !log adjusting swap on es2 [15:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:04:40] So the global policy is "needs local community consensus" [15:04:44] ? [15:05:05] yes, which matches the requirements for Steward [15:06:20] So in a way, the global community is stating what the requirements are for a local community (so an individual community can't set their own standards as to what qualifies as "consensus") [15:06:45] a local community can, but they can't relax the requirement [15:07:13] * AmandaCath shrugs [15:07:45] I remember the global policy being "local community consensus and prove that the stewards can't handle it in a reasonable time" [15:08:10] it used to be, then we changed it when we realised the latter is not measurable [15:08:40] Yeah, defining "reasonable time" is difficult and unrealistic [15:09:01] an active attack with no steward online immediate shows "unreasonable time" but likewise, no one informing a steward also meets that definition which puts fault on a Steward and not the community [15:09:25] True [15:10:06] plus we much rather prefer global cooperation than local [15:10:34] why have 20 local CUs when 6 stewards covering the globe can do the same, if not a better job [15:10:56] Yeah, and there is also no need IMO for CU/OS on individual wikis [15:11:06] so far any requests passed on to Stewards have been done quickly [15:11:24] I don't recall there being a time where requests have waited too long [15:11:47] Fortunately [15:12:04] Although I'd predict that as Miraheze grows, so will disruptive editing [15:12:23] and as it does, so I hope Stewards [15:12:24] if there was a wiki where let's say PI was leaked every single day multiple times, maybe then the question of a local OS could be raised [15:12:33] but as John says, Stewards are better at dealing with such things [15:13:05] I'm more talking about sock puppets on larger wikis [15:13:20] That's more likely to occur than constant PII leakage [15:13:53] AmandaCath: and why could those not be investigated by Stewards? [15:14:11] a separate page called CheckUser Requests could be made if they got out of hand [15:14:47] They could be now [15:14:50] That's a good thing [15:15:07] But like I said, as Miraheze grows, so will the disruption [15:15:17] and as John said, so will the Stewards (hopefully) [15:15:30] If we end up with a community as large as a WMF project... [15:15:31] why prioritize local rights over global rights? [15:15:57] The stewards over at WMF only step in when there are no local right holders [15:16:18] When Miraheze is big enough, I could picture something similar [15:16:24] AmandaCath: well in that case, a user will definitely have enough support to fill the minimum requirements for a Steward and therefore get the rights (however if that happens the requirements will be greater I'm sure) [15:16:46] but I don't think there's any need to speculate the future, we'll have to see how things go [15:16:54] though for the time being, I think the current policy works [15:17:51] IMHO there should be a separate "CheckUser and Oversight Test Wiki" somewhere [15:17:59] Maybe something I'll do in all my spare time [15:18:16] (Not hosted by Miraheze, of course) [15:19:09] AmandaCath: the thing is, it's hard to do a testwiki like that because of the access to the very information that should be private [15:19:19] There was (not sure if it still exists) a testwiki where you could CU yourself [15:19:44] yeah, the wiki run by Skizzerz [15:19:55] I think it's pretty much gone at this point [15:20:00] (Dormant) [15:20:45] Although, with a dedicated CU/OS test wiki, it would be a wiki where nothing really mattered in terms of content, and users would register simply to test the tools l [15:21:22] AmandaCath: yes, though with CU users would have access to other users' IPs, which no one wants [15:21:42] and OS, if there was a PI leakage there would be no way of oversighting it, since everyone would have access [15:21:58] + there isn't really a need to test OS, it isn't that different from admin revdel in terms of how to do it [15:22:07] A privacy policy could include "by registering, you agree that your IP address etc etc may be seen by others, but they should not publicly share it without permission" [15:22:53] well I don't think there would be any clients then [15:23:07] Or alternatively, set up the wiki to automatically spoof everyone's IP [15:23:22] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 7.20, 4.57, 2.78 [15:23:23] (which is possible from a software and technical perspective) [15:23:42] AmandaCath: that would be quite some work, with not much effect. but feel free to set it up :) [15:23:51] by a PP such as that, you'd have to blacklist the EU [15:24:33] oh, didn't think about that [15:24:35] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [15:24:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkG [15:24:38] [02miraheze/puppet] 07paladox 0374563d0 - elasticsearch: Enable tls connections [15:24:40] [02puppet] 07paladox opened pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:25:02] Reception123: it is possible and I know how because I've done it for official purposes. Unfortunately the how-to of that is currently classified within Canada so I can't give more details [15:25:19] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:25:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkn [15:25:22] [02miraheze/puppet] 07paladox 035380aed - Update elasticsearch.pp [15:26:27] spoofing software level isn't really hard at all anyway [15:27:16] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 1.15, 3.36, 2.75 [15:27:33] Well, we have a certain way of doing it but it's been classified along with several other things since the nationwide cyberattack a few years ago [15:29:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkW [15:29:31] [02miraheze/puppet] 07paladox 0350a16a1 - Update es1.yaml [15:29:33] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:29:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkl [15:29:57] [02miraheze/puppet] 07paladox 03c65c35f - Update common.yaml [15:30:01] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:30:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMk8 [15:30:11] [02miraheze/puppet] 07paladox 03f1403fd - Update es1.yaml [15:30:13] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:30:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMk4 [15:30:22] [02miraheze/puppet] 07paladox 03c48446b - Update es2.yaml [15:30:24] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:30:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkB [15:30:32] [02miraheze/puppet] 07paladox 03fe02b68 - Update es3.yaml [15:30:33] [02puppet] 07paladox synchronize pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:34:07] [02puppet] 07paladox closed pull request 03#1042: elasticsearch: Enable tls connections - 13https://git.io/fjMkZ [15:34:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/fjMkV [15:34:10] [02miraheze/puppet] 07paladox 03f248164 - elasticsearch: Enable tls connections (#1042) * elasticsearch: Enable tls connections * Update elasticsearch.pp * Update es1.yaml * Update common.yaml * Update es1.yaml * Update es2.yaml * Update es3.yaml [15:40:27] !log restarting elasticsearch on es* (includes stopping and starting) [15:40:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:41:41] paladox: I feel like that is a direct definition of restarting :P [15:44:06] paladox: heh https://publictestwiki.com/w/index.php?title=Test/en&mobileaction=toggle_view_mobile works [15:44:07] [ Test/en - TestWiki ] - publictestwiki.com [15:44:17] paladox: so the issue only happens on some wikis (like Meta) [15:44:19] JohnLewis haha [15:44:24] Reception123 oh [15:44:27] yeh [15:44:33] also works on att.. [15:44:39] paladox: so what's different on Meta then? [15:44:51] there's nothing in the logs so it doesn't seem to be an error [15:45:04] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [15:45:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [15:46:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMkM [15:46:31] [02miraheze/puppet] 07paladox 030a985c5 - fix [15:50:22] Reception123 i'm not, that lies the problem i guess :) [15:50:55] paladox: I really don't see why it's happening on some wikis and not others [15:57:38] [02puppet] 07paladox created branch 03paladox-patch-7 - 13https://git.io/vbiAS [15:57:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMkp [15:57:41] [02miraheze/puppet] 07paladox 0323b631d - Update elasticsearch.pp [15:57:43] [02puppet] 07paladox opened pull request 03#1043: Update elasticsearch.pp - 13https://git.io/fjMkh [15:59:43] [02puppet] 07paladox synchronize pull request 03#1043: Update elasticsearch.pp - 13https://git.io/fjMkh [15:59:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-7 [+0/-0/±1] 13https://git.io/fjMIf [15:59:46] [02miraheze/puppet] 07paladox 0396c0dcc - Update elasticsearch.pp [15:59:56] [02puppet] 07paladox edited pull request 03#1043: Add elasticsearch user to ssl-cert group - 13https://git.io/fjMkh [16:11:26] PROBLEM - es1 Puppet on es1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [16:16:56] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://git.io/vbiAS [16:16:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+1/-0/±0] 13https://git.io/fjMIl [16:16:59] [02miraheze/puppet] 07paladox 03b539522 - Add ssl::elasticsearch_wildcard [16:17:01] [02puppet] 07paladox opened pull request 03#1044: Add ssl::elasticsearch_wildcard - 13https://git.io/fjMI8 [16:22:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIu [16:22:46] [02miraheze/puppet] 07paladox 0316705f9 - Update elasticsearch.pp [16:22:47] [02puppet] 07paladox synchronize pull request 03#1044: Add ssl::elasticsearch_wildcard - 13https://git.io/fjMI8 [16:23:19] [02puppet] 07paladox synchronize pull request 03#1044: Add ssl::elasticsearch_wildcard - 13https://git.io/fjMI8 [16:23:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIg [16:23:22] [02miraheze/puppet] 07paladox 03b252846 - Update elasticsearch.pp [16:24:10] paladox: would investment not have been better suited to build on ssl::wildcard? [16:24:46] Nope, not unless we want dupes inside that class? [16:24:53] huh? [16:25:08] Since for nginx we need it in /etc/ssl but for es we need it in /etc/elasticsearch//ssl/ [16:25:16] yeah... [16:25:27] so build on the existing, don't dupe it [16:25:35] config exists for a reason [16:25:35] JohnLewis: you're a steward right, look at meta rc and the disregard wiki thing, think some wikis might want cleaning up. [16:26:27] JohnLewis yeh, but i doin't see a way to do it in wildcard without duplicating? [16:26:59] paladox: you don't see a way of doing puppet params, something you've done extensively? [16:27:11] RF1|pingonreply: I'll take a look in a bit [16:27:29] JohnLewis uh, i do, i more mean creating it in both place. [16:27:43] do you need it in both places? [16:27:46] yup [16:27:54] why? [16:27:55] es master needs it for nginx && es needs it [16:28:21] JohnLewis: I can't see a history of a wiki request so I might need a sysadmin to look and check if 8715 won't be blocked by the mess [16:28:27] and you can't configure nginx to use it from the es location? [16:29:24] RF1|pingonreply: can you create 8715? [16:30:06] I guess i could, but will nginx be able to read the file if it has elasticsearch? Or maybe i can use ssl-cert [16:30:07] JohnLewis: no, It already exisits but with the wrong information and disregardwiki now also exisits [16:30:27] RF1|pingonreply: close it as a dupe please [16:30:47] JohnLewis: okay, can u move the crat over or something? [16:32:20] RF1|pingonreply: yeah already done [16:34:18] JohnLewis: can you drop disregardwiki [16:34:50] yeah later [16:36:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIi [16:36:08] [02miraheze/puppet] 07paladox 03e35b560 - Update wildcard.pp [16:36:10] [02puppet] 07paladox synchronize pull request 03#1044: Add ssl::elasticsearch_wildcard - 13https://git.io/fjMI8 [16:36:25] JohnLewis: thx [16:39:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMID [16:39:09] [02miraheze/puppet] 07paladox 03e504019 - Update elasticsearch.pp [16:39:10] [02puppet] 07paladox synchronize pull request 03#1044: Add ssl::elasticsearch_wildcard - 13https://git.io/fjMI8 [16:39:27] [02puppet] 07paladox edited pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:45:10] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:45:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMId [16:45:13] [02miraheze/puppet] 07paladox 0341e7b51 - Update wildcard.pp [16:45:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMIF [16:45:16] [02miraheze/services] 07MirahezeSSLBot 0317cd39a - BOT: Updating services config for wikis [16:45:24] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:45:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-1/±0] 13https://git.io/fjMIb [16:45:27] [02miraheze/puppet] 07paladox 039f3665e - Delete elasticsearch_wildcard.pp [16:47:41] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:47:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIp [16:47:44] [02miraheze/puppet] 07paladox 03e87ff16 - Update elasticsearch.pp [16:47:50] JohnLewis: is title blacklist not on miraheze mw? [16:48:24] it's global [16:48:40] JohnLewis: so should work using local pages [16:48:41] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:48:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIh [16:48:44] [02miraheze/puppet] 07paladox 03a1b8f68 - Update wildcard.pp [16:49:17] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:49:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMIj [16:49:20] [02miraheze/puppet] 07paladox 03741a11c - Update elasticsearch.pp [16:49:51] yes [16:51:20] JohnLewis: thx [16:51:52] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:51:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMLJ [16:51:55] [02miraheze/puppet] 07paladox 035b2363c - Update elasticsearch.pp [16:53:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+1/-1/±0] 13https://git.io/fjMLL [16:53:29] [02miraheze/puppet] 07paladox 03929db14 - Update and rename modules/role/files/elasticsearch/nginx-site.conf to modules/role/templates/elasticsearch/nginx-site.conf.erb [16:53:31] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:56:37] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [16:56:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMLt [16:56:40] [02miraheze/puppet] 07paladox 032e62db2 - Update wildcard.pp [16:57:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMLq [16:57:07] [02miraheze/puppet] 07paladox 0399b12d0 - Update elasticsearch.pp [16:57:08] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [17:00:03] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [17:00:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjML3 [17:00:06] [02miraheze/puppet] 07paladox 03db99c38 - Update wildcard.pp [17:02:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://git.io/fjMLG [17:02:03] [02miraheze/puppet] 07paladox 0389e3cc7 - Update elasticsearch.pp [17:02:05] [02puppet] 07paladox synchronize pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [17:03:22] [02puppet] 07paladox closed pull request 03#1044: Modify ssl::wildcard to support setting paths - 13https://git.io/fjMI8 [17:03:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±2] 13https://git.io/fjMLc [17:03:25] [02miraheze/puppet] 07paladox 03e3a80e1 - Modify ssl::wildcard to support setting paths (#1044) [17:04:47] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [17:04:49] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://git.io/vbiAS [17:05:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMLW [17:05:06] [02miraheze/puppet] 07paladox 03a737c7a - Fix syntax [17:05:25] PROBLEM - es1 Puppet on es1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:07:25] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:10:20] PROBLEM - es2 Puppet on es2 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[GlobalSign.crt] [17:15:32] PROBLEM - es3 Puppet on es3 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[GlobalSign.crt] [17:15:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMLw [17:15:37] [02miraheze/puppet] 07paladox 033b9e719 - Create "/etc/elasticsearch/${es_instance}/ssl" directory [17:16:10] PROBLEM - elasticsearch1 Puppet on elasticsearch1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[nginx-syntax],File[wildcard.miraheze.org],File[wildcard.miraheze.org_private],File[GlobalSign.crt] [17:16:21] RECOVERY - es2 Puppet on es2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:17:34] RECOVERY - es3 Puppet on es3 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:24:10] RECOVERY - elasticsearch1 Puppet on elasticsearch1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:27:26] PROBLEM - es1 Puppet on es1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:03:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMtK [18:03:35] [02miraheze/puppet] 07paladox 033731599 - elasticsearch: Tweek config [18:09:35] PROBLEM - es1 Puppet on es1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:11:31] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [18:13:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMt5 [18:13:21] [02miraheze/puppet] 07paladox 03c2da634 - Update elasticsearch.pp [18:15:04] PROBLEM - es3 Current Load on es3 is CRITICAL: CRITICAL - load average: 6.18, 3.50, 1.63 [18:17:01] RECOVERY - es3 Current Load on es3 is OK: OK - load average: 2.48, 3.10, 1.72 [18:20:14] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMqe [18:20:15] [02miraheze/services] 07MirahezeSSLBot 034e8672e - BOT: Updating services config for wikis [18:33:26] PROBLEM - es1 Puppet on es1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 seconds ago with 0 failures [18:38:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMqs [18:38:09] [02miraheze/puppet] 07paladox 0307156f6 - fix [18:41:44] PROBLEM - es3 Puppet on es3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:42:34] PROBLEM - es2 Puppet on es2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:45:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMql [18:45:14] [02miraheze/services] 07MirahezeSSLBot 0366ff3d6 - BOT: Updating services config for wikis [19:06:30] PROBLEM - es2 Puppet on es2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:27] PROBLEM - es2 Puppet on es2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 36 minutes ago with 0 failures [19:12:30] PROBLEM - es2 Puppet on es2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:31] PROBLEM - es2 Puppet on es2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 42 minutes ago with 0 failures [19:14:58] Paladox: https://phabricator.miraheze.org/T4556 -- You seem to be the search expert [19:14:59] [ ⚓ T4556 Search on AllTheTropes is completely nonfunctional ] - phabricator.miraheze.org [19:15:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/fjMqN [19:15:21] [02miraheze/mw-config] 07Reception123 03ca9622a - note that Tweeki causes mobile view errors [19:15:23] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [19:15:24] [02mw-config] 07Reception123 opened pull request 03#2701: note that Tweeki causes mobile view errors - 13https://git.io/fjMqA [19:15:35] ^ paladox [19:16:05] [02mw-config] 07paladox closed pull request 03#2701: note that Tweeki causes mobile view errors - 13https://git.io/fjMqA [19:16:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMqp [19:16:08] [02miraheze/mw-config] 07Reception123 03c55f8cc - note that Tweeki causes mobile view errors (#2701) [19:16:13] RF1|pingonreply ok [19:16:15] :) [19:16:32] paladox: I've filed a task upstream since the skin is up to date for us, so definitely some issue there [19:16:38] ok [19:16:54] Paladox: Mind adding an update to the task - it's same on meta [19:17:22] paladox: also, the 'help' thing you wrote for VisualEditor doesn't display for some reason [19:22:46] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [19:22:48] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [19:29:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMmt [19:29:15] [02miraheze/puppet] 07paladox 03a9c3923 - Update elasticsearch.pp [19:30:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjMm3 [19:30:50] [02miraheze/puppet] 07paladox 03579589a - Update nginx-site.conf.erb [19:30:55] [02puppet] 07paladox closed pull request 03#1043: Add elasticsearch user to ssl-cert group - 13https://git.io/fjMkh [19:30:59] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-7 [19:31:00] [02puppet] 07paladox deleted branch 03paladox-patch-7 - 13https://git.io/vbiAS [19:31:04] [02puppet] 07paladox deleted branch 03paladox-patch-6 - 13https://git.io/vbiAS [19:31:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-6 [19:31:07] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [19:31:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [19:31:10] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:31:12] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:31:13] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [19:31:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [19:33:26] RECOVERY - es1 Puppet on es1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [19:34:54] !log purge binary logs before '2019-07-19 17:00:00'; [19:34:59]