[02:37:52] hi all. is there any chance we get dotnet core installed on the web grid? I'm so sick of mono bugs [12:46:42] !log admin shutdown toolsbeta-sgegrid-master (cronspam) [12:46:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:30:16] !log tools draining and cordoning tools-worker-1027 for a region migration test [17:30:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:01:33] !log tools pushed updated docker images [19:01:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:14:23] !log tools.zppixbot Restarting bot for config change [19:14:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL [19:25:21] I'm unable to ssh into a cloud vps instance using agent forwarding [19:35:00] Zppix, disable agent forwarding? [19:35:29] * chicocvenancio laughs [19:35:46] It wasn't a joke. [19:35:48] Zppix: what error are you getting? [19:36:18] Krenair: do you mean as a way to solve the not getting in problem or as a safety prective? [19:36:28] both [19:36:37] s/prective/practice/ [19:37:10] Krenair: are we not allowing logins with agent forwarding now? [19:38:19] not sure, I haven't dared try [19:38:35] it still works [19:38:54] * chicocvenancio agrees about the security practice [19:39:41] even though I do dare :) [19:39:45] we don't allow it in prod IIRC [19:40:22] bblack: we definitely shouldn't [19:42:05] I use ssh -a [19:42:11] not -A [19:43:03] my laugh was because, though correct, disabling agent forwarding is probably not going to solve the connection problem [19:44:55] didn't think so either but Zppix thought it was relevant so [19:45:16] figured since some hosts will very much not appreciate agent forwarding, try without [19:45:28] chicocvenancio: I'm just getting public key error [19:45:36] (sorry for my delay in replying) [19:45:54] you're getting an error with agent forwarding and not without? [19:46:17] chicocvenancio: I followed https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_using_agent_forwarding [19:47:06] Zppix: did you try the proxyjump method? its a bit simpler and safer [19:48:14] chicocvenancio: ill try it now [19:51:13] Zppix, is the public key error coming from the bastion or the host you're trying to get to? [20:04:48] Krenair: im able to get into bastion [20:04:58] Krenair: but not the instance in the project im wanting [20:05:19] Zppix: what steps/command are you using? [20:05:44] chicocvenancio: i was following the agent forwarding commands (ive not tried proxyjump yet) [20:15:30] chicocvenancio: Krenair i got agent forwarding to work [20:19:51] !log tools.indic-ocr Rebuild virtualenv and restart webservice [20:19:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-ocr/SAL [20:21:32] Zppix, which instance? [20:21:34] (assuming you still have issues) [20:21:57] Krenair: Nope i managed to get in... i switched from tera term to putty [20:51:33] !log tools reboot tools-package-builder-02 (unresponsive) [20:51:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:50:59] !log deployment-prep restart logstash on deployment-logstash2 while hacking around to see why apifeatureusage doesn't work [22:51:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [22:52:25] !log delete logstash logs in /var/log/logstash generated prior to 2019 [22:52:25] ebernhardson: Unknown project "delete" [22:52:31] !log deployment-prep delete logstash logs in /var/log/logstash generated prior to 2019 [22:52:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [23:27:57] The stretch isn't quite stable. [23:29:44] Just received a report that IABot was down. When I went to check I found the webservice was not started. It seems bigbrother is not monitoring my webservice [23:35:50] Cyberpower678, your tool was shut down deliberately IIRC [23:35:58] Why? [23:36:29] And more importantly, why was I not informed? [23:37:45] excessive resource usage [23:37:46] "It seems bigbrother is not monitoring my webservice" -- I don't think that is exactly possible. Did you check the error.log and service.log? [23:37:48] https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Tools.iabot/SAL&diff=1817267&oldid=1809380 [23:38:36] oh... iabot was the one that got caught filling NFS with log files [23:38:37] it wrote many hundreds of gigs to log files [23:38:42] over NFS [23:38:43] yes [23:39:55] * bd808 sees 3.8G access.log file and remembers we have still not solved that problem [23:40:06] the impact of having to clean this up caused issues for other tools too [23:41:35] Krenair: so IABot errored out? That's unusual. I note that I recently moved to Stretch. [23:42:00] This was not an issue when it ran on Trusty [23:42:05] in fact not just other tools, but toolforge infra itself [23:42:31] truncated log files were left in place to assist debugging [23:42:38] IABot is PHP 7 compatible [23:42:45] Anyone know how to reset 2FA on horizon.wikimedia.org? It should accept the same tokens i use on wikitech, but it doesn't want to. I tried resetting my 2FA on wikitech anyways, but that didn't help [23:43:23] ebernhardson: it is the same tokens (and actually processed by Wikitech via API) [23:43:27] Krenair: I see a shit load of deprecated warning messages in the error log [23:43:40] The problem is, I think this might be a PHP bug [23:43:52] do we know if wikitech and horizon have the same amount of tolerance for inaccurate clocks? [23:44:18] yes, because wikitech actually does all the validation [23:45:00] my knowledge of how it works is probably out of date. didn't keystone/horizon's plugin just deal with the wikitech DB directly? [23:45:16] bd808: hmm, horizon sure claims i can't log in :S [23:45:28] but if it's asking mw .. .hmm [23:45:38] Krenair: is the latest PHP version installed on the Stretch VMs? [23:45:40] Krenair: it did, but we fixed that soon after I built the API so that Striker could enforce 2FA [23:45:47] Cyberpower678, I have no idea, I am not a tools admin [23:46:18] Cyberpower678: "the latest", no. The current PHP 7.2 that is being used in Wikimedia's production wikis, yes. [23:46:24] I rarely touch tools unless I need to interact with the DB replicas or some shared tool or something [23:46:41] are there maybe logs somewhere that say why it's rejecting me? Maybe it's completely unrelated to 2FA [23:46:51] ebernhardson: let me see if I can spot anything useful in logs [23:47:40] bd808: The error flood is coming from PHP. It's telling me that idn_to_ascii is using INTL_IDNA_VARIANT_2003 which is deprecated. Only problem is, I'm not passing in a variant, so it should be using the defaults. [23:48:06] INTL_IDNA_VARIANT_UTS46 is the default according to the PHP docs [23:48:43] Cyberpower678: "7.2.0 INTL_IDNA_VARIANT_2003 has been deprecated; use INTL_IDNA_VARIANT_UTS46 instead. " [23:49:02] the new default you are expecting is not in 7.2, but 7.4 [23:49:05] bd808: The default is INTL_IDNA_VARIANT_UTS46 according to the docs [23:49:11] bd808: oh [23:49:54] You're right. That means I have to modify Wikimedia's CheckIfDead algo. [23:53:24] bd808: Krenair: Looking at the logs, from what I can tell, this deprecation warning is what flooded the logs and drove the size to NFS exhaustion. [23:53:47] seems quite possible [23:54:08] * Cyberpower678 works on patching wikimedia/checkIfDead [23:54:35] ebernhardson: the log message is less than useful :/ '2019-02-26 23:49:39.120542 Login failed for user "ebernhardson", remote address 10.64.0.130.' [23:54:52] so we logged an auth failure and the ip of the varnish cluster :) [23:55:14] nice :) [23:57:51] ebernhardson: just for grins, try using "EBernhardson" as your username when authenticating to Horizon [23:59:10] bd808: no luck, i also tried Ebernhardson earlier which is what wikitech shows