[00:45:09] PROBLEM - Puppet errors on tools-exec-1410 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [00:53:02] 10Labs, 10Tool-Labs: Install python3-tk package on ToolLabs - https://phabricator.wikimedia.org/T166998#3313937 (10bd808) The `matplotlib` python library seems to be install globally for python2, but I get the same error about missing bindings for libtk: ``` $ which python /usr/bin/python $ python Python 2.7.6... [01:20:08] RECOVERY - Puppet errors on tools-exec-1410 is OK: OK: Less than 1.00% above the threshold [0.0] [01:57:38] 10Labs, 10Tool-Labs: Install python3-tk package on ToolLabs - https://phabricator.wikimedia.org/T166998#3313965 (10Facenapalm) As far as I remember, I've never used virtualenv on tool labs, and bash history proves it. But "import matplotlib.pyplot" line cause no exceptions in python2. On the other hand, I stil... [02:11:23] 10Tool-Labs-tools-Xtools: Adminstats isn't labelling administrators properly - https://phabricator.wikimedia.org/T154408#3313973 (10Matthewrbowker) Not yet, you may track the progress at T153112 [02:32:13] PROBLEM - Free space - all mounts on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1417.diskspace._tmp.byte_percentfree (<55.56%) [03:37:08] PROBLEM - Puppet errors on tools-exec-1409 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [04:07:09] RECOVERY - Puppet errors on tools-exec-1409 is OK: OK: Less than 1.00% above the threshold [0.0] [04:49:09] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Rewrite Edit Counter with new backend architecture - https://phabricator.wikimedia.org/T160481#3314047 (10Samwilson) This is done, e.g. http://tools.wmflabs.org/xtools-dev/ec/en.wikipedia.org/samwilson It's still pretty slow, but that will be fixed via {T1632... [08:02:31] 10Labs, 10Tool-Labs: Install python3-tk package on ToolLabs - https://phabricator.wikimedia.org/T166998#3313937 (10valhallasw) https://matplotlib.org/faq/usage_faq.html#what-is-a-backend documents on how to select a backend (you probably want to use 'AGG' or 'PDF'). Make sure to select one //before// importing... [11:09:26] 10Labs, 10Tool-Labs: Install python3-tk package on ToolLabs - https://phabricator.wikimedia.org/T166998#3314463 (10Facenapalm) 05Open>03Resolved a:03Facenapalm Many thanks! `matplotlib.use('AGG')` makes it works in both python2 and python3. I'm sorry for worrying you, should've googled it manually. [14:04:33] 10Labs, 10Horizon, 10Operations, 10Patch-For-Review, 10Puppet: Puppet tab in Horizon unusably slow - https://phabricator.wikimedia.org/T149589#3314826 (10Volans) To add some data here, I'm getting very slow responses when opening an instance page, like `https://horizon.wikimedia.org/project/instances/edb... [14:39:02] 10Labs, 10Tool-Labs: Add bash completion to sql - https://phabricator.wikimedia.org/T167026#3314951 (10Dispenser) [14:56:20] huh ? [14:56:52] -labs isn't canonical anymore ? [14:57:23] WTF?? [14:57:30] :D [14:57:45] more or less my thoughts :d [14:58:03] Why is -labs invite only now? [14:58:11] Are we renaming labs to the cloud? [14:58:18] What [14:58:23] is going on here. [14:58:24] ^^' [14:58:43] bd808 is currently opped, so he must be responsible. [14:59:04] * Alphos will read later [14:59:27] Cyberpower678: yes we renaming -labs to -cloud. https://wikitech.wikimedia.org/wiki/User:BryanDavis/Rebranding_Cloud_Services_products [14:59:47] announced on labs-l weeks ago [15:00:01] not on the ML :/ [15:00:27] (i'm not) [15:00:43] Alphos: now is a good time to sign up! :) -- https://lists.wikimedia.org/mailman/listinfo/labs-l [15:00:49] bd808: I don't recall getting such an email. [15:01:21] https://lists.wikimedia.org/pipermail/labs-l/2017-May/005002.html [15:01:23] bd808: Not that I mind renaming stuff, but can you provide a short explanation why? [15:01:33] https://wikitech.wikimedia.org/wiki/User:BryanDavis/Rebranding_Cloud_Services_products [15:01:43] https://lists.wikimedia.org/pipermail/labs-l/2017-May/005002.html [15:02:55] bd808: when will the domain migration take place? [15:04:29] Cyberpower678: there is currently no planned domain migration. [15:04:44] An easier approach may be to select a new TLD/subdomain as part of the Grid Engine replacement project and migrate tools from https://tools.wmflabs.org/$TOOL/ to https://$TOOL.$NEWTLD/ as they move from the current proxies to the new system. [15:05:01] So I assumed some form of new domain structure is going to happen. [15:05:19] "If we do change it, ...." [15:05:31] weird. I got kicked? [15:05:32] Fortunately, my newest tools are built to be movable to any location and still work. [15:05:44] oh! -cloud! [15:05:45] it probably will happen, but even starting down that road is months away [15:05:48] * Cyberpower678 gives halfak a trout. [15:05:49] halfak: channel renaming and operator migration [15:06:13] in fairness "You have been kicked" is not the most informative thing to be told ;P [15:06:31] * bd808 blames 20 year old protocols [15:06:38] halfak: I gave you the trout. I didn't smack you with it. bd808 is doing the kicking. :p [15:06:40] ;) [15:06:48] Oh! I gotchat [15:06:56] * halfak wields trout [15:07:19] * Cyberpower678 goes to reconfigure his bouncer. [15:17:49] (03PS1) 10BryanDavis: Remove obsolete #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357222 (https://phabricator.wikimedia.org/T166420) [16:34:59] anyone have an idea of how this page used to be used? -- https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Tips [16:35:16] Nope [16:35:54] IIRC it was used to provide tips on various projects [16:36:12] back on toolsserver [16:36:53] last edit before I just touched it was 2013-09-16 so at least ancient history. [16:37:26] I think its okay to archive? [16:37:50] I mean if someone wants it back im sure they will ask [16:37:50] I'm just going to delete it [16:38:43] Sounds good to me :P [16:58:02] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Rewrite Edit Counter with new backend architecture - https://phabricator.wikimedia.org/T160481#3315590 (10MusikAnimal) Interface is a little off, but we will fix that :) I've been quietly observing updates to the backend code and I'm definitely a fan. +1 from me! [16:58:05] (03PS6) 10Jean-Frédéric: Proof of concept to harvest Wikidata into monuments database [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) [16:59:45] (03CR) 10Jean-Frédéric: "Hurrah!" [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/354961 (https://phabricator.wikimedia.org/T165988) (owner: 10Jean-Frédéric) [17:22:06] 10Labs, 10wikitech.wikimedia.org, 10Patch-For-Review: Can we search namespaces on wikitech? - https://phabricator.wikimedia.org/T165725#3315726 (10Andrew) 05Open>03declined ok, let's leave things as they are for now -- I'll re-open if I can find a good example where it's needed. [18:11:19] 10Labs, 10wikitech.wikimedia.org: Can't login to wikitech.wikimedia.org - https://phabricator.wikimedia.org/T167055#3315864 (10Jhernandez) [18:14:32] PROBLEM - High iowait on tools-worker-1006 is CRITICAL: CRITICAL: tools.tools-worker-1006.cpu.total.iowait (>10.00%) [18:18:07] 10Labs, 10wikitech.wikimedia.org, 10User-bd808: Can't login to wikitech.wikimedia.org - https://phabricator.wikimedia.org/T167055#3315908 (10bd808) p:05Triage>03Normal a:03bd808 @Jhernandez the easiest way to fix this is by following the guidance [[https://wikitech.wikimedia.org/wiki/Password_reset#Res... [18:24:31] RECOVERY - High iowait on tools-worker-1006 is OK: OK: All targets OK [18:27:27] 10Wikibugs: Tag detection is broken - https://phabricator.wikimedia.org/T166951#3312649 (10Krinkle) I think this broke the color detection. As of last few days, all tags are blue on IRC regardless of the tag's actual color. {F8383301 height=157} | {F8383299} | {F8383300} [18:27:34] 10Wikibugs: Tag detection is broken - https://phabricator.wikimedia.org/T166951#3315940 (10Krinkle) 05Resolved>03Open [19:08:01] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10kubernetes: Install request on ToolLabs inside Kubernetes containers for DjVuLibre and XPDF packages - https://phabricator.wikimedia.org/T166985#3316109 (10Xelgen) a:05yuvipanda>03None [19:10:03] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10kubernetes: Install request on ToolLabs inside Kubernetes containers for DjVuLibre and XPDF packages - https://phabricator.wikimedia.org/T166985#3316127 (10zhuyifei1999) [19:10:06] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10Tracking: Packages to be installed in Tool Labs Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110#3316128 (10zhuyifei1999) [19:10:33] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10kubernetes: Install request on ToolLabs inside Kubernetes containers for DjVuLibre and XPDF packages - https://phabricator.wikimedia.org/T166985#3313559 (10zhuyifei1999) Can't it be installed via pip? [19:20:49] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316143 (10Krinkle) [19:21:21] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3313559 (10Krinkle) [19:24:15] hmmm. telnet.wmflabs.org seems down :( [19:24:36] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316145 (10Xelgen) >>! In T166985#3316127, @zhuyifei1999 wrote: > Can't it be installed via pip? No, as pip is only for Python packages. And those a... [19:26:11] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316151 (10zhuyifei1999) (oh I thought you meant install the python `requests` package) [19:27:12] cscott ----^ [19:30:45] 10Labs: Neutron networking, with IPv6 at eqiad - https://phabricator.wikimedia.org/T85611#3316174 (10chasemp) [19:30:48] 10Labs, 10Tool-Labs: Distributing tools, deployment-prep to both data centers (availability/redundancy) - https://phabricator.wikimedia.org/T85610#950707 (10chasemp) [19:30:51] 10Labs: Labs available in the new data centre (with Neutron/IPv6) - https://phabricator.wikimedia.org/T85609#3316170 (10chasemp) 05Open>03declined With T85610 also being declined I'm going to say any work towards this end is a ways off and will be tracked in other tasks [19:31:24] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 2 others: Labs test cluster in codfw - https://phabricator.wikimedia.org/T114435#3316187 (10chasemp) [19:31:28] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, 10Operations: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#3316184 (10chasemp) 05Open>03declined For now this is totally off the books [19:33:49] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, 10Operations: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#1753690 (10Dzahn) If this is totally off the books, can we remove the existing remnants? [19:41:50] >>> len('https://www.mediawiki.org/wiki/CoC') [19:41:50] 34 [19:41:50] >>> len('https://phabricator.wikimedia.org/u/conduct') [19:41:50] 43 [19:42:11] legoktm: :) [19:42:21] if we only had an URL shortener... [19:42:30] I can't edit the topic :( [19:42:53] ugh. did we end up with +t here? [19:44:20] it was set as you founded the channel IIRC [19:44:48] *nod* that makes some sense [19:44:50] i think it's default [19:45:08] and it was not send at the old chan, so until it is needed, we can remove it I think [19:47:20] !log telnet sudo service wikipedia-telnet start [19:47:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet/SAL [19:49:20] the list of packages installed in toollabs.. i remember seeing it but now i don't .. it's in puppet, right [19:50:02] mutante: yeah there's a "packages" resource in there somewhere [19:51:18] mutante: ::toollabs::exec_environ [19:51:20] hmm. searching. i am suprised how packages already installed don't "grep -r" in the entire repo [19:51:22] bd808: can you throw an example !help in here, testing notice words [19:51:23] thanks! [19:51:43] !help I'd like to know how to use ssh [19:51:43] bd808: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [19:52:07] that did not throw a ping like I want hm [19:53:24] oh, wait it did but I'm being obtuse [19:53:24] ok [19:54:30] !help how can I reset my 2fa credentials if I lost my phone? [19:54:31] anonbd808: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [19:55:03] anonbd808: that worked fine thanks, name even had me going for a second :) [19:55:12] bd808: For the telnet thing, I'm still getting connection denied. I think you might also have to run setup-firewall command based on docs at https://meta.wikimedia.org/wiki/Telnet_gateway [19:55:15] if there a !help 2fa [19:55:31] !log tools disabling puppet on tools-proxy-01 and -02 for a staged rollout of https://gerrit.wikimedia.org/r/#/c/350494/16 [19:55:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:55:42] anonbd808: for completeness -- https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication [19:55:54] !log project-proxy disabling puppet on novaproxy-01 and -02 for a staged rollout of https://gerrit.wikimedia.org/r/#/c/350494/16 [19:55:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [19:56:29] andrewbogott: good luck [19:57:42] !log telnet Ran setup-firewall per https://meta.wikimedia.org/wiki/Telnet_gateway; service back up and running [19:57:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet/SAL [19:59:32] bd808: Wee, thanks :) [20:00:07] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3313559 (10Dzahn) Packages get installed via Puppet. The list of packages installed in toollabs, on exec nodes, is: `puppet/modules/toollabs/manife... [20:00:36] bawolff: it seems to be busted. All page fetches are failing [20:00:47] yeah. Oh well [20:02:53] I bet its still using the old rest.wikimedia.org endpoint which no longer exists [20:03:26] logs say -- Retrying (1) https://en.wikipedia.org/w/api.php?action=sitematrix&format=json [Error: CERT_UNTRUSTED] [20:10:38] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316324 (10chasemp) p:05Triage>03Normal [20:13:09] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316339 (10zhuyifei1999) The k8s packages are installed via Dockerfiles at rODIT. [20:14:49] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3313559 (10bd808) >>! In T166985#3316302, @Dzahn wrote: > Question is though how the currently installed packages are not listed there. The package... [20:15:37] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3313559 (10chasemp) @dzahn I believe this user is in the k8s environment which is handled separately from SGE I imagine the current state is explain... [20:16:36] we really need a PaaS layer so that people can build custom images with apt packages they need in the Kubernetes cluster. :/ [20:16:57] bloating the shared images is not a great long term solution [20:17:21] 10Labs, 10Tool-Labs, 10Tools-Kubernetes, 10cloud-services-team (Kanban): Install DjVuLibre and XPDF packages for Kubernetes containers on Tool Labs - https://phabricator.wikimedia.org/T166985#3316363 (10chasemp) [20:17:41] true on both accounts [20:17:55] bloating the shared worker node is equally undesirable to me too [20:18:02] but some amount of it atm is inevitable [20:18:25] bd808: how do you feel about me adding a "to be discussed" column? [20:18:29] the grid nodes are a lots cause. we will never clean them up [20:18:29] is to-do equally good? [20:18:39] *lost cause [20:18:40] i'm getting "504 Gateway Time-out" on https://paws.wmflabs.org/ [20:18:42] I meant the k8s workers [20:18:45] but yeah [20:18:52] bd808: grid just needs to go bye bye and go full on k8s [20:19:12] ...is this a general issue? [20:19:34] HaeB: let me look and see if i do as well [20:19:46] chasemp: having a column to track things we need to bring up in team meetings is a good idea. Do you think that goes in the kanban board or the team board? [20:19:51] HaeB: it could be there is a brief but in theory not impactful maint in progress [20:19:52] i got a 502 bad gateway chasemp [20:20:07] andrewbogott: ^ we are seeing bad gateway and general proxy errors possibly [20:20:37] https://tools.wmflabs.org/openstack-browser/ works tho [20:21:00] let me try again [20:21:02] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316387 (10Bawolff) [20:21:18] I think its just PAWS [20:21:19] bd808: I guess I was thinking the kanban board was The Board https://phabricator.wikimedia.org/project/board/2774/ [20:21:20] nope still 502 [20:21:33] bd808: still users getting 2 different errors is odd [20:21:39] chasemp: really? I haven't merged anything yet as far as I know... [20:21:51] andrewbogott: maybe false alarm [20:22:20] chasemp: there is this board too -- https://phabricator.wikimedia.org/project/board/2773/ -- but I'm ok wiht the kanban board being the main one for things like this [20:23:18] madhuvishy: are you around to look at PAWS and see why its sad? [20:23:48] i reloaded https://paws.wmflabs.org/ and now get "502 Bad Gateway" (instead of 504) too [20:23:54] bd808: let's briefly talk about the two boards and difference tomorrow and decide as a group? I'm not sure I totally understand teh current working theory [20:23:58] bd808: yes i can look [20:24:09] tx madhuvishy [20:24:12] chasemp: sounds like a plan. [20:24:22] madhuvishy: cc chasemp bd808 incase its needed im using google chrome latest stable and on android os (Latest stable) [20:24:37] Zppix: its something busted on the backend, but thanks [20:24:53] bd808: hey you never know chrome does weird things [20:33:58] 10Labs, 10Labs-Infrastructure, 10DBA: Decide whether back-compat views for upcoming major schema changes will be provided in the Labs replicas - https://phabricator.wikimedia.org/T166798#3316420 (10bd808) > I don't know if there are any statistics as to how many tools would be broken by these changes, or if... [20:37:50] uhh it's not very happy [20:38:11] madhuvishy: maybe it just needs love [20:51:36] irc is very unhappy with my bouncer atm [20:52:02] chasemp: how so? [20:52:24] keep getting randomly disconnected [20:56:27] chasemp: what bouncer and how are you running it? [20:59:48] 10Labs-project-Wikistats: New wikistats interface takes minutes to load the mediawikis list - https://phabricator.wikimedia.org/T167066#3316475 (10Nemo_bis) [21:01:19] 10Labs-project-Wikistats: New wikistats interface takes minutes to load the mediawikis list - https://phabricator.wikimedia.org/T167066#3316490 (10Dzahn) a:03Dzahn [21:02:36] 10Labs-project-Wikistats: Restore direct links in new wikistats interface - https://phabricator.wikimedia.org/T167067#3316509 (10Nemo_bis) [21:45:42] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316627 (10cscott) Perhaps this has been already fixed? https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet says @bd808 did some maintenance on the host today... [21:48:39] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316387 (10Andrew) ``` >>> banana Sorry! Could not fetch "banana" for you. No worries. There are lots of other pages to read. Pick a different title. >>> waffle Sorry! Could not fetch "waffle" for... [21:49:53] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316639 (10bd808) >>! In T167064#3316627, @cscott wrote: > Perhaps this has been already fixed? https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet says @bd808 did some maintenance on the host... [21:50:45] Krinkle: in your feedback on the renaming proposal, do you have a specific reason for not liking the camel case ToolForge name? [22:08:11] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316687 (10cscott) Hm: ``` cscott@telnet2:/srv/wikipedia-telnet$ node --version v0.10.25 ``` We could hotpatch around this as described at https://thequietinnovator.com/2015/10/11/how-to-fix-nodejs-... [22:12:10] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316693 (10cscott) The ugliest way is: ``` export NODE_TLS_REJECT_UNAUTHORIZED=0 ``` before launching the telnet service (thanks to https://stackoverflow.com/questions/41390965/cert-untrusted-error-... [22:15:22] !log tools Deleted tools.aibot crontab that somehow got locally installed on tools-exec-1436 on 2017-05-24T20:55Z [22:15:23] 10Labs: nova-fullstack is losing instances on creation - https://phabricator.wikimedia.org/T165555#3316697 (10Andrew) | c52c756e-b64e-42f1-8c54-679beec4cebd | fullstackd-1496674006 | ACTIVE | public=10.68.19.139 | Leaked, and ssh fails. Ldap errors as below. | 4f17a862-dcc4-4336-a544-1331fb7306a5 | fullstackd... [22:15:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:16:14] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316698 (10bd808) Building a new VM based on the Debian Jessie image should get you access to nodejs v6.9.1 from the Wikimedia apt repo. [22:20:11] !log telnet Hacked /etc/init/wikipedia-telnet.conf to include "env NODE_TLS_REJECT_UNAUTHORIZED=0" for T167064 and restarted service [22:20:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Telnet/SAL [22:20:13] T167064: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064 [22:21:26] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3316724 (10bd808) The `NODE_TLS_REJECT_UNAUTHORIZED=0` hack seems to work: ``` $ telnet telnet.wmflabs.org Trying 208.80.155.160... Connected to telnet.wmflabs.org. Escape character is '^]'.... [22:32:08] 10Labs, 10wikitech.wikimedia.org, 10MW-1.30-release-notes (WMF-deploy-2017-05-30_(1.30.0-wmf.3)), 10Patch-For-Review: Disable/redirect instance instance management on wikitech - https://phabricator.wikimedia.org/T164875#3316739 (10Andrew) 05Open>03Resolved [22:33:12] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, 10Operations: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#3316744 (10Andrew) Subbu is still using Prometheum. We have half a plan to clean that up but in the meantime we'll need to keep some... [22:55:59] the start of that log line really got my attention. "< bd808> !log telnet Hacked" :) [23:05:11] O_O [23:08:55] mutante: lol. Didn't mean to give you a fit with that :) [23:09:24] bd808: stop hacking its illegal xD [23:09:27] "strategically altered un-puppeted code" [23:12:54] 10Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3316865 (10MacFan4000) p:05Triage>03Normal [23:21:26] hehehe [23:44:00] !log tools Deleted tools.iabot crontab that somehow got locally installed on tools-exec-1412 on 2017-05-24T20:55Z [23:44:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL