[01:16:15] Wіth ⲟur ІᖇC ad ser⋁ice yoᥙ can reаch а glоbaⅼ ɑuԁiᥱnce ⲟf ᥱntrеprеneurs and feᥒtaᥒyl adⅾiсts witһ extrаorԁinarу enɡɑgement rɑtеs! һttpѕ://wіlⅼⅰampitcⲟϲk․cоⅿ/ [01:53:41] Rеаⅾ ᴡһat IᎡC inᴠestiɡаtivе jourᥒаⅼіѕts haᴠе ᥙncovered оᥒ thе frеᥱnode pеⅾoⲣһilⅰa ѕⅽаndaⅼ https://eᥒcуⅽlopeⅾiɑԁramɑticɑ．rs/ᖴreeᥒodеgate [01:54:30] І tһഠught уoᥙ gᥙуѕ ⅿiɡһt be iᥒtеrᥱѕteⅾ іᥒ thіs blog by freеᥒodе ѕtɑff ⅿember Bryan klοеri Ostergɑard һttⲣs፡/⁄brỿаnoѕtᥱrgаard.com⁄ [06:05:19] Ꭱeaԁ ᴡһat IᎡⲤ іnvestіɡatiᴠᥱ jourᥒaⅼiѕtѕ haᴠe uncovered οn the freеnഠde pеdഠⲣhiⅼia sⅽaᥒdaⅼ httpѕ։⧸/eᥒcуϲlopеԁiadraⅿɑtіca․rs∕Frᥱeᥒodеɡɑte [07:36:54] !log codesearch restarted to pick up new config / repository lists [07:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [09:08:42] Witһ οur IᖇC ad ѕеrviϲe уou ϲаᥒ rᥱаcһ a glഠbаl audiencе of еntreprenᥱurѕ ɑnd fᥱᥒtɑᥒyl aԁԁіcts ᴡith extraordiᥒɑrỿ ᥱngɑɡᥱment ratᥱs! һttps:/⧸williaⅿpіtсock.com／ [09:57:04] A fɑsciᥒɑting bloɡ ᴡһеre freᥱnⲟԁe staff ⅿeⅿber Matthew ⅿѕt Тrout rеϲοᥙntѕ hⅰs ᥱхрerіᥱnceѕ of eyе╴raрing уoung chiⅼdrеn httрѕ⠆//ΜattSΤrout.ⅽoⅿ⁄ [10:37:09] Ꮃith оᥙr ΙRC аd ѕerᴠіϲe yoᥙ caᥒ reɑⅽһ a ɡⅼobal aᥙdiᥱnⅽᥱ of entreprenᥱ∪rs anⅾ feᥒtanyl ɑdԁⅰⅽtѕ with еxtrɑorⅾinarу eᥒgagemeᥒt rates! httрs:∕᜵wiⅼlⅰɑmріtcock.com/ [10:37:09] I tһouɡһt уou ɡuуs miɡht be intᥱrᥱsted in tһіs blоg bу frеenode staff mᥱⅿbеr Bryan klοeri Оѕtergaɑrԁ httрs:∕⧸bryaᥒostergaɑrd.cοm/ [10:44:06] I tһouɡht you guys might bᥱ іnterested in thіs blοg bу freeᥒⲟdе ѕtaff ⅿember Ⲃrуaᥒ kⅼοeri Ostergaаrd https://brуаnoѕterɡaɑrd．coⅿ⁄ [11:41:10] Wіtһ οur IRϹ ad ѕervice уoᥙ ϲaᥒ rᥱach ɑ ɡⅼοbal ɑudіеncе ഠf ᥱntreⲣrеnᥱurs and feᥒtaᥒyⅼ aⅾԁiⅽts wⅰth extrаⲟrdiᥒary ᥱᥒgagеⅿeᥒt rаtes! https://wіllіaⅿріtcock.com/ [11:57:33] !log tools T205078 purge packages smbclient libsmbclient libwbclient0 python-samba samba-common samba-libs from trusty machines [11:57:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:57:39] T205078: toolforge: round of package upgrades and cleanups - https://phabricator.wikimedia.org/T205078 [12:12:07] !log tools T205078 upgrade trusty-wikimedia packages (git-fat, debmonitor) [12:12:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:12:13] T205078: toolforge: round of package upgrades and cleanups - https://phabricator.wikimedia.org/T205078 [12:13:30] Rᥱad wһat IᎡC iᥒ∨ᥱѕtigɑtі∨е ϳⲟurᥒаliѕtѕ һave ᥙnϲο∨ᥱreԁ on tһе freeᥒⲟde pedophіlіa scаᥒԁaⅼ httрs:⧸⁄enсуcⅼoⲣeԁiаdrаmatica.rѕ⧸Frеenoⅾegate [12:14:51] !log tools T205078 same for {jessie,stretch}-wikimedia [12:14:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:35:39] !log tools cleanup stalled apt preference files (pinning) in tools-clushmaster-01 [12:35:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:28:04] І tһⲟugһt you guys ⅿigһt bе ⅰnterestеd iᥒ this blog bỿ freenоde stɑff mᥱⅿber Ᏼryan kloeri Οsterɡaаrd һttps∶⁄∕brỿanostеrɡaɑrd．coⅿ/ [13:28:04] Ꮃіtһ oᥙr ІᖇС aԁ sеrvicᥱ you cаn rеаϲһ a ɡⅼⲟbɑl auԁiencе οf еᥒtreрreᥒeᥙrs and fentaᥒуl аdԁісtѕ witһ ᥱⲭtraordinаry eᥒgagement rateѕǃ һttрs﹕⁄/ᴡіlliampitcഠϲk.coⅿ⧸ [13:32:05] hello, I'm trying to install a python module on virtualenv, but i'm getting an error with setuptools version: "cryptography requires setuptools 18.5 or newer, please upgrade to a " any idea on how to install the module? [13:34:46] alchimista: upgrade pip [13:35:02] i.e. pip install -U pip [13:41:54] !log shinken Created shinken-02 for T204562 [13:41:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [13:41:58] T204562: cloudvps: shinken project trusty deprecation - https://phabricator.wikimedia.org/T204562 [13:49:41] Krenair i wonder if we should do that in neutron? [13:50:05] actually ignore me, the old network will view neutron as a floating ip :) [13:50:10] paladox, not yet [13:50:24] there's also the problem of all the old security groups likely not letting us in [13:50:37] yeh [13:51:06] but yes [14:37:52] Ꭺ faѕciᥒɑting bⅼоg ᴡhᥱre freеnode staff membеr Matthеw ⅿѕt Trout rесⲟᥙᥒts his ехperiеnceѕ οf eye˗rаpіnɡ young cһiⅼdren һttрs︓//MattSTrοut．сⲟⅿ/ [15:00:15] !log wmcs-nfs Installed backported packages to nfs server and demonstrated that it can be made to work by following a procedure to be documented in T203254 [15:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmcs-nfs/SAL [15:00:19] T203254: labstore1004 and labstore1005 high load issues following upgrades - https://phabricator.wikimedia.org/T203254 [15:09:19] zhuyifei1999_, worked, thanks [15:11:38] :) [15:16:30] I went into horizon and removed my own membership in a project. It worked, but unexpectedly kicked me out to the login screen. I guess because it was still trying to use that project as the "active" project. Logging back in did work. [15:17:49] anomie, yeah I think horizon does that when you try to have a session in a project where you're not a member [15:19:07] I'd expected it to either pick a new active project or say "you have no active project, go select one", rather than logging me out. Just mentioning it here in case someone wants to treat that as a bug. [15:21:14] anomie: It's dumb but it has to do with how tokens are handled — you have a project-scoped token and as soon as Horizon sees that your token is invalid it kicks you. It turns out to be hard to work around :( [15:36:01] https://bugs.launchpad.net/horizon/+filebug [16:17:34] Ι tһouɡht yoᥙ ɡuys ⅿigһt bе intereѕted iᥒ this blog by freenoԁe stɑff ⅿеmber Brỿan kloeri Оѕterɡɑɑrd һttpѕ:∕/bryanostᥱrɡɑard．cоm/ [16:25:53] Ꮃith our IᎡϹ ad servіce yοᥙ cɑᥒ rᥱɑсh a global audіеᥒcᥱ of еntrерrеᥒеurs aᥒd fentɑnỿl аԁdicts wіtһ еxtraordinarу еnɡaɡеⅿᥱnt ratᥱs! httⲣѕ:⁄/ᴡilliɑmрⅰtϲock․ⅽoⅿ/ [16:26:27] Ι thouɡһt уഠᥙ gᥙys might be interᥱѕtеԁ іᥒ tһiѕ blοɡ by freеnodᥱ staff ⅿᥱmber Bryan klоеri Oѕtergaard https:⁄/brỿаᥒostᥱrgаard.сom/ [16:31:37] (03PS1) 10Urbanecm: Add a workaround symlink to static in src [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461989 [16:32:32] (03PS1) 10Urbanecm: Switch to Flask-JSONLocal as localization library [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461990 [16:32:50] (03CR) 10Urbanecm: [C: 032] Add a workaround symlink to static in src [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461989 (owner: 10Urbanecm) [16:33:00] (03CR) 10Urbanecm: [C: 032] Switch to Flask-JSONLocal as localization library [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461990 (owner: 10Urbanecm) [16:33:11] (03Merged) 10jenkins-bot: Add a workaround symlink to static in src [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461989 (owner: 10Urbanecm) [16:33:22] (03Merged) 10jenkins-bot: Switch to Flask-JSONLocal as localization library [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461990 (owner: 10Urbanecm) [16:43:05] mutante: If it's not handling active traffic, I'll move relic-stretch over to the new region now. [16:43:08] It'll be down while it copies [16:43:37] andrewbogott: i think it does [16:43:43] just very low traffic [16:43:57] oh, -stretch is already the active host? [16:44:00] anyone who is still trying to use the old toolserver.org url or receive email [16:44:12] oh, no, sorry it is not. misunderstood [16:44:28] the old one is still the active host [16:44:38] great, I'll move the new one, then we can move traffic over there [16:44:49] ok :) [16:45:22] I'm also going to merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/461992/ which will (mostly) prevent VMs from being created in the old region [16:45:45] (03PS1) 10Urbanecm: Add message documentation [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461994 (https://phabricator.wikimedia.org/T205119) [16:45:51] gotcha, ok [16:45:53] (03CR) 10jerkins-bot: [V: 04-1] Add message documentation [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461994 (https://phabricator.wikimedia.org/T205119) (owner: 10Urbanecm) [16:46:00] so i need to amend my change with a new IP address, right [16:46:08] (03PS2) 10Urbanecm: Add message documentation [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461994 (https://phabricator.wikimedia.org/T205119) [16:46:22] (03CR) 10Urbanecm: [C: 032] Add message documentation [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461994 (https://phabricator.wikimedia.org/T205119) (owner: 10Urbanecm) [16:46:44] (03Merged) 10jenkins-bot: Add message documentation [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/461994 (https://phabricator.wikimedia.org/T205119) (owner: 10Urbanecm) [16:47:24] (03CR) 10D3r1ck01: "recheck" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/459263 (owner: 10D3r1ck01) [16:47:38] mutante: because floating IPs are region-specific, it will wind up with a different floating IP. That probably means we need DNS changes someplace... [16:48:19] andrewbogott: if that is in Horizon in the DNS settings of that project.. i looked at them. .and yes [16:48:27] there is that fun name :) [16:48:40] things.point.here.something DNS name :) [16:48:59] so far i thought i dont have to touch it though because only the public IP is in there [16:49:25] i understand..region-spefic.. ok [16:49:28] when it comes up, legacy-stretch will have a private IP of 172.16.1.226 and a public IP of 185.15.56.23 [16:49:35] um, sorry, relic-stretch [16:50:00] ok, let me amend my gerrit change [16:53:20] udated message of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455737/ [16:53:29] so then: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/455737/4/hieradata/eqiad/profile/openstack/main/nova/network.yaml [16:53:40] "relic" in this context is not the instance name but the proxy name.. if i get it right [16:53:54] so we are not adding relic-stretch here. we are changing what the relic proxy is [16:54:11] and then in DNS .. hold on ... [16:54:18] not positive but I think you're right [16:54:56] (03CR) 10Merlijn van Deen: [C: 032] Add GCI 2018 work-board to #wikimedia-dev-africa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/459263 (owner: 10D3r1ck01) [16:55:19] have to boot my phone which is down.. to get Google authenticator.. to login on Horizon with 2fa :p just a sec [16:55:32] (03Merged) 10jenkins-bot: Add GCI 2018 work-board to #wikimedia-dev-africa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/459263 (owner: 10D3r1ck01) [16:55:43] (03CR) 10jenkins-bot: Add GCI 2018 work-board to #wikimedia-dev-africa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/459263 (owner: 10D3r1ck01) [16:56:50] mutante: that VM should be back up and reachable, let me know if it looks weird to you [16:56:57] ok, so i am in Horizon in DNS settings for the project [16:57:11] other than host key warning [16:57:12] it has 2 zones. toolserver-legacy.wmflabs.org. and toolserver.org [16:57:41] in the first one we have an A record, that's the funny one: [16:57:52] um, hang on, apache won't start on the VM now [16:57:54] did it run before? [16:58:02] !log tools.wikibugs Updated channels.yaml to: 59937c1b4d3c31cd15c802831117b1a4c43c6b35 Add GCI 2018 work-board to #wikimedia-dev-africa [16:58:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [16:58:06] yea [16:58:27] AH00526: Syntax error on line 47 of /etc/apache2/sites-enabled/50-www-toolserver-org.conf [16:59:04] I'm not clear on how this ever worked, seems like an apache version mismatch [16:59:49] apache 2.2 vs apache 2.4 [16:59:50] oh, it's missing the SSL module ? looking [17:00:09] yea, that actually not too surprising, what paladox says [17:00:23] i thought it was running though, hadnt noticed [17:02:47] Invalid command 'SSLEngine', .. patch coming [17:05:49] !log relic relic-stretcH: a2enmod ssl; starting apache [17:05:49] mutante: Unknown project "relic" [17:06:07] !log toolserver-legacy: relic-stretch: a2enmod ssl; starting apache [17:06:07] mutante: Unknown project "toolserver-legacy:" [17:06:22] !log toolserver-legacy relic-stretch: a2enmod ssl; starting apache [17:06:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [17:06:32] andrewbogott: it's running and puppetizing it now [17:07:01] paladox: actually no other error , just not having ssl module enabled [17:07:07] oh [17:07:08] not the usual 2.2 issues [17:07:16] well, it's running [17:07:43] is it actually surprising we setup SSL while behind the proxy ? [17:08:11] heh [17:08:23] mutante you need ssl for floating ips [17:08:45] ah, ok. i'm thinking like misc-web prod [17:10:20] well, this also means the apache could not have started before on trusty [17:10:24] if we had rebooted that [17:10:46] or the module was pulled in by default on trusty or manually installed [17:10:49] do you have some test URLS to verify that things actually work? [17:10:58] toolserver.org ? [17:11:19] oh, does it only ever display the 'we moved' page? Not deep-link redirs? [17:11:54] hmm, it does this: [17:11:55] RewriteRule ^/(.*)$ https://www.toolserver.org/$1 [L,NE,R] [17:11:56] (I'm about to be on the phone — you have the DNS stuff figured out right? It's just search/replace all the IPs on there) [17:12:01] no, i don't [17:12:06] well maybe [17:12:50] ok [17:14:03] Hi, I'm running into some errors on Toolforge that seem to relate to my exceeding the limit of the number of processes that I can run [17:14:33] `ulimit -u` reports 30; `ps ax | wc -l` reports 270 [17:15:06] I don't believe that I am running 270 processes, but not being a Linux native I don't really understand exactly what these numbers are telling me [17:15:45] Perhaps the real question I should be asking is "I need to run a node.js app on Toolforge, what is the best way to do this"? [17:17:25] !help (Sorry I should have read the welcome message first!) [17:17:26] Smith609: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [17:17:27] Smith609: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#node.js_web_services is probably the best information [17:17:51] Ooh, that does look helpful, thank you [17:18:54] !log admin Running `sudo maintain-meta_p --all-databases --purge` across labsdb10(09|10|11) for T201890 [17:18:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:18:58] T201890: Update data in `meta_p.wiki` in Cloud Services for foundationwiki move - https://phabricator.wikimedia.org/T201890 [17:20:27] Is it possible to run a nodejs webserver from a tool account that is simultaneously running a PHP webserver, also using kubernetes? [17:21:20] Smith609: not today, no. Are the php and node components really the same tool? Or are you putting multiple things into a single tool account? [17:21:42] The PHP script will make a call to the node script [17:22:21] do both serve the web (listening under tools.wmflabs.org)? [17:22:27] The node script is Zotero's Translation Server that extracts metadata from a given URL. My PHP script is a Wikipedia bot that will extract URLs from a webpage and enhance them with metadata where appropriate [17:22:45] So the node.js script does not need to serve the web, only the PHP script [17:23:21] so you want PHP to shell out to node? Or the node process is a web service that you want the PHP to call? [17:23:31] The latter [17:23:45] *nod* I would setup 2 tool accounts in that case [17:23:57] I suspect that it would be possible for the PHP to call on localhost rather than the web per se [17:24:06] Okay, I'll do that, thankyou. [17:24:14] mutante: ok, I'm off the phone, want me to start moving those DNS settings or did you already? [17:24:28] (I assume this should be a graceful transition since we can leave the old instance up for a day) [17:25:20] andrewbogott: i just merged a fix for the apache config on stretch, there is only issue now.. the letsencrypt cert generation.. but that should fix itself once we switch DNS [17:25:50] andrewbogott: i haven't made the changes in Horizon yet.. i wanted to show you that setup there [17:25:58] if you already see it. go ahead [17:26:16] things-point-at-this-ip.toolserver-legacy.wmflabs.org. points to 208.80.155.197 [17:26:44] and then there is the second zone for toolserver.org [17:27:26] hm, 'submit' button is greyed out for me, I wonder what that's about [17:27:32] same for you? [17:27:38] if you 'update' the record [17:28:54] Type and Name are greyed out [17:29:01] but "Records" below is not [17:29:03] that seems fine... [17:29:08] what about the submit button? [17:29:29] oh yea, greyed out [17:29:41] ok — I'm going to do this via the cli and then I'll figure out what's up with Horizon [17:29:51] "This was imported from ldap where it was associated with instance in project toolserver-legacy" [17:29:57] maybe that means we have to change in LDAP/ [17:30:08] I don't think so [17:30:12] ok [17:30:13] no [17:30:19] just means it was imported from there originally [17:30:31] is it possible it was set as a managed record or something? [17:31:55] searching for inurl:toolserver.org in google to get test URLs, i see there was wiki.toolserver.org as well [17:32:25] 50-www-toolserver-org.conf: # Redirect wiki.toolserver.org to mediawiki.org. [17:32:37] so we do that too [17:32:57] wiki.toolserver.org has address 10.68.16.162 [17:33:02] that's relic (old) [17:33:51] status.toolserver.org has address 10.68.16.162 [17:35:21] mutante: dns is updated; you can merge your puppet patch now and then I think that's it [17:36:12] i am making the list of test urls .. and ok! :) [17:37:23] starting here.. https://phabricator.wikimedia.org/P7580 [17:37:27] merging [17:37:48] it's greyed out [17:37:57] because you have to enter a tty value [17:38:07] sorry i just skimmed so maybe missed something :) [17:38:20] mutante andrewbogott ^^ [17:39:06] oh you're right [17:39:14] huh, I wonder why it doesn't populate with the existing setting [17:39:21] enter "1" and that will fix it :) [17:40:13] hehe, confirmed. that was a bit odd [17:40:34] i do see the new IP in web UI [17:40:54] without actually clicking update.. but it wasnt greyed out anymore [17:41:49] so the Letsencrypt cert generation still fails for now [17:41:57] i ran the command thta puppet runs manually for debugging [17:42:05] copy the cert from the old machine [17:42:07] to the new one [17:42:09] it couldn't download the challenge file [17:42:20] yeh [17:42:22] mutante: for reasons I don't quite understand, the toolserver.org is in prod dns as well. I'll make a patch for that [17:42:32] andrewbogott: oh! good catch, thank you [17:43:02] paladox: ok, will do. but doesnt acme usually work ? [17:43:24] mutante nope, i doin't think [17:43:27] maybe i just have to wait longer [17:43:31] toolserver.org has address 10.68.16.162 [17:43:33] i had this problem too [17:43:38] ok [17:44:01] https://gerrit.wikimedia.org/r/#/c/operations/dns/+/462005/ [17:45:32] +1 ed, they both point to relay.toolserver.org [17:46:31] ok, applied — I guess it'll be an hour before anything happens now, so I'll probably go eat lunch :) [17:46:58] ok, i'm copying the cert over [17:50:52] !log toolserver-legacy scp toolserver.org Letsencrypt key and cert from relic to relic-stretch instance for migration [17:50:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [17:51:22] paladox: when you copied it .. what happened later when it was time for renew? [17:51:33] it should be able to renew it [17:51:37] did the renews work from puppet then ? [17:51:40] ok [17:51:56] did you copy over the acme stuff too [17:52:07] ie the keys it generates for acme to confirm the domain [17:52:14] i copied the key and the files in /cert/ [17:52:21] but not more [17:52:42] you may need to also copy the other thing "acme key" [17:52:48] not "acct" and the request in csr [17:52:52] thats some where in /var/www (i think) [17:53:05] if you mean /etc/acme/key/ then i did [17:53:11] ah yep [17:53:51] Info: Applying configuration version '1537552413' [17:53:51] Notice: /Stage[main]/Toolserver_legacy/Letsencrypt::Cert::Integrated[toolserver]/Exec[acme-setup-acme-toolserver]/returns: executed successfully [17:53:54] yay :) [17:54:38] wow, lots of activity in access.log now [17:54:48] Dalvik/2.1.0 user agent [17:55:00] I see that kubernetes is running node 6.11; I need node >8.0.0. Is the best way around this to install node myself? [17:55:14] GEThttp://c.www.toolserver.org/tiles/osm-labels-en/ [17:55:27] he is downloading osm-labels-en .. aa lot of them .. from the toolserver.org URLs [17:55:30] so ..works [17:56:36] both relic and relic-stretch are currently serving stuff [18:01:00] andrewbogott: thanks for the DNS part, i wasnt sure about that setup. it looks like we are done. i will now confirm the email part as well. also enjoy the lunch! [18:01:21] paladox: i'll do the same test as last night.. exim [18:01:26] ok [18:02:01] mutante: great! Thanks for paying attention to that weird instance :) [18:03:10] oops.. and we have an issue with the email part [18:03:16] "The mapping of aliases is in /etc/toolserver.aliases" [18:03:44] we dont have that yet on new host [18:04:04] that is probably not in puppet because it contains email addresses of people [18:04:08] copying it [18:06:06] !log toolserver-legacy scp /etc/toolserver.aliases from relic to relic-stretch (exim legacy aliases for @toolserver.org mail address forwarding), not puppetized, contains personal email addresses [18:06:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [18:06:47] paladox: can you mail paladox@ and dzahn@ toolserver [18:07:18] ok [18:07:33] done [18:07:47] it arrived on the new box :) and noting in exim log on old box .. nice [18:07:52] did you get your reply ? [18:08:08] heh [18:08:15] oh. you did not .. right [18:08:17] and nope [18:08:28] SMTP error from remote mail server after RCPT TO [18:08:30] sigh [18:08:44] oh [18:08:57] it's not allowed to do it [18:08:58] 550 Relay not permitted [18:09:03] the old one as [18:09:05] was [18:09:44] i got my copy because i am in wikimedia.org [18:10:00] oh [18:12:54] "# This version to be used as relay-only for a file of aliases [18:13:25] yea, it has that puppetized config on both [18:36:11] it's an ACL on prod MX servers that have a list of wikimedia networks [18:36:21] and the new eqiad-r network isnt in it yet [18:36:34] herron helped debugging [18:37:06] we need to add 172.16.0.0 or something that matches 172.16.1.226/21 [18:37:35] ..or just that one instance [18:38:50] hostlist relay_from_hosts = <; @[] ; 127.0.0.1 ; ::1 ; <%= scope.lookupvar('network::constants::all_networks').join(" ; ") %> [18:39:39] is it not in all_networks then? hmm [18:40:31] A fascinating bⅼoɡ ᴡһere frеeᥒoԁe staff member Mattһew mѕt Trout rеϲഠunts hiѕ ᥱxⲣеrіеnceѕ of еye-rɑpіᥒg yഠuᥒg chіlԁrеᥒ httⲣs:᜵／ΜattЅᎢrout．coⅿ/ [18:46:25] is eqiad-r network range 172.16.1.0/21 ? [18:47:54] in a ticket i found the phrase "The future use of 172.16/12 space" [19:08:33] andrewbogott: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462012/ [19:10:34] mutante: intenrnal IPS for eqiad1-r are 172.16.0.0/21 [19:10:40] *internal [19:11:05] andrewbogott: yep, thanks. i found it https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations [19:11:30] do we want to allow the entire (eqiad) network [19:12:51] looking at https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462012/4/modules/network/data/data.yaml maybe "external network" isn't the best choice [19:13:10] maybe it should be further down as new labs networks then [19:13:31] mutante: I'm confused about whether we're talking about internal IPs or floating IPs. I don't see anything about the internal IP range from eqiad in that file... [19:14:24] well it already has this: [19:14:25] cloud-instances2-b-eqiad: [19:14:25] ipv4: 172.16.0.0/21 [19:14:43] but the production exim config doesnt seem to honor that [19:14:52] andrewbogott: internal IPs [19:15:25] I thougһt you guуѕ ⅿіgһt be intereѕted iᥒ this bⅼഠɡ by freenodе ѕtaff ⅿember Bryan kⅼoeri Ostᥱrɡaarԁ https:∕/bryɑnostergɑarԁ.cοm⁄ [19:15:59] mutante: so, my question is — how did this work before? I'd expect to see a reference to the old internal IP range someplace, and for you to be adding the new range in the same place [19:16:02] andrewbogott: so we looked at production exim puppet code and it gets a list of "relay_from_host" networks [19:16:04] Maybe I'm overlooking something obvious [19:17:37] ok, soo.. it starts in exim config, we care about the "relay_from_host" line [19:17:45] in puppet source that means [19:17:47] hostlist relay_from_hosts = <; @[] ; 127.0.0.1 ; ::1 ; <%= scope.lookupvar('network::constants::all_networks').join(" ; ") %> [19:17:57] so we go to check "all_networks" right [19:18:28] $all_networks = flatten([$external_networks, '10.0.0.0/8']) [19:18:45] now this uses external_networks and 10.0.0.0/8 [19:18:49] the latter matches relic (old) [19:18:55] ah, right, because it's 10.x it gets picked up by accident by the regular internal range :/ [19:18:55] but both dont match relic (new) [19:19:26] so then we look at external_networks and get [19:19:26] $external_networks = $network_data['network::external'] [19:19:43] that gets us $network_data = loadyaml("${module_path}/data/data.yaml") [19:19:45] (lol) [19:19:57] and now we arrive in that data.yaml i am changing [19:20:17] and i add it to external_networks because.. above .. but that may also not be right [19:20:29] yeah — so it looks like your patch will fix the exim thing but I have no idea what 'external_networks' means elsewhere in the code, so I worry it will open up security holes elsewhere that we don't want [19:20:33] because clearly it should be more like "allow labs range" [19:20:55] So yeah, either add an explicit thing for cloud VMs or let's wait and see what other SREs think about the proposed patch [19:21:13] i would love to just wait for revies [19:21:19] but then .. we broke it [19:22:15] touching the exim config itself rather than this seems almost more invasive [19:22:49] brandon or arzhel might have opinions, let's ping them [19:22:57] meanwhile I'll grep and see if I can see where else that's used [19:23:08] i also want to know what "external" means elsewhere. yea... [19:23:51] I mean, it won't open anything that isn't already open for eqiad [19:23:53] we could add it to "all_networks" [19:24:04] that sounds like it cant be bad.. "all" :p [19:24:11] and it would also fix it [19:24:25] surely it's already in all_networks? [19:24:33] mind if I ping some folks in _security about this? [19:24:39] no,it;s not [19:24:41] because all_networks is just "external" plus 10.0.0./8 now [19:24:49] not at all [19:26:05] heh, so all_networks isn't [19:26:11] * andrewbogott throws up his hands [19:26:29] (03CR) 10Framawiki: [C: 032] output.py: Fix 'fake list' boolean value for empty generators [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/461662 (https://phabricator.wikimedia.org/T204964) (owner: 10Zhuyifei1999) [19:27:13] (03Merged) 10jenkins-bot: output.py: Fix 'fake list' boolean value for empty generators [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/461662 (https://phabricator.wikimedia.org/T204964) (owner: 10Zhuyifei1999) [19:29:43] !log quarry deployed 4b01077 on quarry-web-01 T204964 [19:29:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [19:29:48] T204964: Empty resultsets give broken json - https://phabricator.wikimedia.org/T204964 [19:30:05] mutante: well, that sounds like we might have to revert :( Unless you feel like setting up new mail relays [19:30:47] andrewbogott: no, i don't. i would like to just get rid of trusty [19:30:56] can we revert the eqiad-r move then [19:31:01] but still use the stretch instance [19:31:34] ugh [19:31:39] andrewbogott does that mean mail is broken for example "gerrit" uses it to send mail out. [19:31:44] Does that mean it will now fail? [19:31:48] I've never migrated the other way — It would take a couple of days for me to write a script for that [19:31:53] paladox: I don't know — probably? [19:32:10] hmm [19:33:43] mutante: Is there a reason not to move it back to trusty in eqiad and then back to stretch/eqiad1 when we have a mail relay? That seems like less work and it doesn't hurt anything to leave Trusty running there for another week [19:33:48] (since it'll be months for other projects anyway) [19:34:25] !log discourse-wam deleting project as per note on https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2018_Purge#OBSOLETE_discourse-wam [19:34:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Discourse-wam/SAL [19:34:50] andrewbogott: is the issue that converting relic-stretch back to eqiad isnt easy? [19:35:07] "I've never migrated the other way — It would take a couple of days for me to write a script for that" [19:35:08] Ꮃіth our IᖇC ad sеrᴠіcе you can reɑⅽh ɑ ɡlοbaⅼ aᥙԁіence οf entrepreneurs ɑnd feᥒtaᥒyⅼ adԁісts wіtһ еⲭtraοrdinary еnɡаgeⅿеnt rɑtеsⵑ httрs:/∕wіllⅰamⲣіtⅽοck.cοm／ [19:35:08] i can make relic-stretch2 and use that and be done [19:35:09] ? [19:35:30] andrewbogott so wmcs will have email relays? [19:35:37] then we dont need to convert anything back and kill trusty and use stretch in eqiad [19:35:41] mutante: that's fine although it seems like more work than just reverting for now [19:35:53] since we have to move to eqiad1 soon regardless [19:36:48] is there a relation between using stretch and having to move it? [19:37:51] you could use stretch in eqiad too. it would use the 10.x range so will work with the prod email relays. [19:38:00] that's what i wanted all the time :) [19:38:11] to resolve that specific ticket [19:38:32] * framawiki discovers that mutante is in fact Dzahn [19:42:32] andrewbogott: so is "eqiad" <->" "eqiad-r' switch always on the level of an entire project or each instance [19:44:50] it's per instance [19:45:01] ah, cool! [19:45:05] but also I currently have eqiad disabled for that project. I can re-enable [19:45:16] that would be nice, please do [19:46:58] Isn't it just doing extra, pointless work to build a new stretch VM in eqiad just so you can close the ticket today rather than next week though? [19:48:31] no, not to me. i would like to resolve that ticket and feel the other thing is actually a compeltely different story that will take longer thatn next week [19:48:47] anyway, re-enabled [19:48:48] and since we are puppetized it's easy to apply a role and run it [19:48:54] thank you [19:59:11] mutante: do you understand the mail relay request enough to create a ticket? I definitely don't [20:02:31] andrewbogott: i think it's basically the same as the commit message on https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/462012/ [20:02:38] but a ticket probably needs to be more general [20:02:49] planning network ranges globally [20:03:03] mutante: It sounded to me like someone needs to build a mail relay someplace, probably on a VM? [20:03:05] all the things that Faidon said on the other channel [20:03:55] i think so, yes, that's correct [20:04:10] he said the solution is that there are separate mail relays [20:04:16] and prod mx are not used anymore [21:09:41] А fаѕcіᥒаtіᥒɡ blοg ᴡhere frᥱenοԁe staff mеmbеr Matthew ⅿѕt Trοᥙt reϲouᥒtѕ һiѕ eхperⅰencеѕ of eуe-rapiᥒg yo∪nɡ ϲһiⅼdrеn httpsː／/ΜattЅTrout.coⅿ⁄ [21:16:45] !log toolserver-legacy associating floating IP (the existing one in eqiad) with relic-stretch-eqiad [21:16:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [21:19:10] !log toolserver-legacy update DNS zone entries and remove eqiad-r IP, revert to eqiad IP [21:19:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolserver-legacy/SAL [21:21:54] A faѕcіnating blഠɡ wһerе freenоⅾe staff ⅿеmbᥱr Mattheᴡ ⅿѕt Ꭲrоut rесounts һіs eⅹⲣᥱrieᥒces of eуe-rapinɡ уо∪ng chiⅼdrеn https://MattЅTrⲟᥙt．ⅽoⅿ/ [21:21:54] Wⅰth our IᏒC ɑd sеrvicᥱ you сan rеaϲh а glοbal aᥙdіеncе of entrерrеᥒeᥙrѕ and fᥱntaᥒуl ɑddiϲtѕ ᴡitһ еⅹtraordіᥒɑrу engagеmеnt rаtᥱs! https://ᴡilⅼiampіtϲοck．ⅽoⅿ／ [21:26:52] I tһought you gᥙуs mⅰgһt be interestеd in this bⅼog by freenodᥱ staff meⅿbеr Brỿaᥒ kⅼoerі Ⲟѕterɡaarⅾ һttpѕ:/∕bryanostеrgaɑrd.com/ [21:44:35] * bd808 sighs [21:44:36] spammers [21:56:27] !log toolserver-legacy sending all frozen messages from exim queue on old instance to clean up [21:56:29]