[00:11:00] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674325 (10MusikAnimal) Your comment was at 06:22 UTC on 9 October. Here's what I've got when grepping the logs... [00:16:19] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674332 (10MusikAnimal) Scratch that, the logs clearly say `s5003` and not `s51187` =P The cache has been cleare... [06:15:04] (03CR) 10Hashar: [V: 032 C: 032] "Surely using base64() was done to prevent json to mangle a password somehow :-]" (031 comment) [labs/private] - 10https://gerrit.wikimedia.org/r/383386 (owner: 10Dduvall) [08:07:56] 10Data-Services, 10cloud-services-team, 10DBA, 10XTools: Request to increase active connection quota for user s51187 on analytics.db.svc.eqiad.wmflabs - https://phabricator.wikimedia.org/T177570#3674700 (10Marostegui) Cool - let me know how it goes. I didn't want to add exceptions to two users, if we can a... [08:27:31] 10cloud-services-team, 10Operations: Switch labstore servers to default SSH configuration - https://phabricator.wikimedia.org/T177914#3674761 (10MoritzMuehlenhoff) [08:56:05] !help I'm starting to prepare a webservice that uses php and a simple echo "Test"; isn't showing up. what am I doing wrong? [08:56:05] DatGuy: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [09:45:33] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3674908 (10hashar) + @fgiunchedi At least we seem to run the latest Grafana (4.5.2). >>! In T177747#3669196, @Gilles wrote: > It's Graphite intermittently servi... [09:45:45] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3674910 (10hashar) p:05Triage>03Normal [10:02:26] 10cloud-services-team: unattended-upgrades not upgrading "-wikimedia" packages automatically in wmcs - https://phabricator.wikimedia.org/T177920#3674953 (10fgiunchedi) [10:58:23] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3675045 (10Gilles) Nice find! graphite-web is in sid and buster, but not in stretch? https://packages.debian.org/jessie/graphite-web otherwise I would have suggested... [13:15:56] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3675512 (10fgiunchedi) Nice find indeed @hashar ! @gilles yeah looks like graphite-web was removed from stretch because of django incompatibility https://packages.qa.... [13:20:55] 10Cloud-VPS, 10Graphite: grafana-labs often fails to generate graphs with c.datapoints is undefined - https://phabricator.wikimedia.org/T177747#3675523 (10hashar) Hurrah it works fine on https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?orgId=1&var-project=integration&from=now-7d&to=now So I... [13:52:21] 10Tools, 10I18n: Consider merging "Wikinity-stats-before" and "Wikinity-stats-after"? - https://phabricator.wikimedia.org/T177268#3652846 (10Liuxinyu970226) maybe this user can tell me if this is possible or not? [13:58:18] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @addshore & @Andrew-WMDE - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:10:04] (03PS1) 10Gehel: New role to validate maps with new vector tiles. [labs/private] - 10https://gerrit.wikimedia.org/r/383573 (https://phabricator.wikimedia.org/T153282) [14:11:08] (03CR) 10Gehel: [V: 032 C: 032] New role to validate maps with new vector tiles. [labs/private] - 10https://gerrit.wikimedia.org/r/383573 (https://phabricator.wikimedia.org/T153282) (owner: 10Gehel) [14:12:32] 10Data-Services, 10cloud-services-team (Kanban), 10Wikidata, 10User-Ladsgroup, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3675709 (10daniel) Thank you all for sorting this out! @Andrew For the record: yes, wb_terms is wikida... [14:14:36] (03PS1) 10ArthurPSmith: Group 1 ID has changed due to a merge [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/383576 [14:17:14] (03CR) 10ArthurPSmith: [C: 031] "ptable has been broken for a while due to a merge of the "Group 1" QID - this should fix it." [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/383576 (owner: 10ArthurPSmith) [14:27:36] 10Toolforge: Alert for 'All k8s worker nodes are healthy on checker.tools.wmflabs.org' - https://phabricator.wikimedia.org/T177944#3675759 (10chasemp) p:05Triage>03Normal [14:37:12] 10Toolforge: Alert for 'All k8s worker nodes are healthy on checker.tools.wmflabs.org' - https://phabricator.wikimedia.org/T177944#3675813 (10chasemp) a:03Andrew [14:48:20] (03PS1) 10Gehel: New role to validate maps with new vector tiles. [labs/private] - 10https://gerrit.wikimedia.org/r/383588 (https://phabricator.wikimedia.org/T153282) [14:48:32] (03CR) 10Gehel: [V: 032 C: 032] New role to validate maps with new vector tiles. [labs/private] - 10https://gerrit.wikimedia.org/r/383588 (https://phabricator.wikimedia.org/T153282) (owner: 10Gehel) [15:24:00] 10Toolforge: Alert for 'All k8s worker nodes are healthy on checker.tools.wmflabs.org' - https://phabricator.wikimedia.org/T177944#3675989 (10Andrew) 05Open>03Resolved When I refreshed puppet on the affected host, it included this diff: ``` - name: client-infrastructure user: - token: faketoken... [15:26:35] hey, any known DNS issues with labpuppetmaster1001? [15:26:56] it cannot resolve traffic-misc-varnish5.traffic.eqiad.wmflabs, while traffic-puppetmaster.traffic.eqiad.wmflabs can [15:37:34] ema: afaik production hosts are not setup to resolve teh wmflabs domain [15:37:47] maybe I'm wrong or misunderstanding what you mean [15:38:24] chasemp: so, I've created a new instance on horizon, project "traffic" [15:38:27] hi! [15:38:43] chasemp: by default it uses labpuppetmaster1001 as its puppetmaster [15:38:56] * chasemp nods -- with you so far [15:39:24] and I expected that an instance in the traffic project would have been able to have access to other instances in the same project [15:39:50] define access [15:40:22] right :) given that labpuppetmaster1001 can't resolve wmflabs, every piece of puppet code trying to resolve a hostname such as traffic-misc-varnish5.traffic.eqiad.wmflabs fails [15:40:52] (to state the issue otherwise) [15:41:17] I think that's a known thing and you are not wrong, there is a way around it possibly, this comes up every now and again [15:41:42] andrewbogott: ^ can you give insight [15:41:52] issue easily solved by using a project-specific puppetmaster such as traffic-puppetmaster.traffic.eqiad.wmflabs [15:41:58] um… in a meeting but be there soon [15:42:05] (sure np) [15:42:36] ema: yeah, that's a meh solution but maybe that's teh current course, I'd wait for andrewbogott to come around and I think he'll recall what we did previously [15:42:55] chasemp: cool, thanks man! [15:42:59] not to pawn it off :) but he has rebuilt most of the puppetmaster setup and probably has it at the tip of his brain [15:51:07] ema, chasemp, I'm back now. So far I think you're accurately describing the status quo :) I believe that you can pass in a specific dns resolver to the puppet function that resolves things... [15:51:25] ah yeah...that was the hack I think [15:51:38] It might also makes sense to just have the puppetmaster know about those servers. I can't immediately think of a downside for that [15:52:48] it seems like we've danced around it a few times and I'm not sure why atm [15:52:58] (03PS1) 10Ottomata: Add druid/public/worker.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/383602 [15:53:21] (03CR) 10Ottomata: [V: 032 C: 032] Add druid/public/worker.yaml [labs/private] - 10https://gerrit.wikimedia.org/r/383602 (owner: 10Ottomata) [15:53:56] 10cloud-services-team (Kanban): Should VPS puppetmasters include labs-ns0/ns-1 in their resolv.confs? - https://phabricator.wikimedia.org/T177959#3676095 (10Andrew) [15:54:04] ema, chasemp ^ [15:54:38] andrewbogott: I couldn't find an example of specifying the resolver [15:55:43] There are different ways of resolving things… are we talking about a particular piece of code here? [15:56:59] good question :) [15:57:05] I'll wait or ema to circle back [15:57:09] s/or/for [15:57:34] DatGuy: Sorry we missed you! I don't immediately know the answer to your question, but if you repeat here with more detail someone may be able to help you. (It also might be good to link to whatever docs you're following and mention which step you're on) [15:57:55] andrewbogott: it's alright, figured it out. missed something with the grid engine/k8s transition [15:58:40] chasemp, andrewbogott: +1 for making sure labs puppetmasters can resolve wmflabs hostnames [16:06:03] DatGuy: great! [16:09:45] ema: In the meantime, if you are using ipresolve... [16:09:47] ipresolve(string $name_to_resolve, string $type = '4', string $nameserver = nil) [16:10:24] you can pass in labs-ns0.wikimedia.org or labs-recursor0 [16:10:38] (Though that will probably ugly up your code) [16:12:27] andrewbogott: thanks, I'll use the traffic-specific puppetmaster for now :) [16:23:36] (03PS1) 10Eevans: Add key material for new deployment-prep Cassandra nodes [labs/private] - 10https://gerrit.wikimedia.org/r/383608 [16:25:56] (03CR) 10Eevans: [C: 031] Add key material for new deployment-prep Cassandra nodes [labs/private] - 10https://gerrit.wikimedia.org/r/383608 (owner: 10Eevans) [16:47:32] (03CR) 10Dzahn: [V: 032 C: 032] Add key material for new deployment-prep Cassandra nodes [labs/private] - 10https://gerrit.wikimedia.org/r/383608 (owner: 10Eevans) [18:13:00] 10Cloud-Services, 10cloud-services-team (Kanban), 10Patch-For-Review: Remove non-interactive bots from #wikimedia-cloud - https://phabricator.wikimedia.org/T177427#3658254 (10Dzahn) Also see T177974 for a similar thing that was attempted in the past and lead to removing the channel again now. [18:20:55] (03PS137) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [18:21:05] PS 137.... ouch [18:23:20] Sagan https://gerrit.wikimedia.org/r/#/c/193434/ :) [18:23:21] 150 [18:32:07] (03CR) 10Ricordisamoa: [C: 04-2] "PS137 changes suggest.js into a module" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [18:41:40] 10Toolforge, 10Outreachy (Round-15): Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T175768#3676785 (10bd808) [18:44:45] (03PS138) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [18:52:57] (03CR) 10Ricordisamoa: [C: 04-2] "PS138 splits EntitySuggester into a module" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [19:03:53] (03PS139) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [19:04:20] * Reedy wonders if Ricordisamoa knows the initial commit doesn't need to be perfect [19:09:11] (03CR) 10Ricordisamoa: [C: 04-2] "PS139 fixes some JSDoc issues" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [19:12:34] yes, small changes are nice. embrance many small changes [19:14:12] (03CR) 10Ricordisamoa: [C: 032] Group 1 ID has changed due to a merge [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/383576 (owner: 10ArthurPSmith) [19:14:32] (03Merged) 10jenkins-bot: Group 1 ID has changed due to a merge [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/383576 (owner: 10ArthurPSmith) [19:20:03] (03CR) 10Ricordisamoa: "It seems to have done the trick, thank you!" [labs/tools/ptable] - 10https://gerrit.wikimedia.org/r/383576 (owner: 10ArthurPSmith) [19:45:27] (03PS1) 10EddieGP: Remove gerrit notifications for #wikimedia-codereview [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/383656 (https://phabricator.wikimedia.org/T177974) [20:22:30] 10Toolforge, 10cloud-services-team (Kanban): Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#3677432 (10bd808) The new script is available on Toolforge hosts as `/usr/bin/oge-crontab`. The final step needed is to replace the current `/usr/local/bin/crontab` script... [20:44:18] Does tool labs support non-monolithic FastCGI python scripts? [20:47:11] 10Cloud-Services, 10wikitech.wikimedia.org: Stop using an older release of SMW and get back to tracking their master branch - https://phabricator.wikimedia.org/T75940#3677522 (10Krinkle) [20:52:12] 10Cloud-Services, 10Outreachy (Round-15): Proposal: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T177603#3677543 (10madhuvishy) [21:25:26] Dispenser: I'm not sure why it would not, but you may have to do something tricky to get your PYTHONPATH setup like you need it to be [21:25:55] fancy uwsgi apps are certainly possible and actually pretty easy [21:26:44] Everything I've tried a tutorial it hasn't worked [21:27:08] ok. which tutorial(s) are you following? [21:27:18] maybe I can spot some problems that we can fix [21:27:59] for uwsgi apps -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Python_.28uWSGI.29 [21:28:55] fastcgi is not a completely common Python app structure these days. Our php containers run as fcgi though so it should be possible to figure out [21:30:02] I'll try it again, a simple hello world example code would be nice [21:30:30] https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#the-first-wsgi-application [21:31:22] Dispenser: this builds up from a hello world to something slightly more advanced -- https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool [21:32:00] I wrote my own framework, I don't want to bolt on another framework to it [21:32:53] more of a library than framework [21:33:09] ok. so you have a custom python fcgi container and you would like to run it on Toolforge? [21:33:47] what is a container? [21:34:16] application, script, thinkg [21:34:17] *thinkg [21:34:24] heh. can't type that word -- thing [21:35:02] I tend to think about fcgi as a "container" that holds an app I guess [21:35:11] a base runtime [21:35:27] A single file python script that is called often has clear input/output, already using a pseudo-wsgi wrapper [21:36:34] When I hear container, I think docker-- And nobody given me a clear explanation of pros and cons nor what's happening exactly [21:37:19] an oversimplification, but a Docker image is a tarball that gets extracted and executed inside a chroot jail [21:37:41] (Something about duplication the function of an operating system, presumably to avoid versioning conflicting) [21:38:35] the chroot is not quite a "normal" chroot. it has extensions into kernel space to allow other resource limiting beyond filesystem boundaries. [21:38:50] I'll work on the tutorial and write down my thoughts [21:39:19] ok. I'm glad to help you work through problems [21:39:45] write down my thoughts in a separate editor (to avoid steeling your time)* [21:39:58] *nod* [21:40:43] changing flask for another wsgi aware framework in that tutorial should be trivial I hope [21:41:36] worst case, webservice --backend=gridengine generic $SCRIPT gives control of pretty much everything to a script you provide [21:42:20] that direction requires the script to setup something that speaks http [21:45:45] From "For new projects" I run `webservice --backend=kubernetes python shell`. It responds with "If you don't see a command prompt, try pressing enter." I don't see a prompt, I press enter. Now it crashes when I try re-running it. [21:46:50] Dispenser: hmmm... I haven't seen that behavior. Can I login to your tool and try to reproduce? [21:47:03] Ok figured it out $SHLVL > 2 [21:47:21] ah. inception :) [21:48:16] We should maybe make it print a banner when the shell is entered [21:49:15] 10Toolforge, 10cloud-services-team (Kanban): Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#3677736 (10zhuyifei1999) ``` tools.yifeibot@tools-bastion-02:~$ oge-crontab -e Traceback (most recent call last): File "/usr/bin/oge-crontab", line 337, in ... [21:49:49] Something like "Connecting to Kubernetes backend..." Maybe a fake "Press enter to start" [21:51:24] yeah. I can't remember what the need to hit enter is about. I think it has something to do with the way that the bash process is launched inside the Kubernetes "pod" [21:53:34] /usr/bin/python2: No module named venv [21:54:16] Same with `python2 -m virtualenv` [22:02:25] "Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running\nPod stopped. Session cannot be resumed." Would Session terminated be better? [22:02:56] 10Toolforge, 10Patch-For-Review: Catchpoint tests failing under Toolforge availability product - https://phabricator.wikimedia.org/T177103#3677791 (10chasemp) 05Open>03Resolved looks good now to me, thank you! [22:11:30] Dispenser: probably confusing, but "python" starts a python3 container. If you need python2 you must use "python2" instead. [22:12:21] I used python2 (see prog name) [22:13:44] `webservice --backend=kubernetes python shell` -- that gives you a python3 setup [22:14:19] Ok, need to amend docs then [22:14:20] there is a python2 interpreter in there, but it only has the bare needs for the Debian Jessie startup [22:14:57] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python2_.28Python2_.2B_Kubernetes.29 [22:16:16] we really should have called them "python3" and "python2" instead of "python" and "python2" but that decision was made quite a while ago [22:16:48] the broad thinking is that python2 is dead tech (although we know lots of people still use it out of habit) [22:22:44] the "Session ended..." up to the newline is what Kubernetes prints. We don't control that unfortunately. [22:23:58] `python` referring to python3 is very confusing, especially since with the python executable `python` is python2 [22:24:14] How do I test what to put into requirements.txt? Apparently flup uses datatime, but doesn't list it? [22:24:19] version control ~everything~ [22:25:58] Dispenser: I'm not completely sure I understand the question. You have a python package that does not require all of the things it needs? [22:26:15] yes [22:27:03] 10Cloud-VPS (Quota-requests): Request increased quota for Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677875 (10MarkAHershberger) [22:27:14] I guess the "easy" way to test is to use the python interactive shell or a test script from inside the webservice shell environment. [22:27:56] shell in, activate your venv, add things with pip, test them with ... somehting, exit [22:29:12] flup claims it has no dependencies except with python <2.4 [22:31:28] 10Cloud-VPS (Quota-requests): Request increased quota for Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677902 (10MarkAHershberger) It could well be that I just need a domain added to the proxy, but I don't see how to do that. [22:32:56] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677905 (10bd808) [22:33:40] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677875 (10bd808) >>! In T178012#3677902, @MarkAHershberger wrote: > It could well be that I just need a domain added to the proxy, but I don't see how to do that. To use a domain outs... [22:34:41] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677914 (10MarkAHershberger) I can manage the DNS. So I guess I need a static IP? [22:37:50] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677933 (10bd808) >>! In T178012#3677914, @MarkAHershberger wrote: > I can manage the DNS. So I guess I need a static IP? Sounds like it. We will review the request at our next regula... [22:44:33] 10Toolforge, 10cloud-services-team (Kanban): Rewrite /usr/local/bin/crontab in python; fix bugs - https://phabricator.wikimedia.org/T156174#3677947 (10bd808) >>! In T156174#3677736, @zhuyifei1999 wrote: > Umm... shouldn't the encode be utf-8 instead of latin-1? Alternatively can we open this temp file in text... [22:52:48] bd808: o.O I don't understand why ssh.communicate() returns str not bytes. looking at the docs https://docs.python.org/3/library/subprocess.html#subprocess.Popen without encoding or errors or universal_newlines they should be binary streams [22:53:53] * zhuyifei1999_ loves and hates python3 [22:54:15] zhuyifei1999_: "universal_newlines=True" makes them strings. I needed that for something ... [22:55:21] zhuyifei1999_: oh, it was for input=stdin I think [22:55:42] I don't see where https://phabricator.wikimedia.org/diffusion/LTOL/browse/HEAD/misctools/oge-crontab;1ac45aefdaaa3939ae7dd5343de8b206d483e4f5$87 universal_newlines is specified [22:56:10] input defaults to None for majority of cases [22:56:30] *stdin defaults to None [22:56:57] .... grep universal_newlines /usr/bin/oge-crontab ... matches. what did I do? [22:57:07] huh [22:57:59] https://phabricator.wikimedia.org/diffusion/LTOL/browse/master/misctools/oge-crontab [22:58:02] its there [22:58:17] the HEAD reference ... not sure what I grabbed [22:58:25] s/I/it/ [22:58:36] huh HEAD is not master's HEAD [22:58:58] o.O [22:59:19] this was the python3 hack I made -- https://phabricator.wikimedia.org/rLTOLd7449a2cfecec91d9a6db5db8b87026deb52c323 [22:59:49] ah ok [23:00:49] do you think it'll make sense to open the tempfile in text instead? [23:01:05] (not manual encode/decodes) [23:02:59] it might make more sense, yes [23:03:26] or make the ssh popen stuff all binary [23:03:58] The universal_newlines thing there might not really have been needed [23:04:15] I think I added that before I figured out the *args bug [23:04:34] I think all the regexes operates on strings not bytes [23:04:50] ugh. yeah at some point we need strings [23:05:01] what was the args bug? [23:05:21] I didn't want to assume utf-8, but maybe that's the actually reasonable thing to do [23:05:52] I had the *args splat in the wrong place in the signature [23:06:25] it accidentallly worked in my python2 tests but blew up in python3 in various strange ways [23:07:00] and was all cause by my trying to not wrap/unwrap a list of args on the calling side which was a dumb "optimization" [23:07:41] lol [23:08:44] bd808: I'm going to eat dinner, but it seem I can only run one fastcgi environment? (Using PHP and python) [23:08:57] hopefully everyone’s crontab is utf8 [23:09:30] Dispenser: hmm... possibly. I've never tried to have both at the same time with lighttpd [23:10:01] if someone puts a ton of binary into their crontab... nano should break them anyhow [23:10:03] zhuyifei1999_: encodings are hard :/ I can run some checks on the backend server though [23:11:03] Dispenser: there are some very old guides on lighttpd fcgi on wikitech [23:11:14] might need some digging though [23:14:17] (they were written when we only had lighttpd, and (tomcat and/or apache)) [23:16:16] Dispenser: I'm not sure I understand your edit at https://wikitech.wikimedia.org/w/index.php?title=Help:Toolforge/Web&diff=0&oldid=1770361 -- in this sense, what does "monolithic" mean? Single applications? [23:16:32] (pre-yuvi era lol) [23:17:54] Or is it in opposition to polyglot tools somehow? [23:22:17] bd808: what was universal_newlines for? can I replace it with encoding=utf8 explicitly? [23:22:50] zhuyifei1999_: I'd first try removing it entirely [23:23:26] bytes would be no good for regexes [23:23:45] right... you can decode the stream though [23:26:01] * zhuyifei1999_ prefers opening a stream with an encoding than manually call encode & decode [23:26:15] fair enough [23:26:22] :P [23:28:19] (will do when I get time, probably tomorrow night) [23:28:32] awesome