[00:02:42] <legoktm>	 Krenair: https://www.mediawiki.org/w/index.php?diff=2861216&oldid=2819771&title=Wikibugs&type=revision anything else that needs to be mentioned? also I assume you already got the existing channel set whitelisted?
[00:04:18] <Krenair>	 legoktm, I did give them the list of channels and I think they whitelisted them
[00:04:22] <Krenair>	 but I have no way of checking
[00:05:18] <legoktm>	 ack, ty
[00:13:43] <Krenair>	 legoktm, the one I spoke to was Fuchs
[00:21:10] <davidwbarratt>	 Question... on a Cloud VPS... can you make a HTTP request over the local network to the varnish cache of a wiki? (i.e. get a cached response from the API?)
[00:23:01] <davidwbarratt>	 i.e. could I do something like: http://enwiki/api/rest_api/page/html/Saturn
[00:23:12] <davidwbarratt>	 or would it have to go over the public network?
[00:30:01] <Krenair>	 davidwbarratt, to the varnish cache of a wiki?
[00:30:21] <Krenair>	 you mean, bypassing LVS?
[00:31:23] <Krenair>	 you should just use the public hostname
[00:31:28] <Krenair>	 you'll get the public IP from it, but look:
[00:31:55] <Krenair>	 https://phabricator.wikimedia.org/P7472
[00:32:13] <Krenair>	 no non-wikimedia equipment
[00:33:10] <Krenair>	 you won't be able to talk directly to an individual varnish server, just like the rest of the internet can't
[00:33:30] <Krenair>	 but you don't need to
[00:34:25] <davidwbarratt>	 right, no not the individual server... .uhh... I guess the better question is... do the cloud VPS instances operate on the same local network as the production servers or no?
[00:34:47] <Krenair>	 that's a complicated question
[00:34:55] <davidwbarratt>	 errr... the same network as the load balancer (i.e. the thing i want to connect to)
[00:34:56] <davidwbarratt>	 haha. ;)
[00:35:19] <legoktm>	 davidwbarratt: making requests from cloud services to the mediawiki app servers will go through varnish just like any other request
[00:35:24] <Krenair>	 particularly because prod itself is spread across multiple sites
[00:35:32] <legoktm>	 there's no special internal routing
[00:35:43] <legoktm>	 but because you're in the data center you'll have insanely fast network speeds
[00:35:52] <davidwbarratt>	 legoktm right, I'm saying, can I do that across the local network, or does it have to go over the internet?
[00:35:57] <davidwbarratt>	 legoktm oh ok, so it shoudln't matter?
[00:35:58] <Krenair>	 actually you're not at the moment legoktm 
[00:36:11] <Krenair>	 it is going to a different datacentre
[00:36:24] <legoktm>	 davidwbarratt: it doesn't matter. just make networks requests like normal
[00:36:52] <legoktm>	 Krenair: right, I forgot we were switched over
[00:37:11] <Krenair>	 and with no public labs in codfw..... yeah
[00:38:11] <Krenair>	 sure it's not sub-millisecond ping, but still, 36-39ms, not bad
[00:38:25] <legoktm>	 "not bad" lol
[00:40:18] <Krenair>	 slightly faster than me connecting to esams
[00:41:49] <Krenair>	 davidwbarratt, just use the normal hostnames, from within labs I would expect the packets to only go through wikimedia routers
[00:41:52] <Krenair>	 i.e. not random internet ones
[00:42:37] <legoktm>	 Krenair: !!! you used the bad word ;-) cloud services*
[00:43:10] <Krenair>	 VPS**
[00:43:45] <Krenair>	 still feel the new name is awkward in a sentence sometimes
[00:44:42] <harej>	 "from within Cloud VPS," if we're being pedantic. But I think we all know what "Labs" is
[00:45:04] <Krenair>	 it got changed because lots of people were confused about what 'Labs' is
[00:47:55] <harej>	 this is true
[00:47:59] <harej>	 we three*, at least
[00:48:54] <Krenair>	 yes
[01:03:54] <davidwbarratt>	 Krenair ok, so using a Cloud VPS would be the fastest way to connect to MediaWiki's API and just use the standard public hostnames?
[01:04:14] <Krenair>	 yeah
[01:06:02] <davidwbarratt>	 Krenair okie dokie, thanks!
[05:28:42] <annika>	 has there been a recent change to webservices? i'm getting a lot of errors with my webservice lately
[05:57:25] <legoktm>	 annika: what errors?
[05:57:37] <legoktm>	 what kinds of errors*
[05:58:16] <annika>	 i'm not sure
[05:58:21] <annika>	 response not received, request sent: 800 on socket: …
[05:58:34] <annika>	 unexpected end-of-file …
[06:02:33] <zhuyifei1999_>	 annika: where is that error message from?
[06:02:41] <zhuyifei1999_>	 as in where do you see it?
[06:03:03] <annika>	 from error.log of tools.giftbot
[06:07:59] <zhuyifei1999_>	 annika: that message looks like from lighttpd
[06:08:13] <annika>	 i think so too
[06:08:41] <zhuyifei1999_>	 I don't know how tcl works so idk what I can do
[06:09:37] <annika>	 i was thinking as i didn't change anything that the cause must lie somewhere else
[06:09:59] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/VDyQyEsj/
[06:10:10] <zhuyifei1999_>	 does this process tree look right to you?
[06:10:58] <annika>	 seems right
[06:17:11] <zhuyifei1999_>	 (idk then)
[07:01:42] <arturo>	 legoktm: you around?
[07:01:57] <legoktm>	 arturo: for about 10 more minutes
[07:02:04] <arturo>	 ok
[07:02:14] <arturo>	 regarding T202155
[07:02:15] <stashbot>	 T202155: Approve https://packages.sury.org/php/ as an accptable Debian package source for Toolforge - https://phabricator.wikimedia.org/T202155
[07:02:37] <arturo>	 I have some doubts, because the repo is already imported into apt.wikimedia.org, right?
[07:02:43] <arturo>	 so you only need the wiki update?
[07:03:08] <legoktm>	 only part of the repository is imported
[07:03:27] <legoktm>	 just the packages phabricator needed, plus two more that I asked for in my initial pass
[07:03:36] <arturo>	 and you need all of them?
[07:04:20] <legoktm>	 well, more :) for example, php7.2-zip requires a newer libzip, which wasn't imported (since phab probably doesn't use -zip) so it can't be installed
[07:04:43] <arturo>	 if they are in apt.wikimedia.org, then this fall in `We assume that whatever is good for production is also OK for Toolforge. `
[07:05:08] <legoktm>	 right, I was proposing just using the sury repository directly
[07:05:20] <arturo>	 but how? in Toolforge nodes?
[07:05:46] <arturo>	 i.e, update `/etc/apt/sources.list` in toolforge nodes?
[07:05:55] <legoktm>	 just in the docker images right now
[07:06:13] <legoktm>	 specifically https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/toollabs-images/+/master/php72/base/
[07:07:26] <legoktm>	 also to be clear, there's nothing wrong with using apt.wm.o, it'll probably work fine. I was just thinking that this would be easier for toolforge to move with, since it wouldn't be tied to phabricator's needs
[07:07:27] <arturo>	 so, you would like to use the external repo directly, instead of apt.wikimedia.org
[07:07:35] <legoktm>	 yes
[07:07:57] <arturo>	 the problem I see is that this external repo has many many other packages, not related to php
[07:08:59] <legoktm>	 in https://packages.sury.org/php/pool/main/ ?
[07:10:11] <arturo>	 well no, in the top level https://packages.sury.org/
[07:10:29] <legoktm>	 oh, but I only want to use the /php/ one
[07:10:42] <arturo>	 I see
[07:10:54] <arturo>	 we could batch-import all the php packages into apt.wikimedia.org
[07:11:09] <arturo>	 would that work for you?
[07:12:38] <legoktm>	 It's totally fine for me. I was hoping to reduce work for others by not needing to import stuff into apt.wm.o and being able to use the repo directly
[07:14:00] <arturo>	 ok, fair enough
[07:14:12] <arturo>	 will think about this and follow-up on the phab task
[07:14:17] <legoktm>	 ok
[07:14:20] <arturo>	 thanks legoktm !!
[07:15:02] * arturo afk for a bit
[07:19:54] <jynus>	 zhuyifei1999_ I can merge https://gerrit.wikimedia.org/r/454217 now, unless you tell me that is going to break quarry
[07:20:18] <jynus>	 we can then talk what is the right classes to apply to it
[07:20:53] <zhuyifei1999_>	 I think it's okay
[07:21:05] <zhuyifei1999_>	 quarry is still using the mysql module right now
[07:21:10] <jynus>	 ok, then
[07:22:05] <jynus>	 so in general, packages and config is only what you need, the second to setup the mysql config
[07:22:38] <jynus>	 the other classes are just in special cases, like overriding init /systemd config
[07:22:48] <jynus>	 extra monitoring
[07:22:50] <jynus>	 etc.
[07:23:12] <jynus>	 if you send me patches or something I can help you
[07:25:03] <zhuyifei1999_>	 ok
[07:25:08] <zhuyifei1999_>	 will do
[07:57:25] <zhuyifei1999_>	 jynus: does https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454481/ look right to you?
[07:57:46] <zhuyifei1999_>	 (I'll test it on the test puppetmaster if so)
[07:58:11] <zhuyifei1999_>	 oh wait no
[08:01:56] <zhuyifei1999_>	 what is this modules/quarry/manifests/database.pp:8 wmf-style: class 'quarry::database' declares class mariadb::config from another module
[08:03:14] <jynus>	 so in theory modules should not call other modules
[08:03:47] <jynus>	 roles should include profiles and profiles should use modules
[08:04:10] <jynus>	 https://wikitech.wikimedia.org/wiki/Puppet_coding#Organization
[08:04:30] <zhuyifei1999_>	 so the quarry module should be profiles?
[08:04:46] <jynus>	 according to that page, yes
[08:05:00] <jynus>	 not the module
[08:05:12] <jynus>	 the modules I guess should be mostly the same
[08:05:30] <jynus>	 just the module should not include mysql or mariadb classes
[08:05:33] <zhuyifei1999_>	 I don't think I know how to migrate to this new format
[08:06:33] <jynus>	 let's focus on get the module right first
[08:06:53] <jynus>	 we can do the refactoring later
[08:07:13] <zhuyifei1999_>	 ok
[08:07:14] <jynus>	 my question is if there wasn't any custom config before?
[08:07:34] <zhuyifei1999_>	 what custom config?
[08:08:00] <jynus>	 so there was no special configuration but the one on the config_hash?
[08:08:33] <jynus>	 (for the database)
[08:08:40] <zhuyifei1999_>	 I don't think I see anything in /etc/mysql/my.cnf. let me re-check
[08:09:05] <zhuyifei1999_>	 https://www.irccloud.com/pastebin/mHWIUwjR/
[08:09:19] <jynus>	 and how important is to have the database on /srv/mysql/data or /srv/sqldata ?
[08:09:42] <jynus>	 can it be on the default /var/lib/mysql ?
[08:10:06] <zhuyifei1999_>	 I think /srv can benefit from the larger lvm
[08:10:15] <jynus>	 ok
[08:10:21] <jynus>	 so this is what we are going to do
[08:10:40] <zhuyifei1999_>	 but whether it's on /srv/mysql/data /srv/sqldata I don't think it matters
[08:11:01] <jynus>	 take the old my.cnf
[08:13:16] <jynus>	 parametrize port, socket, datadir, etc, as in https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/role/templates/mariadb/mysqld_config/core-mysql.my.cnf.erb
[08:13:38] <jynus>	 so change port = 3306 for <%= @port %>, etc
[08:14:13] <jynus>	 and call mariadb::config with 
[08:14:26] <zhuyifei1999_>	 so I can't just use the default in mariadb::config?
[08:14:44] <jynus>	 you can
[08:14:56] <jynus>	 but it may have a different config than you want
[08:15:05] <zhuyifei1999_>	 okay
[08:15:11] <jynus>	 you can test it
[08:15:22] <jynus>	 and see if the config works for you with the compiler
[08:15:51] <zhuyifei1999_>	 what do you mean by 'Missing a config file'?
[08:16:25] <jynus>	 so the default should be modules/mariadb/templates/default.my.cnf.erb
[08:16:30] <zhuyifei1999_>	 afaik there's no easily-usable puppet-compiler for cloud instances
[08:16:32] <jynus>	 check if that would work for you
[08:17:12] <jynus>	 for example, it uses 75% of the available memory
[08:17:23] <jynus>	 which is ok if it is a dedicated server
[08:17:36] <jynus>	 but not ok if you run more stuff there
[08:17:38] <zhuyifei1999_>	 well, it shares memory with redis
[08:18:22] <jynus>	 so how much memory would you say it is ok to use?
[08:19:41] <zhuyifei1999_>	 honestly, idk.
[08:19:57] <jynus>	 :-)
[08:20:04] <zhuyifei1999_>	 I rarely fine-tune my config files
[08:20:07] <jynus>	 just do this, copy the current config
[08:20:26] <jynus>	 and upload it to the quarry templates
[08:20:26] <zhuyifei1999_>	 ok
[08:20:38] <jynus>	 and point it in mariadb config with
[08:22:11] <jynus>	 config => 'quarry/quarry.my.cnf.erb''
[08:22:25] <jynus>	 and just add there the current/expected config
[08:22:44] <jynus>	 under modules/quarry/templates/quarry.my.cnf.erb
[08:23:07] <jynus>	 you only need that parameter and it will work well
[08:24:04] <zhuyifei1999_>	 k
[08:27:08] <zhuyifei1999_>	 can I just use the default? I see config.pp has lots of references to basedir and datadir
[08:28:09] <zhuyifei1999_>	 (quarry was using mostly the default while using mysql module anyways)
[08:28:33] <jynus>	 if it works for you, sure
[08:30:21] <zhuyifei1999_>	 I think it'll work
[08:31:28] <jynus>	 I need to atend other things, send your patches and I will review them later or daniel or someone else will
[08:32:17] <zhuyifei1999_>	 ok
[11:52:34] <Amir1>	 !log wikilabels spinning up wikilabels-02 (T168478)
[11:52:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[11:52:37] <stashbot>	 T168478: Keep wmflabs scoring boxes up-to-date - https://phabricator.wikimedia.org/T168478
[12:25:14] <Amir1>	 !log wikilabels 03f71f0 is going to staging
[12:25:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[12:27:20] <Amir1>	 Everything looks fine, moving to prod
[12:28:30] <Amir1>	 !log wikilabels: 03f71f0 is going to production
[12:28:30] <stashbot>	 Amir1: Unknown project "wikilabels:"
[12:29:02] <Amir1>	 !log wikilabels 03f71f0 is going to production
[12:29:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[13:00:35] <arturo>	 !log tools remove all emails in tools-mail.eqiad.wmflabs queue, 3378 bounce msgs, mostly related to @qq.com
[13:00:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:02:14] <arturo>	 !log tools I used this command: `sudo exim -bp | sudo exiqgrep -i | xargs sudo exim -Mrm`
[13:02:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:00:52] <wm-bot>	 Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[14:50:42] <wm-bot>	 Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @CFisch_WMDE & @amir1 - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
[15:27:09] <arturo>	 !log admin cleanup local keystone database in cloudcontrol1003.wikimedia.org (already in m5-master)
[15:27:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:28:21] <arturo>	 !log admin cleanup local glance,keystone databases in cloudcontrol1003.wikimedia.org (already in m5-master)
[15:28:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:49:38] <addshore>	 am I being stupid, but should this work?
[15:49:43] <addshore>	  /usr/bin/jsub -N cron-30 -once -quiet bash /data/project/wmde-access/cronUpdate.sh
[15:49:56] <addshore>	 balls
[15:49:56] <addshore>	 wait
[15:50:00] <addshore>	 i just have the wrong file name
[15:50:06] <addshore>	 updateCron not cronUpdate, lol
[15:50:16] * addshore goes to throw himself down the stairs
[16:08:54] <zhuyifei1999_>	 (lol)
[16:14:09] <Krinkle>	 Hm.. it seems the openstack-browser api might be having issues.
[16:14:19] <Krinkle>	 getting empty responses from https://tools.wmflabs.org/openstack-browser/api/dsh/project/:project
[16:14:25] <Krinkle>	 e.g. https://tools.wmflabs.org/openstack-browser/api/dsh/project/cvn
[16:14:33] <Krinkle>	 affecting Nagf - https://tools.wmflabs.org/nagf/?project=cvn
[16:18:34] <zhuyifei1999_>	 I see '500 Internal Server Error' in that page
[16:20:30] <zhuyifei1999_>	 keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-ba28be7d-2727-4ded-9f81-b034c0afd85c)
[16:21:07] <zhuyifei1999_>	 umm I guess andrewbogott? openstack-browser is generating this error in the logs
[16:22:15] <andrewbogott>	 I'm away for a while yet... bstorm_ might be interested.
[16:22:26] <zhuyifei1999_>	 k
[16:22:51] <bstorm_>	 Hrm
[16:23:40] <bstorm_>	 I can take a look at that in a bit.
[16:24:25] <bstorm_>	 It's probably not entirely unlike issues that affected nfs-exportd
[18:12:55] <Amir1>	 !log wikilabels redirecting traffic of labels.wmflabs.org from wikilabels-01.eqiad.wmflabs to wikilabels-02.eqiad.wmflabs (T168478)
[18:12:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL
[18:12:57] <stashbot>	 T168478: Keep wmflabs scoring boxes up-to-date - https://phabricator.wikimedia.org/T168478
[18:20:55] <harej>	 interesting, "labeling unsourced statements" is a research team project, is it not?
[18:21:03] <harej>	 i thought they had a separate instance of wikilabels
[18:21:08] <harej>	 or was that for another project
[18:22:40] <Amir1>	 harej: it's a research project but they use wikilabels (our infra) 
[18:23:05] <harej>	 how good are we at supporting labeling projects that are not ours?
[18:27:31] <bstorm_>	 Interesting.  The errors that openstack-browser are showing have caused it to think their are 7 instances in the entire environment.
[18:33:20] <bstorm_>	 That said, this is definitely doing what nfs-exportd did.  I think that's quite fixable.
[20:02:09] <wikibugs>	 (03PS1) 10D3r1ck01: Add examples extension to #wikimedia-dev-africa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/454637
[20:07:50] <wikibugs>	 (03PS2) 10D3r1ck01: Add examples extension to #wikimedia-dev-africa [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/454637
[20:54:33] <bstorm_>	 zhuyifei1999_ and Krinkle: I think I got it working
[20:55:29] <bstorm_>	 If you see anything really odd, let me know.  My biggest concern is that some things could be somehow duplicated with the approach I took, but I'm not seeing it so far.
[21:04:42] <Krinkle>	 bstorm_: LGTM, thanks!
[21:04:52] <bstorm_>	 👍🏻
[22:25:11] <mutante>	 zhuyifei1999_: hello!
[22:25:43] <mutante>	 zhuyifei1999_: i saw what jenkins-bot told you about your patch.. and it's not your fault
[22:25:54] <mutante>	 it was only always broken and it notices because you touch it
[22:26:30] <mutante>	 let me try to upload a change before your change that might fix it
[22:28:11] <mutante>	 zhuyifei1999_: which class is really applied on quarry? i see several roles in role/manifests/labs/quarry/ but not a global one
[22:29:52] <mutante>	 is this right https://tools.wmflabs.org/openstack-browser/puppetclass/role::labs::quarry::database