[01:28:46] (03PS1) 10Zoranzoki21: Fix "attachted" typo in nonexistent-article message [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/470307 (https://phabricator.wikimedia.org/T201491) [06:17:49] (03PS1) 10Legoktm: Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) [07:17:30] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/470318 (owner: 10L10n-bot) [07:17:36] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [labs/tools/crosswatch] - 10https://gerrit.wikimedia.org/r/470321 (owner: 10L10n-bot) [09:14:09] (03CR) 10Urbanecm: [C: 032] "Thank you!" [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/470307 (https://phabricator.wikimedia.org/T201491) (owner: 10Zoranzoki21) [09:15:08] (03Merged) 10jenkins-bot: Fix "attachted" typo in nonexistent-article message [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/470307 (https://phabricator.wikimedia.org/T201491) (owner: 10Zoranzoki21) [09:27:12] (03CR) 10Urbanecm: [C: 031] "LGTM" [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [12:37:38] I don't quite understand the situation of Python 3 on Tools [12:38:33] If I understand correctly, there is intentionally no python3 virtualenv installed, instead one needs to use kubernetes https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool#Create_a_Python_virtual_environment_for_the_application%27s_external_library_dependencies [12:38:59] But then a series of packages are missing there, such as tesseract [13:12:42] (03CR) 10Urbanecm: [C: 032] Implement list of monuments feature [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470253 (owner: 10Urbanecm) [13:13:13] (03Merged) 10jenkins-bot: Implement list of monuments feature [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470253 (owner: 10Urbanecm) [13:18:59] Nemo_bis: we may need to add additional packages to our base containers in toolforge [13:20:04] There should certainly be virtualenv with python3. I use it in toolforge [13:20:37] Nemo_bis: Does `virtualenv -p python3 ` not work? [13:21:38] I mean to say, I use it on gridengine, which appears to be what you are trying as well [13:36:27] (03CR) 10Hashar: [V: 032 C: 032] Configure l10n-bot and l10n-bot-watcher [labs/tools] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470312 (https://phabricator.wikimedia.org/T208138) (owner: 10Legoktm) [13:40:53] !log shinken deleted shinken-01 instance [13:40:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [13:48:58] gtirloni: \o/ [13:50:54] :) it still lives in shinken-02 with a newer version and jessie.. just enough to give us some breathing room to not rush the prometheus stuff :) [13:52:43] (03PS1) 10Urbanecm: Pass veaction=edit in monuments.html [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470391 [13:52:45] (03PS1) 10Urbanecm: Allow users to pass "campaign" parameter to automatically include a hashtag in edit summary [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470392 [13:53:13] (03CR) 10Urbanecm: [C: 032] Pass veaction=edit in monuments.html [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470391 (owner: 10Urbanecm) [13:53:32] (03Merged) 10jenkins-bot: Pass veaction=edit in monuments.html [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470391 (owner: 10Urbanecm) [13:56:40] (03PS1) 10Urbanecm: Add quote method into templates [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470393 [13:57:59] (03PS2) 10Urbanecm: Allow users to pass "campaign" parameter to automatically include a hashtag in edit summary [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470392 [13:58:10] arturo: can't we just have another container which provides all the packages which have been selected for installation across the years on the exec hosts? [13:59:51] Nemo_bis: I don't know what the policy is right now [14:01:04] ok [14:01:13] bstorm_: thanks, "virtualenv -p python3" seems to work [14:04:41] but then I don't understand what's up with Pillow in python3... the Ubuntu package seems installed but import fails, build in the virtualenv fails as well [14:04:57] maybe I need to try some ancient version of it [14:05:29] Nemo_bis: in which host are you trying? [14:05:47] tools-bastion-03 [14:06:02] Is there a python3-pillow package installed on ubuntu? [14:06:32] pillow uses C libraries, so compiling in a venv can definitely fail for reasons of missing libraries. [14:06:33] no [14:07:03] Nemo_bis: I don't see a pillow package installed there [14:07:05] "dpkg -s python3-imaging" and "dpkg -s python3-pil" seem to say yes [14:07:19] pillow is in in python3-pil [14:07:29] I thought pil and pillow were different [14:07:46] seems it includes both [14:07:55] :P [14:08:10] heh [14:08:11] import PIL works [14:08:11] Nice [14:08:36] I get: ImportError: No module named 'PIL' [14:08:46] from $ python3 [14:08:47] Python 3.4.3 (default, Nov 28 2017, 16:41:13) [14:09:00] outside of the virtualenv? [14:09:06] platonides@tools-bastion-03:~$ python3 [14:09:06] Python 3.4.3 (default, Nov 28 2017, 16:41:13) [14:09:06] [GCC 4.8.4] on linux [14:09:06] Type "help", "copyright", "credits" or "license" for more information. [14:09:06] >>> import PIL [14:09:08] >>> [14:09:10] right that was the virtualenv [14:09:18] (03CR) 10Urbanecm: [C: 032] Add quote method into templates [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470393 (owner: 10Urbanecm) [14:09:29] remember that if you want to see external packages from inside a virtualenv [14:09:37] you need to pass a special parameter [14:09:45] (03Merged) 10jenkins-bot: Add quote method into templates [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470393 (owner: 10Urbanecm) [14:09:47] --system-site-packages [14:09:47] Give access to the global site-packages modules to the virtual [14:09:50] environment. [14:13:44] ok, so I did virtualenv -p python3 --system-site-packages [14:13:51] and now "import PIL" works [14:13:52] (03PS3) 10Urbanecm: Allow users to pass "campaign" parameter to automatically include a hashtag in edit summary [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470392 [14:14:08] let's see if pip breaks everything now (I'm installing ocrmypdf) [14:15:10] :) [14:16:11] (03CR) 10Urbanecm: [C: 032] Allow users to pass "campaign" parameter to automatically include a hashtag in edit summary [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470392 (owner: 10Urbanecm) [14:16:35] maybe it worked https://paste.debian.net/1049566/ [14:17:19] (03Merged) 10jenkins-bot: Allow users to pass "campaign" parameter to automatically include a hashtag in edit summary [labs/tools/map-of-monuments] - 10https://gerrit.wikimedia.org/r/470392 (owner: 10Urbanecm) [14:17:52] No errors so far! Thanks bstorm_ Platonides \o/ [14:17:53] $ ocrmypdf --version [14:17:53] 4.5.6 [14:20:26] btw I'm also going to install jbig2enc, AFAICS it's all free now https://ocrmypdf.readthedocs.io/en/latest/jbig2.html#jbig2 [14:22:23] (03PS1) 10Vgutierrez: secret: Add dummy LE ACMEv2 staging private key for certcentral2001 [labs/private] - 10https://gerrit.wikimedia.org/r/470403 [14:24:41] I forgot that we have tesseract < 3.04, so there's no leptonica; oh well, no big deal [14:25:48] (03PS2) 10Vgutierrez: secret: Add dummy LE ACMEv2 staging private key for certcentral2001 [labs/private] - 10https://gerrit.wikimedia.org/r/470403 (https://phabricator.wikimedia.org/T208212) [14:26:24] (03CR) 10Vgutierrez: [V: 032 C: 032] secret: Add dummy LE ACMEv2 staging private key for certcentral2001 [labs/private] - 10https://gerrit.wikimedia.org/r/470403 (https://phabricator.wikimedia.org/T208212) (owner: 10Vgutierrez) [14:50:16] bstorm_: is this request for packages to be installed on SGE correct? https://phabricator.wikimedia.org/T204422 [14:51:43] Nemo_bis: I think it is, but it'll be stuck until we finish deprecating Trusty. We are moving to Debian Stretch in the grid as is, but it's a big effort. [14:52:31] It has to be done soon, at least because of the EoL :) [14:53:17] Ah ok, makes sense [14:53:46] For packages that will install on Trusty, those might be doable in general, though. [14:54:07] I think these are all available [14:54:18] Ok cool [15:18:26] Nemo_bis: `dpkg-query: package 'python3-pytest' is not installed and no information is available` *hangs head in shame* [16:31:48] !log wikilabels:f770a79 going prod [16:31:49] Amir1: Unknown project "wikilabels:f770a79" [16:32:02] !log wikilabels wikilabels:f770a79 going prod [16:32:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [16:39:26] Nemo_bis: do you have any hints on https://phabricator.wikimedia.org/T204499#4679304 ? [16:56:50] anyone experienced with OAuth want to help some lost XTools developers? ;) Our login suddenly stopped working over the weekend, with "JWT can't validate". It would seem the access tokens we're getting from the wiki are already expired. I've tried creating a new consumer. Strangely everything works on my local. I'm lost :( [16:57:26] musikanimal: chances are your clock is off [16:58:38] hmm that is a sound theory. The VPS instance is definitely on UTC, let me double check that the application is in the right time zone [17:00:06] !log tools Ran grid engine orphan process kill script from T153281 [17:00:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:00:10] T153281: webgrid-lighttpd queues kill OOM jobs with SIGKILL leaving php-cgi processes behind - https://phabricator.wikimedia.org/T153281 [17:03:10] it's in UTC :( [17:07:04] hm [17:07:18] musikanimal, I don't think he was talking about timezones [17:07:23] everything should be running UTC [17:07:26] oh, ha [17:07:33] I think he meant the actual accuracy of the clock [17:08:30] yeah, so it's failing at https://github.com/wikimedia/mediawiki-oauthclient-php/blob/master/src/Client.php#L344 [17:09:02] yeah, usually it's due to clock drift [17:09:04] `$identity->iat` is about 3 minutes ahead of `$now + static::IAT_TOLERANCE` [17:09:38] hmmm [17:09:41] but what's odd is IAT_TOLERANCE is `2`. That amounts to two milliseconds, I think? in Unix time [17:10:11] I actually don't know if we use ntp in labs [17:10:12] no, that should be seconds [17:10:26] err, no yes you're correct it's seconds [17:13:28] if you want to see the output, try https://xtools-dev.wmflabs.org/login (doesn't spit out anything sensitive) [17:14:22] so `$identity->iat` is 3 seconds ahead of `$now + IAT_TOLERANCE` (not minutes, which I said above) [17:15:28] how can I correct the clock? [17:16:00] andrewbogott, ^ maybe you know how clocks are supposed to work in our setup? [17:17:12] Krenair: I don't know a lot offhand. In theory the clock should re-sync itself periodically. I did see issues of clock drift years ago but haven't seen that in quite a while. [17:17:23] I think there's a way to force a re-sync. [17:17:42] do instances get their time from the hypervisors? [17:17:55] As far as I know it comes from ntp like any other server [17:18:03] I took a quick look on an instance and didn't find an ntp service or ntpdate [17:18:06] but maybe that's not true anymore on e.g. Stretch. [17:18:19] hm yeah I'm looking at a stretch box [17:18:20] ok — I can research then, if you make me a task. Want to finish up the thing i'm in the middle of first. [17:18:28] cool [17:18:42] this one is on eqiad1-r, if that means anything [17:18:49] no ntp stuff on random jessie box either... [17:20:10] we switched from ntp to systemd-timesyncd btw [17:20:12] musikanimal, for the record this is xtools-prod03? [17:20:15] aha [17:20:27] yes prod03 and dev04 [17:20:30] thanks godog [17:20:50] Hm, you're seeing clocks that are head of UTC on eqiad1? Or behind? [17:20:51] musikanimal, can you 'sudo service systemd-timesyncd status' on prod03? [17:21:05] np Krenair, got confused too at the beginning [17:21:10] let me try on dev04 first :) [17:21:31] I just did a spot-check of a recently-migrated VM and its clock looks right [17:23:08] heh we use prod dns servers for ntpd [17:23:14] nice [17:23:14] Krenair: I ran that, didn't fix it. I'm seeing `Timed out waiting for reply from dns2002.wikimedia.org` [17:23:28] yeah no it shouldn't change anything :) [17:23:30] But that is interesting. [17:23:36] I suspect the problem is region-related. [17:23:44] I bet the prod DNS servers won't let the new region talk NTP [17:23:56] timed out on dns1001 too [17:23:57] Here's what I got in the main deployment [17:24:00] Oct 29 17:22:35 deployment-deploy01 systemd-timesyncd[13990]: Synchronized to time server 208.80.154.10:123 (dns1001.wikimedia.org). [17:24:08] Gonna open a subtask of your ticket for Andrew [17:25:09] that timeout is very interesting! [17:26:00] This is going to be another example of the old region having worked by accident because of being in 10.x [17:26:17] darn [17:26:19] https://phabricator.wikimedia.org/T208244 [17:26:23] yeah [17:28:51] standard::ntp::timesyncd is not showing anything special for "labs" so yeah I'd put money on the 10.* being allowed and the new cloud private range being blocked on the ntp service side [17:29:10] FWIW this has inherited high priority from the xtools ticket, not sure if it should be UBN or not [17:29:35] i would think it would be UBN [17:29:47] godog: do you have an opinion about whether the right solution is 1) use a public non-wmf NTP server 2) run our own cloud-specific NTP server 3) open up the firewall so that cloud VMs can use the standard server? [17:29:53] 3 is easiest :) [17:30:47] heh: [17:30:48] class role::recursor { [17:30:48] require role::dnsrecursor [17:30:48] require role::ntp [17:30:49] } [17:31:24] Ah, so we're running ntp already? [17:31:26] * andrewbogott doublechecks [17:31:28] role::ntp contains profile::ntp, which has a ferm::service permitting UDP traffic to the NTP port [17:31:33] andrewbogott, this is the prod recursor role [17:31:46] I don't know if it applies to the labs DNS recursors [17:31:54] ok, checking [17:32:21] though I don't know the extent to which it necessarily makes sense to co-locate ntpd with DNS [17:32:44] I imagine in prod this is done because there are DNS servers in every DC and something needs to serve time locally? [17:33:38] Interestingly this ferm::service doesn't specify saddr [17:33:46] Which makes me wonder if a firewall along the way has a rule for it [17:33:58] wouldn't surprise me [17:36:54] thank you all for looking into this! [17:42:19] I'm actually wondering what's involved in running ntpd properly [17:43:31] we just have to be able to handle some port 123 UDP traffic and talk to some upstream external NTP servers right? [17:44:48] andrewbogott: indeed 3 is the easiest, though I think pointing VMs to cloud-specific hosts is going to be more future-proof while we're at it [17:45:15] if you're assuming vms can/shoudl always be able to talk to the internet then also 1) would work [17:45:35] godog: Option 2 will probably involve a $::realm check in standard/manifests/ntp.pp; can you live with that? [17:47:29] wouldn't hiera do the trick based on realm? anyways I'm not the ntp gate(time?)keeper but we should discuss on phab [17:47:32] also I gotta go! [17:48:25] godog: there's no hiera used /at all/ in the ntp setup currently [17:48:31] So it would be a big refactor I think [17:48:52] who is the ntp timekeeper? [17:55:02] * andrewbogott puts up some strawman patches [17:55:36] thanks andrewbogott! [17:55:50] i also get the timeout when looking at systemd-timesyncd [17:57:16] andrewbogott, do we want to go down the route of shoving this onto the cloud support physical hosts? [17:57:34] maybe! [17:59:37] musikanimal, in the mean time [17:59:51] presumably you need xtools back up and working ASAP and preferably without migrating back to the main region [18:00:37] I wonder if there's some way for us to point systemd-timesyncd at a public NTP server [18:03:06] musikanimal, maybe you can disable puppet, change /etc/systemd/timesyncd.conf to have Servers=0.us.pool.ntp.org and try restarting the systemd-timesyncd service? [18:03:34] I'm pretty afraid of puppet [18:03:51] unless you have a puppetmaster in which case, modules/base/templates/timesyncd.conf.erb [18:03:58] hm [18:04:05] wikimedia servers are a bad place to be scared of puppet :p [18:04:12] XTools login is only used so you can see your own restricted stats https://xtools.readthedocs.io/en/stable/opt-in.html [18:04:20] not a huge deal, but people are complaining [18:04:50] yeah, I do need to learn the puppets [19:37:37] !log shinken restart ircecho which I found broken with https://phabricator.wikimedia.org/P7736 [19:37:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [19:54:59] okay it's a bit broken [19:58:34] brb [20:17:22] back [20:25:07] !log shinken made /var/log/ircecho/* files world-readable so ircecho (running as 'nobody') can read them for output to IRC (they were written by the shinken user) [20:25:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken/SAL [21:45:48] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 [21:45:51] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470504 (owner: 10QChris) [21:46:32] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 [21:46:35] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/ipwatcher] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470505 (owner: 10QChris) [21:53:05] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 [21:53:08] (03CR) 10QChris: [V: 032 C: 032] Allow “Gerrit Managers” to import history [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470506 (owner: 10QChris) [21:53:29] (03PS1) 10QChris: Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 [21:53:32] (03CR) 10QChris: [V: 032 C: 032] Import done. Revoke import grants [labs/tools/gerrit-newcomer-bot] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/470507 (owner: 10QChris) [23:07:59] (03PS1) 10Urbanecm: Use __dir__ constant for loading of config [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470513 [23:09:16] (03PS1) 10Urbanecm: Make monitor.py work with SqlAlchemy [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470514 [23:10:43] (03CR) 10Urbanecm: [V: 032 C: 032] Make monitor.py work with SqlAlchemy [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470514 (owner: 10Urbanecm) [23:11:11] (03CR) 10Urbanecm: [V: 032 C: 032] Use __dir__ constant for loading of config [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470513 (owner: 10Urbanecm) [23:17:08] (03PS1) 10Urbanecm: Fix a typo [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470515 [23:19:34] (03CR) 10Urbanecm: [V: 032 C: 032] Fix a typo [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470515 (owner: 10Urbanecm) [23:24:26] (03PS1) 10Urbanecm: Add support for tox [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470520 [23:27:44] (03PS2) 10Urbanecm: Add support for tox [labs/tools/ipwatcher] - 10https://gerrit.wikimedia.org/r/470520 (https://phabricator.wikimedia.org/T208277)