[00:01:50] it looks like profile::dumps::distribution::datasets::fetcher would get included on labstore1006 and co that is responsible for pulling some files from stat1007.eqiad.wmnet [00:05:28] specifically it sets up dumps::web::fetches::stats which sets up dumps::web::fetches::analytics::job['pageview' [00:05:44] which includes a kerberos::systemd_timer that should be calling rsync [00:05:49] looks like pretty new stuff [00:06:51] mturked, to be clear have you looked at this page before and got more recent hourly pageview dumps? [00:07:14] yes, generally you would get these values updated more promptly [00:07:42] currently it is UTC Dec 16 00:00:07 [00:08:04] so it is really odd that the last file dump was Dec 14 18:00:00 [00:09:12] yeah [00:09:35] I'm gonna suggest we open a task on phab and ping ariel and elukey [00:09:47] not sure how to do it :| [00:11:30] should we open an issue in the public github? [00:12:32] I am mildly suspicious of the fact that https://gerrit.wikimedia.org/r/c/operations/puppet/+/556681 was merged 48h before it stopped working [00:13:32] looks like there's some kerberos stuff in the mix and I don't know what effect that might have [00:13:34] mturked, github?? [00:14:09] !help Hi, I asked about question related to instance. Now I see debian-10.0-buster (deprecated 2019-12-15) still, but when I click on Launch instance and on Source there is debian-10.0-buster without deprecation notice. What I should do? [00:14:10] Zoranzoki21: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [00:16:45] Krenair gh: tmoon [00:17:01] mturked, what is that? [00:17:14] github [00:17:29] I'm not sure I understand [00:17:45] I don't see difference https://prnt.sc/qbaa34 but this confused me [00:18:08] Zoranzoki21, you mean you see the debian buster deprecated image on an existing instance but can't launch a deprecated instance? this is expected, people shouldn't launch instances with deprecated images [00:18:34] nvm sorry, I was proposing whether we should report is as an issue in the public github repo: https://github.com/wikimedia/analytics-wikistats [00:18:44] deprecated images should not show in the list at instance creation time under normal circumstances [00:18:48] but I am mostly clueless about how to report issues [00:19:04] mturked, as a general rule problems with wikimedia things should be reported on Phabricator. [00:19:12] gotcha [00:19:17] Krenair: No. I mean why I see for old instances debian-10.0-buster with deprecation notice, but for new created before ~2 minutes I see debian-10.0-buster without deprecation notice [00:19:36] I am not familiar with that repository but it does look like a mirror of Gerrit, not it's own writeable repository that people directly maintain and monitor [00:20:01] (it also has issues disabled) [00:20:13] Zoranzoki21, presumably the live non-deprecated image changed in that ~2 minute window [00:20:41] mturked, anyway right now I think this should be reported somewhere, I'm just not sure if we should make a new one or comment on https://phabricator.wikimedia.org/T234229 [00:20:58] I think I'll make a new one [00:21:51] yea although should probably also leave a note on that T234229 as we suspect it caused the issue [00:21:52] T234229: Shorten the time it takes to move files from hadoop to dump hosts by Kerberizing/hadooping the dump hosts - https://phabricator.wikimedia.org/T234229 [00:22:01] yep [00:24:06] mturked, opened https://phabricator.wikimedia.org/T240803 [00:24:26] thanks very much! [00:24:52] I don't know what priority they assign to tasks regarding hourly data that's stopped flowing [00:28:39] https://phabricator.wikimedia.org/T240518 [00:29:22] also seems like there might be some infra issue causing the job queue to grow out of control [00:29:46] that'll be the MW job queue, doubt it has anything to do with scheduling pageview dumps. IIRC MW doesn't even know anything about all page views as varnish sits in front and doesn't send all requests to it [04:28:24] andrewbogott: can I have a hard reboot on encoding04.video.eqiad.wmflabs? I saw df -h hangs and some D-state processes so went for `sudo reboot`, after a while, still can't ssh in [04:57:40] zhuyifei1999_: I'm afk but will do in an hour or so... Ping me if I forget [04:57:49] ok thanks [05:06:46] andrewbogott: no need anymore. I think it timed out waiting for d-state or something, so it rebooted normally [05:10:50] thanks anyways [05:52:31] zhuyifei1999_: ok! [06:16:02] Following regarding this issue of not having wiki pageview data dump (https://phabricator.wikimedia.org/T240803 and https://phabricator.wikimedia.org/T240815#5743206)--is it possible to have direct access to the log files? [06:16:15] e.g. hdfs://analytics-hadoop/wmf/data/raw/webrequest/webrequest_text/hourly/2019/12/14 etc ? [06:16:49] for a research project I am doing it would be incredibly helpful to have dependable daily raw data [13:59:05] !log testlabs powering down `puppet-stretch-test` VM to test stuff related to T240851 [13:59:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [13:59:08] T240851: CloudVPS: stretch base images fails to boot - https://phabricator.wikimedia.org/T240851 [16:42:08] hey, I'm trying to create a new vagrant instance (visualeditor-test3.wmflabs.org), but I get an error when I do `vagrant up` [16:42:37] `here was an error executing ["sudo", "/usr/bin/env", "lxc-start", "-d", "--name", "mediawiki-vagrant_default_...` [16:42:43] *There [16:44:48] edsanders: when i've seen that before vagrant generated an LXC config that was invalid. does `sudo lxc-ls` say the config is bad? [16:45:09] it will either say it's bad, or just list the valid named container [16:45:31] just lists the name of the container [16:45:38] hmm, so not the issue i've run into :( [16:45:53] I've done this twice - after it first failed, I deleted the whole instance and started again [16:47:06] all I've done is apply `role::labs::mediawiki_vagrant`, run `sudo puppet agent --test --verbose`, logged out/in, then `vagrant up` [16:48:04] i stood up a machine last thursday in cloud, did the same thing and it worked (untill i turned on a half dozen roles) [16:48:39] sorry, not much help :( [16:49:18] I've really done the minimal steps, and reproduced it twice [16:49:29] who should I ask? [16:51:09] we created another instance a week ago, is this a different image now: https://usercontent.irccloud-cdn.com/file/ZT6WNKuG/image.png [16:51:19] ? (not marked as deprecated) [16:52:42] huh, the instance i spun up last week is: debian-10.0-buster (deprecated 2019-12-15) [16:52:42] Looks like that image is new since yesterday [16:52:46] so, new image [16:54:33] edsanders, ebernhardson: the "deprecated" label for base images means that we have made a newer image. Right now however the newest base image is broken, so all the ones that work are showing that "deprecated" label. [16:55:07] it is not really a problem though. Instances made with the "deprecated" images are fine. [16:55:30] bd808: right, the problem seems to be revese. The deprecated one worked fine, but ed's new instance from the new image wont init mwv [16:55:32] bd808: I can only choose the non-deprecated one when I create a new instance [16:55:53] was that a Buster image or Stretch? [16:55:56] and yeah, the new one don't work :) [16:55:58] buster [16:56:05] *nod* ok [16:56:36] I'm catching up on all the stuff right now. Today is my first day back after 4 weeks of vaction :) [16:56:40] *vacation [16:57:26] nice [16:57:28] But I do know that there are some problems like T240851 and T240660 that folks are looking at [16:57:28] T240660: openstack nova: virsh consoles broken in Ocata - https://phabricator.wikimedia.org/T240660 [16:57:29] T240851: CloudVPS: stretch base images fails to boot - https://phabricator.wikimedia.org/T240851 [16:57:52] shall I file a task? [16:59:16] edsanders: sure! I think I can make some time to try spinning up a new mwv on buster today to see if I can spot the problem [17:05:38] Filed T240875 [17:05:38] T240875: Can't start a vagrant instance on a new buster-10.0 instance - https://phabricator.wikimedia.org/T240875 [17:08:29] so we looked into the log file, and seems apparmor is missing, started working after installing... [17:08:58] ah. Could be some change in what is installed in the base image then [17:09:22] like we are just missing some dependencies in Puppet that were provided accidentally before [17:12:07] I'll leave it yours capable hands! [19:05:24] !log tools deployed the maintain-kubeusers operations pod to the new cluster [19:05:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:57:21] i [20:14:19] !log wikilabels created wikilabels-staging-02 and ran fab initialize_staging_server [20:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [20:16:13] !log mediawiki-vagrant Building instance mwv-t240875 to test proposed fix for T240875 [20:16:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [20:16:19] T240875: Can't start a vagrant instance on a new buster-10.0 instance due to missing apparmor - https://phabricator.wikimedia.org/T240875 [21:27:16] Could I bother someone on here about my Toolforge membership request? I fear it might be a while before it's looked at. [21:36:11] Erutuon: I just approved it. :) You will need to log out of https://toolsadmin.wikimedia.org/ and then log back in again for it to notice your new rights. [21:39:54] Thank you! :) [21:42:29] is the mailserver for tools laggy? I've just tried to send a mail to the tools.wmflabs.org address, but I did not receive that mail yet [21:45:28] Sagan: hmmm... the mail queue there is completely empty, which actually surprises me. There are usually at least a few outbound messages in a hold state [21:48:19] bd808, I mean it can be also my normal mail provider web.de, but it's strange that I did not received the mail I've send to me from a different email [21:53:57] Would somebody mind blocking CHANKAICHE on Phabricator. I'm getting spammed. [21:56:47] Sagan: I will keep looking a bit more. I tried emailing myself and haven't seen it in either the queue or my mailbox yet. [21:57:01] ok, thank you [22:04:21] !log tools Added 'ALLOW IPv4 25/tcp from 0.0.0.0/0' to "MTA" security group applied to tools-mail-02 [22:04:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:07:35] Sagan: something is messed up with the external networking for the SMTP server. I'll open a bug and find someone to figure out what is wrong [22:08:12] ok, thanks [22:08:23] so the last change you logged did not solved it? [22:08:33] it did not :/ [22:10:17] ok, so the mail send in the meantime is lost or will be send later? I guess it's lost? [22:11:30] Sagan: outbound mail looks like it is working. In-bound mail is not. For in-bound mail, it should get queued and retried for delivery by whatever MTA the mail is sent out from. [22:12:06] the problem I am seeing is that I can't connect to port 25 on the published MX server from outside the Cloud VPS network [22:26:00] bd808, it looks like I received just now one of the emails I was waiting for [22:26:49] Sagan: and that was an email sent out from a tool to you or an email sent into a tool that then got delivered to you because you are a maintainer? [22:27:39] huh. I just got one of my tests too. But I still can't seem to connect to the MTA directly? [22:27:45] bd808: it was a mail to my personal tools.wmflabs.org address [22:55:10] Sagan: I think things are working as expected now. My telnet test was bogus because the network I was doing it from is apparently blocking port 25 outbound [22:56:59] ah, ok [22:57:39] If you don’t block port 25 those wily kids will go on Usenet or what have you [22:58:15] I vaguely remember that my ISP decided something goofy like that yeah. [22:58:29] like "nobody can send spam if we block port 25 outbound" [22:58:34] which is ... wrong [22:59:00] also, hi hare! [23:03:29] !log toolsbeta updated the kubeadm-config configmap to match the new init file [23:03:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL