[01:57:18] !log tools.svgtranslate-test Switched to canonical domain name. [01:57:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.svgtranslate-test/SAL [07:01:25] !log devtools phabricator-prod-1001 - removing cron for public task dump (though puppet should have removed it) [07:01:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [10:29:00] how can we be notified of a pod restart for a tool hosted on toolforge/k8s? [11:16:04] don-vip: you mean notified like an email or something? [11:16:23] arturo yes [11:16:54] yes, we would like to have a more robust monitoring service for toolforge [11:18:07] some intelligence can be collected from prometheus metrics, for example: https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?orgId=1&var-namespace=tool-fourohfour&refresh=5m [11:18:18] using prometheus alert manager we could generate per tool alerts [11:18:24] but that's something we don't have as of today [11:20:01] ah, nice, I didn't know [11:26:00] arturo: i am using openstack-browser to check which instances use certain classes. normally works great, but just ran into an issue. if i go to https://openstack-browser.toolforge.org/puppetclass/role::simplelap it lists " [11:26:04] officetools.sugarcrm.eqiad.wmflabs [11:26:21] but if i click that and get to https://openstack-browser.toolforge.org/server/officetools.sugarcrm.eqiad.wmflabs then it is "Unknown server 'officetools.sugarcrm.eqiad.wmflabs'. Are you just guessing?" [11:26:35] does it mean it's been deleted but openstack-browser has not learned it yet? [11:27:45] if so then that is also cool with me, since i just want to know nobody uses that role then [11:41:53] mutante: moreover, it seems the project no longer exists [11:47:42] arturo: ok! thanks for checking. for my purpose that's just fine. though maybe i should make a ticket for openstack-browser to be updated and forget deleted projects? [11:54:30] sure [11:54:47] in this case, I think it should forget about that particular role-->instance association [11:55:04] it already forgot about the project and the instance, but no the role association [12:05:53] alright, i'll create it in a bit. thanks! [12:14:09] thanks you :-) [12:24:07] !log toolsbeta added puppet prefix `toolsbeta-email` (T120225) [12:24:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:24:11] T120225: Toolforge: correctly envelope forwarded email - https://phabricator.wikimedia.org/T120225 [12:48:40] !log toolsbeta allocated floating IP `185.15.56.12` for the VM `toolsbeta-email-01` and FQDN `mail.toolsbeta.wmflabs.org` (T120225) [12:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:48:43] T120225: Toolforge: correctly envelope forwarded email - https://phabricator.wikimedia.org/T120225 [12:51:58] arturo: oh, we probably shouldn't have that letsencrypt::cert::integrated stuff in puppet anymore... [12:52:34] I don't think it's been possible to set up new servers using it since November [12:53:01] and we'll stop being able to renew certs issued through it in a year [12:54:16] looks like prod mail servers need to change as well as toolforge mail relays and toolserver_legacy [12:54:19] Krenair: ok! next week I can try figuring out what to update in puppet, or I can leave it to you [12:55:14] I'm about to lunch here [13:04:42] am gonna take a look [13:09:25] actually maybe not prod mailservers, only WMCS ones? [13:55:37] Can i see who is in the group "admins" in Horizon? [13:56:56] i can click around in items under "Identity" but also get errors i am not authorized. first i could still the groups though. but after a few clicks i get forced logout [13:58:05] what i really want is add myself to a few projects for a temp. task to replace role::simplelamp [13:58:14] and then remove myself again after it's done [13:58:25] should i make a ticket? [14:00:56] mutante: pm'd [14:05:23] Hey folks. I have a volunteer who I am working with on some of our "label extraction" utilities in ORES. Essentially these things process XML dump files looking for talk page templates. [14:05:36] We'd like to access dump files from a cloud VPS instance. What's the best way to do that? [14:09:44] halfak: I can just flip a switch to make dumps available on a given project. It should be /mostly/ harmless although adding NFS mounts is always a bit scary [14:09:45] what project? [14:10:04] ores-staging [14:10:11] ok, let's see if I remember how to do this [14:13:37] halfak: is there a phab task for this? If not, do you mind making one? [14:13:50] (good to have a record of why we're doing this) [14:14:01] Right on. Thanks. One sec. [14:15:31] https://phabricator.wikimedia.org/T252204 [14:17:55] halfak: I'd like to get brooke's sign-off on this. After it's merged it'll take a couple of puppet cycles (so about an hour) before the mounts appear on the VMs. [14:18:21] patch is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/595178/ [14:18:27] Makes sense. Thanks for picking this up! [14:18:43] np [14:58:53] !log git deleting project for T236569 [14:58:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [14:58:58] T236569: "git" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236569 [15:59:48] !log signwriting - signwriting-swis - replaced role(simplelamp) with role(simplelamp2) - before that puppet showed error Package 'mysql-server-5.5' has no installation candidate - this should also fix that, though no DB seems to be in use [15:59:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Signwriting/SAL [16:34:25] !log tools.versions Update to bb4c8a1 (D1182) [16:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.versions/SAL [17:29:16] !log tools Building new jessie-sssd base image (T197930) [17:29:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:29:20] T197930: Replace pykube with a library that works with newer Kubernetes APIs - https://phabricator.wikimedia.org/T197930 [18:17:07] !log tools Building all jessie-sssd derived images (T197930) [18:17:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:17:11] T197930: Replace pykube with a library that works with newer Kubernetes APIs - https://phabricator.wikimedia.org/T197930 [18:20:01] hi! [18:20:01] yesterday afternoon I rebooted the server I’m working on (https://wcdo.wmflabs.org/) and I tried to start up nginx again but apparently it does not work. could someone help me take a look at it? [18:20:01] I don’t think it is any change in the code or configuration in the server. I wonder if there has been any other change. Thanks. [18:26:28] mmecor: what is the actual instance? That link is just to a shared HTTP proxy entry point [18:26:47] https://tools.wmflabs.org/admin/tool/wcdo [18:27:22] what should i do to know the instance? [18:27:24] that's a link to a tool, which would not have the url you gave [18:28:08] the tool would be at https://wcdo.toolforge.org/ and/or https://tools.wmflabs.org/wcdo/ [18:28:40] i have been using https://wcdo.wmflabs.org/ all the time [18:28:47] for the past year and half [18:29:30] mmecor: so that means you are using https://openstack-browser.toolforge.org/project/wcdo [18:29:45] yes, that's right [18:29:46] and wcdo.wcdo.eqiad.wmflabs is the instance name [18:30:16] ok, i'm going to keep this. i didn't know the instance name. i see it's the same i use to connect with ssh. [18:30:17] Toolforge tools are different entirely from Cloud VPS projects [18:32:03] mmecor: on your instance, /var/log/nginx/error.log shows that something is wrong with "uwsgi://unix:/srv/wcdo/src_viz/dash_apps.sock" [18:32:50] dash_apps.ini is the configuration of the uwsgi [18:33:24] the .sock might be the socket it creates. but i did not do any change to the dash_apps.ini config neither to the nginx configuration. [18:33:29] the nginx error log is full of timeouts reading data from the uwsgi container [18:33:43] so check the logs for your uwsgi process [18:34:26] tail -500 /var/log/nginx/access.log [18:34:43] that's the nginx log i look at [18:34:59] and that's the uswgi procses log i defined: [18:35:02] tail -100 /tmp/errlog [18:35:33] mmecor: errors from nginx show up in /var/log/nginx/error.log [18:35:50] and that's where I can see it failing to talk to the uwsgi container [18:36:18] which then makes the next thing to look at the uwsgi container's error logs [18:36:57] isn't the uwsgi container's error log that one /tmp/errlog? [18:37:14] I have no idea. [18:37:44] this log is empty. it is created when the uwsgi starts running. [18:37:44] This is your custom setup. I haven't traced the system out [18:38:49] that log is the one i use to check if the uwsgi is up with its workers. let me see if it is the one you mean. [18:40:35] mmecor: It lunch time for me, but keep looking for what is wrong with your uwsgi process. It looks to me like nginx there is fine, but the upstream you are proxying is not [18:41:41] it seems the nginx is fine, yes [18:41:53] because i see some redirects in the nginx config working well [18:42:10] there's sth wrong with the uwsgi [18:42:17] thanks! i'll keep looking at it. enjoy your meal! [19:02:39] !log admin moving tools-k8s-haproxy-2 from cloudvirt1021 to cloudvirt1017 to improve spread [19:02:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:10:09] bd808: i changed the logs of place, cleant the config in the uwsgi and enabled again the uwsgi service. and it works again! :) thanks. [22:06:13] !log tools.indic-wscontest Add number instead of bullet (T252231) [22:06:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-wscontest/SAL [22:06:16] T252231: Indic-wscontest: Add number instead of bullet - https://phabricator.wikimedia.org/T252231 [23:14:05] !log tools.indic-wscontest Use table instead of bullets list for contests (T252257) [23:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.indic-wscontest/SAL [23:14:09] T252257: indic-wscontest: Use table instead of bullets list for contests - https://phabricator.wikimedia.org/T252257