[00:17:53] bd808 hi, users are reporting whois as slow in #wikipedia-en. [00:18:26] ok [00:18:31] Oshwah ^^ [00:18:46] Thank you. [00:18:53] paladox: do you mean https://tools.wmflabs.org/whois/ ? [00:19:08] yup, i think that's what they ment [00:19:55] https://tools.wmflabs.org/admin/tool/whois [00:20:00] that's the maintainer [00:20:15] Oshwah ^^ [00:20:44] Ah [00:22:41] It looks like the tool was moved to the new Stretch job grid a few days ago, but I don't know if that has anything to do with reports of "slow" [00:24:20] its really just a wrapper that calls out to 3rd party web services [00:24:39] Hello, the WHOIS tool appears to be non-responsive as of now [00:25:23] Kb03 https://tools.wmflabs.org/admin/tool/whois [00:25:28] maintainer is listed there [00:27:30] !log tools.whois Restarted webservice. Looks like someone found a slow IP to look up, handed that URL to a bunch of folks, and eventually locked up all the lighttpd threads with the same slow query [00:27:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.whois/SAL [00:27:56] Thanks bd808 [00:27:58] DDOS by "helping" [00:28:58] thanks bd808 [10:41:42] !log tools flip tools-prometheus proxy back to tools-prometheus-01 and upgrade to prometheus 2.7.1 [10:41:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:47:51] !log deployment-prep shut deployment-prometheus01, unused now [10:47:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [10:56:37] can someone please tell me became of the cronjobs on tools accounts [10:57:23] not to worry [10:59:28] !log tools.openstack-browser Deployed change to show hypervisor in VM list [10:59:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.openstack-browser/SAL [11:03:46] sDrewth, I'm not sure exactly what you mean but it sounds like you've got it figured out :) [11:04:21] yep, one of my co-owners must have made the migration [11:04:42] and removed the cronjob [11:04:48] ah [11:08:30] Krenair, if I had jobs where I pushed with ./.. release=trusty do they now become release=stretch ? [11:09:10] you skipped jessie? [11:09:25] ummm [11:09:52] FYI doesn't look like tools-prometheus can talk to kubelet on tools-workers - T215779 [11:09:53] T215779: Tools prometheus can't talk to kubelet running on tools-worker - https://phabricator.wikimedia.org/T215779 [11:09:54] no idea, just looking at my set and forget cronjob [11:10:44] I think they got rid of the release option for some reason [11:10:58] okay, that makes it easy [11:11:07] it will work or it will crash :-) [11:11:13] I'm not too familiar with tools but "set and forget" sounds really bad [11:11:54] just archiving jobs, and I can see from outside whether they are working [11:12:08] and they always do [14:18:48] !log admin running `maintain-views --all-databases --replace-all` on labsdb1010 [14:18:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:43:50] I'm working on an extended version of the Wikidata Query Service to be able to Query Wikidata History? Is it fine if I keep an anonymous log of the valid queries asked ? [14:44:21] This log would only contain the queries and its order would be shuffled to make association with the HTTP request logs difficult [14:47:29] Tpt[m]: the queries themselves are not an issue I think, it's the IP and UA. If you're behind teh generic reverse proxy the IP should be masked so if you can strip out the UA. Seems OK? Or if it's more complicated than that maybe make a task asking for guidance [14:48:06] I was planning to only log the queries, not the IPs, UA... [14:48:27] and of course not the timestamp [14:48:42] shouldn't be a privacy issue then, at least afa our privacy policy [14:48:54] to my knowledge, unless the queries themselves contain sensitive info? [14:49:56] the query contents could still be an issue, there’s a reason we anonymize them for the official WDQS logs (see https://iccl.inf.tu-dresden.de/web/Wikidata_SPARQL_Logs/en) [14:50:15] e. g. if someone is doing a geospatial search, their location could be part of the query text [14:51:02] (there was also a page on-wiki describing the anonymization schema, but I can’t find it) [14:57:08] indeed [14:57:34] but this endpoint is targetted at doing analytics queries so hopefully no one is going to use it for such use cases [14:57:59] but "hopefully" is definitely not a good argument [14:59:01] Tpt[m]: does this run out of its own project or in toolforge? [14:59:49] it is in its own VPS [15:00:02] NFS is very bad for databases [15:00:17] that's the truth [15:01:24] Tpt[m]: ok, I think this will be OK but maybe a project specific notice page is warranted(?) but if you wouldn't mind logging a task and tagging security-team along with cloud-vps I think we can settle on something officially so down the line we remember [15:03:32] thanks Lucas_WMDE that's really helpful context too [15:05:35] Done: https://phabricator.wikimedia.org/T215804 [15:05:43] Thank you for the advice! [15:08:00] np, thanks for trying to be a good steward (pun intended?) for other ppls data [15:23:19] Lucas_WMDE: I quoted you on task, i didn't figure you would mind but fyi [15:24:20] sure, np [15:28:25] !log admin running `maintain-views --all-databases --replace-all` on labsdb1011 [15:28:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:08:25] !log tools Built tools-manifest_0.17_all.deb and published to aptly repos (T107878) [18:08:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:08:30] T107878: Limit webservice manifest restarts - https://phabricator.wikimedia.org/T107878 [18:12:20] !log tools Upgraded tools-manifest on tools-sgecron-01 to v0.17 (T107878) [18:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:13:16] !log admin cleaned old metrics data in labmon1001 T215417 [18:13:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [18:13:18] T215417: labmon1001: archive-instances not working - https://phabricator.wikimedia.org/T215417 [18:25:13] !log tools Built tools-manifest_0.18_all.deb and published to aptly repos (T107878) [18:25:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:25:18] T107878: Limit webservice manifest restarts - https://phabricator.wikimedia.org/T107878 [18:26:44] !log tools Upgraded tools-manifest on tools-sgecron-01 to v0.18 (T107878) [18:26:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:28:52] !log tools.bd808-test2 Manually killed grid job to test webservicemonitor [18:28:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bd808-test2/SAL [18:57:47] !log tools Built tools-manifest_0.19_all.deb and published to aptly repos (T107878) [18:57:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:57:51] T107878: Limit webservice manifest restarts - https://phabricator.wikimedia.org/T107878 [18:57:52] !log tools Upgraded tools-manifest on tools-sgecron-01 to v0.19 (T107878) [18:57:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:08:59] !log tools Upgraded tools-manifest on tools-cron-01 to v0.19 (T107878) [19:09:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:09:03] T107878: Limit webservice manifest restarts - https://phabricator.wikimedia.org/T107878 [20:06:48] !log tools Ran apt-get clean on tools-sgebastion-07 since it was running out of disk (and lots of it was the apt cache) [20:06:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:05:27] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1426.tools.eqiad.wmflabs [22:05:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:06:18] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1424.tools.eqiad.wmflabs [22:06:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:07:34] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1427.tools.eqiad.wmflabs [22:07:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:18:25] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1428.tools.eqiad.wmflabs [22:18:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:21:36] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1413.tools.eqiad.wmflabs [22:21:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:23:45] !log tools sudo exec-manage depool tools-webgrid-lighttpd-1401.tools.eqiad.wmflabs [22:23:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:34:15] !log tools Decommissioned tools-webgrid-lighttpd-14{01,13,24,26,27,28} [22:34:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:57:15] !log tools Shutoff tools-webgrid-lighttpd-14{01,13,24,26,27,28} via Horizon UI [22:57:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL