[00:00:38] milimetric: I think the python code needs to be installed in the venv [00:00:54] I directly changed it in the venv path [00:01:00] That worked [00:01:27] I think there's a setup.py script? [00:09:00] thanks :) I never really worked on it after all these fancy changes [00:32:10] milimetric: the setup.py thing has been there forever [00:32:18] It's one thing I didn't get to killing [00:32:23] no the venv [00:32:29] Aah yes [00:32:37] I just did a fab deploy and that's fine [00:32:43] Oh yes cool [00:32:48] but the requirements.txt is messed up so it upgraded a bunch of stuff and broke everytihng [00:32:50] sig [00:32:51] h [00:38:53] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [00:39:09] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [00:48:09] 10Tools: Tool "wikifactmine-api" loads assets from swagger.io - https://phabricator.wikimedia.org/T173063#3517642 (10zhuyifei1999) [00:50:19] 10Tools: Tool "wikimap" loads Google Maps - https://phabricator.wikimedia.org/T173064#3517657 (10zhuyifei1999) [00:50:45] 10Tools: Tool "wikimap" loads Google Maps - https://phabricator.wikimedia.org/T173064#3517674 (10zhuyifei1999) The tool maintainer is DB111. I cannot find their Phabricator username. [00:50:56] 10Cloud-VPS, 10Huggle: huggle.huggle.eqiad.wmflabs does not have puppet installed - https://phabricator.wikimedia.org/T166588#3517675 (10bd808) >>! In T166588#3517345, @Petrb wrote: > instance is down Thank you. :) [00:51:05] ok, madhuvishy, fixed it, just needed a simple session.close() in the right spot [00:51:16] thanks very much for the fancy tools, they helped a lot [00:51:30] and I never really got to do a real firefight with them, it was a pleasure [00:53:41] !log tools.mrmetadata Started webservice; asked for on irc by Krinkle [00:53:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mrmetadata/SAL [00:53:48] 10Tools: Tool "wikinity" loads assets from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T173065#3517676 (10zhuyifei1999) [00:54:10] 10Tools: Tool "wikinity" loads assets from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T173065#3517690 (10zhuyifei1999) The other tool maintainer is S3w3nofficial. I cannot find their Phabricator username. [00:54:55] 10Tools: Tool "wikinity" loads assets from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T173065#3517691 (10zhuyifei1999) This same ticket also applies to tool "wikinity-test" [01:00:38] 10Tools: Tool "wikistats" loads assets from Amazon AWS - https://phabricator.wikimedia.org/T173066#3517694 (10zhuyifei1999) [01:03:24] milimetric: yay I'm glad :) [01:05:48] 10Tools: Tool "wikipedia-fetch-content" loads jquery and bootstrap from code.jquery.com and bootstrapcdn - https://phabricator.wikimedia.org/T173067#3517709 (10zhuyifei1999) [01:09:08] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [01:11:34] 10Tools: Tool "wikitrends" loads cc-by buttons from creativecommons and licensebuttons.net - https://phabricator.wikimedia.org/T173068#3517726 (10zhuyifei1999) [01:16:23] 10Tools: Tool "wikitrends" loads cc-by buttons from creativecommons and licensebuttons.net - https://phabricator.wikimedia.org/T173068#3517740 (10zhuyifei1999) The tool maintainer is Johang / johan.gunnarsson. I'm unable to find their Phabricator username. [01:19:41] PROBLEM - Puppet errors on tools-exec-1415 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [01:22:52] 10Tools: Tool "wikiviz" loads assets from google, cloudflare, and bootstrapcdn, mixed http/https - https://phabricator.wikimedia.org/T173069#3517741 (10zhuyifei1999) [01:34:16] 10Tools: Tool "wikiwatchdog" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T173071#3517768 (10zhuyifei1999) [01:34:37] 10Tools: Tool "wikiwatchdog" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T173071#3517783 (10zhuyifei1999) The tool maintainer is Sonet. I cannot find their Phabricator username. [01:35:41] 10Tools: Tool "wle" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T173072#3517784 (10zhuyifei1999) [01:38:03] 10Tools: Tool "wlm-maps" loads tiles from OSM - https://phabricator.wikimedia.org/T173073#3517799 (10zhuyifei1999) [01:41:16] 10Tools: Tool "wscontest" loads fonts from google - https://phabricator.wikimedia.org/T173074#3517815 (10zhuyifei1999) [01:47:42] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3517839 (10zhuyifei1999) a:05zhuyifei1999>03None Done. Special cases from P5825 that I did not create a task: ``` https://tools.wmflabs.or... [01:54:40] RECOVERY - Puppet errors on tools-exec-1415 is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:31] bd808: Thanks [02:04:42] (8 o clock) [02:04:56] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [02:44:57] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [03:43:53] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [03:52:14] PROBLEM - Puppet errors on tools-exec-1405 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [04:17:11] RECOVERY - Puppet errors on tools-exec-1405 is OK: OK: Less than 1.00% above the threshold [0.0] [05:08:51] (03PS2) 10BryanDavis: Change ToolInfo authors to list of strings [labs/striker] - 10https://gerrit.wikimedia.org/r/370134 (https://phabricator.wikimedia.org/T149458) [05:08:53] (03PS2) 10BryanDavis: Encode toolinfo.json as utf8 [labs/striker] - 10https://gerrit.wikimedia.org/r/370135 [05:08:55] (03PS2) 10BryanDavis: Change default mysql collation to utf8mb4_unicode_ci [labs/striker] - 10https://gerrit.wikimedia.org/r/370143 [05:44:33] !log tools.commons-delinquent Disabled crontab jobs and stopped demon due to commons vandalism issue [05:44:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.commons-delinquent/SAL [06:58:38] PROBLEM - Puppet errors on tools-webgrid-generic-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:02:39] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:02:47] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:03:09] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:04:52] PROBLEM - Puppet errors on tools-exec-1403 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:33:37] RECOVERY - Puppet errors on tools-webgrid-generic-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:41] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:37:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:39:54] RECOVERY - Puppet errors on tools-exec-1403 is OK: OK: Less than 1.00% above the threshold [0.0] [07:43:08] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [08:29:06] PROBLEM - Puppet errors on tools-worker-1014 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [08:56:35] !help Why is my ssh key constantly refused by the server? [08:56:35] sau226: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [08:58:34] Thanks [09:07:28] sau226: did you upload the public key? [09:08:00] Yeah [09:08:36] Put it on openstack zhuyifei1999 [09:09:08] RECOVERY - Puppet errors on tools-worker-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [09:09:46] sau226: what's in your ssh debug log? [09:10:02] i.e. ssh -vvv [09:13:21] I ust putty [09:14:03] * zhuyifei1999_ don't know about putty [09:14:39] Server refused our key [09:15:15] https://wikitech.wikimedia.org/wiki/Help:Putty [09:15:36] https://wikitech.wikimedia.org/wiki/Help:Access_to_instances_with_PuTTY_and_WinSCP [09:16:22] also https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP [09:16:29] Thanks [09:16:37] My keys probably 4096 [09:31:12] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3518040 (10Incola) [09:31:14] 10Tools: Tool "personabot" loads assets from bootstrapcdn, ajax.googleapis.com, and w3.org - https://phabricator.wikimedia.org/T172822#3518037 (10Incola) 05Open>03Resolved a:03Incola Thanks, solved as suggested. [09:37:41] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3518044 (10RLuts) [09:37:43] 10Tools: Tool "wle" loads jquery from code.jquery.com - https://phabricator.wikimedia.org/T173072#3518042 (10RLuts) 05Open>03Resolved Done [11:12:46] !log quarry deployed 2834160 on quarry-main-01 [11:12:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [11:32:37] If i have a job with status dt (ie I've run qdel -j ) but the job wont disapear, what's the next suitable action? [12:41:27] looks like the bugstatus tool is down [12:42:05] getting a 500 error code [13:05:09] 10Tools: Tool "raun" loads fonts and analytics from google - https://phabricator.wikimedia.org/T172825#3518271 (10Kenrick95) Hi there, yes I will keep in mind to replace font from Font CDN. And I'll remove usage of Google Analytics, but is there any quick solution to track, simply, how many people visited the to... [13:09:23] 10Tools: Tool "raun" loads fonts and analytics from google - https://phabricator.wikimedia.org/T172825#3518293 (10zhuyifei1999) uh you can parse access.log or make php write to a file/database. [13:27:43] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3518339 (10Kenrick95) [13:27:45] 10Tools: Tool "adas" loads fonts from fonts.googleapis.com - https://phabricator.wikimedia.org/T172271#3518338 (10Kenrick95) 05Open>03Resolved [13:28:38] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [13:31:37] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3518346 (10Kenrick95) [13:31:39] 10Tools: Tool "raun" loads fonts and analytics from google - https://phabricator.wikimedia.org/T172825#3518344 (10Kenrick95) 05Open>03Resolved Yeah, but both solutions provided are not simple changes from my side. Anyway, I've replaced/removed the related assets/scripts. [13:48:23] 10Tools: Tool "suggestor" loads botstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172962#3518396 (10Arlolra) p:05Triage>03Low [13:59:55] fluff: one of the admins can try to delete by job id if you paste it here [14:03:38] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [14:26:39] !log git upgrading gerrit-test3 to gerrit 2.14.3 pre [14:26:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [14:34:05] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3518503 (10Niedzielski) [14:34:43] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3512921 (10Niedzielski) [15:05:21] 10Horizon, 10Marvin: Add @Niedzielski to the reading-web-staging group in horizon - https://phabricator.wikimedia.org/T172911#3512921 (10Krenair) Cloud VPS project membership doesn't work like this - it's not managed through Horizon, it's managed through wikitech, and it's usually done by the project's adminis... [15:14:15] 10Striker, 10User-bd808: Can’t log into Striker, potentially because of accents in logins - https://phabricator.wikimedia.org/T172949#3518588 (10bd808) > so I'm not sure why it is even shown that way in ldapsearch It's just a UI/UX decision that was made in ldapsearch to make dealing with arbitrary binary data... [15:17:59] fluff: is your stuck delete still stuck? I can force delete it for you if you give me the job id [15:42:33] !log recreating phab-01 with stretch [15:42:33] paladox: Unknown project "recreating" [15:42:42] !log phabricator recreating phab-01 with stretch [15:42:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [16:04:39] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3518663 (10Tarrow) [16:04:41] 10Tools: Tool "wikifactmine-api" loads assets from swagger.io - https://phabricator.wikimedia.org/T173063#3518661 (10Tarrow) 05Open>03Resolved Thanks, I didn't actually realise the default swagger did this out of the box. For other people who find this the answer is to set `validatorUrl: null` in the Swagge... [16:06:10] bd808: 7441503 and 7443268 [16:06:14] bd808: Thank you! [16:08:42] fluff: bd808 deleted those two [16:09:06] !log tools qdel -f -j 7441503 [16:09:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:10:48] thanks chasemp [16:14:40] 10Data-Services, 10DBA: LabsDB infrastructure pending work - https://phabricator.wikimedia.org/T153058#3518678 (10jcrespo) [16:15:06] !log rcm Xenon: Running update.sh (again) to include the Security fix [16:15:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [16:16:44] !log phabricator recreating puppet-phabricator (accidentily removed the wrong cert which is for puppet) was meant to remove phab-01. [16:16:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [17:02:28] 10Tools: Tool "wembedder" loads jquery from cloudflare - https://phabricator.wikimedia.org/T172968#3518847 (10Fnielsen) It should be fixed now, e.g., https://tools.wmflabs.org/wembedder/most-similar/Q76#language=en [17:04:25] any able to poke at why phabricator-bug-status is throwing 500 error messages? [17:13:37] Betacommand: it's flapping somehow... 50% of the requests I send 500 and 50% are 200 [17:13:42] * bd808 pokes a bit more [17:17:17] !log tools.phabricator-bug-status Restarted webservice due to intermittent 500 errors [17:17:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabricator-bug-status/SAL [17:25:42] bd808 and chasemp - thank's a lot! [17:26:29] bd808: thanks [17:27:26] (03PS1) 10BryanDavis: Fix author tracking for toolinfo create/edit [labs/striker] - 10https://gerrit.wikimedia.org/r/371504 (https://phabricator.wikimedia.org/T149458) [17:39:38] 10Tool-Pageviews: Userviews not working for some users - https://phabricator.wikimedia.org/T173118#3518955 (10MusikAnimal) [17:57:58] fastcci seems to be giving me 502's [17:58:39] oh, maybe i'm just sending it too big a thing [18:22:21] 10Tools, 10Toolforge-standards-committee, 10Privacy: Hunt for Toolforge tools that loads resources from third party sites - https://phabricator.wikimedia.org/T172065#3519218 (10Arlolra) [18:22:23] 10Tools: Tool "suggestor" loads botstrap from bootstrapcdn - https://phabricator.wikimedia.org/T172962#3519216 (10Arlolra) 05Open>03Resolved Fixed by https://github.com/wikimedia/labs-tools-suggestor/commit/ad2b018a9c4d34328cfdca47f5a3aeda64ba1b3e and deployed. [19:14:33] (03PS1) 10BryanDavis: Use renamed Maintainer.cn attribute [labs/striker] - 10https://gerrit.wikimedia.org/r/371518 [20:30:07] is anyone else's tools periodically hanging for extended periods of time? this has been happening on and off for the /pageviews suite, but apparently CopyPatrol is also affected [20:30:42] meanwhile the old XTools and a few other tools that I checked aren't hanging [20:32:18] no major code changes were recently made to Pageviews, and none to CopyPatrol [20:33:06] musikanimal: I haven't checked. Which tools specifically? [20:33:36] /pageviews and /copypatrol, as I said [20:34:27] I use UptimeRobot to automatically monitor them, they've been going down for a few minutes at a time, on and off, for the past few days [20:36:07] before then we had 100% uptime for many moons [21:27:38] q: reading over the k8s/Admin docs and thought it would be helpful to have links to existing phabricator tasks where they exist to features listed as to be done. This let's folks quickly dial into the current state. Is this acceptable? Like so: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Control_Plane [23:03:07] 10Tools: Tool "wikidata-primary-sources" loads assets from online.swagger.io - https://phabricator.wikimedia.org/T172970#3519901 (10zhuyifei1999) See also T173063 [23:29:37] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]