[02:18:11] !log tools.stashbot Restarting bot to pick up package updates [02:18:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [03:04:25] !log tools.sal Updated to ba29ca1 [03:04:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL, Master [05:17:35] 06Labs, 10Tool-Labs, 10labs-sprint-119, 06Community-Tech-Tool-Labs, 10Diffusion: Figure out a git hosting solution for tools/kubernetes - https://phabricator.wikimedia.org/T117071#2205968 (10mmodell) @bd808: I agree that we should stick with one integrated system as much as reasonably makes sense, it was... [05:30:03] (03PS1) 10BryanDavis: jsub: Add a ton of comments [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283377 (https://phabricator.wikimedia.org/T132475) [05:30:05] (03PS1) 10BryanDavis: jsub: sexier -continuous runner [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283378 [05:33:17] (03CR) 10BryanDavis: jsub: sexier -continuous runner (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283378 (owner: 10BryanDavis) [05:42:00] (03CR) 10Yuvipanda: jsub: sexier -continuous runner (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283378 (owner: 10BryanDavis) [06:52:33] * madhuvishy can't find the Manage web proxies page on wikitech [06:54:17] madhuvishy: it's in horizon now [08:34:52] PROBLEM - Host tools-bastion-01 is DOWN: CRITICAL - Host Unreachable (10.68.17.228) [08:41:59] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2206223 (10MoritzMuehlenhoff) a:03Papaul [08:42:51] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2193908 (10MoritzMuehlenhoff) That box didn't come back up after a kernel reboot and from what I remember the serial console was dead. Papaul, can you please check the... [11:28:10] tools.wmflabs.org is sluggish; ping time ~90ms or timeout (about 1/3 of the time), http problematic as well [11:28:24] not just for me, either [13:36:18] RECOVERY - Puppet run on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [13:43:54] (03PS88) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [13:48:51] (03CR) 10Ricordisamoa: "PS88 updates the copyright years in demo.js" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [14:03:43] 06Labs, 10Beta-Cluster-Infrastructure, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2206880 (10Krenair) [14:08:10] (03PS89) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [14:19:59] (03CR) 10Ricordisamoa: "PS89 moves from 'data-original-type' attribute of string type to 'original-section' datum of Section type" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [14:33:47] 06Labs: unable to change puppet groups via Special:NovaInstace - https://phabricator.wikimedia.org/T130104#2206984 (10fgiunchedi) 05Open>03Invalid local problem, possibly browser caching [14:51:41] 06Labs, 10Horizon: Add some in-line documentation to the horizon login screen - https://phabricator.wikimedia.org/T132694#2207043 (10Andrew) [14:53:25] (03PS90) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [14:57:35] (03CR) 10Ricordisamoa: "PS90 moves DraggableElement construction from DragHelper.prototype.getDraggable into a new Section.prototype.getDraggable" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [15:05:40] (03PS2) 10BryanDavis: jsub: sexier -continuous runner [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283378 [15:06:09] (03CR) 10BryanDavis: jsub: sexier -continuous runner (031 comment) [labs/toollabs] - 10https://gerrit.wikimedia.org/r/283378 (owner: 10BryanDavis) [15:21:42] bd808: Interesting commits today :D [15:21:45] (03PS3) 10MarcoAurelio: Attempting to clean Stewardbot's code from unneeded TAB and empty spaces [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 [15:22:06] The jsub stuff? [15:22:25] yeah. Interesting title for a commit :D [15:22:45] (03CR) 10MarcoAurelio: [C: 032] Attempting to clean Stewardbot's code from unneeded TAB and empty spaces [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 (owner: 10MarcoAurelio) [15:24:06] * mafk wonders why on zuul labs patches are on the mediawiki queue... [15:24:24] (03Merged) 10jenkins-bot: Attempting to clean Stewardbot's code from unneeded TAB and empty spaces [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/282727 (owner: 10MarcoAurelio) [15:24:54] mafk: Greg said cross-repo depencies, and depecies at the test queues [15:31:30] (03PS91) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [15:36:14] (03CR) 10Ricordisamoa: "PS91 moves EditableDraggableElement construction from Section.prototype.getDraggable to LanguageSection.prototype.getDraggable, div.panel " [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [16:37:42] brion: This extension is likely to get a good trial by fire. One brief message on the enwiki vpt and Nihiltres is already trying to break it. :-) [16:37:51] Hm. Wrong window. [16:41:19] (03PS92) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [16:46:10] 06Labs, 10Tool-Labs, 10labs-sprint-116, 10labs-sprint-117, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#2207419 (10valhallasw) So, let me rephrase the problem statement first. The main problem is permissions on files. There are currently three ways to upload fi... [16:46:16] (03CR) 10Ricordisamoa: "PS92 removes a leftover of the transition from PS38 to PS39" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [17:23:31] 06Labs, 10Tool-Labs, 10labs-sprint-116, 10labs-sprint-117, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#2207592 (10valhallasw) Re: umask option -- I thought there was no way to set it for scp, but apparently there is: http://jeff.robbins.ws/articles/setting-the... [18:07:39] RECOVERY - Puppet run on tools-exec-1408 is OK: OK: Less than 1.00% above the threshold [0.0] [18:07:41] andrewbogott: need to query your brain please [18:07:55] matanya: what's up? [18:08:20] we have video2commons and it can't do upload by url, as labs is not whitelisted [18:08:32] so can't upload to prod directly [18:09:02] is there a way around this ? e.g. setting a subdomain or other means of a solution ? [18:09:25] matanya: why do you need upload by url? [18:09:37] larger limits [18:10:37] 'upload by url' means that you're giving commons a url and it slurps up the file without involving your client? [18:11:09] matanya: hm, I thought chunked uploads allowed files as large as server side ones. I know people were testing large uploads at the hackathon (I remember 2GB? not 100% sure) [18:11:34] yes andrewbogott [18:11:45] matanya: what's an example of a url you would give it? [18:11:57] valhallasw`cloud: it was my patch and work in the hackathon [18:12:02] ah :-) [18:12:10] but it was for upload by url only for now [18:12:48] matanya: what are the limits at the moment? https://commons.wikimedia.org/wiki/Commons:Maximum_file_size suggests it's 4GB, but that's clearly wrong :-) [18:13:39] it is not wrong valhallasw`cloud [18:13:57] "However, users who enable the experimental "Chunked uploads" setting in their user preferences benefit from a limit of 4 GiB "? [18:14:05] yes, indeed [18:14:11] with upload wizard [18:14:37] not with pywikibot or other clients, if they do sync uploads [18:14:52] with async uploads might work [18:15:30] so why can't video2commons do chunked uploads....? [18:15:37] * valhallasw`cloud is lost [18:16:20] valhallasw`cloud: video2commons can do chuncked [18:16:29] it can't do upload by url [18:16:44] and that means it hits timeouts on the mediawiki side [18:16:54] which are 180 secs [18:17:09] to quick for a >2 GB file [18:17:18] so you hit the limit [18:17:38] with upload by url, you don't hit that, as the server does the work, and not the client [18:18:01] hope it is clear now valhallasw`cloud [18:18:32] I don't understand why there's a 180s time limit issue -- is that the time mediawiki needs to put the chunked file together after the final part hsa been uploaded? [18:19:11] that, and the time the server waits for the upload to complete [18:19:22] aaaah. I see. [18:19:34] Weird limit, but at least it makes a bit more sense. [18:22:26] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2207838 (10Papaul) p:05Triage>03Normal [18:24:39] matanya: what's an example of a url you would give it? [18:32:10] andrewbogott: v2c.wmflabs.org ? [18:32:53] matanya: and you can't just whitelist that domain specifically? [18:33:10] looking fir the ticket about that [18:33:13] matanya: also, that is not a url… http or https? [18:33:23] https [18:33:40] and that's behind the labs web proxy I presume? [18:34:33] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2207848 (10Papaul) I reset the IDRAC and upgrade the firmware from 1.7 to 2.8, but having the same issue can not redirect to com2. All the settings in the BIOS for the... [18:34:35] yes [18:34:56] andrewbogott: https://phabricator.wikimedia.org/T95714 [18:36:11] but i don't know about subdomain, as it is not mentioned there [18:38:36] matanya: https://phabricator.wikimedia.org/T78167#1470499 states it apparently works? [18:39:40] i didn't said it is not possible, only that i don't know :) [18:50:25] matanya: responded to https://phabricator.wikimedia.org/T95714 [19:09:58] thanks andrewbogott [19:11:57] matanya: am I at least stating the problem correctly? [19:13:18] 06Labs, 06Operations, 10ops-codfw: labtestneutron2001.codfw.wmnet does not appear to be reachable - https://phabricator.wikimedia.org/T132302#2207998 (10Krenair) To be clear, this is a system that I noticed had issues when I got labtest-roots so I opened a ticket to ensure it was not forgotten. It's not a pr... [19:48:41] Wow [19:48:49] 'http://toolserver.org/~kolossos/openlayers'.length === 'https://tools.wmflabs.org/wiwosm/osm-on-ol'.length // 42 [20:01:23] where's that from, Krinkle? [20:01:30] also wat [20:01:41] https://krc.wikipedia.org/w/index.php?title=MediaWiki:Osm.js&diff=prev&oldid=102725 [20:02:04] I was just shocked that when I followed the url (and hoping for a redirect) that the replacement was the exact same length [20:02:09] It didn't do anything in the editor [20:02:32] http/https ts/labs differnent path [20:02:42] :D [20:11:06] 10PAWS: PAWS 404 for users with special characters in their names - https://phabricator.wikimedia.org/T120066#2208210 (10yuvipanda) As an update, 4.2 (which has all of these fixed) is in beta now. It'll be released in a week or something and then we'll upgrade to it. \o/ [20:46:38] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:55:34] ^ result of gerrit 503'ing [21:06:15] Krinkle: I'm thinking of hosting a copy of https://npmcdn.com/ in addition to the cdnjs we already have. Thoughts? [21:06:25] Krinkle: do you think it'll be useful? [21:06:32] YuviPanda: In what context? [21:07:04] YuviPanda: we also still need to clean up cdnjs :/ [21:07:18] (non-free packages and external includes) [21:07:21] YuviPanda: For client-side? [21:07:25] Krinkle: yeah [21:08:01] valhallasw`cloud: yeah :| I don't know what we can do though - blacklist some? I guess not all upstreams are as good about license enforcement as debian is. [21:08:07] YuviPanda: I don't think it'd be very useful in that case, no. The require model itself is incompatible with loading files that way so it would only work for packages that intentionally create a second version of their dist folder for cdn use. [21:08:17] Which is rather rare to then publish into the package itself. [21:08:37] valhallasw`cloud: Although, I think long term, the right way to do that enforcmenet is via CSP headers. [21:08:51] In most cases you'll want it to be part of the larger front-end distribution, in which case you'll need to install it locally anyway so that it can be bundled. [21:08:58] E.g. for browerify or webpack [21:09:01] Krinkle: right, ok. makes sense. [21:09:29] For non-npm and non-require stuff (e.g. traditional JS) it'll be as a single file on cdnjs instead. [21:09:32] valhallasw`cloud: I remember bd808 was talking about it (it -> CSP) in some ticket, trying to find it [21:12:22] https://phabricator.wikimedia.org/T130748 [21:21:40] 06Labs, 10Tool-Labs, 10labs-sprint-116, 10labs-sprint-117, and 2 others: Allow direct ssh access to tools - https://phabricator.wikimedia.org/T113979#2208454 (10valhallasw) Time to write down some more thoughts. current situation ------------------ In the current situation, access is controlled through `... [21:26:38] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0] [21:29:54] REMINDER: about next week's data center migration. There will be NO EDITING for about 30 minutes on both Tuesday, 19 April at 14:00 UTC and Thursday, 21 April at 14:00 UTC. This will affect ALL of the wikis. You will be able to read the wikis and to log in, but you will not be able to edit. Read https://meta.wikimedia.org/wiki/Tech/Server_switch_2016 for more information. Please share this news. [21:29:54] If you find unexpected problems (e.g., can't read for more than a few minutes), then please report those in #wikimedia-tech, which will be monitored by members of the Operations team. #wikimedia-operations will be moderated until the switch is finished and everything is stable again. [22:12:17] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:14:20] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 06Operations, and 2 others: Clean up labs graphite datapoints - https://phabricator.wikimedia.org/T111540#2208576 (10Krenair) [22:18:47] 06Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure: deployment-sentry2 /var is full - https://phabricator.wikimedia.org/T132730#2208581 (10Krenair) [22:47:37] PROBLEM - Puppet run on tools-bastion-02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:22:37] RECOVERY - Puppet run on tools-bastion-02 is OK: OK: Less than 1.00% above the threshold [0.0]