[00:35:25] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:15:26] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [02:12:09] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:47:07] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [02:56:03] 10Tool-Labs-tools-Xtools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3377419 (10Samwilson) [03:20:12] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Write unit tests for Xtools - https://phabricator.wikimedia.org/T165400#3377421 (10Samwilson) [03:33:30] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:34:00] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3377423 (10Samwilson) Yes, let's commit to writing tests for each bug that gets raised from now on (as part of {T165400}). [03:58:29] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [05:30:00] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3376801 (10Marostegui) Hi As per: T168782 this looks like a public wiki but with some restrictions, what about its replication to labs hosts? Does it need to happen as a normal publi... [05:46:29] 10Tool-Labs-tools-Xtools, 10Community-Tech: Weird formatting in "Latest global edits" part of XTool edit counter - https://phabricator.wikimedia.org/T168814#3377507 (10kaldari) [06:04:20] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3377524 (10Samwilson) So, edit counter is currently getting a user's 40 most recent edits by first getting a list of edited wikis from CentralAuth (ordered by edit count) and the... [06:34:50] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:37:56] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:38:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:44:23] 10Tool-Labs-tools-Xtools, 10Community-Tech: Weird formatting in "Latest global edits" part of XTool edit counter - https://phabricator.wikimedia.org/T168814#3377580 (10Samwilson) Hm, strange. That class was only used in Edit Counter, and everything looks fine without it, so I've removed it all together. (Hope... [07:09:08] 10Labs, 10Labs-Infrastructure, 10DBA, 10Tracking: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#3377617 (10Marostegui) [07:09:10] 10Labs, 10DBA, 10Patch-For-Review: Fix broken views in labs DB "ERROR 1356 -- references invalid table(s) or column(s)" - https://phabricator.wikimedia.org/T153213#3377614 (10Marostegui) 05Open>03Resolved a:03Marostegui I have cleaned up the views, the only pending thing would be to merge: https://gerr... [07:09:49] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [07:16:44] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/361403 (owner: 10L10n-bot) [07:17:58] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1426 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:02] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [08:47:35] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:13:04] 10cloud-services-team, 10Operations: Reboots of cloud servers - https://phabricator.wikimedia.org/T168445#3377951 (10ArielGlenn) p:05Triage>03Normal [09:15:34] 10Labs, 10Labs-Infrastructure, 10Operations, 10Scoring-platform-team-Backlog: Keep wmflabs scoring boxes up-to-date - https://phabricator.wikimedia.org/T168478#3377954 (10ArielGlenn) p:05Triage>03Normal [09:27:35] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [11:13:43] !log git upgrade gerrit-test from jessie -> stretch following https://linuxconfig.org/how-to-upgrade-debian-8-jessie-to-debian-9-stretch [11:13:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [12:54:52] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378730 (10Urbanecm) Hi @Marostegui, I think the wiki should be replicated as other public wiki. In fact all other fishbowl (publicly readable, editing restricted) wikis are replicated. [12:55:36] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378732 (10Marostegui) >>! In T168788#3378730, @Urbanecm wrote: > Hi @Marostegui, I think the wiki should be replicated as other public wiki. In fact all other fishbowl (publicly reada... [13:02:37] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378752 (10Urbanecm) There are listed at https://github.com/wikimedia/operations-mediawiki-config/blob/master/dblists/fishbowl.dblist. I can do use _p at toollabs for all of th... [13:16:41] andrewbogott: good morning! Openstack failed to delete a nodepool instance over the week-end :\ [13:17:04] andrewbogott: a one time issue though. it is ci-jessie-wikimedia-711220 69de774d-470c-440f-9edd-6f184d51a15e [13:17:07] are you sure it was over the weekend? There were a bunch of those on Thursday and Friday... [13:17:14] 41 hours ago [13:17:32] hm ,ok [13:17:55] it is apparently stuck in delete state [13:17:58] in general the API seems to be struggling with the new nodepool rate. I'm really surprised that 1 second made that much difference. [13:19:21] could nodepool reach a rate limit on the openstack api side ? [13:19:39] or is it overflowing it ? :( [13:21:16] meanwhile the stuck instance is from thursday 22nd created at 19:48 delete asked at 21:03 ( https://horizon.wikimedia.org/project/instances/69de774d-470c-440f-9edd-6f184d51a15e/ ) [13:21:24] so I guess it got stuck while the reboots were being done [13:21:45] thursday was 41 hours ago? [13:24:01] yeah no nhmm [13:24:03] sorry :D [13:24:12] I took the time as reported by nodepool, which is obviously wrong somehow [13:25:34] So if it happened on Thursday then I don't care and you can just delete it :) [13:25:44] or I can [13:27:59] andrewbogott: I cant delete it from the openstack CLI :( [13:28:08] ok, I'll do it [13:39:10] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378877 (10Marostegui) Thanks @Urbanecm. I have checked them and they are in labs indeed - not that I didn't trust you, I just wanted to double check, better be safe than leaking some... [13:43:34] andrewbogott: it is gone. Thank you! [14:02:06] 10Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: Lower rate of Nodepool requests to OpenStack API - https://phabricator.wikimedia.org/T167803#3344784 (10hashar) a:03hashar [14:02:35] andrewbogott: do you have more info about the openstack API struggling with the nodepool rate? [14:03:04] hashar: nothing concrete — I'll let you now if I'm able to find an actual failure case [14:03:05] looks like it kept trying to delete the stuck instance over and over which might be an explanation [14:03:13] okkk [14:03:32] so I am keeping open the task that tracks the rate change [14:04:03] 10Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10Nodepool, and 2 others: Lower rate of Nodepool requests to OpenStack API - https://phabricator.wikimedia.org/T167803#3378947 (10hashar) Keeping it open for monitoring. The OpenStack API might be struggling with the new rate of requests. [14:13:41] 10Labs, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3378981 (10Urbanecm) Thank you for checking! Unfortunately I can't ping you when the tables are in prod as I can't create it. I can only prepare relevant configuration patches (as you... [14:28:52] 10Quarry: Slowdown of Quarry queries processing - https://phabricator.wikimedia.org/T168803#3379071 (10Mess) Have a look also on the Quarry maintanance request that was posted during the 2016 Community Wishlist Survey, where there are old reports similar to this one: https://meta.wikimedia.org/wiki/2016_Communit... [14:37:34] (03Draft1) 10Paladox: Remove Ores-Compute-01 from load check [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361462 [14:37:36] (03PS2) 10Paladox: Remove Ores-Compute-01 from load check [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361462 [14:37:47] (03CR) 10Paladox: [V: 032 C: 032] Remove Ores-Compute-01 from load check [labs/icinga2] - 10https://gerrit.wikimedia.org/r/361462 (owner: 10Paladox) [15:16:35] 10Tool-Labs-tools-Xtools, 10Community-Tech: Weird formatting in "Latest global edits" part of XTool edit counter - https://phabricator.wikimedia.org/T168814#3379234 (10MusikAnimal) That sounds like a legacy class name (even though it may not have been used), so yes let's remove it. It looks like the Edit Count... [15:27:29] PROBLEM - Free space - all mounts on tools-bastion-02 is CRITICAL: CRITICAL: tools.tools-bastion-02.diskspace._public_dumps.byte_percentfree (No valid datapoints found)tools.tools-bastion-02.diskspace.root.byte_percentfree (<10.00%) [15:28:57] that public/dumps alert looks like some NFS glitch. I'm seeing 39% free [15:33:10] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Optimize edit count queries in XTools - https://phabricator.wikimedia.org/T163284#3379306 (10MusikAnimal) >>! In T163284#3377524, @Samwilson wrote: > So, edit counter is currently getting a user's 40 most recent edits by first getting a list of edited wikis fr... [16:34:15] 10Quarry: Slowdown of Quarry queries processing - https://phabricator.wikimedia.org/T168803#3379652 (10Mess) p:05Triage>03High [16:42:36] 10Labs, 10Tool-Labs, 10User-bd808, 10cloud-services-team (Kanban): Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842#3379728 (10bd808) Reindexing has been done for the indexes that were created with ElasticSearch <2.x. Migration helper is showing a few issues, but I think t... [17:01:40] 10Tool-Labs-tools-Pageviews: Vibrating page on large screens - https://phabricator.wikimedia.org/T152658#3379773 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal This was fixed some time ago, I believe [17:02:12] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestmetal2001.codfw.wmnet - https://phabricator.wikimedia.org/T168891#3379777 (10RobH) [17:02:14] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestservices2002.wikimedia.org - https://phabricator.wikimedia.org/T168892#3379794 (10RobH) [17:02:17] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestservices2003.wikimedia.org - https://phabricator.wikimedia.org/T168893#3379811 (10RobH) [17:02:20] 10Labs, 10Labs-Infrastructure, 10Operations, 10procurement: rack/setup/install labtestcontrol2003.wikimedia.org - https://phabricator.wikimedia.org/T168894#3379828 (10RobH) [17:02:57] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#2421318 (10Mess) >>! In T139162#3288840, @Dvorapa wrote: > Is this still an issue? I haven't seen this issue for a while. Yes, it is indeed @Dvorapa. Check out my recent report here: https://phabricator.wikimedia.... [17:03:00] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestmetal2001.codfw.wmnet - https://phabricator.wikimedia.org/T168891#3379859 (10RobH) a:05RobH>03chasemp @Chasemp: Please review my racking and vlan/IP proposal above and confirm or correct. Once that is done, please ass... [17:03:14] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestservices2002.wikimedia.org - https://phabricator.wikimedia.org/T168892#3379861 (10RobH) a:05RobH>03chasemp @Chasemp: Please review my racking and vlan/IP proposal above and confirm or correct. Once that is done, pleas... [17:03:15] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestservices2003.wikimedia.org - https://phabricator.wikimedia.org/T168893#3379863 (10RobH) a:05RobH>03chasemp @Chasemp: Please review my racking and vlan/IP proposal above and confirm or correct. Once that is done, pleas... [17:03:30] 10Labs, 10Labs-Infrastructure, 10Operations, 10procurement: rack/setup/install labtestcontrol2003.wikimedia.org - https://phabricator.wikimedia.org/T168894#3379828 (10RobH) a:05RobH>03chasemp @Chasemp: Please review my racking and vlan/IP proposal above and confirm or correct. Once that is done, plea... [17:04:26] 10Labs, 10Operations, 10hardware-requests: Codfw: (2) hardware access request for labtest [region 2] - https://phabricator.wikimedia.org/T161766#3379872 (10RobH) 05Open>03Resolved These two hosts are on site and being setup via tasks T168893 and T168894. [17:05:43] So I assume chasemp is the ideal for confirming those 4 new hosts IP/vlan requirements [17:05:55] but if someone else in cloud team knows, those 4 hosts are onsite in codfw and awaiting racking. [17:05:56] =] [17:06:22] andrewbogott may know =] [17:09:32] robh: c.hasemp is the best bet. He should be around tomorrow [17:16:18] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3379891 (10Dvorapa) @Mess Recent queries list shows `queued` but the query is completed actually, you can see its real status if you click on some. The wrong status is tracked by T137517. But I think this issue sho... [17:20:00] !log quarry Add milimetric as project admin [17:20:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [17:35:15] 10Tool-Labs-tools-Xtools, 10Community-Tech: Throttle usage of XTools - https://phabricator.wikimedia.org/T168896#3379916 (10MusikAnimal) [17:40:47] 10Tool-Labs-tools-Xtools, 10Community-Tech, 10Epic: Epic: Rewrite Xtools: RfX Analysis - https://phabricator.wikimedia.org/T165709#3379954 (10Matthewrbowker) p:05Triage>03Low [17:43:21] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3379959 (10Matthewrbowker) @kaldari Not just that, but we are also bound under Intuition on TranslateWiki. [17:58:40] robh: I know what existing systems they should resemble but that's probably less helpful than actually knowing where to put them. [17:59:24] so for 3 of the 4 im 99% sure its public vlan, due to existing sytems [17:59:34] for the labsbaretestmetal i assumed the labs row for instances and hosts [17:59:53] since it eventually has to run its own bare metal test in the same vlan as labs instances [18:00:02] but iw asnt sure if it needs labs vlan for instances, or labs vlan for hsots [18:00:39] so i just assumed hte most restrictive row (row b) for having labs hosts vlan [18:01:06] all the codfw rows for labs hosts other than b are reserved, but not deployed in config. [18:01:34] 'labsbaretestmetal' is not a think I have heard of I think [18:01:37] but yeah, the others are labtestservices200[23] and labcontrol2003 which all have existing systems to copy [18:01:56] indeed its labtestmetal2001, first of its kind [18:01:57] * andrewbogott scrolls up [18:02:08] remove bare, i fucked up hostname in irc [18:02:13] labtestmetal2001. [18:02:19] ok, let me look for a minute [18:02:19] i assume its for bare metal testing [18:02:41] but i just told papaul to stack all 4 up in the corner of the cage and not to rack until chase confirmed =] (so if we cannot figure it out today, its ok) [18:03:15] yeah, I think I don't know what labtestmetal is for, will have to wait on chase for that one [18:31:00] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3380097 (10kaldari) Please replace the current logo with xtools3.svg, which fixes the alignment issue. Also, the logo is still way too big. [18:31:12] 10Quarry: Users blocked from account creation on meta can not use Quarry - https://phabricator.wikimedia.org/T157342#3380100 (10Quiddity) 05Open>03Resolved Marking as resolved. Seems to be fixed (per comments above). [18:32:00] hey, that was mean, some bot kicked me. [18:32:15] madhuvishy: how do you log to https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry? [18:32:43] milimetric: The redirect from the old channel does that! [18:32:45] milimetric, from the old channel name, maybe? [18:32:52] ah! [18:33:00] you log here, like !log quarry [18:33:00] I wasn't in the new one [18:33:06] yup :) [18:33:37] !log quarry Restarted celery workers on quarry-runner-01 and quarry-runner-02 (systemctl restart celery-quarry-worker.service) [18:33:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [18:33:44] it also goes here - http://tools.wmflabs.org/sal/quarry [18:34:41] thanks for the access and the help, Madhu, I'll keep a quarter of an eye on quarry from now on [18:35:23] ooh, I like wikimedia-cloud, sorts much closer to analytics :) [18:36:01] I manually re-order my IRC channel list. That's part of what keeps me on hexchat. [18:36:03] milimetric: no problem :) and yay for quarry support [18:41:17] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Restrict access to users' edit stats unless opted-in - https://phabricator.wikimedia.org/T165401#3380115 (10kaldari) Was able to test. In the previous implementation, a user could see their own restricted stats if they logged in, but that doesn't seem to be th... [19:14:27] o/ [19:14:39] I've been trying to extract some big files on tools-dev [19:14:47] Total will be something like 15GB [19:15:02] I'd like to stay off of nfs because it is super slow. [19:15:11] I tried to work in /tmp but it's too small. [19:15:36] maybe I should just deal with nfs and use /sratch? [19:15:39] *scratch [19:19:31] halfak: if you are going to unpack something on NFS its probably best to do that via grid job [19:20:15] the problem that has been hit before of unpacking directly from one of the bastions is that it can use up all of the NFS bandwidth quota on that instance [19:20:57] the same could actually happen on the grid too, so if there is a way to nice or throttle the unpacking that would be appreciated [19:21:00] So no local disk options for a temp write? [19:21:49] no large ones I guess. there is ~16G free on bastion-02 (tools-dev) [19:22:57] I think /tmp is ~16G on most exec nodes [19:23:46] scratch is certainly the big volume for this kind of thing globally [19:41:59] OK understood [19:43:18] halfak: do you really need to extract it? [19:43:57] yeah. [19:44:02] Well... sort of [19:44:06] mysqlimport is dumb [19:44:15] I could make an extraction pipe I suppose. [20:03:13] a question: is it possible to get my default shell changed to zsh across all instances so that everytime i login to any instance i get zsh as my shell? [20:09:21] probably [20:09:26] I think the shell was a ldap attribute [20:10:36] Platonides: certainly i don't have permissions to do that, know anyone who can or probably you? [20:15:00] hmm, chsh certainly doesn't fly [20:15:04] chsh: user 'platonides' does not exist in /etc/passwd [20:18:28] yeah same [20:18:35] thats why i came here [20:21:57] I do have a loginShell: /bin/bash attribute in ldap [20:23:28] I don't see your entry, though [20:25:36] oh result: 4 Size limit exceeded [20:26:07] ok, found [20:26:32] ldapsearch -b dc=wikimedia,dc=org -D cn=proxyagent,ou=profile,dc=wikimedia,dc=org -w **** -h ldap-labs.eqiad.wikimedia.org -p 389 -b "dc=wikimedia,dc=org" -s sub uid=codezee [20:33:21] modifying entry "uid=platonides,ou=people,dc=wikimedia,dc=org" [20:33:29] and it indeed changed the attribute [20:34:15] but it seems to be ignored [20:36:31] anyway, the incantation was: [20:36:34] ldapmodify -D uid=platonides,ou=people,dc=wikimedia,dc=org -W -h ldap-labs.eqiad.wikimedia.org -Z -p 389 [20:36:45] (enter in stdin:) [20:36:46] dn: uid=platonides,ou=people,dc=wikimedia,dc=org [20:36:46] changetype: modify [20:36:46] replace: loginShell [20:36:46] loginShell: /bin/zsh [20:41:51] what's the point of the isNovaAdmin attribute? [21:08:48] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3380480 (10MusikAnimal) @Ricordisamoa Is it possible to get xtools3.svg but with an uppercase T? Thanks so much for your help! We will be sure to give you credit on the About page :) [21:10:47] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3380482 (10Mess) @Dvorapa It's not true: I've clicked on [[ https://quarry.wmflabs.org/query/19775 | this query ]] and [[ https://quarry.wmflabs.org/query/19728 | this other one]] and their status is actually "que... [21:26:02] Platonides: as far as I know, isNovaAdmin is a long-abandoned ldap setting [21:27:17] 10Quarry: Query runs over 5 hours without being killed - https://phabricator.wikimedia.org/T139162#3380530 (10Dvorapa) @Mess OK then. I experience same slow down as you describe. Sometimes even not very complicated queries take too much time. [21:29:55] strangely whenever i do [21:30:10] i would get invalid credentials even though i'm using the shell password [21:32:01] codezee: are you using the password of proxyagent? [21:32:20] change cn=proxyagent,ou=profile,dc=wikimedia,dc=org if you want to authenticate as yourself [21:34:46] Platonides: so cn= ? [21:35:35] uid=codezee,ou=people,dc=wikimedia,dc=org [21:35:50] you are in the people OU :P [21:38:25] Platonides: ah i see :P thanks! [21:38:55] Platonides: just curious what was the proxyagent above, any forwarding agent? [21:39:27] it is an account used to query ldap [21:39:36] I obscured its password here [21:39:50] but actually any tools user can read it [21:41:23] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3265163 (10kaldari) @Samwilson: Would it be possible to post links to the relevant patches in GitHub or is it spread over too many? [21:44:25] Platonides: i think i entered everything correct in stdin but attribute didn't change, any special character to end with for changes to take place? [21:44:37] *in stdin of ldapmodify [21:46:17] hmm no [21:46:22] I simply copied that above [21:46:29] and ended with a ^D (ie. EOF) [21:47:03] * codezee tries again [21:49:40] beware not having trailing spaces [21:49:43] Platonides: did it...and yes its ignored, couldn't achieve my objective but learnt about ldap, thanks! :P [21:49:54] hehe [21:50:09] now you could open a task requesting it to be honored :P [21:51:24] Platonides: any tag suggestions? don't wanna annoy wrong people with unrelated stuff :D [21:51:50] other than labs… :/ [21:53:06] codezee: Tag it as labs and cc me. I'll see if I'm smart enough to fix it. :) [21:53:34] I looked at /etc/ldap.conf, but I don't see where are the attributes mapped [21:53:40] there's another open task about cleaning up some old /usr/local/bin/sillyshell entries [21:53:56] oh... yeah. that's the other bug then [21:54:20] why isn't ldap using ssl, btw? [21:54:31] there is a magic mapping in (something) that ignores the shell set in ldap and just uses /bin/bash [21:55:00] ah, probably to avoid the sillyshell [21:55:08] magic mapping in (something) :P [21:55:19] Platonides: probably just because nobody has gotten around to setting up TLS support for it. I bet there is an open bug about doing that. [21:55:46] T86668 [21:55:47] T86668: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668 [21:56:08] Apparently we do the remapping in nslcd.conf [21:56:15] "We override it to be /bin/bash everywhere in nss_ldap.conf and that means we can't actually set it to anything other than /bin/bash" [21:56:49] /etc/nslcd.conf is root:root [21:57:24] most everything in /etc will be root:root [21:58:00] * /etc/nslcd.conf is root:root with 440 [21:58:53] it has a password in it, but that same password can be found elsewhere I think... [22:00:32] this is the puppet file template that generates it -- https://github.com/wikimedia/puppet/blob/production/modules/ldap/templates/nslcd.conf.erb [22:00:39] the proxyagent one? [22:00:46] yeah [22:01:01] yeah, I found it in two different files in about 5 minutes poking at ldap configs [22:01:36] I'm not sure why we even have the proxyagent account to be honest. I don't think it can do anything that an anon bind can't do [22:02:16] maybe anon bind is disabled? [22:02:18] my shell alias for poking around in LDAP is 'ldapsearch -xLLL -P 3 -E pr=40000/noprompt -b"dc=wikimedia,dc=org"' [22:02:43] what's pr? [22:02:58] it turns on result paging [22:03:00] I never remember about ldapsearch syntax [22:03:07] have to look up samples each time [22:03:23] so you can get more than the first N (2048?) results [22:04:13] I was skeptic that we had more than 2048 users [22:04:18] $ ldapsearch -xLLL -P 3 -E pr=40000/noprompt -b"dc=wikimedia,dc=org" | grep -c ^dn: [22:04:21] 19660 [22:04:24] heh. yeah [22:04:30] lots of LDAP accounts [22:04:52] grep -c "^dn: [^,]*,ou=people," [22:04:52] 12561 [22:04:56] not all of those dns are users [22:06:03] 1750 are tools. accounts [22:06:38] that's still 10k users, which is impressive [22:06:51] ldap '(objectClass=posixaccount)' dn|grep dn:|wc -l == 12561 [22:07:19] that's the same as my grep -c "^dn: [^,]*,ou=people," [22:07:40] but nicer by filtering on ldap [22:08:07] and only fetching the dn attribute [22:09:04] 10Tool-Labs-tools-Xtools, 10Community-Tech: Weird formatting in "Latest global edits" part of XTool edit counter - https://phabricator.wikimedia.org/T168814#3380664 (10kaldari) 05Open>03Resolved a:03kaldari [22:09:51] I have uid 1237, that's really early given that [22:10:07] * bd808 looks at what it takes to finish T86668 [22:10:16] wow, even earlier than robchurch's [22:13:52] codezee: you should be partially glad for the /bin/bash remapping. zsh is actually installed as /usr/bin/zsh [22:16:11] lol [22:19:01] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3380682 (10kaldari) Ah, I see. So currently, the Intuition repo has [[ https://github.com/Krinkle/intuition/tree/master/language/messages/xtools | a directory containing... [22:27:51] bd808: i had verified that exec /bin/zsh works atleast for this instance [22:39:17] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3380736 (10Samwilson) @kaldari: No, I think it's too many and would be interleaved with other stuff. Probably the best thing, to get an idea of where we're at, is to look at... [22:55:05] 10Labs, 10Labs-Infrastructure, 10LDAP-Access-Requests, 10Operations: Make all ldap users have a sane shell (/bin/bash) - https://phabricator.wikimedia.org/T86668#3380786 (10bd808) The `sillyshell` entries have been cleaned up. ``` $ ldapsearch -xLLL -E pr=40000/noprompt -b 'dc=wikimedia,dc=org' '(&(objectC... [23:00:36] !log video starting v2c workers at encoding02 & 03, somehow they didn't start automatically after reboot [23:00:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:16:43] !log video failed to start because systemd-tmpfiles-setup failed. investigating [23:16:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:24:08] !log video Fixing tmpfiles configuration on encoding0{1..3}, and rebooting; expecing to see workers back alive after reboot [23:24:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:29:34] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3380904 (10MusikAnimal) >>! In T165708#3380682, @kaldari wrote: > I suppose someone is then manually copying them over into the xtools repo. This is indeed a sad state o... [23:32:40] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3380913 (10Ricordisamoa) >>! In T167345#3380480, @MusikAnimal wrote: > @Ricordisamoa Is it possible to get xtools3.svg but with an uppercase T? Thanks so much for your help! We will be sure to g... [23:34:25] !log video Doing that again :( [23:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:49:48] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3380933 (10Ricordisamoa) With Overpass font: {F8533593} [23:50:46] !log video Doing that again, should be finally working [23:50:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:53:42] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Update all of the Xtools with new backend - https://phabricator.wikimedia.org/T165399#3380955 (10kaldari) Looks sane. The directory structure is a bit complicated/confusing, especially the fact that we have a directory called Xtools in the src directory. Shoul... [23:54:41] !log video depool encoding03 [23:54:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL [23:56:38] !log video oops, depooled encoding02 instead [23:56:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Video/SAL