[00:30:56] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, 10Operations: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#3317065 (10chasemp) This task was to make a plan for user mgmt access to bare metal as a service @dzahn to help clarify, which we have... [01:39:39] 10Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, 10Operations: How to handle mgmt lan for labs bare metal? - https://phabricator.wikimedia.org/T116607#3317114 (10Dzahn) Got it, thank you both. Yep! [02:02:03] 10Labs, 10Tool-Labs, 10Tools-Kubernetes: k8s webservice restart failure with `ValueError: get() more than one object; use filter` - https://phabricator.wikimedia.org/T156626#3317136 (10APerson) @zhuyifei1999 (sorry for the delayed response!) ``` NAME READY STATUS RESTARTS AGE in... [02:20:32] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than re-using Tools - https://phabricator.wikimedia.org/T167086#3317175 (10yuvipanda) [03:35:04] 10Tool-Labs-tools-Zppixbot, 10QuarryBot-enwiki, 10User-Zppix: Zppixbot & Quarrybot-enwiki: "Add Code of Conduct.md" - https://phabricator.wikimedia.org/T167087#3317221 (10Zppix) [05:42:53] 10Tool-Labs-tools-Xtools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3317258 (10kaldari) [05:42:58] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Rewrite Edit Counter with new backend architecture - https://phabricator.wikimedia.org/T160481#3317256 (10kaldari) 05Open>03Resolved Code is merged and seems to work (although extremely slowly). [06:12:08] Join [06:18:39] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3317272 (10yuvipanda) [06:46:01] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1428 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:59:10] PROBLEM - Puppet errors on tools-exec-1422 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:21:01] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1428 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:10] RECOVERY - Puppet errors on tools-exec-1422 is OK: OK: Less than 1.00% above the threshold [0.0] [08:33:43] 10Labs, 10Labs-Infrastructure, 10Tool-Labs, 10User-fgiunchedi: Rollout prometheus-node-exporter 0.14 in labs - https://phabricator.wikimedia.org/T166561#3317409 (10fgiunchedi) [09:59:44] (03PS1) 10Tobias Gritschacher: Report contributions to AdvancedSearch to wikimedia-de-tech channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357357 (https://phabricator.wikimedia.org/T167094) [10:06:24] ¬‿¬ [10:18:12] 10Tool-Labs-tools-Attribution-Generator, 10TCB-Team, 10User-Tobi_WMDE_SW: When generating attribution of image under a ported version of the licence, the unported license is shown in the attribution - https://phabricator.wikimedia.org/T136305#2330071 (10Tobi_WMDE_SW) 05Open>03Resolved [10:18:19] 10Tool-Labs-tools-Attribution-Generator, 10TCB-Team, 10User-Tobi_WMDE_SW: Commons shortlinks not supported - https://phabricator.wikimedia.org/T157434#3005045 (10Tobi_WMDE_SW) 05Open>03Resolved [10:18:27] 10Tool-Labs-tools-Attribution-Generator, 10TCB-Team, 10User-Tobi_WMDE_SW: Add additional Text to the PD-Images hint - https://phabricator.wikimedia.org/T165332#3263311 (10Tobi_WMDE_SW) 05Open>03Resolved [10:34:06] 10Labs, 10wikitech.wikimedia.org, 10User-bd808: Can't login to wikitech.wikimedia.org - https://phabricator.wikimedia.org/T167055#3317887 (10Jhernandez) @bd808 Thanks for the instructions. I've left `/home/jhernandez/2fa-reset.txt` in `reading-web-staging-3.reading-web-staging.eqiad.wmflabs` as instructed. [11:09:13] 10Labs, 10Wikidata, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3318062 (10Ladsgroup) [11:44:40] 10Labs: high commonswiki_p replag on labs - https://phabricator.wikimedia.org/T167120#3318211 (10Steinsplitter) [11:45:26] 10Labs, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3318228 (10Ladsgroup) a:03Ladsgroup [12:01:56] 10Labs, 10Commons: high commonswiki_p replag on labs - https://phabricator.wikimedia.org/T167120#3318295 (10Steinsplitter) [12:24:22] 10Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown: Rebuild old timestamp format tables - https://phabricator.wikimedia.org/T151607#3318347 (10Marostegui) 05stalled>03Resolved Closing this as we are not moving tablespaces anymore. We would need to make sure that this isn't an issue wh... [12:24:26] 10Labs, 10Labs-Infrastructure, 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review: Provision db1095 with at least 1 shard, sanitize and test slave-side triggers - https://phabricator.wikimedia.org/T150802#3318349 (10Marostegui) [14:19:22] #wikimedia-labs moved here huh? Was there a discussion about this that I missed? [14:27:41] hello everybody, I have a question: the infrastracture of Wikimedia Labs is hosted in the same datacenters of the servers hosting Wikipedia or is it separate? I recall that some time ago what you saw on the Labs wiki was basically an interface to Amazon AWS [14:27:46] is this still the case? [14:28:33] ShoeMaker: yes. [14:28:36] Labs was never AWS, we've always maintained our own cloud [14:28:40] CristianCantoro: ^ [14:28:53] CristianCantoro: links please. We don't use 3rd party services to host afaik. [14:29:18] The Labs infrastructure is in eqiad which is our primary datacenter for production Wikimedia as well. there's some network isolation between the two clusters though. [14:30:53] ShoeMaker: I encourage you to subscribe to labs-l and labs-announce for things like this, although they're probably about to get renamed as well :) [14:30:57] Do you need subscribe links? [14:31:24] 10Labs-project-other: telnet gateway broken (unable to fetch articles) - https://phabricator.wikimedia.org/T167064#3318833 (10cscott) Thank you for that illuminating music recommendation. [14:32:32] andre__: this was just something I was recalling. Some time ago when I was setting up a service somebody linked me a list of the available sizes for the VMs, it looked like AWS, but maybe it was just a similar interface [14:43:22] CristianCantoro: horizon.wikimedia.org has such an interface I believe [14:43:39] you need an ldap account CristianCantoro see https://wikitech.wikimedia.org/wiki/Help:Getting_Started [14:44:40] chasemp: thanks, I already have an account. My problem is that I use Labs so sporadically that each time I use it the interface is completely changed (for the better, of course) :-) [14:45:38] CristianCantoro: sure understandable, horizon is the primary interface to the VPS/AWS like functionality [14:45:47] if you are interested in toollabs/toolforge check out https://toolsadmin.wikimedia.org/ [15:13:30] 10Labs, 10Wikidata, 10Patch-For-Review, 10User-Ladsgroup, 10Wikidata-Sprint: Open view for term_full_entity_id in wb_terms table in labs - https://phabricator.wikimedia.org/T167114#3319056 (10daniel) I'm dubious about exposing this table to labs tools before it is fully populated. This at least needs a w... [15:22:53] question: if I wish to delete a project from WMFlabs where I can find instruction on how to do so? [15:23:30] CristianCantoro: a full project can only be deleted by admins so making a task is the best thing [15:24:21] chasemp: a pointer for that? [15:24:40] chasemp: question, what are valid reasons to request labs projects? [15:25:04] Zppix: typically some focused effort that can't be done in Tools [15:25:17] CristianCantoro: https://phabricator.wikimedia.org/maniphest/task/edit/form/1/ [15:25:19] chasemp: does testing wmf projects count? [15:25:31] Zppix: that's pretty vague so I'm not sure [15:26:09] chasemp: for example say i wanted to test my own instance of wikilabels [15:26:42] Zppix: probably? but there could also be someone already doing that and we would recommened joining their effort etc [15:26:55] it's about practicality and resource tracking more than anything ideological [15:27:06] Alrighty thanks just curious [15:28:08] np [15:35:09] 10Labs, 10Wikidata: Delete wikidata-ldf project on wmflabs - https://phabricator.wikimedia.org/T167154#3319139 (10CristianCantoro) [15:35:27] chasemp: I should have done it :) [15:36:27] CristianCantoro: you did now its a waiting game [15:36:47] I am not in a rush :-) [15:37:22] Have any other things you need help with CristianCantoro ? [15:37:51] No, thanks a lot for the help Zppix, chasemp [15:38:08] valhallasw`cloud: was my config change to wikibugs deployed or will that happen at a later time (no rush just curious) [15:38:19] CristianCantoro: anytime thats why we are here [15:41:35] 10Labs: Request creation of openipmap labs project - https://phabricator.wikimedia.org/T166671#3303834 (10bd808) "efficient IP range indexing" is a bit confusing to me. An IP address is just a number and range queries are something like `WHERE ip >= X and ip <= Y`. That can certainly be done in MySQL/MariaDB. [15:43:21] 10Labs, 10Labs-Infrastructure: When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3319183 (10Andrew) a:03Andrew [15:56:59] (03Draft2) 10Zppix: Add Code of Conduct.md per policy [labs/tools/quarrybot-enwiki] - 10https://gerrit.wikimedia.org/r/357407 [15:57:14] (03CR) 10Paladox: [V: 032 C: 032] Add Code of Conduct.md per policy [labs/tools/quarrybot-enwiki] - 10https://gerrit.wikimedia.org/r/357407 (owner: 10Zppix) [15:57:24] Wait [15:57:31] woops [15:57:35] Sorry zppix [15:58:00] Its ok [15:58:08] I just have to add bugs to the commit [15:58:36] ok [16:01:31] replag on commons_p :'-( [16:02:14] Its the dbhosts [16:14:16] Zppix: dbhost? [16:14:46] Steinsplitter: store i meant [16:58:28] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3319424 (10RobH) [17:03:13] 10Labs, 10cloud-services-team (Kanban): Rename labs-admin mailing list to cloud-admin - https://phabricator.wikimedia.org/T167155#3319451 (10bd808) [17:04:03] (03PS1) 10BryanDavis: Add cloud-services-team to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357419 [17:06:20] 10Labs, 10wikitech.wikimedia.org, 10User-bd808: Can't login to wikitech.wikimedia.org - https://phabricator.wikimedia.org/T167055#3319458 (10bd808) 05Open>03Resolved `{{done}}` [17:07:34] (03Draft2) 10Zppix: Add Code of Conduct.md per policy [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/357420 (https://phabricator.wikimedia.org/T167087) [17:08:58] (03CR) 10Zppix: [C: 032] Add Code of Conduct.md per policy [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/357420 (https://phabricator.wikimedia.org/T167087) (owner: 10Zppix) [17:09:05] (03Merged) 10jenkins-bot: Add Code of Conduct.md per policy [labs/tools/ZppixBot] - 10https://gerrit.wikimedia.org/r/357420 (https://phabricator.wikimedia.org/T167087) (owner: 10Zppix) [17:12:50] 10Labs, 10wikitech.wikimedia.org, 10User-bd808: Can't login to wikitech.wikimedia.org - https://phabricator.wikimedia.org/T167055#3319483 (10Jhernandez) Thanks @bd808. Already re-enabled and properly saved scratch codes this time. [17:16:44] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3319490 (10RobH) [17:16:49] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2001 - https://phabricator.wikimedia.org/T167160#3319508 (10RobH) [17:18:54] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3319531 (10yuvipanda) I would also like to use T167084 for this cluster, so knowing if that'll work with Neutron will be useful. [17:26:06] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3319558 (10yuvipanda) Note that as is with non-tools labs projects, I'll be responsible for this k8s cluster - not the labs admin te... [17:30:03] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3317175 (10bd808) The concept of keeping PAWS in sync with upstream practices is sound. I want to add support of PAWS to the charter... [17:32:37] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3317175 (10Andrew) I don't have a strong opinion about this. Running on tools = eating our own dogfood, which is often useful... bu... [17:58:16] PROBLEM - Free space - all mounts on tools-webgrid-lighttpd-1426 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1426.diskspace._tmp.byte_percentfree (<100.00%) [18:14:35] 10Labs, 10DBA, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3319781 (10jcrespo) I have reloaded dbproxy1011 configuration, heads up in case the wrong db is being pointed at (downtime expired). [18:17:18] 10Labs, 10DBA, 10Patch-For-Review: Add and sanitize s2, s4, s5, s6 and s7 to sanitarium2 and new labsdb hosts - https://phabricator.wikimedia.org/T153743#3319866 (10jcrespo) Maybe it didn't expire but got lost, but same thing applies. [18:27:07] 10Tool-Labs-tools-Zppixbot, 10QuarryBot-enwiki, 10User-Zppix: Zppixbot & Quarrybot-enwiki: "Add Code of Conduct.md" - https://phabricator.wikimedia.org/T167087#3319975 (10Zppix) 05Open>03Resolved [18:50:19] 10Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3320121 (10demon) >>! In T165624#3274279, @bd808 wrote: > User `MacFan4000` already exists in LDAP. The account was created 2016-08-25T22:53:04Z. Both accounts are r... [19:00:23] (03CR) 10Legoktm: [C: 032] Add cloud-services-team to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357419 (owner: 10BryanDavis) [19:00:58] (03CR) 10Legoktm: [C: 032] Remove obsolete #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357222 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [19:02:19] (03Merged) 10jenkins-bot: Remove obsolete #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357222 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [19:02:26] (03CR) 10jenkins-bot: Remove obsolete #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357222 (https://phabricator.wikimedia.org/T166420) (owner: 10BryanDavis) [19:02:36] (03Merged) 10jenkins-bot: Add cloud-services-team to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357419 (owner: 10BryanDavis) [19:02:45] (03CR) 10jenkins-bot: Add cloud-services-team to #wikimedia-cloud [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357419 (owner: 10BryanDavis) [19:13:54] (03CR) 10Merlijn van Deen: [C: 032] Report contributions to AdvancedSearch to wikimedia-de-tech channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357357 (https://phabricator.wikimedia.org/T167094) (owner: 10Tobias Gritschacher) [19:13:59] (03Merged) 10jenkins-bot: Report contributions to AdvancedSearch to wikimedia-de-tech channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357357 (https://phabricator.wikimedia.org/T167094) (owner: 10Tobias Gritschacher) [19:14:14] (03CR) 10jenkins-bot: Report contributions to AdvancedSearch to wikimedia-de-tech channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/357357 (https://phabricator.wikimedia.org/T167094) (owner: 10Tobias Gritschacher) [20:04:21] Hey folks, I want to edit my hiera for ORES so that it adds a line to /etc/nagios/nrpe.cfg [20:04:42] But I'm not quite sure how that works. I want it to be in hiera because it should only apply to our instances of ORES in labs -- not prod. [20:06:06] halfak: depends on what you want to change [20:06:16] halfak: I'm not sure how straight forward that is, I know there are hiera magics that change values there but entire entries may be difficult potentially? [20:06:17] I don't think /etc/nagios/nrpe.cfg is puppetized? /etc/nagios/nrpe_local.cfg does seem to be [20:06:26] ah [20:06:28] oh yeah. Probably that [20:06:31] paladox, ^ [20:06:34] sound right? [20:06:46] but class nrpe($allowed_hosts='127.0.0.1') { means $allowed_hosts is really the only configurable parameter [20:06:56] but my repo is old, so let me do a clean checkout [20:07:11] I want to add "command[check_disk]=/usr/lib/nagios/plugins/check_disk -w 5% -c 2% -l -e --exclude-type=tracefs" [20:07:22] ah [20:07:55] halfak: look at role/common/labs/nfs/misc.yaml [20:07:59] or so in the puppet repo [20:08:14] in theory I think if you define [20:08:15] nfs O.O [20:08:15] profile::base::check_disk_options: -w 10% -c 5% -l -e -A -i /run/lock/storage-replicate-.*/snapshot -i /exp/.* [20:08:32] it's just an example of overriding check_disk [20:08:37] https://github.com/wikimedia/puppet/blob/production/modules/role/manifests/labs/nfs/misc.pp [20:08:41] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3320503 (10Papaul) [20:08:47] err no, wait [20:08:59] the repo is too complicated >_> [20:09:01] that's the manifest but not the yaml :) [20:09:02] heh [20:09:39] modules/profile/manifests/base.pp: $check_disk_options = hiera('profile::base::check_disk_options', [20:09:42] oh, the hiera yaml. Of course. [20:10:33] So I wouldn't be setting it here? https://wikitech.wikimedia.org/wiki/Hiera:Ores [20:10:40] https://github.com/wikimedia/puppet/blob/production/hieradata/role/common/labs/nfs/misc.yaml [20:10:44] halfak: you can set it there as well [20:11:05] you should be able to override modules/profile/manifests/base.pp value $check_disk_options = hiera('profile::base::check_disk_options' [20:11:08] on that page halfak [20:11:12] if that's all you need to do [20:11:13] there's a hierarchy in config (hence the name hiera), where first the config from hieradata is loaded, and then via some steps you get to wikitech [20:11:14] so instead of [20:11:24] -w 6% -c 3% -l -e -A -i "/srv/sd[a-b][1-3]" --exclude-type=tracefs [20:11:27] you could set it to [20:11:34] -w 5% -c 2% -l -e --exclude-type=tracefs [20:11:44] profile::base::check_disk_options: '-w 5% -c 2% -l -e --exclude-type=tracefs' [20:11:49] that...should work [20:12:00] * halfak tries that [20:12:34] valhallasw`cloud: hiera...hierarchy [20:12:40] I honestly did not put that together [20:12:41] https://wikitech.wikimedia.org/w/index.php?title=Hiera:Ores&diff=1761366&oldid=1761351 [20:13:16] halfak: idk if there is a difference between single and double quotes there [20:13:22] just a thought in case it doesn't work [20:13:53] Is this page read as yaml? [20:13:53] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2001 - https://phabricator.wikimedia.org/T167160#3320517 (10Papaul) @Robh @chasemp we have already a node with the name labtestneutron2001 in row B rack B8 can we make this labtestneutron2002? [20:14:11] yuvi wrote a thing that basically reads the wiki page and converts it yes [20:14:14] iirc [20:14:40] that panda feller [20:14:44] yup [20:16:05] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2001 - https://phabricator.wikimedia.org/T167160#3320521 (10chasemp) @papaul yes, thank you [20:16:09] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2001 - https://phabricator.wikimedia.org/T167160#3320522 (10RobH) @Papaul: good catch! Yes, lets just call this new host labtestneutron2002. [20:16:12] halfak: any luck? [20:16:17] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3320523 (10RobH) [20:16:57] chasemp, testing it out now. [20:17:04] So many strings pulling on me [20:17:27] shake'em off halfak, shake shake shake'em off [20:17:47] lol [20:18:11] chasemp, looks like it worked great [20:18:17] halfak: ok cool [20:18:37] paladox, we have 'command[check_disk_space]=/usr/lib/nagios/plugins/check_disk -w 5% -c 2% -l -e --exclude-type=tracefs' -- that's not what you asked for, but will it work? [20:18:37] so this worked becuase it was already param'd via hiera so we hooked into that etc [20:18:59] I need a map every time I start browsing around the puppet repo [20:19:51] halfak: this cloud services not the maps team :P [20:20:32] lol [20:20:38] We don't have a maps team anymore [20:20:50] :( [20:21:12] rollercoaster of emotions guys [20:22:07] hi, I'm on tools-bastion-02 and anything involving reading/writing from the filesystem is going super slow, e.g. `ls` or tabbing out when doing `cd`, saving a file in vim, etc. Is it just me? [20:22:20] I can check [20:22:40] thanks [20:23:08] i would check but everytime i go to tools im always on -03 [20:23:16] hm, ls on / takes 3s for example [20:23:23] tools-dev.wmflabs.org is -02 [20:23:42] really? i always use that and it says -03 [20:23:44] musikanimal: now it's faster. for you too? [20:23:53] Zppix: you are using tools-login I guess [20:23:55] no, but it goes in and out [20:24:08] nope i used login once (when i first got access to tools) [20:24:14] right now it is still really slow for me [20:24:18] musikanimal: it's not just you, and someone is doing some major scp there [20:24:20] hm, cd .. takes more than 20 seks [20:24:46] halfak hi, sorry for late reply [20:24:51] yes that will work [20:24:51] up for 76 days, ppl have left behind some random things I imagine [20:24:54] but need to also do a service restart [20:24:55] Oh good :) [20:25:01] sudo service nagios-nrpe-server restart [20:25:27] paladox, {{done}} [20:26:13] halfak which host did you do it on? [20:26:16] btw, is there a way to see which shell name belongs to which LDAP user? [20:26:31] paladox, sorry. ores-web-05.eqiad.wmflabs [20:26:37] Sagan: Look in ldap :) [20:26:44] thanks [20:26:58] RainbowSprinkles: never did before, how can I do that? :D [20:27:02] chasemp: i can look at the bastion if you are not already killing the scp [20:27:13] I'm looking, I'm going to to renice them [20:27:17] Sagan: `ldapsearch` :) [20:27:18] if that doesn't work I'll reboot [20:27:20] okay [20:27:47] * Sagan switches to another server before doing that [20:28:45] musikanimal: any better? [20:28:57] RainbowSprinkles: works not for me: It promps for a password, and then it says: additional info: SASL(-13): user not found: no secret in database [20:29:09] no :( [20:29:16] Sagan: `ldapsearch -xLLL -P 3 -E pr=40000/noprompt -b"dc=wikimedia,dc=org" uid=bd808` [20:29:54] bd808: ah, thx. I'm using LDAP at my company as well, but we all have a GUI [20:29:56] That ^ [20:29:59] thank you two :) [20:30:13] ldaCan swap the uid=foo for cn=Bar to search by their wiki name [20:30:36] s/ldaCan/Can/ [20:30:41] !log rebooting tools-bastion-02 as unresponsive (up 76 days and lots of seemingly left behind things running) [20:30:43] chasemp: Unknown project "rebooting" [20:30:45] !log tools rebooting tools-bastion-02 as unresponsive (up 76 days and lots of seemingly left behind things running) [20:30:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:30:53] lol [20:31:20] oh wow, a new wikitech homepage? :) [20:31:43] musikanimal: try again in a few minutes if you would [20:31:58] will do! I got the message about it rebooting [20:31:58] it was insane whackamole trying to clean up all the random running things there so I rebooted [20:34:33] chasemp: working great now! thank you :) [20:37:33] 10Labs, 10wikitech.wikimedia.org, 10MW-1.30-release-notes (WMF-deploy-2017-06-06_(1.30.0-wmf.4)), 10Patch-For-Review: Disable/redirect instance instance management on wikitech - https://phabricator.wikimedia.org/T164875#3320696 (10Jdforrester-WMF) Mass-moving all items tagged for MediaWiki 1.30.0-wmf.3, as... [20:37:50] 10Labs, 10MediaWiki-extensions-OpenStackManager, 10wikitech.wikimedia.org, 10MW-1.30-release-notes (WMF-deploy-2017-06-06_(1.30.0-wmf.4)), 10Patch-For-Review: Remove OpenStackManager from Wikitech - https://phabricator.wikimedia.org/T161553#3320705 (10Jdforrester-WMF) Mass-moving all items tagged for Med... [20:44:40] 10Labs: disable service groups for non-tools projects - https://phabricator.wikimedia.org/T167204#3320803 (10Aklapper) #Labs, I assume? [20:52:35] 10Labs, 10cloud-services-team (Kanban): disable service groups for non-tools projects - https://phabricator.wikimedia.org/T167204#3320882 (10bd808) [20:54:00] 10Labs, 10Operations, 10cloud-services-team (Kanban): tools-k8s-master-01 has two floating IPs - https://phabricator.wikimedia.org/T164123#3320890 (10bd808) [20:54:03] 10Labs, 10Patch-For-Review, 10cloud-services-team (Kanban): Issues in enabling NFS for new projects (was 'adding project to nfs-mounts.yaml does not create directories') - https://phabricator.wikimedia.org/T158883#3320891 (10bd808) [20:54:07] 10Labs, 10Operations, 10cloud-services-team (Kanban): Initial OpenStack Neutron PoC deployment in Labtest - https://phabricator.wikimedia.org/T153099#3320892 (10bd808) [20:56:01] 10Labs, 10Operations, 10cloud-services-team (Kanban): Investigate alternative RAID strategies for labstore1001/2 - https://phabricator.wikimedia.org/T162090#3320899 (10bd808) [20:56:05] 10Labs, 10Operations, 10cloud-services-team (Kanban): Undo special tools-home and tools-project share definitions for NFS - https://phabricator.wikimedia.org/T161834#3320900 (10bd808) [20:56:08] 10Labs, 10Tool-Labs, 10cloud-services-team (Kanban): Setup monitoring for cdnjs git pull - https://phabricator.wikimedia.org/T144215#3320903 (10bd808) [20:56:12] 10Labs, 10Operations, 10cloud-services-team (Kanban): labstore systemd state Icinga alarms - https://phabricator.wikimedia.org/T151322#3320902 (10bd808) [20:56:16] 10Labs, 10Labs-Infrastructure, 10Tool-Labs, 10Wikimedia-Incident, 10cloud-services-team (Kanban): Write a simple script that handles failovering proxies - https://phabricator.wikimedia.org/T143639#3320904 (10bd808) [20:57:21] 10Labs, 10Labs-Infrastructure, 10cloud-services-team (Kanban): monitor some things on all Cloud instances (discussion) - https://phabricator.wikimedia.org/T166845#3320911 (10bd808) [20:57:24] 10Labs, 10cloud-services-team (Kanban): nova-fullstack is losing instances on creation - https://phabricator.wikimedia.org/T165555#3320912 (10bd808) [20:57:27] 10Labs, 10cloud-services-team (Kanban): Add new users to 'bastion' project via a Keystone hook - https://phabricator.wikimedia.org/T165337#3320913 (10bd808) [20:57:30] 10Labs, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Set up external DNS record for wikitech-static - https://phabricator.wikimedia.org/T164290#3320914 (10bd808) [20:57:33] 10Labs, 10Labs-Infrastructure, 10Patch-For-Review, 10cloud-services-team (Kanban): Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3320915 (10bd808) [20:57:36] 10Labs, 10Labs-Infrastructure, 10cloud-services-team (Kanban): When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3320916 (10bd808) [20:59:38] 10Labs, 10Cassandra, 10Services (blocked), 10User-bd808, 10cloud-services-team (Kanban): Request increased quota for services-testbed labs project - https://phabricator.wikimedia.org/T163375#3320923 (10bd808) [20:59:42] 10Labs, 10Tool-Labs, 10Patch-For-Review, 10User-bd808, 10cloud-services-team (Kanban): Update maintain-kubeusers to allow tool's to write to $HOME/.kube - https://phabricator.wikimedia.org/T165875#3320922 (10bd808) [21:13:59] 10Tool-Labs-tools-Xtools, 10Community-Tech: Rewrite Edit Counter with new backend architecture - https://phabricator.wikimedia.org/T160481#3321066 (10DannyH) [21:16:41] 10Tool-Labs-tools-Xtools, 10Community-Tech: Unknown project should not cause an Internal Server Error - https://phabricator.wikimedia.org/T166332#3321082 (10DannyH) [21:17:37] 10Tool-Labs-tools-Xtools, 10Community-Tech: Epic: Rewriting XTools - https://phabricator.wikimedia.org/T153112#3321105 (10Matthewrbowker) [21:17:42] 10Tool-Labs-tools-Xtools, 10Community-Tech: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3321093 (10Matthewrbowker) [21:20:04] 10Tool-Labs-tools-Xtools, 10Community-Tech: If revisions are revdel'd, articleinfo compares the surrounding edits as if it were one edit - https://phabricator.wikimedia.org/T148857#3321124 (10DannyH) [21:22:49] 10Tool-Labs-tools-Xtools, 10Community-Tech, 10I18n: XTools: Cleanup i18n messages - https://phabricator.wikimedia.org/T163468#3321132 (10DannyH) [21:52:27] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3321267 (10MusikAnimal) [22:03:01] 10Labs, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Set up external DNS record for wikitech-static - https://phabricator.wikimedia.org/T164290#3321352 (10RobH) >>! In T164290#3312661, @Andrew wrote: > @robh, is there any reason we can't just register wikitech-static.wikimedia.org with markmoni... [22:11:50] 10Labs, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Set up external DNS record for wikitech-static - https://phabricator.wikimedia.org/T164290#3228802 (10Dzahn) Technically.. we could use wikitech-static.wikiartpedia.biz or wikitech-static.webhostingwikipedia.com or wikitech-static.softwarew... [22:12:49] don't you love those URLs?^ :) [22:15:10] 10Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3321441 (10bd808) User `MacFan4000` has no contributions on wikitech, so we could just delete the local MediaWiki account and its associated LDAP record and then tre... [22:15:11] My toollabs tool, https://tools.wmflabs.org/excel2wiki/, is saying there's no webservice. However, I have started the webservice and there are no entries in the error log. [22:16:16] ahecht: your service.log shows that the service isn't starting. [22:17:13] ahecht: hmmm... something bad is happening. `qstat` shows a big pile of erroring processes [22:17:15] When i do "webservice start" it says "Your job is already running". [22:17:59] can I poke at it for a bit and see if I can get it to work? [22:18:04] Sure [22:18:22] it looks like you are just trying to run a normal lighttpd + php webservice. is that correct? [22:19:21] Yes [22:20:46] the kubernetes backend immediately went into the "CrashLoopBackOff" too. Something is funny here [22:21:07] ownership of the error.log file maybe? [22:21:38] 10Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3321463 (10demon) >>! In T165624#3321441, @bd808 wrote: > User `MacFan4000` has no contributions on wikitech, so we could just delete the local MediaWiki account and... [22:23:10] !log tools.excel2wiki Fixed webservice startup crash by removing error.log file with bad ownership (-rw-r--r-- ahecht:tools.excel2wiki) [22:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.excel2wiki/SAL [22:23:16] ahecht: ^ fixed [22:23:40] Thanks. Looks like that was it. [22:23:59] not very nice that that caused things to crash with no logging [22:31:13] 10Labs, 10Operations, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Set up external DNS record for wikitech-static - https://phabricator.wikimedia.org/T164290#3321486 (10RobH) [23:14:12] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3321646 (10kaldari) [23:21:24] 10Tool-Labs-tools-Xtools: Ensure xTools Rebirth is fully responsive - https://phabricator.wikimedia.org/T165706#3321668 (10kaldari) [23:22:14] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Investigation: Add OAUTH for Xtools - https://phabricator.wikimedia.org/T165401#3321669 (10DannyH) [23:22:26] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Verify all routes work between new and old xtools - https://phabricator.wikimedia.org/T165612#3321670 (10DannyH) [23:22:31] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Convert xtools intuition to its own repository - https://phabricator.wikimedia.org/T165708#3321671 (10DannyH) [23:38:47] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Investigation: Add OAUTH for Xtools - https://phabricator.wikimedia.org/T165401#3321713 (10Samwilson) [23:42:22] 10Tool-Labs-tools-Xtools, 10Community-Tech-Sprint: Investigation: Add OAUTH for Xtools - https://phabricator.wikimedia.org/T165401#3321716 (10Samwilson) I've searched the old code for the opt-in message (for the `nograph` message) and found only the above four locations.