[01:13:19] !log wikilabels staging wikilabels-wmflabs-deploy:3b857d0 [01:13:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [01:51:38] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3343358 (10Papaul) [01:51:43] 10Labs, 10Labs-Infrastructure, 10Operations, 10netops, 10ops-codfw: codfw: labtestpuppetmaster2001 switch port configuration - https://phabricator.wikimedia.org/T167321#3343355 (10Papaul) 05Resolved>03Open [02:24:39] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:59:40] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [04:35:21] (03PS4) 10BryanDavis: Add toolinfo.json style data [labs/striker] - 10https://gerrit.wikimedia.org/r/353909 (https://phabricator.wikimedia.org/T149458) [04:35:24] (03PS1) 10BryanDavis: Add support for "tagging" toolinfo records [labs/striker] - 10https://gerrit.wikimedia.org/r/358505 (https://phabricator.wikimedia.org/T149458) [04:35:27] (03PS1) 10BryanDavis: Expose all toolinfo data for indexing [labs/striker] - 10https://gerrit.wikimedia.org/r/358506 (https://phabricator.wikimedia.org/T149458) [06:33:41] PROBLEM - Puppet errors on tools-exec-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:38:06] 10Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Create a new instance flavor for deployment-prep - https://phabricator.wikimedia.org/T167723#3342471 (10Paladox) What about using the xtra large image to future proof things? [06:39:58] 10Labs, 10DBA: Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3341927 (10Marostegui) If this is going to be a public Wiki, there is nothing that the DBAs have to do. The only pending thing would be the table views created by #cloud-services-team once the tables are in p... [06:40:14] 10Labs, 10DBA: Prepare and check storage layer for atjwiki - https://phabricator.wikimedia.org/T167715#3341927 (10Marostegui) p:05Triage>03Normal [07:13:27] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3343597 (10Addshore) @GoranSMilovanovic have the #analytics team been poked regarding this issue? as they already have some sort of shiny / rstudio setup? [07:13:39] RECOVERY - Puppet errors on tools-exec-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [07:49:58] 10Labs, 10Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3343635 (10hashar) [07:50:02] 10Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban): Create a new instance flavor for deployment-prep - https://phabricator.wikimedia.org/T167723#3343631 (10hashar) 05Open>03Resolved a:03hashar The new flavor I am requesting is the same as `m1.large` ! [09:30:57] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [10:05:58] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [11:04:10] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3344278 (10GoranSMilovanovic) @Addshore Not yet; they are my last resort. I have tagged this ticket by Analytics too, but they've removed the tag, probably thinking that Labs will... [12:22:19] 10Labs, 10DBA: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#3344427 (10Marostegui) If this is a public wiki (T155038#2934291), the DBAs do not have to do anything. #cloud-services-team will need to create the views once the tables are in place [12:22:27] 10Labs, 10DBA: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#3344431 (10Marostegui) p:05Triage>03Normal [12:23:58] 10Labs, 10DBA: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#2931704 (10jcrespo) We do have to do the sanitization of both sanitarium (currently, the 2 of them) and labsdbs. [13:36:05] 10Labs, 10DBA: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#2931704 (10chasemp) #dba gents please let us know when it's ready for views and meta_p fixup and we'll get on it! Thanks [13:37:05] 10Labs, 10DBA: Prepare and check storage layer for wikimania2018wiki - https://phabricator.wikimedia.org/T155041#3344674 (10Marostegui) Will do @chasemp! The DB isn't yet created. Thanks! [13:39:53] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3317175 (10chasemp) Small thought to which I'm not terribly attached but it is worth mentioning: Tools as an ecosystem is more than... [13:42:42] chasemp: good morning :) for when you have time, I would like to have one less Nodepool Trusty node available :D [13:42:49] hashar: sure [13:42:52] we are mostly using Jessie nowadays [13:42:53] https://gerrit.wikimedia.org/r/#/c/356466/2/modules/nodepool/templates/nodepool.yaml.erb [13:43:01] that lower the # of trusty nodes from 5 to 4 [13:43:14] so during busy hour let us run 1 more job on jessie [13:43:20] hashar: how about doing it now? [13:43:23] I am only dropping by 1 to be conservative [13:43:27] if you dont mind yes [13:43:33] rebasing [13:43:43] whenever nodepool run on the host, the yaml conf will be updated and nodepool automatically reread it [13:43:54] * chasemp nods [13:46:08] and I would like to try to lower the rate of queries slightly. It is currently at 6 seconds or 10 queries per minutes. And I would like to try 5 seconds or 12 queries per minutes [13:46:13] that would slightly speed up deletion of nodes [13:46:29] but iirc that causes some stress to the nova api, so would need to find a good window to do it [13:47:02] let's look at that first thing next week then [13:47:12] I'll be gone fri so maybe mon and we can watch for a few days pior [13:47:13] prior [13:47:25] I'm mostly fine w/ that depending on how it goes :) [13:47:39] want me to fill a task about it ? [13:47:45] oh sure yeah please do [13:47:58] * hashar is a phabricator addict [13:48:00] task+patch? :) [13:48:11] I can do both :] [13:49:10] from nodepool debug log: Deficit: ci-trusty-wikimedia: 0 (start: 0 min-ready: 4 ready: 5 capacity: 5) [13:49:14] ie min-ready is 4 :] [13:49:24] hashar: done [13:49:40] and confirmed. Thank you [13:49:45] I think this is a very very low risk change but let me know if things get weird hashar [13:50:11] that is effectively causing one more instance to be busy on the cloud during busy hour [13:50:20] so would potentially consume one morish cpu [13:50:31] we will see tonight when SF folks spam gerrit [13:50:34] kk [13:52:18] thank you ! [14:00:57] 10Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Release-Engineering-Team (Kanban): Lower rate of Nodepool requests to OpenStack API - https://phabricator.wikimedia.org/T167803#3344784 (10hashar) [14:01:32] 10Labs, 10Labs-Infrastructure, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Release-Engineering-Team (Kanban): Lower rate of Nodepool requests to OpenStack API - https://phabricator.wikimedia.org/T167803#3344799 (10hashar) I have quickly talked to @chasemp about it. It is best done early in a... [14:04:49] Hey andrewbogott, you around by any chance? [14:04:58] volans: what's up? [14:05:40] I was on wikitech on the Special:NovaProject page and I noticed that the links to add/remove members in the projectadmin role box are red and send me to an error page [14:05:46] You have requested an invalid special page. [14:06:02] ...title=Special:NovaRole&action=addmember... [14:06:22] while I can add members to the project, I cannot add them to the projectadmin role [14:06:25] has anything changed there? [14:07:10] not that I know of… what project? [14:07:51] puppet3-diff but I've just verified it happens also for automation-framework [14:07:57] so potentially all of them? :) [14:08:01] let me adjust the filter [14:08:29] 10Labs, 10Labs-Infrastructure, 10Patch-For-Review, 10cloud-services-team (Kanban): When an instance is deleted, remove proxy records that point to it - https://phabricator.wikimedia.org/T163765#3344815 (10Andrew) 05Open>03Resolved [14:08:47] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3341596 (10zhuyifei1999) `proxy_read_timeout 20d;` <= Does Shiny server use persistent connections? I'm unsure whether it works fine in the current configuration. (Although WebSoc... [14:09:08] volans: ok, I see what the issue is I think, I'll see if i can fix it [14:09:44] andrewbogott: great! no hurry from my side, I just noticed it and wondering if something has changed or was a bug :) [14:10:49] 10Tool-Labs-tools-Other: svgtranslate tool-Not Working - https://phabricator.wikimedia.org/T164275#3344828 (10Mikey641) Thanks [14:59:39] 10Labs-project-icinga2, 10User-Zppix: Make Icinga2-wm bot use IRC auth - https://phabricator.wikimedia.org/T167807#3344942 (10Zppix) [15:14:50] andrewbogott: thanks for https://gerrit.wikimedia.org/r/#/c/358604/ [15:16:56] volans: it won't fix anything until it gets cherry-picked to the branch, hopefully during SWAT later today [15:17:22] no problem at all ;) [15:36:01] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labcontrol100[34] - https://phabricator.wikimedia.org/T165781#3345077 (10Cmjohnson) [15:36:26] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labcontrol100[34] - https://phabricator.wikimedia.org/T165781#3276700 (10Cmjohnson) Racked in A6 and B7 [16:21:54] !log wikialbels 3b857d0 is going to staging [16:21:54] Amir1: Unknown project "wikialbels" [16:22:42] !log wikilabels 3b857d0 is going to staging [16:22:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [16:23:48] everything looks good, going to prod [16:31:03] !log wikilabels a24433f is going to staging [16:31:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [16:34:16] !log wikilabels deploying a24433f to prod [16:34:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikilabels/SAL [16:34:19] halfak: ^ [16:37:21] halfak: I tested and labels some stuff, everything looks just fine [16:37:25] I'm calling it done [16:40:09] (03PS2) 10Gehel: maps - renamed cassandra passwords for role / profile refactoring [labs/private] - 10https://gerrit.wikimedia.org/r/353068 [16:40:28] (03CR) 10Gehel: [V: 032 C: 032] maps - renamed cassandra passwords for role / profile refactoring [labs/private] - 10https://gerrit.wikimedia.org/r/353068 (owner: 10Gehel) [16:53:14] (03PS1) 10Gehel: maps - refactoring to role/profile: parameter renaming [labs/private] - 10https://gerrit.wikimedia.org/r/358633 [16:57:00] bd808: (or others): Are there docs for how a VPS project admin would add users? [16:57:22] meeple27: lets look! :) [16:57:34] i tried, although granted it was a 2 minute search, not 20 [16:58:06] (03CR) 10Gehel: [V: 032 C: 032] maps - refactoring to role/profile: parameter renaming [labs/private] - 10https://gerrit.wikimedia.org/r/358633 (owner: 10Gehel) [16:59:21] meeple27: the answer is by using https://wikitech.wikimedia.org/wiki/Special:NovaProject but I'm not sure there is a good tutorial. Also it is broken at the moment. :/ [17:00:08] this patch needs to be swatted -- https://gerrit.wikimedia.org/r/#/c/358604/ [17:00:30] so in a perfect world, would there be an "add user" item in the list of user doc "basics" on https://wikitech.wikimedia.org/wiki/Portal:Wikimedia_VPS ? [17:00:47] or is there an "admin docs" section somewhere? [17:00:47] andrewbogott: did you put that in for swat already? [17:01:07] oh duh, there's an admin section right below the user basics...so there maybe? [17:01:15] in a perfect world yeah there would be a "so you are running a project" page [17:01:51] probably belongs in the Project Administration section on the VPS portal [17:02:15] it also seems confusing to have VPS admins and admins of the VPS system. perhaps inevitable, but adjectives might help [17:02:50] help making the docs suck less is always welcome :) [17:02:50] ok. I don't feel qualified to add pages at this point. would it help you for me to create a phab doc task for this, and if so, in what project? [17:03:27] the #labs project and add a #documentation tag too [17:03:34] will do [17:04:24] #labs not cloud ??? :P [17:04:42] that rename hasn't happened in phab yet [17:05:07] #cloud-services-team exits but that is organizational, not product focused [17:05:30] makes sense. can't rename the whole world in a day [17:05:46] T167244 [17:05:46] T167244: Rename and update Cloud Services Phabricator projects - https://phabricator.wikimedia.org/T167244 [17:08:56] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3345372 (10Papaul) @RobH can we check please partman recipe for this system and labtestnet2002. I am stuck at partition during install, Please se... [17:09:17] 10Labs, 10Documentation: WMCS VPS docs should describe how to add users to a project - https://phabricator.wikimedia.org/T167816#3345375 (10ksmith) [17:09:29] Done: T167816 [17:09:32] T167816: WMCS VPS docs should describe how to add users to a project - https://phabricator.wikimedia.org/T167816 [17:10:33] bd808: so if we want to add a user to a project in the next day or two, would we ask you to do it? (we = jaufrecht [17:10:56] I don't think I can even do it at the moment. [17:11:03] bd808: and I need to reset my 2FA on wikitech for [mostly but not completely embarassing reasons] [17:11:11] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3345391 (10RobH) a:05Papaul>03RobH [17:11:37] ok. we will wait patiently [17:12:29] jaufrecht: ok. we need to follow the process from https://wikitech.wikimedia.org/wiki/Password_reset#Reset_two_factor_authentication [17:13:04] the easiest thing is a phab task and you making a file in your $HOME on an instance that references the task [17:13:17] assign to me and I'll take care of it [17:17:46] Quiero ver todos mis mensajed [17:34:37] bd808: yes, for swat this evening [17:36:46] ok. If I have time and there is a free hole in the schedule I may do it sooner [17:37:11] sure — I just don't like to step on releng's process [17:37:47] bd808 I assigned you a phab task and made a proof file: phlogiston-3:/home/jaufrecht/bd808.txt [17:38:15] jaufrecht: cool. let me check and do the needful [17:39:06] andrewbogott: *nod* I on the other hand love to jump to the front of the line :) [17:43:43] jaufrecht: {{done}} [17:44:07] great. I'll send the nigerian money order right away [17:44:18] hmmm... did I not setup wikibugs to ping here for cloud-*? [17:44:36] I have logged in successfully, bd808. [17:44:55] write down/print out your scratch codes this time :) [17:47:58] chasemp: about? i have a question for the racking of the two new eqiad labweb hosts [17:48:05] mainly what vlan they need to reside in [17:48:07] robh: k [17:48:15] https://phabricator.wikimedia.org/T163024 is the procurement task [17:48:19] im making the racking task now =] [17:48:24] they are onsite [17:49:13] robh: californium.wikimedia.org and silver.wikimedia.org that these are reeplacing are both in the public VLAN [17:49:25] that works! [17:49:31] so no row restrictions either [17:49:39] makes chris's job easier ;] [17:49:42] I know there has been some debate historically about where californium should go and I'm not sure where that stood but for the moment best to keep status quo I imagine [17:49:45] robh: nope [17:50:01] yeah if they suddenly need specific labs vlans they may have to relocate depending on what is required [17:50:09] but those are usually reimages to keep things simple [17:50:23] labweb1001 and labweb1002 work for hostname? [17:50:26] no worries, if it comes down to that it's on us [17:50:29] good w/ me [17:50:45] we haven't gone for a cloud* anyting yet on that respect so keep it consistent [17:51:31] andrewbogott: you ok w/ labweb1001 and labweb1002? [17:51:33] atm [17:51:52] for horizon? Sure. [17:52:06] yeah i assume when you guys wanna redo the hostnames you'll also audit them all for accuracy and restructure in a single rolling reimage [17:52:18] or hostname without reimage is also doable but messier [17:52:35] when you do make sure you coordinate with us so we update the labels, switch ports, etc.... [17:52:45] * robh is happy to help with that when the time comes [17:52:47] understood [17:52:57] tx robh [17:53:15] cloudweb1001 is annoying so idk what we'll do [17:54:28] i think cs-hostname [17:54:34] for cloud services servers [17:54:49] keep it short and consistent across the cluster so it doenst take too much hostname characters [17:55:18] so where there is overlap of cloud versus production servers (like cs-db, cs-puppetmaster, etc...) [17:55:37] are dashes friendly in all the places? [17:55:45] afaik we've used them for ages for ms-be [17:55:47] and ms-fe [17:55:49] I know puppet hates them in some places [17:55:53] k [17:56:22] i may spend too many cycles thinking about this kinda shit. [17:56:25] ;] [17:56:43] https://phabricator.wikimedia.org/T167820 [17:56:46] that has the racking plan [17:56:49] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-eqiad: rack/setup/install labweb100[12].wikimedia.org - https://phabricator.wikimedia.org/T167820#3345535 (10RobH) [17:57:12] Can we avoid 2-letter abbreviations? [17:57:32] robh: k tx, hey we were wondering about https://phabricator.wikimedia.org/T161345#3345499 [17:58:13] RainbowSprinkles: why? [17:58:25] RainbowSprinkles: we're taling about labs systems, and having a longer prefix makes the hostnames too long [17:58:27] Ask a newbie: is es2002 elasticsearch? [17:58:43] if they dont check the wikitech page that tells them [17:58:46] then i dunno what to tell them [17:59:06] we cannot go too long on hostnames, because i think we all know labtestpuppetmasterXXXX looks like hell [17:59:07] * RainbowSprinkles just thinks you should kind of know from a hostname what it does without consulting Some List [17:59:23] RainbowSprinkles: yeah, thats a longer discussion that im not gonna have here, cuz everyone in ops feels differently about it ;] [17:59:27] and it'll bikeshed for years [17:59:31] because it has ;] [17:59:52] es = external storage [17:59:56] I know that [17:59:58] And you know that [17:59:58] it existed before elastic search was created [18:00:15] longer hostnames are not feasible when they start to be as long as labtestpuppetmaster is all [18:00:34] RainbowSprinkles: So we have 2 letter hostnames now, we arent going to make them go away that i can see [18:00:43] and what i just suggested for the above was NOT a two letter one [18:00:47] cs-hostname [18:00:53] we also have element names which requires consulting "some list" to know what it is [18:00:53] just pointing that out ;] [18:00:56] and I think that's perfectly fine [18:01:14] i just want to make sure when we rename the labs to cloud it fits without being silly long =] [18:01:16] and I actually think it'd be fine to call one-off WMCS servers with element names as well [18:01:17] * RainbowSprinkles regrets having an opinion [18:01:25] RainbowSprinkles: sorry if it seemed like i was mad [18:01:27] * RainbowSprinkles goes away and finds something better to do that bikeshed [18:01:27] i totally was not! [18:01:37] ive had this conversation a lot though so i just have very set answers ;] [18:02:29] chasemp: no matter what hostname standard settled on, im happy to assist in the migration since there is a lot to tweak [18:02:54] racktables/physicallabel/network config on the onsite site. [18:03:24] great [18:03:40] RainbowSprinkles: sorry if i seemed mad i didnt mean to. i understand what you are saying and we do try to stick to a sensible standard that mostly makes sense for most use cases [18:04:16] its just a difficult process to pick a hostname that doesn't conflict. [18:04:50] I didn't think you were mad. I was mostly kicking myself for caring at all [18:07:28] thats what i do to myself when i find myself actually feeling any actual emotion when discussing hostnames [18:07:34] ive mostly quashed it entirely at this point [18:07:51] as long as its applied consistently and documented on the wikitech page, thats all i care about. [18:45:29] 10Labs, 10Tracking: Existing Labs project quota increase requests (Tracking) - https://phabricator.wikimedia.org/T140904#3345694 (10greg) [18:45:32] 10Labs, 10Labs-Infrastructure, 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External): Create a new instance flavor for deployment-prep - https://phabricator.wikimedia.org/T167723#3345690 (10greg) 05Resolved>03Invalid a:05hashar>03None [18:47:03] !log deployment-prep root@deployment-salt02:~# salt "*" cmd.run "apt-get -y install facter" [18:47:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:52:46] 10Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3345705 (10hashar) So that would be T160990 (high IO on swift instances). deployment-ms-be03.deployment-prep.eqiad.wmflabs should cause the same issue on whatever labvirt it is running on. Would you min... [18:53:59] 10Labs, 10Horizon, 10cloud-services-team (Kanban): Fix watroles to work with new Puppet storage backend for Labs - https://phabricator.wikimedia.org/T151522#3345707 (10Andrew) a:03Andrew [19:15:08] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3345754 (10RobH) [19:15:21] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3345755 (10RobH) a:05Papaul>03RobH [19:15:47] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3319508 (10RobH) I had listed raid10 it seems in the setup, but its a two disk system, so fixed and installing. [19:25:36] 10Labs, 10DBA, 10Epic: Labs database replica drift - https://phabricator.wikimedia.org/T138967#3345782 (10bd808) p:05Triage>03Normal [19:26:19] 10Labs, 10Labs-Infrastructure, 10DBA, 10Tracking: LabsDB replica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#3345788 (10bd808) p:05Triage>03Normal [19:30:28] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3345824 (10GoranSMilovanovic) @zhuyifei1999 Could you explain what do you mean by "persistent connections"? I am not a web-admin. Also, how do you propose the ```proxy_read_timeo... [19:33:05] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3345834 (10RobH) [19:33:11] 10Labs, 10Labs-Infrastructure, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3345835 (10RobH) [19:33:46] So who in cloud team gets the labtestnet2002 and labtestneutron2002 service deployment tasks? [19:33:55] install/puppet/salt is done [19:34:14] im assuming either chasemp or andrewbogott =] [19:34:34] chase i think, for now at least [19:34:39] that's fine [19:35:02] they are going to sit for a bit but we knew that upfront when we forwarded some Q1 things [19:36:10] 10Labs, 10Labs-Infrastructure, 10Operations: rack/setup/install labtestneutron2002 - https://phabricator.wikimedia.org/T167160#3345845 (10RobH) a:05RobH>03chasemp Assigning to @chasemp for service implementation. I also removed the onsite project tags, since all onsite work is completed for this system. [19:36:14] 10Labs, 10Labs-Infrastructure, 10Operations: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3345848 (10RobH) Assigning to @chasemp for service implementation. I also removed the onsite project tags, since all onsite work is completed for this system. [19:36:22] 10Labs, 10Labs-Infrastructure, 10Operations: rack/setup/install labtestnet2002 - https://phabricator.wikimedia.org/T167159#3345850 (10RobH) a:05RobH>03chasemp [19:55:26] 10Labs, 10MediaWiki-Cache, 10wikitech.wikimedia.org, 10MW-1.28-release-notes, 10Wikimedia-log-errors: "Memcached::touch(): touch is only supported with binary protocol" from wikitech as it's not running HHVM - https://phabricator.wikimedia.org/T143464#3345985 (10bd808) [19:56:38] 10Labs, 10BetaFeatures, 10Edit-Review-Improvements, 10wikitech.wikimedia.org, 10Collaboration-Team-Triage (Collab-Team-Q1-Jul-Sep-2017): ERI requesting opt-in on wikitech but not available - https://phabricator.wikimedia.org/T165822#3346001 (10jmatazzoni) [20:00:25] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3346020 (10yuvipanda) One of the things I'd like to do is to use an nginx ingress directly for getting traffic into the cluster, ins... [20:01:09] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3346024 (10zhuyifei1999) I'll experiment with it (try setting one up) later. By "persistent connections" I meant that (it's just my guess) Shiny expects the connection kept open,... [20:05:01] 10Labs, 10cloud-services-team (Kanban): disable service groups for non-tools projects - https://phabricator.wikimedia.org/T167204#3346041 (10Andrew) p:05Triage>03Normal a:05Andrew>03None Unassigning as this is currently blocked on new Striker features. [20:05:50] 10Labs, 10Horizon, 10cloud-services-team (Kanban): Fix watroles to work with new Puppet storage backend for Labs - https://phabricator.wikimedia.org/T151522#3346045 (10Andrew) p:05Triage>03Normal [20:05:55] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3346047 (10chasemp) I was thinking a separate nginx ingress in total, and ignoring tools-proxy-xx here for sure. This is a consider... [20:06:48] 10Striker, 10Epic, 10Patch-For-Review, 10cloud-services-team (FY2017-18): Manage shared tool accounts via Striker - https://phabricator.wikimedia.org/T149458#3346051 (10bd808) [20:06:51] 10Labs, 10cloud-services-team (Kanban): disable service groups for non-tools projects - https://phabricator.wikimedia.org/T167204#3346050 (10bd808) [20:08:46] 10Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3346079 (10Andrew) ms-be03 is on labvirt1001. It's not the biggest CPU user on that host, but it /is/ the second biggest. [20:09:06] 10Labs, 10Labs-Infrastructure: labvirt1006 super busy right now - https://phabricator.wikimedia.org/T165753#3346080 (10Andrew) p:05Triage>03Normal [20:10:11] 10Labs, 10Labs-Infrastructure, 10Patch-For-Review, 10cloud-services-team (Kanban): Audit disk usage on labvirts - https://phabricator.wikimedia.org/T163796#3346091 (10Andrew) a:05Andrew>03None [20:12:56] andrewbogott: for labvirt1006 , on the ms-be instance I have managed to save a bunch of cpu cycles with a low hanging fruit [20:13:02] but I guess most of the load is due to disk I/O :( [20:14:03] 10Labs, 10Labs-Infrastructure, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#3346114 (10Andrew) p:05High>03Normal right now I'm just checking periodically to see if there are new leaks. [20:15:44] 10Labs, 10PAWS, 10Tool-Labs, 10Tools-Kubernetes: Consider moving PAWS to its own k8s cluster, rather than using Tools' k8s cluster - https://phabricator.wikimedia.org/T167086#3346125 (10yuvipanda) On chatting more, if I have to use puppet then using the tools puppetmaster will make my life easier. So I'm... [20:20:54] 10Labs, 10Gerrit, 10wikitech.wikimedia.org: Request to rename LegoFan4000 to MacFan4000 on WikiTech - https://phabricator.wikimedia.org/T165624#3346137 (10bd808) >>! In T165624#3342899, @bd808 wrote: > * rename `MacFan4000` to `Abandoned-MacFan4000` using [[ https://wikitech.wikimedia.org/wiki/Special:Rename... [21:43:51] 10Labs: Need support on hosting an RStudio Shiny Server on a Labs instance behind a proxy - https://phabricator.wikimedia.org/T167702#3346298 (10GoranSMilovanovic) @zhuyifei1999 Are you a Shiny Server user too? Thank you very much for offering support. I have read the material that you have shared, however, I h... [22:20:28] 10Labs, 10Operations: virbr0 interface present in some virt hosts - https://phabricator.wikimedia.org/T83732#3346434 (10chasemp) Post hoc note. I noticed that `/etc/libvirt/qemu/networks/autostart/default.xml` is ensured absent in our nova compute role. This is a file that libvirt seems to generate and the c... [22:25:46] bd808: even if it's just for development purposes that OAuth consumer is pretty scary [22:26:13] tgr: the grant all the rights one? [22:26:14] yeah [22:26:31] but localhost so meh [22:28:42] should have at least a / at the end [22:28:45] our URL matching is pretty stupid [22:29:05] if you want to revoke it go for it [22:29:18] wouldn't bet on something like http://127.0.0.1:8000@evil.com not matching it [22:30:07] I still get pretty confused about the level of paranoia we have. OAuth grants are self-serve everywhere else I have ever used them [22:30:42] we should fix the url matcher if it is really broken [22:30:43] they are usually limited to reading / messing with your own data tho [22:31:10] we should, yes :) [22:31:19] posting to my twitter, editing all the github things I can touch, ... [22:31:27] acting as me with GOOG [22:31:47] yeah OK github is sufficiently scary [22:32:41] where is the url matching? just in the extension? [22:32:42] well I am happy to follow any policy that someone sets up, so far my efforts to generate interest in the topic haven't been very successful [22:33:12] yeah... nobody really cares I fear [22:33:46] https://github.com/wikimedia/mediawiki-extensions-OAuth/blob/master/backend/MWOAuthServer.php#L81-L107 [22:34:38] hmmm... that is pretty wide open [22:34:46] we can do better [22:35:41] there is a consumer that relies on being able to change the port, I think [22:36:00] that one will break then I suppose [22:36:07] fair enough [22:36:10] or we can special case port matching [22:39:30] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3346515 (10kaldari) @Niharika suggested that we get people to bid on UpWork (formerly oDesk). Apparently, you can get a bunch of people to create logos for you and then you just pay whoever has the bes... [22:41:55] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3346522 (10Niharika) See also - https://99designs.com [22:42:30] tgr: do you think we should allow port hopping? [22:42:48] I'm going to compare piecewise with wfParseUrl() output [22:43:01] bd808: better not to [22:43:24] that becomes a hurdle you need to mantain, later [22:43:36] specially if security issues appear [22:44:27] bd808: the use case was a vagrant role for OAuthAuthentication [22:44:50] if it's not too difficult to be fancy about it, handling * for a port would be nice [22:45:37] its as easy as just not checking that component [23:16:35] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3346606 (10kaldari) We will need both a largish logo and a small square favicon version (preferably as a vector graphic). [23:17:41] 10Tool-Labs-tools-Xtools, 10Community-Tech: Create an XTools logo - https://phabricator.wikimedia.org/T167345#3346610 (10kaldari) [23:43:48] 10Tool-Labs-tools-Xtools, 10Community-Tech, 10Epic: Epic: Rewrite Xtools: RfX Vote Calculator - https://phabricator.wikimedia.org/T165710#3346667 (10kaldari) [23:47:26] 10Tool-Labs-tools-Xtools, 10Community-Tech, 10Epic: Epic: Rewrite Xtools: RfX Vote Calculator - https://phabricator.wikimedia.org/T165710#3346670 (10kaldari) p:05Triage>03Low