[06:19:32] 10Phabricator: Resolving a task should not assign it to me - https://phabricator.wikimedia.org/T138806#2410881 (10bmansurov) [06:41:59] 10Phabricator: Resolving a task should not assign it to me - https://phabricator.wikimedia.org/T138806#2410909 (10Peachey88) [06:42:02] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106#2410911 (10Peachey88) [06:46:23] 10Phabricator: Phabricator is auto claiming tasks when closing as resolved (again) - https://phabricator.wikimedia.org/T134106#2410918 (10bmansurov) > What is the use-case for closing tasks but leaving them owned by nobody? That makes it difficult to search for tasks closed by a given user. When signing-off on... [09:49:17] 05Gitblit-Deprecate, 06Operations, 06Release-Engineering-Team, 07Developer-notice, and 2 others: Redirect Gitblit urls (git.wikimedia.org) -> Diffusion urls (phabricator.wikimedia.org/diffusion) - https://phabricator.wikimedia.org/T137224#2411057 (10Danny_B) >>! In T137224#2407860, @Dzahn wrote: > I merged... [09:51:52] 05Gitblit-Deprecate, 06Operations, 06Release-Engineering-Team, 07Developer-notice, and 2 others: Redirect Gitblit urls (git.wikimedia.org) -> Diffusion urls (phabricator.wikimedia.org/diffusion) - https://phabricator.wikimedia.org/T137224#2411064 (10Danny_B) I guess the cause may be the asterisk in ` A spammer: https://phabricator.wikimedia.org/p/Lullu121verma/ [13:30:45] Andre is on vacation, can someone deal with them, please? [15:56:18] Hi, can't think of a better place to ask: I can't access glamtools, it's just stuck loading. Is it down? [15:59:02] #wikimedia-labs would probably be a better place :) [15:59:24] not a development tool [16:03:16] thanks [16:03:18] :) [17:35:14] 10Phabricator, 06Design-Research, 06Team-Practices: Design-Research-Archive tasks seem to remain unresolved forever though there is often nothing left to do - https://phabricator.wikimedia.org/T138662#2406725 (10Capt_Swing) These can be closed. We were following @DarTar's system. Then he changed his system.... [18:37:54] 05Gitblit-Deprecate, 06Operations, 06Release-Engineering-Team, 07Developer-notice, and 2 others: Redirect Gitblit urls (git.wikimedia.org) -> Diffusion urls (phabricator.wikimedia.org/diffusion) - https://phabricator.wikimedia.org/T137224#2412573 (10Dzahn) Yes, the problem appears when these rules are adde... [18:48:36] 05Gitblit-Deprecate, 06Operations, 06Release-Engineering-Team, 07Developer-notice, and 2 others: Redirect Gitblit urls (git.wikimedia.org) -> Diffusion urls (phabricator.wikimedia.org/diffusion) - https://phabricator.wikimedia.org/T137224#2412605 (10Paladox) The redirect Bugzilla looks like https://phabric... [19:09:48] what is the difference between the phabricator ::labs role and the prod role? [19:10:01] is it easy to tell why we cant just use the prod role [19:10:10] since that would be much better for testing [19:10:16] twentyafterfour ^^ [19:10:32] or was it just one specific thing that cant work in labs [19:11:00] and separately, paladox , which error did you get when using the labs role on a new instance? [19:12:18] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item scap::deployment_server in any Hiera data file and no default supplied at /etc/puppet/modules/scap/manifests/target.pp:97 on node git-phab4.git.eqiad.wmflabs [19:12:19] Warning: Not using cache on failed catalog [19:12:19] Error: Could not retrieve catalog; skipping run [19:12:19] trusted_proxies => $cache_misc_nodes[$::site], [19:12:19] line 34 ^^ [19:12:22] mutante ^^ [19:13:10] ah.. hmm [19:13:41] wondering if that just happens because you are in a different project and is fixed in the phabricator project [19:13:56] mutante not sure maybe [19:14:02] because it's a hiera lookup... [19:14:08] Oh. [19:14:25] so you saw phab-01 thrugh phab-05 ? [19:14:38] are they all configured they same way and using that ::labs role? [19:15:07] do you have access for that? [19:16:25] mutante [19:16:28] i doint have access [19:16:31] But you do [19:16:40] And i doint think there configured the same [19:16:42] as each other [19:17:27] which of the instances should be used for testing? [19:17:32] (asking the channel) [19:17:53] for testing the Apache setup that is.. so it's as close to prod as possible [19:18:31] or i can just make yet another one [19:18:43] mutante looks like this one https://wikitech.wikimedia.org/wiki/Nova_Resource:Phab-04.phabricator.eqiad.wmflabs [19:18:46] apply that ::labs role and see what happens [19:18:49] but yes set up another one [19:18:58] please [19:19:16] i would like it if it wasnt't a "labs" role [19:19:34] mutante we could try prod puppet role [19:19:36] ok, what made you pick -04 ? [19:19:55] Since it is the only one showing it as using the phabricator labs role. [19:20:03] and was running ubuntu trusty [19:20:08] same as the prod machine [19:20:08] mutante: The prod role has a bunch of things that don't work in labs, but could probably be dealt with [19:20:39] you can use any of the phab-0* for testing [19:21:10] twentyafterfour i think phab-01 keeps going down due to small instance [19:21:23] i think all phab instances needs to be a medium or higher [19:21:27] twentyafterfour: ah! hi! that would be cool as a medium term goal. have an instance that is using the prod role and has no other changes. so some instance is for staging new stuff and one is just always like prod [19:21:36] since i had the problem with running phd the daemons and it crashing. [19:21:48] paladox: ah, you're probably right [19:21:49] twentyafterfour: short term it's just where we test those rewrite rules that DannyB wrote.. [19:22:13] mutante: yeah like paladox said, phab-04 sounds like a fine one or any of them really [19:22:16] twentyafterfour yep, would there be a way to swith phab-01 to a medium to prevent any more crashes please. [19:22:23] I was working on phab-tin for deployment via scap [19:22:30] but that's not quite set up right atm [19:22:45] paladox: I don't know how to switch it other than destroy and re-create [19:22:50] oh [19:23:11] it's probably better to destroy and re-create [19:23:15] anyways [19:23:26] twentyafterfour could you help us test the rewrites for git.wikimedia.org please. [19:23:35] Since we broke bugzilla by mistake. [19:23:51] so these rules seem to be ok when they are standalone [19:24:01] but once combined with the existing setup for bugzilla [19:24:03] it breaks [19:24:34] mutante would havent it set on a different apache port work [19:24:48] that's why we need tests with the same apache config and it's not sufficient to run on git.wmflabs or a laptop [19:24:53] I found with gerrit and jenkins on a test instance i had to do that to allow them to work. [19:25:37] but we dont want to run 2 Apache instances in prod anyways [19:25:46] so the rules need to work together [19:26:04] oh ok [19:26:44] that would mean 2 different varnish backends on the same instance.. etc.. just for this [19:26:58] it's about the VirtualHosts and the order they are in [19:27:05] oh [19:27:17] brb [19:27:53] or something.. we need to compare the Apache config part between 04 and prod too [19:28:32] the root of the issue remains that nothing is a real test if we have diffs [19:34:03] paladox: I'm on train duty [19:34:09] I can try to help test [19:39:53] Im back [19:39:58] twentyafterfour Ok [19:40:19] mutante ^^ [19:40:50] twentyafterfour hi, could you have another look at https://phabricator.wikimedia.org/D273 please or do you have a better way please [19:42:59] twentyafterfour: unrelated..i got pinged about rebooting iridium tomorrow during phab maintenace.. for kernel upgrade [19:43:28] mutante do you want to use phab-04 or should we create another instance [19:43:31] to test on please [19:45:32] mutante: I can probably do the reboot from the shell but it wouldn't hurt to have an opesen around just in case [19:45:47] twentyafterfour: ok, i can be around, np [19:46:21] twentyafterfour currently i think https://phabricator.wikimedia.org/D273 is the only way for now. [19:46:31] Since it will cause refs/cache* to break repos [19:46:45] currently without that patch [19:48:57] mutante :) [19:50:16] paladox: did you want to request access to phab project? [19:50:37] mutante im not sure, how do i do that. [19:50:43] please [19:51:04] not sure either [19:51:19] Oh, maybe i could ask twentyafterfour [19:51:44] twentyafterfour hi, would i be able to access the https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator please [19:51:46] per mutante [19:51:54] paladox: added you to the project [19:51:59] :) [19:52:14] I thought you were already a member :) [19:52:25] twentyafterfour thanks, and no, i was on a different project. [19:52:27] :) [19:52:45] made you a projectadmin as well [19:52:53] twentyafterfour thanks :) [19:53:04] mutante should i create the instance? [19:55:21] paladox: sure, it won't hurt to make a new one. as Ryan always said, we should treat instances "like cows not pets" [19:55:29] create, test, remove [19:55:33] mutante ok :) [19:55:37] which size [19:55:39] please [19:55:55] default / same as the other existing ones [19:55:55] medium? [19:56:04] Ok thanks twentyafterfour [19:56:18] how do i ssh. Do i do the same. ssh into bastion. [19:56:22] ? [19:56:28] yes it's the same [19:56:59] I have *.wmflabs set up in my .ssh/config so it automatically uses the bastion [19:57:12] twentyafterfour oh, thanks do i set the os as trusty [19:57:16] ? [19:57:38] please [19:57:59] paladox: yes [19:58:04] Ok thanks [19:58:12] paladox: ssh just like we set it up for git.wmflabs.org [19:58:20] that instance i made in the other project [19:58:25] mutante ok, tahnks [19:58:27] thanks [19:58:37] except you have to put .phabricator. in host name for this [19:58:39] mutante and twentyafterfour ive set it up as phab-05 [19:58:45] medium and os is trusty [19:58:46] :) [19:58:48] phab-04.phabricator.eqiad.wmflabs [19:58:52] same style as this [19:59:00] and yes, use trusty [19:59:04] because prod is trusty [19:59:18] Ok [19:59:20] :) [19:59:58] thanks paladox, i will be back in a few minutes [20:00:10] mutante ok, and your welcome. [20:00:12] then we can compare Apache config [20:00:14] with iridium [20:00:21] after you applied the role [20:01:55] ok [20:02:23] twentyafterfour i think phab-01 has crashed again since website is not working properly and i carn't seem to ssh into it. [20:02:27] What should we do. [20:02:46] Should we destroy it and re create it as a medium instance to prevent it crashing all the time. [20:03:16] i think there are 2 separate use cases here [20:03:30] one is to test new stuff before it gets to prod [20:03:34] Yep [20:03:44] and the other is to test changes on something that is like the existing prod [20:04:18] we could define which instance is for what [20:04:28] Yep [20:04:31] or use slightly different names [20:04:35] instead of just the numbers [20:04:42] mutante i think one of the instances doint work. [20:05:02] make a new phab-01 too but separate from what you just did [20:05:30] -staging vs -somethingelse .. hmm [20:05:33] mutante should i delete phab-01 and recreate it [20:06:42] mutante rebooting phab-01 brings up ERROR [20:07:39] hmm. do not delete it [20:07:46] mutante oh, what do i do [20:07:55] How do i get it to work [20:08:02] please [20:08:18] re-open https://phabricator.wikimedia.org/T137270 [20:08:23] and ask there what to do [20:08:31] ok [20:10:44] 10Phabricator, 06Labs: Upgrade phab-01.wmflabs.org - https://phabricator.wikimedia.org/T127617#2412816 (10Paladox) [20:10:48] 10Phabricator, 06Labs: https://phab-01.wmflabs.org returns a core exception - https://phabricator.wikimedia.org/T137270#2412814 (10Paladox) 05Resolved>03Open Re opening this. I tried ssh into it but failed I looked at the website and it showed http 500 error so I rebooted it but now it failed to reboot.... [20:12:00] mutante i can setup another instance and migrate it from phab 01 since we can re point the domain dns to the new instance then fix up phab-01 and re point it back to it. [20:12:57] mutante says this now [20:12:58] channel 0: open failed: connect failed: No route to host [20:12:58] stdio forwarding failed [20:12:58] ssh_exchange_identification: Connection closed by remote host [20:15:54] mutante it says this [20:15:55] Messageinternal error: No PCI buses available [20:15:58] twentyafterfour ^^ [20:17:16] paladox: ok, let's handle that via the ticket. let's focus on the redirects for now [20:17:44] mutante ok [20:18:00] that pci buses notice I believe is related to general labs overload [20:18:03] re-pointing DNS is just clicking in wikitech ui too btw [20:18:17] "manage proxies" [20:18:19] I don't have a minute to try to shuffle that instance now but if there is a task cc me on it [20:18:42] chasemp oh this one https://phabricator.wikimedia.org/T137270 please [20:19:23] paladox: let's look at -05 [20:19:25] yeah that's fine there is a general ticket for instances struck by overload [20:19:31] but I can't get to this right this moment [20:19:36] mutante ok [20:19:39] idk why we have so many phab instances tbh [20:20:02] and chasemp i doint think we have a ticket for the issue like that. But we have one for phab-01 going down [20:20:25] because you never know what the others have been used for, so you create a new one to be sure it's "clean" [20:20:43] Yep [20:20:46] i think that is why [20:21:39] only of them is configured to use the ::labs role when paladox just checked [20:21:46] Yep [20:21:47] and none with the prod role [20:21:54] so the others are manual testing [20:24:02] what we wanted is just one of them to be actually like prod otherwise you dont know if the test is a real test [20:24:14] yep [20:24:16] we should clean sweep those VMs [20:24:39] the labs role comes w/ a different banner since no oauth, local mysql, and none of the email crazy routing or prod monitoring [20:24:52] afa phab itself is concerned there isn't a difference [20:25:03] it's the ecosystem that has to be self contained or nonexistent [20:26:06] ok, thanks, we wanted to find out just that, what the difference is in the ::labs role [20:26:31] chasemp is there a way to swap phab-01 to a medium instance [20:26:40] without deleting please [20:26:42] no, other than delete it and set up a new instance [20:26:50] and we need ot remove some of these to do that [20:26:54] Oh ok [20:26:56] we are literally on overload atm in labs [20:26:56] actually all we need right now is the same Apache config .. and a new proxy [20:27:06] mutante we can rename the instance [20:27:15] then setup a new phab-01 [20:27:34] you cannot rename the instance like that in the VM itself if that's what you mean [20:27:46] we just change the proxy config , paladox [20:27:57] but let's stick to one of the issues at a time [20:28:04] chasemp oh, but it shows the option to do it. [20:28:26] in horizon? [20:28:40] Yep [20:28:41] I wouldn't trust that at all, there are too many things associated and I'm not sure it's been tested [20:28:48] puppet certs, etc [20:28:51] chasemp ok. [20:30:06] paladox: did you apply a role on the new instance yet? [20:30:18] mutante im going to do that now [20:30:21] cool [20:30:35] mutante how do i apply the role again in puppet on instance [20:30:39] not on wikitech [20:30:40] please [20:31:01] you click "configure instance" in wikitech [20:31:12] mutante yep, i mean in instance [20:31:15] The command [20:31:20] please [20:31:25] you can't afaik [20:31:28] Oh [20:31:34] i thought you run puppet [20:31:37] puppet run [20:31:43] puppet agent -tv [20:31:46] Thanks [20:31:47] yes, you do that [20:32:01] but after telling it via wikitech which role class to use [20:32:14] It says [20:32:14] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: DNS lookup failed for phab-tin.phabricator.eqiad.wmflabs Resolv::DNS::Resource::IN::A at /etc/puppet/modules/scap/manifests/target.pp:98 on node phab-05.phabricator.eqiad.wmflabs [20:32:14] Warning: Not using cache on failed catalog [20:32:14] Error: Could not retrieve catalog; skipping run [20:32:48] hmm.. that means the ::labs role is broken? [20:33:02] try running puppet on the -04 instance [20:33:08] same error ? [20:33:41] or failed DNS lookups are also symptoms of the general labs overload? [20:34:36] if that's the case we can't really test it i guess [20:34:56] maybe we should postpone the announced change [20:35:09] no dns should be ok, no clue whwat the deal is there [20:35:13] mutante phab-04 is suffering the same as phab-01 [20:35:23] plus i coulden ssh into phab-04 [20:35:24] that's a scap issue [20:35:31] probbly because they moved deploy of phab to scap3 [20:35:35] sigh, ok. so we have instances without a role.. and the ones with the broken role [20:35:44] and maybe none of this labs stuff was ever dealt w/ [20:35:49] that's a releng question I guess [20:35:52] how hae they been testing? [20:35:53] mutante we can create a new puppet rule without scap ? [20:35:53] idk [20:36:03] yes, what chase just said [20:36:11] i wanted to say the same [20:36:23] Ok [20:36:32] i also dont know [20:37:03] twentyafterfour can we help [20:37:04] please [20:37:07] paladox: can you see the Apache config on -04 though? [20:37:12] mutante nope [20:37:15] .. [20:37:16] i carn't ssh into it [20:37:21] plus it is suffering the same fault [20:37:24] as phab-01 [20:38:08] did you keep track of the number of issues ?;p [20:38:17] since we "just" wanted to test [20:38:35] can you ssh to any other instance? [20:38:38] let me try as well [20:39:20] mutante ok, i will try the 2nd and 3rd one [20:40:15] ssh root@phab-04.phabricator.eqiad.wmflabs [20:40:15] channel 0: open failed: connect failed: No route to host [20:40:15] stdio forwarding failed [20:40:20] yea.. no [20:40:36] Yep [20:40:40] I get that error [20:41:26] i dont know what to say, but nothing workds today [20:41:32] Yep [20:45:11] i am on -05 now, but doesnt help because of the puppet issue [20:46:17] so now we could try to manually copy the Apache config .. and then we have yet another instance with manual stuff :p but we can delete it right after we used it [20:46:24] at least [20:47:34] that also means manually installing Apache, unless we have another role that currently does that [20:47:55] there used to be webserver::php5 but pretty sure that is historic and doesnt work anymore [20:48:03] looks [20:48:22] Oh [20:48:54] do you see any other role that instance use, that installs Apache? [20:49:14] well, it's also puppet groups _per project_ .. [20:49:27] so we'd have to add that in phab project context [20:50:10] mutante [20:50:11] apache: [20:50:11] [20:50:11] [20:50:11] role::simplelamp [20:50:28] ah, right. wanna try that on -05 ? [20:50:40] mutante ok [20:50:43] I will do that now [20:51:20] It says [20:51:21] Notice: Run of Puppet configuration client already in progress; skipping (/var/lib/puppet/state/agent_catalog_run.lock exists) [20:51:23] mutante ^^ [20:51:53] that's when puppet is already running [20:52:01] Oh [20:52:01] you can just wait [20:52:03] Ok [20:52:06] or tail -f /var/log/syslog now [20:52:10] Ok [20:52:12] to see the output of the ongoing run [20:52:44] how do i exit it [20:52:46] mutante [20:52:46] it is started via cron, so that was just timing [20:52:58] meaning the screen [20:53:01] please [20:53:20] exit from "tail -f?" ctrl+c [20:53:30] Thanks [20:53:46] i see it installed apaache and mod_rewrite. which we need [20:53:47] good [20:53:53] Notice: /Stage[main]/Role::Labs::Instance/Exec[enable_sites_local]/returns: executed successfully [20:53:53] Notice: Augeas[Apache2 logs](provider=augeas): [20:53:53] --- /etc/logrotate.d/apache2 2016-05-04 17:07:02.000000000 +0000 [20:53:53] +++ /etc/logrotate.d/apache2.augnew 2016-06-28 20:53:43.052539723 +0000 [20:53:53] @@ -1,7 +1,7 @@ [20:53:55] /var/log/apache2/*.log { [20:53:57] - weekly [20:53:59] + daily [20:54:01] missingok [20:54:03] - rotate 52 [20:54:05] + rotate 30 [20:54:07] compress [20:54:09] delaycompress [20:54:10] yea, ok, no need to paste all that [20:54:11] notifempty [20:54:13] Notice: /Stage[main]/Apache::Logrotate/Augeas[Apache2 logs]/returns: executed successfully [20:54:15] mutante ^^ [20:54:22] yea, that's normal [20:54:29] Ok sorry [20:54:30] doesnt matter for us for testing [20:54:37] Ok [20:55:04] ok, hold on.. getting config from prod [20:55:13] mutante http://phab-05.wmflabs.org/ [20:55:15] Ok [20:55:26] you can try making a proxy [20:55:42] mutante already done http://phab-05.wmflabs.org/ [20:56:04] oh, ok. do we need to delete the old one? [20:56:09] so git.wmflabs is free again [20:56:12] and use it for this [20:56:18] Ok [20:56:48] yea, in the context of the project "git", you can delete that, and also the instance if you want [20:57:05] and then re-create it and let it be used by phab-05 [20:57:24] so we can test against git.wmflabs.org [20:57:27] Ok [20:57:35] Actually i also use project git [20:57:40] but i can delete the domain [20:57:43] if you want [20:57:47] then just delete the domain, yea [20:57:49] the proxy [20:57:50] Ok [20:58:47] mutante done https://git.wmflabs.org/ [20:59:43] on iridium, prod phab [20:59:53] we have 50-phabricator.conf [21:00:04] 6 ServerName phabricator.wikimedia.org [21:00:04] 7 ServerAlias phab.wmfusercontent.org [21:00:15] but that file does not have the Bugzilla rules [21:00:51] going back to DannyB's change [21:01:41] Ok [21:02:14] yea, so it doesnt work like that either.. hrmm [21:02:19] Oh [21:02:25] we need the result of the change that had to be reverted [21:02:42] i cant just copy existing config [21:02:47] mutante what do you mean by results of change. [21:02:58] https://gerrit.wikimedia.org/r/#/c/296085/2/modules/phabricator/manifests/init.pp [21:03:16] https://gerrit.wikimedia.org/r/#/c/296138/ [21:03:18] mutante ^^ [21:03:39] yea, so put that on this intance [21:03:49] mutante oh, how do i do that [21:03:51] please [21:04:01] that's what i mean by "we need the result of the change" [21:04:26] the content of the template https://gerrit.wikimedia.org/r/#/c/296138/2/modules/phabricator/templates/gitblit_vhost.conf.erb [21:04:36]