[00:00:31] any puppet failures being reported now are mine [00:04:48] starts merging [00:04:58] hahaha [00:05:00] i had the same thought [00:05:12] haha :P [00:05:14] 17:05 the magic number :p [00:05:15] on this channel only! [00:05:32] makes you think of merges, nbut only Friday [00:05:40] I had actually forgotten it's friday... [00:05:43] and 5pm... [00:05:49] bonus: before offsite [00:06:32] PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [00:06:35] but that also means get it done quick before [00:06:36] hehe [00:06:46] there we go [00:06:51] *g* [00:06:52] those are the puppet failures I was tlaking about [00:07:10] 60% broken, so just kind of [00:08:17] hahah [00:26:48] goddamit puppet [00:26:53] why won't you recognize a modulepath!? [00:31:36] chasemp: now that you are in the doc, please let me know if I'm doing useless things, or if I can show some other things that are more useful for your team. ;-) [00:32:16] sure, thanks this is really great [00:46:34] RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:27:39] PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:29:32] PROBLEM - Puppet failure on tools-proxy-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:36:01] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-117: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1717003 (10yuvipanda) Done! (ish!). Still needs I think to figure out how to make hiera also look at the secret repo for... [01:37:34] RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:35] RECOVERY - Puppet failure on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:04:56] PROBLEM - Puppet failure on tools-puppetmaster-01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:13:33] PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [02:14:57] RECOVERY - Puppet failure on tools-puppetmaster-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:28:36] RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:54:36] PROBLEM - Puppet failure on tools-k8s-master-01 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [03:04:33] RECOVERY - Puppet failure on tools-k8s-master-01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:07:21] PROBLEM - Puppet failure on tools-proxy-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [03:15:48] I think that's all covered now [03:25:54] 6Labs, 10Tool-Labs: Expose revision.rev_content_format on replicated wikidatawiki - https://phabricator.wikimedia.org/T56164#1717074 (10liangent) Seems completely resolved? [03:27:23] RECOVERY - Puppet failure on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [04:18:56] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:19:37] uhm [04:19:39] wt [04:19:50] that's indeed true [04:19:52] wtf [04:20:47] no [04:20:49] not true [04:23:49] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 900816 bytes in 3.074 second response time [04:26:28] * legoktm hugs yuvipanda [04:27:01] saw IRQ overload on the machines [04:27:03] no idea why [04:27:17] thanks legoktm :) [04:27:33] IRQ? [04:33:40] legoktm: interrupt requests [04:33:46] possibly a network burst [04:34:02] nothing I can do about now [04:34:07] o.O [04:35:23] nope [04:35:27] no spikes for tools-proxy-02 [04:44:15] I'm going to let it be [04:44:35] wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Admin will allow switching over the proxy if need be [05:44:48] 6Labs, 10Tool-Labs, 3Labs-Sprint-114, 3Labs-Sprint-115, and 2 others: Add support to dynamicproxy for kubernetes based web services - https://phabricator.wikimedia.org/T111916#1717122 (10yuvipanda) Do we still need python3-requests that's different? I don't find any actual errors there, but not sure if tha... [05:58:26] PROBLEM - Puppet failure on tools-proxy-02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:02:18] it's ok puppet [06:02:20] that's me [06:02:25] I'll get it fixed up in no time, etc [06:03:37] 6Labs, 10Tool-Labs, 3Labs-Sprint-114, 3Labs-Sprint-115, and 2 others: Add support to dynamicproxy for kubernetes based web services - https://phabricator.wikimedia.org/T111916#1717127 (10yuvipanda) Yes, it looks like it's being read from `/usr/local` (the requests library) so it was definitely imported fro... [06:07:39] 6Labs, 10Tool-Labs, 3Labs-Sprint-114, 3Labs-Sprint-115, and 2 others: Add support to dynamicproxy for kubernetes based web services - https://phabricator.wikimedia.org/T111916#1717128 (10yuvipanda) Ok, so the token authentication is sorted out now \o/ [06:12:13] 6Labs, 10Tool-Labs, 3Labs-Sprint-114, 3Labs-Sprint-115, and 2 others: Add support to dynamicproxy for kubernetes based web services - https://phabricator.wikimedia.org/T111916#1717129 (10yuvipanda) They're also running on a readonly proxy-infrastructure account now! [06:13:24] RECOVERY - Puppet failure on tools-proxy-02 is OK: OK: Less than 1.00% above the threshold [0.0] [06:14:57] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-117: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1717130 (10yuvipanda) I think this is all done now. [06:15:15] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-117: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1717131 (10yuvipanda) There's no secret repo on the labs puppetmaster and that seems to be ok. [06:15:26] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1717134 (10yuvipanda) [06:15:28] 6Labs, 10Tool-Labs, 5Patch-For-Review, 3labs-sprint-117: Setup a way to store secrets and access them from puppet inside the Tool Labs project - https://phabricator.wikimedia.org/T112005#1717132 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [06:16:38] 6Labs, 10Tool-Labs, 3Labs-Sprint-114: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1717135 (10yuvipanda) It's all good now! We have a 'client-infrastructure' account for kubelet / kube-proxy, and 'admin' and 'namespaced' accounts. This is all controlled from `hie... [06:16:41] 6Labs, 10Tool-Labs, 3Labs-Sprint-114: Setup and verify authentication for Kubernetes - https://phabricator.wikimedia.org/T111904#1717136 (10yuvipanda) 5Open>3Resolved a:3yuvipanda [06:16:42] 6Labs, 10Tool-Labs: Initial Deployment of Kubernetes to Tool Labs (Tracking) - https://phabricator.wikimedia.org/T111885#1618964 (10yuvipanda) [06:17:28] 6Labs, 10Labs-Team-Backlog: Make sure that the 'secret' repo in self hosted puppetmasters is back-upable - https://phabricator.wikimedia.org/T115177#1717139 (10yuvipanda) 3NEW [06:17:38] 6Labs, 6operations, 10Labs-Team-Backlog: Make sure that the 'secret' repo in self hosted puppetmasters is back-upable - https://phabricator.wikimedia.org/T115177#1717146 (10yuvipanda) [06:24:23] 6Labs, 10Tool-Labs, 5Patch-For-Review: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1717155 (10yuvipanda) I think adding [trusted=yes] is a much simpler solution, and also mirrors our labsdebrepo setup. Going to implement that... [06:29:25] 6Labs, 10Tool-Labs, 5Patch-For-Review: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1717166 (10yuvipanda) We can maybe keep this task open (or open new ones?) to have gpg keys setup, but I guess the real way to do it would be to have a keyring with all the to... [06:58:44] air please help me [06:59:03] sir please help me [07:18:51] RECOVERY - Puppet failure on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [09:46:37] PROBLEM - Puppet staleness on tools-k8s-bastion-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [15:58:58] hi all - about a year ago I had signed up for a basic account here, perhaps that gave privs to spawn a VM ... I am revisiting some notes.. [15:59:20] "Your user rights were changed by Tim Landscheidt. You are now a member of this group: shell." [15:59:43] https://wikitech.wikimedia.org/wiki/User_talk:Darkblueb#Your_shell_access_was_granted [16:00:56] can anyone point me to current info on my status ? [16:04:42] I have logged in with password auth at that page now [16:05:12] i.e. this is actually me talking ;-) [16:17:01] dbb, umm [16:17:03] dbb, you should be able to connect to bastions [16:17:22] bastions is a person ? [16:17:35] but not much more [16:17:37] no [16:17:39] bastion servers [16:18:23] ok - I am not current on the infra here.. so a basic review, and the status of my login, would be great [16:22:31] I can physically go to San Francisco to present ID if that is what is needed [16:22:40] (I am in the SF Bay Area) [16:28:09] no, that's not needed [16:29:04] the bastion servers control access to the labs internal network and therefore the actual servers in most projects [16:29:13] ok [16:29:22] this doesn't mean you can create instances yet [16:29:28] to do that you need to be admin of a project [16:29:34] what do you want to do in labs exactly? [16:29:36] ok that is what I am asking [16:30:03] depending on what you want to do, there may be existing servers you can use etc. [16:31:28] e.g. if you just wanted to query the replicas of wikimedia production database, access to the tools project is all that's needed. [16:31:50] ok - well I am writing a proposal .. it is related to geo data [16:32:13] I have never gotten anyone from the "geohack" to respond, in several years.. but that is very much related to what I am writing about [16:34:08] eventually multi-language support would be great, but right now my emphasis will be in English [16:46:25] 6Labs, 10Tool-Labs, 5Patch-For-Review: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1717466 (10scfc) 5Open>3Resolved a:3scfc [16:47:00] 10MediaWiki-extensions-OpenStackManager: RC logging issue with instance creation - https://phabricator.wikimedia.org/T38515#1717468 (10Krenair) [16:47:06] 6Labs, 10Tool-Labs: Sign packages on aptly server with Tool Labs packages key - https://phabricator.wikimedia.org/T112901#1717474 (10scfc) [16:47:07] 6Labs, 10Tool-Labs: Create Tool Labs packages key - https://phabricator.wikimedia.org/T112905#1717471 (10scfc) 5Open>3declined a:3scfc With T112699 resolved, no longer necessary. [16:47:11] 6Labs, 10Tool-Labs, 5Patch-For-Review: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1643006 (10scfc) [16:47:13] 6Labs, 10Tool-Labs: Sign packages on aptly server with Tool Labs packages key - https://phabricator.wikimedia.org/T112901#1717475 (10scfc) 5Open>3declined a:3scfc With T112699 resolved, no longer necessary. [16:49:47] dbb: well ive been banned for the last year, sorry [16:50:23] ?? [16:51:09] You're in geohack, Dispenser? [16:51:45] all my user groups were removed [16:51:53] maybe using the word "hack" is bad luck :-/ [16:52:03] AWS is crawling with bad actors [16:52:24] 6Labs: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#1717483 (10Krenair) Isn't this done? We have Hiera:$Project/host/$instancename now: ```mysql> select page_title from page where page_title like '%/host/%' and page_namespace = 666; +-----------------------------... [16:52:39] 6Labs, 10MediaWiki-extensions-OpenStackManager: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#1717485 (10Krenair) [16:53:03] anyway ive got an interface improvement to fix geohacks usability issues [16:53:38] Dispenser: I would like to programatically access the result set of POINT-of-interest or BBOX-of-interest [16:54:30] if there is some nice way to get links to wikipedia pages for referenced things, that would be great [16:54:32] wrote that in 2009 was it [16:54:37] sure [16:54:42] 6Labs: OpenStack APIs opened to end users - https://phabricator.wikimedia.org/T85612#1717487 (10Krenair) Are you announcing that they have been opened? Or is this task actually supposed to be about opening them? [16:54:52] have to fix wma's implementation [16:57:13] 6Labs, 10Tool-Labs, 5Patch-For-Review: Packages from the aptly server are not installable - https://phabricator.wikimedia.org/T112699#1717489 (10scfc) a:5scfc>3yuvipanda Removed. [17:17:38] devunt, around? [17:18:32] 6Labs, 10wikitech.wikimedia.org: Can't login into a newly created nova instance through ssh - https://phabricator.wikimedia.org/T113451#1717505 (10Krenair) Is it still broken? [17:19:50] Dispenser: pointer to recent docs ? [17:20:12] or, are you in some bad status .. maybe someone else [17:27:22] 6Labs: Cleanup leftover lucid instances on Labs - https://phabricator.wikimedia.org/T113199#1717507 (10Krenair) labs-vmbuilder-lucid and nginx-dev1 are not running (Shutdown), opengrok-web is in error status and conventionextension-trial is shutoff but it's Power State is NOSTATE... [17:30:19] 6Labs: project members don't see "Living-style-guide.design" instance in Nova_Resource:Design on wikitech - https://phabricator.wikimedia.org/T113214#1717509 (10Krenair) 5Open>3Resolved a:3Krenair [17:30:29] 6Labs: project members don't see "Living-style-guide.design" instance in Nova_Resource:Design on wikitech - https://phabricator.wikimedia.org/T113214#1657568 (10Krenair) a:5Krenair>3Andrew [17:34:28] 6Labs: no route to host trying to reach labs instances - https://phabricator.wikimedia.org/T113025#1717522 (10Krenair) So are these both fixed? [18:31:00] I am in my account on wikitech.wikimedia.org .. what is the "token" ? [18:31:20] where do I find out about that ... [18:31:44] dbb: you don't need to use it if you do not have two factor authentication enabled [18:32:05] hi yuvipanda - thx for the reply.. I am looking to get current on the shell account I was issued last year [18:32:21] (I havent used anything for ten months or so) [18:32:30] ah [18:32:33] for toollabs? [18:32:54] what would you recommend, to make a small program that responds to a URL with a geo location in it.. to get "geohack" type results [18:33:04] should I think about a whole VM, or use a service or what [18:33:18] ah [18:33:22] yes, you should use toollabs [18:33:35] https://wikitech.wikimedia.org/wiki/Help:Tool_Labs [18:33:39] that's what geohacks looks [18:33:40] err [18:33:43] works [18:35:34] I do not want to do big setup today, but i do want to ... [18:35:46] a) be current in my login [18:35:55] b) get an idea of the services I have access to [18:36:24] c) write that into a small example program, say in IPython Notebook making curl type calls [18:36:39] that way I can put the proposal forward, and see if it sticks [18:37:30] thx for any tips [18:38:38] I'm on my phone and not able to help much outside of pointing to docs today, unfortunately [18:38:44] ok [18:38:46] sorry! [18:38:50] np [18:39:39] all your questions should be answered from the docs though. you should have info on ssh, list of services you have access to, and how to get an example program going - I suggest using the uwsgi webservice setup [18:40:32] ??ssh [18:40:41] help ssh [18:40:44] ?ssh [18:41:15] oh hm you have a robot [18:41:52] wm-bot help [18:45:22] should be in https://wikitech.wikimedia.org/wiki/Help:Tool_Labs :) [18:45:33] I don't know if wm-bot actually has anything useful [18:47:51] o [18:48:22] it has some stuff useful for MW, but not labs [18:49:21] dbb, you should be able to ssh dbb@bastion.wmflabs.org [18:49:29] if you added your public key to your preferences in wikitech [18:50:16] "no matching mac found" [18:50:27] oh hm - have to review that.. thats why I am here [18:50:47] does my ssh client have to register its MC address ?? [18:50:50] MAC [18:54:46] dbb: what OS are you on? [18:56:47] I have a Mac laptop here.. linux would be connecting later [18:56:53] no windoze :-p [18:57:39] which version of OS X? older ssh clients have outdated MACs (which is part of the ssh encryption, not your MAC address) [18:57:50] Krenair: do you know if we have docs on this? [18:58:08] also what's your wikitech username? you should just connect to tools-login.wmflabs.org and not worry about bastion.wmflabs.org [18:58:11] oohh yes - macosx 10.7 [18:58:17] since the latter isn't of much use to toollabs [18:58:22] my login is darkblueb [18:58:26] yuvipanda, on macs? [18:58:47] on specifying local MAC overrides.... [18:58:54] 10.8 and 10.9 are when they tried to make everything connect to their store and look like an iPhone :-( [18:59:08] Krenair: can you also check if dbb is on tools already? [18:59:09] so I dont run 10.8 or 0.9 [18:59:19] darkblueb please [18:59:22] yuvipanda, he wasn't when I checked [18:59:56] I can theoretically add people but I only just got admin on tools the other day and I'm not really sure if I'm supposed to use it for anything :/ [19:00:04] wait you got admin on tools?! [19:00:06] wat [19:10:04] dbb: looks like you have to ask for toollabs access. As Krenair says, you do not have it [19:10:15] it'll be approved pretty quickly (I can probably find ways to do it from my phone) [19:48:33] 6Labs: no route to host trying to reach labs instances - https://phabricator.wikimedia.org/T113025#1717584 (10Smalyshev) 5Open>3Resolved a:3Smalyshev Yes. I don't know why the issue happened and what caused it, but restart fixed both and they are fine now. [20:00:23] 6Labs, 10MediaWiki-extensions-OpenStackManager: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#1717601 (10Smalyshev) If /host/ works, could we add docs about it to https://wikitech.wikimedia.org/wiki/Puppet_Hiera#In_Labs? Right now it mentions only `Hiera: 6Labs, 10MediaWiki-extensions-OpenStackManager: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#1717602 (10yuvipanda) Done! [20:09:57] 6Labs: Document labs SSH Fingerprints in sha256 format - https://phabricator.wikimedia.org/T112993#1717603 (10Krenair) Someone else noticed this issue (I think it might've been @Jdforrester-WMF?) If I can figure out how to generate these myself I'll add them in. [20:16:14] valhallasw`cloud: Can you help me with your gerrit patch uploader? It fails every time, I tried it with three different patch generators, but the upload failed... [20:19:03] 6Labs: Document labs SSH Fingerprints in sha256 format - https://phabricator.wikimedia.org/T112993#1717633 (10Krenair) See also T111867 [20:22:26] ok, found the error [21:12:59] 6Labs, 10MediaWiki-extensions-OpenStackManager: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#1717666 (10Krenair) 5Open>3Resolved a:3Krenair [21:13:08] 6Labs, 10MediaWiki-extensions-OpenStackManager: Setup per-instance configurability for labs in hiera - https://phabricator.wikimedia.org/T1356#23852 (10Krenair) a:5Krenair>3None [21:24:19] 6Labs, 6operations: 10.68.18.65 resolves to two different instances - https://phabricator.wikimedia.org/T115194#1717673 (10Krenair) 3NEW [21:26:02] 6Labs, 6operations: RDNS for 10.68.18.65 resolves to two different instances - https://phabricator.wikimedia.org/T115194#1717680 (10yuvipanda) [21:28:54] 6Labs, 10Education-Program-Dashboard, 10Labs-Infrastructure: Stable home for WMF dashboard server - https://phabricator.wikimedia.org/T107939#1717684 (10Base) [21:30:09] 6Labs, 10Education-Program-Dashboard: Stable home for WMF dashboard server - https://phabricator.wikimedia.org/T107939#1717693 (10yuvipanda) [22:19:42] Luke081515: uh. not entirely sure. are you creating the patch based on the most recent git version? [22:24:46] valhallasw`cloud: Was another error, I fixed this, patch is now online ;) [22:24:53] ok, great :-) [22:45:07] hm [22:46:57] I see Shell Request/Darkblueb - completed yes [22:47:35] now.. did I set a key ? sorry, I just havent used this in a year [22:57:53] dbb: sorry, I'm heading to bed, but you should be able to set it under https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack [22:58:11] dbb: and https://wikitech.wikimedia.org/wiki/Help:Tool_Labs should have info on the rest [23:36:31] thx looking