[00:01:08] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3689329 (10bd808) I do have access; I think all deployers do as well, although that may require using the mwdeploy shared agent from tin rather than dire... [00:04:50] (03PS161) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [00:09:15] (03CR) 10Ricordisamoa: [C: 04-2] "PS161 changes some strings into template strings" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [00:11:30] (03PS162) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [00:15:40] (03CR) 10Ricordisamoa: [C: 04-2] "PS162 adds a JSDoc typedef" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [00:27:04] (03PS163) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [00:31:59] (03CR) 10Ricordisamoa: [C: 04-2] "PS163 changes some vars into consts" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [06:12:16] 10Tools, 10InternetArchiveBot, 10Privacy: Tool "iabot" loads assets from google and hotjar - https://phabricator.wikimedia.org/T172605#3689550 (10Josve05a) [07:19:46] 10Data-Services, 10DBA, 10Tracking: Wikireplica service for tools and labs - issues and missing available views (tracking) - https://phabricator.wikimedia.org/T150767#3689572 (10jcrespo) ok to me, if someone retag those tickets. [08:05:31] 10Data-Services, 10DBA, 10InternetArchiveBot: User log table creation on tools.labsdb failing intermittantly for IABot interactive UI - https://phabricator.wikimedia.org/T178294#3689661 (10Marostegui) 05Open>03Resolved The graph looks stable and back to the normal pattern: https://grafana.wikimedia.org/d... [08:08:25] 10Data-Services, 10DBA: labsdb1005's mysql crashed - https://phabricator.wikimedia.org/T178272#3689667 (10Marostegui) 05Open>03Resolved a:03jcrespo The load seems back to previous levels: https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&panelId=19&fullscreen&orgId=1&var-server=la... [08:41:56] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3689706 (10MarcoAurelio) @bd808 Yep, that's them . Should they still nee... [08:56:59] 10Cloud-Services, 10Outreachy (Round-15): Proposal: Improvements for the Toolforge 'webservice' command - https://phabricator.wikimedia.org/T177603#3689736 (10Sowjanyavemuri) Hi @bd808, Could you please confirm my eligibility for Outreachy(Round-15) with the help of the answers/documents/links I've provided in... [12:24:05] !log git switching gerrit-test gerrit-new.wmflabs.org to ldpa using gerrit-test3 gerrit-test3.git.eqiad.wmflabs:1389 (mediawiki vagrant). [12:24:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [12:38:52] !log git starting gerrit-new.wmflabs.org up with ldap auth [12:38:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [12:46:48] bah, how does one force a puppet run on a instance again? [12:53:24] addshore puppet agent -tv [12:53:30] though you have to do sudo [12:53:33] hmm [12:53:34] in https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup#puppet [12:53:43] Use sudo /usr/local/sbin/puppet-run &. Don't use sudo puppet agent -t, because that is not what cron uses and leads to inconsistencies with e.g. umask and other factors affecting default values used at runtime. [12:53:52] but, meh, puppet-run doesnt seem to exist :D [12:54:08] oh [12:54:32] /usr/local/sbin/puppet-run [12:54:38] sudo puppet agent -t also fails with issues "Warning: Unable to fetch my node definition, but the agent run will continue:" [12:55:24] hmm [12:55:31] and the views in horizon just don't show me what roles are applied >.> [12:55:32] is it using a local puppet master? [12:55:48] integration-puppetmaster01.integration.eqiad.wmflabs [12:56:34] https://ask.puppet.com/question/6644/unable-to-fetch-my-node-definition-but-the-agent-run-will-continue-warning-403-forbidden/ [12:56:37] addshore ^^ [13:17:27] bah, stupid stuff [14:25:40] addshore: are you still frustrated by puppet? I can have a look [14:26:06] andrewbogott: I threw away the instance and recreated it and everything worked this time! [14:26:20] that is both bad news and good news :) [14:39:13] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3690781 (10jcrespo) p:05Normal>03High a:05Andrew>03jcrespo @Ladsgroup There is a duplicate database on s7 called amwikimedia. I ass... [14:43:34] Nikerabbit, Nemo_bis, I'm noticing that the 'ttmserver-salt01' instance is unhappy since there aren't any puppet classes for salt left anymore (we deprecated salt a while ago). Can that VM just be deleted at this point? [14:48:18] andrewbogott: salt master is probably unimportant, but let me take a backup of the code on the other instance just in case [14:48:28] 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 10User-bd808: Remove 'importers' (note the ending 's') group from wikitech - https://phabricator.wikimedia.org/T171682#3690816 (10bd808) 05Open>03Resolved a:03bd808 ``` (wikiadmin@silver) [labswiki]> select * from us... [14:50:17] ugh, why can't I log into horizon [14:59:24] Nikerabbit: ? [15:02:04] andrewbogott: I was planning to check the instance name but could not login (generic "invalid credentials" error) by using my wikitech details from my password manager. [15:02:38] the instance is ttmserver-salt01 [15:02:51] other instances in that project are: ttmserver-elasticsearch01, ttmserver-mediawiki01 [15:03:17] you have 2fa set up on wikitech? [15:03:27] andrewbogott: yeah [15:04:46] I can log in to wikitech, so chances of user error should be low [15:05:18] hm, strange [15:06:08] worked now after I tried again after logging in to wikitech [15:06:21] that's even weirder :/ [15:27:58] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3690925 (10Ladsgroup) hmm, when at first I tried to make the wiki, due to lack of documentation I used "fawiki" instead of "aawiki" because... [15:31:43] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3690929 (10jcrespo) Sorry, I phrased the "how" badly (I do not care much if there was a bug on the documentation/procedure). What I wanted... [15:34:25] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3690930 (10Ladsgroup) I was more into explaining that this was a one time thing and won't happen so we should not be worried about future c... [15:37:05] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) - https://phabricator.wikimedia.org/T176043#3690933 (10jcrespo) [15:39:03] 10Data-Services, 10cloud-services-team (Kanban), 10DBA, 10Patch-For-Review: Prepare and check storage layer for amwikimedia (including dropping s7 version of the wiki) - https://phabricator.wikimedia.org/T176043#3690935 (10Marostegui) [16:04:45] andrewbogott: I lied [16:04:49] It is still broken! :D [16:05:18] It feels like the first run works (this time I perhaps added the role I wanted before the first run so it got included) but puppet agent -t still fails [16:05:39] addshore: I'll look in a moment, what's the instance? [16:05:57] integration-slave-docker-c2-m4-d40-1004.eqiad.wmflabs [16:07:58] hm, I seem not to have sudo on that box [16:08:02] does it have a special policy? [16:08:14] * andrewbogott uses root key [16:08:41] addshore: so, I apologize if I'm stating the obvious here, I don't know how deep into this you are [16:08:57] When a new VM comes up it always uses the 'normal' puppetmaster, labs-puppetmaster.wikimedia.org [16:09:08] ack [16:09:17] but this project seems to have a universal puppetmaster override set [16:09:35] so that after that first puppet run, the puppetmaster is always set to something else (seemingly integration-puppetmaster01.integration.eqiad.wmflabs) [16:09:52] switching puppetmasters always breaks existing certificates (since it's trying to certify the new puppetmaster with the old certs) [16:10:05] so that will ALWAYS happen with any new VM in this project [16:10:20] The solution for this kind of thing is usually to rm -rf /var/lib/puppet/ssl [16:10:22] to get fresh certs [16:10:34] but then you may or may not have to explicitly sign the new request on the custom puppet master [16:11:11] interesting, None of this happened when I created the 1005 instance (a few weeks ago) as far as I remember [16:11:20] basically https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes#Switch_to_new_puppetmaster [16:11:27] but for whatever project you are in [16:11:43] addshore: it's possible to set overrides based on hostname prefixes, so it may depend on the instance name [16:12:09] look at the 'puppet' tab for your project in Horizon, both at 'Project Puppet' and 'Prefix Puppet' [16:13:10] So, "Project Puppet" doesn't actually have the class I am applying [16:13:36] wait, no, that's on a different page, when applying roles to an instance... [16:14:24] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3677875 (10chasemp) +1 [16:14:28] 10Cloud-VPS (Project-requests): Request creation of reading-lists VPS project - https://phabricator.wikimedia.org/T178110#3681085 (10chasemp) +1 [16:15:55] 10cloud-services-team (Kanban), 10DBA, 10Operations, 10Ops-Access-Requests: Access to raw database tables on labsdb* for wmcs-admin users - https://phabricator.wikimedia.org/T178128#3691050 (10chasemp) p:05Triage>03Normal a:03madhuvishy @madhuvishy is going to take a tour here and document from our e... [16:17:53] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3691058 (10bd808) +! [16:18:47] 10Cloud-VPS (Quota-requests): Request increased quota for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178134#3681841 (10bd808) +1 [16:19:09] 10Cloud-VPS (Quota-requests): Request static ip for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178134#3691066 (10bd808) [16:19:29] 10Cloud-VPS (Quota-requests): Request increased quota for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178332#3691071 (10bd808) +1 [16:20:21] andrewbogott: those docs really helped, let me update the integration docs with a link to that section! [16:20:42] 10Cloud-VPS (Project-requests): Request creation of reading-lists VPS project - https://phabricator.wikimedia.org/T178110#3691073 (10bd808) +1 [16:21:13] Hmm, perhaps I spoke to seen, I see get "Could not evaluate: Could not retrieve information from environment production source(s) file:/var/lib/puppet/client/ssl/certs/ca.pem" [16:26:32] addshore: I'm a little it concerned that you're plunked into the middle of an existing project that clearly already has certain internal practices and assumptions… yet are asking for help from people who have had nothing to do with that project historically. Is there no one around who actually knows what's going on there? [16:27:26] andrewbogott: well, I created integration-slave-docker-c2-m4-d40-1005 2 weeks and 4 days ago and this didn't happen! [16:27:42] addshore: get help from hashar :) [16:28:01] bd808: Yup, I will do tomorrow, no hashar today though! [16:28:47] we haven't messed with how Puppet works globally in the last 2-3 weeks so this is something in the project itself. The ci project is "delicate" as I recall [16:29:16] addshore: you haven't applied anything custom to that vm yet, right? Just trying to get a basic, initial puppet run? [16:30:04] andrewbogott: 1005 or 1004? [16:30:29] um… whichever one you're having trouble with [16:30:38] It had role::ci::slave::labs::docker applied [16:31:00] Although, Horizon doesn't seem to show that roles are actually applied. [16:31:31] I bet that role wants to borrow the puppet cert and is causing a chicken and egg problem with the first run [16:33:37] Right, I might just get hashar to create this instance tomorrow and update the docs! [16:33:50] Typically you'll need to do this one slow step at a time. Get a clean, error-free puppet run before applying any custom config, then applying roles or config one thing at a time. [16:34:16] Of course having a ready-made project-wide config interferes with that strategy, but… as much as you can [16:34:57] the big problem is switching puppetmasters. completely automating that is very hard [16:36:23] we have similar issues with some of the the Toolforge hosts [16:36:43] Final question, how can I remove a role from a node? (as it doesn't appear as applied in horizon) [16:37:18] addshore: you have to figure out *how* it is applied. It may be via hiera at a project, prefix, or instance level [16:38:56] bd808: instance level (I clicked apply role on the instance in horizon) [16:39:31] hmmm... and now you can't find it in the UI to uncheck? [16:40:06] nope [16:40:35] addshore: this is integration-slave-docker-c2-m4-d40-1005? [16:40:39] and what role? [16:41:10] wait, 1005 has it in Hiera Config! [16:41:35] Which means that I am actually trying to create 1004 in a different way! [16:42:26] no, to many tabs, integration-slave-docker-1003 has it in Hiera config.... [16:43:20] bd808: I was trying to remove role::ci::slave::labs::docker from integration-slave-docker-c2-m4-d40-1004, which I added using the horizon UI [16:44:40] hmmm... I don't see that as applied via horizon... [16:45:12] bd808: indeed I also don't, and that also happened with integration-slave-docker-c2-m4-d40-1005, yet it is applied on both. [16:45:31] I do see it on https://tools.wmflabs.org/openstack-browser/server/integration-slave-docker-c2-m4-d40-1004.integration.eqiad.wmflabs [16:45:54] oooh, thats a nice UI [16:49:53] addshore: there is some kind of bug. I just tried applying it again to see what would happen and it did not give any errors, but it still doesn't show as applied in the horizon ui [16:51:09] addshore and/or bd808 can you make me a bug? I've never seen this before. [16:55:01] bd808: I imagine you could write a better bug than me? :) [16:55:41] I suppose I can also write my own [16:55:57] 10cloud-services-team: create a wmcs alerting group in icinga - https://phabricator.wikimedia.org/T178405#3691161 (10chasemp) [16:56:12] 10Cloud-VPS (Project-requests): Request creation of reading-lists VPS project - https://phabricator.wikimedia.org/T178110#3691174 (10chasemp) a:03chasemp [16:56:40] 10Cloud-VPS (Quota-requests): Request increased quota for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178332#3691179 (10chasemp) p:05Triage>03Normal a:03chasemp +1 [16:56:53] 10Cloud-VPS (Quota-requests): Request static ip for cyberbot Cloud VPS project - https://phabricator.wikimedia.org/T178134#3691183 (10chasemp) p:05Triage>03Normal a:03chasemp +1 [16:57:06] 10Cloud-VPS (Quota-requests): Request increased quota for mwstake Cloud VPS project - https://phabricator.wikimedia.org/T178012#3691186 (10chasemp) p:05Triage>03Normal a:03chasemp [17:07:05] bd808: andrewbogott I'll make one now [17:08:59] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691257 (10Addshore) [17:09:03] ^^ [17:10:27] thanks addshore [17:11:17] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691276 (10bd808) Re-adding the role via Horizon does not change anything. No error message for the add, but still not showing as applied on... [17:13:58] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691296 (10bd808) Interestingly, the view at https://tools.wmflabs.org/openstack-browser/puppetclass/role::ci::slave::labs::docker shows the... [17:14:27] addshore: ^ I've got a hunch this is related to the length of the instance names [17:14:33] ooooooooh [17:15:18] Would be curious to see if "integration-docker-1006" works just fine [17:15:30] Well, I might as well give that a go now! [17:15:45] that also gets rid of the horrible "slave" naming ;) [17:16:53] Hah, well, i'm not sure if hashar wants to get rid of that, but let me try setting up integration-slave-docker-1006 [17:21:01] 10Wikibugs: Add wikibugs to - https://phabricator.wikimedia.org/T178410#3691325 (10Steinsplitter) [17:21:26] 10Wikibugs: Add wikibugs to #wikimedia-commons-sd - https://phabricator.wikimedia.org/T178410#3691338 (10Paladox) [17:21:51] paladox: Thanks. copy&past error. [17:23:26] i-s-d-c2-m4-d40-1005 :) [17:40:28] your welcome :) [17:41:05] bd808: I now remember having an issue editing the hiera config in horizon when I created the 1005 instance too, I wonder if that was also due to name length? [17:42:35] Indeed, it works for me with instances with shorter names, but not for those with the longer names [17:42:37] do we now the max lenght yet? [17:42:39] know [17:42:51] I guess integration-slave-docker-c2-m4-d40-1004.integration.eqiad.wmflab* [17:43:02] based on what bd808 put in https://phabricator.wikimedia.org/T178409#3691296 [17:43:38] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691443 (10Addshore) I now remember (and just tested) that when I first created the 1005 instance I couldn't edit the hiera config. That cou... [17:44:12] interesting, that is exactly 64 [17:44:20] seems plausible [17:44:22] heh, suspicious number [17:44:25] indeed [17:44:32] used online character count form thingie :) [17:44:43] http://www.charactercountonline.com lazy [17:44:53] bwhahaha :P [17:45:24] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691257 (10Dzahn) That's exactly 64 characters there. Seems suspicious and plausible that it is about the max length. [17:47:36] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3691257 (10Paladox) see https://bugs.launchpad.net/horizon/+bug/1276389 [17:48:46] addshore: paladox found the upstream bug :) [17:48:47] https://bugs.launchpad.net/horizon/+bug/1276389 [17:48:50] Ooh [17:49:04] well, or almost [17:49:06] fixed in "OpenStack Dashboard (Horizon) 2014.1 "icehouse"" [17:49:09] project name. but it is 64! [17:49:11] dosen't sound like it is fixed [17:49:24] "message": "Project name should not be greater than 64 characters.", "code": 400, " [17:49:35] that's not instance name.. but .. 64 even more likely now [17:49:48] that's another upstream bug for no checking on instance name? [17:50:52] Nikerabbit: did you figure out what to do about that salt master? [17:52:05] https://bugs.launchpad.net/horizon/+bug/1279590 [17:52:09] that's for instances ^^ [17:52:51] paladox: that seems a perfect match, thanks! [17:52:53] "There is an allowed maximum length for instance name, while current code doesn't check / restrict the instance name field when update an instance." [17:53:22] Hmmm, that still doesn't sound right thoughm [17:53:24] ? [17:53:38] 10wikitech.wikimedia.org: New e-mail-created wikitechwiki user "Per Magnus" can't set their password - https://phabricator.wikimedia.org/T178417#3691507 (10Jdforrester-WMF) [17:53:41] it seems it has been fixed [17:53:43] Unless update instance also means create instance [17:53:45] "Also changed the max_length value from 80 to 255 in Create Instance [17:53:45] to match the backend's restriction. [17:53:47] but possibly another bug some where? [17:54:02] so the backend says up to 255 is ok? [17:54:09] but the frontend messed it up after 64 ? [17:54:32] but why does he say "from 80" then :p [17:54:54] I spent a while trying to add regexp validation to the instance name field on Horizon and got nowhere [17:54:58] but it's definitely a thing we need [17:58:38] paladox, mutante, while you're here… is one of you active in 'wikistats'? Puppet is failing on two instances there, wikistats-cowgirl.wikistats.eqiad.wmflabs and wikistats-kraken.wikistats.eqiad.wmflabs [17:59:04] i think one of those are mutante prod instances [17:59:06] or act as prod [17:59:23] The last Puppet run was at Thu Oct 5 01:32:26 UTC 2017 (18266 minutes ago). [17:59:24] wow [17:59:44] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item profile::wikistats::wikistats_host in any Hiera data file and no default supplied at /etc/puppet/modules/profile/manifests/wikistats.pp:6 on node wikistats-kraken.wikistats.eqiad.wmflabs [17:59:44] Warning: Not using cache on failed catalog [17:59:44] Error: Could not retrieve catalog; skipping run [18:00:33] i can do the fix [18:00:37] simple hiera fix [18:00:37] profile::wikistats::wikistats_host' [18:01:00] hey, i just read this now. wait a sec [18:01:05] andrewbogott: yes, i am active there [18:01:21] thx [18:02:17] paladox: ok, please do :) thanks [18:02:25] ok your welcome :) [18:02:26] done [18:04:49] andrewbogott: currently cant ssh into wikistats-cowgirl yet, looking why. i used to have the root access [18:04:59] andrewbogott: but if you are on it, feel free to run puppet after paladox' fix [18:05:10] puppet is running on karaken [18:05:18] i mean wikistats-kraken [18:05:27] puppet fails with this [18:05:28] Error: /usr/bin/git pull --quiet returned 1 instead of one of [0] [18:05:28] Error: /Stage[main]/Wikistats/Git::Clone[operations/debs/wikistats]/Exec[git_pull_operations/debs/wikistats]/returns: change from notrun to 0 failed: /usr/bin/git pull --quiet returned 1 instead of one of [0] [18:05:29] now [18:05:31] mutante ^^ [18:06:16] paladox: ok thanks, i have issues logging in. it might be me. not sure yet [18:06:23] ok [18:06:45] Notice: /Stage[main]/Wikistats/Git::Clone[operations/debs/wikistats]/Exec[git_pull_operations/debs/wikistats]/returns: error: insufficient permission for adding an object to repository database .git/objects [18:06:49] ah [18:06:58] that's new to me [18:07:24] wikistats-cowgirl seems fixed [18:07:28] remembers applying that role and it working [18:07:33] andrewbogott: cool:) [18:07:35] -kraken is upset, as paladox notes [18:07:51] yep, seems due to permissions problems. [18:08:00] can you see failed logins from me by any chance? [18:08:02] though i am not sure where it is cloning too [18:08:05] will check [18:08:53] mutante: which one can't you log in to? [18:08:57] either [18:09:36] try cowgirl again? [18:09:38] oh. i might be trying to connect as root to the restricted bastion.. [18:10:09] i see Oct 17 18:05:36 wikistats-kraken sshd[22425]: Accepted publickey for root from 10.68.18.66 port 52264 ssh2: RSA SHA256: [18:11:30] mutante: I don't see any evidence that you're getting as far as wikistats-cowgirl. [18:11:49] but I have to step out for a bit, sorry [18:12:00] ok, thanks.dont worry about it. it must be on my side [18:12:04] trying to fix it [18:12:21] puppet passes [18:12:22] now [18:12:29] i fixed the permissions in /srv/wikistats [18:12:55] thank you! can't say i know why they were changed. puppet doesnt re-break them, right? [18:13:04] sudo chown -R wikistatsuser:wikistatsuser ./ in /srv/wikistats [18:13:25] i think it may be because we did something manual in there possibly as root [18:13:30] though i carn't remember [18:13:43] did puppet change anything about it after you ran it first time after doing the chown? [18:13:46] it sets with correct perms here https://github.com/wikimedia/puppet/blob/bf158c15ba1deec0683dd1c4388d2996140829cb/modules/wikistats/manifests/init.pp#L65 [18:13:49] nope [18:13:52] puppet passes :) [18:13:57] fine then :) [18:14:32] :) [18:19:38] 10wikitech.wikimedia.org: New e-mail-created wikitechwiki user "Per Magnus" can't set their password - https://phabricator.wikimedia.org/T178417#3691507 (10bd808) There is no LDAP record with cn="Per Magnus". What does "e-mail-created" mean? [18:20:28] 10wikitech.wikimedia.org: New e-mail-created wikitechwiki user "Per Magnus" can't set their password - https://phabricator.wikimedia.org/T178417#3691644 (10bd808) ``` 2017-10-16T21:48:50 User account Per Magnus (talk | contribs | block) was created by JForrester (talk | contribs | block) and password was sent by... [18:23:05] login issues fixed. pebcak :p [18:24:00] puppet run ok on both. installing package upgrades :) [18:25:37] bd808: andrewbogott I think I solved my puppet issue [18:25:41] https://www.irccloud.com/pastebin/wwAnZzud/ [18:26:06] that's ... ugly [18:27:04] right, all annoying issues solved, time to eat thai and sushi!!!! [18:27:34] so you clear the certs from the global puppetmaster and force a run (that's normal). What is the copy of the ca cert for? The docker role? [18:28:11] !log wikistats wikistats-cowgirl installing apache,systemd, misc upgrades [18:28:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [18:28:18] I have no idea :D I looked at .bash_history for root on 1003, but it doesnt work until I do that copy :) (the copy is in the bash history) [18:28:20] paladox: systemd itself got new version :p [18:28:27] heh :) [18:29:04] thcipriani made those instances, the bash history of 1001 and 1002 is a mess (I guess as he was figuring out what to do) [18:29:31] puppet now runs perfectly on integration-slave-docker-1006 :) *is a happy bunny* [18:30:18] addshore: great [18:30:28] * bd808 backs away slowly :) [18:32:31] bd808: andrewbogott one final question, how do I make classes appear in the project list for integration? Could I add role::ci::slave::labs::docker to that list to avoid going into "All" each time? [18:33:54] addshore: You would need to submit a puppet patch. The project tags are in code comments; if you grep for 'filtertags' you'll see how it works. [18:34:05] andrewbogott: thanks! [18:34:06] (or I can do it if you've never made a puppet patchbefore) [18:34:20] * addshore has made many! I'll get to it this evening! [18:36:38] 10Wikibugs, 10Structured-Data-Commons, 10Wikidata: Add wikibugs to #wikimedia-commons-sd - https://phabricator.wikimedia.org/T178410#3691698 (10Legoktm) [18:39:32] 10wikitech.wikimedia.org: New e-mail-created wikitechwiki user "Per Magnus" can't set their password - https://phabricator.wikimedia.org/T178417#3691507 (10Krenair) I'm assuming it's just where you create an account while already logged in, for another user, providing their email address. It will email them thei... [18:53:05] !log wikistats-cowgirl upgrading linux-meta-4.9 as last package to upgrade. [18:53:06] paladox: Unknown project "wikistats-cowgirl" [18:53:18] !log wikistats wikistats-cowgirl upgrading linux-meta-4.9 as last package to upgrade. [18:53:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [19:46:14] andrewbogott: sorry I got distracted by non-work life. If it is okay, give me a few days and I check what I should backup and turn/kill the instances. [19:57:37] Nikerabbit: how dare you have non-work life! ;) [19:59:42] bd808: yeah it's horrible isn't it [20:00:21] I think I remember having a non-work/volunteer life... [20:49:12] !log run puppet on tools-worker-1007.tools.eqiad.wmflabs to fix token issue (why is this getting a bad token and from where????) [20:49:13] chasemp: Unknown project "run" [20:50:49] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3692107 (10chasemp) [20:54:09] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3692118 (10chasemp) ~/git/wmf/labs/private grep -Ri faketoken * hieradata/labs/tools/common.yaml: token: faketoken hieradata/labs/tools/common.yaml: token: faketoken hieradata/labs/tools/common.y... [20:54:18] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3692119 (10chasemp) 05Resolved>03Open [20:59:57] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3692145 (10chasemp) I can see the flaps: ```tools-puppetmaster-01:/var# grep -Ri faketoken * lib/puppet/reports/tools-worker-1007.tools.eqiad.wmflabs/201710172048.yaml: message: "\n--- /etc/kube... [21:16:09] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3675748 (10Krenair) I'm pretty sure this is a wider issue as I've seen it before on deployment-prep with other hiera data [21:29:08] 10Toolforge: k8s nodes sometimes getting bad token value from hiera - https://phabricator.wikimedia.org/T177944#3692268 (10Andrew) Current theory is that this happens when the labs-private repo is in the process of being rebased. [22:09:18] 10Horizon: Applied puppet classes not appearing in horizon for integration-slave-docker-c2-m4-d40-1005.integration.eqiad.wmflabs - https://phabricator.wikimedia.org/T178409#3692368 (10Addshore) So, I have removed both integration instances that were causing me issues and replaced them with instances with shorter... [22:57:24] 10Toolforge, 10Tools, 10cloud-services-team (FY2017-18), 10Community-Liaisons (Oct-Dec 2017), 10Goal: Promote Toolforge Tools and their maintainers within Wikimedia communities - https://phabricator.wikimedia.org/T176677#3692448 (10Quiddity) p:05Normal>03High [23:06:22] (03Draft1) 10Paladox: [labs/private] - 10https://gerrit.wikimedia.org/r/384902 (https://phabricator.wikimedia.org/T178385) [23:06:24] (03PS2) 10Paladox: Gerrit: Replace certificates with tokens for its-phabricator [labs/private] - 10https://gerrit.wikimedia.org/r/384902 (https://phabricator.wikimedia.org/T178385) [23:15:43] (03PS164) 10Ricordisamoa: Initial commit [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 [23:17:09] are there DB clusters on deployment-prep? or all the DBs are together? [23:22:01] (03CR) 10Ricordisamoa: [C: 04-2] "PS164 changes some JSDoc comments" [labs/tools/wikidata-slicer] - 10https://gerrit.wikimedia.org/r/241296 (owner: 10Ricordisamoa) [23:43:06] tgr: Beta cluster has its own db servers [23:43:15] for the beta wikis [23:43:47] I *think* there are 2 vms that do master/replica there