[00:02:11] (03CR) 10Legoktm: [C: 032] Redis2irc: more detailed logging [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190731 (owner: 10Merlijn van Deen) [00:02:23] (03Merged) 10jenkins-bot: Redis2irc: more detailed logging [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190731 (owner: 10Merlijn van Deen) [00:02:34] (03CR) 10Legoktm: [C: 032] wikibugs: add --ask debug parameter [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190732 (owner: 10Merlijn van Deen) [00:02:45] (03Merged) 10jenkins-bot: wikibugs: add --ask debug parameter [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190732 (owner: 10Merlijn van Deen) [00:04:19] (03CR) 10Legoktm: [C: 032] Better IRC project list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190733 (https://phabricator.wikimedia.org/T1231) (owner: 10Merlijn van Deen) [00:04:30] (03Merged) 10jenkins-bot: Better IRC project list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190733 (https://phabricator.wikimedia.org/T1231) (owner: 10Merlijn van Deen) [00:04:34] (03CR) 10Legoktm: [C: 032] Make sure URL is always present [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190741 (owner: 10Merlijn van Deen) [00:04:46] (03Merged) 10jenkins-bot: Make sure URL is always present [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190741 (owner: 10Merlijn van Deen) [00:05:19] !log tools.wikibugs legoktm: Deployed d5922a4d10169ec8870e55de1b74ea9e39dc8c5c Make sure URL is always present wb2-irc [00:05:25] Logged the message, Master [00:05:30] !log tools.wikibugs legoktm: Deployed d5922a4d10169ec8870e55de1b74ea9e39dc8c5c Make sure URL is always present wb2-phab [00:05:32] Logged the message, Master [00:44:48] 2Wikibugs, and 1 other: Find a way to report tasks with only an archived project - https://phabricator.wikimedia.org/T1231#1040614 (10Legoktm) 5Open>3Resolved a:3Legoktm [00:45:07] 2Wikibugs, and 1 other: Find a way to report tasks with only an archived project - https://phabricator.wikimedia.org/T1231#21175 (10Legoktm) a:5Legoktm>3valhallasw [00:45:23] 2Wikibugs, and 1 other: Add sane project ordering - https://phabricator.wikimedia.org/T86448#1040617 (10Legoktm) 5Open>3Resolved a:3valhallasw [00:45:49] 2Wikibugs, and 1 other: Always show matched project - https://phabricator.wikimedia.org/T86759#1040620 (10Legoktm) 5Open>3Resolved a:3valhallasw [00:46:02] valhallasw`cloud: we should blacklist "Patch-to-Review" from the "1 other" ^ [00:46:27] 2Wikibugs, and 1 other: Limit number of projects - https://phabricator.wikimedia.org/T88011#1040623 (10Legoktm) 5Open>3Resolved a:3valhallasw [00:47:37] (03PS2) 10Legoktm: Add online project scrape test [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189051 (owner: 10Merlijn van Deen) [00:48:12] (03CR) 10Legoktm: [C: 032] Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen) [00:48:24] (03Merged) 10jenkins-bot: Provide more CTCP responses [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189257 (https://phabricator.wikimedia.org/T88070) (owner: 10Merlijn van Deen) [00:48:57] (03PS2) 10Legoktm: Extend logging [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189322 (owner: 10Merlijn van Deen) [00:49:08] (03CR) 10Legoktm: [C: 032] Add online project scrape test [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189051 (owner: 10Merlijn van Deen) [00:49:20] (03Merged) 10jenkins-bot: Add online project scrape test [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189051 (owner: 10Merlijn van Deen) [00:49:34] (03CR) 10Legoktm: [C: 032] Extend logging [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189322 (owner: 10Merlijn van Deen) [00:49:42] (03CR) 10Legoktm: [C: 032] Make sure logging uses utf-8, or else entries might disappear [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189323 (owner: 10Merlijn van Deen) [00:50:26] (03Merged) 10jenkins-bot: Extend logging [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189322 (owner: 10Merlijn van Deen) [00:51:47] (03PS1) 10Legoktm: Send Wikibase stuff to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190754 [00:51:52] (03PS2) 10Legoktm: Make sure logging uses utf-8, or else entries might disappear [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189323 (owner: 10Merlijn van Deen) [00:52:06] (03CR) 10Legoktm: [C: 032] Make sure logging uses utf-8, or else entries might disappear [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189323 (owner: 10Merlijn van Deen) [00:52:27] (03Merged) 10jenkins-bot: Make sure logging uses utf-8, or else entries might disappear [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/189323 (owner: 10Merlijn van Deen) [00:52:49] (03CR) 10Legoktm: [C: 032] Send Wikibase stuff to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190754 (owner: 10Legoktm) [00:53:19] (03Merged) 10jenkins-bot: Send Wikibase stuff to #wikidata [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190754 (owner: 10Legoktm) [00:53:51] !log tools.wikibugs legoktm: Deployed 4c82585a9c01bceeb91acabbc5b481ea5928327d Merge "Send Wikibase stuff to #wikidata" wb2-phab [00:53:54] Logged the message, Master [00:54:01] !log tools.wikibugs legoktm: Deployed 4c82585a9c01bceeb91acabbc5b481ea5928327d Merge "Send Wikibase stuff to #wikidata" wb2-irc [00:54:03] Logged the message, Master [04:33:04] (03PS1) 10Greg Grossmeier: Add Blocked-on-RelEng to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190769 [04:33:49] Yuvi|Vacation: see? ^^ I like it that much :) [04:34:02] awww :) [04:34:18] * greg-g goes to bed for real now [04:34:35] (03CR) 10Yuvipanda: [C: 032] Add Blocked-on-RelEng to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190769 (owner: 10Greg Grossmeier) [04:34:41] legoktm: ^ does it still autodeploy? [04:34:47] (03Merged) 10jenkins-bot: Add Blocked-on-RelEng to -releng [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190769 (owner: 10Greg Grossmeier) [04:35:02] !log tools.wikibugs Updated channels.yaml to: eb4a51a4628a6b26d4e798b3e55d8749231bb72c Add Blocked-on-RelEng to -releng [04:35:08] Logged the message, Master [05:30:26] Does anyone know how to use the API to download all contributions from an ip address within a date range? [06:44:49] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040833 (10Nikerabbit) 3NEW [06:48:58] 2Wikibugs: Color floods into the main text and url - https://phabricator.wikimedia.org/T89633#1040846 (10Nikerabbit) 3NEW [07:19:49] 2Wikibugs: Color floods into the main text and url - https://phabricator.wikimedia.org/T89633#1040873 (10valhallasw) That's clearly not as intended. Which client are you using? [07:20:44] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040876 (10valhallasw) The colors are supposed to match the Phabricator colors, but that idea is of course worthless if the text is then unreadable. I'll test with a black background. [07:22:18] 2Wikibugs: Report 'and X more' in a different way - https://phabricator.wikimedia.org/T89634#1040883 (10valhallasw) 3NEW [07:26:34] 2Wikibugs: Color floods into the main text and url - https://phabricator.wikimedia.org/T89633#1040890 (10Nikerabbit) I don't think it has anything to do with my client, but I'm using irssi. [07:42:44] 2Wikibugs: Color floods into the main text and url - https://phabricator.wikimedia.org/T89633#1040894 (10valhallasw) irssi : F41539 irccloud: F41541 [09:24:55] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040973 (10Nemo_bis) [09:26:57] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040977 (10fgiunchedi) FWIW this is what I'm seeing (disregard the fact that wikibugs appears for me as NOTICE) {F41568} [09:36:53] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040986 (10Nikerabbit) @fgiunchedi The color bleeding thing is reported separately as T89633. [09:51:41] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1040997 (10valhallasw) I'll look at this tonight; To do list: * map 'blue' project color to 'teal' instead of dark blue (issue 1/2) * switch to underline instead of bold for highlighting projects (issue 3) I'm planning t... [10:09:26] legoktm: I'm thinking we should maybe show 'hidden' projects but only if they fit [10:09:42] legoktm: i.e. ordering 'matched tags', 'projects/teams', 'other projects' [10:17:03] 2Wikimedia-Labs-General: Project-wide Puppet classes and variables - https://phabricator.wikimedia.org/T66980#1041028 (10Joe) 5Open>3stalled p:5Normal>3Low [10:20:01] 2Wikimedia-Labs-General: Project-wide Puppet classes and variables - https://phabricator.wikimedia.org/T66980#1041032 (10yuvipanda) @joe I would also like to have hiera_include('classes') at some point. Should investigate when I'm back. [10:42:27] 2Tool-Labs, 13Labs: WIWOSM not working in Wikipedias - https://phabricator.wikimedia.org/T87038#1041081 (10Aklapper) Still not sure who of the #Labs team could look into this... [11:03:56] 13Labs, and 1 other: New Labs project requests (Tracking) - https://phabricator.wikimedia.org/T76375#1041123 (10Aklapper) [11:03:57] 13Labs: Requesting for developer access for native iOS app (in development) - https://phabricator.wikimedia.org/T85450#1041121 (10Aklapper) 5Resolved>3Invalid (Changing task status as no change was needed here, hence nothing fixed) [11:08:07] 2Wikibugs: Report 'and X more' in a different way - https://phabricator.wikimedia.org/T89634#1041130 (10TTO) Does it even need to show "patch for review"? Seems like a waste of space. IMHO "tag" projects, like easy, patch for review, design etc (those with the tag icon here on Phabricator) don't need to be rel... [11:11:01] 2Tool-Labs-tools-Quentinv57's-tools, and 1 other: SUL info tool fails with unicode usernames - https://phabricator.wikimedia.org/T67144#1041134 (10Aklapper) >>! In T67144#953607, @Tgr wrote: > Can you push the tool code to Github @tgr: Who is "you" in that sentence? [11:36:59] PROBLEM - Puppet staleness on tools-exec-15 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0] [11:41:29] (03PS1) 10KartikMistry: CX: Add dummy Yandex API key for testing on Beta Cluster [labs/private] - 10https://gerrit.wikimedia.org/r/190798 [11:46:19] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] CX: Add dummy Yandex API key for testing on Beta Cluster [labs/private] - 10https://gerrit.wikimedia.org/r/190798 (owner: 10KartikMistry) [13:14:02] 2Wikibugs: make sure to match all projects against all regexes - https://phabricator.wikimedia.org/T89644#1041281 (10valhallasw) 3NEW [13:16:45] 2Wikibugs: wikibugs says "…and 1 other [project]" which is stupid, it should just say the project - https://phabricator.wikimedia.org/T89645#1041292 (10matmarex) 3NEW [13:17:37] 2Wikibugs: wikibugs says "…and 1 other [project]" which is stupid, it should just say the project - https://phabricator.wikimedia.org/T89645#1041299 (10valhallasw) [13:17:38] 2Wikibugs: Report 'and X more' in a different way - https://phabricator.wikimedia.org/T89634#1041300 (10valhallasw) [13:32:51] !log tools.lolrrit-wm bot dead, no reply to commands [13:32:56] Logged the message, Master [13:38:03] 2Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1041358 (10faidon) The current status quo is unusable for me right now, because of both of those issues (I'm using irssi). Problem is, the bot was so useful in the first place that I've grown accustomed to it, almost depen... [13:40:26] 2Tool-Labs, 2Wikidata: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#1041363 (10Aklapper) So how to proceed here (asking as this has Priority "Unbreak now", if that urgency is realistic)? [13:42:02] grrrit-wm: ? [13:43:43] !log tools.lolrrit-wm restarted [13:43:45] Logged the message, Master [13:45:22] 3Spam, Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1041366 (10valhallasw) fdsafdsadfsa [13:50:11] 3Tool-Labs, Wikidata: Lost connection to MariaDB server during query - https://phabricator.wikimedia.org/T76699#1041370 (10coren) p:5Unbreak!>3High It's a serious pain point for some tool mainteners, but can be fixed tool-side also. I'm lowering to "high" because we realistically want to do some tweaking se... [13:51:21] !log tools.wikibugs reverted locally (git revert ) and restarted as it was breaking people's workflows [13:51:26] Logged the message, Master [14:19:43] 3Labs, Labs-Vagrant, Tool-Labs, Wikimedia-Labs-wikistats, Wikimedia-Labs-wikitech-interface, Wikimedia-Labs-General, Beta-Cluster, Wikimedia-Labs-Other, Project-Creators, Wikimedia-Labs-Infrastructure, Tool-Labs-tools-Article-request, Wikimedia-Labs-extdist, MediaW... - https://phabricator.wikimedia.org/T89270#1041474 [14:22:00] i think there's a stale socket preventing my webservice from starting. from tool's error.log: [14:22:03] 2015-02-16 14:16:16: (mod_fastcgi.c.978) bind failed for: unix:/tmp/cgstat-fcgi.sock-0 Address already in use [14:22:03] 3MediaWiki-extensions-OpenStackManager: Special:NovaPuppetGroup "All projects" section is confusing - https://phabricator.wikimedia.org/T60520#1041481 (10Aklapper) p:5Triage>3Volunteer? [14:22:06] 2015-02-16 14:16:16: (mod_fastcgi.c.1399) [ERROR]: spawning fcgi failed. [14:22:09] 2015-02-16 14:16:16: (server.c.938) Configuration of plugins failed. Going down. [14:22:13] 3MediaWiki-extensions-OpenStackManager: Non-Admin users can't see anything in manage addresses interface - https://phabricator.wikimedia.org/T57897#1041483 (10Aklapper) p:5Triage>3Volunteer? [14:22:22] 3MediaWiki-extensions-OpenStackManager: OpenStackManager: Nova resource pages are endlessly prepended with line break whitespace on each update by Labslogbot - https://phabricator.wikimedia.org/T58316#1041485 (10Aklapper) p:5Triage>3Volunteer? [14:22:23] tools.cgstat@tools-webgrid-03:~$ ls -lh /tmp/cgstat-fcgi.sock-0 [14:22:23] srwxr-xr-x 1 tools.catgraph tools.catgraph 0 Feb 11 19:30 /tmp/cgstat-fcgi.sock-0 [14:22:37] 3MediaWiki-extensions-OpenStackManager: OpenStackManager: Instance actions via Special:NovaInstance should be protected by a nonce - https://phabricator.wikimedia.org/T58275#1041487 (10Aklapper) p:5Triage>3Volunteer? [14:22:50] can a labs op please 'rm /tmp/cgstat-fcgi.sock-0' on tools-webgrid-03? [14:22:58] 3MediaWiki-extensions-OpenStackManager: Special:NovaProject&action=addmember should treat underscores in usernames as spaces - https://phabricator.wikimedia.org/T62586#1041489 (10Aklapper) p:5Triage>3Volunteer? [14:23:10] 3MediaWiki-extensions-OpenStackManager: Better error message needed when adding a member to a project he's already a member of - https://phabricator.wikimedia.org/T62620#1041491 (10Aklapper) p:5Triage>3Volunteer? [14:23:16] 3MediaWiki-extensions-OpenStackManager: Removing a service group leaves sudo policies behind - https://phabricator.wikimedia.org/T63186#1041493 (10Aklapper) p:5Triage>3Low [14:24:49] JohannesK_WMDE: That normally won't cause an Address already in use unless there is an actual process listening on that socket - are you sure there isn't a stray daemon running? [14:24:50] 3MediaWiki-extensions-OpenStackManager: Special:NovaPuppetGroup should list number of instances having a class applied - https://phabricator.wikimedia.org/T65300#1041495 (10Aklapper) p:5Triage>3Volunteer? [14:24:51] 3MediaWiki-extensions-OpenStackManager: Special:NovaAddress should list instance IP - https://phabricator.wikimedia.org/T60444#1041497 (10Aklapper) p:5Triage>3Low [14:24:51] 3MediaWiki-extensions-OpenStackManager: Some usernames are all lowercase, leading to inconsistent behavior with rights - https://phabricator.wikimedia.org/T52515#1041499 (10Aklapper) p:5Triage>3Volunteer? [14:24:52] Coren: i think not. "webservice status" says "not running", web page displace corresponding error, also this: [14:24:52] tools.cgstat@tools-webgrid-03:~$ ps aux|grep lighttpd.*cgstat [14:24:52] 52450 25010 0.0 0.0 33340 944 pts/1 S+ 14:24 0:00 grep lighttpd.*cgstat [14:24:52] 3MediaWiki-extensions-OpenStackManager: Improve OpenStackManager's Echo-based notification emails - https://phabricator.wikimedia.org/T58202#1041502 (10Aklapper) p:5Triage>3Low [14:25:02] Coren: so i don't think it's running... [14:25:08] Hm, how odd. Allright, lemme kick the socket. [14:25:09] hey, I want to install labs vagrant on my own VM [14:25:14] so I've checked out the operations/puppet repo [14:25:54] trying to just execute an init.pp with one line: include ::role::labs::vagrant [14:25:57] 3MediaWiki-extensions-OpenStackManager: Fatal error: Call to undefined method GoogleLoginAuth::connect() in nova/OpenStackNovaLdapConnection.php on line 21 - https://phabricator.wikimedia.org/T70718#1041504 (10Aklapper) p:5Triage>3Low [14:25:57] but it can't find it [14:25:59] JohannesK_WMDE: {{done}} [14:26:08] 3MediaWiki-extensions-OpenStackManager: Sanity check submitted SSH keys on (Tool) Labs - https://phabricator.wikimedia.org/T77902#1041507 (10Aklapper) p:5Triage>3Volunteer? [14:26:15] 3Wikimedia-Labs-General, MediaWiki-extensions-OpenStackManager: adding/removing project members updates project page, changing projectadmins does not - https://phabricator.wikimedia.org/T73164#1041508 (10Aklapper) p:5Triage>3Low [14:26:41] werdna: You probably just want to include the class from the instance config on wikitech [14:26:53] * werdna blankstare [14:27:03] I'm not using a labs VM [14:27:18] werdna: Oh, "own VM" sorry. [14:27:20] Coren: working now, thanks [14:27:41] it's my own Linode :) [14:28:34] well here's some progress [14:28:34] root@li273-119:~/puppet# puppet apply init.pp --modulepath=`pwd`/modules [14:28:44] Error: Could not find dependency Package[git-core] for Exec[git_clone_vagrant] at /root/puppet/modules/git/manifests/clone.pp:147 [14:29:49] werdna: I... honestly don't know then. I'm pretty sure that attempting to apply a class from the operations puppet manifest outside its normal context is doomed to failure though - not only are there odd dependencies but it requires a facter with custom facts and so on. [14:29:59] nod [14:30:11] so how does labs actually set up that environment? [14:30:26] I mean, I have a clean VM so I'm happy to do it the exact same way that labs does [14:31:33] werdna: Well, it sets it up in the context of labs with all the bells and whistles and presumptions but, iirc, deploying vagrant on a $random_vm is a fairly simple process - but I've never done it so I couldn't tell you offhand. [14:31:44] nod [14:31:48] Lemme see if I can dig it up for you. [14:31:52] I can just read labs_vagrant.pp though [14:32:28] werdna: That's almost certainly going to refer to piles of stuff from the manifest though, like base:: [14:32:59] https://gist.github.com/8e2847c3295f5d810d86 [14:33:01] https://www.digitalocean.com/community/tutorials/how-to-install-vagrant-on-a-vps-running-ubuntu-12-04 seems a bit old, but is probably useful [14:34:13] https://www.mediawiki.org/wiki/MediaWiki-Vagrant#Quick_start seems more useful. [14:37:39] everything worked with this: https://gist.github.com/werdnum/d9accb2d4a0815171052 [14:38:18] a bit quick and dirty but who cares :p [16:17:49] werdna: That seems like a mostly sane set of steps to run labs-vagrant on a random host. It might be neat to figure how to add your labs-vagrant.pp manifest to the mw-vagrant repo. You could put it in the support directory if nothing else. [16:18:19] nod [16:18:38] bd808: I wrote a GIST with idiotproof instructions [16:18:42] so could also stick that in the repo [16:18:56] bd808: (background: my laptop is too slow to run a virtual machine these days) [16:19:00] Yeah. I was just reading it [16:19:12] boo for too slow laptops [16:19:24] I'll buy a new one in a month or two [16:19:31] just need to be vaguely productive for the time being :p [16:19:32] I really really need to find a place to play with lxc as the container [16:19:38] lxc? [16:19:51] hey, any idea where HHVM sends its error messages? [16:19:55] /vagrant/logs has no hhvm directory :( [16:20:36] lxc a linux only light weight vm-like container system. https://linuxcontainers.org/ [16:20:50] which should make things much nicer for linux users if it works [16:21:18] and would let labs-vagrant change to actually using vagrant and our custom plugin in theory [16:21:40] ahh nice [16:21:41] docker started as a set of scripts to configure lxc [16:21:43] I like the sound of that [16:22:16] It has been on my list of things I should do when "volunteering" for quite a while [16:24:49] :p [16:24:52] "volunteering" [16:24:58] werdna: hmmm... I'm not seeing hhvm errors on my mw-vagrant vm either. Maybe I broke that at some point [16:24:59] it's great when WMF can double dip like that :p [16:25:14] there's supposed to be a server.ini in /etc/hhvm [16:25:16] but it's not there on vagrant [16:25:44] Try /etc/hhvm/fcgi.ini [16:25:48] but I don't understand what's *supposed* to send the HHVM errors through [16:26:19] hhvm pushes them to syslog I think (or at least it did at some point) [16:26:34] then rsyslog rules should write them to disk [16:26:47] let me look at our puppet rules [16:26:54] nod [16:28:36] This should write them to /vagrant/logs/hhvm/error.log -- https://github.com/wikimedia/mediawiki-vagrant/blob/master/puppet/modules/hhvm/templates/rsyslog.conf.erb [16:28:51] assuming they get into syslog in the first place [16:29:37] it could be that I fucked up permissions on /vagrant/logs [16:30:11] I was trying to sync my own vagrant directory and fucked it up so I restored from an archive which didn't store permissions, so I just extracted the archive and chown vagrant:wikidev'd it [16:30:27] "hhvm.log.use_syslog = true" is in /etc/hhvm/fcgi.ini so ... it "should" work [16:31:09] Try creating /var/log/hhvm and making it 0775 root:syslog [16:31:37] err. I meant /vagrant/logs/hhvm [16:33:45] chmod'd to 777 to see [16:35:39] hmmm, still nothing [16:35:42] restarted rsyslog, hhvm [16:35:56] and chmod'd /vagrant/logs/hhvm to 777, chown'd to root:syslog [16:51:10] werdna: hmmm... are you doing things that you know should be triggering hhvm errors? Not that that is too hard with our current codebase, but the hhvm.log that should be created wold only have the things that would have gone to the apache2/error.log under a php5 runtime. [16:51:26] well I'm getting a runtime error [16:51:52] i.e. I'm going to a page that should work but instead displays "Function name must be a string" [16:51:57] that seems like something that should end up in the hhvm.log all right [16:52:32] yeah [16:52:38] thought about adding the ELK role [16:52:43] maybe it would show up there [16:58:32] I notice that there is no /vagrant/logs/hhvm on a more conventionally set up labs-vagrant instance too [16:59:18] hmmm... that's true for me too on a recent labs-vagrant instance. [17:00:34] /etc/rsyslog.d/40-hhvm.conf [17:00:39] :programname, startswith, "hhvm" /var/log/hhvm/error.log [17:00:39] & ~ [17:00:41] 3Labs, Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1041755 (10Lydia_Pintscher) [17:00:56] It probably has something to do with permissions there too. When we run in a VM /vagrant/logs is mounted from the host computer in a way that would hide permissions problems [17:01:40] nod [17:01:45] could it be apparmor or so? [17:01:49] oh! I overrode that in hiera for labs-vagrant then I bet [17:01:58] the /var/log/hhvm/error.log location [17:02:14] I do note, however, that I have a functioning erorr log at /var/log/hhvm/error.log – it just doesn't include the script errors [17:02:16] yeah I have /var/log/hhvm/error.log on labs-vagrant [17:02:24] just a million times this: Failed to initialize central HHBC repository:\n Failed to open /var/www/.hhvm.hhbc: 14 - unable to open database file\n [17:05:33] so maybe the problem is not the logging but that the incorrect things are being logged? [17:05:34] that message should be non-fatal. it's hhvm running as the www-data user trying to open the secondary byte code cache file in ~/.hhvm.hhbc [17:06:15] you could shut it up by touching the file and chown it to www-data [17:06:20] nod, but if errors are going there then why are no code errors going there? [17:06:37] 3Labs, Wikimedia-Labs-wikitech-interface: Separate wikitech machine from labs nova controller - https://phabricator.wikimedia.org/T85297#1041778 (10Andrew) 5Open>3Resolved a:3Andrew This is done -- wikitech has its own server (Silver) and seems happy. [17:07:45] my guess is still permissions. For me that file is 0644 www-data:www-data which means syslog can't write there [17:08:16] I wonder if our /etc/hhvm/php.ini (cli config) writes the file directly [17:09:12] hmmm.. nope. "hhvm.log.use_syslog = true" in there too [17:09:35] 3Labs, operations: MySQL on wikitech keeps dying - https://phabricator.wikimedia.org/T88256#1041786 (10Andrew) Thanks to the wikitech move to silver, we've now gone several days without this happening! Still, virt1000 is ailing a bit. I propose throwing more memory at the problem, as per T89266 [17:09:59] 3hardware-requests, Labs, ops-eqiad, operations: virt1000 memory upgrade - https://phabricator.wikimedia.org/T89266#1031811 (10Andrew) [17:10:00] 3Labs, operations: MySQL on wikitech keeps dying - https://phabricator.wikimedia.org/T88256#1041789 (10Andrew) [17:11:14] -rw-r--r-- 1 www-data www-data 1 Feb 16 17:04 error.log [17:11:16] same here [17:11:18] 3Labs, operations: OOM on virt1000 - https://phabricator.wikimedia.org/T88256#1041793 (10Andrew) p:5Unbreak!>3High [17:12:01] what port am I supposed to go to to see the ELK error log? [17:12:03] I thought 8080 [17:12:05] but apparently not [17:17:30] 3Labs: Wikitech should use shared misc-host mysql - https://phabricator.wikimedia.org/T88311#1041809 (10Andrew) 5Open>3declined a:3Andrew I've thought better of this. there are advantages to keeping wikitech with its own db. [17:17:31] 3Labs: Move wikitech web interface to a dedicated server - https://phabricator.wikimedia.org/T88300#1041812 (10Andrew) [17:20:25] 3Labs, Labs-Vagrant, Tool-Labs, Wikimedia-Labs-wikistats, Wikimedia-Labs-wikitech-interface, Wikimedia-Labs-General, Beta-Cluster, Wikimedia-Labs-Other, Project-Creators, Wikimedia-Labs-Infrastructure, Tool-Labs-tools-Article-request, Wikimedia-Labs-extdist, MediaW... - https://phabricator.wikimedia.org/T89270#1041816 [17:20:47] 3hardware-requests, Labs, ops-eqiad, operations: virt1000 memory upgrade - https://phabricator.wikimedia.org/T89266#1041819 (10Andrew) Thursday is fine. Chris, if you can pick a window that's not already blocked on the deployment calendar, I'll notify greg and the labs list. The good news is, this outage won't... [17:24:14] 3Labs, Wikidata: wdq.wmflabs.org does not update (data week old) - https://phabricator.wikimedia.org/T89583#1039840 (10Lydia_Pintscher) [17:59:33] 3Tool-Labs-tools-Quentinv57's-tools: SUL info tool fails with unicode usernames - https://phabricator.wikimedia.org/T67144#1041919 (10Tgr) Whoever owns/operates the tool I guess. [18:03:02] 3Wikibugs: Color floods into the main text and url in irssi - https://phabricator.wikimedia.org/T89633#1041928 (10Legoktm) [18:03:15] 3Wikibugs: Color floods into the main text and url in irssi - https://phabricator.wikimedia.org/T89633#1040846 (10Legoktm) It bleeds for me in irssi, but not in quassel. [18:19:07] PROBLEM - Free space - all mounts on tools-webgrid-02 is CRITICAL: CRITICAL: tools.tools-webgrid-02.diskspace.root.byte_percentfree.value (<100.00%) [18:29:03] Salut Coren... Es-tu l [18:29:06] la? [18:29:24] AndyRussG: Partiellement [18:29:40] Heh, c'est mieux que rien :) [18:30:13] Coren: I'm helping anow, an intern with the Outreach Program for Women, try to log in to labs :) some ssh trouble [18:30:27] Need moar detailz! [18:31:21] She's uploaded her ssh key, and is able to log in via Putty, but not via a normal terminal (on Ubuntu) [18:31:35] Coren: http://pastebin.com/eGWGtMAi [18:32:04] She's working with the WMF and the Wiki Education Foundation on stuff for the Education program [18:32:21] I'll bet you a token that her username is different than that of the Labs account. :-) [18:32:52] Ah, it is, but she did take that into account. Nevermind. :-) [18:33:00] Hmm.... [18:33:04] (Single most common mistake) [18:33:37] Ah, but the log shows that her private key isn't in one of the usual places and she didn't otherwise specify it on the command line. [18:33:56] Right (I don't know if I have tokens to bet, though... have some icicles, though) [18:34:10] Hmmm [18:34:25] Her key should be named ~/.ssh/id_[dr]sa [18:34:35] (depending on whether it's a DSA or RSA key) [18:34:40] Coren: hi, thanks for the effort .. :) [18:34:49] Or it needs to be specified with '-i /path/to/the/key' [18:34:55] my key is named like this.. [18:35:10] and is in the standard place. as far as I can see - [18:36:14] Hm, indeed. And that's the same key that has been shown to work with putty? [18:36:38] anow: what do you get when you say "ls ~/.ssh" [18:36:38] Hm. "debug1: Server accepts key: pkalg ssh-rsa blen 279" [18:36:39] ? [18:37:19] config config~ id_rsa id_rsa.pub known_hosts [18:37:35] OK that looks good... [18:37:47] anow: could you make another pastebin with your .ssh/config file, please? [18:37:56] yes, one sec [18:38:02] thanks! [18:38:04] AndyRussG: Not necessary, I see the issue I think. [18:38:18] Coren: ah OK cool :) sorry about that [18:38:27] http://pastebin.com/4ERNy4Q4 [18:38:29] That said, that issue is supposed to be "impossible" [18:38:37] andrewbogott_afk: Ping when you're around [18:38:43] sounds interesting [18:38:59] anow: What's your Wikitech username? [18:39:07] sputnizy [18:39:14] no: sputniza [18:39:20] Your account has access to the tools project, but not the generic bastions - which is not supposed to be possible. [18:39:42] https://wikitech.wikimedia.org/wiki/User_talk:Sputniza#Welcome_to_Tool_Labs [18:39:55] Your shell access was granted [18:39:59] this is not it? [18:39:59] anow: You should be able to log onto the tools' bastions (login.tools.wmflabs.org) [18:40:42] anow: Yes, you *should* have access to the general bastions too - it should not be possible for your account not to be. Yet, it is. *grumble* [18:40:58] But if all you need is access to Tool Labs, you already got that and can access it directly. [18:41:06] Coren: why would it be working from Putty tho? [18:41:37] AndyRussG: It shouldn't - unless she connected to Tool labs with putty (which would work) [18:41:40] I see! I could login here: ssh sputzina@login.tools.wmflabs.org [18:41:58] AndyRussG: we had a spelling mistake in my username before, when we tried it [18:42:01] And in fact, as far as I can tell, she never logged onto a generic bastion in the past. [18:42:24] (That said, it's a real issue - at least it won't affect her for now) [18:42:30] anow: aaah OK that's the reason [18:42:38] I was about to say that we had tried that... [18:42:52] anow: so now you're logged into login.tools.wmflabs.org? [18:42:58] yeah, sorry - didn't realize before [18:43:02] yes! [18:43:12] anow: fantastic! :) [18:43:27] AndyRussG: Coren: great! thanks! [18:43:41] Coren: merci bien! :) [18:44:35] So, I have to leave late and dark here already .. in europe. thanks again and have a nice day .. [18:44:52] anow: bye! as always please ping anytime [18:45:43] AndyRussG: Filed as https://phabricator.wikimedia.org/T89667 [18:46:04] 3Wikimedia-Labs-wikitech-interface, Labs: Wikitech may fail to add the project-bastion group when shell access is granted - https://phabricator.wikimedia.org/T89667#1042046 (10coren) [18:46:46] Coren: cool... [18:48:22] 3Labs: Wikitech created a broken LDAP entry for a new instance - https://phabricator.wikimedia.org/T89001#1042050 (10coren) p:5High>3Unbreak! T89667 may be an additional symptom of the same problem. Given that this now points to increasingly troublesome LDAP corruption (or at least, desynchronization with t... [20:01:52] 3Wikibugs: Problems with the new color scheme - https://phabricator.wikimedia.org/T89632#1042192 (10valhallasw) After checking, the white-and-black-safe color palette is: * 3/green * 4/light red * 5/red * 6/dark purple * 7/orange * 10/teal * 11/cyan * 12/blue * 13/magenta * 14/gray * 15/light gray with 2/dar... [20:03:15] Cyberpower678: did you take care of the intuition integration on xtools? https://translatewiki.net/wiki/Thread:Support/About_Intuition:X%27stools-others/en [20:04:05] Didn't know it existed. [20:10:44] 3Wikibugs: Color floods into the main text and url in irssi - https://phabricator.wikimedia.org/T89633#1042199 (10valhallasw) Color bleeding: ```2015-02-16 07:33:27,627 - irc3.wikibugs - DEBUG - > PRIVMSG #mediawiki-feed : \x0313Phabricator\x03, \x0313operations\x03, and 1 other: Mysql search issues flagged by... [20:20:20] 3Wikibugs: Color floods into the main text and url in irssi - https://phabricator.wikimedia.org/T89633#1042205 (10Nikerabbit) So it's difference in clients on how they handle commas not followed by another number (the background color). It's been way too long since I wrote my own IRC bots to spot this earlier ;) [20:30:28] (03PS1) 10Merlijn van Deen: Improve wikibugs color scheme [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190858 (https://phabricator.wikimedia.org/T89632) [20:34:16] (03CR) 10Merlijn van Deen: [C: 04-1] "Running online test atm." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190858 (https://phabricator.wikimedia.org/T89632) (owner: 10Merlijn van Deen) [20:34:44] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042224 (10valhallasw) [20:36:22] 3Wikibugs: get redis2irc to show formatting errors - https://phabricator.wikimedia.org/T89674#1042231 (10valhallasw) 3NEW [20:36:41] 10Wikibugs: get redis2irc to show formatting errors - https://phabricator.wikimedia.org/T89674#1042231 (10valhallasw) 3NEW [20:38:04] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042238 (10valhallasw) fdsafdsadfsa [20:38:07] 10Wikibugs, 6Phabricator, and 1 other: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042238 (10valhallasw) fdsafdsadfsa [20:46:38] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042249 (10valhallasw) now with more debug logging! [20:47:42] (03PS2) 10Merlijn van Deen: Improve wikibugs color scheme [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190858 (https://phabricator.wikimedia.org/T89632) [20:48:03] (03PS3) 10Merlijn van Deen: Improve wikibugs color scheme [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190858 (https://phabricator.wikimedia.org/T89632) [20:52:01] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042252 (10valhallasw) fdafdsfdsafs [20:52:13] what the heck :< [20:59:53] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042267 (10valhallasw) was this just redis borking? [20:59:57] 10Wikibugs, 6Phabricator, and 1 other: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042267 (10valhallasw) was this just redis borking? [21:00:19] odd [21:00:40] 10Wikibugs, ... [21:00:46] 10Wikibugs, ... [21:00:58] ... [21:02:28] \x1f\x0310Wikibugs\x0f looks OK to me [21:02:37] as ^_ == \x1f [21:03:16] 3Wikibugs, Tool-Labs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042268 (10valhallasw) adding tool-labs to see if that helps with underlining (second project?) [21:03:22] 10Wikibugs, 6Phabricator, 10Tool-Labs, and 1 other: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042268 (10valhallasw) adding tool-labs to see if that helps with underlining (second project?) [21:04:37] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042271 (10valhallasw) [21:04:39] 10Wikibugs, 6Phabricator, and 1 other: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042271 (10valhallasw) [21:05:15] maybe it's .strip()ed somewhere... [21:06:51] because \x1f is also a record separator :(((( [21:08:55] well, then we first do color and then formatting [21:16:48] dsdsadsa [21:19:42] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042279 (10valhallasw) whitespace, whitespace everywhere. Now with colors first! [21:20:09] :< [21:21:26] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042288 (10valhallasw) I keep fighting with redis. Isn't redis supposed to make life easy? [21:21:32] 10Wikibugs, 6Phabricator, and 1 other: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042288 (10valhallasw) I keep fighting with redis. Isn't redis supposed to make life easy? [21:21:37] YESSSSSSSSSSS [21:24:42] valhallasw`cloud: what about redis? [21:25:00] legoktm: I dunno, I keep having issues with bugs not being sent from wikibugs to redis2irc [21:25:11] basically every time above where wikibugs did report it bug phawkes didnt [21:25:19] o.O [21:25:25] locally or on labs? [21:25:37] on laba [21:26:45] are you using a different queue key? [21:27:02] yeah [21:27:03] wikibugs2-dev [21:27:19] I'm not sure what's happening, maybe disconnects or something [21:28:20] (03PS4) 10Merlijn van Deen: Improve wikibugs color scheme [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190858 (https://phabricator.wikimedia.org/T89632) [21:30:26] (03PS1) 10Merlijn van Deen: Match all projects, even if a channel is already used [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190933 (https://phabricator.wikimedia.org/T89644) [21:38:29] (03PS1) 10Merlijn van Deen: Always show four tags, most relevant first [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190936 (https://phabricator.wikimedia.org/T89634) [21:42:01] 3Wikibugs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042337 (10valhallasw) [21:44:58] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042339 (10valhallasw) [21:45:11] legoktm: it's broken again. [21:45:12] :( [21:45:18] :| [21:45:21] 2015-02-16 21:44:57,852 - wikibugs.wb2-phab - DEBUG - {'title': 'wikibugs test bug', 'projects': {'Wikibugs': {'shade': 'blue', 'uri': '/tag/wikibugs/', 'tagtype': 'briefcase', 'disabled': False}}, 'user': 'valhallasw', 'url': 'https://phabricator.wikimedia.org/T1152#1042339'} [21:45:21] Push? (y/N)y [21:45:29] and then it just doesn't show up in redis2stdout [21:45:56] and I have no clue how to debug this stuff [21:46:09] maybe the redis cli tool [21:52:19] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042345 (10valhallasw) dumdiedum [21:53:17] 3Wikibugs: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042348 (10valhallasw) [21:53:18] 10Wikibugs, 7Easy, 5Patch-For-Review, 7Pywikibot-General: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042348 (10valhallasw) [21:54:55] 3Wikibugs, Tool-Labs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#19999 (10valhallasw) [21:55:30] 3Wikibugs, Tool-Labs, Phabricator: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042350 (10valhallasw) and now it will work! really! [21:55:36] 10Tool-Labs, 10Wikibugs, 6Phabricator, 7Easy, 5Patch-For-Review: wikibugs test bug - https://phabricator.wikimedia.org/T1152#1042350 (10valhallasw) and now it will work! really! [21:57:31] !toolshelp [21:57:32] https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help [21:58:29] (03PS2) 10Merlijn van Deen: Always show four tags, most relevant first [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/190936 (https://phabricator.wikimedia.org/T89634) [21:58:39] Coren: do we have a list of limitations for tool accounts? [21:58:43] legoktm: well, seems to work now. Took me the best part of my evening, but hey. [21:58:48] Betacommand: limitations in what sense? [21:59:24] valhallasw`cloud: concurrent myswl connections, disk space, ect [22:00:10] Betacommand: ah. I think there are no limits in general, unless they are there by default [22:00:16] which would be the case for mysql, I think [22:01:19] valhallasw`cloud: we have a user on the mailing list complaining about hitting the too many connections limit and was looking for number to give them [22:01:29] | max_user_connections | 0 | [22:01:34] | max_connections | 1000 | [22:01:41] that's on enwiki.labsdb [22:02:11] yeah, thats 'too many connections' = too many connections overall [22:03:26] Coren: I’m trying to wrap up jessie on labs (finally). Any thoughts about https://phabricator.wikimedia.org/T87309? Is there a simple OS check we can put in puppet for that? [22:05:04] andrewbogott: is there a max connections per tool on labs? [22:05:20] Betacommand: I don’t know. [22:05:24] Betacommand: no, there isn't [22:06:05] valhallasw`cloud: Im pretty sure there is, or at least was [22:06:42] Betacommand: I'm pretty sure max_user_connections=0 means there isn't. [22:06:48] there was one on the toolserver [22:13:56] 3Labs, Wikimedia-Labs-Infrastructure: No init script for idmapd on labs jessie instances - https://phabricator.wikimedia.org/T87309#1042381 (10faidon) So, Debian ships an nfs-common package that contains a SysV init script with the same name that spawns all the needed daemons. Individual daemons like idmapd can... [22:16:15] paravoid: is that just a question of adding an os switch to manage nfs-common on deb and idmapd on ubuntu? [22:17:06] that + /etc/default [22:17:24] $ grep -B 1 IDMAPD /etc/default/nfs-common [22:17:25] # Do you want to start the idmapd daemon? It is only needed for NFSv4. [22:17:25] NEED_IDMAPD= [22:17:37] that needs to become NEED_IDMAPD=yes [22:18:13] ok [22:18:20] I’ll write a patch [22:18:34] I fixed my other jessie bug so now I want it to be perfect :) [22:18:44] :D [22:18:49] what was the other one? [22:19:22] um… just a thing with parted and lvs. The parted script behaved slightly differently on debian, ignoring a command when a tty wasn’t attached. [22:19:41] stupid but effective fix: https://gerrit.wikimedia.org/r/#/c/190934/ [22:20:02] haha [22:37:30] 3Labs, Wikimedia-Labs-Infrastructure: No init script for idmapd on labs jessie instances - https://phabricator.wikimedia.org/T87309#1042437 (10Andrew) Proposed bandaid: https://gerrit.wikimedia.org/r/#/c/190948/ [22:58:07] Hello [22:59:52] I have a problem of connection to tools-login.wmflabs.org with ssh: I alaways have "Agent admitted failure to sign using the key. Permission denied"(publickey,hostbased)." [23:00:07] Someone would have the skills to help me? Thank you [23:04:00] (re) [23:08:40] Someone here? [23:15:28] Please, a little hope [23:24:56] Baccanal: did you upload a public key ? [23:25:16] matanya: yes, it is in my preferences tab [23:26:07] I can log in with Windows, but now with Linux I don't know how logging in [23:27:09] Baccanal: try with ssh -vvvv tools-login.wmflabs.org and post the output to dpaste.org [23:28:56] Baccanal: what is your login name? [23:29:01] matanya: voilà https://dpaste.de/0LdZ [23:29:10] andrewbogott: botomatik [23:29:29] Baccanal: line 7 and 8 tells you [23:30:52] matanya: what I should do? [23:31:00] Baccanal: fix your key ? [23:31:08] but how? [23:32:23] I ever copied the public SSH key for my preference tab to ~/.ssh/id_rsa.pub [23:33:26] so how did you create it? [23:34:16] hmm... maybe I should add a public key to my preferences tab (it is a public key by client or OS I guess) [23:35:12] Baccanal: you can either copy the one from windows, or upload another key [23:42:55] matanya: I uploaded an other key generated with ssh-keygen, but I still have the error "debug3: Could not load "/home/emmanuel/.ssh/id_rsa" as a RSA1 public key [23:43:20] id_rsa is the private key [23:43:25] whereas the public key is obviously in "id_rsa.pub" and not in "id_rsa" [23:43:31] yes, but why this error, then ? [23:43:31] right [23:44:01] just asking: is .ssh permission 700 and id_ras 600 ? [23:44:05] *id_rsa [23:44:41] yes, exactly [23:45:39] Baccanal: https://stackoverflow.com/questions/13606363/could-not-load-id-rsa-as-a-rsa1-public-key [23:45:46] so it is ok to get this error [23:47:02] debug3: Could not load "/home/emmanuel/.ssh/id_rsa" as a RSA1 public key [23:47:02] but ssh botomatik@tools-login.wmflabs.org does'nt work [23:48:35] i compared it with my output [23:48:46] the server doesn't accpet your key [23:49:00] I get: debug1: Server accepts key: pkalg ssh-rsa blen 279 [23:49:05] you don't [23:51:48] so Baccanal what is wrong with your keys ? [23:54:19] I really don't know, I just tried know again with a new key, copied from vim to the preferences tab, and it is not recognized then by "ssh -vvvv tools-login.wmflabs.org [23:54:19] " or "ssh botomatik@tools-login.wmflabs.org [23:54:20] " [23:55:41] you prbably need someone with more rights/knowledge than me now. Coren / andrewbogott [23:56:35] thanks for your help and your availability matanya [23:56:47] sure [23:57:13] Baccanal: try ssh -i /path/to/the/key/you/really/mean botomatik@tools-login.wmflabs.org [23:58:15] same problem: If you are having access problems, please see: https://wikitech.wikimedia.org/wiki/Access#Accessing_public_and_private_instances [23:58:15] Permission denied (publickey,hostbased).