[00:10:11] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 2 others: @scfc cannot add a user to the Tools project: "You must be a member of the projectadmin role in project to perform this action." - https://phabricator.wikimedia.org/T126937#2027665 (10scfc) IRC showed that the... [00:36:41] andrewbogott, still around? [00:42:41] 6Labs, 10Labs-Infrastructure, 10MediaWiki-extensions-OpenStackManager, 10Tool-Labs, and 2 others: @scfc cannot add a user to the Tools project: "You must be a member of the projectadmin role in project to perform this action." - https://phabricator.wikimedia.org/T126937#2027687 (10Krenair) 5Open>3Resolv... [00:43:13] never mind :) [00:51:14] !log tools.stashboterr Intentional invalid servicegroup [00:51:15] tools.stashboterr is not a valid project. [00:58:41] !log tools.stashboterr Intentional invalid servicegroup [00:58:42] tools.stashboterr is not a valid project. [01:01:01] !log tools.stashbot Upgraded to 64fcdcf; !log project name validation in #wikimedia-labs; fixed bug in Phab mention regex [01:01:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [03:14:00] !log tools.sal Cleaned up junk data from before project validation was in place [03:14:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL, Master [03:59:45] !log ores Restarted the uwsgi-ores-web service on ores-web-02 [03:59:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [04:01:44] !log ores Ammending last message -- Restarted the uwsgi-ores-web service on *ores-web-01*. The service on ores-web-02 was left alone. [04:01:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [08:36:10] 6Labs, 10Wikimedia-Fundraising, 10Wikimedia-Fundraising-CiviCRM, 7Tracking: Create new labs project: fundraising-integration - https://phabricator.wikimedia.org/T88599#2027965 (10hashar) [08:52:43] bd808: I see on https://wikitech.wikimedia.org/wiki/Logstash that when logging to logstash with GELF, type should be set to "gelf". Are there any other gotcha? Would you have time to have a look in logstash-beta and let me know if I should recategorize anything for elasticsearch logging ? [09:16:20] !log deployment-prep re-enabling puppet on elastic05 (https://phabricator.wikimedia.org/T126891) [09:16:20] Please !log in #wikimedia-releng for beta cluster SAL [09:16:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [12:25:41] andrewbogott: what about databases that were bound to be on the same server as the redirect for a certain replica? can i request a move or do i have to move them myself? [12:26:58] gifti: user databases? You need to move them yourself. [12:28:13] :( [12:28:29] is there a documentation, valhallasw`cloud? [12:28:38] of what? [12:28:45] moving user databases [12:29:10] other than 'don't store permanent data on replica databases', no, probably not [12:29:22] you can use mysqldump probably? but I'm not sure if labsdb1002 is online at all [12:29:30] duh [12:30:29] gifti: c2.labsdb is completely down, unfortunately [12:30:33] (=labsdb1002) [12:30:38] no problem [14:19:46] 10Tool-Labs-tools-Other, 6TCB-Team, 7German-Community-Wishlist: "pageviews" tool link at bottom creates invalid HTML syntax (due to missing HTML encoding of characters like '&') - https://phabricator.wikimedia.org/T126975#2028690 (10Aklapper) [14:31:01] 6Labs, 6Operations, 10ops-eqiad: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2028710 (10Cmjohnson) labsdb1002 is a CISCO server. I can pull a disk out of one of the decommissioned servers and replace but we should really start working on replacing this server. [15:39:29] hello. i am apparently an administrator of the "phabricator" project in labs. i learned about it because i started getting puppet alert emails every day announcing that it has keeled over. i would like to no longer be one, how do i remove myself? [15:45:45] 6Labs, 10Labs-Infrastructure: Horizon: login refused with "Unable to retrieve authorized projects." - https://phabricator.wikimedia.org/T126981#2028874 (10hashar) 3NEW [15:47:22] 10Tool-Labs-tools-Other, 6TCB-Team, 7German-Community-Wishlist: "pageviews" tool link at bottom creates invalid HTML syntax (due to missing HTML encoding of characters like '&') - https://phabricator.wikimedia.org/T126975#2028882 (10PatoLogic) - sorry for the German, I thought this was a problem only relevan... [16:19:24] 6Labs: MediaWiki Extension "ImportArticles" project - https://phabricator.wikimedia.org/T89208#2028959 (10Danny_B) [16:19:51] 6Labs, 10Wikimedia-Fundraising, 10Wikimedia-Fundraising-CiviCRM: Create new labs project: fundraising-integration - https://phabricator.wikimedia.org/T88599#2028961 (10Danny_B) [16:26:31] !log ores restarted workers on ores-worker -02 and -03 [16:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ores/SAL, Master [16:46:30] 10Tool-Labs-tools-Other, 6TCB-Team, 7German-Community-Wishlist: "pageviews" tool link at bottom creates invalid HTML syntax (due to missing HTML encoding of characters like '&') - https://phabricator.wikimedia.org/T126975#2029056 (10MusikAnimal) I understand this is an on-wiki issue on dewiki, and not with t... [16:59:36] RECOVERY - Puppet failure on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [0.0] [17:09:30] MatmaRex: under https://wikitech.wikimedia.org/wiki/Special:NovaProject [17:11:54] valhallasw`cloud: thank you! [17:28:26] 6Labs, 5Patch-For-Review, 7Tracking: Support instance manipulation, proxies, dns with Horizon (Quarterly goal tracking bug) - https://phabricator.wikimedia.org/T124181#2029261 (10Andrew) [17:28:28] 6Labs, 5Patch-For-Review: Unable to change projects in horizon - https://phabricator.wikimedia.org/T123310#2029258 (10Andrew) 5Open>3Resolved a:3Andrew This is now resolved. Note that for unrelated reasons I've disabled all nova actions in Horizon, so Horizon is currently useful only for viewing project... [17:47:31] 6Labs, 10Labs-Infrastructure: Horizon: login refused with "Unable to retrieve authorized projects." - https://phabricator.wikimedia.org/T126981#2029325 (10Andrew) 5Open>3Resolved a:3Andrew This should be fixed. Note that the glance login name has changed -- now it should be the same as your wikitech use... [18:15:38] I don't suppose anyone here knows how to restart Rezabot? It's having problems related to the SessionManager system changes. Brad has a theory that a restart would clear old cookies and possibly get it to login properly again. [18:16:41] The layout of /data/project/rezabot doesn't make it obvious how things are controlled unfortunately [18:19:15] bd808: I think I need to go kick my cable modem... [18:19:21] k [18:20:54] chasemp, valhallasw`cloud: would/could you issue a job restart for the SGE job named "red" owned by the "rezabot" tool to test our theory that a restart will clear old cookies and allow the bot to login again? [19:04:13] bd808: just a sec [19:05:01] valhallasw`cloud: hey, thanks. There are actually 4 jobs running and I can't figure out which one (or ones) are having problems [19:05:09] ah, I see. [19:05:33] and I can't find any docs in the prject homedir about how it all workd :/ [19:06:09] * valhallasw`cloud nods [19:06:49] bd808: qstat -u "tools.rezabot" lists the jobs, then qstat -j tells you what script it's running [19:06:58] how do we know it's having sessionmanager issues? [19:07:11] Logs in production Logstash [19:07:24] ah. Does that include an ip? [19:07:39] because the jobs are fairly spread out [19:07:46] it might. let me look [19:09:16] valhallasw`cloud: yuck. the ips in Logstash are from the Varnish servers, not the client [19:09:20] òk [19:09:57] oh, darn. The jobs are all running in the 'task' queue [19:09:59] so can't restart them [19:11:48] bd808: leave a talk page message on https://fa.wikipedia.org/wiki/%D8%A8%D8%AD%D8%AB_%DA%A9%D8%A7%D8%B1%D8%A8%D8%B1:Yamaha5 ? [19:11:48] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:49] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:49] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:50] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:50] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:50] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:51] D8: Add basic .arclint that will handle pep8 and pylint checks - https://phabricator.wikimedia.org/D8 [19:11:59] yuck [19:12:02] that's a bug [19:12:05] WTF?? [19:14:50] valhallasw`cloud: yeah. I'll do that [19:18:14] Hi, i'm still trying to complete the Git/gerrit config [19:18:39] 6Labs, 10Labs-Infrastructure: Horizon: login refused with "Unable to retrieve authorized projects." - https://phabricator.wikimedia.org/T126981#2029578 (10hashar) Confirmed login works fine with my regular user as well as `nodepoolmanager`. Awesome thank you. [19:19:18] i pushed from my station to gerrit.wikimedia but i have this error message: [19:19:36] committer email address tools.vocabulary-index@tools-bastion-01.tools.eqiad.wmflabs [19:20:40] does not match your user account. [19:20:49] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: Labs project admin can not delete per project image on Horizon - https://phabricator.wikimedia.org/T110936#2029585 (10hashar) [19:21:25] 6Labs, 10Labs-Infrastructure, 5Continuous-Integration-Scaling: Labs project admin can not delete per project image on Horizon - https://phabricator.wikimedia.org/T110936#1590423 (10hashar) Took a screenshot. Not a priority since we can get rid of the images from labnodepool via the openstack cli. [19:21:26] Youni: right, you're not allowed to push using other people's email addresses (well, email addresses not registered to your account) by default [19:22:02] Youni: so you either need to turn that off in gerrit, or you need to change the email address used by git (git config user.name and user.email, I think?) [19:23:44] it's all about the same mail adress oliv71@laposte.net [19:24:48] Youni: the email in the git commit is 'tools.vocabulary-index@tools-bastion-01.tools.eqiad.wmflabs', which is why gerrit is confused [19:25:08] oups... [19:27:52] valhallasw`cloud: Sorry how can i corretc it ? [19:28:25] Youni: set up the username and email in git (with git config --- the git commit message should have told you how) [19:28:32] then git commit --amend --reset-author [19:29:04] Youni: `git config user.name Youni`, then `git config user.email oliv71@etc` [19:40:23] i config git than try to commit --amend... than i reach a vi text file i basicaly write a msg on it save and quit [19:41:11] than i git push but failed again with the same error email adress [19:42:16] I hit an error "DB connection error: Unknown database 'p50380g50497__wikidb_zhwiki' (zhwiki.labsdb)" [19:42:18] any known issue? [19:43:52] liangent: /topic [19:44:25] valhallasw`cloud: hmm thx [19:44:30] but replica is working [19:44:44] yes, because it has been moved to another database server [19:45:03] by git config -l i can see that i have 2 user.email=oliv71@... and 2 user.name one is bad [19:45:26] Youni: that's.. odd. if you git log -1, does it show the incorrect username? [19:45:31] eh, incorrect email? [19:46:24] valhallasw`cloud: will user db contents be lost? [19:46:40] liangent: unknown, but possible [19:47:18] liangent: keep in mind that user databases on replicas should not be used for data that needs long-term storage [19:47:27] bad name with git log -1 [19:47:41] valhallasw`cloud: I run mysqldump in cron every day.. [19:48:55] liangent: that sounds like a good idea :-) [19:49:17] (assuming you need it to join w/ other databases on that shard) [19:51:09] Youni: huh. I'm confused. [19:51:26] If I run git config -l as tools.vocabulary-index, no user.* entries show up [19:52:40] Youni: so try setting user & email again, I guess? [19:53:12] i'm running git on my local station not on tools.vocabulary-index [19:53:50] Youni: I don't get your setup :/ [20:02:28] i change the git config via the tool account, than i tried git commit --amend --reset-author ; now i can't get out from vi [20:08:56] i have the same error [20:13:47] should i use the push command from my station or from the labs tool directory? first case i have an email config problem and second case i have permission denied [20:17:16] Youni: you should be able to push from tools using https [20:17:38] Youni: but you have to use the password at https://gerrit.wikimedia.org/r/#/settings/http-password [20:24:03] dawned i don't understand anymore i need a break sorry this stuff is making me crazy, thanks i'll be back ;-P [20:26:55] Youni: alternatively, create a github account and follow their simple tutorial instead of trying to get gerrit to do what you want :/ [20:27:48] !log tools.stashbot Updated to 3314de8; Guard against %D\d matching Pabricator echo; De-dup Phabricator echos [20:27:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL, Master [20:44:29] 6Labs, 10Labs-Infrastructure, 10labs-sprint-117, 10labs-sprint-118, and 3 others: Move project membership/assignment from ldap to keystone mysql - https://phabricator.wikimedia.org/T115029#2029747 (10Andrew) [21:27:00] valhallasw`cloud: i would try later , i think i'm very close to reach. It's a lot of things to assimilate and probably the main dificulty for me is Git. By one way, ssh or http, i should push soon on "my" gerrit account. Lot of thanks and see you soon. [22:02:00] PROBLEM - SSH on tools-exec-1221 is CRITICAL: Server answer [22:04:14] chasemp ^ might be another NFS hang [22:05:26] although it seems to hang earlier than other times (at debug1: expecting SSH2_MSG_KEX_ECDH_REPLY) rather than when entering the shell [22:07:01] RECOVERY - SSH on tools-exec-1221 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2~wmfprecise2 (protocol 2.0) [22:11:26] or maybe it's a temporary issue? *logs in* [22:12:09] no, it's still hanging, but there is an initial handshake, so shinken doesn't notice. Evil! [22:23:02] PROBLEM - SSH on tools-exec-1221 is CRITICAL: Server answer [22:29:46] !log phabricator rebooting instance phab-01 [22:29:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, dummy [22:32:09] !log phabricator rebooting instance phab-02 [22:32:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, dummy [22:35:32] RECOVERY - Puppet staleness on tools-webgrid-lighttpd-1208 is OK: OK: Less than 1.00% above the threshold [3600.0] [22:52:53] !log phabricator stashed local changes in /var/lib/git/operations/puppet on deploy instance, checked out ‘production’ branch, updated [22:52:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, dummy [22:54:08] !log phabricator Andrew encourages future puppet hackers to commit their local changes so that this mess doesn’t reappear [22:54:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL, dummy [23:10:39] 6Labs: broken labs instances (ssh or perms), do we care? - https://phabricator.wikimedia.org/T126323#2029952 (10Krenair) [23:12:05] 6Labs: broken labs instances (ssh or perms), do we care? - https://phabricator.wikimedia.org/T126323#2011201 (10Krenair) [23:16:32] 10MediaWiki-extensions-OpenStackManager, 10Notifications, 3Collaboration-Team-Current, 5Patch-For-Review: Update OpenStackManager notifications to new language and format - https://phabricator.wikimedia.org/T125691#2029956 (10Etonkovidova) Checked in betalabs osm notifications: osm-instance-build-complete... [23:27:42] hi there. A user database of my tool pb is not available at the moment. It was located on dewiki.labsdb, so I guess that this is caused by the c2.labdsdb failure, and that the user databases have not been moved to a different server like dewiki & co. Is this plausible? [23:33:06] dewiki was on c2, yeah [23:33:14] user databases were not moved afaik [23:33:22] you probably shouldn't have user databases on c[1-3] [23:36:18] yes, I just read that in the documentation [23:36:39] AFAIR that was not part of the help page once I created the database ;) [23:36:57] anyway, I restored a backup on tools-db, and that works fine. [23:37:42] so the question is: are the changes on c2.labsdb lost, or is there a chance to see them once the server is up again?