[00:03:30] !log db1050 alive. do not repool. [00:03:36] Logged the message, Master [00:09:08] There's no LD scheduled but FYI Flow deploy completed some time ago. Thanks [00:29:32] (03PS1) 10Ori.livneh: Release ordered_json.rb under the terms of the Apache license [operations/puppet] - 10https://gerrit.wikimedia.org/r/112828 [00:35:35] (03CR) 10Ryan Lane: [C: 031] "Excellent. Thanks!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112828 (owner: 10Ori.livneh) [00:35:51] (03CR) 10Ori.livneh: [C: 032] Release ordered_json.rb under the terms of the Apache license [operations/puppet] - 10https://gerrit.wikimedia.org/r/112828 (owner: 10Ori.livneh) [01:14:37] db crash again? [01:15:04] Function: User::saveOptions Error: 1213 Deadlock found when trying to get lock; try restarting transaction (*ip address*) [02:18:45] !log LocalisationUpdate completed (1.23wmf13) at 2014-02-12 02:18:44+00:00 [02:18:51] Logged the message, Master [02:22:35] who is the right person to ask about gerrit permissions? [02:29:10] gwicke: I'm not the right person, necessarily, but I can probably help [02:37:38] !log LocalisationUpdate completed (1.23wmf12) at 2014-02-12 02:37:38+00:00 [02:37:46] Logged the message, Master [03:13:44] (03CR) 10Jforrester: [C: 04-1] "For discussion." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112590 (owner: 10Jforrester) [03:16:07] (03CR) 10Jforrester: [C: 04-1] "Need to notify the community first." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/111149 (owner: 10Jforrester) [03:17:13] (03CR) 10Jforrester: "Max: The OTRS admins, whose call it is." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100927 (owner: 10Jforrester) [03:19:05] !log LocalisationUpdate ResourceLoader cache refresh completed at 2014-02-12 03:19:05+00:00 [03:19:14] Logged the message, Master [03:22:40] RECOVERY - mysqld processes on db1036 is OK: PROCS OK: 1 process with command name mysqld [03:24:10] about topic branch names, if every single one is unique, isnt it just repeating the first line of the commit message? doesn't it make more sense to have a few topic branch names that are being reused, so like categories [04:45:31] (03CR) 10GWicke: "The salt issue is tracked at https://github.com/saltstack/salt/issues/10339" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112640 (owner: 10GWicke) [04:47:58] !log disabling mysql monitoring on db48 (prep before starting OTRS upgrade) [04:48:05] Logged the message, Master [04:48:16] !log disabling https/smtpd monitoring on iodine [04:48:24] Logged the message, Master [05:11:20] PROBLEM - ElasticSearch health check on elastic1002 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1340: active_shards: 3895: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 0 [05:12:20] RECOVERY - ElasticSearch health check on elastic1002 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3896: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [05:26:14] Jeff_Green: You up still? [05:26:42] I get a fatal error on changing user rights as a steward. [05:26:55] ? [05:26:57] PHP fatal error in /usr/local/apache/common-local/php-1.23wmf13/includes/specials/SpecialUserrights.php line 155: [05:26:57] Call to undefined method UserRightsProxy::clearInstanceCache() [05:27:40] ohhh. I'm sort of tied up at the moment with an OTRS upgrade. [05:27:49] Oh okay. :) [05:27:55] I reported it already, sDrewthedoff. [05:27:59] ok cool [05:28:03] https://bugzilla.wikimedia.org/show_bug.cgi?id=61252 [05:28:18] jeff it seems to be a wmf13 issue [05:28:24] and only xwiki [05:28:25] ohhh. I'm sort of tied up at the moment with an OTRS upgrade. [05:28:38] This is a Reedy thing. [05:28:47] Or Tim. [05:28:48] Reedy:. [05:28:51] TimStarling [05:28:57] Thanks, Bsadowski1. [05:29:03] Gloria! :D [05:29:20] How did you all not notice until now? [05:29:20] everything will work as long as we don't need to change any xwiki rights [05:29:31] what is xwiki? [05:29:35] Crosswiki. [05:29:39] Like X-mas, or something. [05:29:43] Gloria: wmf13 only rolled five hours ago [05:29:49] TimStarling: https://bugzilla.wikimedia.org/show_bug.cgi?id=61252 [05:31:26] Gloria: changing xwiki rights is less often undertaken task [05:31:43] sDrewthedoff: Right, right. Sorry, I thought the deploy was longer ago. [05:32:23] Gloria: I don't change user rights 24/7. [05:32:28] :| [05:32:32] * sDrewthedoff is magnaminous and forgives [05:33:33] I think I remember this change. [05:33:35] To the code. [05:33:55] It was something to do with a rights conflicts or something, I think. [05:34:31] that is where I fail, I wouldn't knwo where to start looking [05:35:00] https://gerrit.wikimedia.org/r/#/c/110869/1/includes/specials/SpecialUserrights.php,unified [05:35:04] I think it's that. [05:35:57] "Note: I can't reproduce the issue or verify this patch." [05:36:03] that looks a pretty goo dmatch to the error [05:36:49] I'm still not sure why https://bugzilla.wikimedia.org/show_bug.cgi?id=38989 can't just use the same logic as edits. [05:36:57] With a start timestamp. [05:37:55] "Beware of bugs in the above code; I have only proved it correct, not tried it." :) [05:38:22] Gloria: We were trying rights at different wikis. [05:38:24] Not the same. [05:38:29] How could it be conflicting? [05:38:46] I tried to give myself rights at mediawikiwiki. [05:39:00] Bsadowski1: That code change was made to address a separate issue. [05:39:07] The separate issue should be solved differently. [05:39:23] in reflection, yes [05:39:23] And that code change likely caused your issue. [05:39:34] Are you sure? [05:39:48] No. That's why I said likely. [05:40:56] Gloria, and 38989 was due to you and I rights dancing at mw [05:41:06] sDrewthedoff: Indeed. :-) [05:41:36] legoktm: Are you a shell user yet? [05:41:43] I'm not. [05:41:46] Lame. [05:41:48] how urgent is it? [05:41:51] I can look at exception logs though [05:42:01] no xwiki rights changes until fixed [05:42:20] do you want it fixed in 1 minute or 10? [05:42:22] so as I said earlier, as long as we have no spambots and other ballistic issues, we can wait [05:42:30] 10-60-120 are all okay [05:43:09] TimStarling: 2 hours is fine, longer is okay [05:43:30] up to one day is acceptable, not preferable [05:43:33] That's quite the negotiation tactic. [05:44:14] legoktm: I think a straight revert is fine for now. [05:44:19] Gloria: always try to be reasonable and practicable [05:44:22] oh [05:44:25] I have the fix ready [05:44:27] agreed [05:45:13] [09:45:06 PM] (PS1) Legoktm: Only call ->clearInstanceCache() if $targetUser instanceof User [core] - https://gerrit.wikimedia.org/r/112847 [05:45:53] Special:GlobalGroupMembership is also probably broken [05:46:04] No topic, -1. [05:46:17] legoktm: need it tested? [05:46:30] I think it is better to introduce clearInstanceCache() to UserRightsProxy [05:46:36] I am testing a fix along those lines [05:47:27] We'd also need to add it to CentralAuthUser too then [05:47:46] that's what you get for duplicating code [05:48:38] legoktm: yes, ... Call to undefined method CentralAuthGroupMembershipProxy::clearInstanceCache() [05:48:52] thought so [05:49:25] globalgroup membership is less likely to have duelling rights [05:49:42] the other was local <-> central duel [05:52:30] $user = self::newFromId( $wikiID, $id ); // FIXME: newFromId is undefined [05:52:32] * legoktm sighs [05:52:45] Louder. [05:53:04] Hi pakaran. [05:58:13] taking too long [05:58:31] is legoktm's patch tested? [05:58:51] I tested that Special:GlobalGroupMembership works [05:59:14] what's the worst thing that can happen? [05:59:14] I don't have cross-wiki userrights set up, so that part isn't tested [05:59:25] !worstcase ;) [06:05:55] !log tstarling synchronized php-1.23wmf13/includes/specials/SpecialUserrights.php [06:06:03] Logged the message, Master [06:06:39] sDrewthedoff, Bsadowski1: ^ [06:06:46] here for testing, and if it fails, hardly a mega issue [06:07:10] testing [06:08:14] we have a win [06:08:21] thx legoktm [06:08:23] :) [06:08:34] will test a little further soon [06:08:55] and s:ggm is a tick too [06:09:13] Special:GlobalGroupMembership [06:14:38] (03PS1) 10TTO: Initial setup for legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 [06:28:35] !log restarted apache on mw1184 [06:28:43] Logged the message, Master [06:29:20] RECOVERY - Apache HTTP on mw1184 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 808 bytes in 0.058 second response time [06:41:14] (03CR) 10TTO: "recheck (see bug 61246)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [06:57:17] (03PS1) 10Ryan Lane: Code documentation for trebuchet's deployment module [operations/puppet] - 10https://gerrit.wikimedia.org/r/112855 [07:11:50] (03CR) 10TTO: "recheck" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/112850 (owner: 10TTO) [07:20:43] !log aaron synchronized php-1.23wmf13/includes/filebackend/SwiftFileBackend.php '288941aa53adffb7de1af8ff4fbae4c1f0c26937 and 1bb7ef5a7816a57b38caa4a302fbff3e53f16d16' [07:20:49] Logged the message, Master [07:25:57] ori: https://upload.wikimedia.org/wikipedia/en/thumb/0/02/Barbie_and_the_Magic_of_Pegasus_poster.jpg/402px-Barbie_and_the_Magic_of_Pegasus_poster.jpg [07:25:59] ugh [07:27:37] the pegasus in the distant background knows something we don't, clearly [07:28:32] who says it is in the background, maybe it is a faerie on a mini-pony [07:28:53] maybe it clearer when you see it in 3D [07:29:03] perspective is everything [07:30:44] * ori gets that Thai food. [08:19:34] (03PS3) 10GWicke: Remove old parsoid init script [operations/puppet] - 10https://gerrit.wikimedia.org/r/112640 [08:32:50] PROBLEM - ElasticSearch health check on elastic1009 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1340: active_shards: 3895: relocating_shards: 6: initializing_shards: 1: unassigned_shards: 0 [08:32:50] PROBLEM - ElasticSearch health check on elastic1004 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1340: active_shards: 3895: relocating_shards: 6: initializing_shards: 1: unassigned_shards: 0 [08:32:51] PROBLEM - ElasticSearch health check on elastic1012 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1340: active_shards: 3895: relocating_shards: 6: initializing_shards: 1: unassigned_shards: 0 [08:33:50] RECOVERY - ElasticSearch health check on elastic1004 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3896: relocating_shards: 6: initializing_shards: 0: unassigned_shards: 0 [08:33:51] RECOVERY - ElasticSearch health check on elastic1009 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3896: relocating_shards: 6: initializing_shards: 0: unassigned_shards: 0 [08:33:51] RECOVERY - ElasticSearch health check on elastic1012 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3896: relocating_shards: 6: initializing_shards: 0: unassigned_shards: 0 [08:35:01] * apergos raises an eyebrow [08:35:08] concise and actionable, no? [08:35:18] i like the color-coding [08:35:24] I don't have color coding [08:35:32] it's spelled out [08:35:39] unless you mean it has a color in th line which I would have to dig out [08:35:40] status: red [08:37:35] (03CR) 10Alexandros Kosiaris: [C: 032] Remove old parsoid init script [operations/puppet] - 10https://gerrit.wikimedia.org/r/112640 (owner: 10GWicke) [08:42:50] PROBLEM - Parsoid on wtp1004 is CRITICAL: Connection refused [08:43:42] morning [08:46:21] hashar: good morning to you too [08:48:13] that is interesting... parsoid listening on another port [08:48:17] hmmm [08:48:59] (03PS1) 10Matanya: zuul: remove lookupvar and replace with same scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112862 [08:49:10] RECOVERY - Parsoid on wtp1004 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.004 second response time [08:49:18] hashar: morning ^ :) [08:49:58] got restart comp [08:49:59] !log restarted parsoid on wtp1004. After https://gerrit.wikimedia.org/r/112640 is was for some reason listening on 56547 and not 8000. The restart fixed it [08:50:08] Logged the message, Master [08:51:41] akosiaris: I think parsoid attempt to listen on whatever port is configured (8000) and if it is busy listen on another port [08:51:49] or might be related to some env variable not being set properly [08:51:58] why would it do that ? [08:52:04] what sense does it make ? [08:52:20] I think that is for testing/dev purposes [08:52:44] you probably want to bug fill it [08:53:15] api/ParsoidService.js: var port = process.env.VCAP_APP_PORT || process.env.PORT || 8000; [08:53:33] I already mailed gwicke. I 'll wait his assesment. And if it confirms yours, a bug it is [08:53:50] so environment variables with the fallback being 8000 [08:54:09] yeah and VCAP_APP_PORT is for a cloud service ( https://www.appfog.com ) [08:54:11] but a restart had it to 8000 and not 56657 (or whatever it was) [08:54:25] 56547 [08:54:25] I am pretty sure some folks were talking about that env variable yesterday night [08:54:51] I guess 56547 would be a port assigned automatically whenever you open a listen socket on port 0 [08:55:00] i.e. the OS gives you a free port [08:55:02] yeah, an ephemeral port [08:55:23] so might be because one of the env variable is unproperly set [08:55:49] so one of those variables being set to 0 ... [08:55:59] * | 92b3b1d - Rename PARSOID_PORT to PORT and PARSOID_INTERFACE to INTERFACE (11 hours ago) [08:56:00] but it must also be some race [08:56:01] nice [08:56:27] how do we know which parsoid we are running btw ? [08:56:33] git log in the deploy repo ? [08:57:04] I think it is deployed using git deploy, so potentially by inspecting the parsoid process running, you can find the path it runs from and then look at that git repo [08:57:32] that is a bit messy :( [08:57:40] it it git-deploy indeed [08:57:44] it is* [08:57:56] i was hoping something more generic though [08:57:58] anyway [08:59:08] seems like we are a bit before that commit hashar [08:59:10] Gabriel is working on a debian package to ship parsoid + its node modules [08:59:23] yeah I know [08:59:28] akosiaris: yup probably, but the init / upstart conf might already have been updated in puppet [08:59:36] potentially leading to the env var not being set properly somehow [08:59:44] could be [09:03:15] (03CR) 10Hashar: "I keep forgetting why we had/have to use scope.lookupvar(). Apparently the @statsd_host should just work." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112862 (owner: 10Matanya) [09:03:22] at least it is monitored [09:03:49] akosiaris: easy one to review ^^ ? :) [09:07:05] scope.lookupvar() must die apparently [09:07:16] if you get it merged, i can apply the fix on Zuul server right away [09:08:45] heh. scope.lookupvar in the same scope ? [09:09:06] can't remember the details [09:09:35] probably caused by puppet noobiness back in that time :-] [09:10:12] merging [09:10:36] (03CR) 10Alexandros Kosiaris: [C: 032] zuul: remove lookupvar and replace with same scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112862 (owner: 10Matanya) [09:11:08] thanks akosiaris :) [09:11:26] running puppet [09:13:22] nothing changed \O/ [09:21:08] (03CR) 10Tim Landscheidt: "@ottomata: Please adjust the commit message in such cases in the future then :-). Seeing "DO NOT MERGE" in the log of merged changes can " [operations/puppet] - 10https://gerrit.wikimedia.org/r/111523 (owner: 10Ottomata) [09:58:43] YuviPanda: do you do mobile stuff ? [09:58:51] some of it, matanya [09:58:53] why [09:59:00] have you seen https://play.google.com/store/apps/details?id=animaonline.android.wikiexplorer ? [09:59:34] matanya: looking [09:59:43] matanya: ah, I haven't! [09:59:47] this one is up my alley [09:59:48] installing [09:59:57] you can grabe some ideas for it, really a great one :) [10:00:51] was published yesterday iirc [10:01:13] matanya: thanks! [10:01:26] my pleasure :) [10:01:29] matanya: the wikipedia app reboot is going to be out in a month or two. Will have editing! [10:01:38] YAY! [10:02:01] looking forward for it, the current one annoyies me a lot ... [10:02:27] matanya: yup. do you have an android device? [10:02:37] yes, i do [10:02:41] (03CR) 10Alexandros Kosiaris: [C: 032] Bind nrpe to a primary address [operations/puppet] - 10https://gerrit.wikimedia.org/r/112698 (owner: 10Alexandros Kosiaris) [10:02:44] matanya: we put out builds every other wednesday. I can get you one now to try it out with [10:02:46] matanya: what version of Android? [10:03:09] last time i rooted it, i through 4.0 [10:03:17] but didn't change that in months [10:03:28] so it is probably still it [10:03:49] matanya: that's fair enough. can you give me your email address so I can mail you a recent build? [10:03:58] the last public one was 2 weeks ago, a lot has changed since then [10:04:57] matanya: sent! [10:05:00] matanya: it has rough edges still [10:05:27] thanks a lot, i'll test and push you bugs :) RTL would be the first most likely [10:06:07] matanya: :D [10:06:16] matanya: I already did an RTL pass with aharoni a few weeks ago [10:06:28] matanya: so it shouldn't be *too* bad, but I am sure it has edges [10:06:36] ah, stole my thunder [10:06:59] when i'll get home tonight i'll give it a shot [10:07:39] matanya: :) [10:07:50] matanya: I tried out WikiExplorer. Honestly even unfinished I like our app better :P [10:07:57] even without the editing [10:08:13] why may i ask? [10:08:52] matanya: their font rendering seems terrible, and they seem to be stripping inline styles not too properly [10:08:59] matanya: plus if I tap a link, I have to tap again to actually read it [10:09:37] matanya: wikigrid is an interesting concept but thoroughly unoptimized for a mobile device [10:09:53] makes sense [10:10:19] matanya: the 'you have to tap twice to follow links' got annoying really fast [10:10:44] didn't note it until you memtioned it [10:11:27] matanya: the font rendering is also 'off'. I don't know why [10:11:54] matanya: they have attempted to restyle infoboxes and such, but feels a bit meh. [10:12:20] PROBLEM - ElasticSearch health check on elastic1006 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3898: relocating_shards: 6: initializing_shards: 1: unassigned_shards: 0 [10:13:09] matanya: it's also a trademark violation. I'll usually be ok with letting it go, but there are ads on it... [10:13:20] RECOVERY - ElasticSearch health check on elastic1006 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3898: relocating_shards: 1: initializing_shards: 0: unassigned_shards: 0 [10:13:37] so poke legal [10:14:22] matanya: might. [10:15:46] matanya: done [10:17:30] matanya: I'm now super interested in having you try out the app I link I gave you :) [10:18:49] if you will still be aroud at approx 17:00 UTC you will be able to find some feedback [10:18:57] * YuviPanda does some math [10:19:04] matanya: yup, will definitely be around [10:19:31] ok, i'll look for you [10:19:34] matanya: thanks :) [10:19:46] matanya: I'll send you another build if I fix some CentralAuth issues in the meantime :) [10:19:54] great, thanks [10:20:35] matanya: :) [10:28:18] YuviPanda: you won't be happy :P [10:28:25] hmm? [10:28:27] i showed it to a fellow at work [10:28:39] 'it' -> ? [10:28:44] the app [10:28:59] oh, the build I sent you? [10:29:02] he said: "on english it is great, but on hebrew it is bad" [10:29:05] yes [10:29:21] matanya: heh :D I *am* happy, because we have people who care and who can tell me what's wrong and then I can fix it :) [10:29:46] i'll through his comments at you [10:29:56] and you decide what to do with it, fair? [10:30:06] matanya: sure! [10:30:12] but we should probably move to some -mobile channel [10:30:16] sure [10:30:18] #wikimedia-mobile [10:30:19] is there one? [10:33:50] PROBLEM - ElasticSearch health check on elastic1004 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1341: active_shards: 3898: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [10:34:50] RECOVERY - ElasticSearch health check on elastic1004 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 11: number_of_data_nodes: 11: active_primary_shards: 1342: active_shards: 3901: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [11:01:20] (03PS1) 10Patchon: Adding new configuration option "listenLocalOnly" [operations/debs/lucene-search-2] - 10https://gerrit.wikimedia.org/r/112871 [11:05:18] (03PS1) 10Matanya: base: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112872 [11:12:15] akosiaris: is realm a top scope var that can be used in .erb templates as @realm ? [11:12:56] matanya: exactly [11:13:20] but do avoid referencing it deep inside modules please [11:13:29] oh, a big new batch of commits coming then :) [11:13:37] it more like for role classes [11:13:48] no, only fixing exsiting ones [11:14:11] can't see scope.lookupvar all over the place [11:18:55] (03PS1) 10Matanya: search: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112876 [11:52:01] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [12:13:25] paravoid: regarding hk --> commons issue, have a sec? [12:32:43] (03PS1) 10Matanya: nginx: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112881 [12:44:26] gone for a little while (lunch out) [12:46:03] (03PS1) 10Matanya: torrus: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112884 [12:52:14] (03PS1) 10Matanya: swoft: remove lookupvar and replace with fact @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112885 [13:02:01] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [13:07:19] (03CR) 10MaxSem: [C: 04-1] "lsearchd is curently in a (very) maintenance mode. Third-party users are not recommended to use it, instead they're encouraged to try the " (033 comments) [operations/debs/lucene-search-2] - 10https://gerrit.wikimedia.org/r/112871 (owner: 10Patchon) [13:09:58] (03CR) 10MaxSem: "s/connect to/bind to/" [operations/debs/lucene-search-2] - 10https://gerrit.wikimedia.org/r/112871 (owner: 10Patchon) [13:19:00] (03PS1) 10Matanya: misc: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112887 [13:38:33] (03PS1) 10Matanya: ganglia: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112888 [13:44:10] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [13:45:45] (03CR) 10Patchon: ">> lsearchd is curently in a (very) maintenance mode. Third-party users are not recommended to use it, instead they're encouraged to try t" [operations/debs/lucene-search-2] - 10https://gerrit.wikimedia.org/r/112871 (owner: 10Patchon) [13:45:51] (03PS1) 10Matanya: ganglia_new: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112889 [13:50:25] (03PS1) 10Matanya: solr: remove lookupvar and replace with top scope @ var [operations/puppet] - 10https://gerrit.wikimedia.org/r/112892 [14:12:19] apergos: i have over 30 patches pending review, mind reduce that a bit? [14:17:34] (03CR) 10Matanya: (WIP) Completely overhaul admins.pp & modularize (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/107848 (owner: 10Faidon Liambotis) [14:20:08] matanya: how old is the oldest? [14:20:48] about month i guess [14:21:13] jan 13 [14:21:27] a month tommorow [14:31:25] not too bad [14:42:21] Nemo_bis: i don't how that is not too bad, but meh [14:42:51] *don't see [14:46:11] https://www.mediawiki.org/wiki/Gerrit/Reports/Oldest_open_changesets [14:46:47] You do hard and needed work so ops review your stuff quickly, but most are not so lucky. ;) [14:53:23] (03CR) 10Ottomata: [C: 031] "Oh, I totally read that patchset wrong. This looks good to go." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112613 (owner: 10Dzahn) [14:57:26] matanya: for simple changes such as the lookupvar/@ thing, you can just submit a single patchset that touches multiple modules [14:57:48] thanks paravoid didn't know that [14:58:07] (03CR) 10ArielGlenn: "actually this is already handled; sahar was already in admins.pp, and if he gets stat1002 access he doesn't need stat1... he also got adde" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112613 (owner: 10Dzahn) [14:59:17] !log just finished in place (Cirrus) reindex of all group1 wikis except commons [14:59:25] Logged the message, Master [15:05:12] (03PS1) 10Faidon Liambotis: Switch Hong Kong, Macao & Bhutan to ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/112897 [15:06:39] (03CR) 10Faidon Liambotis: [C: 032] Switch Hong Kong, Macao & Bhutan to ulsfo [operations/dns] - 10https://gerrit.wikimedia.org/r/112897 (owner: 10Faidon Liambotis) [15:11:26] Ops, do you have opinions on how to handle bounces in MediaWiki? https://bugzilla.wikimedia.org/show_bug.cgi?id=46640#c4 [15:11:45] Or should this be asked on the ops list? :) [15:17:10] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [15:46:08] (03CR) 10Ottomata: "Looks good, only one suggestion (and an idea!)" (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/112855 (owner: 10Ryan Lane) [15:59:44] apergos: do you deal with media storage wierdnesses? [15:59:59] a little of it [16:00:03] what's up? [16:00:10] compare https://he.wikipedia.org/wiki/%D7%A6%D7%94%D7%9C%D7%94 [16:00:20] with file page: https://he.wikipedia.org/wiki/%D7%A7%D7%95%D7%91%D7%A5:Zahala.jpg [16:00:32] with the full lengh file: https://upload.wikimedia.org/wikipedia/he/7/75/Zahala.jpg [16:01:35] what should I be seeing? [16:01:49] wrong rotationg? 90* [16:02:33] they are all vertical except for the oldest thumb for me [16:02:52] I am logged in viewing them [16:03:35] ok, so anon complaing it wrong, caching issue? [16:04:39] yes, and maybe a purge of the image would get it [16:05:12] (03CR) 10Faidon Liambotis: "Why aren't you doing this on the MediaWiki side?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/112805 (owner: 10Yurik) [16:06:40] (03CR) 10Yurik: "Because many responses (especially API) are not varied on X-CS." [operations/puppet] - 10https://gerrit.wikimedia.org/r/112805 (owner: 10Yurik) [16:19:51] (03CR) 10Chad: "I see no harm in merging this if it helps someone in the short term, once the style nits are taken care of that Max pointed out." [operations/debs/lucene-search-2] - 10https://gerrit.wikimedia.org/r/112871 (owner: 10Patchon) [16:20:38] <^d> MaxSem: I'll probably merge ^ if he amends to fix your style complaints. It's basically no big deal. [16:21:09] no objctions from me [16:21:33] I just had to warn him that we don't care [16:21:59] thanks [16:22:08] -1 was only on stylistics grounds:) [16:22:53] <^d> MaxSem: Yeah, we don't care, but it's harmless :) [16:43:20] RECOVERY - Host labnet1001 is UP: PING OK - Packet loss = 0%, RTA = 1.04 ms [17:14:33] !log stopping hadoop daemons on an12, oracle java 6 is running there, don't know why yet, apt says it is not installed….grrr [17:14:41] Logged the message, Master [17:14:43] !log demon synchronized php-1.23wmf12/extensions/CirrusSearch 'Performance fixes' [17:14:52] Logged the message, Master [17:15:14] !log demon synchronized php-1.23wmf13/extensions/CirrusSearch 'Performance fixes' [17:15:22] Logged the message, Master [17:17:40] PROBLEM - Hadoop DataNode on analytics1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [17:17:50] PROBLEM - Hadoop NodeManager on analytics1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:20:50] ottomata is on it: ^^ [17:21:04] (03PS1) 10ArielGlenn: Add shell account for santosh, admins restricted + stats1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/112912 [17:22:16] (03PS3) 10Se4598: Replace easter egg by a more explaining message [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110926 (owner: 10Hashar) [17:23:30] (03PS4) 10Se4598: Replace easter egg by a more explaining message [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/110926 (owner: 10Hashar) [17:25:40] RECOVERY - Hadoop DataNode on analytics1012 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.datanode.DataNode [17:25:50] RECOVERY - Hadoop NodeManager on analytics1012 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [17:25:54] ok [17:26:08] !log purged oracle java 6 from analytics1012 and restarted hadoop daemons [17:26:16] Logged the message, Master [17:26:21] (03PS2) 10Nemo bis: Add shell account for santosh, admins restricted + stats1002 [operations/puppet] - 10https://gerrit.wikimedia.org/r/112912 (owner: 10ArielGlenn) [17:30:36] hey Ryan_Lane, when I run git deploy start [17:30:40] should that create a .deploy file? [17:33:48] yo jgage, yt? [17:45:47] !log Updated scholaships.wm.o to fbffb96; l10n updates [17:45:55] Logged the message, Master [17:46:01] greg-g: Done [17:46:10] bd808: yay [17:50:11] (03PS1) 10Dzahn: lower TTL for bugzilla, upcoming switch [operations/dns] - 10https://gerrit.wikimedia.org/r/112915 [17:50:24] greg-g: fyi: i'm not igniring your email re deployment calendar - been a crazy week - sorry [17:52:26] (03CR) 10Dzahn: "in case you're wondering why, see f.e. http://techtalk.virendrachandak.com/migrating-servers-using-dns-ttl-for-minimum-downtime/" [operations/dns] - 10https://gerrit.wikimedia.org/r/112915 (owner: 10Dzahn) [17:53:05] Lydia_WMDE: no worries :) [17:53:06] thaks [17:53:07] +n [18:14:24] manybubbles: elastic1007 is fixed [18:14:44] cmjohnson1: sweet! [18:15:00] I'll get it back in the loop! [18:16:47]