[00:29:48] (03PS13) 10GWicke: WIP: Add restbase role [puppet] - 10https://gerrit.wikimedia.org/r/171741 [00:31:38] (03PS12) 10GWicke: WIP: RESTBase puppet module [puppet] - 10https://gerrit.wikimedia.org/r/167213 [00:31:59] (03PS14) 10GWicke: WIP: Add restbase role [puppet] - 10https://gerrit.wikimedia.org/r/171741 [00:34:08] PROBLEM - puppet last run on search1015 is CRITICAL: CRITICAL: Puppet has 1 failures [00:36:01] gwicke: you should still change provider => 'init' to provider => 'debian' [00:41:43] ori: I don't really like the idea of starting the service before the puppet run [00:42:30] I know that puppet in its current setup doesn't do much for consistency either [00:42:46] but I'm hoping that we can switch away from trebuchet to fix that [00:43:09] that strikes me as very odd [00:43:26] why is it an issue to have the service start before a puppet run? [00:43:38] we don't want the service to start with old code [00:43:57] a box can be down & miss a few deploys [00:44:43] (03PS1) 10MZMcBride: Set $wgEnableCaptcha to false for test.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171998 [00:45:29] (03CR) 10Ori.livneh: [C: 032] "Let's try it." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171998 (owner: 10MZMcBride) [00:45:39] (03Merged) 10jenkins-bot: Set $wgEnableCaptcha to false for test.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171998 (owner: 10MZMcBride) [00:46:38] ori: in practice this doesn't help much either, so it's more a symbolic gesture to remind ourselves about the issue [00:46:53] gwicke: whatever floats your boat :) [00:48:21] ori: if you know of a way to actually ensure consistency, then that'd be great of coure [00:48:24] *course [00:49:38] starting the daemon with old code should not be catastrophic, that's a red flag imo. depending on how the service is architected, a node with stale code should simply not have work delegated to it, or other nodes should refuse to peer with it [00:50:51] sadly cassandra doesn't check versions of its clients [00:51:29] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 310 seconds [00:51:30] it's also really the job of the deploy system to deploy the right code [00:51:54] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 310 seconds [00:52:05] RECOVERY - puppet last run on search1015 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:52:08] fencing off outdated nodes can help to contain the problem, but it doesn't magically update the code [00:52:30] preventing init from launching the service doesn't magically update the code either [00:52:49] no, but a deploy system can update it before running init [00:53:05] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:53:26] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -1 seconds [01:21:57] bd808: ping [01:21:58] gwicke: what's up? [01:21:59] bd808: I was wondering if I need to add any config to the labs logstash in order to start sending logs to it [01:22:00] am working on the restbase puppetization [01:22:00] If you send in via an existing protocol (GELF I'd assume), it should "just work" [01:22:00] yeah, gelf [01:22:00] stole the host data from parsoid [01:22:00] deployment-logstash1.eqiad.wmflabs, port 12201 [01:22:00] yup. that sounds right [01:22:02] okay, must be something on my end then [01:22:02] thanks! [01:22:18] gwicke: Is your app deployed in the deployment-prep project? I bet our firewall rules don't allow access to the logstash server from outside the project. [01:22:41] hmm, good point -- it's in the services project [01:23:08] If you want to test things inside your own project that should be possible. I tried to write the roles so they could be used in multiple palces. [01:23:12] *places [01:23:49] I think I'll wait until the cassandra & restbase modules are merged before testing in deployment-prep [01:24:57] maybe trebuchet will work there as well [01:25:03] It should [01:25:13] we deploy several thigns with trebuchet in beta [01:25:24] k, that's promising [01:25:44] <^demon|away> trebuchet for java? [01:26:01] no, nodejs [01:26:08] <^demon|away> Ah ok, I was thinking the cassandra bit. [01:26:20] we are using packages for cassandra [01:26:31] elasticsearch uses trebuchet for the plugins [01:26:41] so it's possible [01:26:43] <^demon|away> And it's horribly non-worky in deployment prep :) [01:27:01] * gwicke sighs [01:27:17] <^demon|away> for jar stuff, i'm sure it's fine for other stuff. [01:27:41] Yeah, scap, kibana, ocg are all trebuchet deploys in beta [01:27:57] <^demon|away> we don't use kibana package? [01:29:18] nope [01:29:19] I'm a bit worried about updating to kibana4. They added a dumb ruby proxy server to it. [01:29:35] <^demon|away> "how can we make kibana better?" [01:29:35] (03PS13) 10GWicke: Initial RESTBase puppet module [puppet] - 10https://gerrit.wikimedia.org/r/167213 [01:29:40] <^demon|away> "more ruby, obviously" [01:30:14] v1 was php, v2 was ruby, v3 is all client side, v4 goes back partly to ruby [01:30:41] But right now just as a reverse proxy in front of elasticsearch [01:30:54] which we already do with apache [01:31:28] But I'm guessing their new closed source access controls will have some kind of kibana ruby plugin bit [01:31:31] (03PS15) 10GWicke: WIP: Add restbase role [puppet] - 10https://gerrit.wikimedia.org/r/171741 [01:32:02] (03PS16) 10GWicke: Add a simple restbase::labs role [puppet] - 10https://gerrit.wikimedia.org/r/171741 [01:32:10] <^demon|away> bd808: I haven't looked into it. I know there was a plugin that already attempted acl/security stuff. [01:32:51] <^demon|away> I wonder if they just took that as their base to start from, like how marvel is just kibana. [01:33:19] seems likely. [01:33:40] dinner bell just rang so I'm out. Have a good weekend everybody. [01:33:45] <^demon|away> adios. [01:34:18] bd808: have a great weekend! [01:40:55] ^demon|away: I was just wondering if it's possible to change the 'full name' in gerrit somehow [01:41:33] there is an input in https://gerrit.wikimedia.org/r/#/settings/contact, but it's greyed out & not editable [01:44:42] <^demon|away> not really, it's based on your cname in ldap. [01:44:46] <^demon|away> which also == your login name. [01:44:54] <^demon|away> they can differ, but historical reasons. [01:45:40] hmm, okay [01:46:50] thanks! [01:47:32] <^demon|away> yw. ok, i'm out, dinner time. [01:47:35] <^demon|away> have a good weekend [02:16:52] I have a request for the backend mobile beta who do i talk to about that ? [02:20:26] (03PS1) 10Dereckson: Remove Anexo namespace on pt.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172012 (https://bugzilla.wikimedia.org/73164) [02:21:35] (03CR) 10Dereckson: [C: 04-1] "ON HOLD. A bot moved some pages into the main namespace, but not all." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172012 (https://bugzilla.wikimedia.org/73164) (owner: 10Dereckson) [03:58:04] Dereckson: you can remove the Anexo namespace from ptwiki if the problem is the bot job, it left only 30 pages that must be verified and moved manually [04:06:14] PROBLEM - puppet last run on db2034 is CRITICAL: CRITICAL: Puppet has 1 failures [04:14:13] PROBLEM - RAID on nickel is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:24:13] RECOVERY - puppet last run on db2034 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:34:03] RECOVERY - RAID on nickel is OK: OK: Active: 3, Working: 3, Failed: 0, Spare: 0 [06:27:54] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:04] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:24] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:34] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:55] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:24] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:44] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:24] PROBLEM - puppet last run on db1047 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:24] PROBLEM - puppet last run on db1017 is CRITICAL: CRITICAL: Puppet has 2 failures [06:45:34] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:45:54] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:08] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:08] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:46:34] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:46:35] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [06:46:56] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:49:45] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:25] RECOVERY - puppet last run on db1047 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [07:02:36] RECOVERY - puppet last run on db1017 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [07:07:44] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [07:09:11] PROBLEM - Disk space on ms-be3003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:57:47] (03PS4) 10Yuvipanda: dataset: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170492 (owner: 10John F. Lewis) [07:59:58] (03CR) 10Yuvipanda: [C: 032] dataset: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170492 (owner: 10John F. Lewis) [08:00:37] (03CR) 10Yuvipanda: [V: 032] dataset: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/170492 (owner: 10John F. Lewis) [08:07:32] ok, jenkins back up [08:23:16] (03PS1) 10Ori.livneh: Add tests for pybal.util.ConfigDict [debs/pybal] - 10https://gerrit.wikimedia.org/r/172018 [08:35:25] (03PS1) 10Ori.livneh: Add .travis.yml file to enable automated tests on Travis CI [debs/pybal] - 10https://gerrit.wikimedia.org/r/172019 [09:13:20] * YuviPanda puts https://bugzilla.wikimedia.org/show_bug.cgi?id=73166 in _joe_'s backscroll, since he was the last person seen doing anything with ocg [09:34:46] <_joe_> YuviPanda: uh? [09:35:16] <_joe_> YuviPanda: try jeff green please [09:35:25] <_joe_> I firefighted some mess [09:35:30] ah ok [09:35:33] will do [09:35:43] <_joe_> :) [11:07:57] !log ran makelost+found on /srv/postgres on labsdb1007 to kill cronspam [11:08:01] Logged the message, Master [11:09:15] !log ran makelost+found on /srv/postgres on labsdb1004 to kill cronspam [11:09:18] Logged the message, Master [12:32:37] PROBLEM - Disk space on ms-be2011 is CRITICAL: DISK CRITICAL - /srv/swift-storage/sda1 is not accessible: Input/output error [12:32:46] PROBLEM - RAID on ms-be2011 is CRITICAL: CRITICAL: 1 failed LD(s) (Offline) [12:32:46] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 1 failures [13:01:39] RECOVERY - Disk space on ms-be2011 is OK: DISK OK [14:39:03] Some user is reporting that he's not seeing the option to enable HHVM. [14:39:06] Logged in [14:39:51] ori: ^ [14:40:12] Glaisher: Oh dear? :P [14:40:23] Not me [14:40:25] He's very likely asleep at this time [14:40:28] Which wiki? [14:40:35] enwiki [14:40:44] I just invited him tot his channel [14:41:01] Reedy: royh ^ [14:41:18] ohai o/ [14:42:29] royh: Do you see any beta features? [14:43:09] Reedy: Yes. https://en.wikipedia.org/wiki/Special:Preferences#mw-prefsection-betafeatures lists "New search", "Hovercards", "Compact personal bar", "VisualEditor", "VisualEditor language tool", "Compact language links" and "Other projects sidebar" [14:43:46] Weird it just doesn't show hhvm [14:44:53] Do you see it on othe wikis? [14:45:10] Reedy: Which ones should I check? Enabled on all? [14:45:34] Yeah, it's supposed to be opt in on all wikis [14:46:47] "For information about the settings on this page, see Help:Preferences. Your language setting British English is not recommended." [14:46:49] Isn't that nice [14:48:03] Reedy: I don't see it on meta or nowiki. [14:51:01] <_joe_> british english [14:51:11] <_joe_> isn't that like "english english"? [14:51:23] english for english people [14:51:54] Nor on wikidata. [14:52:36] There's nothing conditional about it at all [14:52:48] That's because many custom /en-gb interface messages are not in sync with the actual message on enwiki [14:52:54] Right [14:52:59] and was causing confusion [14:53:01] lol [14:53:11] But surely the same can be said if you use anything != en [14:53:30] yeah, you want to read enwiki, but you still want your interface in de [14:53:48] they don't have many custom messages for other languages [14:53:50] I think [14:54:01] Right [14:54:18] But the modifications similarly won't have been translated on enwiki [14:54:45] I'll probably get a lot of stick for this, but the editing interface should only be in English. We're making it too easy for people with atrocious English ability to write articles. [14:55:07] heh [14:55:33] Wikipedia, the Encyclopedia anyone can edit! (as long as your English skills are halfway decent) [14:56:25] (03CR) 10Glaisher: "I really think we should revert this. This was not discussed and Special:Random (linked on the sidebar) also now shows pages in Draft name" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [14:57:00] I don't think Jimmy or Larry thought editing Wikipedia would become a way for people to learn/teach themselves English [14:57:05] (03PS1) 10Manybubbles: Update elasticsearch plugins to fix regex issue [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/172032 [14:57:26] (03CR) 10Manybubbles: [C: 04-1] "Will +1 when I've tried it in beta." [software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/172032 (owner: 10Manybubbles) [14:57:29] I'm popping out for a bit. Highlight my nick and I'll get notified. :) [14:57:37] And thanks for the help thus far. [15:00:42] Reedy: can we get that drafts-are-content-too patch reverted today? [15:01:14] Has James_F|Away commented? [15:01:28] No. [15:01:31] "Sorry for not highlighting it to the team." [15:01:52] Helpful :/ [15:01:54] Caturday deploys are the EVIL! [15:01:58] It's already causing trouble for people at enwiki [15:02:24] hoo: Because you never deploy on the weekend... [15:02:25] ;D [15:02:34] Not this weekend at least [15:02:36] so far :D [15:05:55] (03CR) 10Chad: "Ouch, I had forgotten about the Special:Random implications too." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [15:14:13] (03CR) 10Bartosz Dziewoński: "Already on the VIllage Pump… https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Short_pages_and_Draft_article_space" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [15:14:20] (03CR) 10Bartosz Dziewoński: "Also https://en.wikipedia.org/wiki/Wikipedia_talk:Drafts#Draft_namespace_added_to_ContentNamespaces" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [15:14:26] fun ^ [15:14:35] James_F: ^ [15:14:48] (03CR) 10Bartosz Dziewoński: "Bug asking for revert: https://bugzilla.wikimedia.org/show_bug.cgi?id=73136" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [15:20:12] <^demon|away> Happy fun times. [15:20:14] <^demon|away> I sent James an e-mail. [15:20:16] <^demon|away> I'm going ahead and reverting. [15:20:20] <^demon|away> I'll ask forgiveness from him later :) [15:20:39] Please do! :) [15:20:49] "ASSUMING DAMAGE CONTROL" [15:22:01] <^demon|away> Revert patch incoming. It'll be a rather length commit msg :) [15:24:52] (03PS1) 10Chad: Revert "enwiki: Add Draft: namespace to wgContentNamespaces" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172034 [15:25:10] <^demon|away> It's a straight revert but somebody plz review & I'll deploy [15:26:35] (03CR) 10Hoo man: [C: 032] "Go for it" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172034 (owner: 10Chad) [15:26:37] (03CR) 10Glaisher: [C: 031] Revert "enwiki: Add Draft: namespace to wgContentNamespaces" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172034 (owner: 10Chad) [15:26:42] <^demon|away> hoo: Thanks [15:26:43] ^demon|away: there [15:29:13] <^demon|away> Figures, jenkins backed up. [15:29:27] (03CR) 10Chad: [V: 032] Revert "enwiki: Add Draft: namespace to wgContentNamespaces" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172034 (owner: 10Chad) [15:30:17] !log demon Synchronized wmf-config/InitialiseSettings.php: Gerrit I46b151ff: Reverting addition of Draft namespace to enwiki (duration: 00m 04s) [15:30:24] Logged the message, Master [15:38:51] <^demon|away> Ok, back to my weekend. [15:51:36] Glaisher, Reedy: new users don't have the beta feature at the moment, so we can randomly assign them either HHVM or Zend and see if it has an impact on whether they stick around and help out [15:51:50] PROBLEM - Apache HTTP on mw1192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:52:22] royh: Are you a new user? [15:52:24] interesting [15:52:34] Glaisher: Yes. [15:52:50] That explains it then, I guess. [15:53:19] yes, sorry about that :/ everyone will be on HHVM pretty soon (within a month or so) [15:53:44] you may already have it -- what's your username? [15:53:53] ori: Royhvaara. [15:54:50] ori: I would very much like to have it, if possible. [15:55:37] royh: you're not in the HHVM group, sorry :/ On the flip side, you'll probably get it in the next week or two, and you're nominally less likely to run into bugs in the interim. [15:57:06] <_joe_> ori: good night! [15:57:44] ori: I see. Thanks :) [15:57:45] _joe_: i wish, it's 8 AM and my three year old decided to jump on my face an hour ago [15:58:04] <_joe_> ori: eheh [15:58:17] <_joe_> that's actually a joy, admit it :) [15:58:43] yes, but the timing! [15:59:17] i'm going out to the park, then i'll see if i can uncover anything re: the issue bblack is investigating [15:59:30] have a good evening [15:59:59] RECOVERY - Apache HTTP on mw1192 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 2.229 second response time [16:00:20] royh: HHVM stuff aside, awesome to see new editors ramp up so quickly! hope you stick around [16:02:20] <^demon|away> ori: Beautiful day for a park :) [16:03:09] PROBLEM - Apache HTTP on mw1192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:06:00] RECOVERY - Apache HTTP on mw1192 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 400 bytes in 0.079 second response time [16:09:20] PROBLEM - Apache HTTP on mw1192 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:09:27] ori: Thanks :) [16:25:41] (03CR) 10MZMcBride: "I'd like to know why this was proposed, merged, and deployed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/171024 (owner: 10Jforrester) [16:38:52] (03CR) 10MZMcBride: "Thank you for this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172034 (owner: 10Chad) [18:31:56] (03PS1) 10Ottomata: Add .bash_aliases for mysql for otto [puppet] - 10https://gerrit.wikimedia.org/r/172036 [18:32:55] (03CR) 10Ottomata: [C: 032 V: 032] Add .bash_aliases for mysql for otto [puppet] - 10https://gerrit.wikimedia.org/r/172036 (owner: 10Ottomata) [18:35:50] (03PS1) 10Ottomata: Fix otto mysql aliases [puppet] - 10https://gerrit.wikimedia.org/r/172037 [18:35:59] (03CR) 10Ottomata: [C: 032 V: 032] Fix otto mysql aliases [puppet] - 10https://gerrit.wikimedia.org/r/172037 (owner: 10Ottomata) [20:01:23] !log Jenkins/Zuul appear stuck. Disconnect/Re-enable Gearman from Jenkins. [20:01:26] Logged the message, Master [20:17:47] !log Jenkins/Zuul was still stuck. Disconnected and relaunched slave agents on lanthanum and gallium. This fixed it (slaves in labs were fine). [20:17:52] Logged the message, Master [22:37:01] (03PS1) 10Ori.livneh: Add unit tests for `pybal.util.LogFile` [debs/pybal] - 10https://gerrit.wikimedia.org/r/172089