[00:00:04] Respected human, time to deploy Holiday (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150119T0000). Please do the needful. [00:00:39] Please deploy that holiday to Germany [00:06:12] haha [00:34:06] heh https://github.com/mattofak/jouncebot/issues/1 [00:40:45] andrewbogott: hey! for some reason my client doesn’t do nick autocomplete in _security :) [00:40:48] YuviPanda: this could be keystone and not ldap at all, right? [00:41:01] andrewbogott: does keystone interact with MW?!! [00:41:22] sure, auth involves getting a keystone token. That's how OSM works [00:42:45] andrewbogott: ah, right. yes, could be theat [00:42:46] *that [00:42:51] YuviPanda: did I fix it? [00:42:56] andrewbogott: didn’t realize *MW* login had anything to do with keystone [00:42:58] andrewbogott: checking [00:42:59] !log restarted keystone service on virt1000 [00:43:07] Logged the message, Master [00:43:36] YuviPanda: I think the only alternative would be for mw to cache the login/password and reuse it when you hit an OSM page… seems right the way it is. [00:43:43] But, yeah, super different from a normal mw install. [00:43:54] right. hadn’t realized how different :) [00:43:58] andrewbogott: yup, works now [00:44:30] Keystone was running before but with an empty log. Lemme dig and see if it had any last words before dying [00:44:41] (although it sounds like pretty much everything was dying a few hours ago) [00:44:49] yeah, probably swapped out [00:45:34] yeah, nothing in the logs, just silence. [00:46:09] hmm right [00:47:10] ok, so, the only remaining (immediate) issue is: do we start puppet? [00:47:21] andrewbogott: I’d say let’s wait for paravoid? I suppose that’s him [00:47:45] Yeah, I agree. [00:47:53] andrewbogott: I think I’ll add wikitech monitoring to the infrastructure setup on shinken. [00:48:00] yes please! [00:48:36] doing now [00:52:42] YuviPanda: my pizza is just about to arrive so I'm going to turn my attention there :) Will keep an eye on email; let me know if you turn up other wikitech breakages. [00:53:18] (03PS1) 10Yuvipanda: shinken: Add wikitech check for labs infra [puppet] - 10https://gerrit.wikimedia.org/r/185788 [00:53:20] andrewbogott: :) ok. I’m in SF now [00:53:24] andrewbogott: without mobile internet. [00:53:29] andrewbogott: so will be less available than usual [00:53:51] Then take some time to enjoy the city away from your laptop :) [00:54:05] I will see you tomorrow or Tuesday AM! [00:54:34] (03PS2) 10Andrew Bogott: shinken: Add wikitech check for labs infra [puppet] - 10https://gerrit.wikimedia.org/r/185788 (owner: 10Yuvipanda) [00:54:46] (03CR) 10Andrew Bogott: [C: 031] shinken: Add wikitech check for labs infra [puppet] - 10https://gerrit.wikimedia.org/r/185788 (owner: 10Yuvipanda) [01:54:14] paravoid: here yet? [02:06:20] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Puppet has 1 failures [02:12:33] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:12:37] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-19 02:12:37+00:00 [02:12:42] Logged the message, Master [02:12:47] Logged the message, Master [02:20:37] !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 01s) [02:20:40] !log LocalisationUpdate completed (1.25wmf15) at 2015-01-19 02:20:40+00:00 [02:20:42] Logged the message, Master [02:20:45] Logged the message, Master [02:24:09] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [03:46:24] Hello. A user mailed in to OTRS saying his ISP blocks HTTP (not HTTPS) traffic, and he would like Wikimedia to automatically redirect all users from that range to HTTPS. Who do I ask them to contact about it, or who do I forward the ticket to? [03:46:59] Is https@wikimedia.org regularly checked, i.e. is it ok to ask them to forward their email to this address? [03:47:52] svetlana: if his isp blocks http traffic, wouldn't our redirect attempt get blocked as well? [03:48:37] Hm. Probably. (I think they type 'en.wikipedia.org' into a browser and think that the choice is made server-side.) [03:49:35] yeah, i can't think of a way that that would work [03:50:01] svetlana: the EFF provides a browser extension that works with Firefox, Chrome and Opera and that makes many sites known to be available on HTTPS default to HTTPS [03:50:06] it's available here: https://www.eff.org/https-everywhere [03:51:25] Okay, I'll explain them that this is done client-side then, and refer them to this extension. [03:51:40] tangential question: is our really-long-term plan still to enable HSTS on all WMF sites? [03:51:41] the ruleset it ships with includes *.wikimedia.org and *.wikipedia.org and the other major project TLDs [03:53:13] jackmcbarn: I hope not, but I don't know [03:53:21] you hope not? why not? [03:53:37] Thanks ori jackmcbarn. [03:56:16] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jan 19 03:56:15 UTC 2015 (duration 56m 14s) [03:56:24] Logged the message, Master [03:56:54] jackmcbarn: HTTPS is slower, harder to implement, and not universally available [03:58:21] HSTS wouldn't affect users that couldn't use HTTPS at all though, since they'd need to have it work successfully once for it to get set [04:00:36] jackmcbarn: well, they could use HTTPS at one time for HTTPS to be set, but that doesn't guarantee that it is available to them on their current network [05:43:28] jackmcbarn: never mind, that's a stupid reason. I read a little bit more about HSTS and yes, it makes sense to enable it. [06:28:39] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:10] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:10] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:29] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:40] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:29] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:45:59] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:59] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:19] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:46:20] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:51:29] PROBLEM - puppet last run on mw1195 is CRITICAL: CRITICAL: Puppet has 1 failures [07:08:10] RECOVERY - puppet last run on mw1195 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [08:44:41] (03CR) 10Amire80: [C: 031] Enable VisualEditor on 'Draft' (118) namespace at hewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185449 (https://phabricator.wikimedia.org/T87027) (owner: 10Glaisher) [09:10:40] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [09:27:29] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [10:17:40] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [10:26:30] (03CR) 10Yuvipanda: [C: 032] shinken: Add wikitech check for labs infra [puppet] - 10https://gerrit.wikimedia.org/r/185788 (owner: 10Yuvipanda) [10:30:59] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [10:33:09] (03PS1) 10Yuvipanda: shinken: Check wikitech on https rather than http [puppet] - 10https://gerrit.wikimedia.org/r/185816 [10:33:16] (03CR) 10jenkins-bot: [V: 04-1] shinken: Check wikitech on https rather than http [puppet] - 10https://gerrit.wikimedia.org/r/185816 (owner: 10Yuvipanda) [10:33:27] (03PS2) 10Yuvipanda: shinken: Check wikitech on https rather than http [puppet] - 10https://gerrit.wikimedia.org/r/185816 [10:33:37] (03CR) 10Yuvipanda: [C: 032] shinken: Check wikitech on https rather than http [puppet] - 10https://gerrit.wikimedia.org/r/185816 (owner: 10Yuvipanda) [13:38:39] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [500.0] [13:38:39] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 1 failures [13:51:50] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [13:56:39] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [14:33:13] (03CR) 10Jgreen: "There's background info in phabricator T86209" [puppet] - 10https://gerrit.wikimedia.org/r/185472 (owner: 10Jgreen) [14:56:03] (03PS1) 10Springle: repool db1054 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185831 [14:56:29] (03CR) 10Springle: [C: 032] repool db1054 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185831 (owner: 10Springle) [14:57:47] !log springle Synchronized wmf-config/db-eqiad.php: repool db1054, warm up (duration: 00m 05s) [14:57:55] Logged the message, Master [15:20:20] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00666666666667 [15:30:39] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [16:13:02] it seems all the wikidata json dumps mysteriously disappeared https://dumps.wikimedia.org/other/wikidata/ any ideas? [16:21:54] !log springle Synchronized wmf-config/db-eqiad.php: depool db1060 (duration: 00m 05s) [16:22:00] Logged the message, Master [16:24:09] never mind found the problem, working on patch [17:05:11] (03PS1) 10JanZerebecki: Do not delete the Wikidata dump we just created. [puppet] - 10https://gerrit.wikimedia.org/r/185840 [17:07:36] (03CR) 10Alexandros Kosiaris: [C: 032] Do not delete the Wikidata dump we just created. [puppet] - 10https://gerrit.wikimedia.org/r/185840 (owner: 10JanZerebecki) [17:13:30] akosiaris: could you start the cron job ahead of shedule so there is a dump? 'wikidatajson-dump' or /usr/local/bin/dumpwikidatajson.sh as user datasets on host snapshot1003.eqiad.wmnet [17:22:30] jzerebecki: yeah I just did. We would have to wait for like a week otherwise :-( [17:22:48] I see it runs on Mondays 03:15 UTC [17:24:10] akosiaris: thx :) [17:27:40] !log manually running wikidatajsondump.sh in a screen on datasets1003 after https://gerrit.wikimedia.org/r/185840 was merged [17:27:45] Logged the message, Master [17:34:44] (03PS3) 10Alexandros Kosiaris: parsoid: Include base::firewall on parsoid hosts [puppet] - 10https://gerrit.wikimedia.org/r/185610 (https://phabricator.wikimedia.org/T87105) (owner: 10Yuvipanda) [17:35:10] akosiaris: yay [17:36:23] YuviPanda: I 'll +2 and merge either today or tomorrow. But for sure after breakfast [17:36:59] :) [18:17:09] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:50] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59701 bytes in 2.068 second response time [19:16:29] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet last ran 1 day ago [19:17:39] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [19:23:54] !log disable puppet on wtp* hosts for https://gerrit.wikimedia.org/r/#/c/185610/ merge [19:24:00] Logged the message, Master [19:24:11] (03CR) 10Alexandros Kosiaris: [C: 032] parsoid: Include base::firewall on parsoid hosts [puppet] - 10https://gerrit.wikimedia.org/r/185610 (https://phabricator.wikimedia.org/T87105) (owner: 10Yuvipanda) [19:28:49] PROBLEM - Disk space on dataset1001 is CRITICAL: DISK CRITICAL - free space: /data 1520639 MB (3% inode=99%): [19:28:49] PROBLEM - puppet last run on wtp1022 is CRITICAL: Timeout while attempting connection [19:28:50] PROBLEM - puppet last run on wtp1005 is CRITICAL: Timeout while attempting connection [19:29:09] PROBLEM - puppet last run on wtp1007 is CRITICAL: Timeout while attempting connection [19:29:50] RECOVERY - puppet last run on wtp1022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:29:59] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:10] RECOVERY - puppet last run on wtp1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:30:52] !log https://gerrit.wikimedia.org/r/185610 merged, tested on wtp1024, wtp1023, caused 0 problems, rolling out to the rest of parsoid machines [19:30:57] Logged the message, Master [19:40:08] !log Set email of commons user Tatobot to the email of the owning account [19:40:15] Logged the message, Master [19:40:56] greg-g: let’s do it here [19:41:13] fine. [19:41:14] :) [19:41:20] greg-g: so we definitely don’t want bc to be cross-cluster this quarter [19:41:24] so that’s a relief to hear :) [19:41:36] yeah, I'm surprised by the idea myself :) [19:41:39] greg-g: :) [19:41:49] well, maybe *I* just misunderstood. [19:41:58] * greg-g shrugs [19:42:05] greg-g: anyway, if we do it in codfw, it is greenfield by definition [19:42:12] there's been a lot of talk lately with various ideas [19:42:21] true. lots of intense conversations :) [19:42:24] right [19:42:56] greg-g: _joe_ and mark and others are gonna be in the same room shortly. I’ll poke around. [19:43:09] kk, have fun [19:43:23] YuviPanda: I won't be there tomorrow (in office) but I'll be there wed-fri [19:43:34] but, feel free to corner antoine/mukunda/dan :) [19:44:05] YuviPanda: not sure if you saw https://etherpad.wikimedia.org/p/relengallhandsconversations, whic is basic notes on what we (releng) want to talk about while we can see people's faces [19:44:28] greg-g: yeah, I added things! [19:44:33] see === Potential BetaCluster target === [19:44:33] awesome! [19:44:35] that is all me [19:44:48] YuviPanda: behave [19:44:53] ah, sweet [19:45:04] * YuviPanda stabs Reedy [19:45:13] woah, I’m the only one with a loudish keyboard! [19:45:17] * YuviPanda types incassently [19:45:46] greg-g: I won’t be in the office tomorow either. Ops offsite [19:53:28] (03PS1) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [19:53:34] (03CR) 10jenkins-bot: [V: 04-1] Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 (owner: 10Yuvipanda) [19:53:35] man, gerrit is so much faster here [19:53:43] (03PS2) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [19:53:48] (03CR) 10jenkins-bot: [V: 04-1] Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 (owner: 10Yuvipanda) [19:54:42] (03PS3) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [19:55:50] mark: see backscroll with greg-g. he doesn’t paraticularly want betacluster in codfw, although we could use that as a ‘from scratch’ starting point. definitely no cross-DC functionality tho [19:56:43] hm... [19:56:46] (03PS4) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [19:59:32] (03PS2) 10Giuseppe Lavagetto: parsoid: use hiera, role [puppet] - 10https://gerrit.wikimedia.org/r/184925 [20:01:06] greg-g: mark agrees! no cross-dc replication! [20:04:10] 3Beta-Cluster, operations: Either add base::firewall to parsoid hosts or remove them from beta - https://phabricator.wikimedia.org/T87105#985741 (10akosiaris) 5Open>3Resolved Option 1 was chosen and https://gerrit.wikimedia.org/r/185610 by @Yuvipanda was merged some minutes ago. Everything seems fine so reso... [20:08:14] _joe_ wanna see a really funky puppet error? [20:08:22] Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol [20:08:49] funky in the sense it’s been around for months now and I’ve no idea what it means [20:09:50] bd808: hiiiii! when are you here? [20:10:13] YuviPanda: tomorrow afternoon [20:10:23] 3Beta-Cluster: Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 - https://phabricator.wikimedia.org/T87197#985752 (10yuvipanda) 3NEW [20:10:30] bd808: HFCS coke is here! SOO GOOD [20:11:23] 3Beta-Cluster, operations: Puppet failures on deployment-pdf01 Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol - https://phabricator.wikimedia.org/T75506#985759 (10akosiaris) [20:11:24] 3Beta-Cluster: Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 - https://phabricator.wikimedia.org/T87197#985760 (10akosiaris) [20:12:37] YuviPanda: :) glad you like our highly refined poisons. [20:13:54] (03PS2) 10Giuseppe Lavagetto: restbase: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184926 [20:13:56] (03PS2) 10Giuseppe Lavagetto: lvs: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184927 [20:13:58] (03PS3) 10Giuseppe Lavagetto: parsoid: use hiera, role [puppet] - 10https://gerrit.wikimedia.org/r/184925 [20:14:00] (03PS2) 10Giuseppe Lavagetto: virt: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/185153 [20:14:02] (03PS2) 10Giuseppe Lavagetto: redis: user role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184929 [20:14:04] (03PS2) 10Giuseppe Lavagetto: snapshot: unify node declarations, use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/185152 [20:14:06] (03PS2) 10Giuseppe Lavagetto: memcached: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184928 [20:14:08] (03PS2) 10Giuseppe Lavagetto: services: create sca role [puppet] - 10https://gerrit.wikimedia.org/r/185154 [20:16:21] 3Spam-Spam: Testing - https://phabricator.wikimedia.org/T86746#985769 (10yuvipanda) 5Open>3declined a:3yuvipanda [20:28:05] Hello. I can't seem to connect to any WIkipedia site nor Wikimedia Commons. I can however connect to Toollabs, facebook, irccloud.com... is there some sort of problem on my side or some server dead in Europe or something? I have tried rebooting my router. [20:29:36] could you give us your IP? [20:29:55] you can use a private message to me if you want [20:30:07] paravoid: I believe it is 80.217.42.136 [20:30:07] also a traceroute, if you know how to run one of those [20:30:21] I have no idea what a traceroute is.... [20:30:35] What OS are you on? [20:30:41] ChromeOS [20:30:43] that's fine, no traceroute needed [20:31:49] your ISP is having issues [20:32:01] does anyone know how the ‘cluster’ salt grain is set? [20:32:44] 3Beta-Cluster, operations: Set 'cluster' salt grain appropriately for all instances in beta cluster - https://phabricator.wikimedia.org/T87199#985792 (10yuvipanda) 3NEW [20:33:59] (03CR) 10Giuseppe Lavagetto: [C: 032] parsoid: use hiera, role [puppet] - 10https://gerrit.wikimedia.org/r/184925 (owner: 10Giuseppe Lavagetto) [20:34:08] paravoid: Is this a traceroute? http://i.imgur.com/UHdyw9L.png [20:34:09] PROBLEM - puppet last run on analytics1003 is CRITICAL: CRITICAL: Puppet last ran 3 days ago [20:34:26] paravoid: You sure it's my ISP? [20:34:36] yes [20:35:04] they are having trouble with their Amsterdam interconnection [20:37:16] paravoid: How do you know? [20:39:48] He'll have looked at traffic routes back to your IP address [20:40:02] Josve05a: http://www.chromestory.com/2011/04/how-to-run-tracert-in-chrome-os-devices/ [20:40:28] Josve05a: For reference, toollabs doesn't go to amsterdam, so hence that being ok [20:40:52] ah. That's why I'm having trouble w/ twitter as well [20:41:12] Possibly [20:41:22] If you can, go shout at your ISP [20:41:36] at 21:41 PM? :P [20:41:45] yes, why not [20:41:51] Some ISPs have somove lovely 24/7 indian tech support centres [20:41:52] ANy "specific things" I can say? [20:41:53] * Reedy coughs [20:42:13] issues with their AMS-IX connection [20:42:21] that's the Amsterdam Internet Exchange [20:42:43] Reedy: That link worked :) Now I know that :P http://i.imgur.com/MqRM6gi.png [20:44:02] Reedy: behave [20:45:49] they know...automated responce saying that it is affecting the whole Sweden by the way that we can't acces specific websites. They are currently working on it. (since it is closed since 21 PM) [20:46:10] 21 PM? [20:46:15] 9 PM * [20:46:22] 21:00 normal time :P [20:47:21] seems a fairly big outage then [20:48:06] Greeat. NO Wikipedia for me.... I'm supposed to help out students doing their school assignments on Wikipedia (campus volunteer....greeeeaaat). Any way I can bypass the Amsterdam connection? [20:48:28] Yeah [20:48:33] * Josve05a has no idea how internet works :P [20:48:34] You need to hack your hosts file [20:48:39] But that's messy [20:48:41] I can't do that... :/ [20:48:51] toor? [20:48:54] The school has "locked" this crappy Chromebook... [20:48:58] s/oo/o/ [20:49:07] well, that would require many hosts entries [20:49:22] and my W8.1 is...W8.1, no need to say more [20:50:04] springle: what’s the port range on db1069 again? [20:50:06] * YuviPanda keeps forgetting [20:50:14] 3301 to 7? [20:51:00] oh found it [20:51:07] 3311 - 17 [20:52:59] (03PS1) 10Alexandros Kosiaris: monitoring::service: mimic monitoring::host group handling [puppet] - 10https://gerrit.wikimedia.org/r/185854 [20:54:55] RECOVERY - puppet last run on analytics1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [21:00:01] (03CR) 10Giuseppe Lavagetto: [C: 032] monitoring::service: mimic monitoring::host group handling [puppet] - 10https://gerrit.wikimedia.org/r/185854 (owner: 10Alexandros Kosiaris) [21:00:03] (03PS2) 10Alexandros Kosiaris: monitoring::service: mimic monitoring::host group handling [puppet] - 10https://gerrit.wikimedia.org/r/185854 [21:03:37] (03PS1) 10Alexandros Kosiaris: Followup commit to 0333eab [puppet] - 10https://gerrit.wikimedia.org/r/185889 [21:04:25] (03CR) 10Giuseppe Lavagetto: [C: 032] Followup commit to 0333eab [puppet] - 10https://gerrit.wikimedia.org/r/185889 (owner: 10Alexandros Kosiaris) [21:11:14] (03PS1) 10Giuseppe Lavagetto: base::resolving: get rid of the global domain_search variable [puppet] - 10https://gerrit.wikimedia.org/r/185912 [21:21:12] (03PS5) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [21:21:14] (03PS1) 10Yuvipanda: Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 [21:21:18] (03CR) 10jenkins-bot: [V: 04-1] Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 (owner: 10Yuvipanda) [21:23:10] 3Beta-Cluster: Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 - https://phabricator.wikimedia.org/T87197#985848 (10akosiaris) Issue does not seem to be present on pdf02 that has the exact same set of classes. I even verified /var/lib/puppet/state/classes.txt -... [21:23:19] (03PS2) 10Yuvipanda: Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 [21:23:24] (03CR) 10jenkins-bot: [V: 04-1] Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 (owner: 10Yuvipanda) [21:25:14] (03PS3) 10Yuvipanda: Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 [21:25:51] (03PS6) 10Yuvipanda: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 [21:25:53] (03PS4) 10Yuvipanda: Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 [21:31:23] (03CR) 10Yuvipanda: [C: 032] Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 (owner: 10Yuvipanda) [21:31:29] (03Merged) 10jenkins-bot: Support specifying portnames with hostnames [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185848 (owner: 10Yuvipanda) [21:31:40] (03CR) 10Yuvipanda: [C: 032] Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 (owner: 10Yuvipanda) [21:31:46] (03Merged) 10jenkins-bot: Allow public dbs to be ignored [software/labsdb-auditor] - 10https://gerrit.wikimedia.org/r/185914 (owner: 10Yuvipanda) [21:31:54] springle: I set the yaml file with that info in your homedir on terbium [21:32:00] a looot of tables in a lot of dbs [21:33:42] 3Beta-Cluster: Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 - https://phabricator.wikimedia.org/T87197#985849 (10yuvipanda) Why do we have two pdf servers anyway? [21:34:06] 3Beta-Cluster: Error: Sysctl::Parameters[wikimedia base]: Could not evaluate: can't dup Symbol on deployment-pdf01 - https://phabricator.wikimedia.org/T87197#985853 (10yuvipanda) (am inclined to agree, btw. Let's get rid of the pdf01 instance once @cscott chimes in) [21:39:58] springle: https://phabricator.wikimedia.org/P145 [21:42:08] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#985857 (10Reedy) 3NEW [21:42:19] YuviPanda: Got a second? [21:43:19] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#985864 (10Reedy) It's also on 1.25wmf2 (a bit out of date) [21:43:59] (03CR) 10Alexandros Kosiaris: "So, a meta role that includes roles. I am not sure on this one tbh." [puppet] - 10https://gerrit.wikimedia.org/r/185154 (owner: 10Giuseppe Lavagetto) [21:44:53] 3operations: wikitech-static is out of date - https://phabricator.wikimedia.org/T87202#985866 (10Reedy) p:5Triage>3Normal [21:50:17] find /var/log/wikidatadump/ -regex '.*dumpwikidatajson-[0-4]\..*' -delete [21:50:21] YuviPanda: ori ^ [21:50:30] on snapshot1003 as datasets or root [21:51:19] !log Updated Wikimania scholarships to 1f8d8f8 (disable PDO persistent connections; review display fixes) [21:51:25] Logged the message, Master [21:51:38] hoo: done [21:51:54] ori: Thanks :) [21:52:08] springle: https://phabricator.wikimedia.org/P146 [21:55:21] (03CR) 10Alexandros Kosiaris: [C: 032] "We talked with Giuseppe on the issue. I was sold on the idea and this is an intermediate step to a better design so +2" [puppet] - 10https://gerrit.wikimedia.org/r/185154 (owner: 10Giuseppe Lavagetto) [21:58:44] springle: https://phabricator.wikimedia.org/P222 [21:58:58] springle: that’s just all tables with names that are very delete-worthy [21:59:05] no cur, blob etc in there [21:59:20] thanks [22:02:42] YuviPanda: thanks for the wikitech stuffs and agree re: logging! [22:03:06] ori: yup! SAL is terrible, I think. [22:06:28] (03PS1) 10Yuvipanda: logstash: Experimental logstash irc logging on production logstash [puppet] - 10https://gerrit.wikimedia.org/r/185918 [22:06:29] ori: ^ perhaps, to start with? [22:06:32] bd808: ^ [22:07:08] logstash is currently overloaded [22:07:12] yeah [22:07:21] so no point merging it now, I think. [22:07:25] wasn’t there a patch to unoverload it? [22:07:27] badly, meltingly, crushingly overloaded [22:07:40] needs more RAM [22:07:48] https://gerrit.wikimedia.org/r/#/c/185482/ specifically? [22:07:48] there's a phab task that bd808 filed [22:08:08] hmm, we could temporarily use the lsearchd boxes but they don’t have warranty [22:08:26] they don't have enough disk [22:08:29] oh [22:08:30] hmm [22:08:31] right [22:08:47] have we started on procuerment? [22:08:50] bd808: link to ticket? [22:09:02] <3 YuviPanda [22:09:15] *procurement [22:09:21] https://phabricator.wikimedia.org/T87078 [22:09:23] YuviPanda: https://phabricator.wikimedia.org/T87078 [22:09:57] (03PS2) 10Yuvipanda: logstash: Experimental logstash irc logging on production logstash [puppet] - 10https://gerrit.wikimedia.org/r/185918 [22:10:20] ^ just applies this on the labs logstash, as an experiment [22:10:24] oh, so we just need more RAM? [22:10:26] not new machines? [22:10:32] heh, or both [22:10:45] long term -- https://phabricator.wikimedia.org/T84958 [22:10:55] (03PS1) 10Giuseppe Lavagetto: coredb::s1: use hiera, role [puppet] - 10https://gerrit.wikimedia.org/r/185921 [22:10:55] which we should figure out this week too [22:11:14] <_joe_> springle: I'd really like your feedback on this ^^ [22:11:29] I think 3 "search" class boxes to move the elasticsearch services too would take us a long way [22:11:32] (03CR) 10Ori.livneh: [C: 04-1] "Either hard-code the channel names in the role (as before), or parametrize the channels but pass them in site.pp. But it doesn't make sens" [puppet] - 10https://gerrit.wikimedia.org/r/185918 (owner: 10Yuvipanda) [22:11:44] <_joe_> I think it makes it easier to find all the parameters [22:12:03] the current logstash hosts are small misc servers (6 cores; 16G) that we added disk to a while ago [22:12:47] <_joe_> It kinda feel strange to interact non-via-irc :) [22:12:50] ori: hmm, you’re right. I’m too lazy to figure out how to do this properly (don’t want to setup a required param for a role class) [22:12:51] _joe_: inorite [22:13:15] YuviPanda: just hard-code, as before [22:13:23] oh are you guys already at CQ? [22:13:31] ori: we’re in the office [22:13:32] <_joe_> ori: yes [22:13:33] ori: sitting around a table [22:13:36] <_joe_> we're in office now [22:13:36] wattt [22:13:38] talking on IRC :P [22:13:39] i was near the office like an hour ago [22:13:41] with my son [22:13:44] damnit [22:13:46] ori: waaaaatttttt. [22:13:54] #lame [22:13:57] ya [22:13:58] and your son wants to get into WMF ops? [22:14:05] certainly [22:14:16] mark: he'd be better at unracking rather than racking [22:14:20] everyone wants to get in on it [22:14:38] i've been an ops-wannabe since before it was cool [22:15:20] heh [22:15:25] (03PS3) 10Yuvipanda: logstash: Experimental logstash irc logging on production logstash [puppet] - 10https://gerrit.wikimedia.org/r/185918 [22:16:09] ori: bd808 don’t know if we can actually get any RAM (or any DC work) for the next couple of weeks, though [22:16:10] :( [22:16:11] oh well [22:16:51] couple of weeks doesn't seem like the end of the world. i'd say let's wait with the logging until we have a setup that we feel good about. [22:17:13] yeah [22:17:19] it's been a year getting to this point. a few weeks won't change much [22:17:36] let me merge that patch anyway. shohuld be interesting to see what happens in deployment-prep [22:17:45] (03PS4) 10Yuvipanda: logstash: Experimental logstash irc logging on production logstash [puppet] - 10https://gerrit.wikimedia.org/r/185918 [22:18:37] YuviPanda: more fun with logstash if your bored -- https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/puppet+branch:production+topic:logstash,n,z [22:18:51] you're [22:18:57] you welcome. [22:18:57] no you [22:19:24] * bd808 can't spell and has horrible grammar and isn't too ashamed [22:20:28] bd808: yeah, looked. I don’t know enough about logstash to actually merge / review those. [22:21:30] (03CR) 10Yuvipanda: [C: 032] logstash: Experimental logstash irc logging on production logstash [puppet] - 10https://gerrit.wikimedia.org/r/185918 (owner: 10Yuvipanda) [22:24:07] bd808: hmm, have we looked at what logs are being generated and see if some are really spammy? [22:24:08] (03CR) 10Springle: [C: 031] "No issues with this idea, however note that coredb will go away soon in favour of mariadb::core." [puppet] - 10https://gerrit.wikimedia.org/r/185921 (owner: 10Giuseppe Lavagetto) [22:24:18] (03PS1) 10Reedy: Disable xff logs. Bloated and no one uses them [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 [22:24:39] bd808: like ^ [22:24:54] many/most/all are pretty spammy [22:25:06] we should spend some time killing some! [22:25:07] jobrunner was a big one last time I checked [22:25:42] hadoop seem to generate a lot of logs atm (3x the next biggest, ocg) [22:25:47] (03CR) 10Yuvipanda: [C: 031] Disable xff logs. Bloated and no one uses them [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 (owner: 10Reedy) [22:25:59] where are you guys checking all this in? [22:26:00] fluorine? [22:26:15] <_joe_> jobrunner now is not that spammy [22:26:38] 3MediaWiki-Core-Team, operations: Audit log sources and see if we can make them less spammy - https://phabricator.wikimedia.org/T87205#985909 (10yuvipanda) 3NEW [22:26:41] I filed ^ [22:27:43] (03PS3) 10Giuseppe Lavagetto: restbase: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184926 [22:27:50] 3Analytics, operations: Hadoop logs on logstash are being really spammy - https://phabricator.wikimedia.org/T87206#985916 (10yuvipanda) 3NEW [22:28:04] (03CR) 10Hoo man: "I occasionally used them in the past, when we had security bugs regarding "weird" ip addresses showing up or things like that. I haven't d" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/185923 (owner: 10Reedy) [22:28:07] (03CR) 10Giuseppe Lavagetto: [C: 032] restbase: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184926 (owner: 10Giuseppe Lavagetto) [22:28:15] hadoop is loud but jgage likes it that way to debug the occasional problem. The trouble with logs is that to get enough data to diagnose runtime problems you often end up with lots of "junk" when things are running fine [22:29:17] YuviPanda: btw, can you help me word an android app bug report? I figured you might know the precise widget name or whatever. When I log in via the android app, I start typing "ori.livneh", but after "Ori." it automatically inserts a space. Is there some way to annotate a text field to indicate that it does not contain prose? [22:29:18] The problem today is that we are running 2 JVM apps and a redis instance per host and the hosts only have 16G of RAM. Elasticsearch would really like ~48G just for itself. [22:29:39] ori: aha. yes, you want to disable autocorrect on the username text field. [22:29:54] yep I tend to agree bd808 they are mostly useful when things go wrong so it is hard to know in advance what you'll need [22:30:17] godog: that is my wife's approach to packing for travel [22:30:44] Is your kitchen sink well travelled? [22:30:48] and how [22:30:55] do we / can we configure logstash to drop older messages? [22:31:02] ageism [22:31:10] yeah it only keeps 30 days of data [22:31:21] 1 index per day; 30 indices [22:31:22] don't trust anyone over 30 [22:31:31] old guys are creepy [22:31:54] ori: hahaha light travel FTW [22:32:12] the only thing I own that doesn’t fit into one of my two bags is a chair [22:32:39] anyway, I still think we should put the lsearchd boxes on these for a while [22:32:41] godog: ^ [22:32:48] (03PS3) 10Giuseppe Lavagetto: lvs: use role, hiera [puppet] - 10https://gerrit.wikimedia.org/r/184927 [22:33:51] they aren’t even powered off, maybe? [22:34:03] If we can figure out how that would be fine with me. We could spread out over more hosts with the same replica count to reduce the disk per host needed [22:34:12] bd808: you should corner godog when you’re here and bribe him with M&Ms [22:35:05] Or we could just figure out the new hardware, order it and wait things out [22:35:32] or that, yeah. [22:35:48] we should also audit our log sources and kill some [22:35:55] or be more aggressive, maybe [22:36:08]