[00:01:07] Popups extension (Hovercards Beta Feature) is working on mediawiki.org. bd808 thanks so much for Ctrl+C. [00:02:14] problem: [20-Mar-2014 00:01:37] Fatal error: require_once() [function.require]: Failed opening required '/usr/local/apache/common-local/php-1.23wmf18/extensions/Popups/Popups.php' (include_path='/usr/local/apache/common-local/php-1.23wmf18/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/usr/local/apache/common-local/php-1.23wmf18:/usr/local/lib/php:/usr/share/php') at /usr/local/apache/common- [00:02:52] that was server mw1201, checking... [00:05:33] also fatals from other mw12NN servers, the file isn't there [00:06:15] but it's on e.g. mw1060. Help ?! [00:06:36] spagewmf: Does it fatal only on 1201, 1202, 1203, 1204? [00:06:41] Those servers were moved recently [00:06:55] Are they in /etc/dsh/group/mediawiki-installation [00:06:57] ? [00:07:34] They are, but … we had problems last week with servers that had moved [00:07:41] RoanKattouw: seems that way (and thanks!) FWIW I Ctrl+C'd my first scap after it stalled with 50 hosts to go [00:08:13] They were in subnets that weren't allowed by the rsync servers [00:10:01] bd808: so the sync-dir of wmf-config went out to them, but not the scap? Anyway, how best to get each one to sync-common [00:10:35] from SAL on 03-13 "8:46 bd808: mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 in mediawiki-installation dsh group on tin; not sure why they didn't get the scap request" [00:11:25] Same hosts again [00:12:48] spagewmf: I'll fix them [00:13:03] bd808 I love you [00:13:04] !log dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' [00:13:09] Logged the message, Master [00:14:52] bd808: I was about to `ssh mw1201` then `sync-common`, would that have been sufficient? [00:15:02] This is the exact same thing that happened a week ago [00:15:26] spagewmf: You'd still need to rebuild the cdbs. Same thing. but you've got a few other machines to then do aswell ;) [00:15:28] spagewmf: That plus running scap-rebuild-cdbs [00:15:52] dsh ftw [00:16:12] !log finished dsh -c -M -m mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 -- '/usr/local/bin/sync-common; /usr/local/bin/scap-rebuild-cdbs' [00:16:16] Logged the message, Master [00:16:24] spagewmf: Try your tests some more [00:18:00] To find the broken hosts I ran: `dsh -g mediawiki-installation -M -F 80 -- 'ls -ld /usr/local/apache/common-local/php-1.23wmf18/extensions/Popups'|grep -v -- drwxr-xr-x` [00:18:22] It was reporting the list of hosts in the !logs above. Now it's clean [00:21:11] !log scaps failed because "@ERROR: access denied to common from mw1202.eqiad.wmnet (10.64.48.34)" [00:21:15] Logged the message, Master [00:21:45] !log Subnet for row D still not in rsync server config? [00:21:51] Logged the message, Master [00:21:57] Reedy: Didn't you make a patch to fix that? [00:22:16] I believe it was merged [00:23:00] https://gerrit.wikimedia.org/r/#/c/118309/ [00:23:07] Add 10.64.48.0/22 to scap allowed hosts [00:23:31] bd808: I'm just tailing fatal.log and exception.log and it's looking good. [00:23:33] What's the rsync config on tin show? [00:23:48] The red states were winning :) http://tinyurl.com/n3twd8k [00:24:07] Reedy: "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/24 208.80.152.0/22 10.64.32.0/22 10.64.48.0/22" [00:24:57] Reedy: But on mw1010 it's "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22" [00:25:12] So the slaves didn't get the update [00:25:36] Ah [00:25:51] Wonder why puppet didn't update it [00:26:11] !log scap-slave rsync servers have "hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22"; missing new 10.64.48.0/22 subnet [00:26:16] Logged the message, Master [00:26:17] The last Puppet run was at Thu Mar 20 00:12:25 UTC 2014 (13 minutes ago). [00:26:59] Reedy: class misc::deployment::scap_primary. I bet there's a second list [00:27:02] Please don't say puppet doesn't manage the config on this hosts... [00:27:14] bd808: I didn't see any errors from scap, certainly nothing mentioning mw12XX hosts [00:27:40] spagewmf: I believe you. I didn't last week either. [00:27:51] Probably a bug in the new scap code [00:28:19] sure, no worries. [00:28:29] NFS is out of date.. but we don't rsync from there [00:28:30] manifests/nfs.pp: hosts_allow => ['10.0.0.0/16', '10.64.0.0/22', '10.64.16.0/24', '208.80.152.0/22', '10.64.32.0/22'], [00:28:50] Hah, no [00:28:52] That's a bug too [00:29:52] (03PS1) 10Reedy: Update misc::nfs-server::home::rsyncd hosts_allow [operations/puppet] - 10https://gerrit.wikimedia.org/r/119667 [00:29:54] mutante|away: ^^ Apache config won't be synced to all servers using rsync.. [00:30:49] * bd808 is still trying to remember how the scap-slaves get rsync config [00:31:38] is that replacing sync-apache? [00:31:47] class role::deployment::deployment_servers::production { [00:31:50] urgent? [00:32:02] Nope [00:32:15] Just I know you love the apaches so so much ;) [00:32:42] just wonders how come we never needed that before to sync apache [00:33:13] hmm, where's the other fix you made [00:33:30] (03PS1) 10Reedy: Update docroot_dir_allows in role::deployment::deployment_servers::production [operations/puppet] - 10https://gerrit.wikimedia.org/r/119668 [00:33:31] Reedy: That one is the role for tin that makes trebuchet work [00:33:55] Still out of date ;) [00:34:02] for sure [00:34:19] Why we don't just do 10/8 I don't know [00:34:37] Reedy: that looks like a network for elastic search to me,..ack? [00:34:57] Row D right? [00:35:15] checks one last thing [00:35:36] mw1202 is 10.64.48.34/255.255.252.0 [00:36:07] in DNS i see dbstore and elastic in there [00:36:20] in racktables i see mw1201-1203 [00:36:30] mw1201,mw1202,mw1203,mw1208,mw1209,mw1210 [00:36:37] that is row D D5 [00:36:46] those are the boxes that scap is breaking for [00:36:54] (03CR) 10Dzahn: [C: 032] "row D network" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119667 (owner: 10Reedy) [00:37:01] and Sam is finding other config that is missing that subnet [00:37:24] We still don't have the fix for the scap rsync slaves boxes I don't think [00:37:29] (03PS1) 10Reedy: Allow all apache ranges in copy-by-url-proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/119670 [00:38:30] Nope... [00:38:43] I can't see anywhere obvious where it's also set [00:39:15] * bd808 hopes they are in puppet somewhere [00:39:38] One way to easily test... [00:39:40] (03CR) 10Dzahn: "no 10.64.48.0/22 here?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119670 (owner: 10Reedy) [00:39:51] Get a root to change it on one host then force a puppet run [00:39:54] see if it's "corrected" [00:40:49] mw1070 definitely doesn't use the same sort of management for rsyncd as tin does [00:41:05] :( [00:41:09] which box do you want a puppet run on [00:41:37] we're just trying to confirm if the rsyncd conf is managed by puppet on the rsync proxy boxes [00:41:40] mutante|away: mw1070 after making some change to /etc/rsyncd.conf [00:41:54] (03PS2) 10Reedy: Allow all apache ranges in copy-by-url-proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/119670 [00:42:40] mw1010 and mw1070 are the boxes we're concerned about. They need to have "10.64.48.0/22" added to the hosts allow for rsyncd [00:43:08] (03CR) 10Dzahn: [C: 032] Allow all apache ranges in copy-by-url-proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/119670 (owner: 10Reedy) [00:43:18] class misc::deployment { [00:43:18] system::role { "misc::deployment": description => "Deployment host" } [00:43:18] include misc::deployment::scap_primary, [00:43:36] are you in a deployment window? [00:43:43] I though that was only for tin [00:43:57] mutante|away: No we are way past the deploy window [00:44:06] But it broke spagewmf's deploy [00:44:18] and we're trying to figure out how to fix it [00:44:19] bd808: well, we did not change rsyncd conf , did we [00:44:26] bd808: I think it is [00:44:28] tin is a Wikimedia Deployment host (misc::deployment). [00:44:35] mw1010 doesn't show that [00:44:50] which means it's not got that puppet class [00:44:51] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [00:44:58] mw1010 and mw1070 would need a different role [00:45:18] because the serve from a different root [00:45:46] why do you expect mw1010 to be a deployment host [00:45:50] They serve from /usr/local/apache/common-local instead of /a/common [00:45:54] because they are those slaves? [00:46:32] mutante|away: Yes. I don't think they should have the same conf as tin but puppet should be managing it [00:46:40] and I can't find where that's happening [00:47:04] and the /etc/rsyncd.conf doesn't have a puppet warning at the top [00:47:29] or have the same dir structure under /etc that the rsync class uses to manage the file [00:47:42] TimStarling: How did you setup the scap proxies? [00:47:52] (one way to find out) [00:48:17] appeal to authority is not a valid rhetorical device :) [00:48:45] (03CR) 10Dzahn: "ranges look good, yep, eqiad ranges (10.in-addr.arpa)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119670 (owner: 10Reedy) [00:49:31] hang on [00:51:01] modules/rsync/templates/rsync.default.erb:# This file is used instead of the default /etc/rsyncd.conf [00:51:04] ? [00:51:54] ### THIS FILE IS MANAGED BY PUPPET [00:51:54] ### puppet:///files/rsync/rsync.default [00:52:35] reedy@mw1070:~$ cat /etc/rsyncd.conf [00:52:35] hosts allow = 10.0.0.0/16 10.64.0.0/22 10.64.16.0/22 10.64.32.0/22 208.80.152.0/22 [00:52:43] We're trying to work out where those values come from [00:52:44] in https://gerrit.wikimedia.org/r/#/c/44526/ I added a misc::deployment::scap_proxy class and added the conf file to puppet [00:52:57] but that class is not invoked from anywhere [00:53:16] and I'm pretty sure I never did it at a later time [00:53:31] i see this in that template above 21 RSYNC_CONFIG_FILE=<%= rsync_conf %> [00:53:50] misc::deployment::scap_proxy isn't there any more [00:54:00] I think I just set them up manually and committed an incomplete puppet manifest with the idea of fixing it (or getting someone to fix it) later [00:54:41] Neither is files/rsync/rsyncd.conf.scap, which is where we would be amending the config in question [00:55:22] * bd808 is filing an rt ticket [00:55:25] https://github.com/wikimedia/operations-puppet/commits/production/files/rsync/rsyncd.conf.scap [00:55:39] andrewbogott removed it in Ia792a461778e726ffa2473e5224a8431e7e0e72a [00:55:57] (03PS1) 10Reedy: Revert "Remove misc::deployment::scap_proxy as it appears unused." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119676 [00:56:12] (03PS2) 10Reedy: Revert "Remove misc::deployment::scap_proxy as it appears unused." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119676 [00:56:34] * andrewbogott hopes that didn't cause any harm [00:57:06] andrewbogott You were right about it being unused [00:57:16] It's a case of it probably should've been used ;) [00:57:35] fortunately gerrit kept it safe and sound for us [00:58:25] Reedy, mutante|away, TimStarling, paravoid : https://rt.wikimedia.org/Ticket/Display.html?id=7080 [00:59:28] Jan 17 22:37:13 !log installed rsyncd on 6 apache servers for network-aware scap testing [00:59:43] Hmm. I wonder if it'd be better to copy and amend class misc::deployment::scap_primary { [00:59:48] Changing paths to match etc [01:02:43] bd808: might the mw12XX problem explain scap hanging with 50 hosts left? I don't have the set of ssh s that were in Sleep state [01:02:54] * bd808 looks [01:02:57] bd808: Do you have a list of the scap proxies to hand? [01:03:33] $ cat /etc/dsh/group/scap-proxies [01:03:33] # eqiad row A [01:03:33] mw1010.eqiad.wmnet [01:03:33] # eqiad row B [01:03:33] mw1070.eqiad.wmnet [01:03:59] The others were in pmtpa [01:04:06] and they are gone now [01:04:35] We need to add some more in eqiad I think but that could come later [01:04:48] I'd like to see one slave per row [01:04:58] s/row/subnet/ [01:05:49] "19:57 < bd808> I really need to get done playing in labs and get back to porting the rest of the scap scripts" sucks that's taking a while :/ [01:05:53] * greg-g is catching up [01:06:17] greg-g: https://rt.wikimedia.org/Ticket/Display.html?id=7080 [01:06:23] (03PS1) 10Reedy: Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 [01:06:41] (03Abandoned) 10Reedy: Revert "Remove misc::deployment::scap_proxy as it appears unused." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119676 (owner: 10Reedy) [01:07:28] (03PS2) 10Reedy: Manage scap proxy rsync config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 [01:07:39] I do love how some files are tab indented [01:07:41] and others space ;) [01:08:43] The coolest ones are the half tab, half space versions [01:11:22] I prefer 60/40 [01:11:41] cuz then you have a nagging feeling "oh, I should just fix the 40%" but never do it [01:12:05] My internal clock says 10.5hrs is long enough for today. Check with you guys tomorrow [01:12:45] later [01:12:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [01:13:26] bd808: thanks for the help with this deploy [01:16:21] RECOVERY - ElasticSearch health check on logstash1003 is OK: OK - elasticsearch (production-logstash-eqiad) is running. status: green: timed_out: false: number_of_nodes: 3: number_of_data_nodes: 3: active_primary_shards: 35: active_shards: 93: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [01:16:21] RECOVERY - ElasticSearch health check on logstash1001 is OK: OK - elasticsearch (production-logstash-eqiad) is running. status: green: timed_out: false: number_of_nodes: 3: number_of_data_nodes: 3: active_primary_shards: 35: active_shards: 93: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [01:19:59] With racktables being private, it'd be useful if we had a list of what rows which apaches are in on wikitech or similar [01:21:25] (RE: the comment about adding more scap proxies - as it stands you've gotta ask ops for server lists) [01:23:15] priyanka has an account on it [01:23:19] just ask her [01:24:26] ok so there's an ACL that omits priyanka [01:24:35] anyway, point stands, you should just have an account [01:26:52] !log gave Sam Reed access to racktables [01:27:03] Logged the message, Master [01:27:31] Thanks [01:28:06] I think Popups is OK, if not my phone is on officewiki. Thanks again [01:28:17] I sent you the password by encrypted email [01:28:55] you can change it after login [01:30:48] everybody hates mix of tabs/spaces, but also everybody hates the lint changes, but you can only hate one:) [01:32:25] mutante|away: well... https://gerrit.wikimedia.org/r/77131 ... [01:32:28] * hoo|away runs away [01:33:53] (03CR) 10Springle: [C: 032] Remove PMTPA remnants [operations/software] - 10https://gerrit.wikimedia.org/r/118939 (owner: 10Reedy) [01:33:58] hoo|away: i know i know. attempt 2 with some time in between, we're getting there https://gerrit.wikimedia.org/r/#/c/118794/ [01:34:26] (03PS6) 10Springle: Read local ganglia.xml if exists (useful for debugging) [operations/software] - 10https://gerrit.wikimedia.org/r/118947 (owner: 10Reedy) [01:34:56] yep... it might ahve been better to do WS only first and the rest later [01:35:01] this is almost impossible to review [01:35:13] (03CR) 10Springle: [C: 032] Read local ganglia.xml if exists (useful for debugging) [operations/software] - 10https://gerrit.wikimedia.org/r/118947 (owner: 10Reedy) [01:35:32] hoo|away: 4 people, 4 opinions, no way to do it right [01:35:37] also runs away for now [01:35:44] yeah... [01:35:46] good night [01:36:35] (03PS2) 10Springle: Fixup a few syntax and validator errors [operations/software] - 10https://gerrit.wikimedia.org/r/118954 (owner: 10Reedy) [01:36:58] (03CR) 10Springle: [C: 032] Fixup a few syntax and validator errors [operations/software] - 10https://gerrit.wikimedia.org/r/118954 (owner: 10Reedy) [01:37:48] (03PS4) 10Springle: Fix invalid css [operations/software] - 10https://gerrit.wikimedia.org/r/118950 (owner: 10Reedy) [01:38:04] (03CR) 10Springle: [C: 032] Fix invalid css [operations/software] - 10https://gerrit.wikimedia.org/r/118950 (owner: 10Reedy) [01:44:39] springle: Thanks! [01:45:07] looks like the js syntax error wasn't a js syntax error [01:45:35] scary dynamic js is scary [01:47:46] hmm [01:48:57] i'm just fixing it up with a bit of reordering/requoting [01:49:29] :) [01:50:35] / this is why i'm not a frontend developer. [01:50:35] / love, [01:50:35] / asher feldman, [01:50:58] heh that made me lol when i first read it... then cry a bit later ;) [01:55:22] (03PS1) 10Reedy: Fixup syntax error from Id866ac01c812d591d2 [operations/software] - 10https://gerrit.wikimedia.org/r/119684 [01:59:31] (03CR) 10Springle: [C: 032] Fixup syntax error from Id866ac01c812d591d2 [operations/software] - 10https://gerrit.wikimedia.org/r/119684 (owner: 10Reedy) [01:59:51] PROBLEM - ElasticSearch health check on elastic1003 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1712: active_shards: 5059: relocating_shards: 1: initializing_shards: 1: unassigned_shards: 0 [01:59:51] nice [02:00:51] RECOVERY - ElasticSearch health check on elastic1003 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1713: active_shards: 5060: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [02:01:09] "CRITICAL - elasticsearch (production-search-eqiad) is running." [02:07:24] (03PS1) 10Reedy: Stylize and function parameter and return type docblocks [operations/software] - 10https://gerrit.wikimedia.org/r/119685 [02:17:34] !log LocalisationUpdate completed (1.23wmf17) at 2014-03-20 02:17:34+00:00 [02:17:42] Logged the message, Master [02:32:10] !log LocalisationUpdate completed (1.23wmf18) at 2014-03-20 02:32:10+00:00 [02:32:10] (03PS1) 10Reedy: Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 [02:32:15] Logged the message, Master [02:39:11] PROBLEM - ElasticSearch health check on elastic1009 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1712: active_shards: 5059: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [02:39:11] PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1712: active_shards: 5059: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [02:41:11] RECOVERY - ElasticSearch health check on elastic1005 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1715: active_shards: 5062: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [02:41:12] RECOVERY - ElasticSearch health check on elastic1009 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 1715: active_shards: 5062: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [03:23:40] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Mar 20 03:23:37 UTC 2014 (duration 23m 35s) [03:23:46] Logged the message, Master [03:35:07] (03CR) 10Tim Landscheidt: Update misc::nfs-server::home::rsyncd hosts_allow (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119667 (owner: 10Reedy) [03:45:51] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [04:13:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [06:12:47] Coren: are you about? [06:13:39] sorry to have to bother you ... https://de.wikipedia.org/wiki/Benutzer_Diskussion:Lustiger_seth says that Seth is away for four weeks from 8 March [06:14:22] he has a search tool "searchsbl" which looks to need a kick of webservices [06:22:32] ack, that was a fail [06:46:51] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [07:14:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [08:42:31] RECOVERY - Check status of defined EventLogging jobs on vanadium is OK: OK: All defined EventLogging jobs are runnning. [08:53:50] hey springle, any idea why db1047 went down around midnight, march 18? [08:53:57] http://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&c=MySQL+eqiad&h=db1047.eqiad.wmnet&jr=&js=&v=202766&m=mysql_uptime&vl=secs&ti=mysql_uptime [08:54:10] is this the work of crazy researcher queries? [09:11:55] ori: yes, that was the root cause [09:13:16] i think the eventlogging database needs to run on separate hardware :/ [09:13:59] the separate slaves we spoke about via email are being setup now [09:14:03] so, soon [09:14:25] you're a saint, i'm mostly frustrated by analytics here, and myself [09:14:53] the job that writes events repeatedly failed to connect and hit upstart's respawn limit [09:15:10] :\ [09:15:15] this triggered an icinga alert, but no one responded to it [09:15:26] i got reports from both the multimedia team and the communications team that image performance metrics and blog visit data stopped being logged [09:15:53] no one responded because maintenance responsibilities are currently fuzzy [09:16:41] did that overlap with icinga-wm being off-channel? [09:16:57] nope, i found out about the alert by browsing the channel irc logs [09:18:21] eventlogging was my baby for a while, so i still jump in to fix things, as a matter of pride. it is a cardinal sin, of course -- you can never hand off maintenance like that. [09:18:54] do you recall why it went onto db1047 in the first place? [09:19:02] instead of m1 or m2 [09:20:53] i was fairly new when i requested it and the circumstances around me writing an analytics framework were a bit bizarre [09:21:13] so i think asher was probably not betting on longevity, or at least not holding his breath [09:22:14] so it got assigned to the query loony bin. i would have probably made the same call in his position :) [09:22:47] ok :) well, even if analytics leave db1047 alone, EL probably deserves to move if people notice when it's not working. [09:24:12] yeah, I think so. the daemon that decodes and validates incoming events sits on vanadium, which IIRC is not a database-class machine, in terms of specs [09:24:30] but the daemon itself could probably live on the same box as the database [09:25:14] this would diminish availability somewhat -- i should note that the log-file writers continued to process events while the db writer was down. [09:25:43] i'd prefer the deamon live separately and put the database onto m2. save having any database box with special config [09:25:56] gain HA with m2 master&slaves [09:26:15] i like the sound of that very much [09:26:46] vanadium still seems logical [09:26:48] is it fair for me to make this ask, or is it expected that ottomata handle analytics database machines? [09:27:00] i have no idea :) [09:27:20] i'll take whatever help i can get [09:27:59] so would you wipe and reprovision as a database machine, or simply add a role? [09:28:19] oh wait [09:28:33] i meant vanadium still seems like a good place for the daemon [09:28:39] oh right, separate [09:28:40] m2 will be the database [09:28:43] yep [09:28:59] keeping it there would certainly be convenient [09:29:07] (the daemon on vanadium) [09:31:04] so what would be the best way for me to proceed, do you think? i can mention it in the postmortem i'm about to send to the analytics list, and i can file an RT ticket, if that helps [09:31:26] or should i just sit tight? [09:31:38] 226G in log db. woo [09:31:54] RT ticket [09:33:37] well, the data comes in as json. i tried selling the analysts on a document store like mongo or couchedb, but the intellectual investment in SQL is too high; when people have a job to do they don't want to learn some map-reduce library [09:33:44] will need downtime for dump/import. plus folks who access db1047 for log will need fair warning [09:35:29] * ori nods [09:35:49] i'll start communicating / logging this [09:35:54] thanks [09:37:38] might be able to wean analysts off SQL with something like https://mariadb.com/kb/en/cassandra-storage-engine/ [09:38:07] it's not actually meant for analytics, but as a stepping stone, maybe [09:38:34] with the added temptation ofr users to go a whole lot faster if they bother to get away from sql [09:39:47] MongoDB is webscale! [09:40:03] :) [09:40:11] we're talking analytics-scale [09:41:15] you know, a research paper by an analyst at yahoo! mentioned in a footnote that an internal study at yahoo! revealed that the average dataset analysts were working on was something like ten megabytes in size [09:42:20] massive datasets are mostly used for basic counting; they're too noisy for anything much more involved [09:47:51] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [09:47:53] people will eventually realize that data doesn't become sentient at large volumes and they'll start to look for systems that emphasize curation and discovery rather than scalability [09:49:10] that's my wager, anyway. but i'm rambling and it's late [09:49:30] someone or something has to process the large noisy data down to something useful though, plus keep eveything around for when someone wants a slightly different 10mb :) [09:52:26] well, parallelism, concurrency, and capacity will still matter, of course. but the emphasis now is exaggerated [09:55:51] the Bigtable paper from 2006 shows Google mostly trying to move away from SQL because they wanted people to express computation in a way that could be easily parallelized [09:57:17] but with dremel and spanner they went back to SQL because they realized servers are cheaper than people [09:58:03] and making people write mapreduce jobs means transferring responsibility from query planners to humans [09:59:10] heh [09:59:58] some combination of the two is needed, which is why i'm interested in MariaDB engines that talk to nosql. Analytics have asked for both interfaces [10:03:43] yeah, the cassandra storage engine looks interesting [10:15:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [10:46:50] i get a lot of cache misses for meta, and the site is quite slow when editing [11:02:19] Pausing 21 seconds due to database server lag from pywikibot [11:03:05] something is going on [11:14:26] (03CR) 10Mark Bergsma: "Ideally we'd pull the allowed prefixes from network.pp, to prevent this from happening again. We will be adding more subnets often..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [11:43:43] (03CR) 10Ori.livneh: [C: 031] "LGTM. I added this to the SWAT deployment window for today (Thursday), 23:00 UTC." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [11:51:27] ori wears his SWAT suit all day [11:52:05] i like how it makes me feel super important [11:53:34] also the change is super-cool. have you seen this stuff from the multimedia team? like navigation timing, but for image loading, using the resource timing api: http://www.w3.org/TR/2011/WD-resource-timing-20110524/ [11:55:39] https://meta.wikimedia.org/wiki/Schema:MultimediaViewerNetworkPerformance [11:56:19] because MultimediaViewer uses ajax to fetch images, they are able to inspect response headers from javascript [11:56:46] so they're logging varnish cache hits / misses info along with the timing data [11:56:54] awesome [12:33:17] ori: around ? [12:48:51] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [13:16:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [13:49:21] (03CR) 10Reedy: "Seems a good idea. From looking around I found a few other places that were using the same set of ranges in different ways." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [14:06:40] (03PS1) 10Aude: Remove legacy settings for wikibase setup without the build [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119748 [14:21:07] (03PS1) 10Matanya: icinga: remove dvanliere from analytics contact group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 [14:22:28] (03CR) 10Dzahn: [C: 032] icinga: remove dvanliere from analytics contact group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 (owner: 10Matanya) [14:22:59] faster than the sun [14:23:01] (03PS1) 10Hashar: contint: override .jshintrc file on gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 [14:23:47] (03CR) 10Hashar: "Timo, that should fix the annoying issue." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [14:24:07] (03CR) 10Dzahn: "are you adding all the another new analytics people?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 (owner: 10Matanya) [14:24:24] (03PS1) 10Aude: Add Wikibase repoSiteName setting for client [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119751 [14:24:56] (03CR) 10Matanya: "in a separate patch. but i need there names for it." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 (owner: 10Matanya) [14:25:37] (03CR) 10Dzahn: contint: override .jshintrc file on gallium (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [14:26:40] (03CR) 10Dzahn: "yea and you need contacts who are in private repo, i'll make a patch" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 (owner: 10Matanya) [14:27:32] (03CR) 10Matanya: "thanks. i'll leave this to you." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119749 (owner: 10Matanya) [14:31:58] (03PS1) 10Dzahn: add members to analytics contact group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119753 [14:32:33] (03CR) 10Matanya: [C: 031] add members to analytics contact group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119753 (owner: 10Dzahn) [14:32:52] (03CR) 10Matanya: "the topic seems somewhat off" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119753 (owner: 10Dzahn) [14:33:13] (03CR) 10Dzahn: "need to create the contacts in private repo first, but the people can be confirmed already" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119753 (owner: 10Dzahn) [14:42:05] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 311 seconds [14:42:15] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 316 seconds [14:42:55] (03PS1) 10Ottomata: Enabling cgi scripts for stats.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 [14:44:14] (03PS2) 10Ottomata: Enabling cgi scripts for stats.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 [14:44:42] (03CR) 10Ottomata: "Alex, any reason I shouldn't do this?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 (owner: 10Ottomata) [14:47:05] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 121 seconds [14:47:15] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 121 seconds [14:50:18] (03CR) 10Dzahn: [C: 032] add members to analytics contact group [operations/puppet] - 10https://gerrit.wikimedia.org/r/119753 (owner: 10Dzahn) [14:56:57] !log restarted logstash on logstash1001.eqiad.wmnet [14:57:03] Logged the message, Master [14:58:27] MaxSem: I see we have one SWAT today, but the person who added it doesn't seem to be on IRC yet. But you did PS2 on the change, so if you want to be responsible I'd go for that.. [14:58:38] matanya: so halfak also wants to be on it.. but who else are the researchers would be the next question [14:58:54] * MaxSem looks [14:59:02] yes mutante i think a licket should be created [14:59:34] anomie, yep - I'll do it [14:59:38] mutante: DarTar, lzia and Ironholds [14:59:38] matanya: +1 [14:59:45] halfak: :) hi, thanks [14:59:57] *ticket [14:59:59] np I can get a list of email addresses if that is more helpful. [15:00:10] it is [15:00:37] Is there a ticket I could post on or should just email 'em to you? [15:00:44] !log logstash stopped ingesting logs at 2014-03-19T22:37:54.000Z. [15:00:49] Logged the message, Master [15:00:58] (03CR) 10MaxSem: [C: 032] Upload to non-CentralAuth wikis locally from mobile [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118328 (owner: 10Jdlrobson) [15:01:03] halfak: rt [15:01:06] (03Merged) 10jenkins-bot: Upload to non-CentralAuth wikis locally from mobile [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118328 (owner: 10Jdlrobson) [15:01:08] ops-requests [15:01:16] halfak: ideally what matanya said:) [15:01:56] Hmm... Are you guys asking me to create the ticket or asking me to search for it? [15:02:10] create [15:02:25] halfak: by sending the mail you create it automatically [15:02:26] just mail ops-request with the user/s wnat access [15:02:54] you don't need to hassle with the web ui unless you prefer it [15:03:00] !log maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/118328' [15:03:04] Logged the message, Master [15:03:07] I'm lacking a little bit of context to be able to write up a ticket. ping milimetric [15:03:22] done [15:03:48] mutante, I'll make sure the ticket gets filed. [15:06:16] halfak: perfect, will add the researchers [15:07:36] (03PS1) 10Dzahn: retab contactgroups.cfg - tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/119756 [15:08:07] (03CR) 10Matanya: [C: 031] "+2 if i could ..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119756 (owner: 10Dzahn) [15:08:53] (03PS2) 10Dzahn: retab contactgroups.cfg - tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/119756 [15:15:34] (03PS1) 10Reedy: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119758 [15:15:36] (03PS1) 10Reedy: Update Wikipedias to 1.23wmf18 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119759 [15:15:38] (03PS1) 10Reedy: group0 to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119760 [15:15:52] (03CR) 10Dzahn: [C: 032] retab contactgroups.cfg - tabs to spaces [operations/puppet] - 10https://gerrit.wikimedia.org/r/119756 (owner: 10Dzahn) [15:16:08] (03CR) 10Reedy: [C: 032] Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119758 (owner: 10Reedy) [15:16:15] (03Merged) 10jenkins-bot: Add symlinks [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119758 (owner: 10Reedy) [15:17:51] (03PS1) 10Dzahn: add more new analytics contacts to contactgroup [operations/puppet] - 10https://gerrit.wikimedia.org/r/119761 [15:18:03] (03CR) 10Hashar: contint: override .jshintrc file on gallium (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [15:18:10] (03PS2) 10Hashar: contint: override .jshintrc file on gallium [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 [15:18:21] (03CR) 10Hashar: "Pending Timo review" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [15:21:13] !log reedy Started scap: testwiki to 1.23wmf19 and build l10n cache [15:21:18] Logged the message, Master [15:23:01] it's a reedy! [15:23:15] Sam is back and ready to scap! [15:23:47] bd808: btw, I added your follow up to the 'deploy' outage/post-mortem last night [15:23:58] (03CR) 10Dzahn: [C: 032] add more new analytics contacts to contactgroup [operations/puppet] - 10https://gerrit.wikimedia.org/r/119761 (owner: 10Dzahn) [15:36:22] bd808: Reedy got my email? [15:36:45] !log reedy Finished scap: testwiki to 1.23wmf19 and build l10n cache (duration: 15m 32s) [15:36:49] Logged the message, Master [15:37:01] fast scap is fast [15:37:22] grrrr, need to backport my patch to wikimedia messages extension [15:39:14] https://gerrit.wikimedia.org/r/#/c/119764/ [15:40:21] greg-g: re: my question about incident "numbers", yes i was confused, somehow i thought we have numbers assigned to incidents similar to CVE's, as opposed to just dates, but no need [15:42:50] (03CR) 10Krinkle: [C: 04-1] "Per bug, this would mask the problem as an empty object essentially disables most checks and makes the job pass giving a false sense of se" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [15:44:44] (03CR) 10Krinkle: [C: 031] "whatev." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [15:46:56] mutante: yeah, no thanks [15:46:58] :) [15:48:10] (03PS1) 10Ottomata: Setting recurse_submodules => true on labs self hosted puppet clones [operations/puppet] - 10https://gerrit.wikimedia.org/r/119766 [15:48:54] ottomata: I'm looking at a labs project called 'mobile-sms' managed by you, diederik and erosen. [15:48:58] Any idea what that's about? [15:49:39] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [15:50:40] uhhhh [15:50:53] sounds like the old vumi project? dunno [15:50:59] i think you can delete it [15:51:14] ottomata: ok… who is erosen? Should we check with them first? [15:51:21] Krinkle: isn't .jshintrc with  {} applying the default settings ? [15:51:30] (03PS2) 10Dzahn: Revert "remove mw1163 from dsh groups" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119650 [15:51:32] Krinkle: i.e. the exact same as running jshint without a .jshintrc ? [15:51:34] evan rosen, no longer works with us [15:51:45] ok, then you're the last man standing! I'll mark it for death [15:51:48] thans [15:51:52] *thanks [15:52:04] yup [15:52:14] * mark thinks we should do these labs migrations every year, nice cleanups ;) [15:52:27] yeah! [15:52:30] :( [15:53:30] mark: sometimes I think each project should send a keepalive email every few months. Like, 'respond to this email if you don't want me to delete your project' [15:54:02] bd808: Reedy greg-g we might want https://gerrit.wikimedia.org/r/#/c/119751/ and https://gerrit.wikimedia.org/r/#/c/119764/ when we run localisation cache again on wmf19 [15:54:36] yes, i agree [15:54:41] then wikimedia messages submodule updated [15:54:42] i suggested something like that in the beginning too ;) [15:54:44] bd808: mw1163, you manually synced it, i'm putting it back in dsh and pybal.. right [15:54:49] (03CR) 10Hashar: Allow user to specify mount point for role::labs::lvm::mnt (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119524 (owner: 10BryanDavis) [15:55:02] aude: sure cc bd808 Reedy [15:55:11] mutante: Let me check it quickly [15:55:21] i suppose before mediawiki.org is put on wmf19 [15:55:42] just to make sure our new magic word is working well [15:56:01] * aude mumbles evil magic words [15:57:25] (03CR) 10Hashar: "> Per bug, this would mask the problem as an empty object essentially disables most checks and makes the job pass giving a false sense of " [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [15:57:49] mutante: It didn't get the scap that Sam just did. Did it get taken out of the dsh group? [15:58:58] * bd808 will update it manually again [15:59:18] bd808: that's what i wanted to do [15:59:28] bd808: https://gerrit.wikimedia.org/r/#/c/119650/2 [15:59:47] bd808: but it's also depooled, so making sure it's either both, or none [16:03:25] mutante: I did the sync and am running the l10n rebuild on mw1163 now, but we shouldn't repool until after it's back in the dsh group [16:03:42] It will get out of date immediately otherwise [16:05:27] !log Ran /usr/local/bin/sync-common && /usr/local/bin/scap-rebuild-cdbs on mw1163. Should not repool until it's back in the dsh group. Should me manually synced just before repooling. [16:05:32] Logged the message, Master [16:06:02] (03PS17) 10Alexandros Kosiaris: lint admins.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 (owner: 10Dzahn) [16:06:25] (03PS1) 10Hashar: Disable Zero configuration on beta cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 [16:07:25] (03PS3) 10Dzahn: Revert "remove mw1163 from dsh groups" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119650 [16:08:01] (03CR) 10Hashar: "The Zero .json files can not be fetched from noc.wikimedia.org causing puppet to yield on beta cluster with:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [16:08:24] aude: "when we run localisation cache again"? [16:08:45] Meaning we need another full scap? [16:08:46] bd808: i don't know when, but https://test.wikipedia.org/wiki/Main_Page is broken [16:09:01] gorram l10n [16:09:01] i don't know why it can't find the magic word, but then those are usually trouble [16:09:23] (03PS1) 10Andrew Bogott: Add git-review to toollabs bastion [operations/puppet] - 10https://gerrit.wikimedia.org/r/119771 [16:09:24] our patches make the contents of the magic word translatable, where as now it's hardcoded [16:09:42] e.g. "Wikidata" versus being able to write "wikidata" in hebrew etc. [16:09:55] (03PS2) 10Tim Landscheidt: Tools: Install packages git-review and joe [operations/puppet] - 10https://gerrit.wikimedia.org/r/118595 [16:10:06] (03CR) 10Dzahn: [C: 032] ""revert remove" = add :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119650 (owner: 10Dzahn) [16:10:11] hashar: No, not the same. [16:10:11] (and allow us to not hard code "wikidata" in the code and not say "Wikibase" in the UI) [16:10:53] bd808: ^ [16:10:54] pretty much every new magic word does this to test.wikidata [16:10:55] hashar: Also, there is no default. Providing no file or providing an empty file, means you have all restriction options disabled, and all relaxing options enabled. [16:10:57] (03CR) 10Hashar: [V: 031] "Tested on instance deployment-cache-mobile03.eqiad.wmflabs by cherry-picking the change on deployment-salt.eqiad.wmflabs in /var/lib/git" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [16:11:04] err test.wikipedia [16:11:36] (03PS3) 10Andrew Bogott: Tools: Install packages git-review and joe [operations/puppet] - 10https://gerrit.wikimedia.org/r/118595 (owner: 10Tim Landscheidt) [16:11:42] we found no better solution, however than the magic word in this case [16:12:02] (03Abandoned) 10Andrew Bogott: Add git-review to toollabs bastion [operations/puppet] - 10https://gerrit.wikimedia.org/r/119771 (owner: 10Andrew Bogott) [16:12:06] (03CR) 10Alexandros Kosiaris: [C: 032] "Ran it through differ and after a couple of rounds, found it to be a noop. Thanks for the lint Daniel!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 (owner: 10Dzahn) [16:12:48] (03CR) 10Dzahn: "hah, "joe" just keeps showing up every few months until it found the way in" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118595 (owner: 10Tim Landscheidt) [16:12:55] hashar: Running without config means it only does bare syntax checking and a few other things. Which can be good enough, and is better than a default config because defaults might change and break stuff. [16:15:02] (03CR) 10Andrew Bogott: [C: 032] Tools: Install packages git-review and joe [operations/puppet] - 10https://gerrit.wikimedia.org/r/118595 (owner: 10Tim Landscheidt) [16:17:39] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [16:18:00] <^d> Heh, joe's back? brion will be happy. [16:18:50] Just in toolabs! [16:19:06] Volunteers get to use whatever editor they want :) [16:19:31] yay, git-review! [16:20:16] <^d> yay and git-review don't belong in the same sentence aude :p [16:20:27] heh [16:22:00] be careful with "whatever editor" :) [16:22:09] mc [16:22:58] <^d> Can we install notepad? [16:23:43] (03CR) 10Alexandros Kosiaris: [C: 04-1] "Apart from the syntax error, CGI scripts are no longer (a long time now) the best approach for anything the is going to be heavily used an" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 (owner: 10Ottomata) [16:24:06] joe makes old guys who spent a lot of time with turbo pascal happy :) [16:25:26] and there it goes:) cya [16:25:29] (03CR) 10Ottomata: "Erik wants to use this for a very minimal search feature on stats.wikimedia.org, which is just a bunch of html files." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 (owner: 10Ottomata) [16:25:46] aude: so, you have those updates needed for wikidata anyways, and there's l10n breakage... should we just redo the initial sacp to testwiki to get it sorted now? [16:25:53] I mean, that's my leaning :) [16:26:13] * andrewbogott is one of the world's leading experts on running notepad on linux. [16:26:16] No kidding! But we'd need X [16:26:33] greg-g: sure [16:26:54] <^d> andrewbogott: One of? There's more? [16:26:57] I have no idea when Reedy will be back... [16:26:59] <^d> Do you guys have a conference? [16:27:15] magic words always cause an issue [16:27:24] ^d: https://en.wikipedia.org/wiki/Wine_%28software%29 [16:27:31] They have a room at fosdem [16:28:27] https://gerrit.wikimedia.org/r/#/c/119764/ is more trivial and then i can update the mediawiki core submodule [16:28:37] bah, whatever, I'm going to save some of bd808's sanity and just wait for Reedy to return [16:28:43] ok :) [16:28:49] <^d> andrewbogott: "In a 2007 survey by desktoplinux.com of 38,500 Linux desktop users, 31.5% of respondents reported using Wine to run Windows applications." [16:28:50] not a hurry [16:28:52] * greg-g nods [16:29:11] aude: I'll try to remind sam about your patches when he's back online [16:29:16] ij [16:29:17] Ah, 2007, the year of the linux desktop [16:29:18] gah, ok [16:29:25] aude: :) off by one [16:29:38] Or at least that's what they told us in 2006 [16:30:06] ^d: "How to run Gerrit on Windows " :) stackoverflow:) [16:30:37] alright, I'm migrating to a coffee shop, I have no calls today, going to take advantage of not being tied to a office-type place [16:30:44] ok [16:30:49] mutante: I'm pretty sure you can run cygwin on wine, and then install wine in cygwin, etc. [16:30:52] <^d> mutante: Hmm, it's all in the jvm so it shouldn't be hard actually. [16:31:02] <^d> (In theory, never tried) [16:32:46] andrewbogott: fun recursion is recursive. but i can confirm SSH runs fine in cygwin, so i could SSH into the windows box [16:33:10] ^d: true, Java.. platform independent etc.. in theory [16:33:45] <^d> heh, http://scn.sap.com/docs/DOC-42271 [16:37:38] Ohai [16:37:44] <^d> It's a Reedy! [16:37:45] Sorry, other distractions taking time atm [16:37:58] no hurry [16:38:32] * Reedy cleans up testwiki till window [16:38:35] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki back to 1.23wmf18 till window [16:38:41] Logged the message, Master [16:38:42] Right [16:38:50] What else is on the prep TODO list? [16:38:54] mark: labs.wikimedia.org ipv6/ipv4 stuff is no more, right? [16:39:04] mutante: Can/will you force a puppet run on tin to get https://gerrit.wikimedia.org/r/#/c/119650/ applied there? [16:39:11] Found them on nl.wiki and gag.wiki still [16:39:54] Reedy: https://gerrit.wikimedia.org/r/#/c/119764/ and https://gerrit.wikimedia.org/r/#/c/119751/ (and it's dependency) for us [16:40:10] i can update submodules (need to update hte wikidata submodule also) [16:40:53] then make sure the magic word is good [16:46:50] Krinkle: no [16:47:34] mark: Removed it from nlwiki, gagwiki and tpiwiki :) [16:47:42] thanks [16:47:44] Thanks to /bin/mwgrep [16:47:56] Love that tool [16:48:28] you mean /usr/local/bin/mwgrep? ;) [16:48:41] Yeah [16:48:51] ah, case sensitive, found another one when matching for ipv6 instead of IPv6 [16:48:54] https://sc.wikipedia.org/wiki/MediaWiki:Common.js [16:49:12] How did it even get there... [16:56:31] (this is a bumping channel this morning) [16:58:36] !log repooling mw1163 (it's back in dsh as well) [16:58:41] Logged the message, Master [16:59:42] andre__: i wonder if we can just do this license switch https://gerrit.wikimedia.org/r/#/c/119726/1/Bugzilla/BugMail.pm [17:00:02] mutante: thank you (re 1163) [17:00:25] it looks surprising, but if upstream uses MPL 2, and those are contributors from Mozilla.. i guess .. [17:00:53] greg-g: welcome [17:00:53] has it only just gone back in dsh? [17:01:07] aude: Sure, if you could update the submodules that'd be great [17:01:16] Netscape (C) 1998 :p [17:01:23] doing [17:01:35] Reedy: it's complicated ;) [17:01:43] Reedy: bd808 did a manual sync after your last scap [17:01:52] then i added back in dsh, then back in pybal [17:02:04] great [17:02:08] that's what I was going to ask :) [17:02:11] :) [17:02:24] I just re-ran the sync to be sure. It should match the rest of the clsuter [17:02:47] cool [17:04:32] greg-g: https://gerrit.wikimedia.org/r/#/c/119762/ [17:04:38] "@reedy Can the Wikimedia setup handle JSON i18n without B/C already? The same was merged for MobileFrontend yesterday, but I just realised that the way Wikimedia builds the cdb files is different..." [17:04:51] MobileFrontend could be rather broken in 1.23wmf19 [17:05:41] So a revert might be needed... [17:05:49] And some more prep/dev work [17:05:51] I'm really not sure [17:06:24] !log Reloading Zuul to deploy Ie800ed90b51c47d5a1 [17:06:30] Logged the message, Master [17:07:24] bd808: AaronSchulz ^^ re i18n json file merge https://gerrit.wikimedia.org/r/#/c/119762/ [17:07:46] In theory we should be able to test this on beta... [17:08:17] bryan's working on it... [17:08:39] but, you know, non-ops doing ops-like things is tough [17:08:40] Aha [17:08:41] I'm messing wtih beta.eqiad [17:09:01] bd808: not to say you aren't capable :P just, slower than people who do it every day [17:09:11] Ha [17:10:21] Is speed legal over there? [17:11:31] Reedy: I think it is legally *required* to work on wallstreet [17:11:42] sweet [17:11:49] I dunno about the json i18n files. I *think* that if LocalisationCacheBulkLoad() can read the files everything should be fine [17:11:51] not so much non-profits [17:12:14] git clone https://git.wikimedia.org/summary/labs/tools/grrrit.git [17:12:28] Cloning into 'grrrit' .. fatal: https://git.wikimedia.org/summary/labs/tools/grrrit.git/info/refs not valid: is this a git repository? [17:12:40] /summary [17:12:45] right:) thx [17:13:02] ssh://gerrit.wikimedia.org:29418/labs/tools/grrrit.git [17:13:12] waiting for jenkins to merge wikidata updates [17:13:16] then can update submodules [17:13:21] Reedy: http://idahomethproject.org/ says I'm supposed to stay away from drugs [17:15:06] bd808: it's a good idea, unless you want to spend multiple tens of thousands of dollars on new teeth. [17:15:30] * greg-g had a roommate/buddy who was an addict for too long, no longer is, but he's paying for it in multiple ways [17:16:07] (03PS1) 10Dr0ptp4kt: WIP: DO NOT MERGE YET. Split up tagging for baselining period. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119781 [17:17:50] (03CR) 10Dr0ptp4kt: [C: 04-1] "-1'ing to emphasize that this is a WIP." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119781 (owner: 10Dr0ptp4kt) [17:22:20] (03PS1) 10RobH: Adding cobalt to public1-b-eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/119782 [17:22:57] (03CR) 10MaxSem: "Instead of killing a way to test some aspects of Zero, why not create a labs-specific version of these resources?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [17:23:34] idles in #mediawiki-feed where all the bots are but no people:) [17:25:10] bblack, you around? [17:25:41] (03CR) 10RobH: [C: 032] Adding cobalt to public1-b-eqiad [operations/dns] - 10https://gerrit.wikimedia.org/r/119782 (owner: 10RobH) [17:26:41] there is one bot there [17:26:57] err, 2 bots :) [17:27:32] aude: but it outputs just a few repos, i just looked at the config of grrrit-wm , that's why [17:27:39] ah [17:27:57] and the bugzilla bot is there as well [17:28:04] * aude nods [17:41:53] !log reedy synchronized php-1.23wmf19 'Update Wikidata and WikimediaMessages' [17:41:58] Logged the message, Master [17:45:40] (03PS1) 10Faidon Liambotis: varnish: add X-Range to upload for Flash workaround [operations/puppet] - 10https://gerrit.wikimedia.org/r/119786 [17:56:38] * aude back in ~30 min [17:56:48] * greg-g makes note ;) [17:58:46] bd808: can you please explian to me the relation between gdash and graphite for monitoring stuff using check_graphite ? [17:59:13] matanya: Ummm… no :) [17:59:30] andre__: meeting ? [17:59:39] I think the relationship is that both gdash and check_graphite read data from graphite [18:00:07] bd808: i'm refering to https://gerrit.wikimedia.org/r/#/c/118435/1/templates/icinga/checkcommands.cfg.erb [18:00:17] jgage might know ? [18:00:38] matanya: gdash would just be an interface to show a subset of things on graphite that can be public.. afaik [18:01:09] so i didn't understand how i query it [18:01:13] matanya: Ah. So you need the right graphite function to get a data set [18:01:21] hang on... [18:01:55] Blerg, but you're not in wmf group so you can't see the explorer UI [18:02:39] * greg-g finds that thread [18:04:03] matanya: Does gdash have a graph with that title or were you just describing the requirement? [18:04:48] bd808: there is such a graph, but as said in the patch, it was a wild guess [18:05:50] i'm very famailir with graphite UI, but can't guess the stored info and graphs without access [18:06:00] * bd808 nods [18:06:22] so my patch is mostly useless [18:07:06] You can find some of the info via the image urls in gdash, but yeah you really need access to the schema browser to find the right query to monitor [18:07:22] yup [18:07:44] bug greg-g to bug "Legal" to get the NDA users access to this stuff :/ [18:07:51] :( :( [18:07:56] sad but true [18:08:08] (03PS1) 10Ori.livneh: Krinkle: access to LogStash cluster [operations/puppet] - 10https://gerrit.wikimedia.org/r/119790 [18:08:24] bd808: heh, why is "legal" in scare-quotes? [18:08:38] i'm pretty sure they passed the bar and everything [18:08:42] not sure it is 100% NDA issue, more like ldap group setting, isn't it? [18:09:11] Because it has been used as a stall tactic on this issue in my opinion [18:09:26] matanya: Sort of. [18:09:50] we need a volunteer ou, and add the NDA people to it [18:10:03] and grant some access to stuff to that group [18:10:06] * bd808 nods [18:10:22] it will most likely solve the WMDE issues too [18:10:25] Now we just need a way to know who has signed an NDA [18:10:37] and have that access monitored approriately [18:10:44] i'm sure there is a list somewhere [18:10:44] and there's the gorram problem [18:10:54] You'd think so wouldn't you? [18:11:05] lets say i hope [18:11:29] meh, I'll give m-ark more than 8 days to schedule, have meeting, and report back on his meeting with legal [18:11:39] * greg-g just looked at thread on the topic [18:12:00] matanya: there's no one canonical place... yet [18:12:04] (03CR) 10BryanDavis: [C: 031] "Bring on the co-maintainers!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119790 (owner: 10Ori.livneh) [18:12:54] csteipp: did you here about operation windigo ? [18:13:14] matanya: Nope? [18:13:25] greg-g: i'm sure if you could create the awsome incident page, you can fix this [18:13:34] csteipp: http://www.welivesecurity.com/wp-content/uploads/2014/03/operation_windigo.pdf [18:13:55] matanya: I would, I have ideas, i've shared them, but I have no power in this situation [18:14:05] matanya: i really really really want to jfdi [18:14:36] a quick review of our infrastructure to verify we aren't infected would be a god idea csteipp [18:14:48] greg-g: no rush, at least on my side [18:14:57] *good [18:16:04] (hear), csteipp if you aren't the one to poke, sorry [18:16:09] (03PS1) 10Alexandros Kosiaris: Fix parsoid sudo rules [operations/puppet] - 10https://gerrit.wikimedia.org/r/119793 [18:20:52] (03CR) 10Alexandros Kosiaris: [C: 032] Fix parsoid sudo rules [operations/puppet] - 10https://gerrit.wikimedia.org/r/119793 (owner: 10Alexandros Kosiaris) [18:21:16] Reedy, hi, when would be a good time to bug you with the new wiki? thx :) [18:22:37] wait for him to get on a boat [18:23:38] YuviPanda, Reedy is on a boat? [18:23:43] dr0ptp4kt: somewhat :) [18:23:45] sometimes. [18:23:58] bblack, ha! [18:26:33] dr0ptp4kt: those PowerApp + XFB checks look somewhat suspicious. XFB is set from an explicit list of forwarders, would we not be checking for a specific one? [18:28:19] greg-g: so while i think we need legal to say that somebody _has_ an NDA, if we already know they have one and access to other systems that need NDA.. then.. what else is there to ask them [18:28:47] (03CR) 10BBlack: "We'll probably have to, perhaps as a static set of files somewhere that are fetchable, but that the Zero team can update (with perhaps a f" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [18:29:16] mutante: legal, probably nothing. but then then "we" (someone) need to figure out the right groupings/levels of access [18:29:47] greg-g: gotcha [18:30:12] eg https://www.mediawiki.org/wiki/User:Greg_%28WMF%29/Scratch/NDA_access_groups [18:33:58] greg-g: "RT account" -> "privileged RT account".. adjusted [18:34:34] matanya: (now that I was finally able to open that) I'm the wrong person to ask. It would be someone in ops, although that pdf didn't give much to go on for intrusion characteristics. [18:34:37] back [18:35:59] csteipp: thanks, there is an appendix for that. but i'll poke one of ops [18:36:13] aude: and Reedy should be getting back soon-ish, if the 45 minute estimate holds. [18:36:36] (03CR) 10BBlack: "I've created some new urls that are (a) publicly-accessible and (b) don't contain any actual carrier data. They just have the internal TE" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [18:36:42] Yeah... hashes aren't super reliable since they usually get a version upgrade after their published. A list of C&C ips would have been nice.. [18:37:28] But yeah, it would be about 5 minutes to write a nagios check for those hashes, if you wanted to. [18:38:08] no hurry [18:39:22] bblack, we only care about proxies as 'opera' or 'other'. if we added a proxy to the proxies configuration, the value may be non-empty, actually. [18:42:45] bblack, yurik's looking at an alternative approach based on X-Analytics, although we'd need to just double check the delimeter parsing i the existing analytics scripts [18:48:16] (03PS1) 10Yurik: Enhanced X-Analytics header for zero [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 [18:48:23] bblack, dr0ptp4kt ^ [18:50:39] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [18:51:40] (03CR) 10Reedy: [C: 032] Update Wikipedias to 1.23wmf18 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119759 (owner: 10Reedy) [18:51:57] (03Merged) 10jenkins-bot: Update Wikipedias to 1.23wmf18 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119759 (owner: 10Reedy) [18:56:49] Are there known issues in Bugzilla sending e-mails? [18:57:09] I just changed a bug and got "There was an error sending mail from 'bugzilla-daemon@wikimedia.org' to 'moriel@gmail.com': error when closing pipe to /usr/lib/sendmail: Unknown Error (1)" [18:57:16] whoa [18:57:27] andre__: ^^ [18:57:28] Yeah, not fun. [18:57:30] (Also, the change didn't save.) [18:57:37] that shouldn't block.... [18:57:40] but, perl [18:57:52] (03CR) 10Dzahn: "thanks a lot! this resolved https://bugzilla.wikimedia.org/show_bug.cgi?id=60277 btw :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 (owner: 10Dzahn) [18:57:52] Hmm, I didn't change anything lately. [18:58:04] Related to the eqiad move, somehow? [18:58:10] what eqiad move? [18:58:13] that was ages ago [18:58:15] James_F, comment should still be added. [18:58:16] * James_F speculates wildly. [18:58:18] the one you did.. yeah, a while ago [18:58:25] OK, never mind me then. [18:58:27] * greg-g lags a bit [18:58:30] five weeks later we get sendmail errors? ;)) [18:58:32] Disc full? [18:58:44] (03PS1) 10QChris: Remove qchris from Icinga's analytics conactgroup [operations/puppet] - 10https://gerrit.wikimedia.org/r/119796 [18:58:47] #1 cause of random server errors for us. [19:00:08] hexmode: andre__: bugzilla is giving errors (unable to send mail), its configured to use sendmail [19:00:15] last I checked it was conifugred to use smtp which works fine [19:00:17] https://bugzilla.wikimedia.org/editparams.cgi?section=mta [19:00:18] Krinkle: ^^^ Read up. [19:00:29] :) [19:00:31] andre__: Sorry, bug comment /was/ added, but didn't send e-mails (so no wikibugs-l, so no wikibugs bot). [19:00:32] I have [19:00:34] !log disk full on zirconium - gzipping some etherpadlite.sql dump i found [19:00:39] Logged the message, Master [19:00:46] hey, yes, it is disk space [19:00:49] nice [19:00:54] Bugzilla is on zirconium. [19:00:55] and i don't think even BZ related at all [19:00:57] James_F++ [19:00:57] Called it! [19:00:58] yes it is [19:01:00] (03PS2) 10QChris: Remove qchris from Icinga's analytics contactgroup [operations/puppet] - 10https://gerrit.wikimedia.org/r/119796 [19:01:01] but more than that [19:01:04] mutante: Thanks. [19:01:11] (03PS1) 10Ottomata: RT 7085 - adding analytics contact group to researchdb icinga alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 [19:01:37] akosiaris: anything related to etherpad going on or coincidence? [19:02:08] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Wikipedias to 1.23wmf18 [19:02:13] Logged the message, Master [19:02:27] andre__: hexmode: James_F: Do you know why bugzilla uses sendmail now instead of smtp? [19:03:02] Hm.. maybe because it was configured to use smtp.pmtpa.wmnet, perhaps that one got kicked? [19:03:08] mutante: not that I can see of. [19:03:22] Krinkle: no. Plus unrelated. [19:03:27] k [19:03:41] akosiaris: there was 1.1G etherpadlite.sql , that's why i asked [19:03:53] but i just looked for random large files [19:04:11] mutante: ah that was left from the migration from the old etherpad [19:04:19] do we still need it? [19:04:20] manually populated files :-( [19:04:28] nope [19:04:36] but we got a vg there [19:04:44] has 1T free space [19:04:49] Krinkle: who cares? [19:05:05] so let's just move some on it [19:05:11] Krinkle: it's disk space :) [19:06:07] i just see swap being an LV [19:06:36] so let's create an lv and pull whatever is a disk hog on it [19:06:48] (03CR) 10Reedy: [C: 032] group0 to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119760 (owner: 10Reedy) [19:06:53] root (/) is fine on /dev/md0 [19:07:05] but an lv for whatever needs disk spaces sounds ideal [19:07:21] (03Merged) 10jenkins-bot: group0 to 1.23wmf19 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119760 (owner: 10Reedy) [19:08:39] !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: group0 wikis to 1.23wmf19 [19:08:44] Logged the message, Master [19:08:48] (03PS2) 10Reedy: Remove legacy settings for wikibase setup without the build [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119748 (owner: 10Aude) [19:08:56] ick [19:09:06] (03CR) 10Alexandros Kosiaris: [C: 032] Remove qchris from Icinga's analytics contactgroup [operations/puppet] - 10https://gerrit.wikimedia.org/r/119796 (owner: 10QChris) [19:09:14] Reedy: ... [19:09:39] (03CR) 10Dr0ptp4kt: [C: 04-1] "I would like to see this even go further." (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [19:09:51] aude: I only pressed rebase :P [19:09:59] test.wikipedia is on wmf19? [19:10:10] yup [19:10:17] or it should be [19:10:18] ok, [19:10:28] i looked first and it was wmf18 and can't login [19:10:29] [53117823] /wiki/Special:Version Exception from line 318 of /usr/local/apache/common-local/php-1.23wmf19/includes/MagicWord.php: Error: invalid magic word 'wbreponame' [19:10:30] Haha [19:10:33] I better run scap [19:10:50] yes [19:10:55] !log reedy Started scap: Rebuild 1.23wmf19 l10n cache for wikibase [19:11:00] Logged the message, Master [19:12:06] so mutante. apache logs ? perhaps /var/log lv ? [19:12:24] it will need some minimal downtime [19:12:52] people wont love bugzilla not responding for 3-5 minutes [19:12:54] akosiaris: i was wondering if we want them or not [19:13:21] i see a lot of etherpad and planet in other_vhosts.log as opposed to BZ [19:13:39] we can split that up [19:13:45] housecleaning [19:13:57] but not much to do other than that [19:14:21] well, we have 3.2G free [19:14:27] where did all that just come from [19:14:29] (03CR) 10Krinkle: "Yes, an empty jshintrc is essentially the same as no jshintrc in the file path." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119750 (owner: 10Hashar) [19:14:34] apt-get clean [19:14:41] /var/cache/apt/archives [19:14:56] nice, i would have done that next, was still looking at logs [19:15:15] I say /var/log a 50G LV and be done with it [19:15:39] some 5 mins downtime for apache and we are done [19:16:14] i say that sounds good, if we want to be logging in the first place [19:16:31] ok. I 'll do it then [19:16:32] for downtime scheduling , i'll ask andre__ [19:16:39] thanks! [19:16:40] and blame the 5 mins on electromagnetic radiation from satellite debris [19:16:45] andre__: ?:P [19:16:51] mutante, Put stuff down whenever you want. [19:17:03] just mention that you needed to take the server down for 5 minutes to install the NSA wiretapping device [19:17:15] LeslieCarr: aaah yes [19:17:27] mutante: I don't need this Bugzilla thingy plus all those people reporting stuff that they think is important [19:18:18] (03CR) 10Faidon Liambotis: [C: 032] add tree command [operations/puppet] - 10https://gerrit.wikimedia.org/r/119127 (owner: 10Rush) [19:18:39] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [19:18:40] (03PS1) 10Ottomata: Adding configs for new install of cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/119801 [19:19:37] (03PS2) 10Ottomata: Adding configs for new install of cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/119801 [19:22:56] !log reedy Finished scap: Rebuild 1.23wmf19 l10n cache for wikibase (duration: 12m 01s) [19:23:03] Logged the message, Master [19:24:26] (03PS3) 10Ottomata: Adding configs for new install of cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/119801 [19:24:28] (03CR) 10Hoo man: [C: 031] Remove legacy settings for wikibase setup without the build [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119748 (owner: 10Aude) [19:24:30] (03CR) 10Ottomata: [C: 032 V: 032] Adding configs for new install of cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/119801 (owner: 10Ottomata) [19:24:44] aude: done ;) [19:25:08] yay! [19:26:09] (03CR) 10Hoo man: [C: 031] Add Wikibase repoSiteName setting for client [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119751 (owner: 10Aude) [19:26:18] when https://gerrit.wikimedia.org/r/#/c/119751/ gets merged/ deployed then the magic word will be translatable [19:26:23] doesn't have to be now though [19:27:06] Might aswell get them done [19:27:07] (03PS2) 10Aude: Add Wikibase repoSiteName setting for client [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119751 [19:27:15] ok [19:27:17] will we see strange things with WD between merging that and rebuilding the cache? [19:27:22] no [19:27:29] it will just say "Wikidata", the default [19:27:31] (03CR) 10Reedy: [C: 032] Remove legacy settings for wikibase setup without the build [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119748 (owner: 10Aude) [19:27:38] (03Merged) 10jenkins-bot: Remove legacy settings for wikibase setup without the build [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119748 (owner: 10Aude) [19:27:38] (the defaults are just example) [19:27:42] ah right... it has a robust fallback [19:27:49] I think I even tested that :P [19:28:09] and will be seen on test wikidata only [19:28:10] (03CR) 10Reedy: [C: 032] Add Wikibase repoSiteName setting for client [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119751 (owner: 10Aude) [19:28:17] (03Merged) 10jenkins-bot: Add Wikibase repoSiteName setting for client [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119751 (owner: 10Aude) [19:28:42] and will be seen on test wikidata only << not true, this is a client thing [19:28:43] err test2 and test wikipedia [19:28:46] testwikipedia [19:28:47] (03CR) 10Dzahn: [C: 031] RT 7085 - adding analytics contact group to researchdb icinga alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 (owner: 10Ottomata) [19:28:51] yep :P [19:28:55] !log reedy synchronized wmf-config/ 'Wikibase config updates' [19:29:01] Logged the message, Master [19:30:57] /dev/mapper/zirconium-varlog 50G 1.5G 46G 4% /a [19:31:00] nice akosiaris [19:33:10] and done [19:33:21] icinga did even have time to whine :-) [19:33:25] did not* [19:33:35] the wmbot-4 noticed, but that's all i saw [19:33:54] thanks! [19:34:10] the feed will be disabled, until you re-enable it by typing @rss+ bugzilla [19:34:13] and re-enabled [19:34:58] (03PS1) 10Dr0ptp4kt: Add Opera Mini tagging for 515-03. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119804 [19:35:12] !log created a 50G LV for /var/log on zirconium, stopped all services, moved data to it, mounted it and restarted all services [19:35:17] Logged the message, Master [19:35:20] neeeext [19:35:47] :) you're on a roll. that week must have been crazy [19:36:04] yeah, but next is gonna be sweeeet [19:36:25] scare the person who was it then ,heh [19:36:28] has [19:38:24] (03PS3) 10Reedy: Manage scap proxy rsync config in puppet. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 [19:38:42] (03PS2) 10Reedy: Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 [19:39:43] (03CR) 10jenkins-bot: [V: 04-1] Manage scap proxy rsync config in puppet. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [19:39:46] bblack, https://gerrit.wikimedia.org/r/#/c/119804 should be safe regardless of the other patches yurik and i submitted today. would you please review 119804 and, if it looks okay, merge & deploy? [19:39:51] (03CR) 10jenkins-bot: [V: 04-1] Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 (owner: 10Reedy) [19:39:53] (03CR) 10Dzahn: "I think it's ok, there's just one thing i'm not sure about. If you have a service with crit => true, that means it's going to try and page" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 (owner: 10Ottomata) [19:40:18] (03PS2) 10Dzahn: RT 7090 - adding analytics contact group to researchdb icinga alerts [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 (owner: 10Ottomata) [19:40:44] (03PS4) 10Reedy: Manage scap proxy rsync config in puppet. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 [19:41:37] (03CR) 10jenkins-bot: [V: 04-1] Manage scap proxy rsync config in puppet. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 (owner: 10Reedy) [19:42:18] (03PS2) 10Reedy: Update docroot_dir_allows to use network::constants::mediawiki_appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119668 [19:43:04] (03CR) 10jenkins-bot: [V: 04-1] Update docroot_dir_allows to use network::constants::mediawiki_appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119668 (owner: 10Reedy) [19:43:34] fail [19:43:59] (03PS5) 10Reedy: Manage scap proxy rsync config in puppet. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119677 [19:45:38] (03PS3) 10Reedy: Update docroot_dir_allows to use network::constants::mediawiki_appservers [operations/puppet] - 10https://gerrit.wikimedia.org/r/119668 [19:45:42] (03PS3) 10Reedy: Add mw1161 and mw1201 as scap proxies for EQIAD row C and D [operations/puppet] - 10https://gerrit.wikimedia.org/r/119686 [19:46:59] (03PS1) 10Catrope: Enable the VE Math plugin by default on mediawikiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119807 [19:47:59] (03PS1) 10RobH: using titanium rather than cobalt, swapping IP into use [operations/dns] - 10https://gerrit.wikimedia.org/r/119808 [19:48:47] (03PS1) 10Dzahn: remove sq67,sq68,sq69,sq70 [operations/dns] - 10https://gerrit.wikimedia.org/r/119809 [19:53:18] ori: did you get a chance to look into the zlib change too? [19:53:19] (03CR) 10RobH: [C: 032] using titanium rather than cobalt, swapping IP into use [operations/dns] - 10https://gerrit.wikimedia.org/r/119808 (owner: 10RobH) [19:53:56] AaronSchulz: not yet. do you mean that there is an existing patch, or are you asking if i got around to creating it? [19:54:08] the later [19:55:36] (03CR) 10Ottomata: "I guess if they don't have phone numbers listed they can't get paged?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119797 (owner: 10Ottomata) [19:55:38] no. let me do it now [19:57:19] AaronSchulz: are you sure it's worth it? [19:57:56] AaronSchulz: does the size trump the extra cost in decoding? [20:01:20] I'm not worried about decoding time, though we could get specific lzma vs gzip size data [20:01:22] I guess it can wait [20:02:06] (03PS1) 10Ottomata: Cobalt had a serial problem, using titanium for new archiva server instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/119811 [20:02:11] AaronSchulz: https://code.google.com/p/memcached/issues/detail?id=317 [20:02:16] see notes about lz4 esp. [20:02:19] (03CR) 10jenkins-bot: [V: 04-1] Cobalt had a serial problem, using titanium for new archiva server instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/119811 (owner: 10Ottomata) [20:02:20] which hhvm supports [20:02:25] (03PS2) 10Ottomata: Cobalt had a serial problem, using titanium for new archiva server instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/119811 [20:03:51] (03CR) 10Ottomata: [C: 032 V: 032] Cobalt had a serial problem, using titanium for new archiva server instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/119811 (owner: 10Ottomata) [20:05:23] (03PS2) 10Catrope: Enable the VE Math plugin by default on mediawikiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119807 [20:05:51] (03PS1) 10Dzahn: remove ssl1-4 - decom [operations/dns] - 10https://gerrit.wikimedia.org/r/119815 [20:06:11] actually I mean fastlz vs zlib [20:07:13] isn't fastlz currently enabled? [20:07:17] (03PS2) 10Dzahn: remove sq67,sq68,sq69,sq70 [operations/dns] - 10https://gerrit.wikimedia.org/r/119809 [20:07:54] (03CR) 10Dzahn: [C: 032] remove sq67,sq68,sq69,sq70 [operations/dns] - 10https://gerrit.wikimedia.org/r/119809 (owner: 10Dzahn) [20:08:16] !log DNS update - remove sq67-70, former varnish testing [20:08:22] Logged the message, Master [20:12:50] (03PS2) 10Dzahn: remove ssl1-4 - decom [operations/dns] - 10https://gerrit.wikimedia.org/r/119815 [20:14:04] (03CR) 10Dzahn: [C: 032] remove ssl1-4 - decom [operations/dns] - 10https://gerrit.wikimedia.org/r/119815 (owner: 10Dzahn) [20:14:29] !log DNS update - remove ssl1-4 [20:14:34] Logged the message, Master [20:25:20] another one(4) bites the dust [20:25:54] #6099 is getting smaller and i'm so happy with that [20:34:26] what is 6099 matanya [20:34:28] ? [20:34:33] the RT for pmtpa migration? [20:36:15] yup [20:36:23] "what's left in Tampa" [20:37:01] any clue what we are going to do with the servers? [20:37:26] depends on the server [20:37:33] if its under warranty it goes to new datacenter buildout [20:37:55] if its not under warranty we'll be selling them off to some it resales place and applying any income from that to new center [20:39:05] In the past we've donated them to other non-profits, but the scope of the migration and the current bandwidth of operations doesn't allow for it this time. [20:39:32] I don't understand what it means. [20:39:46] 'scope of migration' and 'bandwith of operations' [20:40:21] robh: thanks :-] [20:40:22] its too much work to review the hundreds of applications for free servers we get when we open up for them [20:40:46] it involves reading then confirming on government registries for each state that the applying nonprofit is in fact a nonprofit [20:40:57] and then making the judgement call on which nonprofits get them [20:41:19] plus it was always done by chris or i and neitehr one of us has the time =] [20:41:33] yeah yeah I am not blaming you robh :-] [20:41:36] was just wondering [20:41:40] nope, didnt think you were [20:41:47] (03CR) 10Yurik: Enhanced X-Analytics header for zero (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [20:41:49] no ones asked in public channels though so i was happy to explain =] [20:41:51] sending the good one to the new center and getting rid of the oldest one sounds sane [20:42:04] +1 for the logged explanation we can refers folk to hehe [20:42:10] yea, it was a tough call for me to just resale the old servers versus donating to other non profits [20:42:15] cuz i really, really lied doing the latter [20:42:19] +k [20:42:30] or you're just a big fat liar [20:42:30] wasnt just my call mind you [20:42:48] is Reedy back? [20:43:12] yurik: I *believe* he's still less than 100% time on WMF stuff, but he did the deploy today :) [20:43:48] hmm.. that's a good news, right? :) [20:44:39] (03PS1) 10Alexandros Kosiaris: Create a user for access to OSM db [operations/puppet] - 10https://gerrit.wikimedia.org/r/119869 [20:45:16] yurik: yeah :) [20:45:23] yurik: saved bd808 some sanity today :) [20:45:35] Reedy's already gone mad, so no harm there :P [20:45:47] alright, time to migrate back home and get some lunch [20:45:55] bon apetite [20:47:51] anyone know why the job queue has exploded? [20:47:59] manybubbles? [20:48:00] it's been a few days now and it's not getting better [20:48:08] what is up? [20:48:20] any indexing going on? [20:48:58] nothing abnormal planned [20:49:17] oh, but it's only 400k, nothing like the ten millions we had few months ago [20:49:20] http://ganglia.wikimedia.org/latest/graph.php?r=month&z=xlarge&c=Miscellaneous+eqiad&h=terbium.eqiad.wmnet&jr=&js=&v=372073&m=Global_JobQueue_length [20:49:32] lots of parsoid jobs queued [20:49:39] but beyond that doesn't look abnormal [20:49:45] and Parsoid does that from time to time anyway [20:52:18] (03PS1) 10Alexandros Kosiaris: Tune labsdb postgresql [operations/puppet] - 10https://gerrit.wikimedia.org/r/119870 [20:52:39] Coren: labstore1001 puppet? [20:53:31] paravoid: that's me. [20:53:42] It's disabled to support a temporary keypair for data migration. [20:53:58] oh, ok. Coren said that he'd reenable it in 6h last time, that's why I'm pinging him [20:54:00] hashar: mutante deployed. Thanks for the aptch :) [20:54:21] paravoid: Oh, I hope he doesn't, I'm still migrating stuff. [20:54:43] YuviPanda: oh:) thanks [20:54:48] and what's the conflict with puppet? [20:55:24] paravoid: puppet is pretty militant about ensuring that only puppetized authorized keys are present. It wasn't obvious how to set up an authorized root key w/out turning off puppet. [20:55:34] authorized_keys2 [20:55:49] well… that's too easy :/ [20:55:50] * andrewbogott tries it [20:55:57] just put the extra key in a file called .ssh/authorized_keys2 but please remember to remove it when you're done [20:56:31] ok, done. [20:56:32] !log Updated scholarships.wikimedia.org to cb2ef4c (fix for bug 62464) [20:56:37] Logged the message, Master [20:56:45] But now you've made me reluctant to enable puppet in case Coren was doing something as well :) [20:56:56] ori: does https://gerrit.wikimedia.org/r/#/c/117916/ look fine? [20:57:15] paravoid: wait, isn't the set of authorized key files enumerated someplace? In a file that is itself puppetized? [20:57:16] (03CR) 10Aude: [C: 04-1] "typo" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119870 (owner: 10Alexandros Kosiaris) [20:58:02] greg-g: {{done}} [20:58:19] {{citationneeded}} [20:58:20] (03CR) 10Alexandros Kosiaris: Tune labsdb postgresql (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119870 (owner: 10Alexandros Kosiaris) [20:58:22] (03PS2) 10Alexandros Kosiaris: Tune labsdb postgresql [operations/puppet] - 10https://gerrit.wikimedia.org/r/119870 [20:59:35] (03CR) 10Aude: [C: 031] "looks sane :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119870 (owner: 10Alexandros Kosiaris) [21:01:05] paravoid, re parsoid jobs: it would be good to check if all parsoid job runners are indeed running /cc AaronSchulz [21:01:27] (03CR) 10Aude: [C: 031] "not sure about the cidr (looks sane), everything else looks correct to me and will be useful to have this" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119869 (owner: 10Alexandros Kosiaris) [21:02:13] AaronSchulz: in a meeting, will review after [21:03:01] although the parsoid cluster is also busier than normal right now [21:03:20] !log reedy updated /a/common to {{Gerrit|Iaa99d2162}}: Add Wikibase repoSiteName setting for client [21:03:25] Logged the message, Master [21:03:27] (03PS1) 10Reedy: Add mediaviewer.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119872 [21:03:45] (03CR) 10Reedy: [C: 032] Add mediaviewer.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119872 (owner: 10Reedy) [21:03:52] (03Merged) 10jenkins-bot: Add mediaviewer.dblist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119872 (owner: 10Reedy) [21:04:22] looks like somebody is creating a dump of cebwiki [21:06:00] !log reedy synchronized docroot and w [21:06:06] Logged the message, Master [21:06:39] (03PS1) 10Reedy: Add contact pages for legal to testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119873 [21:10:46] (03PS2) 10Reedy: Add contact pages for legal to testwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119873 [21:11:13] (03CR) 10Erik Zachte: [C: 031] "All of wikistats is kind of legacy for several years. Until WMF find resources to replace it, we keep it running as good as we can with < " [operations/puppet] - 10https://gerrit.wikimedia.org/r/119754 (owner: 10Ottomata) [21:12:05] (03CR) 10Reedy: [C: 04-1] Add contact pages for legal to testwiki (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119873 (owner: 10Reedy) [21:13:27] (03PS2) 10Hashar: Vary Zero configuration by $::realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 [21:13:47] (03CR) 10Hashar: "Awesome! Hopefully PS2 properly vary the URL by realm :-]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [21:18:03] (03PS1) 10Ori.livneh: Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119875 [21:19:25] (03CR) 10Greg Grossmeier: [C: 031] "Doit" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119875 (owner: 10Ori.livneh) [21:19:45] (03CR) 10Deskana: [C: 031] "Needed for HHVM page on mediawiki.org." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119875 (owner: 10Ori.livneh) [21:19:55] (03CR) 10Ori.livneh: [C: 032] Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119875 (owner: 10Ori.livneh) [21:20:07] !log ori updated /a/common to {{Gerrit|Id6222f4db}}: Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist [21:20:13] Logged the message, Master [21:20:21] ori: how did you setup "boogs" on labs? it's confusing to me because it's supposed to be puppetmaster::self and has the role selected but it does not appear to even look at stuff in /etc/puppet/ [21:20:41] !log ori synchronized wmf-config/InitialiseSettings.php 'Id6222f4d: Add RSS of Bugzilla query of open HHVM bugs to mediawikiwiki's whitelist' [21:20:45] Logged the message, Master [21:21:48] (03CR) 10Jforrester: [C: 031] Enable the VE Math plugin by default on mediawikiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119807 (owner: 10Catrope) [21:22:21] paravoid: No, it can be reenabled now, I just forgot. [21:22:47] (And sorry, I needed a nap. two weeks of twelve hour days is beginning to show its toll) [21:23:14] Although, honestly, I only wanted to rest my eyes for a few minutes, I didn't expect a 4h nap. :-) [21:24:13] RECOVERY - Puppet freshness on labstore1001 is OK: puppet ran at Thu Mar 20 21:24:10 UTC 2014 [21:34:53] mutante: I used my attempt to Puppetize Bugzilla along with puppetmaster::self; if you recall it was eventually deemed unacceptable because it was relying on third-party cvs and I abandoned it [21:35:41] (03PS2) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 [21:35:56] dr0ptp4kt, bblack is this better ^ [21:36:10] ori: yea, i remember that part, i didn't even mean the code though, i pulled the latest stuff. it's just that it is supposed to be ::self but when i very obviously change things in /etc/puppet/ the puppet run doesnt change, as if it's looking elsewhere [21:36:53] ori: and second part .. if you say you don't need any of that, then we're gonna just setup a fresh one and not bother moving it [21:37:08] ori: i see you were testing TargetProcess's KanBan board etc [21:37:29] mutante: i never really needed it; i set it up for andre__. the kanban board stuff can be nuked from my perspective. [21:37:36] just looked to help andre migrate :) [21:37:50] ori: then we are all fine with that, cool:) [21:37:56] yay [21:38:12] thanks everybody [21:38:56] great, cya later then. we'll just make a new one [21:40:07] (03PS1) 10Ori.livneh: Follow up Id6222f4db to amend sort order in feed URL [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119881 [21:40:23] andre__: thanks, convey my thanks to mutante too if you chat with him soon [21:40:34] sure [21:42:33] (03CR) 10Greg Grossmeier: [C: 031] "Of course." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119881 (owner: 10Ori.livneh) [21:42:46] (03CR) 10Ori.livneh: [C: 032] Follow up Id6222f4db to amend sort order in feed URL [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119881 (owner: 10Ori.livneh) [21:43:01] !log ori updated /a/common to {{Gerrit|If51eda243}}: Follow up Id6222f4db to amend sort order in feed URL [21:43:06] Logged the message, Master [21:43:42] !log ori synchronized wmf-config/InitialiseSettings.php 'If51eda243: Follow up Id6222f4db to amend sort order in feed URL' [21:43:47] Logged the message, Master [21:47:33] (03CR) 10Ottomata: Adding archiva module (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 (owner: 10Ottomata) [21:55:33] (03CR) 10Dr0ptp4kt: "Does unset need to be removed?" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [21:57:09] (03CR) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [21:57:14] dr0ptp4kt, ^ [22:01:03] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 968.707080447 [22:01:25] ottomata: ^ [22:01:29] (03CR) 10Dr0ptp4kt: "Yeah, I think it makes more sense to move it than to delete it, even if deleting things is fun." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:02:58] (03PS3) 10BBlack: Vary Zero configuration by $::realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [22:03:04] (03CR) 10BBlack: [C: 032 V: 032] Vary Zero configuration by $::realm [operations/puppet] - 10https://gerrit.wikimedia.org/r/119769 (owner: 10Hashar) [22:03:08] \O/ [22:03:28] OOOO [22:03:58] (03PS2) 10BBlack: Add Opera Mini tagging for 515-03. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119804 (owner: 10Dr0ptp4kt) [22:04:04] (03CR) 10BBlack: [C: 032 V: 032] Add Opera Mini tagging for 515-03. [operations/puppet] - 10https://gerrit.wikimedia.org/r/119804 (owner: 10Dr0ptp4kt) [22:04:56] bblack: if you have any contacts with the Zero team can you advertise them the change possibly? [22:05:39] I'll send them an email just in case, but it shouldn't be a big deal either way [22:05:48] (03PS1) 10MarkTraceur: Turn MMV on by default on betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 [22:06:20] (03CR) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:06:25] dr0ptp4kt, ^ [22:06:51] bblack, what do you think about my patch? (no rush, just thoughts) [22:07:38] ah yurik, have a look at https://gerrit.wikimedia.org/r/119769 :D [22:07:46] yurik: brings up support for zero.json on beta cluster :-] [22:07:57] hashar, already saw that [22:08:00] good work :) [22:08:19] 99% brandon :-] [22:08:30] yurik: what patch? [22:08:39] Weird! jgage, yt? [22:08:43] bblack, https://gerrit.wikimedia.org/r/119795 [22:08:48] another random leader election for all topics [22:08:52] 22 is now leader for all topics again [22:09:01] this time its different though, i don't see flapping zookeeper connections [22:09:03] hashar, yep, i was about to comment on it that you are disabling betacluster, but than saw all the comment [22:09:16] just one flap, which caused the replica controller to move to 22 [22:14:04] (03PS3) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 [22:15:29] halfak: what's up with the snuggle labs project? Defunct? [22:15:57] andrewbogott, not defunct, but the project is in a weird spot. I intend to make use of it still. [22:16:15] Are my instanced holding up some migration stuff? [22:16:18] (03CR) 10BryanDavis: Allow user to specify mount point for role::labs::lvm::mnt (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119524 (owner: 10BryanDavis) [22:16:25] (03CR) 10Hashar: [C: 031 V: 031] "Applied on beta cluster in eqiad. We have a puppet master on deployment-salt.eqiad.wmflabs which has this change applied (with XFS)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119534 (owner: 10BryanDavis) [22:16:29] (03PS2) 10BryanDavis: Allow user to specify mount point for role::labs::lvm::mnt [operations/puppet] - 10https://gerrit.wikimedia.org/r/119524 [22:16:58] halfak, are you subscribed to labs-l? In theory you need to claim a project to avoid having me shut it down. [22:17:01] And yours is next on the list. [22:17:23] "claim a project"? [22:17:26] (03CR) 10Hashar: [C: 031] "Good to me :-]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119524 (owner: 10BryanDavis) [22:17:28] (03CR) 10Dr0ptp4kt: "Interesting stuff." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:17:37] ^yurik don't punch me [22:17:43] halfak: say you're willing to do the migration [22:17:44] So I take it that's a 'no' w/respect to reading labs-l :( [22:18:02] Well, I am subscribed and I read all of the things that seem relevant. [22:18:07] halfak: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto [22:18:39] It's a lot to ask for. I also don't read every mobile-l, wmfall, wikitech-l post. :( [22:18:44] But I am sorry for the trouble. [22:19:06] andrewbogott: mark it as **important** :) [22:19:23] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Wed 19 Mar 2014 07:10:56 PM UTC [22:19:25] Right now, i can take down those instances completely, but I'd like to not lose the project. [22:20:26] (03CR) 10Dr0ptp4kt: "Oopsie, need to publish comment on PS2." (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:20:30] halfak: https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Howto#Feeling_lazy.3F [22:20:46] Is that what you want? [22:20:51] ^^^there yurik, now don't punch me. sorry, didn't see rebase. [22:20:53] andrewbogott, I see that. It's not worth it to mothball. [22:21:01] I can just take the instances down. [22:21:06] ok, that's easy then :) [22:21:12] Will that get me out of the way? [22:21:37] Mostly just adding a note to the progress page about what you're doing will get you out of the way, for now. [22:21:42] (03CR) 10BBlack: "A couple of things:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:21:56] You can just move your project to the 'migrated' column without doing anything, and that will mean that the pmtpa instances will get erased when pmtpa is shut down. [22:22:02] https://wikitech.wikimedia.org/wiki/Labs_Eqiad_Migration_Progress [22:22:03] RECOVERY - Kafka Broker Messages In on analytics1021 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 2101.86446881 [22:23:01] When you say "migrated", do you mean "Finished migration"? [22:23:15] Yeah, that must be what I mean :) [22:23:25] * halfak wants to be sure.  [22:24:29] andrewbogott, OK. Moved the listing. Is that all you need from me? [22:24:41] halfak: yep! thank you. [22:25:10] No problem. Sorry for not being on top of this. [22:25:52] (03CR) 10Cmcmahon: [C: 031] "enable MMV by default on beta labs" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 (owner: 10MarkTraceur) [22:26:08] (03PS2) 10MarkTraceur: Turn MMV on by default on betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 [22:29:37] (03CR) 10Gergő Tisza: [C: 031] Turn MMV on by default on betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 (owner: 10MarkTraceur) [22:31:00] (03PS1) 10ArielGlenn: re-enable adds-changes dumps out of cron (caught up backlog) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119889 [22:31:15] (03PS4) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 [22:32:33] (03CR) 10ArielGlenn: [C: 032] re-enable adds-changes dumps out of cron (caught up backlog) [operations/puppet] - 10https://gerrit.wikimedia.org/r/119889 (owner: 10ArielGlenn) [22:34:04] (03PS6) 10Ottomata: Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 [22:35:15] (03CR) 10MarkTraceur: [C: 032] "/me crosses self" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 (owner: 10MarkTraceur) [22:35:21] (03PS7) 10Ottomata: Adding archiva module and role, applying on titanium [operations/puppet] - 10https://gerrit.wikimedia.org/r/117024 [22:35:28] (03Merged) 10jenkins-bot: Turn MMV on by default on betalabs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119886 (owner: 10MarkTraceur) [22:38:15] (03CR) 10Yurik: "Brandon - good catch, fixed" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:38:46] bblack, thx, fixed [22:39:00] dr0ptp4kt, you think we should rename that header into something else? [22:39:04] (03CR) 10Dr0ptp4kt: "Cool. I think that's the trick." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [22:39:31] dr0ptp4kt, ^^ [22:39:31] yurik, seems like a reasonable idea to me. X-Yurikbot [22:39:49] kidding [22:39:55] but seems reasonable [22:40:17] dr0ptp4kt, but that would mean touching all the zero lines :( [22:40:57] yurik, what's that now? [22:40:58] you know how varnish & ops are - the less changes were done ,the more likely it will get +2 ;) [22:41:20] and renaming a header into something else would mean refactoring all the zero file [22:41:53] yurik, why would it require refactoring? [22:42:15] or are you just messing? [22:42:31] dr0ptp4kt, search/replace all 23 of req.http.X-Forwarded-By in zero.vcl [22:43:47] yurik, would it be possible to do: [22:44:13] req.http.X-Yuri-XFB = req.http.X-Forwarded-By; [22:44:20] unset req.http.X-Forwarded-By; [22:44:24] ? [22:44:29] dr0ptp4kt, yes yes, but ugly :) [22:44:55] we are talking about a very rare event of non-null proxy [22:45:11] in which case it will be a one-off thing - re-generation of the object [22:45:36] thus slowly cleaning up the cache [22:46:27] at the moment, it's just one set of articles on the languages visited by users of that proxy. you should probably just double check if that's cool. i seem to recall that thing being unset very intentionally in the past, i just can't remember because it bounced around a few times (yes, i know there's the pickaxe) [22:46:48] ^yurik [22:47:36] that is to say, just see if that's cool with opsen with an interest. i agree it's pretty rare. ideally we could get the xfb stuff updated relatively soon to make it stop varying (although we'll need it again eventually for certain things under esi). [22:48:10] by 'pretty rare' i mean pretty rare in the sense we by configuration only have one type of proxy defined at the moment [22:49:09] (03PS1) 10Spage: Enable Flow on two Beta Features discussions [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119892 [22:49:15] paravoid: labs projects 'swift' and 'swiftupgrade' -- any preference on what happens to them, migrationwise? [22:52:12] dr0ptp4kt, i'm still confused - the issue here is that there are a cached pages that vary on XCS+XFP. If we don't unset XFP, than only for those carriers that allow proxy, XCS+XFP will be different from original if coming via proxy, thus forcing a page refresh. The result will no longer rely on XFP, thus becoming XCS-dependent only [22:53:09] the next time the same request comes in, since that page will be XCS dependent, it won't care that XFP is set or not [22:53:24] greg-g, quiddity : I added enabling Flow on the Beta Features pages to SWAT window [22:53:38] page(s)? [22:53:43] I thought it was just hovercards? [22:53:53] * yurik loves new FF29 design! [22:53:55] * greg-g may have misunderstood [22:54:29] yurik, i think we better videoconference. i think we may be talking the same thing, but i'm not sure. okay if i video call you? [22:54:34] I said "going forward any new BetaFeature...", so, as hovercard is the only new BF recently... ? [22:54:38] dr0ptp4kt, ok [22:55:21] yurik, calling [22:56:05] greg-g: they want to enable Flow on the mediawiki.org pages about the betafeatures, IIRC [22:56:15] greg-g: Jared mentioned [22:56:15] Soon to launch beta features include, Hovercards[2], Compact Personal [22:56:18] Bar[3], and Winter(fixed header) [22:56:48] in his e-mail to #wikitec-ambassadors. Talk:Winter is already Flow-enabled. Easier now than later. [22:57:24] YuviPanda: I know :0 [22:57:28] :) [22:57:45] greg-g, spagewmf, I believe the "pages" was a mistake. it's on the calendar as just the 1 page, for today. [22:57:46] spagewmf: but that one doesn't meet any of the criteria he mentioned [22:57:49] greg-g: ok :) sorry, limechat's stalker window encourages these kind of responses, which sometimes end up being really stupid :) [22:57:56] quiddity: there gerrit chagne says otherwise :) [22:58:07] https://www.mediawiki.org/w/index.php?title=Compact_Personal_Bar&action=history [22:59:12] When are Compact Personal Bar, and Winter, actually becoming available as Beta Features? [22:59:21] "When a Beta feature publicly launches (goes from [22:59:21] labs server to mediawiki.org and then other Foundation wikis), or when an [22:59:24] existing Beta Feature recives a major update," [22:59:27] soon. [22:59:33] looking for code for compact personal bar, anynone have a link? [22:59:38] today soon? next week soon? [22:59:43] not today [23:00:58] quiddity: which "calendar" are you referring to? greg-g I can't find Jared, but it's less disruptive to have Flow enabled before people start discussing (again) [23:01:03] month. [23:01:20] spagewmf, https://wikitech.wikimedia.org/wiki/Deployments#Week_of_March_17th [23:01:43] spagewmf: I understand, I just want us to make good on what we say, it doesn't appear that there's much movement with compact personal bar, so I don't understand the hurry, and it's also just not what we agreed on during that huge thread where this was all banged out [23:01:51] (but as greg noted, I didn't look at the gerrit patches contents) [23:02:25] greg-g I agree. threads vs. bugs vs. gerrit patches %$()%*#$)(R%*)$%*()!@*!!!!!~! [23:02:45] I'll amend the patch and it'll teach %$*(#&! managers to be clearer [23:02:49] sure, but thread and jared's email say one thing, only the (just now) created gerrit change says otherwise :) [23:03:07] oh, I was looking at the wednesday entry. >.> [23:03:10] spagewmf: unless I missed another thing that contradicted the thread and jared's email :) [23:03:30] Woo, resolution :) [23:04:05] quiddity: yeah, the calendar entry you see for today was just added 11 minutes ago, which is fine, but not gospel ;) [23:04:19] (03PS2) 10Spage: Enable Flow on Hovercards Beta Features [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119892 [23:05:31] so, SWAT team! engage! [23:05:39] we'll get the procedure aligned with the %$#@! intent, which AIUI is use Flow for *all* Beta Features discussions [23:05:43] RoanKattouw: Go go go. [23:05:49] greg-g: I'll take today's [23:05:54] tell me when I have 10 minute countdown, so that I can archive the old page [23:06:07] * spagewmf isn't riled, must be the keyboard :) [23:06:14] spagewmf: haha [23:06:43] spagewmf: Is there something special about https://gerrit.wikimedia.org/r/#/c/119892/ ? James_F suggested there was ... [23:06:45] spagewmf: get those F, C, K, U, I, N, G keys checked [23:06:52] quiddity: Or is that ---^^ what you want the 10-second countdown on>? [23:08:06] RoanKattouw: when 119892 syncs out the current page contents will be inaccessible, so quiddity archives first. [23:08:16] Oh OK [23:08:25] quiddity: I'd archive right now, then. [23:08:33] Yeah, archive it now [23:08:35] will do [23:08:37] I'll deploy in a little bit [23:08:46] (03CR) 10Catrope: [C: 032] Send Flow specific logs to fluorine [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118020 (owner: 10EBernhardson) [23:08:54] (03Merged) 10jenkins-bot: Send Flow specific logs to fluorine [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118020 (owner: 10EBernhardson) [23:09:39] spagewmf: for the avoidance of doubt: we should pick a time to flow-ify 'old' betafeature talk pages (ie: ones that don't fit into "new or major updates") [23:10:05] (03CR) 10Catrope: [C: 032] Increase the network performance sampling rate [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:10:08] (03CR) 10jenkins-bot: [V: 04-1] Increase the network performance sampling rate [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:10:09] greg-g: sure, I'll add that [23:10:21] next week during you guys' window? [23:10:57] RoanKattouw, All ready at my end [23:11:49] (03PS2) 10Catrope: Increase the network performance sampling rate [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:13:55] (03CR) 10EBernhardson: [C: 031] Enable Flow on Hovercards Beta Features [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119892 (owner: 10Spage) [23:17:17] quiddity: Sorry, got held up because the MV change is incredibly confusing [23:17:52] greg-g: I'm dropping the MV change from this window on the grounds of 1) incredible confusion and 2) unavailability of Gilles and MarkH (I have Gergo but he doesn't know what's going on) [23:17:59] greg-g: So please reschedule it for some other window [23:18:14] RoanKattouw: /me nods thank you [23:18:56] np, no rush [23:19:11] RoanKattouw: we have a ghost changeset, i think thats the cause of the problem [23:19:14] https://gerrit.wikimedia.org/r/#/c/118774/ [23:19:32] (03CR) 10Catrope: [C: 04-2] Increase the network performance sampling rate (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:19:53] tgr: Aha, maybe that change should have gotten merged before the config change happened [23:20:00] Either way, I -2ed and Gilles can explain himself [23:20:03] Later [23:20:10] For now, I'm gonna move on with this deploy [23:20:20] (03CR) 10Catrope: [C: 032] Enable the VE Math plugin by default on mediawikiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119807 (owner: 10Catrope) [23:20:28] (03CR) 10Catrope: [C: 032] Update symlinks for notcirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118778 (owner: 10Chad) [23:20:30] (03Merged) 10jenkins-bot: Enable the VE Math plugin by default on mediawikiwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119807 (owner: 10Catrope) [23:20:36] (03Merged) 10jenkins-bot: Update symlinks for notcirrus [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118778 (owner: 10Chad) [23:20:38] (03CR) 10Catrope: [C: 032] Enable Flow on Hovercards Beta Features [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119892 (owner: 10Spage) [23:20:40] (03PS5) 10Yurik: Enhanced X-Analytics header with HTTPS and Proxy information [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 [23:20:48] dr0ptp4kt, ^ [23:21:00] (03Merged) 10jenkins-bot: Enable Flow on Hovercards Beta Features [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119892 (owner: 10Spage) [23:21:50] Alright sorry for that confusion, everything else is merged now, going to deploy in a bit [23:22:06] !log catrope updated /a/common to {{Gerrit|Ia08c65d40}}: Enable Flow on Hovercards Beta Features [23:22:11] Logged the message, Master [23:22:20] !log catrope synchronized docroot/noc/createTxtFileSymlinks.sh [23:22:26] Logged the message, Master [23:22:36] !log catrope synchronized wmf-config/InitialiseSettings.php [23:22:38] quiddity: Flow deploy going now [23:22:40] Logged the message, Master [23:22:43] ty [23:22:49] !log catrope synchronized wmf-config/CommonSettings.php [23:22:54] Logged the message, Master [23:22:54] aaand all done [23:23:14] greg-g: SWAT done, could you update the deployment schedule for the fact that the MV change was bounced? [23:24:51] yurik: have you tried that one out on a test host yet? PS5 is good to go? [23:24:54] RoanKattouw: yeah, but why do I not see it on the calendar? [23:25:16] It... was there [23:25:30] gerrit:119269 Increase the network performance sampling rate [23:25:34] oh, that [23:25:35] Sorry the name is confusing [23:25:42] * greg-g was looking for something with capital M's and V's [23:25:43] :) [23:25:59] will do [23:26:16] (03PS1) 10Jforrester: Enable VisualEditor TOC widget on enwiki Beta Labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119896 [23:26:20] RoanKattouw: ^^^^ [23:26:32] bblack, no, haven't tried it on test [23:27:54] bblack, copying files to test, will know in a bit [23:27:56] (03CR) 10Catrope: [C: 04-1] Enable VisualEditor TOC widget on enwiki Beta Labs (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119896 (owner: 10Jforrester) [23:28:41] hey RoanKattouw, I'm late to the show but I'm available for SWAT help. Anything you want to delegate to me? [23:29:05] ori: No, we're all done [23:29:19] "good timing" [23:30:03] cool. could I maybe add one, then? a cherry-pick of to prod branches [23:30:09] greg-g: ^ [23:30:14] ori: That's fine [23:30:48] The issue was with Gilles submitting a patchset that didn't merge, and rebasing it introduced a rebase conflict that indicated something mentioned in the commit summary had happened and said some vague things about what should be done once that happened [23:30:55] suuure [23:31:01] (to ori) [23:31:04] So I was confused, and even more so when I found that Mark had added config vars that don't even exist [23:31:20] So I just slapped a big fat -2 on it and excluded it from the SWAT deploy on the grounds of confusing the crap out of the deployer [23:31:26] :) [23:31:30] solid [23:31:46] If RoanKattouw can't figure it out it's busted :) [23:32:18] Indeed. [23:32:21] greg-g: Also, here's a policy question for ya [23:32:25] RoanKattouw: ^^^ Merge already. [23:32:28] shoot [23:32:40] Should patches with European owners be in the window that's scheduled at 1am European time? [23:32:48] (03CR) 10Jforrester: Enable VisualEditor TOC widget on enwiki Beta Labs (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119896 (owner: 10Jforrester) [23:33:04] (03CR) 10Catrope: [C: 032] Enable VisualEditor TOC widget on enwiki Beta Labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119896 (owner: 10Jforrester) [23:33:06] (03PS1) 10MarkTraceur: Add MMV settings to labs config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119898 [23:33:14] (03Merged) 10jenkins-bot: Enable VisualEditor TOC widget on enwiki Beta Labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119896 (owner: 10Jforrester) [23:33:30] rdwrer: ^^^ [23:33:55] Heh [23:33:59] ori: Cherry-pick to which branches exactly? [23:34:15] James_F: What are you ^^^ing me for [23:34:33] rdwrer: I bounced Gilles's MV config patch from the SWAT window on the grounds of confusing the crap out of the deployer [23:34:33] (03CR) 10MarkTraceur: [C: 032] "Let's try this again." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119898 (owner: 10MarkTraceur) [23:34:33] rdwrer: MV SWAT deploy was pulled 'cos you weren't around. [23:34:36] See the -2 w/ comment [23:34:38] Huh. [23:34:43] You fled right before I started deploying [23:34:44] (03Merged) 10jenkins-bot: Add MMV settings to labs config [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119898 (owner: 10MarkTraceur) [23:34:44] I am around [23:34:47] And Gergo was confused too [23:34:59] (03CR) 10Gergő Tisza: Increase the network performance sampling rate (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:34:59] RoanKattouw: probably not, no :) unless they are night owls [23:35:24] Also per greg-g it was Gilles's patch so he shouldn't have scheduled it during a window during which he would be asleep [23:35:47] https://gerrit.wikimedia.org/r/#/c/119897/ [23:35:57] Indeed [23:36:04] bblack, just tested, works good [23:36:46] ori: all's clear, Roan is done [23:37:37] (03CR) 10Dr0ptp4kt: [C: 031] "This seems to make sense. Have not run it, though." [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [23:38:23] !log catrope synchronized php-1.23wmf18/resources/mediawiki/mediawiki.inspect.js [23:38:28] Logged the message, Master [23:39:10] greg-g: OK it sounds like ori was the one that scheduled Gilles's patch for deployment, not Gilles. He apparently meant to be its shepherd/delegate/whatever but neglected to document this on the deployments page, so I told him to do so in the future. Perhaps the process docs should be more explicit about that [23:40:07] * greg-g wordsmiths [23:40:20] (03CR) 10Yurik: "I just uploaded it on https://en.wikipedia.beta.wmflabs.org, seems to work (ignore bad cert warning) - both https & Good to go. ONLY que" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [23:40:24] dr0ptp4kt, ^ [23:40:38] Alright, deployment over [23:40:39] rdwrer: looks like we forgot to merge https://gerrit.wikimedia.org/r/#/c/119396/ [23:40:55] yurik, cool [23:41:01] https://wikitech.wikimedia.org/w/index.php?title=SWAT_deploys&diff=106284&oldid=105506 [23:41:03] dr0ptp4kt, read the comment [23:41:32] greg-g: my bad, thanks [23:41:41] Yeah [23:41:44] tgr: I learned that now [23:41:48] tgr: Mind review? [23:41:49] ori: np [23:41:50] (03CR) 10Dr0ptp4kt: "Probably https" [operations/puppet] - 10https://gerrit.wikimedia.org/r/119795 (owner: 10Yurik) [23:43:25] RoanKattouw: Flow is go on that page, thanks [23:52:50] (03PS3) 10Ori.livneh: Increase the network performance sampling rate [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:54:19] (03PS4) 10Ori.livneh: Increase the network performance sampling rate [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/119269 (owner: 10Gilles) [23:54:56] RoanKattouw: dropped unrelated stuff from commit message; see updated https://gerrit.wikimedia.org/r/#/c/119269/ [23:55:48] ori: OK, but what about the whole thing about when we enable MV by default we need to go back to 1000?