[00:00:00] looks good! [00:01:03] aude: ok you should be live [00:01:06] excellent :) [00:01:32] thanks! [00:02:27] !log ebernhardson synchronized php-1.24wmf1/extensions/MultimediaViewer/ 'I684d44a0b5' [00:02:34] Logged the message, Master [00:02:34] tgr: ok, your now on 1.24wmf1 as well [00:03:23] ok SWAT deploy completed, afaik we got them all [00:05:36] ebernhardson: thanks! [00:05:44] sorry for the commit flood [00:09:18] tgr: no problem [00:09:24] ebernhardson, ori, greg: my understanding was that most of the SWAT tasks are per extension/branch, and an extra commit only means an extra click in gerrit, so doing everything at once is on the whole less work for SWATters [00:09:32] did I get that wrong? [00:10:09] would it help in a similar case if I squashed unrelated commits together instead of cherry-picking them one by one? [00:15:27] (03PS1) 10Ori.livneh: Add hhvm::dev; provision on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/130535 [00:16:19] tgr: IMO you were perfectly fine, it was the sum total of your patches plus everyone else's that made the window a bit cramped [00:20:09] (03PS3) 10Rush: debian packaging directory with our details [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 [00:20:11] (03PS4) 10Rush: change .gitreview for debian branch [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130142 [00:23:44] (03CR) 10Aaron Schulz: [C: 031] Add hhvm::dev; provision on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/130535 (owner: 10Ori.livneh) [00:29:10] (03CR) 10Chad: [C: 031] Add hhvm::dev; provision on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/130535 (owner: 10Ori.livneh) [00:40:42] (03CR) 10BryanDavis: [C: 031] Add hhvm::dev; provision on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/130535 (owner: 10Ori.livneh) [00:52:12] PROBLEM - Disk space on gallium is CRITICAL: DISK CRITICAL - free space: /var/lib/jenkins-slave/tmpfs 17 MB (3% inode=99%): [01:34:13] RECOVERY - Disk space on gallium is OK: DISK OK [01:58:39] !log Deploying Ia82779635d762a3 on Zuul, reloading services [01:58:48] Logged the message, Master [02:22:26] !log LocalisationUpdate completed (1.24wmf1) at 2014-04-30 02:22:23+00:00 [02:22:33] Logged the message, Master [02:33:15] !log LocalisationUpdate completed (1.24wmf2) at 2014-04-30 02:33:12+00:00 [02:33:21] Logged the message, Master [02:51:52] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Tue Apr 29 23:51:10 2014 [03:06:14] (03CR) 10Ori.livneh: [C: 032] Add hhvm::dev; provision on osmium [operations/puppet] - 10https://gerrit.wikimedia.org/r/130535 (owner: 10Ori.livneh) [03:12:13] PROBLEM - DPKG on osmium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [03:13:13] RECOVERY - DPKG on osmium is OK: All packages OK [03:13:22] RECOVERY - Puppet freshness on osmium is OK: puppet ran at Wed Apr 30 03:13:18 UTC 2014 [03:23:33] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 30 03:23:13 UTC 2014 (duration 23m 12s) [03:23:39] Logged the message, Master [04:04:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:06:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:08:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:10:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:12:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:14:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:16:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:18:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:20:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:22:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:24:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:26:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 03:58:50 2014 [04:28:25] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Wed Apr 30 04:28:21 UTC 2014 [04:30:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 04:28:21 2014 [04:32:35] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Last successful Puppet run was Wed Apr 30 04:28:21 2014 [04:33:08] so, analytics1013 has a few puppet checks, eh? [04:49:42] PROBLEM - ElasticSearch health check on elastic1014 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [04:49:42] PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [04:49:43] PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [04:49:43] PROBLEM - ElasticSearch health check on elastic1011 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [04:50:42] RECOVERY - ElasticSearch health check on elastic1015 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [04:50:42] RECOVERY - ElasticSearch health check on elastic1014 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [04:50:43] RECOVERY - ElasticSearch health check on elastic1007 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [04:50:43] RECOVERY - ElasticSearch health check on elastic1011 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [04:58:13] RECOVERY - Puppet freshness on analytics1013 is OK: puppet ran at Wed Apr 30 04:58:10 UTC 2014 [06:52:51] (03PS1) 10Aaron Schulz: Removed "GetLocalFileCopy" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130563 [06:53:33] (03PS2) 10Aaron Schulz: Removed "GetLocalFileCopy" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130563 [06:53:45] (03CR) 10Aaron Schulz: [C: 032] Removed "GetLocalFileCopy" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130563 (owner: 10Aaron Schulz) [06:53:53] (03Merged) 10jenkins-bot: Removed "GetLocalFileCopy" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130563 (owner: 10Aaron Schulz) [06:54:31] !log aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed GetLocalFileCopy pool counter entry' [06:54:38] Logged the message, Master [07:01:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 04:00:28 2014 [07:11:53] PROBLEM - ElasticSearch health check on elastic1010 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [07:12:53] RECOVERY - ElasticSearch health check on elastic1010 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [07:16:06] (03PS1) 10Aaron Schulz: Removed "TMHTransformFrame" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130565 [07:38:16] (03PS1) 10Matanya: releases: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130569 [07:41:42] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 3.38983050847% of data exceeded the critical threshold [500.0] [08:32:21] (03PS1) 10ArielGlenn: sdtpa is gone, remove all entries to devices, links, etc [operations/dns] - 10https://gerrit.wikimedia.org/r/130573 [08:32:59] (03CR) 10ArielGlenn: "Someone who knows for sure what's gone there needs to look at this (are the links really all gone?)" [operations/dns] - 10https://gerrit.wikimedia.org/r/130573 (owner: 10ArielGlenn) [08:36:05] we like to massmove + 50000 files on commons by bot. [08:36:06] are there som issuise with the filestorage when doing so? [08:38:28] matanya: docs.puppetlabs examples don't follow style guide :) [08:38:30] for me it sounds a bit dangerous. [08:39:37] that's 50 k moves from one container to another in the swift backend, which may or may not mean that the data gets copied around (plus the replicated copies) [08:40:00] depends how fast it does that? [08:40:00] how big are these files? [08:40:46] afair there are mass uploader scripts that can be run on a bastion [08:41:05] mutante: where ? [08:41:18] matanya: http://docs.puppetlabs.com/references/latest/type.html#cron-attributes [08:41:32] matanya: no quoting [08:41:42] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1% data above the threshold [250.0] [08:42:08] i don't know, ar typofixes in the filesames. the bot operator will do 2 moves in the minute [08:42:53] I'm going to say that 2 moves a minute is ok [08:43:04] okay, thx^^ [08:43:08] but I would feel better if [08:43:10] heh [08:43:23] we knew that the typical size is not 50mb for example [08:43:32] Steinsplitter: [08:44:49] I guess the thumbs get moved or purged at the same time... I should really look at the move code [08:47:26] (03PS9) 10Dzahn: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [operations/puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [08:49:20] apergos: ok [08:58:30] apergos: please comment on my patch above please :) [09:03:10] which, sorry/ [09:06:02] apergos: https://gerrit.wikimedia.org/r/130569 [09:11:55] matanya: it looks ok except I presume you meant it to go inside the role::releases class instea of dangling? [09:12:57] it is in role::releases [09:13:26] oh, i see what you mean [09:14:41] (03PS2) 10Matanya: releases: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130569 [09:16:49] lgtm [09:29:04] !log dbstore100[12] replicating m2 eventlogging [09:29:10] Logged the message, Master [09:33:53] (03PS1) 10Springle: Make dbstore/db1047 repliction of m2 eventlogging permanent. Also avoid slave autostart for these multi-source replicants. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130576 [09:33:57] (03PS1) 10Dzahn: fix Yuvi's user name in role for mobile releases [operations/puppet] - 10https://gerrit.wikimedia.org/r/130577 [09:34:34] * springle loves writing 'replicants'. feels like blade runner [09:35:15] <_joe_> eheh [09:35:33] (03CR) 10Springle: [C: 032] Make dbstore/db1047 repliction of m2 eventlogging permanent. Also avoid slave autostart for these multi-source replicants. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130576 (owner: 10Springle) [09:37:04] (03CR) 10Dzahn: [C: 032] "RT #7068" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130577 (owner: 10Dzahn) [09:42:10] _joe_: wrote what you asked about, and then figured out that it is already written in other words [09:44:14] (03PS1) 10Dzahn: rename ypanda to yuvipanda [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 [09:50:38] (03PS2) 10Dzahn: rename ypanda to yuvipanda [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 [09:53:10] (03CR) 10Yuvipanda: [C: 031] "TY!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 (owner: 10Dzahn) [09:53:49] (03CR) 10Dzahn: [C: 032] rename ypanda to yuvipanda [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 (owner: 10Dzahn) [09:55:50] (03CR) 10Dzahn: "yuvipanda: @caesium: /User[yuvipanda]/groups: groups changed '' to 'mobileupld' try again now" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 (owner: 10Dzahn) [09:57:17] (03CR) 10Yuvipanda: "GREAT SUCCESS!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130579 (owner: 10Dzahn) [09:57:26] mutante: ^ :) [09:58:47] YuviPanda: :) [10:02:03] (03PS1) 10Matanya: caesium: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130581 [10:02:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 04:00:28 2014 [10:02:38] (03CR) 10Matanya: [C: 04-1] "depends on https://gerrit.wikimedia.org/r/130569" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130581 (owner: 10Matanya) [10:02:47] (03PS2) 10Matanya: caesium: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130581 [10:04:12] RECOVERY - Puppet freshness on osmium is OK: puppet ran at Wed Apr 30 10:04:02 UTC 2014 [10:06:27] mutante: dns recursor needs access to port 53 from outside ? [10:06:47] or is goes through lvs ? [10:10:05] what is outside? [10:14:43] the internet [10:17:22] paravoid: i.e if a user is doing recursive dns query, how does it work internally, and how the dns recursor's involved. [10:18:18] you should in general make your questions a little less specific [10:18:39] in this case, I don't think you understand how recursors are supposed to work :) [10:19:06] what exactly are you trying to accomplish? [10:19:25] find out if the dns recursor roles need firewalling [10:20:26] not for now, no [10:20:26] no they don't [10:20:42] the interesting part to know, is why [10:20:49] i guessed they don't [10:20:55] but not sure i know why [10:21:50] there's not much point having a firewall if all it does is open a hole for the only port open on a box [10:22:21] I wouldn't mind having a firewall on every box with a public IP, but those boxes are low on the list [10:22:37] the exposure there is low [10:22:45] plus, firewalling DNS usually needs to be done with care [10:22:57] increase nf_conntrack limits and such, as it's easy to run over them because of the nature of DNS [10:23:19] my final goal is to have base::firewall on all nodes [10:23:28] this will never happen :) [10:23:40] a more realistic goal is to have base::firewall on all nodes with a public IP [10:23:44] "all" --> allt hose that need it [10:23:48] external firewalls can provide ddos protection etc.... [10:24:13] and still we'll need exceptions even in that more limited goal [10:24:33] any list with that? [10:24:44] what do you mean? [10:25:19] list of host with public IP (i know dns has, but is that SOA?) [10:26:18] oh that's easy: all hosts that have .wikimedia.org (instead of .something.wmnet) [10:27:05] i started with misc-web-servers because it was obvious they need [10:27:52] if i compile a list of servers with public IP, will one prioritize the firewalling for me ? [10:28:09] * matanya is looking at paravoid / akosiaris :) [10:29:06] yup, I could help with that! [10:29:09] that's awesome, thanks :) [10:29:17] no, paravoid doesn't have time for that [10:32:23] (03PS1) 10Giuseppe Lavagetto: Add another check using graphite, small fixes. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130582 [10:38:43] PROBLEM - ElasticSearch health check on elastic1015 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [10:38:43] PROBLEM - ElasticSearch health check on elastic1011 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [10:38:43] PROBLEM - ElasticSearch health check on elastic1007 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [10:38:43] PROBLEM - ElasticSearch health check on elastic1014 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 2: initializing_shards: 1: unassigned_shards: 0 [10:39:24] (03PS2) 10Matanya: antimony: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130306 [10:39:42] RECOVERY - ElasticSearch health check on elastic1011 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6078: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [10:39:43] RECOVERY - ElasticSearch health check on elastic1014 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6078: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [10:39:43] RECOVERY - ElasticSearch health check on elastic1007 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6078: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [10:39:43] RECOVERY - ElasticSearch health check on elastic1015 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6078: relocating_shards: 2: initializing_shards: 0: unassigned_shards: 0 [10:52:45] (03PS2) 10Matanya: magnesium: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130322 [11:06:09] paravoid: although you don't have time for it, fyi : https://etherpad.wikimedia.org/p/nodes_with_a_public_IP [11:10:52] matanya: calcium = robh , and i replaced a few other ? there [11:11:01] (03PS3) 10Alexandros Kosiaris: Introduce role::puppetmaster classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/130310 [11:12:25] thanks mutante [11:14:33] (03PS11) 10Alexandros Kosiaris: Torrus: add torrus to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 (owner: 10Matanya) [11:14:56] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce role::puppetmaster classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/130310 (owner: 10Alexandros Kosiaris) [11:16:14] (03PS2) 10Alexandros Kosiaris: Usage of a weighted loadfactor in puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/130331 [11:16:17] (03PS12) 10Alexandros Kosiaris: Torrus: add torrus to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 (owner: 10Matanya) [11:17:18] (03PS10) 10Dzahn: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [operations/puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [11:18:12] (03CR) 10Alexandros Kosiaris: [C: 032] Torrus: add torrus to netmon1001 [operations/puppet] - 10https://gerrit.wikimedia.org/r/108314 (owner: 10Matanya) [11:20:39] (03PS11) 10Dzahn: Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [operations/puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [11:20:58] (03PS1) 10Matanya: otrs: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 [11:22:23] (03CR) 10Dzahn: [C: 032] Make puppet cronjob to run SecurePoll/cli/purgePrivateVoteData.php [operations/puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [11:23:13] (03PS1) 10Matanya: manutius: remove torrus [operations/puppet] - 10https://gerrit.wikimedia.org/r/130587 [11:23:28] (03CR) 10jenkins-bot: [V: 04-1] manutius: remove torrus [operations/puppet] - 10https://gerrit.wikimedia.org/r/130587 (owner: 10Matanya) [11:24:29] (03CR) 10Dzahn: "Misc::Maintenance::Purge_securepoll/Cron[purge_securepollvotedata]/ensure: created" [operations/puppet] - 10https://gerrit.wikimedia.org/r/74592 (https://bugzilla.wikimedia.org/51574) (owner: 10Reedy) [11:25:24] (03PS2) 10Matanya: manutius: remove torrus [operations/puppet] - 10https://gerrit.wikimedia.org/r/130587 [11:26:32] PROBLEM - ElasticSearch health check on elastic1002 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 0 [11:26:32] PROBLEM - ElasticSearch health check on elastic1005 is CRITICAL: CRITICAL - elasticsearch (production-search-eqiad) is running. status: red: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2032: active_shards: 6075: relocating_shards: 0: initializing_shards: 1: unassigned_shards: 0 [11:27:32] RECOVERY - ElasticSearch health check on elastic1002 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [11:27:32] RECOVERY - ElasticSearch health check on elastic1005 is OK: OK - elasticsearch (production-search-eqiad) is running. status: green: timed_out: false: number_of_nodes: 16: number_of_data_nodes: 16: active_primary_shards: 2033: active_shards: 6076: relocating_shards: 0: initializing_shards: 0: unassigned_shards: 0 [11:28:36] (03PS3) 10Matanya: manutius: remove torrus [operations/puppet] - 10https://gerrit.wikimedia.org/r/130587 [11:47:49] (03PS3) 10Dzahn: split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 [11:54:25] (03PS4) 10Dzahn: split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 [11:56:40] (03PS5) 10Dzahn: split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 [12:00:04] (03CR) 10Matanya: split out apache sync scripts from mw deploy (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 (owner: 10Dzahn) [12:01:32] (03PS1) 10Nuria: [WIP] Upstart follows fork when starting celery. [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130588 (https://bugzilla.wikimedia.org/63819) [12:05:38] (03PS6) 10Dzahn: split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 [12:07:36] (03CR) 10Matanya: split out apache sync scripts from mw deploy (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 (owner: 10Dzahn) [12:17:32] (03PS7) 10Dzahn: split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 [12:21:31] (03CR) 10Matanya: [C: 031] split out apache sync scripts from mw deploy [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 (owner: 10Dzahn) [12:22:13] (03PS21) 10Matanya: etherpad: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/107567 [12:22:55] (03PS1) 10Dzahn: sudo ALL for jkrauska on role pmacct [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 [12:23:47] (03PS2) 10Dzahn: sudo ALL for jkrauska on role pmacct [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 [12:24:17] (03PS6) 10Matanya: torrus: move into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/108498 [12:29:50] (03PS3) 10Dzahn: add sudo ALL as pmacct user for Joel [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 [12:38:23] (03PS6) 10Matanya: sudo: convert into a module [operations/puppet] - 10https://gerrit.wikimedia.org/r/111189 [12:44:49] (03PS1) 10Giuseppe Lavagetto: Move cluster definition to the node level. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130591 [12:45:26] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Work in progress, please do not merge." [operations/puppet] - 10https://gerrit.wikimedia.org/r/130591 (owner: 10Giuseppe Lavagetto) [12:49:19] (03PS4) 10Dzahn: full sudo ALL for jkrauska on pmacct (rhenium) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130589 [12:51:15] (03PS1) 10Alexandros Kosiaris: torrus: Conditionalize declaration of rewrite [operations/puppet] - 10https://gerrit.wikimedia.org/r/130592 [12:53:13] (03CR) 10Alexandros Kosiaris: [C: 032] torrus: Conditionalize declaration of rewrite [operations/puppet] - 10https://gerrit.wikimedia.org/r/130592 (owner: 10Alexandros Kosiaris) [13:05:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 10:04:02 2014 [13:11:43] (03Abandoned) 10coren: Add new labsudb role for Labs users' database [operations/puppet] - 10https://gerrit.wikimedia.org/r/74158 (owner: 10coren) [13:23:03] (03CR) 10Giuseppe Lavagetto: [C: 04-2] "This change will break something that did not need fixing: as $is_labs_puppetmaster is defined at node level, referring to it as $is_labs_" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120512 (owner: 10Matanya) [13:23:49] <_joe_> matanya: please read my comment there and ask for clarifications if something is not clear [13:25:59] yes, you are right [13:26:19] (03Abandoned) 10Matanya: puppetmaster: qualify var [operations/puppet] - 10https://gerrit.wikimedia.org/r/120512 (owner: 10Matanya) [13:28:01] osmium puppet runs: that actually works, but the cron jobs is disabled? (vs. agent disabled) [13:28:20] as in "if you run it manually, it will recover" [13:37:16] new firefox is so much nicer in gerrit [13:39:43] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Copy and paste errors." (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120518 (owner: 10Matanya) [13:40:30] (03CR) 10Dzahn: [C: 032] "not touching mw deployment stuff, just Apache. and on tin these scripts never worked because they rely on NFS home and on fenari they were" [operations/puppet] - 10https://gerrit.wikimedia.org/r/129399 (owner: 10Dzahn) [13:40:32] it is really perrty [13:44:17] (03PS3) 10Matanya: mha: fix var scope search [operations/puppet] - 10https://gerrit.wikimedia.org/r/120518 [13:46:51] (03CR) 10Matanya: "shame on me. fixed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120518 (owner: 10Matanya) [13:47:00] (03CR) 10Hashar: "Well it might not be a good idea:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/125991 (https://bugzilla.wikimedia.org/63939) (owner: 10Hashar) [13:49:31] (03CR) 10Rush: [C: 032] "seems good man" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130582 (owner: 10Giuseppe Lavagetto) [13:53:22] (03CR) 10Mark Bergsma: "So... where's our modifications patch then? Typically that's done using a patch in debian/patches/... did you put that in a branch?" (031 comment) [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 (owner: 10Rush) [13:54:26] (03PS1) 10Jgreen: initial stab at firewall rules for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/130596 [13:56:51] (03CR) 10Rush: "so..." [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 (owner: 10Rush) [13:56:55] (03CR) 10Matanya: [C: 04-1] "duplictae of https://gerrit.wikimedia.org/r/#/c/130586/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130596 (owner: 10Jgreen) [13:58:54] (03CR) 10Dzahn: "06:56 < mutante> hashar: Finished catalog run in 380.94 seconds . [hashar]/uid: uid changed '519' to '1010'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130152 (owner: 10Andrew Bogott) [14:00:18] (03CR) 10RobH: [C: 04-1] "I think that the cross-links for sdtpa to eqiad are now simply plugged into pmtpa. I am not certain if those addresses were migrated over" (034 comments) [operations/dns] - 10https://gerrit.wikimedia.org/r/130573 (owner: 10ArielGlenn) [14:01:12] (03CR) 10Hashar: "I had sessions left over on gallium and tin. Once killed and puppet passed around, the uid and my homedir got fixed :-]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130152 (owner: 10Andrew Bogott) [14:02:05] (03CR) 10Alexandros Kosiaris: [C: 04-1] initial stab at firewall rules for iodine (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130596 (owner: 10Jgreen) [14:05:36] (03PS2) 10Matanya: otrs: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 [14:08:43] (03CR) 10Jgreen: [C: 04-1] "missing rule: http from public, also name these otrs-http and otrs-https to avoid duplicate-name-conflict" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:09:10] (03Abandoned) 10Jgreen: initial stab at firewall rules for iodine [operations/puppet] - 10https://gerrit.wikimedia.org/r/130596 (owner: 10Jgreen) [14:11:12] (03CR) 10Matanya: "1) what do you mean missing the pubic?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:20:00] <_joe_> matanya: you lost an l :) [14:20:31] yes, noted. didn't mean it to sound this way :) [14:22:03] (03CR) 10Jgreen: "You fixed the things I commented on while I was writing the comment. :-) Looks fine now." [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:27:42] (03CR) 10Ottomata: Fix provision from scratch (031 comment) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130499 (owner: 10Milimetric) [14:28:14] (03CR) 10Jgreen: "actually: missing SMTP from WMF subnets. this box needs to receive mail relayed through our mx's to function" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:28:28] (03CR) 10Jgreen: [C: 04-1] "actually: missing SMTP from WMF subnets. this box needs to receive mail relayed through our mx's to function" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:29:18] (03CR) 10Matanya: "Is that externally (through the internet) or within the WMF LAN?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:29:20] (03CR) 10Ottomata: "Sure sounds good to me! I remember trying several variations on this when I was writing the upstart config originally, but don't remember" [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130588 (https://bugzilla.wikimedia.org/63819) (owner: 10Nuria) [14:31:05] (03PS2) 10Milimetric: Fix provision from scratch [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130499 [14:31:34] (03CR) 10Milimetric: Fix provision from scratch (031 comment) [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130499 (owner: 10Milimetric) [14:32:36] (03PS1) 10Dzahn: fix apache-graceful-all for use in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/130600 [14:34:10] (03CR) 10Jgreen: "sending mx --> mchenry/sodium --> iodine-smtp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [14:36:58] (03PS3) 10Alexandros Kosiaris: Usage of a weighted loadfactor in puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/130331 [14:38:32] (03PS3) 10Matanya: otrs: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 [14:48:10] (03PS4) 10Alexandros Kosiaris: Usage of a weighted loadfactor in puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/130331 [14:49:17] (03CR) 10Jgreen: [C: 032 V: 031] otrs: add ferm rule [operations/puppet] - 10https://gerrit.wikimedia.org/r/130586 (owner: 10Matanya) [15:03:03] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Usage of a weighted loadfactor in puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/130331 (owner: 10Alexandros Kosiaris) [15:03:26] (03PS1) 10Withoutaname: Disable query pages for closed wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) [15:04:46] (03PS1) 10Dzahn: fix sync-apache for use in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/130610 [15:06:49] (03PS2) 10Dzahn: fix sync-apache for use in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/130610 [15:09:44] (03PS1) 10Faidon Liambotis: Add FIXME with a removal date for gettingStarted [operations/puppet] - 10https://gerrit.wikimedia.org/r/130612 [15:09:46] (03PS1) 10Faidon Liambotis: Add 2>/dev/null to processEcho maintenance cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/130613 [15:10:03] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add FIXME with a removal date for gettingStarted [operations/puppet] - 10https://gerrit.wikimedia.org/r/130612 (owner: 10Faidon Liambotis) [15:10:15] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Add 2>/dev/null to processEcho maintenance cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/130613 (owner: 10Faidon Liambotis) [15:12:05] are these tickets duplicate? maybe I'm not understanding: https://rt.wikimedia.org/Ticket/Display.html?id=4968 and https://rt.wikimedia.org/Ticket/Display.html?id=4784 [15:14:01] (03PS1) 10Dzahn: fix apache-fast-test for use in eqiad [operations/puppet] - 10https://gerrit.wikimedia.org/r/130614 [15:15:22] chasemp: yea, mostly duplicate. one was was general "puppetize that some time", and the other specifically "move it away from pmtpa host" for Tampa decom [15:15:57] ok I'm going to close https://rt.wikimedia.org/Ticket/Display.html?id=4968 in favor of https://rt.wikimedia.org/Ticket/Display.html?id=4784 then as a dupe [15:15:59] cool? [15:16:18] yea, that works. or we could merge them (in the links section) [15:17:06] chasemp: yes, do that, 4968 wasn't linked to the tracking ticket [15:17:44] ah, heh, it is a dependency, technically correct but taking it too far [15:18:26] ah crap didn't see your merge comment [15:18:29] before I killed it [15:20:04] nah, all fine like that [15:42:11] (03CR) 10Mark Bergsma: "I talked to Chase about this on IRC, and we agreed that in this specific case, having the Wikimedia modified source tree as 'upstream' bra" [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 (owner: 10Rush) [15:58:29] (03PS1) 10Giuseppe Lavagetto: Fixed files and diff leaking. [operations/software] - 10https://gerrit.wikimedia.org/r/130618 [15:59:54] <_joe_> If I took the time to write unit tests, that won't have happened. [16:00:49] (03CR) 10Giuseppe Lavagetto: [C: 032] "I should have known better." [operations/software] - 10https://gerrit.wikimedia.org/r/130618 (owner: 10Giuseppe Lavagetto) [16:06:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 10:04:02 2014 [16:16:00] zomg why is icinga going berzerk re. fundraising boxes... [16:16:14] !log Deploying Id248bd6706f32a on Zuul and reloading service [16:16:21] Logged the message, Master [16:19:45] !log restarting stuck Jenkins [16:19:52] Logged the message, Master [16:41:55] greg-g: New rule of maximum 8 patches for SWATs? [16:42:20] greg-g: If so, who decides which 8? [16:44:06] James_F: the timestamp :) (first come first serve) [16:44:12] Ouch. [16:44:33] with reasonable exceptions for high priority/important bug fixes, of course [16:44:40] * James_F nods [16:45:28] And is this gerrit IDs, submodule patches, or semantic changes? Do config changes count? If I squash all fifteen things I want changed in VE into one cherry-pick, is that one or 15, etc.? :-) [16:45:37] * James_F is just predicting people bickering endlessly. [16:46:10] * YuviPanda squashes all of mediawiki into one commit [16:46:13] yeah, well, you can always work around rules, but the point was that last night was too crazy, need a way to limit that/keep our SWATers sane [16:46:14] (03PS1) 10Matanya: iodine: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130624 [16:46:19] Yeah. [16:46:20] suggestions welcome :) [16:46:27] (it's a first pass) [16:46:29] Jeff_Green: ^ [16:46:31] greg-g: part of it was the size of tgr's patches. it was 12 patches that were like +1 -2 [16:46:34] well, I guess second pass [16:46:41] ebernhardson: /me nods [16:46:44] matanya: wooOoo. reviewing [16:46:50] How about "Greg Almighty is the absolute arbiter of whether something goes or not."? :-) [16:46:59] -Almighty [16:47:11] "His highness Greg ..." [16:47:24] * greg-g bites tounge [16:47:26] "His holiness Greg The Gee" [16:47:36] s/tounge/tonga/? [16:47:39] "His Serene Highness" is probably best. [16:47:58] So..., I can be arbitrator of last resort, but I want the SWATers to feel empowered to do it all themselves without my interaction [16:48:33] part of what would make it easier, is if the patches were all merged and a master branch update was prepared, so regardless of the number of changes to 1 extension its just the 1 master to merge? or could that cause other issues? [16:48:48] for example wikidata had it preped that way yesterday [16:48:58] (but i've never prepped my swat patches that way) [16:48:59] that's how they do it generally, given their setup [16:49:06] (composer etc) [16:49:18] but yeah, more work pre-done the better, in my opinion [16:50:28] (03CR) 10Jgreen: [C: 032 V: 031] iodine: add firewall [operations/puppet] - 10https://gerrit.wikimedia.org/r/130624 (owner: 10Matanya) [16:51:17] greg-g, zero is ready, and i had a q for you - is there a special procedure for PrivateSettings.php depl? [16:51:23] need a few vals for that file [16:54:52] I'm not exactly sure, Reedy ? [16:54:53] ^^ [16:55:53] i think it the same sync-file should work, and since i haven't found any other notes on the topic, will proceede with those assumptions. greg-g, are we good to go? [16:56:01] (03PS4) 10MarkTraceur: Fourth batch of pilot sites for Media Viewer [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/125034 [16:56:11] yurikR: yep, just don't pastebin it ;) [16:56:26] dope, already posted it on github :D [17:05:48] (03CR) 10Aaron Schulz: [C: 032] Removed "TMHTransformFrame" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130565 (owner: 10Aaron Schulz) [17:05:56] (03Merged) 10jenkins-bot: Removed "TMHTransformFrame" pool counter entry [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130565 (owner: 10Aaron Schulz) [17:07:03] !log aaron synchronized wmf-config/PoolCounterSettings-eqiad.php 'Removed "TMHTransformFrame" pool counter entry' [17:07:10] Logged the message, Master [17:09:46] (03PS1) 10Matanya: otrs: fix allowed smtp address [operations/puppet] - 10https://gerrit.wikimedia.org/r/130631 [17:10:20] !log aaron synchronized php-1.24wmf2/thumb.php '93a33d733fa81a9a5396083ded6aa28a74f08a98' [17:10:27] Logged the message, Master [17:11:25] !log yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/ [17:11:31] Logged the message, Master [17:13:56] (03CR) 10Ottomata: [C: 032 V: 032] Fix provision from scratch [operations/puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/130499 (owner: 10Milimetric) [17:16:03] (03CR) 10Jgreen: [C: 032 V: 031] otrs: fix allowed smtp address [operations/puppet] - 10https://gerrit.wikimedia.org/r/130631 (owner: 10Matanya) [17:17:38] !log yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/ [17:17:43] Logged the message, Master [17:19:43] wow, what just happened with cirrussearch ext? [17:20:04] lots of timeouts it seems in fatalmon [17:20:25] greg-g, ^ [17:20:42] marktraceur: Shall I rename you right now? [17:20:57] andrewbogott: wfm [17:21:07] ok, stay tuned... [17:21:15] * marktraceur updates SSH configs [17:21:59] seems like both 24-1 & 24-2 are logging timeouts in search backend [17:24:49] (03PS1) 10Andrew Bogott: Rename mholmquist to marktraceur and sync UID with labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/130633 [17:25:08] andrewbogott: Will it rename my home directory automatically? [17:25:27] marktraceur: I will rename and chown things manually after puppet runs [17:25:30] 'kay. [17:25:37] Check my spelling &c in ^ ? [17:26:21] andrewbogott: You got it all right [17:26:22] manybubbles: ^demon|away ^^ see yurikR's comment [17:26:26] (03CR) 10MarkTraceur: [C: 031] Rename mholmquist to marktraceur and sync UID with labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/130633 (owner: 10Andrew Bogott) [17:26:28] 'k thanks [17:26:54] yurikR: I look [17:27:19] (03CR) 10MaxSem: [C: 04-1] "Just use 'closed' instead of listing every wiki explicitly. Adding a comment with bug name would be also good." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) (owner: 10Withoutaname) [17:27:25] manybubbles, seems like it was a one time fluke - all items are gone from the fatalmon [17:27:56] Reedy, are you still interested in ^^^ ? [17:28:02] (03CR) 10Andrew Bogott: [C: 032] Rename mholmquist to marktraceur and sync UID with labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/130633 (owner: 10Andrew Bogott) [17:28:10] yurikR: probably timed with a load spike - I'm building indexes now and I think I've not being careful enough [17:28:24] (03PS1) 10BBlack: Use whole subnets in squid.php list for XFF acceptance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130634 [17:28:42] marktraceur: the next step is to wait 40 mins for puppet to catch up. Best to steer clear of production in the meantime. [17:28:48] tisk tisk ... [17:29:02] :D [17:29:37] (03PS2) 10BBlack: Use whole subnets in squid.php list for XFF acceptance [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130634 [17:31:43] (03CR) 10Withoutaname: "Thanks, I didn't realize 'closed' was also a keyword." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) (owner: 10Withoutaname) [17:36:29] !log yurik synchronized wmf-config/PrivateSettings.php [17:36:36] Logged the message, Master [17:37:20] !log yurik Added $wmgZeroRatedMobileAccessApiUserName / password [17:37:26] Logged the message, Master [17:37:27] (03PS2) 10Withoutaname: Disable query pages for closed wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130609 (https://bugzilla.wikimedia.org/42436) [17:41:11] (03PS4) 10Rush: debian packaging directory with our details [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 [17:41:13] (03PS5) 10Rush: change .gitreview for debian branch [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130142 [17:46:03] (03CR) 10Mark Bergsma: [C: 031] debian packaging directory with our details [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 (owner: 10Rush) [17:47:19] (03CR) 10Rush: [C: 032 V: 032] debian packaging directory with our details [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130145 (owner: 10Rush) [17:49:10] (03CR) 10Rush: [C: 032 V: 032] "self merging as it's just the .gitreview file" [operations/debs/ircd-ratbox] (debian) - 10https://gerrit.wikimedia.org/r/130142 (owner: 10Rush) [17:56:47] greg-g, not done yet, need a few more minutes, fighting with gerrit [17:57:14] you'll lose [17:57:40] "possile spoofing detected" issue :( [17:58:19] either ru KGB is on my case again, on wmf-config repo is not letting me submit patches [18:01:10] (03PS1) 10Yurik: Switch zero ext to use config from zerowiki instead of meta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130640 [18:01:33] dr0ptp4kt, could you review ^ right away and i will depl it [18:02:14] yurikR looking [18:04:18] dr0ptp4kt, +2 pls [18:05:52] (03CR) 10Dr0ptp4kt: [C: 032 V: 031] "Will need to revert IMMEDIATELY if any problems, though." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130640 (owner: 10Yurik) [18:05:57] gwicke: so, what's wrong with jenkins exactly? [18:06:07] yurikR +2 code review +1 checked [18:06:21] shows as merged [18:07:42] dr0ptp4kt, deploying... [18:08:39] !log yurik synchronized wmf-config/mobile.php [18:08:46] Logged the message, Master [18:09:09] dr0ptp4kt, no errors in logs, hope all's good :) [18:09:31] bd808: ping [18:10:29] greg-g, all seems to be quiet, go ahead with the next depl [18:10:33] paravoid: pong (from a quarterly review meeting) [18:11:18] bd808: greg-g mentioned a request during the SoS for labs; he said precise VM in labs, but I assumed from context (hhvm) that it's trusty that you're requesting? [18:11:50] paravoid: Yes, trusty (14.04) [18:12:01] That was my fault :) [18:12:20] thanks! [18:12:22] My brain wanted one thing and typed another [18:17:09] bblack, hi, any updates on this? https://gerrit.wikimedia.org/r/#/c/126829/ [18:20:14] yurikR: we *just* had the SoS [18:20:19] it was mentioned there [18:20:22] that's why we have them :) [18:42:46] we're getting tons of Google Webmaster Tools about *.zero.wikimedia.org and https [18:42:57] about the certificate not matching, which is expected [18:43:17] the problem is why Google is trying to index https://*.zero.wikimedia.org/ URLs [18:43:20] I really have to go now [18:43:25] yurikR: ^ [18:44:06] can someone from ops make sure this is handled? [18:44:23] paravoid: is that https://rt.wikimedia.org/Ticket/Display.html?id=7395 or unrelated? [18:44:25] hmm, no idea why google is going there [18:45:25] unless google is looking at this patch and treats text as link [18:45:25] https://gerrit.wikimedia.org/r/#/c/130640/1/wmf-config/mobile.php [18:46:08] andrewbogott: that would fix it, but links shouldn't have been public before that gets fixed [18:46:20] if they are public, users are affected [18:46:51] paravoid: so, should I prioritize fixing, or chasing down the premature urls? [18:47:10] andrewbogott: I am around to help and was about to ask the same question [18:47:10] the latter [18:47:35] really got to go now, I'm 47' late :) [18:47:36] ciao [18:48:07] chasemp: unless you already know how to change the cert… do you want to hunt down the urls while I figure the cert out? [18:48:44] andrewbogott: looks like "https://lij.zero.wikipedia.org" and a few other subdomains of zero.wikipedia.org [18:49:29] want to bring this to pm? but in general maybe we should both try to revert whatever change was made first [18:49:35] not sure where to start with it honestly [18:49:39] yurikR: So, what changed? Were links like that valid previously for http, and the issue is just that people are now hitting https? [18:50:54] andrewbogott, links like this should never exist [18:51:17] there is no *.zero.wikimedia.org, only zero. itself, which is a private wiki [18:51:34] yurikR: Ah, so the link is invalid whether http or https [18:51:43] this is different from *.zero.wikipedia [18:51:45] right [18:52:15] any thoughts on why google would be trying to validate something like this https://myv.zero.wikipedia.org [18:53:09] wait, are we talking about .wikimedia.org, .wikipedia.org, or something else? [18:54:49] zero.wikimedia.org -- private configuration wiki for zero project, should never be linked to [18:55:08] mark, RobH, mutante, someone else: Can someone tell me what paravoid means about Google Webmaster Tools? I don't really have the context to understand this problem. [18:55:09] *.zero.wikipedia.org -- a no-images version of the *.m.wikipedia.org [18:56:49] unfortunatelly *.zero.wikipedia.org was never included into the wildcard ssl cert, hence certificate errors [18:57:37] yurikR: ok, I feel like we're going in circles here. [18:58:18] Hits on *.zerowipedia.org are legit? [18:58:25] And *.zero.wikimedia.org not? [18:58:28] Is that the upshot? [18:58:56] *.zerowikipedia.org should be *.zero.wikipedia.org i think - I never looked at hat domain [18:59:31] zero.wikimedia.org is not the same as zero.wikipedia.org - first is our internal config site, second is the wikipedia for zero [19:07:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 10:04:02 2014 [19:07:14] andrewbogott, am i explaining it right? [19:07:35] yurikR: I think so, I just still don't know what the problem is :) Chase and I are digging in [19:09:42] yurikR: are there any changes to zero type stuff today that would explain where google is getting these links to validate? [19:10:21] chasemp, i did do a depl about an hour ago [19:10:46] they seem somewhat random but all cases are https://.zero.wikipedia.org [19:10:49] i am still a bit confused which links you actually get [19:10:58] which can't validate for ssl and google desperately wants it to [19:11:03] what's your email [19:11:07] I will forward you a few [19:11:14] yastrakhan@wikimeedia [19:11:18] wikimedia [19:11:26] or yurik [19:11:27] :) [19:15:16] yurikR: I forwarded you a few of the many [19:15:24] paravoid: I messing around with http://etherpad.wikimedia.org/p/swift-thumbnails (just some conjecturing) [19:15:27] thanks, will look at it shortly [19:15:35] * AaronSchulz goes to lunch [19:15:40] anything in that deployment which could possibly relate? The timeframe does line up [19:15:45] -ish [19:24:16] chasemp, shouldn't, but who knows, stranger things have happened :) [19:25:01] <_joe_> yurikR: in chrome, http://fr.zero.wikipedia.org redirects to https [19:25:16] yurikR: can you please investigate? Or point us to the diff in the config that you deployed? [19:25:19] <_joe_> while in ff or via curl, it works as expected [19:25:39] that is interesting [19:26:10] _joe_, thing is, you shouldn't navigate to fr.zero - that should only work when you are on the carrier's network [19:26:10] <_joe_> maybe someone has given my browser back an HSTS header at some point? [19:26:27] <_joe_> yurikR: so I should just receive an error page [19:26:31] and yes, there might be forceHttps cookie [19:26:45] bounce to m. i think [19:26:50] need to revisit that logic [19:26:52] <_joe_> yurikR: mmmh let me see. [19:28:21] <_joe_> yurikR: we set a forceHTTPS cookie that does that redir even for domains where the SSL is broken? WTF? [19:28:44] exactly :( [19:28:45] <_joe_> is this a known issue? [19:28:56] well, the missing ssl cert is the real issue that should be fixed [19:29:03] ops have said that its in the works i think [19:29:10] <_joe_> of course, and yes it is [19:29:23] <_joe_> still, if this is new, we should correct it [19:29:24] so once it gets deployed, we are golden :) [19:29:33] correct - big if [19:29:39] <_joe_> if it's a long-standing issue, then it's ok [19:29:48] now, have you seen any *.zero.wikimedia.org issues ? [19:30:11] <_joe_> apart from the http -> https redir, no. [19:30:51] I don't seem to be redirected in safari or chrome ? for http://pam.zero.wikipedia.org [19:31:20] <_joe_> chasemp: it's not the browser, you just need to have the forceHTTPS cookie set to 1 [19:31:28] ah [19:33:03] yurikR: did the forceHTTPS change with your recent release? [19:35:59] <_joe_> yurikR: so the situation here is that google spider began to be redirected to https even on *.zero,wikipedia.org [19:36:21] <_joe_> this may be a coincidence with the release, but I am not inclined to beleive it [19:36:29] i meant - have you seen any requests to any subdomains under *.zero.wikimedia.org ? or has it only been the *.zero.wikiPEDIA.org issues? [19:37:22] yurikR: just went through about 20 I have saved, all PEDIA [19:37:30] <_joe_> yurikR: pedias [19:38:00] ok, good, so the initial issue paravoid mentioned was all about pedia, not media. good to know [19:38:18] otherwise it would have been too weird [19:39:53] <_joe_> yurikR: so zero is unavaliable to anyone with forceHTTPS=1 in his cookies; I guess that's what happened to the google spider, hence the alarms [19:40:21] ok, i just checked -- using privacy mode in chrome -- fr.zero.wikipedia.org works as expected [19:40:27] <_joe_> yurikR: still, why do we have this behaviour? is it due to the relelease? if so, we should roll back [19:40:33] i think so too [19:40:40] <_joe_> yurikR: I confirmed with curl, it's just the cookie [19:41:42] where is the changeset for the release [19:41:49] or is that a naive question [19:45:13] yurikR: are you planning to revert your changes? [19:48:22] chasemp, if its just the cookie, than nothing has changed - it has been this way before [19:48:33] can you identify when this started happening? [19:48:56] <_joe_> yurikR: yes, by reading the diffs of releases we may do that I guess [19:49:25] sorry, i meant - when did it showed up in google logs? [19:50:04] <_joe_> yurikR: it seems today :) [19:50:13] i synced new zero code at 17:11:25 [19:50:16] <_joe_> let me see the time it started [19:50:28] yurikR: trying to track down first instance [19:50:44] <_joe_> yurikR: 17:11 UTC? [19:50:48] yep [19:51:11] <_joe_> I have the first email at 7:35 UTC [19:51:19] <_joe_> mh, seems pretty well related [19:51:20] not related :) [19:51:29] <_joe_> yurikR: are you kidding? [19:51:47] <_joe_> it's the result from a spidering activity, it's not a monitoring tool per se :) [19:52:12] hold on, 7:35 AM, whereas depl 5:11pm - seems there is a time paradox at work :) [19:52:36] <_joe_> oh sorry [19:52:40] <_joe_> *17:35 [19:52:47] <_joe_> typo :) [19:52:51] ah, ok, yes, does seem causal [19:53:04] there would be some delay so seems plausible [19:53:13] assuming no other deployments in the recent past before yours [19:55:59] there was really very few changes made to zero, and i think only one related to redirection [19:56:30] link? [19:57:52] https://gerrit.wikimedia.org/r/#/c/129578/ [19:58:13] looking at the logic right now [19:58:26] <_joe_> one may be enough [20:02:31] so maybe I'm crazy but from '$toScheme !== 'https' || $config['enableHttps'] ) && preg_match to... [20:02:49] if ($toScheme !== 'https' || $config['enableHttps'] ) [20:03:01] honestly, the change broke things, regardless if we can determine why it broke [20:03:06] if ( $toHost === 'upload.wikimedia.org' && $config['showImages'] ) [20:03:07] it worked before said change, and now doesnt. [20:03:15] elseif ( preg_match( '/^([^.]+\.)?([^.]+\.[^.]+)\.org$/', $t [20:03:21] it may be in the best interests of everyone to revert that stuff sooner than later ;] [20:03:30] anything matching that regex that has enableHTTPS will be redirected here [20:03:33] yes? [20:03:40] https://gerrit.wikimedia.org/r/#/c/129578/3/includes/PageRenderingHooks.php [20:03:51] my php is not perfect but seems to match what we are seeing [20:03:59] makes sense but its also just me guessing since we know the result [20:04:13] RobH, not exactly - most people don't care about this at all - zero.wikipedia is almost never used, except by a very small (currently sleeping) group from what i remember [20:04:23] yes, but if a change breaks something [20:04:29] letting it sit isnt ideal [20:04:40] i dunno why anyone would argue otherwise [20:04:44] yurikR: that if / elseif block is a logic change where [20:04:50] breaking something for wikipedia zero is not nice =[ [20:05:00] it's gonig to match almost anything with enableHttps ? [20:05:05] seems like a revert is in order [20:05:10] i agree 100% [20:05:15] RobH, true :) but its not really a breakage of browsing, its a breakage of google bot [20:05:25] true enough [20:05:31] that we're aware of so far ;] [20:05:39] disagree, it's redirect to invalid ssl [20:05:39] true ;) [20:05:42] google bot just alerts on it [20:05:46] <_joe_> yurikR: that's a breakage of browsing also!!!! [20:05:46] and indeed, it is https as well [20:05:49] browsers break [20:06:06] _joe_, its a breakage of browsing IF you already got that cookie [20:06:18] <_joe_> ok, so? [20:06:20] chasemp: since you have done the most footwork you should remain point and revert the change. I'll happily +1 it for you so you arent alone [20:06:32] which is what people wouldn't get if they were browsing zero wiki [20:06:56] in any case, i will revert this patch and reexamine it. The problem is that i don't think it will be possible to repo this [20:07:27] RobH: I can do that for sure, I just figured easier to yurik [20:07:33] true! [20:07:33] I don't even have this repo locally yet :) [20:07:47] if yurikR has the moment to do it it would indeed be best [20:07:53] yurikR: thank you, our inboxes are appreciative as well [20:07:54] but he isnt in ops and i dont like telling him to do shite [20:07:55] heh [20:08:20] sorry RobH, didn't see your comment, yes, lets try reverting the patch. [20:08:35] :))) [20:08:36] cool, thank you [20:08:37] hi :) [20:08:55] hi bblack, sorry, minor smoking pile (not burning yet), reverting an issue :) [20:09:54] <_joe_> thanks yurikR [20:10:02] I see that, but still catching up [20:11:18] RobH, chasemp _joe_ https://gerrit.wikimedia.org/r/#/c/130667/ [20:11:33] feel free to +2 )) [20:12:08] the BIG problem is that the issue that patch solves (which works) is much bigger than the SSL cert bug :( [20:12:19] yurikR: sorry if i was coming across heavy handed, wasnt trying to demean the work done for it or anything of that fashion, i just didnt want it to sit broken =] [20:12:47] i think zero is possibly the coolest shit we've done as a new project since wikipedia itself ;] [20:12:57] RobH, totally ok, i agree that we shouldn't break things... beyond reasonable :) [20:13:04] <_joe_> RobH: completely agree :) [20:13:28] greg-g, can i depl now? [20:13:38] looks like it [20:14:25] I verified the revert in gerrit, but I am unsure of the deploy mechanisms under the hood [20:14:58] we're in the middle of parsoid deployment window on the schedule [20:15:04] but i'd still push this now personally [20:15:21] yurikR: so for what its worth, i think its ok to deploy now ;] [20:15:30] gwicke: you doing parsoid now or nah? [20:15:54] (he is signed up for the parsoid window from 20:00 to 21:00 GMT) [20:16:11] yurikR: if none of them answer i think it means that window isnt happening (seems legit to me) [20:16:26] oki, sec [20:16:34] and thanks!!! [20:16:36] all of you ;0 [20:16:37] :) [20:16:48] thanks for deploying so we dont have to touch scap ;] [20:18:27] So for anyone who wondered how I knew what deployment was going on (99.99999% already know this but never hurts to repeat) https://wikitech.wikimedia.org/wiki/Deployments [20:18:44] and yurikR was just following procedure by asking greg, as he is the master of the deployment schedule [20:18:57] RobH: good to know thanks [20:19:14] RobH, we canceled today as Jenkins was broken & rt testing showed an issue that we could not fix in time [20:19:23] so all yours [20:19:26] gwicke: cool, just wanted to ensure we didnt step on you during it, thanks! [20:19:50] stepping onto us would require you to deploy Parsoid ;) [20:21:27] marktraceur: your rename is (finally) finished. Please let me know if you run into any trouble. [20:21:32] yea, i didnt think it would be an issue [20:22:51] but i like to err on the side of 'touching this will cause all servers to burst into flame' side of things. [20:25:27] it takes forever to get any revert merged :( [20:26:13] andrewbogott: I've been running things without issue :) thansk [20:26:17] thanks even [20:26:26] yurikR: waiting for reviewers? cuz if so im happy to go harass folks =] [20:26:57] waiting for the bot :) [20:27:27] jenkins is just not as fast as it used to be :( [20:27:35] i have no goats to sacrifice to the zuul gods at this time. [20:27:48] ran out doing ssl submission changes during athens. [20:28:50] jenkins seems slow to me in general, or not speedy anyway [20:31:57] !log yurik synchronized php-1.24wmf1/extensions/ZeroRatedMobileAccess/ [20:33:42] (03CR) 10JGonera: "All required patches in VectorBeta have been merged." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130266 (owner: 10JGonera) [20:34:16] almost done reverting, please let me know if SSL cert issues stop [20:34:57] !log yurik synchronized php-1.24wmf2/extensions/ZeroRatedMobileAccess/ [20:38:04] chasemp: ^ you were seeing the errors, i didnt try it since i was doing ssl cert csr generation [20:42:09] RobH, chasemp _joe_ could you email me if the issue resurfaces? 1am here, falling asleep :( dr0ptp4kt might be able to tell you more about the redirect logic [20:42:42] yurikR: I will email if I see more of it, I'll be around for awhile on email at the very least [20:42:49] dr0ptp4kt, if you missed the issue - we are getting fludded with bad SSL cert for *.zero.wp [20:43:10] apparently there is a redirect somewhere somehow to switch to https [20:43:18] yurikR, i'm on the google webmaster tool notifications. it's part of their bots. [20:43:38] https://rt.wikimedia.org/Ticket/Display.html?id=7071 [20:43:59] just fyi I think RobH is in the process of that now [20:44:08] (03PS1) 10Jforrester: Enable VE language editor Beta Feature in whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130717 [20:44:10] (03PS1) 10Jforrester: Remove Nearby BF from whitelist [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130718 [20:44:12] (03PS1) 10Jforrester: Remove VE formula editor from BF whitelist (graduated) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130719 [20:45:13] (03CR) 10Jforrester: [C: 04-1] "Do not merge until wmf3 is everywhere (8 May)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130719 (owner: 10Jforrester) [20:45:49] dr0ptp4kt: indeed, i put in the submission for that about 20 minutes ago and plan to roll it live tomorrow [20:46:03] RobH, thanks man [20:46:11] sorry it took so long, i just had a lot of other items taking up my attention, tampa shutdown, athens meet, etc... but it still shoudnt have taken this long ;_; [20:46:49] RobH, it will be okay. i know you're swamped. [20:47:11] i appreciate the understanding [20:48:29] (03PS1) 10BBlack: Fix IPv6 DNS entries for neon and mchenry [operations/dns] - 10https://gerrit.wikimedia.org/r/130731 [20:50:24] (03CR) 10BBlack: [C: 032 V: 032] Fix IPv6 DNS entries for neon and mchenry [operations/dns] - 10https://gerrit.wikimedia.org/r/130731 (owner: 10BBlack) [21:26:58] (03PS1) 10Andrew Bogott: Rename tfinc to tomasz to match labs. Also change uid to 1155. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130740 [21:30:27] (03PS2) 10Andrew Bogott: Rename tfinc to tomasz to match labs. Also change uid to 1155. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130740 [21:33:36] (03CR) 10Andrew Bogott: [C: 032] Rename tfinc to tomasz to match labs. Also change uid to 1155. [operations/puppet] - 10https://gerrit.wikimedia.org/r/130740 (owner: 10Andrew Bogott) [21:38:07] (03PS1) 10BBlack: Remove 4x dead hostnames from DNS in esams [operations/dns] - 10https://gerrit.wikimedia.org/r/130750 [21:39:18] (03CR) 10BBlack: [C: 032 V: 032] Remove 4x dead hostnames from DNS in esams [operations/dns] - 10https://gerrit.wikimedia.org/r/130750 (owner: 10BBlack) [21:46:48] https://ganglia.wikimedia.org/latest/?c=Swift%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [21:46:48] a lot of hosts red, why? [22:02:08] (03CR) 10Ori.livneh: Add another check using graphite, small fixes. (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130582 (owner: 10Giuseppe Lavagetto) [22:04:45] (03CR) 10Giuseppe Lavagetto: Add another check using graphite, small fixes. (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/130582 (owner: 10Giuseppe Lavagetto) [22:08:12] PROBLEM - Puppet freshness on osmium is CRITICAL: Last successful Puppet run was Wed Apr 30 10:04:02 2014 [22:20:22] (03PS6) 10BBlack: Update Zero netmapper data from zero.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/126829 [22:37:04] (03PS1) 10Chad: Configure Swift-backed elasticsearch backups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 [22:37:30] (03CR) 10Chad: [C: 04-2] "Cannot be merged until plugin is live, wanted to get config up for review though." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 (owner: 10Chad) [22:38:09] (03PS7) 10BBlack: Update Zero netmapper data from zero.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/126829 [22:41:17] (03PS2) 10Chad: Configure Swift-backed elasticsearch backups [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/130760 [22:43:08] (03CR) 10BBlack: [C: 032 V: 032] "Giving it a spin" [operations/puppet] - 10https://gerrit.wikimedia.org/r/126829 (owner: 10BBlack) [22:45:25] ok that didn't work :P [22:52:57] (03PS1) 10BBlack: Rename manifest to fix class-lookup issue? [operations/puppet] - 10https://gerrit.wikimedia.org/r/130763 [22:53:15] (03CR) 10BBlack: [C: 032 V: 032] Rename manifest to fix class-lookup issue? [operations/puppet] - 10https://gerrit.wikimedia.org/r/130763 (owner: 10BBlack) [22:55:48] (03PS1) 10BBlack: fix pathnames for /etc/zerofetcher [operations/puppet] - 10https://gerrit.wikimedia.org/r/130764 [22:56:14] (03CR) 10BBlack: [C: 032 V: 032] fix pathnames for /etc/zerofetcher [operations/puppet] - 10https://gerrit.wikimedia.org/r/130764 (owner: 10BBlack) [23:04:19] SWAT time! [23:04:51] RoanKattouw: you doing it or should i? [23:05:00] i don't mind [23:05:11] If you wanna do it, go for it [23:05:11] wee [23:05:23] I'll finish merging things in VE:wmf2 [23:05:38] There, done [23:05:44] cool, i'm on it [23:05:58] ori: All you need to do now is a standard extension update for VE to the wmf2 branch [23:06:15] There was a change in the inner submodule as well so be sure to recursively update it on tin [23:07:44] * ori nods [23:13:50] * greg-g goes to the bus stop [23:13:59] (which is way less cool than the car wash) [23:14:36] !log ori synchronized php-1.24wmf2/extensions/VisualEditor 'Ibaf0cc823bfe: Update VisualEditor for cherry-picks' [23:14:40] ^ RoanKattouw [23:14:45] Logged the message, Master [23:14:48] Thanks man [23:14:56] np [23:16:22] (03PS1) 10BBlack: Fix a couple small bugs in zerofetch.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/130766 [23:18:03] (03CR) 10jenkins-bot: [V: 04-1] Fix a couple small bugs in zerofetch.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/130766 (owner: 10BBlack) [23:19:31] (03PS2) 10BBlack: Fix a couple small bugs in zerofetch.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/130766 [23:21:53] (03CR) 10BBlack: [C: 032 V: 032] Fix a couple small bugs in zerofetch.py [operations/puppet] - 10https://gerrit.wikimedia.org/r/130766 (owner: 10BBlack) [23:40:35] (03PS1) 10BBlack: zerofetch.py: satisfy pep8 while actually still working [operations/puppet] - 10https://gerrit.wikimedia.org/r/130777 [23:41:22] (03CR) 10BBlack: [C: 032 V: 032] zerofetch.py: satisfy pep8 while actually still working [operations/puppet] - 10https://gerrit.wikimedia.org/r/130777 (owner: 10BBlack) [23:47:28] (03PS1) 10BBlack: use carriers.json for zero data [operations/puppet] - 10https://gerrit.wikimedia.org/r/130780 [23:51:00] (03CR) 10BBlack: [C: 032 V: 032] use carriers.json for zero data [operations/puppet] - 10https://gerrit.wikimedia.org/r/130780 (owner: 10BBlack) [23:57:46] ( Reedy: TimStarling: mark: ori: ) [23:58:01] yes? [23:58:47] Krinkle: Krinkle: Krinkle: Krinkle [23:58:57] !log mobile caches now sync zero carriers/proxies from zero.wm.org rather than noc(fenari) temp hack solution [23:59:04] Logged the message, Master