[00:14:13] (03CR) 10Ori.livneh: [C: 032] role::performance: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163368 (owner: 10BryanDavis) [00:14:58] (03CR) 10Ori.livneh: [C: 032] ocg: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163375 (owner: 10BryanDavis) [00:24:53] (03PS4) 10Ori.livneh: role::parsoid: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163373 (owner: 10BryanDavis) [00:24:58] (03CR) 10Ori.livneh: [C: 032 V: 032] role::parsoid: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163373 (owner: 10BryanDavis) [00:33:46] (03PS4) 10Ori.livneh: role::elasticsearch: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163364 (owner: 10BryanDavis) [00:34:02] (03CR) 10Ori.livneh: [C: 032 V: 032] role::elasticsearch: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163364 (owner: 10BryanDavis) [00:34:05] go ori go :) [00:37:29] (03CR) 10Ori.livneh: [C: 032 V: 032] role::analytics: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163366 (owner: 10BryanDavis) [00:38:45] (03PS4) 10Ori.livneh: role::ci::slave: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163365 (owner: 10BryanDavis) [00:41:32] (03CR) 10Ori.livneh: [C: 032] role::ci::slave: Convert to package<|provider==trebuchet|> [puppet] - 10https://gerrit.wikimedia.org/r/163365 (owner: 10BryanDavis) [00:43:51] (03PS4) 10Ori.livneh: Remove deployment::target [puppet] - 10https://gerrit.wikimedia.org/r/163376 (owner: 10BryanDavis) [00:43:58] (03CR) 10Ori.livneh: [C: 032 V: 032] Remove deployment::target [puppet] - 10https://gerrit.wikimedia.org/r/163376 (owner: 10BryanDavis) [00:44:57] bd808: it's gone! thanks [00:45:10] w00t! [00:45:11] i applied each of those on the relevant hosts to confirm [00:51:14] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 313 seconds [00:51:47] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 348 seconds [00:52:47] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [00:53:27] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:54:29] humph [01:16:13] (03PS1) 10Dzahn: ssl_ciphersuite - add new compat mode [puppet] - 10https://gerrit.wikimedia.org/r/166710 [02:15:32] !log LocalisationUpdate completed (1.25wmf2) at 2014-10-15 02:15:31+00:00 [02:15:41] Logged the message, Master [02:28:00] !log LocalisationUpdate completed (1.25wmf3) at 2014-10-15 02:28:00+00:00 [02:28:09] Logged the message, Master [02:32:26] (03PS2) 10Ori.livneh: mediawiki::monitoring::webserver: tidy [puppet] - 10https://gerrit.wikimedia.org/r/166196 [02:32:34] (03CR) 10Ori.livneh: [C: 032 V: 032] mediawiki::monitoring::webserver: tidy [puppet] - 10https://gerrit.wikimedia.org/r/166196 (owner: 10Ori.livneh) [03:34:50] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Oct 15 03:34:50 UTC 2014 (duration 34m 49s) [03:34:58] Logged the message, Master [06:22:00] PROBLEM - Apache HTTP on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:22:20] PROBLEM - HHVM rendering on mw1114 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:50] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.050 second response time [06:24:11] RECOVERY - HHVM rendering on mw1114 is OK: HTTP OK: HTTP/1.1 200 OK - 67869 bytes in 0.185 second response time [06:28:29] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [06:28:30] PROBLEM - puppet last run on db2036 is CRITICAL: CRITICAL: puppet fail [06:28:40] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: puppet fail [06:28:50] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: puppet fail [06:28:51] PROBLEM - puppet last run on amssq46 is CRITICAL: CRITICAL: puppet fail [06:29:09] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:21] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:29] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:33] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:39] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:40] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:40] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:50] PROBLEM - puppet last run on amssq47 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:00] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:19] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:20] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:20] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:21] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:29] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:30] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:39] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:39:10] PROBLEM - puppet last run on db1027 is CRITICAL: CRITICAL: Puppet has 1 failures [06:43:09] PROBLEM - Host mw1205 is DOWN: PING CRITICAL - Packet loss = 100% [06:45:09] ACKNOWLEDGEMENT - RAID on db1051 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) Sean Pringle RT 8650 [06:45:36] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:39] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:45:49] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:11] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:46:20] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:34] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:46:35] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 62 seconds ago with 0 failures [06:46:39] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:50] RECOVERY - puppet last run on db2036 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:46:59] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:47:00] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:47:09] RECOVERY - puppet last run on amssq46 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:47:10] RECOVERY - puppet last run on amssq47 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:47:19] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:47:20] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:48:30] (03PS1) 10ArielGlenn: beta: bastion ssh rule needs the class that defines bastion ip [puppet] - 10https://gerrit.wikimedia.org/r/166717 [06:49:41] (03CR) 10ArielGlenn: [C: 032] beta: bastion ssh rule needs the class that defines bastion ip [puppet] - 10https://gerrit.wikimedia.org/r/166717 (owner: 10ArielGlenn) [06:55:50] RECOVERY - puppet last run on db1027 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:58:43] <_joe_> !log restarting hhvm on mw1114 to avoid memory exhaustion [06:58:50] Logged the message, Master [07:26:55] (03CR) 10TTO: "No results for `git grep langlist` in operations/puppet, and it looks like operations/dns has its own language list [1]. I couldn't see a " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/166281 (https://bugzilla.wikimedia.org/43697) (owner: 10TTO) [07:39:19] PROBLEM - puppet last run on lvs3002 is CRITICAL: CRITICAL: puppet fail [07:40:19] _joe_: memleak ftw! [07:41:25] <_joe_> paravoid: :/ [07:52:13] !log ongoing schema changes rev_content_(model|format) multiple shards, ok to kill osc_host.sh jobs on terbium in emergency [07:52:23] Logged the message, Master [07:55:12] (03CR) 10Giuseppe Lavagetto: [C: 031] "LGTM, but care to explain the changes in the chiphersuite?" [puppet] - 10https://gerrit.wikimedia.org/r/166710 (owner: 10Dzahn) [07:59:00] RECOVERY - puppet last run on lvs3002 is OK: OK: Puppet is currently enabled, last run 65 seconds ago with 0 failures [08:01:11] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "One small implementation detail, but also - we should be careful with this on the API cluster, where I have repeatedly seen servers in a b" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/130296 (owner: 10ArielGlenn) [08:06:32] !log Jenkins: upgrading Gearman plugin to Patchset 9 of https://review.openstack.org/#/c/125755/ [08:06:40] Logged the message, Master [08:09:40] !log restarting Jenkins [08:09:45] Logged the message, Master [08:10:26] (03PS4) 10Giuseppe Lavagetto: rsync: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/159462 (owner: 10Matanya) [08:40:57] (03PS5) 10Giuseppe Lavagetto: rsync: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/159462 (owner: 10Matanya) [08:59:04] (03CR) 10Giuseppe Lavagetto: [C: 032] "http://puppet-compiler.wmflabs.org/419/change/159462/html/" [puppet] - 10https://gerrit.wikimedia.org/r/159462 (owner: 10Matanya) [09:11:53] (03CR) 10Filippo Giunchedi: [C: 031] ssl_ciphersuite - add new compat mode [puppet] - 10https://gerrit.wikimedia.org/r/166710 (owner: 10Dzahn) [09:13:55] (03CR) 10Filippo Giunchedi: [C: 04-1] "shouldn't this be already in the debian package?" [puppet] - 10https://gerrit.wikimedia.org/r/166686 (owner: 10John F. Lewis) [09:15:18] paravoid: re: the ms-be1013 issues on friday, it turns out sdl was failing too :( [09:24:19] !log enable container sync for commons containers [09:24:26] Logged the message, Master [09:38:05] godog: :( [09:44:12] (03CR) 10Hashar: [C: 04-1] import LogFormat s from apache2 package [puppet] - 10https://gerrit.wikimedia.org/r/162541 (owner: 10Jeremyb) [09:54:52] (03PS2) 10Filippo Giunchedi: Fix doc header for ensure_link [puppet] - 10https://gerrit.wikimedia.org/r/159174 (owner: 10BryanDavis) [09:54:59] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] Fix doc header for ensure_link [puppet] - 10https://gerrit.wikimedia.org/r/159174 (owner: 10BryanDavis) [10:27:28] <_joe_> !log depooling mw1114, stopping puppet for debugging purposes [10:27:35] Logged the message, Master [10:35:28] (03PS1) 10ArielGlenn: Revert "Remove deployment::target" [puppet] - 10https://gerrit.wikimedia.org/r/166720 [10:35:34] PROBLEM - Apache HTTP on mw1114 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.031 second response time [10:35:42] (03PS2) 10ArielGlenn: Revert "Remove deployment::target" [puppet] - 10https://gerrit.wikimedia.org/r/166720 [10:38:35] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.066 second response time [10:40:04] (03CR) 10ArielGlenn: [C: 032] Revert "Remove deployment::target" [puppet] - 10https://gerrit.wikimedia.org/r/166720 (owner: 10ArielGlenn) [10:40:52] ew on that commit message :) [10:40:59] not line-wrapped [10:47:16] <_joe_> !log repooled mw1114 with reduced load, using jemalloc with prof_leak enabled for sampling. will depool again soon [10:47:21] Logged the message, Master [10:48:48] (03PS1) 10ArielGlenn: use the old deployment::target class for trebuchet test repo [puppet] - 10https://gerrit.wikimedia.org/r/166729 [10:51:01] (03CR) 10ArielGlenn: [C: 032] use the old deployment::target class for trebuchet test repo [puppet] - 10https://gerrit.wikimedia.org/r/166729 (owner: 10ArielGlenn) [10:52:23] PROBLEM - HHVM rendering on mw1114 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.007 second response time [10:53:12] PROBLEM - Apache HTTP on mw1114 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50350 bytes in 0.011 second response time [10:54:03] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.138 second response time [10:54:35] RECOVERY - HHVM rendering on mw1114 is OK: HTTP OK: HTTP/1.1 200 OK - 67869 bytes in 0.585 second response time [10:59:50] !log AMS-IX renumbering: adding second IP, peering with RS1 [10:59:56] Logged the message, Master [11:38:04] <_joe_> !log depooling mw1114 again [11:38:12] Logged the message, Master [11:51:42] PROBLEM - puppet last run on amssq62 is CRITICAL: CRITICAL: puppet fail [11:52:35] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [12:05:54] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [12:11:23] RECOVERY - puppet last run on amssq62 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [12:15:18] (03CR) 10Ori.livneh: "This aspect of Trebuchet's design is pretty dumb. Rather than map a deployment target to an arbitrary string that is usually the same as t" [puppet] - 10https://gerrit.wikimedia.org/r/166720 (owner: 10ArielGlenn) [12:21:16] (03CR) 10Ori.livneh: "(This becomes pretty obvious when you look at the config file: . It's pretty useless.)" [puppet] - 10https://gerrit.wikimedia.org/r/166720 (owner: 10ArielGlenn) [12:29:41] (03PS1) 10Ori.livneh: trebuchet: derive the grain name from the repo name [puppet] - 10https://gerrit.wikimedia.org/r/166736 [12:29:59] ^ apergos [12:30:24] that means people cannot use one grain for several repos [12:30:49] which was a feature of the previous setup [12:30:55] ori_: [12:31:12] not really a feature, if you ask me [12:31:27] how's the offsite, ori? [12:31:27] where is it anyway? :) [12:31:36] well hashar uses it, as an example [12:31:51] san diego [12:31:51] mark: so-so, tbh [12:32:20] several integration repos, one grain to cover them instead of a grain for each [12:32:23] apergos: because the relationship between the repositories ends up being declared in a giant hash blob in the configuration of trebuchet in role::deployment. the proper way to do it would be to enclose them in the role where they belong. [12:32:25] ah, california, but at least not san francisco [12:33:26] yes, having them in the role would be ok, as long as the same flexbility is there [12:33:35] apergos: what makes more sense: to have role::apt, in which you declare apt::virtual_package { 'integration': packages => ['foo', 'bar', buzz'] } [12:33:43] (03CR) 10CSteipp: [C: 031] ssl_ciphersuite - add new compat mode [puppet] - 10https://gerrit.wikimedia.org/r/166710 (owner: 10Dzahn) [12:33:56] or to have role::integration, in which you have: package { [ 'foo', 'bar', buzz' ]: } [12:34:05] so your offsite is still going on at 5am? [12:34:33] !log Zuul frozen \O/ [12:34:39] Logged the message, Master [12:35:20] my gut feeling is to go for the second tbh [12:35:34] yeah, i think so too [12:35:37] but you should probbly check with a couple ctive users of the system [12:37:17] I see the '' key is bck to its old tricks... [12:37:22] * apergos glres at the keyboard [12:37:22] do you have any spare time to help me with that? i'm not sure i have the time to do that adequately [12:37:52] ori_: I am happy to a) chat about it with you, b) review stuff, test stuff [12:38:01] if you need a c) you will hve to sk [12:38:43] lso do you want a bz report for this? [12:38:46] c) but you should probbly check with a couple ctive users of the system [12:39:01] yep I cn do that also [12:39:15] what do you think is appropriate? imo, we can just perform this migration [12:39:35] if it's not going to tke more than a day I sy don't bother with the report [12:39:44] long as it's documented in gerrit, which it is [12:39:58] it's back! (the 'a') [12:40:17] let me not push my luck :-D [12:40:24] chasemp: hah, we can't actually use shinken autodiscovery, since that's with 2.0 and we don't have packages for that... [12:40:30] 2.0 package just landed on sid... [12:40:42] !log disabled/reenabled gearman plugin at https://integration.wikimedia.org/ci/manage [12:40:50] apergos: george perec wrote an entire book in french without using the letter 'e', and someone translated it to english http://en.wikipedia.org/wiki/A_Void [12:40:50] Logged the message, Master [12:41:52] "Augustus, who has had a bad night, sits up blinking and purblind. Oh what was that word (is his thought) that ran through my brain all night, that idiotic word that, hard as I'd try to pun it down, was always just an inch or two out of my grasp - fowl or foul or Vow or Voyal? - a word which, by association, brought into play an incongruous mass and magma of nouns, idioms, slogans and sayings, a confusing, amorphous outp [12:41:52] ouring which I sought in vain to control or turn off but which wound around my mind a whirlwind of a cord" ... [12:42:14] might have to read that! an odd variant is to write without any gender pronouns and have it be natural [12:42:51] nice that the translators stuck to the same rule [12:43:13] yes, it's almost a greater achievement, because it adds an additional constraint [12:43:30] (you have to be faithful to the original *and* avoid the letter) [12:44:34] yep [12:44:51] mark: we were supposed to come up with a mission statement on day 1, day 2 ended without success and arthur gave up and told us to do it at home [12:45:11] it's now homework for robla [12:45:19] * ori_ isn't making this up [12:46:13] ok [12:46:20] while True should always be banned [12:46:24] i see [12:46:29] wrong chan [12:46:32] RECOVERY - RAID on db1051 is OK: OK: optimal, 1 logical, 2 physical [12:49:27] 'while 1:' is faster, but i doubt that's what you meant [12:49:43] clearly you must recurse upon yourself and make sure your language supports tail calls [12:49:55] err, optimizes tail calls [12:51:24] ocaml! [12:53:04] assembly! [12:55:11] YuviPanda: it really is faster (at least in cpython), if only nominally: https://dpaste.de/pLXj/raw [12:55:29] ori_: don't you need to change the custom provider to add the grain with the repo name instead of the base name? [12:55:53] haha :D [12:55:59] apergos: probably [12:56:59] YuviPanda: it's because you can reassign True in the loop [12:57:08] so it has to check at every iteration [12:57:11] wait, you can re-assign True in python? [12:57:16] yes [12:57:22] True = False [12:57:23] oh god [12:57:24] is valid [12:57:25] I didn't realize [12:57:27] just tried [12:58:03] ori_: can't re-assign 1, I guess. [12:58:04] hence LOAD_GLOBAL, POP_JUMP_IF_FALSE on each iteration [12:58:42] right, that makes sense... [12:58:49] but True being assignable, not so much... [12:59:00] hmm, so if I set True = False, I can't get True back... [12:59:01] oh wait [12:59:02] of course I can [12:59:09] True = (True == False) [12:59:11] would get that back [12:59:22] and it does [12:59:55] or 'True = not True' [13:00:45] we need an icinga alert for that ;) [13:01:45] CRITICAL: I don't even [13:02:38] apergos: i'll amend the patch when i'm actually awake [13:02:41] SANITY is CRITICAL: world ending [13:03:07] I can't believe you're even on line now [13:25:33] (03CR) 10ArielGlenn: "also needs modification of the trebuchet provider to use the repo name for the grain instead of the base." [puppet] - 10https://gerrit.wikimedia.org/r/166736 (owner: 10Ori.livneh) [13:29:30] !log dist-upgrade and reboot indium [13:29:38] Logged the message, Master [13:32:33] any SWATters around? [13:32:46] I'm unsure how to cherry-pick two patches that are dependent on each other to the branch... [13:32:50] I suppose I could just squash them [13:33:10] ? [13:33:18] just cherry pick them both? [13:33:45] Reedy: I cherry picked https://gerrit.wikimedia.org/r/#/c/166543/ but attempting to cherry-pick https://gerrit.wikimedia.org/r/#/c/166673/ shows a merge conflict [13:34:06] do it locally? [13:34:14] lazy :P [13:34:15] but fine [13:34:39] you might not even have to rebase or anything locally [13:34:44] if it's just jgit being shit [13:35:02] Reedy: I'll need to give 'em new change-ids tho [13:35:04] but that's not too hard [13:35:09] No you won't [13:35:36] oh? [13:35:53] if it's on a different branch/repo it can share a change-id [13:36:23] https://gerrit.wikimedia.org/r/#/q/I85d857f018e759cec4fc0e04ee6f195242096704,n,z [13:37:53] Reedy: yeah, done. thanks [13:40:22] Reedy: since you're around (what is it with you guys, all on line at ridiculous-o-clock?)... any idea who manages deployment-memc02 and 03 in beta? asking you cause your nme was on a changeset with them in it :-P [13:40:41] apergos: I'm at home [13:40:48] I didn't get to go to the core offsite :( [13:40:56] ah, ugh [13:40:57] mark___: When are you doing the Ams-ix renumbering thing? You're one of the few peers still in the old range [13:41:15] multichill|work: paravoid did some stuff earlier [13:41:44] we are in the process of renumbering, yes [13:42:07] what do you mean "still in the old range" though [13:42:07] apergos: Anyway. I don't think anyone specifically manages them. Just a handful of people that will do stuff if necessary. Wassup? [13:42:10] we're not, we like the old range better [13:42:15] most peers are in both ranges atm [13:42:23] I see they hve a salt mster running on them, not that their minions talk to it... [13:42:28] seemed pretty bizarre to me [13:42:29] I know, I'm in both too [13:43:03] apergos: Sounds like someone got box checking happy or something [13:43:07] On a Juniper the change is very easy [13:43:21] you who? [13:43:21] apergos: With the dedicated salt/puppet master, seems a bit redundant for them to be salt masters too [13:43:24] I would just shoot it and deinstall tht package (I looked at the configure list .. course, if someone checked it and then unchecked it the package would be round forever) [13:43:58] I'm gonna go ahead and toss then [13:44:05] WFM [13:44:49] Who are you refering to paravoid? I'm 195.69.144.252 / 80.249.208.252 / AS1126 ;-) [13:45:15] aka "knams" to us ;) [13:45:21] ah, I didn't know [13:45:43] although that's a -really- misleading name now [13:46:03] For a change I'm actually sitting in the DC. It's still in Amsterdam and Kennisnet is still a customer [13:46:39] true, although we have nothing to do with kn anymore [13:55:42] mark: You're welcome to rename it to vaams :P [13:55:46] We call it asd01 [13:55:54] we don't rename ever ;) [13:56:10] too much work for too little gain, hehe [14:00:46] Hi all, it looks like some e-mail sent to our OTRS queue get lost. Does someone know if there's any way to figure out what is wrong? [14:01:21] Which queue is that? [14:01:28] Reedy: wm-cz@wikimedia.org [14:01:49] Do you know of any specific emails that are missing? ie from a certain address maybe? (you don't need to post it publicly) [14:02:50] Reedy: reportedly this one: Oct 14 23:10:56 pravy postfix/smtp[30275]: 35C30400B7: to=, orig_to=, relay=polonium.wikimedia.org[208.80.154.90]:25, delay=2.4, delays=0.02/0.01/1.3/1.1, dsn=2.0.0, status=sent (250 OK id=1Xe9Mt-0001j2-QK) [14:03:20] Reedy: (the timezone is UTC+2) [14:04:37] Reedy: these are relayed from our mailserver through info@wikimedia.cz address [14:06:03] mark: True, we had to rename .sara.nl to vancis.net, I just put all the new stuff in the new domain and kept the old stuff in the old one [14:06:17] that's what we did too [14:06:30] i think we just finally got rid of the .knams. that were actually moved to esams back in 2008 ;) [14:07:01] I remember that. Around christmas and some battle scars? [14:07:26] yep [14:09:43] Something else Mark. Talked with Sillke the other day when I was in Berlin. She doesn't really have a good donor for the recent servers. You any ideas or want to claim it? [14:09:53] Three decent recent Dell servers + storage [14:10:50] i already replied to her once [14:11:04] yeah those recent ones we could take, even though we can'd do all that much in amsterdam with it [14:11:06] but can't hurt to mount it there [14:11:12] the rest is... scrap metal [14:11:33] <_joe_> !log powercycling mw1205, down since this morning, console blank [14:11:40] Logged the message, Master [14:12:16] !log reimaging mw1035 for great justice!!! (HHVM) [14:12:21] Logged the message, Master [14:13:17] In de Waarderpolder there's a company that buys old hardware, used them once or twice. Could be used for the scrap metal so at least it won't cost any money [14:14:23] RECOVERY - Host mw1205 is UP: PING OK - Packet loss = 0%, RTA = 1.38 ms [14:14:30] yup, i offered her to take care of that [14:14:38] we have some old equipment that needs to go too [14:14:56] Good to bundle, you might even get some money or toys from it [14:15:18] i was waiting for the toolserver stuff anyway [14:15:22] might as well do it all at once [14:15:26] it's just sitting in a rack now [14:16:36] !log AMS-IX renumbering: peering with (renumbered) top-10 ASNs + ASNs with large number of prefixes [14:16:45] Logged the message, Master [14:19:29] The old Dell squid servers? [14:19:38] those too indeed [14:19:41] some of them [14:20:06] !log Not reimaging mw1035 after all; hhvm is in our base, killing our ramz. [14:20:11] Logged the message, Master [14:20:17] Damn, classics! Making room for some new caching servers? [14:22:14] not really room we need soon [14:32:53] !log AMS-IX renumbering: move all remaining ASNs to the new space [14:33:01] Logged the message, Master [14:35:36] (03PS1) 10Filippo Giunchedi: ganglia: use install2001 public name [puppet] - 10https://gerrit.wikimedia.org/r/166746 [14:36:07] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] ganglia: use install2001 public name [puppet] - 10https://gerrit.wikimedia.org/r/166746 (owner: 10Filippo Giunchedi) [14:37:44] OK...let's see what's on the SWATdar today [14:38:13] * YuviPanda waves at marktraceur [14:38:20] I've a few things for SWAT, but they're for wikitech! [14:38:30] I have no idea what that means [14:38:32] hopefully andrewbogott comes by in the meantime so he can run sync-common [14:38:37] Ah. [14:38:47] So I can get it up but it won't do anything, basically? [14:38:53] yeah, but that's ok. [14:39:00] if andrew doesn't turn up I can always bug Core.n [14:39:23] Looks like editing have been busy. [14:39:33] Nothing too complicated [14:40:37] PROBLEM - Host payments1003 is DOWN: PING CRITICAL - Packet loss = 100% [14:41:56] oops, forgot to enable down-for-maintenance before rebooting payments1003, pls ignore ^^^ [14:42:14] ok [14:45:29] RECOVERY - Host payments1003 is UP: PING OK - Packet loss = 0%, RTA = 0.76 ms [14:52:24] (03CR) 10Alexandros Kosiaris: [C: 032] Beta: Define cxserver port [puppet] - 10https://gerrit.wikimedia.org/r/166542 (owner: 10KartikMistry) [14:52:53] marktraceur: I'm stepping away for a few mins but should be back in 10 [14:53:07] KK [14:53:13] I can do James's stuff first [14:54:35] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Added initial Debian packaging [debs/contenttranslation/apertium-en-es] - 10https://gerrit.wikimedia.org/r/165471 (owner: 10KartikMistry) [14:59:13] All right, it's time to play everyone's favorite game show: Sync or Swim! I'm your host marktraceur, here with the lovely Scap Dancers, ready for a fun morning ahead. [14:59:44] Our first contestant is James_F|Away. Oh, dear, has he stepped out to the restroom? [14:59:54] I was going to ask you to do yuvi's patches first, but maybe James has dibs [15:00:05] manybubbles, anomie, ^d, marktraceur: Respected human, time to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141015T1500). Please do the needful. [15:00:09] andrewbogott: YuviPanda's not here for about 2 more minutes [15:00:23] marktraceur: I'll stand in for James [15:00:26] But right now neither of...aha [15:00:29] I wrote the patches anyway :) [15:00:32] OK, RoanKattouw first [15:00:43] Would it help if I told you that I was wearing a brightly colored printed shirt and showed great enthusiasm during the pre-show interview? [15:00:55] andrewbogott: heya! SWAT is now, I think, and I've the api patches set up for it. [15:00:58] andrewbogott: Yes, but RoanKattouw is Dutch, and that's weird. [15:01:07] I'm not listed because I'm not usually awake at 8am ever, but jetlag [15:01:10] fair point [15:01:49] All right, +2 in, waiting on Jenkins [15:03:10] andrewbogott: pre-show interview? [15:04:46] Based on my exuberant game show entrance [15:05:00] aaaah, I think my network made me miss that [15:07:35] Hm… is Jenkins broken? [15:07:35] mark: Hey. [15:07:37] Bah. [15:07:39] marktraceur: Hey. [15:07:41] marktraceur: Did you not get my text? [15:07:51] Oh, er, no [15:07:57] obviously, I got your text [15:08:08] Damn tab-complete phones [15:08:11] mark: Sorry for mis-ping. [15:08:13] ;) [15:08:28] OK well, James_F and RoanKattouw, I'm doing yours now, it got merged finally [15:08:35] Thanks marktraceur. [15:09:20] Has anyone ever actually had a security patch in an extension? [15:09:30] I check every time but always disappointments [15:09:42] our extensions are always secure [15:09:55] CA does semi regularly [15:10:00] James_F: Oh, got it now [15:10:10] marktraceur: Helpful. [15:10:51] First I'll do the core changes, then VE, sound sane James_F? [15:11:03] marktraceur: It's one patch for a reason. No need to split it. [15:11:09] Ugh [15:11:26] I guess sync-dir for the entire core directory, then [15:11:29] marktraceur: But the core change is fine on its own if you push that first. [15:11:47] I'd rather, yeah. Maybe just symbolic. :) [15:12:10] are you making a link? :) [15:12:26] Booooo [15:12:53] !log marktraceur Synchronized php-1.25wmf3/resources/lib/oojs-ui: [SWAT] [wmf3] Update OOjs UI to v0.1.0-pre (d74a46ca6a) and VisualEditor-MediaWiki to Ie06056b (duration: 00m 06s) [15:13:01] Logged the message, Master [15:13:42] !log marktraceur Synchronized php-1.25wmf3/extensions/VisualEditor/modules/ve-mw/ui: [SWAT] [wmf3] Update OOjs UI to v0.1.0-pre (d74a46ca6a) and VisualEditor-MediaWiki to Ie06056b (duration: 00m 05s) [15:13:45] Much faster [15:13:47] Logged the message, Master [15:13:48] Or something. [15:13:50] James_F: Testy test! [15:13:55] YuviPanda: You're next [15:14:08] * YuviPanda touches up make up, paints nails again quickly [15:14:17] OK then. [15:14:28] *five* patches? [15:14:53] I think I can do them all in two syncs [15:15:04] Oh, no, because submodule bump [15:15:07] marktraceur: yeah. [15:15:15] one of them is a security fix! Sort of [15:15:20] YuviPanda: What have you wrought?! ;-) [15:15:26] James_F: wikitech :) [15:15:34] marktraceur: wmf3 is a no-op as well, since wikitech is on wmf2 [15:15:39] Super duper [15:15:42]