[00:01:25] RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay 0 seconds [00:02:12] !log shutting down mysql on db35, rebuild fail [00:02:14] Logged the message, Master [00:04:49] !log streaming hotbackup of db1021 to db35 [00:04:50] Logged the message, Master [00:15:07] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [00:21:27] PROBLEM - mysqld processes on db35 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:26:37] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay 0 seconds [01:23:28] New patchset: Asher; "i want a single metric to query if both slave threads are running" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2330 [01:25:27] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2330 [01:25:28] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2330 [02:03:25] New patchset: Demon; "Forking svn2git so we can maintain our own hacks" [operations/software] (master) - https://gerrit.wikimedia.org/r/2331 [02:03:26] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2331 [02:05:57] New patchset: Demon; "Forking svn2git so we can maintain our own hacks" [operations/software] (master) - https://gerrit.wikimedia.org/r/2331 [02:08:51] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 959s [02:23:11] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1819s [02:41:26] New patchset: Demon; "Tweak metadata message to be more useful for us" [operations/software] (master) - https://gerrit.wikimedia.org/r/2332 [02:41:27] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2332 [02:49:31] RECOVERY - mysqld processes on db35 is OK: PROCS OK: 1 process with command name mysqld [03:09:11] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 51s [03:29:41] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 48s [04:59:32] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [04:59:32] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [10:05:15] New patchset: Dzahn; "add the existing index.html of noc.wm, as i changed it per bz 23004.." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2333 [10:05:33] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2333 [10:07:01] New patchset: Dzahn; "add the existing index.html of noc.wm, as i changed it per bz 23004.." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2333 [10:07:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2333 [10:11:26] New review: Dzahn; "per https://bugzilla.wikimedia.org/show_bug.cgi?id=23004" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2333 [10:50:25] hashar: re: RT 2315 (https on mail.wm / mail.wp): any idea which is linked in more places? mail.wikimedia or mail.wikipedia? [10:50:44] noooo ideaa :-) [10:51:25] hashar: try accessing mail.wm with enabled HTTPS everywhere now [10:52:28] hashar: just mail.wp would still give you the wrong cert [10:53:36] I need to reinstall that https everywhere extension :D [10:53:45] I disabled it because there was too many links in trouble [10:53:46] for that i would need an additional ip on mchenry.. hrmm [10:54:16] ah,ok, nevermind then. well they do listen on SSL now, unlike before [10:54:31] I am rebuilding the extension [10:56:37] hmm..it will redirect to http://lists first, but then immediately to https://lists again [10:57:31] we can also redirect it all to https:// in the first step [11:04:30] can't manage to rebuild it again :( [11:04:32] anyway [11:06:21] mutante: I think httpS://mail.wikimedia.org/ should redirect to httpS://lists.wikimedia.org/ [11:06:24] to keep the secure link [11:11:22] hashar: i agree, redirecting http _and_ https to https [11:15:02] hashar: done.. wish i could call it resolved now (besides puppetizing), but it will be reopened for "wrong cert on mail.wp", right:) [11:18:53] can't you get the wp.org cert there? [11:19:28] we can have either one. thats why i asked which might be linked to in more places [11:19:47] to have both at the same time i would need an extra IP [11:21:33] (SSL + name based virtual hosts + (lighttpd)) [11:22:16] :-( [11:22:26] so maybe just leave it as is [11:22:35] I am not sure which domain is more used anyway [11:22:52] probably the wm.org one anyway [11:22:53] or we ask Leslie if we get an IP and let networking decide [11:23:23] that would be wasting a public IP [11:23:28] just for a barely used redirect. [11:23:54] depends, i also wrote "waste" but on the other hand, SSL vhosts is a valid reason at RIPE when requesting them.. shrug [11:24:05] :))))) [11:24:31] "barely used" ..hmm yea, lets see if there are logs [11:27:01] hashar: wp is more popular. 3643 times in access.log and wm just 423 times (starting 05/Feb/2012:06:46:24 until now) [11:27:29] \o/ [11:54:43] New patchset: Mark Bergsma; "virtual_module couldn't be defined twice, so let's make module virtual instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2334 [11:55:02] New patchset: Mark Bergsma; "Deploy new style apache configuration on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2335 [11:55:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2334 [11:55:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2335 [11:55:24] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2334 [11:55:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2334 [11:57:58] New patchset: Mark Bergsma; "Webserver::Apache::Site[torrus.wikimedia.org] depends on package torrus-apache2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2336 [11:58:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2335 [11:58:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2335 [11:58:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2336 [11:58:22] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2336 [12:00:41] New patchset: Mark Bergsma; "Oops, two definitions should be classes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2337 [12:00:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2337 [12:01:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2337 [12:01:07] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2337 [12:06:10] New patchset: Mark Bergsma; "Fix template" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2338 [12:06:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2338 [12:12:15] New patchset: Mark Bergsma; "Fix SSL certificate paths" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2339 [12:12:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2339 [12:12:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2338 [12:12:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2338 [12:13:00] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2339 [12:13:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2339 [12:13:43] New patchset: Mark Bergsma; "Add missing ?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2340 [12:14:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2340 [12:14:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2340 [12:14:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2340 [12:17:20] PROBLEM - Puppet freshness on srv226 is CRITICAL: Puppet has not run in the last 10 hours [12:19:49] New patchset: Mark Bergsma; "Fix scoping" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2341 [12:20:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2341 [12:20:15] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2341 [12:20:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2341 [12:22:01] New patchset: Mark Bergsma; "Fix resource dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2342 [12:22:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2342 [12:22:25] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2342 [12:22:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2342 [12:24:28] New patchset: Mark Bergsma; "Fix $mpm variable reference" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2343 [12:24:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2343 [12:24:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2343 [12:24:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2343 [12:24:52] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2343 [12:32:33] New patchset: Mark Bergsma; "Logic for mismatching apache module / deb package names" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2344 [12:32:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2344 [12:33:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2344 [12:33:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2344 [12:39:34] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:39:52] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2345 [12:41:32] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:41:51] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2345 [12:45:50] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:46:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2345 [12:46:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2345 [12:46:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:47:59] New patchset: Mark Bergsma; "Fix mod perl's package name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2346 [12:48:24] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2346 [12:48:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2346 [12:52:04] New patchset: Mark Bergsma; "Fix redirect->rewrite, apache enabled sites link" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2347 [12:52:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2347 [12:52:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2347 [12:52:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2347 [12:58:31] New patchset: Mark Bergsma; "Notify the apache service on config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2348 [12:58:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2348 [13:18:41] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 509 seconds [13:18:41] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 509 seconds [13:30:29] New patchset: Mark Bergsma; "Add Torrus automatic discovery/DDX support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2349 [13:31:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2349 [13:31:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2349 [13:32:03] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2348 [13:32:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2349 [13:32:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2348 [13:34:11] New patchset: Mark Bergsma; "Fix exec name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2350 [13:34:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2350 [13:34:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2350 [13:34:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2350 [13:39:07] New patchset: Mark Bergsma; "Small torrus discovery fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2351 [13:39:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2351 [13:39:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2351 [13:39:33] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2351 [14:02:27] New patchset: Mark Bergsma; "Simply process all .ddx files automatically" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2352 [14:02:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2352 [14:03:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2352 [14:03:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2352 [14:23:48] New patchset: Mark Bergsma; "Small fixes, torrus-discovery already does a torrus compile" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2353 [14:24:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2353 [14:24:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2353 [14:24:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2353 [14:33:33] New patchset: Mark Bergsma; "Make sure discovery is done before compilation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:33:50] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2354 [14:36:16] New patchset: Mark Bergsma; "Make sure discovery is done before compilation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:36:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2354 [14:36:41] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2354 [14:36:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:39:05] New patchset: Mark Bergsma; "Multithreading is NOT SUPPORTED by current perl interpreter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2355 [14:39:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2355 [14:39:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2355 [14:39:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2355 [15:10:20] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [15:10:20] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [15:23:35] hi mark [15:51:36] I've got another question :) [15:51:47] in the webstatscollector code, we are filtering out a number of ip blocks [15:51:52] (see http://svn.mediawiki.org/viewvc/mediawiki/trunk/webstatscollector/filter.c?revision=68218&view=markup) [15:52:30] the comment says "IP addresses from which duplicate requests originate" [15:52:35] is this still an issue? [15:54:03] if yes, are these ip blocks still up-to-date or do we need to check that? [16:00:34] New patchset: Mark Bergsma; "Don't use scope.lookupvar for locally scoped variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2356 [16:01:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2356 [16:01:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2356 [16:03:30] New patchset: Mark Bergsma; "Port 80 vhost wasn't being generated" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2357 [16:03:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2357 [16:03:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2357 [16:11:31] New patchset: Mark Bergsma; "Fix syntax errors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2358 [16:11:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2358 [16:11:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2358 [16:24:25] New patchset: Mark Bergsma; "s/Core_switches/Core_routers/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2359 [16:24:53] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2359 [16:24:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2359 [16:35:51] New patchset: Mark Bergsma; "Add mandatory param snmp-port" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2360 [16:36:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2360 [16:36:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2360 [16:38:10] New patchset: Mark Bergsma; "Doh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2361 [16:38:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2361 [16:38:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2361 [16:38:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2361 [16:53:11] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:54:04] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:54:59] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:55:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2362 [16:55:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:57:20] New patchset: Mark Bergsma; "Fix scope lookup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2363 [16:57:46] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2363 [16:57:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2363 [17:09:02] mark: hi [17:10:06] robh: u there? [17:10:08] hi [17:10:17] cmjohnson1: yep [17:10:45] mark: i am installing new labstore in d1 pmtpa...i want to clean up power cords going to cr2-pmtpa...they are too long and in the way. [17:10:49] New patchset: Mark Bergsma; "Add support for adding custom config lines to apache vhost configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:10:58] it is redundant...is it ok to replace one at a time? [17:11:08] cmjohnson1: in theory yes [17:11:15] but when you do it, you should have leslie or me present [17:11:30] right...that is why i ping'd u =] [17:11:39] if you want to do it now you can [17:11:42] just be VERY careful [17:11:50] if anything goes down, this will cause site downtime [17:12:12] RobH - have u scheduled db9 move yet? [17:12:17] ok [17:12:39] woosters: asher may be able to migrate it with no downtime before this weekend, but if not the plan is on saturday, but its not set yet [17:12:51] the plan was to wait until later in the week and then set the time to move it [17:12:53] !log swapping power cables for cr2-pmtpa [17:12:54] Logged the message, Master [17:13:10] let me know so we can notify engr [17:13:32] woosters: i will touch base with asher later today and set the time for the move [17:13:38] ok, thx [17:13:40] New patchset: Mark Bergsma; "Add support for adding custom config lines to apache vhost configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:14:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2364 [17:14:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2364 [17:14:23] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:16:19] !log power cable swap cr2-pmtpa complete [17:16:20] Logged the message, Master [17:16:53] oops [17:19:20] New patchset: Mark Bergsma; "Torrus is now completely in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2365 [17:19:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2365 [17:19:56] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2365 [17:19:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2365 [17:28:30] New patchset: Mark Bergsma; "Add dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2366 [17:28:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2366 [17:28:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2366 [17:42:14] New patchset: Mark Bergsma; "Better subclassing, dependencies" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2367 [17:42:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2367 [17:42:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2367 [17:52:19] maplebed: Do you have a minute to help me think about ssh tunneling? [17:52:27] sure! [17:52:32] (and welcome home) [17:52:42] (thanks!) [17:53:07] So, I have an irc proxy running on a labs machine on port 9999. And I want to connect colloquy on my laptop to that proxy. [17:53:23] This works: $ssh -A -L 9999:localhost:9999 bastion1 ssh -L 9999:localhost:9999 -N utils-abogott [17:54:08] My first question is: Does opening up that port on bastion1 worry you, security-wise? [17:54:17] (Or is it for some reason only opened for me?) [17:54:37] it's not publically open [17:54:38] hmm. [17:54:39] maybe [17:54:47] /shouldn't be [17:55:01] the first obvious answer is no, because you didn't use the '-g' flag, so you can only connect to it from localhost. [17:55:17] but, there are many people that have access to bastion. [17:55:23] Oh, cool. [17:55:30] I'm not too familiar with how IRC proxies work; [17:55:35] Um... I don't understand that, quite, but I will take your word for it for the moment :) [17:55:45] if I logged into bastion and connected to localhost:9999, what would happpen? [17:55:51] would I be connected to your session? [17:55:56] Oh -- an IRC proxy just stays logged into the channel all the time and logs traffic, then forwards it to me when I want to read the backscroll. [17:56:11] If yes, then anybody with access to bastion can impersonate you, which is bad. [17:56:52] I believe that you would have to actually log into IRC as me in order to impersonate me. But -- want to try? [17:57:05] Hm... now I'm not so sure of that last. [17:58:56] can I just telnet to the port? [17:59:05] or do I need to use a proxy-aware IRC client? [17:59:56] I predict that you can telnet to the port, no problem. But that if you connect with an IRC client you'll just get a normal-ish IRC session. (The client doesn't have to be proxy-aware.) [18:01:29] quite so. telnetting gave me IRC protocol noise. [18:02:59] So that's already imperfect, I guess. Would be nice to have a private tunnel. [18:03:05] But the fact that it requires two hops confounds me. [18:03:06] * maplebed tries with an IRC client [18:03:14] why? [18:03:32] why am I confounded, or why would it be nice to be private? [18:03:41] why are you confounded. [18:04:37] Only because it's something I haven't thought much about before. [18:04:48] is your tunnel currently up? [18:05:22] it is. [18:05:39] so I connected to the port on bastion, [18:05:47] and my IRC client says 'got your hostname', [18:06:10] but I can't connect through to a channel. [18:06:20] I'm probably doing it wrong... [18:06:29] JOIN :#channel [18:06:40] join space colon hash channel? [18:06:48] actually you need to take a NICK first [18:06:53] NICK something [18:06:56] (no slash?) [18:07:01] no slash in the protocol [18:07:19] Yes slash, if the client you're using expects one. [18:07:21] * mark tries to recall if anything else was needed [18:07:27] it's been over 10 years since I wrote irc clients ;) [18:07:28] huh. [18:07:37] not if telnet is your client ;-) [18:07:37] it asked me for /QUOTE PASS to login [18:08:05] ah good! I thought the password only protected my session but maybe the whole thing is locked up. [18:08:46] oh, so that's a proxy password? [18:09:12] um... I think so. [18:09:26] Unless you did /Nick [18:10:01] I might have... [18:10:31] no, it's giving me the password request when I JOIN no when I NICK. [18:10:39] it looks to me like you're protected. [18:10:42] great. [18:10:59] But, ok, impersonation aside, it seems that now anyone on bastion has access to a possibly-insecure 3rd party protocol on my labs machine. Which seems... well, bad in theory but maybe harmless in practice. [18:12:59] IMO it's sufficiently protected to be ok. [18:13:24] ok [18:27:38] !log updating dns for mgmt on db59-60 and labstore1-4 [18:27:39] Logged the message, RobH [18:38:59] ping mark AaronSchulz - just fyi, I'm going to change the upload squids from using the swift1_thumbs acl to swift2_thumbs in 5m [18:40:54] ohhh today is more deploy isn't it? yay [18:41:02] this one's easy. [18:41:08] one line change of existing config [18:41:12] rather than introducing new config. [18:41:15] yep [18:41:18] still exciting [18:41:23] I cannot eait for it to be done. [18:41:27] seriously. [18:41:27] switching from only matching /a/a2 URLs to /a/* and /b/* [18:41:52] nice [18:41:59] 1/8? [18:43:27] yeah. [18:43:52] cool! [18:43:56] * maplebed <3 short diffs. [18:43:57] -cache_peer_access 10.2.1.27 allow swift1_thumbs [18:43:57] +cache_peer_access 10.2.1.27 allow swift2_thumbs [18:44:09] oh wow! the ruling on prop 8 is out already [18:44:40] http://www.reuters.com/article/2012/02/07/us-usa-gaymarriage-california-idUSTRE8160HO20120207 [18:45:05] Yup [18:45:09] I haven't read it yet [18:45:12] \o/ [18:48:21] !log deployed squid config to send 1/8th of all thumbnail traffic to swift [18:48:22] Logged the message, Master [18:49:34] and there goes the traffic! [18:49:34] http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=&tab=v&vn=swift [18:50:07] very colourful [18:51:18] yeah, a little easier to get meaningful info from that URL than http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=&c=Swift+pmtpa&h=ms-fe1.pmtpa.wmnet&tab=m&vn=&mc=2&z=medium&metric_group=ALLGROUPS [18:51:59] quite a lot of stats [18:53:33] guillom, ^ Might be nice to get it into the engineering report that we've got an 12.5% of the thumb traffic over to swit so far [18:54:10] Reedy, I'll put it in the February report. [18:54:47] Thanks for the heads-up. [18:54:47] and the Feb report should say 100%! [18:54:50] (I hope) [18:54:52] héhé [19:06:40] hi maplebed! [19:07:31] hi diederik - mtg atm. back in 10? [19:07:40] sure [19:09:09] !log Deployed torrus on server manutius, migrated DNS over [19:09:11] Logged the message, Master [19:14:20] New patchset: Mark Bergsma; "Setup a ganglia collector (for torrus aggregated stats) on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2368 [19:14:34] looking at the traffic now [19:14:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2368 [19:14:51] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2368 [19:23:01] New patchset: Mark Bergsma; "Temp fix gmetad for torrus usage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2369 [19:23:27] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2369 [19:23:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2369 [19:28:26] woosters: rt 2407 and 2408 (also RobH) [19:29:25] maplebed: cool, i will request the quotes for you now [19:29:53] New patchset: Mark Bergsma; "Saving streber from its misery as a full fledged ganglia collector" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2370 [19:30:05] maplebed: does this mean 2126 is now invalid? [19:30:17] it was open for swift storage in tampa pending the c series testing [19:30:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2370 [19:30:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2370 [19:30:35] maplebed: also you had 1478 for swift in eqiad [19:30:41] i assume that is also now invalid? [19:31:04] RobH: yeah, I'll do some cleanup. [19:31:09] no need [19:31:15] i have them all open, i am linking and resolving [19:31:18] i just wanted to confirm [19:31:22] oh, and I made a mistake on the dates for eqiad; it should be sooner. [19:31:55] (or the pmtpa hardware later, but that wouldn't be as much fun.) [19:31:56] I am going to treat all of the swift stuff as high priority. [19:32:14] it's cool for it to take a while; that's why I made these early. [19:32:19] I have been telling Dell that we have a tight timeframe, so its fine ;] [19:32:31] robh 2126 is for thumbnails & I believe u ordered them [19:33:16] yea, my bad on that one [19:33:20] the other is invalid though now [19:33:32] 1478 is now invalid. [19:33:32] 1478 has to be amended to be the same cfg as 2126 [19:33:53] ah yes, the new one from maplebed invalidates it [19:33:59] at the end of this week is swift going to be doing all thumbs? [19:34:01] maplebed: the eqiad ticket includes the storage for both thumb and full right? [19:34:07] 1478 was for a test cluster. the new one is for prod hardware. [19:34:09] * AaronSchulz assumes so [19:34:15] apergos: yes. [19:34:19] RobH: yes. [19:34:29] maplebed: awesome, thanks =] [19:34:32] ok [19:34:41] maplebed: now when do we kill ms5? :) [19:34:59] AaronSchulz: we need to change the image scalers first. [19:35:10] sure [19:35:16] but after that... [19:35:37] did I miss something, is swift serving all thumbs yet? ;-) [19:35:57] it's now time to resume thinking more carefully about the plan going forward; when we do which step, etc. [19:36:07] mark: no, today's only 12.5%. Thursday will be 100%. [19:36:33] I would do another step in between tomorrow [19:36:36] it's just nobody's ever content with what we've got now... [19:36:42] like 50% traffic tomorrow [19:36:42] always wanting more. [19:36:42] or so [19:36:47] mark: that's the plan. [19:36:50] ok [19:36:57] http://wikitech.wikimedia.org/view/Swift/Deploy_Plan_-_Thumbnails#gradual_rollout_schedule [19:37:48] i'm not convinced yet swift can handle the load [19:38:02] hmm [19:38:25] I'm pretty sure it can, but we'll certainly find out tomrorow and thursday. [19:38:29] ok [19:38:34] :D [19:38:41] I just mean it has less headroom than I would like ;) [19:39:24] RobH's telling Dell to hurry up, and we can also do as you suggested and split traffic b/w swift and ms5 [19:39:47] I think the metric to watch is avg response time for swift, [19:40:14] but I'd love suggestions of other metrics (something from the squids?) that will help us determine whether everything's cool. [19:40:51] New patchset: Mark Bergsma; "Fix rrd dir for Bits caches pmtpa" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2371 [19:41:08] squid also has response time metrics [19:41:11]