[00:00:31] !log asher synchronized wmf-config/db.php 'pulling db35' [00:00:33] Logged the message, Master [00:01:25] RECOVERY - MySQL Replication Heartbeat on db1021 is OK: OK replication delay 0 seconds [00:15:07] RECOVERY - MySQL Slave Delay on db1018 is OK: OK replication delay 0 seconds [00:21:27] PROBLEM - mysqld processes on db35 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:26:37] RECOVERY - MySQL Replication Heartbeat on db1005 is OK: OK replication delay 0 seconds [01:23:28] New patchset: Asher; "i want a single metric to query if both slave threads are running" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2330 [01:25:27] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2330 [01:25:28] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2330 [02:03:25] New patchset: Demon; "Forking svn2git so we can maintain our own hacks" [operations/software] (master) - https://gerrit.wikimedia.org/r/2331 [02:03:26] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2331 [02:05:57] New patchset: Demon; "Forking svn2git so we can maintain our own hacks" [operations/software] (master) - https://gerrit.wikimedia.org/r/2331 [02:08:51] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 959s [02:23:11] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1819s [02:25:18] !log LocalisationUpdate completed (1.18) at Tue Feb 7 02:25:17 UTC 2012 [02:25:24] Logged the message, Master [02:33:17] Hi. Meta is borked with " Missing X-TransactionID header" on all pages. [02:36:48] Symptom is gone now. [02:41:26] New patchset: Demon; "Tweak metadata message to be more useful for us" [operations/software] (master) - https://gerrit.wikimedia.org/r/2332 [02:41:27] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2332 [02:49:32] RECOVERY - mysqld processes on db35 is OK: PROCS OK: 1 process with command name mysqld [02:58:28] !log asher synchronized wmf-config/db.php 'adding db35 back' [02:58:30] Logged the message, Master [03:09:11] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 51s [03:29:41] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 48s [04:59:32] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [04:59:32] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [10:05:15] New patchset: Dzahn; "add the existing index.html of noc.wm, as i changed it per bz 23004.." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2333 [10:05:34] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2333 [10:07:01] New patchset: Dzahn; "add the existing index.html of noc.wm, as i changed it per bz 23004.." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2333 [10:07:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2333 [10:11:26] New review: Dzahn; "per https://bugzilla.wikimedia.org/show_bug.cgi?id=23004" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2333 [11:52:20] Hi, I have a technical problem which I have described here: http://meta.wikimedia.org/wiki/Wikimedia_Forum#Network_connectivity_from_82.113.119.254 - can anyone see what is wrong? TIA [11:54:43] New patchset: Mark Bergsma; "virtual_module couldn't be defined twice, so let's make module virtual instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2334 [11:55:02] New patchset: Mark Bergsma; "Deploy new style apache configuration on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2335 [11:55:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2334 [11:55:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2335 [11:55:24] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2334 [11:55:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2334 [11:57:06] RicZzz: let me try to reproduce, i am also on O2 de [11:57:58] New patchset: Mark Bergsma; "Webserver::Apache::Site[torrus.wikimedia.org] depends on package torrus-apache2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2336 [11:58:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2335 [11:58:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2335 [11:58:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2336 [11:58:22] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2336 [12:00:41] New patchset: Mark Bergsma; "Oops, two definitions should be classes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2337 [12:00:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2337 [12:01:07] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2337 [12:01:07] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2337 [12:02:47] RicZzz: hmm, works for me. also being behind their NAT ...anything else that changed since yesterday? [12:03:13] No change and totally repeatable at the moment - no signle connection to en.wikipedia.org whereas wikimedia seems to work nicely. :( [12:04:36] you are tethering i suppose, because you said you are using wireshark.. does it work directly in the phone browser ? [12:06:10] New patchset: Mark Bergsma; "Fix template" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2338 [12:06:29] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2338 [12:12:15] New patchset: Mark Bergsma; "Fix SSL certificate paths" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2339 [12:12:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2339 [12:12:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2338 [12:12:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2338 [12:13:00] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2339 [12:13:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2339 [12:13:43] New patchset: Mark Bergsma; "Add missing ?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2340 [12:14:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2340 [12:14:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2340 [12:14:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2340 [12:15:44] not aware of any changes, tried to disable proxy and a few things [12:16:28] RicZzz: i'll create a ticket for that, since it does look like the load balancer in Amsterdam does reset your connection [12:17:20] PROBLEM - Puppet freshness on srv226 is CRITICAL: Puppet has not run in the last 10 hours [12:19:49] New patchset: Mark Bergsma; "Fix scoping" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2341 [12:19:52] thanks [12:20:09] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2341 [12:20:15] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2341 [12:20:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2341 [12:22:01] New patchset: Mark Bergsma; "Fix resource dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2342 [12:22:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2342 [12:22:25] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2342 [12:22:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2342 [12:24:29] New patchset: Mark Bergsma; "Fix $mpm variable reference" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2343 [12:24:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2343 [12:24:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2343 [12:24:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2343 [12:24:52] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2343 [12:32:33] New patchset: Mark Bergsma; "Logic for mismatching apache module / deb package names" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2344 [12:32:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2344 [12:33:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2344 [12:33:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2344 [12:39:34] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:39:52] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2345 [12:41:32] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:41:51] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2345 [12:43:45] !log reedy synchronized php-1.18/extensions/ShortUrl/ 'r110841' [12:43:48] Logged the message, Master [12:45:50] New patchset: Mark Bergsma; "Add standard apache modules" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:46:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2345 [12:46:20] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2345 [12:46:21] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2345 [12:47:59] New patchset: Mark Bergsma; "Fix mod perl's package name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2346 [12:48:24] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2346 [12:48:25] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2346 [12:52:04] New patchset: Mark Bergsma; "Fix redirect->rewrite, apache enabled sites link" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2347 [12:52:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2347 [12:52:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2347 [12:52:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2347 [12:58:31] New patchset: Mark Bergsma; "Notify the apache service on config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2348 [12:58:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2348 [13:18:41] PROBLEM - MySQL Slave Delay on db1043 is CRITICAL: CRIT replication delay 509 seconds [13:18:41] PROBLEM - MySQL Replication Heartbeat on db1043 is CRITICAL: CRIT replication delay 509 seconds [13:25:13] !log reedy synchronized php-1.18/extensions/ShortUrl/ 'r110845' [13:25:15] Logged the message, Master [13:30:29] New patchset: Mark Bergsma; "Add Torrus automatic discovery/DDX support" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2349 [13:31:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2349 [13:31:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2349 [13:32:03] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2348 [13:32:03] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2349 [13:32:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2348 [13:34:11] New patchset: Mark Bergsma; "Fix exec name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2350 [13:34:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2350 [13:34:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2350 [13:34:40] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2350 [13:39:07] New patchset: Mark Bergsma; "Small torrus discovery fixes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2351 [13:39:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2351 [13:39:33] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2351 [13:39:33] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2351 [13:55:25] Hi Tim, are you around? [13:57:53] drdee, unlikely as it's 1am for him [13:57:56] Can anyone else help? [14:02:27] New patchset: Mark Bergsma; "Simply process all .ddx files automatically" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2352 [14:02:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2352 [14:03:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2352 [14:03:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2352 [14:03:49] Reedy: thx :) quick question about some logic in C [14:04:11] would you be able to help? [14:23:48] New patchset: Mark Bergsma; "Small fixes, torrus-discovery already does a torrus compile" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2353 [14:24:08] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2353 [14:24:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2353 [14:24:17] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2353 [14:33:33] New patchset: Mark Bergsma; "Make sure discovery is done before compilation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:33:50] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2354 [14:34:41] Who can restart gerrit-wm_? That underscore looks awkward. [14:35:02] And also please restart wikibugs_ on #mediawiki for same reason. [14:36:16] New patchset: Mark Bergsma; "Make sure discovery is done before compilation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:36:34] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2354 [14:36:42] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2354 [14:36:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2354 [14:39:05] New patchset: Mark Bergsma; "Multithreading is NOT SUPPORTED by current perl interpreter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2355 [14:39:25] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2355 [14:39:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2355 [14:39:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2355 [15:10:20] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [15:10:20] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [15:30:10] !log reedy synchronized wmf-config/InitialiseSettings.php 'set project namespace for mrwikisource' [15:30:13] Logged the message, Master [15:35:56] !log reedy synchronized wmf-config/InitialiseSettings.php 'Extra namespaces for mwikisource bug 33907' [15:35:57] Logged the message, Master [15:36:47] !log reedy synchronized wmf-config/InitialiseSettings.php 'Extra namespaces for mwikisource bug 33907' [15:36:49] Logged the message, Master [16:00:34] New patchset: Mark Bergsma; "Don't use scope.lookupvar for locally scoped variables" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2356 [16:01:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2356 [16:01:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2356 [16:03:30] New patchset: Mark Bergsma; "Port 80 vhost wasn't being generated" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2357 [16:03:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2357 [16:03:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2357 [16:11:31] New patchset: Mark Bergsma; "Fix syntax errors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2358 [16:11:54] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2358 [16:11:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2358 [16:24:25] New patchset: Mark Bergsma; "s/Core_switches/Core_routers/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2359 [16:24:53] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2359 [16:24:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2359 [16:26:36] anyone knows how often the osm-stuff at toolserver is updated? [16:26:57] or better: how fast changes in osm make it to toolserver? [16:35:51] New patchset: Mark Bergsma; "Add mandatory param snmp-port" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2360 [16:36:16] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2360 [16:36:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2360 [16:38:10] New patchset: Mark Bergsma; "Doh" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2361 [16:38:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2361 [16:38:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2361 [16:38:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2361 [16:53:11] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:54:04] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:54:59] New patchset: Mark Bergsma; "Generate missing site-global.xml" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:55:28] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2362 [16:55:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2362 [16:57:20] New patchset: Mark Bergsma; "Fix scope lookup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2363 [16:57:46] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2363 [16:57:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2363 [17:10:49] New patchset: Mark Bergsma; "Add support for adding custom config lines to apache vhost configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:13:40] New patchset: Mark Bergsma; "Add support for adding custom config lines to apache vhost configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:14:00] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2364 [17:14:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2364 [17:14:23] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2364 [17:19:20] New patchset: Mark Bergsma; "Torrus is now completely in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2365 [17:19:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2365 [17:19:56] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2365 [17:19:56] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2365 [17:28:30] New patchset: Mark Bergsma; "Add dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2366 [17:28:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2366 [17:28:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2366 [17:42:14] New patchset: Mark Bergsma; "Better subclassing, dependencies" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2367 [17:42:39] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2367 [17:42:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2367 [18:32:56] !log Updated the payments cluster to r110857 [18:32:57] Logged the message, Master [18:52:54] Hello Reedy and RoanKattouw. Don't know if you're still working but if you can just make a little modification that will require two minutes, that would be really helpful [18:53:12] that's about a change on a sitelogo include path [18:53:51] lol, still working, I've got meetings in a few hours from now ;) [18:56:12] Reedy: so you cannot change the site logo ? :( [18:56:27] (I've wondering if you're still working cause in France it's 8 PM now) [18:57:03] It's 8pm here too, and 7pm for Reedy, but that doesn't mean we're not working :) [18:57:11] 11am in SF, so meetings [19:14:20] New patchset: Mark Bergsma; "Setup a ganglia collector (for torrus aggregated stats) on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2368 [19:14:51] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2368 [19:14:51] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2368 [19:15:43] RoanKattouw: so you can't do the logo change ? please, only two minutes to change the path and I do the rest :D [19:15:57] Quentinv57: File a BZ ticket and I'll do it tomorrow [19:16:05] * RoanKattouw got up at 5am today and needs to go to sleep at some point [19:16:21] ah I see, sure [19:16:37] RoanKattouw: should I assign directly to you ? [19:16:41] Yes [19:17:31] 5 am? ouch [19:18:29] Yeah, I drove away at 6am [19:18:42] In the parking garage at 7:30, at the consulate around 8:15 [19:18:50] They made us wait in the cold (-11 C) for a while, too [19:19:03] Once I was in it was fairly quick, I was out by 9 [19:20:19] good for that at least [19:20:28] here I was going to go see about my paperwork ... [19:20:35] except that yesterday they announced a 24 hour strike [19:20:44] which I think is 48 hours for govt workers [19:20:49] so much for this week... [19:21:22] Well, I mean, I spent 3h in the car, took 350 pages of stuff, and all they were interested in was ~5 pages of paperwork, my passport, and a very short interview (~5 questions), and then they sent me home [19:22:02] They didn't ask about any of the 332 pages I had to run around Brussels for last weekend [19:22:16] of course [19:22:27] but if you hadn't walked in with them... [19:22:28] It's funny, the law firm prepares me for the worst, gives me lots of paperwork [19:22:32] yup [19:22:39] And when I get there, it's super quick [19:22:54] That's happened to me for two visa-free US entries, and now at the consulate too [19:22:55] congrats on getting it done [19:22:58] Thanks [19:23:01] New patchset: Mark Bergsma; "Temp fix gmetad for torrus usage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2369 [19:23:15] All that separates me from my actual visa at this point is the postal service [19:23:27] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2369 [19:23:28] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2369 [19:23:28] which one, yours or the us? [19:23:40] Ours [19:23:59] The AMS consulate will mail my passport (with added visa) back to me [19:24:07] ah [19:24:19] Of course I had to provide a self-addressed envelope with like $10 in postage (registered mail is expensive), but whatever [19:29:53] New patchset: Mark Bergsma; "Saving streber from its misery as a full fledged ganglia collector" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2370 [19:30:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2370 [19:30:29] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2370 [19:37:15] RoanKattouw: I just sent the bug, thanks by advance :) [19:37:24] and good luck for your work :P [19:40:51] New patchset: Mark Bergsma; "Fix rrd dir for Bits caches pmtpa" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2371 [19:41:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2371 [19:41:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2371 [19:41:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2371 [19:46:58] New patchset: Mark Bergsma; "Install wikimedia.css file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2372 [19:51:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2372 [19:51:49] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2372 [20:24:20] Would it be bad for any reason (e.g. prevention of caching) to use {{INT:Lang}} in two links on the de.wikipedia main page? [20:55:45] ... alternatively I could just put it in and then look at ganglia if some servers melt ;-) [21:00:24] someone in Pakistan using FF10 sees Wikipedia w/o stylesheets [21:00:44] nick = Snake_lover [21:01:14] hmm [21:01:46] zzz [21:02:08] Snake_lover: explain it [21:02:22] when I open any page of wikipedia it looks like a person who just learnt how to do HTML has made the page and placed in front of me. I have tried 3 different browsers and all of them show the same thing. It has been happening for over a week now [21:02:42] cache? [21:02:53] just information written randomly shows up. Pictures don' open [21:03:11] Cleared it just 2 minutes ago [21:03:16] yeah [21:03:46] Any javascript errors? [21:04:02] nope. [21:04:14] Every other website is working [21:04:33] It takes more then 5 minutes to open the Wikipedia. [21:04:40] A single page. [21:04:54] Pastebin a traceroute? [21:05:11] any other reports from Pakistan with the same problem? [21:05:49] if this has gone on for over a week, this is a big problem [21:05:51] nope [21:06:21] As far as I have seen using Wikipedia in my friends house, it was working fine there [21:06:31] Same ISP? [21:06:38] No. [21:06:41] Tried a different browser/different computer on the same connection? [21:06:53] I even tried it using proxy but it did't work [21:06:58] yes [21:07:26] and? [21:07:42] 3 different computers and even moblies and currently I am using firefox. I tried chrome in internet explorer. [21:07:48] Same result [21:07:52] so it's connection related [21:08:03] Pastebin a traceroute and we can poke our network people [21:08:44] I don't think so. I mean it just stopped working approximately 8-10 days ago. Before that it was perfect. [21:08:53] And? [21:09:05] That's not to say routing hasn't changed, your ISP might have broken something [21:09:26] If it's only happening on your connection, on different machines, it's likely connection related [21:09:59] Some big pages e.g Bill Gates, Taj Mahal then don't even open up even after opening them for more then 10 minutes. [21:10:26] Are you logged in or not? [21:10:38] Well if it was a connection releated problem other websites should also have shown the same result. [21:10:45] Not necesserily [21:10:52] You don't know what routes they take around the world [21:10:52] I can't open my account even after trying. [21:11:30] May be. I am not an expert, thats why wanted to ask you guys because you are the experts. [21:11:34] Yes [21:11:40] What should I do? [21:11:44] Answer the questions [21:11:48] Provide the requested information [21:11:49] okay [21:12:27] Sure. Ask them [21:12:34] Are you logged in or not? [21:12:40] No [21:12:50] New patchset: Ottomata; "Movied wurfl.py file" [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2373 [21:13:43] Can you run a traceroute to en.wikipedia.org and give me a link to a paste of it on http://dpaste.com ? [21:13:54] Frankly speaking I don't really know. The login page opens up but when I enter the user name and password the page becomes blank. [21:14:20] Use a private browsing tab or similar [21:15:03] Did try that too. [21:15:16] How do I use this dpaste thing? [21:15:44] I saw the example above. But it's some kind of code. [21:16:10] shoudl I upload it to photobucket and then send you the link? [21:16:37] you can if that's easier for you [21:16:45] Okay. [21:19:37] http://i1197.photobucket.com/albums/aa431/abdullah_ahmad/Wiki.png [21:19:50] !log reboot dataset1001, new kernel 2.6.38, should deal with kswap bug we just ran into under heavy i/o load, need to test [21:19:51] Logged the message, Master [21:20:58] rats! it's my day off [21:21:03] suckered into work again... [21:21:09] * apergos hangs head in shame [21:21:18] poor apergos [21:21:41] So what do you say? [21:22:04] Snake_lover, Go to start, type cmd, press enter. in the black box type tracert en.wikipedia.org [21:22:08] Press enter [21:22:13] and wait for it to finish [21:22:15] Okay [21:22:19] New patchset: Ottomata; "More documenation, added tests for AccessLogPipeline methods." [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2374 [21:22:20] New patchset: Ottomata; "Merge branch 'otto/pipeline' of ssh://gerrit.wikimedia.org:29418/analytics/reportcard into otto/pipeline" [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2375 [21:22:26] That is presumably a bits related error as you're not getting any CSS [21:22:56] Yeah that was what I was saying. Simple HTML shows up [21:24:45] The trace is complete [21:24:53] What should I tell you from it. [21:25:03] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2375 [21:25:17] It's 17 lines [21:25:22] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2374 [21:25:32] I want the text from that [21:25:42] another screenshot would work [21:25:50] Okay [21:25:59] New review: Diederik; "Ok." [analytics/reportcard] (otto/pipeline); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2373 [21:26:00] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2375 [21:26:00] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2374 [21:26:00] Change merged: Diederik; [analytics/reportcard] (otto/pipeline) - https://gerrit.wikimedia.org/r/2373 [21:26:49] http://i1197.photobucket.com/albums/aa431/abdullah_ahmad/cmd.png [21:28:22] And it also takes approximately 5 or more minutes to open up a single page and the big ones don't open. [21:31:45] So any good news? [21:31:57] any bad news? [21:32:15] What do you get if you visit http://bits.wikimedia.org/en.wikipedia.org/load.php?debug=true&lang=en&modules=ext!wikihiero|mediawiki!legacy!commonPrint|mediawiki!legacy!shared|skins!vector&only=styles&skin=vector [21:33:12] It's not opening. [21:34:12] I just got an error [21:34:18] and the error is? [21:34:28] Connection to bits.wikimedia.org Failed [21:34:45] and that's your issue [21:35:38] if you go back to the console, what does bits.wikimedia.org [21:36:17] connection time out [21:37:33] Should I send a snap shot of it too? [21:39:14] please [21:39:18] right, this is your issue [21:39:23] nslookup bits.wikimedia.org [21:39:54] is mark always opped? [21:40:20] <^demon|away> PiRSquared17: Yeah pretty much. And? [21:40:37] ok, just windering [21:40:39] *o [21:40:44] <^demon|away> freenode's policy is silly :) [21:41:06] he looks like ChanServ [21:42:11] Reedy: http://i1197.photobucket.com/albums/aa431/abdullah_ahmad/bits.png [21:42:27] Snake_lover, what does "nslookup bits.wikimedia.org" (not in the quotes) give you? [21:43:02] in the console [21:43:23] I don't understand what you mean by that. [21:43:33] in your console [21:43:34] type [21:43:38] nslookup bits.wikimedia.org [21:43:40] press enter [21:44:13] babylon search tool bar appeared [21:44:44] ping bits.wikimedia.org [21:46:58] ping result [21:47:00] http://i1197.photobucket.com/albums/aa431/abdullah_ahmad/ping.png [21:50:25] so? [21:51:19] should I ask my ISP to look into this matter? [21:53:41] reedy, should I ask my ISP to look into this matter? [21:54:43] Bare with me a few minutes [21:54:53] Okay [22:05:37] Snake_lover, a quick look at a few things suggests that you're maybe the only person with this issue [22:05:50] Okay [22:05:59] So contacting your ISP would be a good starting point, I can link all your images to a bugzilla bug if you wanted [22:06:10] Then if you have no luck, we can take it further, or close it if they fix it [22:06:24] It would be really helpful. [22:06:28] Basically tell them you're having issues with the bits.wikimedia.org host, but other things would be fine [22:06:34] Have you got an account on our bugzilla? [22:06:45] No [22:11:47] Snake_lover, https://bugzilla.wikimedia.org/show_bug.cgi?id=34256 [22:11:55] You can create an account and CC yourself to that bug [22:12:01] that will allow you to comment on it [22:13:00] Reedy I will ask my ISP to check this and thanks for helping me. [22:13:12] I will create my account on bugzilla [22:13:21] Thank you for your time [22:13:37] Bye [22:26:19] (re-post) Would it be bad for any reason (e.g. prevention of caching) to use {{INT:Lang}} in two links on the de.wikipedia main page? ... alternatively I could just put it in and then look at ganglia if some servers melt ;-) [22:28:56] PROBLEM - Puppet freshness on srv226 is CRITICAL: Puppet has not run in the last 10 hours [22:46:53]