[00:48:10] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [01:41:25] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 288 seconds [01:42:19] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 236 seconds [01:47:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 8 seconds [01:49:13] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 650s [01:51:10] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 7 seconds [01:52:04] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 19s [02:00:32] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [02:51:32] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [06:48:54] PROBLEM - Host db1047 is DOWN: PING CRITICAL - Packet loss = 100% [07:35:17] New patchset: Dereckson; "(bug 37327) Configure chr.wikipedia site logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11741 [07:35:25] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11741 [08:02:05] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [08:40:38] New patchset: Petrb; "(bug 37662) change wgUploadNavigationUrl @ dawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11745 [08:40:44] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11745 [08:43:56] Reedy: can you take a look on that ^ [08:46:09] New review: Raimond Spekking; "(no comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11745 [09:03:47] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [09:11:36] is there someone who could review and push it? :) [09:11:50] !g 11745 this I mean [09:11:50] https://gerrit.wikimedia.org/r/11745 [10:44:51] After reinstall of stat1 /mnt/data mounts are missing. At bayes these are: /mnt/htdocs -> 10.0.5.8:/home/wikipedia/htdocs/wikipedia.org/wikistats /mnt/data -> 208.80.152.185:/data [10:48:46] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [11:50:45] New patchset: Alex Monk; "(bug 37457) Add French and Italian Wikibooks as an import source at Vietnamese Wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11746 [11:50:51] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11746 [12:01:35] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [12:03:12] New patchset: Alex Monk; "(bug 37661) Replace the current Vietnamese Wikibooks logo with File:Wiki.png" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11747 [12:03:21] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11747 [12:06:32] New review: Alex Monk; "(no comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11741 [12:22:23] New review: Alex Monk; "Siebrand, see https://gerrit.wikimedia.org/r/#/admin/projects/operations/mediawiki-config,access" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11082 [12:52:07] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [13:08:02] New review: Siebrand; "Thanks, Alex. That's not a reply I can do something with, as Antoine has complained when these peopl..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11082 [13:15:36] New patchset: Alex Monk; "Let the groups on mediawikiwiki with autoreview also have autopatrol, and allow admins to grant the autochecked group." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11748 [13:15:42] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11748 [13:15:52] wow that was fast [13:20:44] New review: Alex Monk; "Oh, okay. Speaking of reviewers... https://gerrit.wikimedia.org/r/#/c/11067/" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11082 [13:26:37] After reinstall of stat1 two nfs mounts are missing. Anyone can help restore those? [13:39:27] that was the goal, erik [13:39:30] we want to get rid of NFS [13:39:37] other solutions have been found instead of NFS [13:39:41] talk to andrew otto about that [13:52:30] @mark I'm trying to restart fixed wikstats asap. Can you advise me how to access dump server from scripts then? If not I'll ask andrew tomorrow. [13:53:25] Ryan_Lane1, mark: http://barry.wordpress.com/2012/06/16/nginx-spdy-and-automattic/ [13:53:38] wordpress sponsored nginx SPDY support and put it into production already [13:55:46] ezachte: iirc the htdocs one is no longer needed, as stat1 will serve stats.wikimedia.org itself (run apache) [13:56:14] i'm not entirely sure what was decided for the dumps server mounts... I believe it was a read only mount, and rsync for writing [13:57:56] @mark agreed on htdocs but that needs some work (resync of content to stats /a) and was hoping to update a bug notice on all reports before that, today (let's skip that) [13:58:20] that's all I know right now [13:59:03] @mark yes read only access for dump server and rsync for updates and that was implemented already before re-install and working [13:59:21] now /mnt/data is empty [13:59:58] @mark it was on bayes: /mnt/data -> 208.80.152.185:/data [14:08:11] paravoid: unless they are using spdy without encryption, that site isn't spdy for me [14:20:34] Ryan_Lane: only for https:// [14:20:42] the blog post mentions that [14:22:29] hmm, it's not via https either (just checked) [14:22:30] who knows [14:24:24] heh [14:24:52] heh [14:24:57] and they load mixed content [14:26:01] anyway, election time [14:26:03] bye-bye [14:26:07] see ya [14:33:08] we should really get a single certificate with all the domains added [14:33:41] "Chrome https://groups.google.com/forum/#%21msg/spdy-dev/UW0_X2GaMSQ/6sx-_So4aikJ across hosts that share the same IP and certificate." [14:34:02] though that wouldn't help us any [14:34:30] the domains loaded are bits, upload, geoiplookup and text, and we don't want them on the same ip [14:34:36] and they already share the same cert [14:34:54] bah, stupid client copied the url and not the text [14:34:55] fail [15:02:17] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [15:03:38] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 1.75 ms [15:07:32] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:13:23] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.037 second response time [16:26:04] New review: Dereckson; "Comment missing." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11746 [16:32:34] New patchset: Alex Monk; "(bug 37457) Add French and Italian Wikibooks as an import source at Vietnamese Wikibooks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/11746 [16:32:40] New review: jenkins-bot; "Build Successful " [operations/mediawiki-config] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/11746 [16:35:25] New review: Dereckson; "(no comment)" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11746 [18:03:24] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [19:04:41] PROBLEM - Puppet freshness on db29 is CRITICAL: Puppet has not run in the last 10 hours [20:20:32] RECOVERY - Puppet freshness on virt0 is OK: puppet ran at Sun Jun 17 20:20:14 UTC 2012 [20:40:30] New review: Petrb; "can we have a link to ticket in bugzilla related to this change" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/11748 [20:51:35] New review: Alex Monk; "No because there isn't one as far as I know." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/11748 [22:02:12] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [22:53:12] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: Puppet has not run in the last 10 hours [23:38:30] New review: Skizzerz; "Patch looks syntactically correct, and there appears to be developer support for getting this change..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/11748 [23:41:38] PROBLEM - Host search23 is DOWN: PING CRITICAL - Packet loss = 100% [23:42:50] RECOVERY - Host search23 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [23:44:16] New review: Dereckson; "(no comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/11747