[01:13:01] was the nagios-password changed during the last two hours? [01:13:19] nagios-mysql-password [01:14:33] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [01:20:35] DaBPunkt, Asher puppetised db31 at 2300... [01:20:43] dunno if that's related at all [01:21:45] * binasher looks at nagios.. doesn't see any of our mysql monitors failing [01:22:01] someway a new grant for nagios was insert into all databases of the toolserver [01:22:09] yes [01:22:19] the password should not have changed though [01:23:03] we use differend passwords for nagios. But it seems that mysql honors entry for Ips higher then entries for hostnames :( [01:24:26] well, this is one of the more interesting ways i've ever broken something. [01:25:38] binasher: the grant was not defined for s6 here, maybe you should take a look at s6 if everything is correct on your side [01:27:43] oh, thanks! looks like on s6, root@fenari doesn't have grant access and i missed the failure [01:28:55] DaBPunkt: are you going to drop the new nagios mysql.user rows on toolserver? [01:30:29] I had run "drop user 'nagios'@'10.%';" on our cluster, yes. If the problem re-occure, we will rename the nagios-user on our side [01:31:22] New patchset: Ryan Lane; "libvirt-bin's upstart script is rediculous, and doesn't let you change daemon options in the default file. So, let's update it's upstart to appease its stupidity." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1751 [01:32:08] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1751 [01:32:09] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1751 [01:32:27] hmm. the lint check almost never writes a message to the channel anymore :( [01:36:34] gn8 folks [01:36:51] nacht [01:44:02] PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1412s [01:46:44] New patchset: Ryan Lane; "Weirdness with -'s?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1752 [01:46:58] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1752 [01:46:58] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1752 [01:50:57] New patchset: Ryan Lane; "Work around upstart_job's failure." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1753 [01:51:38] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1753 [01:51:40] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1753 [02:05:02] !log LocalisationUpdate completed (1.18) at Thu Dec 29 02:05:01 UTC 2011 [02:05:16] Logged the message, Master [02:29:26] New patchset: Ryan Lane; "%s in this context is the short hostname, not the FQDN. This causes a certificate error." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1754 [02:29:39] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1754 [02:29:39] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1754 [02:29:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1754 [02:56:36] !log enabled instance live migration on virt2-4 [02:56:45] Logged the message, Master [02:57:20] now to figure out why libvirt won't start on virt4... [03:44:07] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:07:57] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [05:26:02] !log [14:32] !log restarting virt4 because upstart is an unreliable POS (from #mediawiki) [05:26:02] Logged the message, Master [05:26:02] I logged it in -operations [05:26:02] what kind of weird TZ is that? [13:37:24] hmm, any problems with the servers? Seems to have a hard time saving & finding revisions here, but unsure if it is perhaps my side :) [13:39:50] eia-writing, where? [13:40:16] nlwiki and enwiki [13:45:36] eia-writing, could it be for that Template loop detected: Template:Convert/F [13:50:41] Nemo_bis: well, fixing that doesn't help exactly, and I had also problems saving on nlwiki without such template :) [13:51:34] I guess you'll need a real expert to figure it out then :-p [13:52:14] I mean it saves allright, but it takes quite a while :) [13:54:51] PROBLEM - Backend Squid HTTP on knsq11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:03:01] PROBLEM - Host knsq11 is DOWN: PING CRITICAL - Packet loss = 100% [14:16:51] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [14:55:00] are there any rev delete experts in here? [15:13:51] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33410 - Allow IP edits on wikimania2012.wikimedia.org' [15:14:01] Logged the message, Master [15:37:05] woot [15:37:48] yay! [15:43:54] New patchset: Jgreen; "fix notification and puppetize offhost_backups-storage3, typo in dump_fundraisingdb" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1762 [15:44:07] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1762 [15:44:10] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1762 [15:44:11] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1762 [17:42:47] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [17:51:17] New patchset: Jgreen; "moving misc::fundraising out of misc-servers and into its own file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1763 [17:51:31] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1763 [17:55:27] New review: Jgreen; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1763 [17:55:27] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1763 [18:31:44] !log reedy synchronized wmf-config/InitialiseSettings.php 'bug 33407 33408 Babel stuff for iswiki' [18:31:53] Logged the message, Master [20:20:18] I don't know of a way to do that, but the people here may [20:23:05] Question. Suppose I wanted to cache some pages from some wiki, and want to be able to fetch a page only if it's newer than the one in my cache. If I stored the whole rendered HTML, I could just fetch the page from the server with an If-Modified-Since HTTP header. [20:23:21] That works, I checked. But what if I want to store the source instead, with Special:Export ? [20:23:39] How do I specify a limit date to Special:Export so that it sends me a page only if it's newer than the version I have? [20:24:06] Ah, maybe I need limit=1 and offset=time but not use curonly. [20:24:09] Let me try that. [20:31:59] no good, limit=1 returns the _oldest_ version [20:34:12] how then? any ideas? [20:34:28] ah, there's a dir parameter [20:35:00] no good: [20:35:43] if I submit with dir=desc&limit=1&offset=2011-10-01T00:00:00Z then I get the latest revision as of that date [20:35:53] that can make sense, but it's not what I want [20:36:19] argh