[00:20:59] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1613 [00:21:00] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1613 [01:43:08] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [02:00:18] !log LocalisationUpdate failed (1.18) at Sat Dec 17 02:00:18 UTC 2011 [02:00:27] Logged the message, Master [02:02:08] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [02:04:08] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [02:25:38] PROBLEM - Memcached on magnesium is CRITICAL: Connection refused [04:31:18] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours [05:10:42] anyone around? [05:11:02] hi [05:11:06] sup [05:11:17] nada [05:11:47] does anyone know why, when I type "WIKI:" followed by anything, into the search box on Wikipedia, it takes me to http://c2.com/cgi/wiki?Whatever ? [05:11:55] is this some new feature I have missed? [05:12:45] Mark_Ryan: its an interwiki prefix [05:13:13] why is "WIKI" the interwiki prefix for c2.com? [05:13:24] or perhaps more pertinently, what is c2.com? heh [05:14:04] Mark_Ryan: http://meta.wikimedia.org/wiki/Interwiki_map [05:14:34] http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap&sifilteriw=!local [05:15:53] given it appears to be broken in any event, maybe it would be worth seeing about removing the "wiki" prefix from referring to that specific random external site? [05:16:58] Mark_Ryan: i think it's neither new nor random nor broken [05:17:10] it's been like that for at least 7 years [05:17:15] http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/maintenance/interwiki.sql?annotate=6445 [05:18:17] actually over 8 years [05:19:17] golly [05:20:03] http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/Interwiki.php?revision=1284&view=markup&pathrev=1589#l94 [05:20:29] that's the beginning of time as far as is currently easily accessible. i think. it's there at the beginning [05:20:44] (there's CVS before that but i think it's harder to access) [05:21:12] we should probably leave it alone then. don't want to break lots of articles that use that prefix [05:21:29] hint: there was a wiki before wikipedia [05:21:45] hint: read https://en.wikipedia.org/wiki/Wiki [05:22:12] hint: I've been using Wikipedia since 2001, don't be patronising [07:28:20] !log thumb cleaner started up for the day [07:28:28] Logged the message, Master [07:37:46] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [08:36:06] PROBLEM - Puppet freshness on maerlant is CRITICAL: Puppet has not run in the last 10 hours [12:28:08] RECOVERY - MySQL slave status on es1004 is OK: OK: [13:01:30] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [13:01:30] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [13:01:30] PROBLEM - Puppet freshness on es1002 is CRITICAL: Puppet has not run in the last 10 hours [14:35:42] !log thumb cleaner to bed for the night... about 2 days left I think [14:35:51] Logged the message, Master [15:54:15] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: Puppet has not run in the last 10 hours [16:27:27] New patchset: Catrope; "Fix for r1516: also remove files from manifest" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1617 [16:30:23] New patchset: Catrope; "Remove stray slash" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1618 [16:37:46] New review: Catrope; "You cherry-picked this without cherry-picking https://gerrit.wikimedia.org/r/1048 , which introduced..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1558 [16:42:31] New patchset: Catrope; "Put mysql::client in its own class, and install it on bastion hosts too." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1619 [16:46:40] !log Found out why LocalisationUpdate was failing. Would have been fixed already if puppet had been running on fenari, but it's throwing errors. See r1617 and my comment on r1558 [16:46:50] Logged the message, Mr. Obvious [16:47:52] RoanKattouw: FYI: /var/log/l10nupdate hasn't been updated since november [16:48:08] I know [16:48:14] It's now logging to /var/log/l10nupdatelog/ [16:48:22] But something's blanking the log file every morning at 6:40 [16:48:26] it confused me [16:48:34] I've got a copy of last night's run in /home/catrope/l10nupdate.lo [16:48:35] g [16:48:52] (Got it by putting a cp in my crontab scheduled for 05:00) [16:49:18] I found the issue (/usr/local/bin is not in $PATH so calling l10nupdate-2 fails with command not found) [16:49:42] But, in an unrelated commit, I already removed that file and any references to it. That change would've been deployed if puppet hadn't been broken for fenari [16:49:57] Fenari hasn't had a successful puppet run for ~2 days [16:50:01] yeah I just looked those gerrit stuff [16:50:06] Yeah [16:50:13] r1617 fixes what would probably be an error [16:50:38] But the error caused by r1558 occurs first and is fatal [16:51:28] And now no one in ops is around to merge and deploy these changes :( [17:56:23] !log catrope synchronized wmf-config/CommonSettings.php 'Remove SVN dir setting, this is now passed in on the command line' [17:56:31] Logged the message, Master