[00:18:47] New patchset: Asher; "varnish3 pkg wants to restart with the varnish3 init, while puppet installs to just init.d/varnish breaking pkg upgrades on hosts that reference geoip libs in their vcl this is a hackish workaround that should work on a second puppet run, until the pkg is" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1821 [00:19:04] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1821 [00:19:04] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1821 [00:19:40] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [00:19:45] !log depooling cp3002, upgrading varnish [00:19:46] Logged the message, Master [00:23:37] !log repooled cp3002 [00:23:39] Logged the message, Master [00:23:48] zzz [00:27:30] PROBLEM - Host mw1102 is DOWN: PING CRITICAL - Packet loss = 100% [00:36:10] RECOVERY - Host mw1102 is UP: PING OK - Packet loss = 0%, RTA = 26.45 ms [00:45:50] PROBLEM - RAID on ms1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:52:10] PROBLEM - Disk space on mw68 is CRITICAL: DISK CRITICAL - free space: /tmp 51 MB (2% inode=95%): [01:11:36] New patchset: Asher; "use varnish 3.0.2 on all hosts" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1822 [01:11:51] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1822 [01:11:52] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1822 [02:02:33] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Tue Jan 10 02:02:25 UTC 2012 [02:05:16] !log LocalisationUpdate completed (1.18) at Tue Jan 10 02:05:15 UTC 2012 [02:05:17] Logged the message, Master [02:14:13] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [04:16:35] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:21:45] RECOVERY - Disk space on es1004 is OK: DISK OK [04:39:21] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:00:41] RECOVERY - Disk space on mw68 is OK: DISK OK [06:23:17] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [06:50:28] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [06:53:38] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [07:20:47] !log nikerabbit synchronized php-1.18/extensions/Translate/TranslateEditAddons.php 'r108497' [07:21:00] Logged the message, Master [10:01:16] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 457027 MB (3% inode=99%): [10:02:56] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 448463 MB (3% inode=99%): [10:25:16] RECOVERY - MySQL slave status on es1004 is OK: OK: [11:39:07] !log nikerabbit synchronized php-1.18/extensions/Translate/MessageGroups.php 'Translate bugfix r108500' [11:39:10] Logged the message, Master [12:11:37] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [12:23:48] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [13:53:32] zzz =_=; [15:06:08] Reedy: are you about? [15:06:34] yes [15:06:47] great - https://bugzilla.wikimedia.org/show_bug.cgi?id=29114#c2 - are Liquid Thread deployments still on hold? [15:07:15] Not sure [15:07:21] Need to ask I guess [15:07:41] LQT isn't being really maintained now... [15:07:49] but I can find out [15:08:03] As I think there's a few requests now [15:08:04] okay thanks, yes because there's a thread on meta asking about it being deployed, http://meta.wikimedia.org/wiki/Meta:Babel#Enabling_Liquid_Thread_on_meta [15:08:08] Yeah [15:08:21] If meta can have it, I can't see why other wikis can't [15:09:41] well the thread is about getting support for it, so we need to know if we can get it on meta before supporting a discussion ;) [15:11:06] I'll speak to Robla/Alolita/Erik about it later [15:22:15] considering to switch from "planet" to "planet-venus" [15:22:27] "Planet Venus is a refactoring of Planet" [15:22:58] and as opposed to the classic planet, it's available as distro package [15:23:31] (the version we are running is an outdated nightly build) [15:23:55] planet as in our blog aggregator thing? [15:23:59] yes [15:24:18] it is half-puppetized currently, vhost setup, users and stuff is there [15:24:30] but the application files itself need to be deployed [15:25:14] so either build our own .deb, or instead switch to distro package.. and i think the latter is better, as long as it still does what we want it to do..and the existing config is at least compatible a bit [15:26:31] http://intertwingly.net/code/venus/ [16:04:25] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1470s [16:07:53] @replag [16:07:55] malafaya: No replag currently. See also "replag all". [16:09:14] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1760s [16:32:22] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [17:25:19] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:27:09] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:31:29] RECOVERY - RAID on ms1002 is OK: OK: State is Optimal, checked 2 logical device(s) [17:54:13] * jeremyb waves AaronSchulz [17:55:08] AaronSchulz: can you suggest some initial flaggedrevs settings for nyc.wikimedia.org ? too many choices for me [17:55:49] AaronSchulz: basic idea is most pages are open to everyone, a couple are held for review by autoconfirmed and a couple are held for review by sysop [17:57:40] and the flagging is to be just yes/no [17:58:07] was there a line missing? :) [17:59:46] i just want to be able to set a page to require review approximately the same way i would do page protection. and have all pages default initally to no review required [18:00:23] that sounds like enwiki's config, doesn't it?! [18:00:26] would you still want all pages to be reviewable but just not always using the stable version instead of the current? [18:00:41] hoo: enwiki's config is much longer than some of the other configs... [18:01:04] AaronSchulz: i guess only if it's set to require review? [18:02:03] AaronSchulz: do WMF wikis usually allow use of [[Special:Stabilization]]? [18:02:41] no, a few that used it early on still have it available [18:02:52] but you could certainly use that for this wiki [18:03:01] hoo: FYI, see http://noc.wikimedia.org/conf/highlight.php?file=flaggedrevs.php [18:03:36] AaronSchulz: in case you can't tell i've never used it as a sysop and mostly not even as a regular user [18:04:05] mh, ok [18:04:14] (i'm trying a little extra hard to get it right the first time so we don't have to keep doing shell reqs) [18:05:31] I need some help about Special:EmailUser feature. [18:05:32] I noticed all mails are received from wiki-mail.wikimedia.org and they show an WMF IP address, [18:05:34] Does it happen all the times? is it possible to get any info from original sender from email headers, eg IP? [18:06:33] nice way to CU users [18:07:03] No, the email originates from the server [18:07:05] not the sender [18:07:08] MikeOscar: of course they come from a WMF email server [18:08:18] Brownout_: I'm asking this for my own privacy :) [18:08:20] I just want to be sure that there is no way to trace the original sender. [18:09:09] MikeOscar: it sends them your email address... [18:10:18] That's OK, they need to know my email anyway, but only that? [18:10:40] MikeOscar: send me one and i'll see what i can find? [18:21:14] jeremyb: something like http://pastebin.com/yrNWk26r ? [18:30:46] New patchset: Lcarr; "Pushing ganglia aggregator correction for misc pmtpa" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1823 [18:31:35] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1823 [18:31:35] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1823 [18:34:12] AaronSchulz: so, flaggedrevs has no way to propose a page move? it just follows page protection settings and reviewer rights? [18:34:46] right [18:38:56] To be sure again, does using Special:EmailUser keep my IP private all the times? [18:38:58] I couldn't find any useful info about privacy on [[en:WP:EMAIL]]... [18:39:22] Yes [18:39:36] Unless you are an anon, nothing in MW will ever reveal your IP to anyone who's not a checkuser [18:40:12] well... depends on other things. like what custom user scripts you have enabled [18:40:23] surely most scripts are not security reviewed [18:40:38] (JS) [18:41:57] jeremyb: Even unreviewed JS has no way of getting your IP [18:42:10] Wait, I take that back [18:42:12] Of course they do [18:42:24] You'd need a helpful 3rd-party site but that's easily arranged [18:42:38] * jeremyb agrees :) [18:42:51] RoanKattouw: 1.18 and 1.18wmf are not synced yet ? [18:43:28] RoanKattouw: cause it seems this: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/103074 might not be in the 1.18wmf deploys.... [18:43:29] That's a Reedy question [18:43:40] Reedy: question ^^^ [18:43:49] gtg home now. [18:43:50] Looks like you're right [18:44:07] see also the bug report that was reopened because of that. [18:44:21] About JS, only if somewhere in the code a 3rd-party address is mentioned, right? [18:44:28] Yes [18:44:38] But you can obfuscate the hell out of that if you want [18:56:08] RECOVERY - Puppet freshness on storage3 is OK: puppet ran at Tue Jan 10 18:55:58 UTC 2012 [19:25:25] notpeter: can you join us at #wikimedia-labs please [19:31:12] Thanks everyone for your help, see you later :) [20:13:55] New patchset: Lcarr; "changing name of ganglia redirect making the tmpfs mount puppetized" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1824 [20:17:13] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1824 [20:17:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1824 [21:28:10] New patchset: RobLa; "Telling the world mediawiki rocks cuz it duz" [test/mediawiki/core] (master) - https://gerrit.wikimedia.org/r/1826 [21:32:34] New review: Catrope; "Clearly, this breaks lots of things :)" [test/mediawiki/core] (master) C: -1; - https://gerrit.wikimedia.org/r/1826 [21:33:37] nightyyy [21:36:08] New review: Catrope; "This also needs to support logging to different channels for different projects. I just noticed revi..." [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/1794 [22:09:13] New patchset: Lcarr; "Fixing redirect on nickel" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1827 [22:11:01] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1827 [22:11:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1827 [22:21:29] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [22:25:59] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Tue Jan 10 22:25:50 UTC 2012 [22:50:36] > I used to be able to view Wikipedia on my IE browser [on my mobile], but now I get a message that says: "Alert. This pages contains errors and cannot be displayed." This happens on any article, any time from any link to mobile Wikipedia? [22:50:57] from an OTRS email. Any pointers/ideas? Ask them not to use Windows mobile devices? (-; [22:51:45] A more descriptive error message would be helpful [22:53:11] They gave more info about their handset but not the error message... [23:02:05] gn8 folks [23:26:55] I created a bug. https://bugzilla.wikimedia.org/show_bug.cgi?id=33649 [23:26:59] report. [23:27:10] snaps for juancarlos. [23:29:59] New patchset: Asher; "initial carrier tagging code for wikipedia zero" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1828 [23:30:16] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1828 [23:30:46] New review: preilly; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/1828 [23:39:13] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1828 [23:39:14] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1828 [23:58:39] !log preilly synchronized php-1.18/extensions/MobileFrontend/ 'weekly update to mobile frontend' [23:58:40] Logged the message, Master [23:58:50] !log weekly mobile frontend push [23:58:51] Logged the message, Master