[00:00:33] New patchset: MaxSem; "Postgres module for OSM" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36155 [00:00:41] New patchset: MaxSem; "WIP: OSM module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222 [00:03:46] that's not really it [00:05:20] PROBLEM - Host constable is DOWN: PING CRITICAL - Packet loss = 100% [00:05:27] New patchset: Reedy; "Expose CDB files for user download" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52315 [00:05:55] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52315 [00:07:01] New patchset: MaxSem; "WIP: OSM module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222 [00:08:03] Hey greg-g and Reedy, is there a time this week when we can schedule deploying https://gerrit.wikimedia.org/r/#/c/21322 ? [00:08:10] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:08:01 UTC 2013 [00:08:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:08:50] PROBLEM - Puppet freshness on virt11 is CRITICAL: Puppet has not run in the last 10 hours [00:09:04] New patchset: Reedy; "Kill asterix" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52316 [00:09:15] !log hashar synchronized wmf-config/InitialiseSettings-labs.php [00:09:20] Logged the message, Master [00:09:26] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52316 [00:09:28] !log snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2) [00:09:33] Logged the message, Master [00:09:33] apergos: ^^^^ snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2) [00:09:46] hashar: It got reinstalled recently ish [00:09:51] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:09:40 UTC 2013 [00:09:51] csteipp: Any reason we can't just do it now? ;) [00:10:20] Reedy: if Ryan_Lane is ok with that, I'm all for it. [00:10:30] fine with me [00:10:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:10:40] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:10:38 UTC 2013 [00:10:45] I'm going to be around for at least another couple of hours [00:10:54] Reedy: should still be out of dsh groups I guess [00:11:01] I'll notify ops list, so they know it changed [00:11:13] hashar: just need someone to run sync-common as root on it [00:11:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:11:48] csteipp: Do you want to do it? [00:12:03] Reedy: I can [00:12:14] Just merge, pull, and sync file, right? [00:12:16] New patchset: Reedy; "(bug 39380) Enabling secure login (HTTPS)." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/21322 [00:12:21] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:12:12 UTC 2013 [00:12:27] Yup [00:12:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:12:50] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:12:48 UTC 2013 [00:13:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:14:30] !log hashar synchronized wmf-config/InitialiseSettings-labs.php [00:14:35] Logged the message, Master [00:15:47] New patchset: Ryan Lane; "Cleanup backups older than 7 days on wikitech" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52321 [00:16:23] notpeter: nagios is red! :-] Regarding Lucene, the search boxes are part of the dsh mediawiki-installation group. They do receive copies of the InitialiseSettings.php file under /usr/local/apache/common-local. I guess we can get rid of the cronjob. [00:17:08] Change merged: CSteipp; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/21322 [00:18:47] csteipp: (typing what I just voice) yeah, whenever you wanna do it today :) [00:19:04] * greg-g likes transparency, or something [00:19:55] hashar: that's just the search indexers [00:21:00] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:20:51 UTC 2013 [00:21:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:21:49] !log csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login' [00:21:55] Logged the message, Master [00:24:15] notpeter: :( [00:25:01] New patchset: MaxSem; "WIP: OSM module" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36222 [00:25:42] Change abandoned: MaxSem; "Merged with https://gerrit.wikimedia.org/r/#/c/36222/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/36155 [00:26:27] hashar: I mean, we totally could just make every search node a mediawiki installation [00:26:35] but this seems like a very heavy-handed approach.... [00:26:40] hm, are there supposed to be lock icons next to the create account and log in links on the user login page with this, csteipp? [00:27:03] notpeter: I am not sure we are willing to add 30 or so boxes to the main sync scripts [00:27:12] notpeter: so that was probably not a good idea :-] [00:27:21] Those are there now... it would be nice if they were styled a bit more [00:27:38] so: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=SSL%2520cluster%2520eqiad&tab=m&vn= [00:27:43] and: https://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&s=by+name&c=SSL%2520cluster%2520esams&tab=m&vn= [00:27:58] that's how we'll know if we need to revert it :) [00:29:08] binasher: how does the naming in gdash-dashboards/ work? [00:31:26] AaronSchulz: the directory names map to URI path names while the dash.yaml naming is how it appears in the menu [00:31:41] csteipp: <3 [00:31:55] this makes using the sites so much less annoying [00:31:58] AaronSchulz: graph order is just by lexical sort of the .graph files under a dash directory [00:32:11] * AaronSchulz just wants to split lockmanger and streamfile [00:35:27] AaronSchulz: oh yeah :) i'll review if you'd like [00:36:25] mutante-away: i reviewed the SPF thing and commented on the ticket [00:37:32] !log added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia [00:37:37] Logged the message, Master [00:38:49] PROBLEM - Solr on vanadium is CRITICAL: Average request time is 460.14417 (gt 400) [00:41:55] New patchset: Pyoungmeister; "site.pp changes for pmtpa x1 shard" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52328 [00:42:42] solr on vanadium thinks it's lsearchd [00:42:59] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52328 [00:43:45] binasher: lol [00:44:09] New review: Hashar; "Works the same as the other helpers and Chris confirmed that is pretty much the authoritative db." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/52253 [00:45:29] hm, having a production service used by mediawiki running on vanadium probably isn't a good idea [00:45:48] New patchset: Ryan Lane; "Cleanup backups older than 7 days on wikitech" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52321 [00:46:09] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 00:46:04 UTC 2013 [00:46:39] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [00:46:59] PROBLEM - MySQL Replication Heartbeat on db38 is CRITICAL: NRPE: Unable to read output [00:47:49] PROBLEM - mysqld processes on db38 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [00:49:09] PROBLEM - MySQL Replication Heartbeat on db36 is CRITICAL: NRPE: Unable to read output [00:50:20] RoanKattouw: constable was stuck on formatting :-/ may finish this up tomorrow [00:50:23] stupid body [00:51:03] OK take your time [00:51:49] RECOVERY - mysqld processes on db38 is OK: PROCS OK: 1 process with command name mysqld [00:52:05] New patchset: Asher; "adding twemproxy to mediawiki::packages" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52333 [00:52:17] notpeter: does ^^ look ok to you? [00:53:12] binasher: do you specifically want that on every mediawiki installtion? [00:53:15] or just every apache? [00:54:08] every mediawiki installation that might talk to memcached [00:54:51] New patchset: Aaron Schulz; "Split out LockManager/StreamFile graphs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52335 [00:54:56] * AaronSchulz really wants the arbitrary name hashing [00:55:14] binasher: I just hacked that up ^ [00:56:49] these will be more interesting when a real LockManager is in use :) [00:58:14] Can I get someone else to help test the ssl login, by unchecking "Stay connected to HTTPS after login", and seeing if they go back to http? [00:58:49] I'm getting kept in ssl.. [00:59:04] trying to confirm if it's just me.. [00:59:24] AaronSchulz looks ok, i'm going to merge [00:59:25] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52335 [01:00:09] RECOVERY - mysqld processes on db36 is OK: PROCS OK: 1 process with command name mysqld [01:00:35] csteipp: Yeah, seems to stay on ssl [01:01:06] binasher: I need to add some stats to jobq too [01:01:11] * AaronSchulz is done for today [01:01:47] i think the jobq stats from mediawiki are still a lil janky, at least for deduplication [01:02:08] csteipp: Almost sounds like it's following $wgSecureLoginDefaultHTTPS, not the checkbox value [01:02:30] if ( $wgSecureLoginDefaultHTTPS && $this->mAction != 'submitlogin' && !$this->mLoginattempt ) { [01:02:30] $this->mStickHTTPS = true; [01:02:30] } [01:02:30] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [01:04:28] Yeah, the if ( $wgSecureLogin && !$this->mStickHTTPS ) { should account for that on redirect though [01:04:44] Lets flip it and see if behavior still follows [01:04:44] moment [01:05:45] !log reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing' [01:06:13] The checkbox is still checked :/ [01:06:30] hmm, memory [01:07:12] yeah, it's off for me on enwiki [01:07:30] PROBLEM - Puppet freshness on ms1004 is CRITICAL: Puppet has not run in the last 10 hours [01:07:40] but... logging in I still get ssl [01:07:44] Same behaviour though [01:07:46] indeed [01:08:27] !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS' [01:08:30] At which point should it take you back to http? [01:09:04] When you do the login post, you should get a redirect back to http [01:09:05] The return to [last page]? [01:09:12] ah, straight away then [01:09:55] Yeah [01:09:56] New patchset: Ryan Lane; "Fix nova config for network bonds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52338 [01:10:14] Guess we need to check if ( $wgSecureLogin && !$this->mStickHTTPS ) { in executeReturnTo [01:10:45] We do on line 992, right? [01:11:00] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [01:11:12] New patchset: Hashar; "adapt Lucene configuration file to support beta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52340 [01:11:19] Yeah. I mean dump the values on that line to confirm it's what it's supposed to be [01:11:35] ah, gatcha [01:12:12] The weird issue is that locally, it works fine [01:12:30] (my local dev instances) [01:12:46] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52321 [01:14:05] New patchset: Ryan Lane; "Fix nova config for network bonds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52338 [01:15:00] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [01:17:06] New patchset: Ryan Lane; "Fix nova config for network bonds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52338 [01:17:20] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 01:17:10 UTC 2013 [01:17:40] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [01:17:56] Ugh [01:18:33] Better we make them stay on https than always back to http [01:19:05] I agree... but I feel bad for screwing people who really really want http [01:19:46] New review: Asher; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/52340 [01:20:26] We should make a special site for them to browse wikipedia on ie6 [01:20:28] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52338 [01:20:51] I support that :) [01:21:11] So, I need to leave the office soon... Is this bad enough that we should revert? [01:21:20] this only affects ie6? [01:21:26] No, everyone [01:21:28] ah [01:22:00] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Wed Mar 6 01:21:59 UTC 2013 [01:22:39] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52333 [01:25:10] PROBLEM - Varnish traffic logger on cp1033 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [01:25:15] csteipp: well, it's broken.... [01:25:27] I'd say let's revert [01:25:36] Cool. I've got a draft for it.. [01:25:37] at minimum it may confuse users [01:25:46] Yep [01:25:56] https://gerrit.wikimedia.org/r/#/c/52344/ anyone? [01:25:57] but it may also affect some users who don't have https, though I have a feeling it's a very, very small set of users [01:28:06] Reedy: mind if I reset to HEAD in common, and then push a revert patch? [01:28:10] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Wed Mar 6 01:28:05 UTC 2013 [01:28:28] I thought I'd reverted that already, haha [01:28:30] Yeah, feel free [01:29:43] Why won't my notifications go away on wikitech? :/ [01:31:20] RECOVERY - Puppet freshness on virt9 is OK: puppet ran at Wed Mar 6 01:31:12 UTC 2013 [01:31:23] Reedy: can you +2 https://gerrit.wikimedia.org/r/#/c/52344/ ? [01:32:39] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52344 [01:33:50] !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch' [01:34:44] * csteipp cries a little on the inside [01:35:01] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [01:35:56] New patchset: Ryan Lane; "Ok. I give up trying to do this a sane way." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52348 [01:37:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/52348 [01:40:12] Ryan_Lane: Can I get some more rights (sysop?) on new wikitech please? Was going to setup a couple of gadgets but I can't edit-interface :( [01:40:17] RECOVERY - Varnish traffic logger on cp1033 is OK: PROCS OK: 3 processes with command name varnishncsa [01:43:35] Reedy: hm. [01:44:47] so… there's an issue with doing so [01:45:14] that would allow you to hijack others' accounts [01:45:23] having sysop? [01:45:26] yes [01:45:39] Can we add a group with editinterface then? [01:45:44] same same [01:45:48] Ohh [01:46:36] Reedy: want me to install it? [01:46:50] all ops folks can do this [01:46:59] I can likely give you additional permissions [01:47:03] I wonder if there's a nice way we can export the relevant pages [01:47:14] Cause from memory it's a little fiddly getting all the messages set [01:47:24] but, if I do that, you'll need to have two factor auth enabled [01:47:57] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [01:47:58] hm. there's no real way to enforce, that for editing interface, though [01:48:07] s/,// [01:55:00] Ryan_Lane: Duh [01:55:03] I forgot MaxSem added it [01:55:04] https://www.mediawiki.org/wiki/Special:Gadgets/export/Navigation_popups [01:55:07] ^ Export feature! :D [01:55:11] heh [01:55:27] Import that, and then on MediaWiki:Gadgets-definition add Navigation_popups|popups.js|navpop.css [01:55:30] Should be somewhere near.. [01:55:57] Lots of JS.. [01:56:03] oh [01:56:07] transwiki import works too [01:57:02] Even better [01:57:05] is the import source for that mw? [01:57:36] Presumably.. [01:57:50] Never tried transwiki importing gadgets though [01:58:00] I think it doesn't work [01:58:40] https://www.mediawiki.org/w/index.php?title=Special%3AExport&pages=MediaWiki%3Agadget-Navigation_popups%0D%0AMediaWiki%3AGadget-popups.js%0D%0AMediaWiki%3AGadget-navpop.css%0D%0A&wpDownload=1&templates=1 [01:58:52] Wonder if putting that in open file will work.. [01:59:24] hm [01:59:30] I don't see gadgets listed in the preferences [01:59:56] i'm guessing you've got memcached setup? [02:00:02] yeah [02:02:28] bad formatting [02:02:37] it's there now [02:02:40] aha [02:03:12] Yup, and it works. Yay [02:03:20] Thanks [02:03:25] yw [02:03:26] i think we can live with [02:03:29] I need to add a message for it [02:03:37] I'm going to export it from mediawiki [02:04:05] It should almost be in WikimediaMessages or something [02:04:31] done :) [02:04:54] https://wikitech.wikimedia.org/w/index.php?title=W:Wikipedia:Tools/Navigation_popups&action=edit&redlink=1 [02:04:58] Aweosme link :D [02:05:21] bleh [02:05:46] MediaWiki:gadget-Navigation_popups [02:06:09] hm [02:06:14] why does w: not work? [02:06:41] https://wikitech.wikimedia.org/w/api.php?action=query&meta=siteinfo&siprop=interwikimap&sifilteriw=local [02:06:45] https://noc.wikimedia.org/conf/interwiki.cdb [02:06:51] * Ryan_Lane groans [02:06:56] $wgInterwikiCache = "$IP/cache/interwiki.cdb"; [02:07:08] should I just download that cdb? [02:07:42] saves you having to mess around with the dumpInterwiki maintenance script [02:08:02] At worst it might need updating from time to time [02:08:12] seems that didn't help [02:08:52] New patchset: Aude; "Settings for deploying wikidata to more wikipedias [WIP]" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52351 [02:09:13] Reedy: don't merge it yet :) [02:09:18] heh [02:11:27] PROBLEM - Puppet freshness on es3 is CRITICAL: Puppet has not run in the last 10 hours [02:13:09] New review: Aude; "still need to add settings for sort order of interwiki links, prepending, etc." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/52351 [02:13:26] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [02:14:26] PROBLEM - Puppet freshness on cp1003 is CRITICAL: Puppet has not run in the last 10 hours [02:26:56] PROBLEM - Varnish traffic logger on cp1028 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [02:28:21] !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013 [02:28:43] Krenair: [02:28:44] 22 req] Exception from line 202 of /usr/local/apache/common-local/php-1.21wmf10/includes/Autopromote.php: Unrecognized condition APCOND_FR_EDITSUMMARYCOUNT for a [02:28:44] utopromotion! [02:28:44] 12 Exception from line 202 of /usr/local/apache/common-local/php-1.21wmf10/includes/Autopromote.php: Unrecognized condition APCOND_FR_EDITSUMMARYCOUNT for autopr [02:28:44] omotion! [02:29:01] Oh, I wonder if they're used too early [02:29:14] maybe not [02:33:06] PROBLEM - Varnish traffic logger on cp1023 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [02:33:06] PROBLEM - Varnish traffic logger on cp1033 is CRITICAL: PROCS CRITICAL: 2 processes with command name varnishncsa [02:35:46] New patchset: Reedy; "Disable trwiki entries for wmgAutopromoteOnceonEdit" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52353 [02:36:02] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52353 [02:36:35] !log reedy synchronized wmf-config/InitialiseSettings.php [02:36:52] New review: Reedy; "Looks like it's slightly buggy" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/38252 [02:44:06] RECOVERY - Varnish traffic logger on cp1033 is OK: PROCS OK: 3 processes with command name varnishncsa [02:47:06] PROBLEM - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is CRITICAL: Connection timed out [02:49:37] RECOVERY - Puppet freshness on virt5 is OK: puppet ran at Wed Mar 6 02:49:29 UTC 2013 [02:49:37] RECOVERY - Puppet freshness on virt10 is OK: puppet ran at Wed Mar 6 02:49:29 UTC 2013 [02:49:37] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Wed Mar 6 02:49:30 UTC 2013 [02:49:37] RECOVERY - Puppet freshness on virt11 is OK: puppet ran at Wed Mar 6 02:49:30 UTC 2013 [02:49:47] RECOVERY - Puppet freshness on virt6 is OK: puppet ran at Wed Mar 6 02:49:37 UTC 2013 [02:49:47] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 02:49:39 UTC 2013 [02:49:47] RECOVERY - Puppet freshness on virt1007 is OK: puppet ran at Wed Mar 6 02:49:41 UTC 2013 [02:50:32] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [02:50:58] RECOVERY - Varnish traffic logger on cp1028 is OK: PROCS OK: 3 processes with command name varnishncsa [02:50:58] RECOVERY - Varnish traffic logger on cp1023 is OK: PROCS OK: 3 processes with command name varnishncsa [02:51:09] ACKNOWLEDGEMENT - LVS HTTP IPv4 on parsoidcache.svc.pmtpa.wmnet is CRITICAL: Connection timed out LeslieCarr yep [02:52:27] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 02:52:23 UTC 2013 [02:52:46] !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013 [02:53:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [03:08:25] who wants to boot morebots? [03:09:19] * jeremyb_ nominates LeslieCarr [03:23:57] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 03:23:47 UTC 2013 [03:24:27] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [03:28:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 8.630 second response time [03:41:47] PROBLEM - Puppet freshness on virt1005 is CRITICAL: Puppet has not run in the last 10 hours [04:03:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 6.826 second response time [04:11:47] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: Puppet has not run in the last 10 hours [04:11:47] PROBLEM - Puppet freshness on amssq46 is CRITICAL: Puppet has not run in the last 10 hours [04:11:47] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [04:11:47] PROBLEM - Puppet freshness on ssl3003 is CRITICAL: Puppet has not run in the last 10 hours [04:13:01] PROBLEM - Puppet freshness on amslvs2 is CRITICAL: Puppet has not run in the last 10 hours [04:13:01] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: Puppet has not run in the last 10 hours [04:13:01] PROBLEM - Puppet freshness on amssq32 is CRITICAL: Puppet has not run in the last 10 hours [04:13:01] PROBLEM - Puppet freshness on amssq38 is CRITICAL: Puppet has not run in the last 10 hours [04:13:01] PROBLEM - Puppet freshness on amssq43 is CRITICAL: Puppet has not run in the last 10 hours [04:14:27] PROBLEM - Puppet freshness on amssq31 is CRITICAL: Puppet has not run in the last 10 hours [04:14:27] PROBLEM - Puppet freshness on amssq34 is CRITICAL: Puppet has not run in the last 10 hours [04:14:27] PROBLEM - Puppet freshness on amssq33 is CRITICAL: Puppet has not run in the last 10 hours [04:14:27] PROBLEM - Puppet freshness on amssq37 is CRITICAL: Puppet has not run in the last 10 hours [04:14:27] PROBLEM - Puppet freshness on amssq40 is CRITICAL: Puppet has not run in the last 10 hours [04:14:28] PROBLEM - Puppet freshness on amssq41 is CRITICAL: Puppet has not run in the last 10 hours [04:14:28] PROBLEM - Puppet freshness on amssq35 is CRITICAL: Puppet has not run in the last 10 hours [04:14:29] PROBLEM - Puppet freshness on amssq52 is CRITICAL: Puppet has not run in the last 10 hours [04:14:29] PROBLEM - Puppet freshness on amssq53 is CRITICAL: Puppet has not run in the last 10 hours [04:14:30] PROBLEM - Puppet freshness on amssq59 is CRITICAL: Puppet has not run in the last 10 hours [04:14:30] PROBLEM - Puppet freshness on amssq61 is CRITICAL: Puppet has not run in the last 10 hours [04:14:31] PROBLEM - Puppet freshness on cp3021 is CRITICAL: Puppet has not run in the last 10 hours [04:14:31] PROBLEM - Puppet freshness on knsq16 is CRITICAL: Puppet has not run in the last 10 hours [04:14:32] PROBLEM - Puppet freshness on cp3009 is CRITICAL: Puppet has not run in the last 10 hours [04:14:32] PROBLEM - Puppet freshness on amssq56 is CRITICAL: Puppet has not run in the last 10 hours [04:14:33] PROBLEM - Puppet freshness on knsq18 is CRITICAL: Puppet has not run in the last 10 hours [04:14:33] PROBLEM - Puppet freshness on knsq19 is CRITICAL: Puppet has not run in the last 10 hours [04:14:34] PROBLEM - Puppet freshness on amssq42 is CRITICAL: Puppet has not run in the last 10 hours [04:14:34] PROBLEM - Puppet freshness on amssq49 is CRITICAL: Puppet has not run in the last 10 hours [04:14:35] PROBLEM - Puppet freshness on knsq24 is CRITICAL: Puppet has not run in the last 10 hours [04:14:35] PROBLEM - Puppet freshness on knsq21 is CRITICAL: Puppet has not run in the last 10 hours [04:14:36] PROBLEM - Puppet freshness on ssl3001 is CRITICAL: Puppet has not run in the last 10 hours [04:14:36] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: Puppet has not run in the last 10 hours [04:14:37] PROBLEM - Puppet freshness on amssq62 is CRITICAL: Puppet has not run in the last 10 hours [04:14:37] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [04:15:27] PROBLEM - Puppet freshness on knsq26 is CRITICAL: Puppet has not run in the last 10 hours [06:12:43] PROBLEM - Puppet freshness on virt0 is CRITICAL: Puppet has not run in the last 10 hours [06:12:43] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Puppet has not run in the last 10 hours [06:29:42] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 06:29:36 UTC 2013 [06:30:22] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [06:30:43] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 06:30:34 UTC 2013 [06:31:23] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [06:31:52] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 06:31:44 UTC 2013 [06:32:22] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [07:01:08] New review: Hashar; "(1 comment)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52340 [07:01:15] New patchset: Hashar; "adapt Lucene configuration file to support beta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/52340 [07:05:33] morebots still needs boot [07:05:38] maybe Ryan_Lane or paravoid ? [07:06:31] done [07:06:55] !log test [07:06:59] hm [07:07:12] !log restarted morebots [07:07:18] Logged the message, Master [07:07:48] !log 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS' [07:07:51] !log 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch' [07:07:54] Logged the message, Master [07:07:55] !log 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013 [07:07:59] Logged the message, Master [07:08:04] Logged the message, Master [07:08:06] !log 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php [07:08:09] !log 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013 [07:08:11] Logged the message, Master [07:08:16] Logged the message, Master [07:09:07] !log 01:05:45 <+logmsgbot_> !log reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing' [07:09:12] Logged the message, Master [07:09:21] * jeremyb_ reorders onwiki [07:09:29] Ryan_Lane: danke [07:11:13] PROBLEM - ircecho_service_running on neon is CRITICAL: Connection refused by host [07:11:52] PROBLEM - MySQL disk space on neon is CRITICAL: Connection refused by host [07:13:58] huh, the bot logged that last one i did to the wiki but never ACK'd in the channel [07:15:11] [01:09:12 AM] Logged the message, Master [07:16:21] boxofjuice: what timezone? [07:16:27] CST [07:16:31] oh [07:16:35] i meant the first time around [07:16:47] i count [07:16:54] 06 01:05:45 <+logmsgbot_> !log reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing' [07:16:57] you making 6 [07:16:57] 06 01:08:20 -!- morebots [~morebots@wikitech-static.wikimedia.org] has quit [Ping timeout: 252 seconds] [07:17:11] and 6 responses [07:17:17] it never ACK'd but it was already on the wiki [07:17:22] oh [07:17:28] idk then [07:17:35] (TZ=UTC there) [07:45:19] RECOVERY - ircecho_service_running on neon is OK: PROCS OK: 3 processes with args ircecho [07:45:39] RECOVERY - MySQL disk space on neon is OK: DISK OK [08:07:59] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 08:07:55 UTC 2013 [08:08:29] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [08:08:39] RECOVERY - Puppet freshness on spence is OK: puppet ran at Wed Mar 6 08:08:30 UTC 2013 [08:09:29] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [08:44:53] where is the http 301 response to "http://en.wikipedia.org" requests configured? [08:47:45] apache-config repo? [08:47:51] ori-l: 301 for what? [08:48:14] to Main_page [08:48:42] i read a good article on mobile web performance http://queue.acm.org/detail.cfm?id=2441756 [08:48:48] which prompted me to poke around a little [08:49:06] and i noticed that if a user types 'en.wikipedia.org' on a mobile device we redirect them twice [08:49:34] see http://i.imgur.com/J4CAueo.png [08:49:58] yeah, not surprising [08:50:24] Alias /wiki /usr/local/apache/common/docroot/mediawiki/w/index.php [08:50:26] we should just push mobile user-agent detection to the initial request [08:50:27] RewriteRule ^/$ /w/index.php [08:51:23] there's various optimizations we could do [08:51:36] * ori-l is tempted to set that as the title :P [08:51:56] the fact that text is still squid isn't helping much, but even so there's room for improvements [08:52:07] feel free :) [08:52:33] I'd love to see them (and hence gladly review/possibly merge) [08:52:50] i think mobile text is varnish, no? [08:52:56] it is [08:53:03] which means the ua detection is vcl, which means having to convert it to apache syntax [08:53:34] nah, I don't think this can fly [08:53:36] i was hoping to make a splash with the discovery and leave the implementation work to you lot :P [08:53:59] why not? [08:54:12] ua detection isn't a cheap operation and produces multiple variants (21 iirc) of e.g. / [08:54:31] this might be expensive [08:54:44] it'll need some careful benchmarking [08:54:44] * ori-l is tempted to set *that* as the title [08:55:06]