with some articles [03:41:03] it doesn't makes big changes in stubs [03:41:33] not sure if it would affect complex tables though [03:43:05] Ryan_Lane: ping? [03:43:54] jeremyb: ? [03:44:10] depends on the browser and what justify it uses, some use (or used in the pasted) force justify or left justify (which will alter the output as well, also screen size comes into play as well) [03:44:30] Ryan_Lane: you cleaned the SAL but it's still on twitter (and i guess identi.ca) someone just mailed wikitech to inform us. idk who has access to those accounts [03:44:54] yes, identi.ca too [03:44:58] ugh [03:45:11] not sure I have the twitter account info [03:45:19] I think I do have the identi.ca [03:45:23] twitter is via identi.ca so only it has to be deleted and the delete will propagate to twitter [03:45:42] (it says right on it via identica) [03:46:37] three people are running morebots on wikitech? [03:46:55] rly?! [03:47:07] how do you figure? [03:47:20] (process list?) [03:47:24] because I can see it in ps ;) [03:47:29] huh [03:47:34] 2 deadish? [03:47:49] probably [03:47:51] what's lsof say? [03:47:58] (for network connections) [03:48:20] maybe from netspilts or something and they are dangling over the ledge [03:48:49] What's this, https://twitter.com/#!/wikimediatech O_o [03:48:50] hmm.. I have zero technical understanding so I can't see what could go wrong [03:49:14] hopefully part of training should have been "Hey this goes out to the public" [03:49:20] no clue. just going to kill all three and restart it [03:49:28] also, cleaned twitter and identi.ca [03:49:52] techman224: it's a twitter/identica feed of the server admin log [03:51:11] Ryan_Lane: i'm replying or do you want to? [03:51:19] replying to what? [03:51:42] probably the post to wikitech [03:51:51] Ryan_Lane: wikitech [03:52:08] * jeremyb is near done writing a reply fwiw [03:52:12] I'll reply [03:52:49] * jeremyb goes to see about the other recent !log vandalism being on twitter still or not [03:53:29] well, we don't get vandalism of it too often. [03:54:04] i noticed [03:54:44] Well we are a back of the woods type channel, there generally isn't much point in trying to troll this channel [03:54:54] yep [03:55:53] I thought morebots had a whitelist? [03:56:00] nope [03:56:11] could have swore it used to [03:56:21] I'm pretty sure it never has. [03:56:40] I rewrote most of it in the Labs channel, and don't believe I saw any whitelist feature [03:56:53] Ryan_Lane: the other dents FYI: 90679058 90683405 90684163 90678942 [03:57:02] since I thought it used to only catch !logs from logmsgbot [03:57:36] nah. that's for deployments. ops generally doesn't use that [03:59:36] is the old src for it anywhere on svn? [04:00:17] OverlordQ: tools > adminlogbot [04:03:53] huh, thought I tried it before and it ignored me [04:04:24] You need to remember the !log at the front [04:04:34] i've even logged to it in the past [04:04:34] I did, this was quite a while back though [04:04:47] around when it started calling people master [04:04:58] what did it say before that? [04:05:19] not a clue, it's been a while [04:05:47] it calls different people diff things [04:05:53] but defaults to master iirc [04:11:49] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [04:13:14] looks like it used to not reply at all [04:16:45] ah, till july 15th '09 it used to be though private message instead of in the channel, that might be what I was thinking of [04:45:05] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [05:22:07] RECOVERY - Disk space on db30 is OK: DISK OK [05:22:34] RECOVERY - MySQL disk space on db30 is OK: DISK OK [05:36:04] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [06:26:44] Stupid question, the extended image syntax ( http://en.wikipedia.org/wiki/Wikipedia:Extended_image_syntax ) works on all wmf wikis, right? [06:31:48] PROBLEM - Squid on brewster is CRITICAL: Connection refused [06:34:41] Snowolf: Yes, well apart from the templated annonation overlay [06:35:04] p858snake|l: great, thanks a lot! [06:51:48] any comments? https://meta.wikimedia.org/w/index.php?title=System_administrators&diff=3464909&oldid=3397644 [06:52:30] also, is it kept up to date? [06:53:28] i see some people missing that i *think* maybe should be there? or should be moved now. does hashar have root now? lcarr, hexmode [06:53:38] sumanah? [06:53:46] Joan: ^ [06:55:09] and should hashar's name be fixed? (i assume but wanted to double check) [06:56:24] * jeremyb sleeps [07:11:25] I don't think it's kept up to date at all. [07:17:52] it would be fine if it were, not reasonnot to [07:19:45] why does that have a root col... seems redundant [07:20:09] also the shell col as well [07:20:27] there are people who can do shel req who aren't root [07:20:43] Thgen they are in the bottom "shell" half of the table [07:21:53] shrug [07:22:38] * apergos is too sleepy to care [08:10:29] PROBLEM - Lucene on search9 is CRITICAL: Connection timed out [08:11:59] PROBLEM - LVS Lucene on search-pool1.svc.pmtpa.wmnet is CRITICAL: Connection timed out [08:14:14] RECOVERY - Lucene on search9 is OK: TCP OK - 0.002 second response time on port 8123 [08:14:23] RECOVERY - LVS Lucene on search-pool1.svc.pmtpa.wmnet is OK: TCP OK - 0.001 second response time on port 8123 [08:32:59] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [08:58:11] PROBLEM - Lucene on search4 is CRITICAL: Connection refused [09:53:45] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [09:54:48] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.006 second response time on port 8123 [09:56:09] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:58:42] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [10:00:57] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [10:37:49] RECOVERY - Lucene on search15 is OK: TCP OK - 8.994 second response time on port 8123 [10:46:04] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [11:04:22] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [11:05:25] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.005 second response time on port 8123 [11:13:09] RECOVERY - Lucene on search15 is OK: TCP OK - 9.008 second response time on port 8123 [11:20:39] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [11:23:39] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [11:26:03] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.005 second response time on port 8123 [11:29:41] New patchset: Pyoungmeister; "making lucene check even less sensitive" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2671 [11:30:48] Hi. [11:30:59] What's wrong with search? [11:31:08] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2671 [11:31:12] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2671 [11:32:46] AVRS: One of the search clusters is down more than likely [11:32:58] p858snake|l: ok, thanks [11:33:04] e.g. https://ru.wikipedia.org/wiki/%D0%A1%D0%BB%D1%83%D0%B6%D0%B5%D0%B1%D0%BD%D0%B0%D1%8F:Search?search=%D0%9D%D0%BE%D0%B2%D0%BE%D0%B7%D0%B0%D0%B2%D0%BE%D0%B4%D1%81%D0%BA%D0%B0%D1%8F doesn't find anything [11:33:26] like https://ru.wikipedia.org/wiki/%D0%9D%D0%BE%D0%B2%D0%BE%D0%B7%D0%B0%D0%B2%D0%BE%D0%B4%D1%81%D0%BA%D0%B0%D1%8F_%D1%83%D0%BB%D0%B8%D1%86%D0%B0_(%D0%9C%D0%BE%D1%81%D0%BA%D0%B2%D0%B0) [11:46:41] works now [11:58:17] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [12:01:17] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [12:04:17] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [12:04:17] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [12:05:47] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [12:06:50] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.001 second response time on port 8123 [12:20:44] any sysadmin online? [12:37:35] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [12:42:50] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.006 second response time on port 8123 [12:57:59] @sysadmin [12:58:07] hmmm [12:58:45] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [12:59:48] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.002 second response time on port 8123 [13:05:08] axpde: if you don't say what you want, it's unlikely anyone will respond [13:05:23] https://bugzilla.wikimedia.org/show_bug.cgi?id=34512 [13:05:59] I thought it would be annoying to post my request without anyone able to go for it ... [13:34:18] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [13:35:21] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.004 second response time on port 8123 [13:39:42] PROBLEM - LVS Lucene on search-pool3.svc.pmtpa.wmnet is CRITICAL: Connection timed out [13:39:43] PROBLEM - LVS Lucene on search-pool2.svc.pmtpa.wmnet is CRITICAL: Connection timed out [13:40:45] RECOVERY - LVS Lucene on search-pool2.svc.pmtpa.wmnet is OK: TCP OK - 0.003 second response time on port 8123 [13:43:27] RECOVERY - LVS Lucene on search-pool3.svc.pmtpa.wmnet is OK: TCP OK - 0.006 second response time on port 8123 [13:52:18] RECOVERY - Lucene on search15 is OK: TCP OK - 8.998 second response time on port 8123 [14:00:32] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [14:12:50] PROBLEM - Puppet freshness on bast1001 is CRITICAL: Puppet has not run in the last 10 hours [14:45:50] PROBLEM - Puppet freshness on search1001 is CRITICAL: Puppet has not run in the last 10 hours [15:10:53] PROBLEM - SSH on lvs6 is CRITICAL: Server answer: [15:14:00] RECOVERY - SSH on lvs6 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:22:43] is there some kind of "something's broken" component? [15:23:05] or just throw it in the component for the feature that's broken? [15:23:14] 34512 [15:25:35] i'll just throw it in the same place as 34381 [15:37:06] PROBLEM - Puppet freshness on fenari is CRITICAL: Puppet has not run in the last 10 hours [16:00:41] The page [[Commons:Deletion_requests/2012/02]] is in Category:Pages where template include size is exceeded (automatically by mediawiki) Is it possible to get a higher limit for all Commons:Deletion_requests subpages? [16:01:09] that is disturbing our work at commons [16:05:49] it was already raised enormously [16:07:50] then.. let's raise it more ;) [16:07:56] hm [16:08:03] servers are there to work, aren't they? [16:08:06] ifexist doesn't seem a good idea there [16:08:11] dunno, not my choice [16:08:14] :D [16:08:16] sure [16:08:16] Does the search server work? [16:08:41] Nemo_bis: ifexist is there to avoid many red links, I guess [16:08:50] do you think just this is a problem? [16:09:21] as I understand the error it is not this function causing the problem [16:10:19] on it.wiki we use a parserfunction which automatically finds which pages to transclude depending on the date [16:10:29] some templates here are trying to be too smart [16:10:40] (pobably) [16:10:41] btw: the discussion is here: https://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard#Commons:Deletion_requests.2F2012.2F02_is_in_Category:Pages_where_template_include_size_is_exceeded [16:11:23] the deletion requests page is anything but smart - just a page with subpages included which themselves do include subpages [16:12:25] the page is already 1 MB HTML... [16:13:11] and? [16:13:22] even my stoneage pc can handle that [16:14:16] I mean: would a exception for the del.req pages put so much more load on the servers? I can't imagine [16:14:22] surely not [16:14:57] if there would be thousands of such big pages - yes - but it is just about 3 [16:19:06] PROBLEM - Puppet freshness on brewster is CRITICAL: Puppet has not run in the last 10 hours [16:21:25] it's surely impossible to configure it that way [16:21:48] Just split the page, there's no reason to have it together. My personal suggestion. [16:22:37] But the limit is something like 4 MB, you probably have some stupid unneeded overhead. [16:46:24] RECOVERY - Lucene on search15 is OK: TCP OK - 0.010 second response time on port 8123 [16:48:59] Nemo_bis: the reason to have it together is that it is that way ;) E.g. that is great for in-browser-search [16:49:12] we have prefix: now [16:49:40] that is not as fast an convenient as directly in browser [16:49:59] try to jump to the next hit using f3 ;) [16:50:22] I will just file a bug - sunday is a bad day for tech channel ;) [16:54:31] https://bugzilla.wikimedia.org/show_bug.cgi?id=34519 [16:54:39] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [17:11:15] RECOVERY - Lucene on search15 is OK: TCP OK - 0.019 second response time on port 8123 [17:20:51] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [17:46:12] RECOVERY - Lucene on search15 is OK: TCP OK - 0.002 second response time on port 8123 [17:54:45] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [18:03:45] RECOVERY - Lucene on search15 is OK: TCP OK - 0.000 second response time on port 8123 [18:12:00] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [18:30:12] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:31:24] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [18:34:06] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [19:42:52] RECOVERY - Lucene on search15 is OK: TCP OK - 8.994 second response time on port 8123 [19:51:07] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [19:52:56] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2578 [19:54:05] New review: Hashar; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2611 [20:15:07] RECOVERY - Lucene on search15 is OK: TCP OK - 0.003 second response time on port 8123 [20:16:41] Am I allowed to make this room a general tech help room? [20:16:50] Like the IRC version of https://meta.wikimedia.org/wiki/Tech ? [20:16:57] I think I'm going to. [20:18:53] Er, my understanding was that it was more for server admin stuff. [20:18:57] But I'm new. [20:19:01] So what do I know. [20:19:15] Well, it was at one point. [20:19:19] I think that phase is over. [20:19:23] heh. [20:19:32] It's now just those damn bots. [20:19:43] Yeah, I find that rather annoying as well. [20:19:47] huh, what? [20:19:48] Most of the oeprations stuff moved to -operations. [20:20:00] Operations, even. [20:20:12] usually here is comparatively low level of general tech help [20:20:14] (I'd probably find it less annoying if I got off my ass and reconfigured irssi. But whatever.) [20:20:43] Saibo: Right. I'm talking about changing that. [20:20:52] ah [20:21:07] About making this channel more of a general Wikimedia wiki tech help channel. [20:21:37] It'd probably be a bit of relaying to #mediawiki too. [20:21:54] then the server nag bots should be only in #operations [20:22:06] and the topic be changed ;) [20:22:16] Yes. :-) [20:22:21] But I want to make sure nobody freaks out first. [20:22:33] There's already #wikimedia-dev [20:22:40] I don't really mind.. just some help texts need to be rewritten.. [20:22:41] There is. [20:22:48] so therefore it is not good [20:22:49] I lean in the other direction. [20:23:08] The dev channel isn't really for general tech help. [20:23:10] -ops for triage, -tech as the watercooler for all of the Tech people [20:23:22] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [20:23:24] -ops is a whole different channel, sadly. [20:23:25] Which would mean moving some of the bots out of here :) [20:23:28] yes. [20:23:31] You mean -opeartions. [20:23:33] Oh. [20:23:36] I figure this is where people come when stuff is broken (whatever it is) [20:23:36] I can't type. :-( [20:23:37] Heh. I was being lazy. Yes. [20:23:50] apergos: Which is kind of like a technical village pump. [20:23:57] But I'd envision a bit more than "it's not working." [20:23:59] I guess it depends on what you imagine as "tech help". [20:24:02] in operations we try to do work so (when things are broken) it's hard to have conversation with other folks in there [20:24:12] Like, "Open Office doesn't work!" == bad [20:24:12] when it's urgent that is [20:24:42] well it's not working can be "there's this problem with the js on project X" [20:24:55] for discussions in here, I mean [20:24:58] Right. [20:25:04] it covers a wide range. [20:25:04] That's what I'd like to see a bit more of. [20:25:06] Instead of nagios. [20:25:06] sure [20:25:08] Right, which totally makes sense. [20:25:10] Much as I love nagios. [20:25:16] I think nagios should stay in here [20:25:18] I love Nagios... in -operations. [20:25:26] I would love to get rid of it in there but mark disagrees [20:25:30] I'd like to advertise [[m:tech]] a bit more and push people here for real-time interaction. [20:25:40] the reason I would love for it to be gone in there is that when something goes all to hell [20:25:48] it's really hard to do work in there cause of all the messages [20:25:56] I mean, I'm in favor of creating a #wikimedia-ops-status or something for the firehose of bots [20:26:01] anyways, since it's an information source I think it should remain here [20:26:07] I don't want to have yet another channel [20:26:07] apergos: More incentive to not have things go wrong. ;D [20:26:08] Yeah, it seriously interrupts conversation. [20:26:20] Joan: right. because we don't have any incentives now :-P [20:26:30] And I ignore channels with the bots most the time because I assume the updates are usually the bots. [20:26:38] Bots == People aren't reading [20:27:05] guess we would need to take a poll [20:27:14] but I prefer watching them in here along with the general conversation [20:27:38] I think that whatever happens, people would be happiest if there was a logic to the rooms and distinctions. [20:27:56] I joined last week. It was really hard figuring these things out. [20:27:56] ha [20:27:57] This channel has general conversation? [20:28:00] that will never happen [20:28:02] it used to be... [20:28:05] wikitech. [20:28:07] So let's make it happen :) [20:28:08] mediawiki. [20:28:09] That's my point. [20:28:15] We need a chart! [20:28:19] and that was really enough. I liked it [20:28:25] There is a chart somewhere. [20:28:28] guillom did one. [20:28:29] Diversity in rooms is fine. [20:28:36] We just need to be better at communicating it. [20:28:37] now there's enough channels I can't autojoin them all. so I miss stuff. [20:28:40] And what the options are. [20:28:47] You... can't? [20:29:26] dschoon: Oh, you're the new hire. [20:29:40] Are you the dinosaur one? [20:29:44] I am. [20:29:46] zak did a chart iirc, its in his userspace [20:29:53] The one with the silly hats and the mohawk, too. [20:30:02] Yeah, I stalked you on LinkedIn. [20:30:08] no, I can't. freenode will flood out after the tenth autojoin cause the pidgin developers (my irc client) refuse to put in an option for staggering autojoins [20:30:24] mohawk? excellent [20:30:26] Oh, I guess I ought to do something about LinkedIn. I kind of hate that site, so I never update it. [20:30:35] Everyone hates it. [20:30:47] I'm davidschoonover.com if you want to stalk 4 realz. [20:30:47] They still don't allow arbitrary order of things. [20:30:50] Or dateless things. [20:30:56] I don't understand the rigidity. [20:31:24] apergos: pidgin? not to start client wars... but i even recommend mirc over that (on the win side) =( [20:32:10] Pidgin for IRC seems like it'd be weird. I use it for IM at work. [20:32:26] dschoon: I recently discovered "T-Rex Trying..." [20:32:26] I'm on linux [20:32:44] I want a client that supports multiple protocols, which pidgin does, for all its flaws [20:32:52] gtalk, irc, icq, etc etc [20:32:58] there's even a skype chat plugin [20:33:07] Joan: this happened on friday :) http://instagr.am/p/HIUcp2Qu4L/ [20:33:26] Haha [20:33:28] apergos: on osx, Adium does all that. i don't use it for IRC, though, as I want a separate app to alt-tab to. [20:33:42] yeah, no adium on linux [20:33:44] Joan: dinosaursloveyou.com [20:33:50] it's the same underlying library, libpurple [20:33:56] yeah. <3 libpurple [20:34:46] ah the dinosaur! [20:34:48] :-) [20:35:43] I'm not sure why watching dinosaurs do human tasks is so amusing. But it is. [20:36:10] T-Rex would have many documented difficulties, though... [20:36:40] He does! [20:36:50] I have documented several of them. [20:38:30] However: http://art.less.ly/dinosaurs/061113-dinos-do-art.png [20:38:53] hah. [20:40:32] :-D [20:49:13] Hey, I have two more cases like the one in this bug https://bugzilla.wikimedia.org/show_bug.cgi?id=33775 about edits leftovers after user renames [20:49:26] should I reopen that and include them or open separate tickets for each case? [20:49:39] can you open bug reports for them and assign them to me please? "ariel @ ..." [20:49:51] I prefer to have them as separate bugs [20:49:56] okay [20:50:00] sweet [20:57:39] done, and thank you! [20:58:05] thank you! [21:10:47] RECOVERY - Lucene on search15 is OK: TCP OK - 2.995 second response time on port 8123 [21:17:05] PROBLEM - Puppet freshness on search1002 is CRITICAL: Puppet has not run in the last 10 hours [21:19:11] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [21:59:05] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [22:02:05] PROBLEM - Puppet freshness on search1003 is CRITICAL: Puppet has not run in the last 10 hours [22:05:05] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [22:05:06] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [22:16:11] RECOVERY - Lucene on search15 is OK: TCP OK - 0.004 second response time on port 8123 [22:29:08] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [23:13:41] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [23:14:26] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.82 ms [23:17:26] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [23:25:23] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.046 second response time [23:31:59] RECOVERY - Lucene on search15 is OK: TCP OK - 2.997 second response time on port 8123 [23:40:23] PROBLEM - Lucene on search15 is CRITICAL: Connection timed out [23:51:05] https://meta.wikimedia.org/w/index.php?title=Special%3ACentralAuth&target=federico :-O