[05:33:00] paravoid: more squid3 https://upload.wikimedia.org/wikipedia/commons/7/73/Aerodrom_Portoroz.jpg [05:39:12] paravoid: filed as https://bugzilla.wikimedia.org/show_bug.cgi?id=41449 [06:07:03] thedj: https://bugzilla.wikimedia.org/show_bug.cgi?id=41449 [06:46:30] Ryan_Lane: do you know something about more squid3 servers? https://bugzilla.wikimedia.org/show_bug.cgi?id=41449 [07:09:18] that still isn't fixed ? [07:09:26] paravoid: ping [07:12:11] odds are slim, it's early morning on a saturday [07:12:33] iirc the issue was "squid3 installed on a precise host, need to reinstall with lucid" [07:14:07] hi apergos [07:14:13] hello [07:15:40] possibly, but now they are/it is in the squid pool [07:15:49] which i believe isn't the intent [07:41:25] no, squid was supposed to have been stopped over there but I bet puppet restarted it [07:42:52] no, I think I'm wrong [07:43:08] there is a frontend process and a backend process on each squid, [07:43:24] the frontend process had been stopped [07:44:14] thedj where are you located? [07:49:56] apergos: Europe like us I think [07:50:01] eh [07:50:16] not useful for me [07:51:49] the user without problems at https://commons.wikimedia.org/wiki/Commons:VP#Inaccessible_image is in USA it seems [07:54:30] well that confirms what I think [07:54:42] but it's aking me forever to figure out how to fix it [08:00:18] hey [08:00:22] just fixed it. [08:00:47] /etc/init.d/squid3 stop on sq82 [08:00:53] ok I thought that [08:00:59] nm [08:01:00] I give up [08:01:57] depooling it is a better solution on the long-ter though [08:02:43] but you can't do it with pybal [08:02:51] well, /just/ with pybal [08:03:00] because that's only the front end [08:03:03] * apergos stabs [08:03:04] exactly [08:03:12] really I hate how complex our setup is [08:03:13] you need to disable it in the squid configs [08:03:17] it's not that complex [08:03:23] that part anyway :P [08:03:40] it is that cmplex [08:03:51] because that url is till broken for europe [08:03:58] and it's still broken cause amssq50 (I think) has it [08:04:19] I don't understand why it thinks it has anything, but whatever [08:04:37] no it's not [08:04:48] your browser cached it probably [08:04:59] I had given a shift r to bypass that [08:05:08] other frontend caches wouldn't cache a 400 for more than 5' [08:05:11] but of course a second check shows it working [08:05:28] seriously I would have spent another hour on this and never found it. I really do give up [08:06:12] (having already spent an hour) [08:06:37] I think it'd be fast to reprovision that server with lucid [08:06:40] rather than trying to find a way to depool it :P [08:06:51] and in fact I'll do that just now, unless you want to [08:07:04] no, I don't want to, I'll likely screw it up [08:07:23] I don't think you will [08:09:31] paravoid: what's your username/userpage? [08:10:00] why do you ask? [08:10:42] paravoid: because I can't find it [08:10:50] and e.g. it wasn't immediate to cc you on bugzilla [08:11:02] as I didn't remember your real name at first, only IRC name :) [08:11:40] I don't have a userpage with my realname on it [08:12:24] and thanks for reporting it/trying to get ahold of us [08:16:59] hm [08:19:07] I do have my name on /whois around here [08:19:13] and my client is always connected [08:19:15] :-) [08:19:50] also WMF people have my phone number [08:20:41] you should blame apergos' kindness [08:20:48] for not wanting to wake me up on a Saturday morning [08:20:51] for not calling you? [08:20:56] yeah I wasn't going to do that [08:21:03] hehehe :) [08:22:09] yeah, we're all amazed by your being up to fix squids :p [08:22:42] another simple thing to do would be adding IRC name to bugzilla name [08:23:03] but yes, the necessary info is already around :) [08:23:15] so why doesn't puppet restart the front end and back end? it seems like it should [08:23:20] (of the squids) [08:23:52] as I look at the squid service stanza [08:25:11] I haven't looked at the manifest but did you take into account that the init script is now named "squid3" instead of just "squid"? [08:25:25] in any case, I think it doesn't make sense to find why, just reinstall the box [08:25:28] I'll do that just now. [08:25:49] be my guest [08:26:06] I just want to understand for the next time, my hope is that someday it won't take me hours to *not* find the problem [08:26:48] so, how's that for a debugging strategy [08:27:01] take the URL, see if it fails [08:27:18] run it with e.g. LWP and get the HTTP headers look at X-Cache etc. [08:27:26] then try running it against pmtpa and eqiad [08:27:32] see if it also works or fails [08:27:33] I looked at the headers [08:27:41] (it might have worked for you but fail for someone in the US for example) [08:27:46] and I ran it from fenari and bast1001 but got success from both of them [08:27:57] including using curl with the right accept encoding [08:28:09] it failed from firefox so I looked at the headers there [08:28:17] and it seemed to indicate a hit from amssq50 [08:28:27] with no mention of sq82 [08:28:41] (you don't have to go through fenari btw, a GET -H "Host: upload.wikimedia.org" http://upload-lb.eqiad.wikimedia.org/ should be sufficient) [08:29:46] amssq50 was reporting "X-Cache: localhost" which is indeed suspect [08:30:00] that was from the misconfigured sq82 [08:30:25] tbh both times around I just put the pieces together and knew what happened before I looked at any headers :P [08:31:07] I saw the new squid error page, remembered how Chris reprovisioned the box, remembered how we switched to a precise stock install, remembered how precise includes squid3 and how that would be a problem for us [08:31:17] and just guessed so and verified by sshing to sq8 [08:31:19] *sq82 [08:31:44] and this time I just ssh'ed to sq82 and ran "ps aux |grep squid3", "uptime", "/etc/init.d/squid3 stop" [08:31:45] so if squid3 isn't in puppet anywhere (having just done a grep -r for the string).. how did it wind up installed and configured over there? [08:32:34] ii squid 3.1.19-1ubuntu3.12. dummy transitional package from squid to squid3 [08:32:39] :-) [08:32:45] oh joy [08:35:37] oh, sigh, we have sq* boxes with varnish [08:35:38] fun [08:35:48] how is that possible? [08:36:30] we have a rule to not ever rename boxes [08:36:39] and we're trying to squid from switch to varnish [08:37:09] and we're not wasteful enough to buy new boxes just because we don't like their names :P [08:37:39] squid from switch is awesome [08:37:52] can't we rename them to cp something? [08:38:20] Nemo_bis: hahahahaha [08:38:29] lol amazing [08:38:36] too early for me I guess :) [08:38:55] apergos: I've been told that we have a rule to *never* rename boxes [08:39:07] maybe that rule needs to be revisted [08:39:15] that was introduced before me [08:39:26] otherwise we wind up with a pile of different naming schemes that go back to prehistory [08:39:33] which is exactly what we have now [08:39:55] a rename would mean a reinstall in my opinion but so what [08:39:56] you mean how we have a swift testing cluster with boxes named owa*? :) [08:40:03] grrrr [08:40:13] :-) [08:40:32] we also have a similar rule of never naming two boxes with the same name [08:40:48] and I did ask (and chris/mark agreed) to not apply it in the ms-be* replacement case [08:41:12] so we don't lose ms-be1 through 12 already [08:41:14] uh huh [08:41:22] right [08:41:26] since they never really were very operational :-D [08:41:44] I'd hate to have a swift cluster with boxes named ms-be13-24 [08:41:59] oh we will someday [08:42:21] we have sq31-86 for example [08:42:26] cause 1-30 are gone. [08:43:13] :-) [08:43:34] so you know where we are on those right? as in, that we don't have any replacements yet, and we're waiting for them to send us one so we can doublecheck that the hardware is right? [08:43:51] (there was chatter in the channel but I don't know if you saw it) [08:45:15] I didn't but I asked CT yesterday [08:45:16] and he told me that [08:45:20] this sucks [08:45:25] yep [08:45:40] oh look at that, chris already added the sq82 stanza [08:45:52] yeah yesterday he was doing that [08:45:56] so, straight reinstall [08:45:57] easy-peasy [08:46:17] wanna do it to gain some confidence? :) [08:46:26] no! [08:46:36] not righ tnow [08:46:42] thank you for asking [08:46:43] I'm fine if you don't want to because it's a saturday [08:47:14] I'm not fine if you don't want to because you think you'll screw up (your words) [08:49:02] this is something where later in the day (if I were around, but I'll be running errands) I would be willing to do it [08:49:53] but not righ tnow whiel I'm already pretty crabby about it [08:50:07] I thought of leaving it for Chris yesterday so he can work out the details, but I'm feeling uncomfortable with leaving it for someone else a second time [08:50:46] well, for not doing it immediately I mean, I didn't mean to imply it was Chris' fault [08:50:53] so I'm just going to do it now [08:50:58] ok [09:17:16] some other squid is not behaving I think [09:17:18] https://bugzilla.wikimedia.org/show_bug.cgi?id=41447 [09:18:53] but not from europe, for sure [09:28:58] Nemo_bis: looking. [09:30:27] wfm from pmtpa/eqiad too [09:30:41] not for me [09:30:45] you got an error? [09:31:26] maybe I tested it wrong [09:32:30] http://upload-lb.pmtpa.wikimedia.org/wikipedia/commons/thumb/9/97/Goering-corpse.jpg/300px-Goering-corpse.jpg [09:34:07] I am getting some other error, I must be testing this wrong somehow [09:34:44] of course you do [09:34:51] this doesn't work, you need to set Host: upload.wikimedia.org [09:34:56] I did [09:35:06] oh, of course I did not set the host header correctly [09:35:16] I set it but it was wrong [09:35:21] not your day, is it? :P [09:35:26] no [09:35:37] this is why I didn't want to do the install without taking a break [09:35:51] ok, now that works for me, 200 [09:36:10] the machine has rebooted but needs a puppet run, wanna do it? :) [09:36:11] I need to get going though (teach me to look at something I tohught would be 5 minutes), I am late already [09:36:14] (sq82) [09:36:24] oh okay [09:36:26] I sohuld have been somewhere at noon, I'm still in my pjs [09:36:27] see you around! [09:36:29] sorry [09:36:33] have a nice sat [09:36:38] I'll be on later in the afternoon or evening [09:36:38] no reason to be sorry for, it's a friggin Saturday [09:36:40] same to you [09:36:47] thanks for bailing me out [09:36:58] don't worry about it [10:42:12] Attempting to create my userpage on wikidata results in "[6d0bf224] 2012-10-27 10:41:49: Fatal exception of type MWException" [10:44:40] QueenOfFrance: Known [10:44:49] hoo: k, thanks! [20:48:03] Meta slow? [20:48:27] Doesn't seem slower than usual from here. [20:49:06] maybe not [22:10:13] hi all, does anyone know why wikitable sortable doesnt work in Chrome? (FF, IE works fine). For example, see http://nl.wikipedia.org/wiki/Lijst_van_bestverkopende_Belgische_muziekartiesten in chrome [22:11:35] works for me [22:12:10] hmmz strange, no add ons running here [22:12:43] im using chrome 22.0.1229.94 m, nothing special [22:13:00] ah never mind [22:13:19] I see some of my local css/js is blocking it, logging out and it was back :p [22:13:33] anyway, thanks for your help