[00:30:32] !log tstarling synchronized php-1.21wmf4/extensions/WikimediaMaintenance/fixBug41778.php [00:30:40] Logged the message, Master [00:31:19] !log tstarling synchronized php-1.21wmf3/extensions/WikimediaMaintenance/fixBug41778.php [00:31:25] Logged the message, Master [00:49:52] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [01:39:59] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 260 seconds [01:40:16] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 279 seconds [01:43:16] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:46:31] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [01:59:43] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 233 seconds [02:00:37] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 289 seconds [02:15:01] PROBLEM - Squid on brewster is CRITICAL: Connection refused [02:23:19] !log LocalisationUpdate completed (1.21wmf4) at Mon Nov 19 02:23:19 UTC 2012 [02:23:29] Logged the message, Master [02:24:37] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [02:31:41] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [02:43:07] !log LocalisationUpdate completed (1.21wmf3) at Mon Nov 19 02:43:07 UTC 2012 [02:43:14] Logged the message, Master [03:01:40] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [03:48:37] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [03:50:43] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 10 seconds [04:30:54] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [04:30:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [04:30:54] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [04:30:54] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [04:54:09] RECOVERY - Lucene on search1016 is OK: TCP OK - 0.027 second response time on port 8123 [05:49:04] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [06:36:58] PROBLEM - Lucene on search14 is CRITICAL: Connection timed out [06:37:07] RECOVERY - Squid on brewster is OK: TCP OK - 0.001 second response time on port 8080 [06:39:58] RECOVERY - Lucene on search14 is OK: TCP OK - 0.018 second response time on port 8123 [06:51:47] PROBLEM - Lucene on search14 is CRITICAL: Connection timed out [06:52:32] apergos: now search14 ? [06:52:40] it incremented ;P [06:53:53] just got on line [06:56:14] * apergos waits for nagios to wise up [06:56:26] RECOVERY - Lucene on search14 is OK: TCP OK - 0.008 second response time on port 8123 [06:56:31] thanks [06:56:43] !log restarted lucene search on search14 [06:56:50] Logged the message, Master [08:36:07] !log running sync-common on mw46 [08:36:14] Logged the message, Master [08:37:07] PROBLEM - Puppet freshness on dobson is CRITICAL: Puppet has not run in the last 10 hours [08:44:10] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [09:31:20] link fix [10:34:26] DanielK_WMDE: see topic :-] [10:34:33] ah, cool [10:34:50] ...or not ;) [10:34:56] so, who *is* on duty? [10:37:19] DanielK_WMDE: Faidon has nickname paravoid [10:37:35] he is in Greece :-] as well as apergos [10:38:32] on duty = working? me, eventually mark I suppose, and eventually paravoid (based on order when we usually show up active in the channel) [10:38:44] Hi all! [10:39:16] DanielK_WMDE: https://labsconsole.wikimedia.org/wiki/Help:Self-hosted_puppetmaster [10:39:47] DanielK_WMDE: once an instance is build simply check the box: [ ] puppetmaster::self [10:39:49] then run puppet [10:39:50] ;-) [10:47:24] Recently, I worked on puppet recipes for Wikidata on labs. I submitted some stuff for review which installs either a Wikidata client or a repo via puppet. Some people from the labs channel commented on it, but I'm somehow stuck: I am not a coder so I did not realize I have to ask for a style guide before starting. Now I got a lot of contradictory comments from the "tabs vs. spaces universe", but nobody has actually given me feedback [10:47:24] in the form of "grades" so merging would get any closer. Do you have any ideas how to get on with this? [10:50:41] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [10:50:41] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [10:51:32] Silke_WMDE: you're unlikely to get anywhere if you don't give links... [10:52:18] jeremyb :) Sorry! https://gerrit.wikimedia.org/r/#/c/30593/ [10:52:48] cat to vet :-( [12:25:25] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [12:26:54] Silke_WMDE_: indeed, the git tutorial and siblings should contain a visible warning: the likelihood of your commit to be actually reviewed is inversely proportional to the number of whitespace changes and to the square of the number of trailing whitespaces you added [12:32:28] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: Puppet has not run in the last 10 hours [12:35:36] Silke_WMDE_: hello :-] I talked with Daniel Kinzler this morning about your puppet change [12:35:44] Silke_WMDE_: definitely ignore anything regarding whitespaces :-] [12:52:17] DanielK_WMDE: our PHPUnit code is really messy :-] [12:52:35] you don't say ;) [12:53:35] so the new setupTestDB and teardownTestDB you wrote have been added to MediaWikiTestCase which is the parent class for any of our test class [12:53:57] made static so they could be run from the MediaWikiPHPUnitCommand [12:54:10] I guess we should move them from MediaWikiTestCase to MediaWikiPHPUnitCommand [12:54:29] seems they are to be run before and after the whole test suite [12:56:20] DanielK_WMDE: I will move your stuff to the MediaWikiPHPUnitCommand [12:56:26] and refactor some stuff while I am at it [12:57:42] hashar: that means that MediaWikiTestCase has to know MediaWikiPHPUnitCommand. [12:58:10] setupTestDB is not done before all tests are run, it's done before the first test runs that needs the database (kind of lazy initialization) [12:58:18] ahh [12:58:23] so it is generally triggered by MediaWikiTestCase [12:58:53] it's not really a problem, except that I dislike circular dependencies like this. [12:59:21] (or... does MediaWikiPHPUnitCommand know about MediaWikiTestCase?) [13:00:33] my idea was to get the testDB logic in the PHPUnitCommand class [13:00:43] and make MediaWikiTest case to execute it whenever needed [13:00:49] that is simply moving static classes aroun [13:00:50] d [13:00:53] or maybe another class [13:02:06] naw, put it where you like it, I don't care :) [13:02:59] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [13:03:34] lets merge all the mess first :-] [13:12:21] DanielK_WMDE: merged both [13:15:49] re [13:16:23] hashar: OK. :) SO whom can ask to review/give feedback/merge this? [13:17:00] Silke_WMDE_: andrewbogott did have a look at it, I guess you can follow up [13:17:09] err .. HE can follow up [13:17:09] yes [13:17:10] I think [13:17:40] Silke_WMDE_: I also told Daniel I was going to have a look at it :-] [13:17:43] doing that right now [13:17:53] great, thanks! [13:18:12] Silke_WMDE_: I will comment in the inline diff [13:18:18] ok [13:18:38] hashar: thanks! [13:18:55] oh my [13:19:35] if only we had something like: mediawiki::extension { name => "Diff", dest => "/some/path" } [13:20:51] Should be possible to create something like that. [13:52:38] New review: Hashar; "I dont think the two templates are actually required:" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/30593 [13:52:50] Silke_WMDE_: did a quick review, untested though [13:52:57] the apache conf could use some cleanup [13:53:11] and I am not sure why you add two template files which do not seems to be used anywhere [14:03:13] hashar: True, I think are even three unneeded files which were still included in the first submission. When I did the patchsets I ... erm ... just didn't know how to throw them out. [14:03:24] thanks for the comments! [14:03:28] ahh you will need some git magic [14:03:43] get the patch then ask git to remove the files with: git rm [14:04:06] "git status" will show you the file are staged for deletion [14:04:18] then amend your commit, that will add the file deletion to the commit [14:04:20] git commit --amend [14:04:33] you might want to try out in a sandbox first ;-] [14:05:10] cd /tmp ; mkdir gittest ; cd gittest; git init; touch FOO; git commit -a -m "adding FOO" file; [14:05:26] then git show to see the commit [14:05:33] then delete the file with git rm FOO [14:05:38] git status <-- show what is staged [14:05:43] git commit --amend [14:05:49] and finally git show to see the new commit :-] [14:06:04] or poke Jeroen :-] [14:06:12] git might be a bit confusing [14:13:40] apergos: how's ms-be7? [14:15:45] paravoid: status is the same. we left it on friday w/the installer not writing to /sdm/n [14:17:05] apergos: did a work around where we could install the OS on /sdm and /sdn but I don't think it is something we would want to do on all of them [14:17:30] apergos believes are issues are bug related https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1012629 [14:20:08] I told apergos on Friday to try with a different bootdev and see what happens [14:20:27] and ping me if that didn't work [14:23:44] parvoid: i added this to the partman recipe d-i grub-installer/bootdev string /dev/sdm /dev/sdn [14:23:55] !log putting ms-fe1 back into the pool [14:24:02] Logged the message, Master [14:24:05] but the installer sill insisted on writing to /sda [14:32:13] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [14:32:13] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [14:32:13] PROBLEM - Puppet freshness on ms-fe1 is CRITICAL: Puppet has not run in the last 10 hours [14:32:13] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [14:32:55] why is ms-be7 different from ms-be6? [14:35:44] mark: they're not different. when the initial OS was loaded on be6 it was installed on /dev/sda and b. We had to fix it to load on sdm and sdn which it then did w/out a problem. [14:36:16] however, /sda/sdb did not get written over ...apergos had to go in and manualy add them to the file system. [14:38:21] ms-be7 works right now but we had to go in to fdisk and create a single primary partition first..than installer wrote to /sdm and n [14:38:35] i see [14:40:53] we can get it back to original state by dd'ing /dev/sda [14:48:06] !log reedy synchronized php-1.21wmf4/cache/interwiki.cdb 'Updating interwiki cache' [14:48:13] Logged the message, Master [14:48:47] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache' [14:48:54] Logged the message, Master [14:51:49] mark: isn't brewster a SPOF for all apt updates/installs? [14:52:08] yes [14:52:12] how is that "critical" [14:52:48] if that's critical, then everything is critical [14:53:24] anyway, I believe iron is brewster's counterpart in eqiad [14:53:43] so it would be far more productive to make sure that's a brewster replica than to migrate brewster itself [14:55:31] indeed, this makes sense [14:56:09] sorry, carbon [14:56:10] not iron [14:56:48] but yeah, what I meant with critical was "spof, affects day-to-day ops" [14:57:30] every service affects day to day stuff of some dept or person ;) [14:57:41] sure, it's not unimportant [14:58:00] but putting that server on one with warranty is not really gonna help anything either is it [14:58:27] if it's gonna be down for a few days we'll have it migrated to another box by the time the new parts arrive [14:58:35] it would be good to ensure good backups of our apt repo though [14:58:40] hehe I guess so [14:58:42] I think we had them but not sure how well it's still working [14:59:19] don't get me wrong, builting a counterpart for redundancy is a much better idea in my opinion too [15:02:35] a large part of brewster is in puppet [15:03:11] meh [15:03:16] is that batch out of warranty too [15:03:22] I still think of brewster as a newish server [15:03:28] i've been here too long ;) [15:06:52] how interesting [15:07:12] almost all of the FIN_WAITs on ms-fe.svc are with imagescalers [15:07:55] I think because squids etc have shorter fin_timeouts [15:08:01] while most of the ESTABLISHED ones are with cp [15:10:36] well, you're right that we have fin_timeout set to 3 on squid/varnish/swift [15:10:57] but otoh, why would it timeout on internal traffic? [15:11:05] it's not like we have broken tcp stacks on our machines [15:12:12] so, neither of ms-fe or srvNNN have FIN_WAIT on their netstat [15:12:16] but LVS thinks so [15:16:33] I've put ms-fe1 back into the pool for almost an hour now and it still gets 1/10 of the traffic than the rest [15:16:42] 18 connections vs. 180 for each of the other three [15:18:00] New patchset: Silke Meyer; "Added puppet files for Wikidata on labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30593 [15:20:18] New review: Silke Meyer; "Removed the three obsolete template files (also the apache template), using the template from the ap..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/30593 [15:26:03] New review: Silke Meyer; "I forgot: the "keep_up_to_date" naming is Andrew's, he has used it in his mediawiki recipe, so I can..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/30593 [15:35:13] New patchset: Hashar; "Gerrit notifications for Wikidata to their channel" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26042 [15:35:13] New patchset: Hashar; "cleanup/refactor gerrit logging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120 [15:35:13] New patchset: Hashar; "Gerrit hook tests now creates hookconfig.py" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26040 [15:35:14] New patchset: Hashar; "Gerrit hook tests extended coverage" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/26041 [15:48:11] !log reedy synchronized php-1.21wmf4/extensions/OAI [15:48:13] DanielK_WMDE: ^^ [15:48:17] Logged the message, Master [15:50:12] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [15:58:02] Reedy: our hero :-) [15:58:03] !log powercycling ms-be3, unresponsive [15:58:08] off to get my daughter [15:58:09] Logged the message, Master [16:01:59] mark: nope, it's not fin_timeout [16:02:36] however I do see one possible reason: there are some long-lived cp/sq connections, they're probably using keepalives [16:02:45] and pipelining [16:03:04] while imagescalers do not/can't [16:03:14] so that part makes sense [16:03:33] RECOVERY - Host ms-be3 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [16:04:15] yay [16:04:26] amazing, it came back up :) [16:11:18] New patchset: Mark Bergsma; "Create new Ganglia cluster(s), and add ms-be300x to it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34093 [16:17:39] hm [16:17:41] come to think about it [16:17:52] this might be the entire reason behind the traffic being unbalanced [16:18:31] persistent connections from varnish/squid [16:19:08] new realservers won't get much traffic, since most of the traffic passes through the existing connections [16:19:43] am I blind or just losing it [16:19:56] https://gerrit.wikimedia.org/r/#/c/34093/1/manifests/ganglia.pp [16:19:59] what's wrong here [16:20:10] i don't see it [16:20:41] doh [16:20:42] I do see it [16:21:08] New patchset: Mark Bergsma; "Create new Ganglia cluster(s), and add ms-be300x to it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34093 [16:21:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34093 [16:21:45] missing $cluster? [16:22:09] oh no, missing } [16:22:28] mark: see what I said above? [16:22:43] yes [16:22:43] about persistent http from backend proxies? [16:22:51] they are using persistent http yes [16:23:17] yeah, so I think that's why ms-fe1 gets less than 3% of the traffic despite being pooled back for 2 hours now [16:23:28] right [16:23:34] might want to set a max on the nr of requests ;) [16:23:51] we can do balancing inside varnish as well of course [16:24:08] mmhmm [16:24:54] pybal is nice though :-) [16:27:06] so, varnish has no way to close backend connections after a number of requests [16:27:11] and neither does swift it seems [16:27:13] heh [16:28:17] hmm [16:28:19] I think you could in vcl [16:28:22] but it's hacky [16:29:29] let's not [16:29:40] amazing, bitten by optimizations [16:30:31] we had persistent connections disabled until half a year ago [16:30:35] when I enabled it again on apaches [16:30:45] knowing that it would reduce equal load balancing [16:30:55] but the distance of eqiad caches to tampa apaches was more important [16:31:35] indeed [16:32:36] it's the same for swift now that we have caches in eqiad [16:32:47] yes [16:35:57] robh: please send license for ms-be7 when you get a chance...ST 2FLGYV1 [16:36:08] apergos: around? [16:36:28] yes [16:36:36] saw the comments above about ms-be7? [16:37:04] want me to take it over? [16:37:08] I saw you asked how things went [16:37:31] cmjohnson1's update is pretty much on the money [16:38:06] checkd the logs and saw the bootdev did seem to get set to sdm, n by the installer [16:38:07] and yet [16:38:18] it tried grub to sda and failed [16:38:37] I tested your theory about the partition table, that indeed let it complete [16:39:26] but the first puppet run would I guess not complete properly, as the filesystem wouldn't getmade for that partition [16:39:49] so we'd have manual intervention a couple of times, not so exciting [16:40:04] did you want to look at it some to see if you can figue out something better? [16:41:04] cmjohnson1: silly question: are the SSDs are connected to the RAID controller? is it possible to connect them to the onboard controller like we had it in the C2100s? [16:41:11] also, I don't know what state it's in tirhgt now; this weekend, maybe Saturday, I saw it was bouncing according to nagios, so I checked and it was at some other random step in the installer, failed. I left it in a shell from the installer [16:42:04] but I'm not sure where cmjohnson1 left it, except that it had gotten through the grub step (it was quite late at that point, I crawled off to sleep) [16:42:45] paravoid: they are connected to the raid controller and no, we are not able to connect them directly to the onboard controller. [16:43:00] worth a shot :) thanks [16:43:02] the c2100's were more flexible than the 720's [16:43:15] apergos: i left it in the shell [16:43:59] hmm [16:44:15] who knows what happened, grrr [16:44:18] I tend to prefer inflexible 720s though :) [16:44:30] yeah, this is annoying but I much prefer these boxes [16:47:56] New patchset: Mark Bergsma; "Set ms-be3001/2 as ganglia aggregators" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34097 [16:47:58] paravoid apergos: the additional 10 720's are supposed to be delivered today. I am going to get ms-be8 and 10 up and once we figure out the install they will be ready [16:48:10] ok that's great [16:48:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34097 [16:48:16] the install should be fine for systems without SSDs [16:48:25] also, I'd like to reserve a few of them for my ceph testing [16:48:26] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [16:48:56] just let me know which ones are to be swift with ssds, which ones are to be swift without ssds [16:49:00] not too many though, we'll have enough to be replacing boxes for a monthr or two [16:49:05] and I can leave the rest or do a base install as you prefer [16:49:28] we used to have 4 boxes with SSDs in the C2100 cluster iirc [16:49:32] I think that's enough for now [16:49:53] only 4? huh [16:50:28] paravoid: so you don't want ssd's in all of them? [16:50:35] no [16:50:59] that was the case with C2100s too [16:51:39] 5-12 had them it says here (according to the netboot config) [16:51:48] right [16:51:59] they were not in them but i recall Ben wanted ssds in all of them...let me know know which ones you want to leave out? [16:52:29] right but i had an old ticket for 1-4...the c2100's were nixed before we ever got that point [16:53:14] I think it's a waste to have it in all of them, although I wasn't around when they did that performance testing [16:53:24] maybe mark knows? [16:54:00] it was only for the container objects I think? [16:54:05] yes [16:54:10] but ben generally liked consistency, so probably for that reason alone [16:54:22] containers and accounts, although we have a single account so... [16:56:51] cmjohnson1: trying to find the damned link for the drac software [16:57:44] found it, pulling the key [16:57:44] ok....still have like 25 days before it expires [16:58:07] ok..keep it handy will have a few more for you today [16:58:10] no problem, ill have dell email it to me and i will forward you the email [16:58:27] or you can drop an RT ticket for each one [16:58:30] and I can attach the key [16:58:37] that may be easier for you to keep track of [16:58:47] since the key doesnt really have info saying its for such and such [16:58:55] ok...let's do that so I can refer back to it [17:01:15] their site for this stuff is really crappy [17:01:26] (in any browser ive tested) [17:02:34] constant error popups and timeouts from whatever crap server side stuff they are running [17:02:39] paravoid: so do you want to go w/ the current set up on which ones have and do not have ssd's? be1-4 do not 6-12 do...let me know or add to ticket 3829 [17:02:49] yeah that's fine [17:03:04] ok [17:03:33] 5-12 [17:03:34] do you know the ETA for the eqiad cluster? [17:04:11] New review: Andrew Bogott; "Regarding tabs vs. spaces... I really would like you to change these files to follow the existing st..." [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/30593 [17:04:12] projected time is first week of December [17:04:27] nothing certain yet...still in production [17:05:29] sigh [17:07:06] https://bugzilla.wikimedia.org/show_bug.cgi?id=20409 - would NE be needed on all rewrites that might deal with special chars? [17:07:19] Krenair: I think so. [17:07:48] And it's apache-config redirects.conf? [17:08:15] I think so :) [17:08:46] cmjohnson1: https://rt.wikimedia.org/Ticket/Display.html?id=3939 [17:08:54] thats ms-be7 key file [17:08:59] cool thx [17:09:04] just drop one ticket per system in the pmtpa queue and assign to me [17:09:15] i'll assign and attach key and assign abck to you [17:09:31] okay [17:09:46] ah so paravoid I see you asked if I want you to take over. I am out of ideas for sure [17:09:52] !log reedy synchronized php-1.21wmf4/cache/interwiki.cdb [17:09:58] Logged the message, Master [17:10:00] but I"m happy to follow along on the backread during the times I'm not here [17:10:07] or follow along live when I am [17:10:18] I notice lots of these redirect to specifically HTTP versions, without considering HTTPS... [17:10:22] wow Silke's client sensed Andrew's comment and quit before it [17:10:58] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb [17:11:04] Logged the message, Master [17:11:26] Reedy: any idea on where to redirect https://secure.wikimedia.org/w/extensions/skins/Donate/images/banners/Banner_88x31_0000_A.jpg ? [17:12:19] <^demon> paravoid: Same path, but just use the host we're already on. [17:12:28] er sorry? [17:12:30] <^demon> eg: https://en.wikipedia.org/w/extensions/skins/Donate/images/banners/Banner_88x31_0000_A.jpg [17:12:53] <^demon> https://fr.wikiquote.org/w/extensions/skins/Donate/images/banners/Banner_88x31_0000_A.jpg [17:12:55] <^demon> Etc. [17:12:57] what do you mean "we're already on"? [17:12:59] Depending on how they're actually including it, fundraising should possibly fix their code [17:13:05] there's no host here [17:13:08] that's the URL as you see it. [17:13:21] New patchset: Reedy; "Kill static-master" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34098 [17:13:23] ugh they're not actually including from secure.wm.o are they? [17:13:29] paravoid: Point it at meta [17:13:38] no, that's from a random blog that has sourced it [17:13:50] but I'm wondering about the general case here, not just this blog. [17:13:53] or URL [17:13:54] <^demon> Meta's prolly fine. [17:14:04] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34098 [17:14:04] redirect what? /w/extensions/ ? [17:14:07] Yeah [17:14:07] or /w/ ? [17:14:23] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.001 second response time on port 11000 [17:14:24] <^demon> I'd say /w/ is probably fine. [17:14:26] <^demon> Reedy? [17:14:29] Ditto [17:14:39] thanks :-) [17:15:42] There shouldn't be that much traffic ging there [17:16:05] no [17:16:22] but I'd like to drop the proxying eventually [17:16:30] since it bypasses caches and everything [17:16:32] <^demon> And if there's any traffic from us going there, we should fix it. [17:16:36] ^ [17:16:39] ideally, I'd like to move it to the SSL cluster, rather than singer [17:16:41] Have we got reasonable access logging? [17:16:53] no [17:16:59] i.e. we log all secure.wm.org hits [17:17:07] which isn't reasonable, but it's useful in this case :-) [17:18:45] New patchset: Faidon; "secure.wikimedia.org: add redirect for /w/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34100 [17:19:35] New patchset: Faidon; "secure.wikimedia.org: add redirect for /w/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34100 [17:21:29] New patchset: Faidon; "secure.wikimedia.org: make the redirects permanent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34101 [17:21:34] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34100 [17:21:42] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/34101 [17:26:10] Nov 19 17:22:36 208.80.152.162 apache2: PHP Warning: Directive 'magic_quotes_gpc' is deprecated in PHP 5.3 and greater <-- looks like that server is out of sync with puppet [17:40:51] Is faidon on IRC? [17:44:42] http://pastebin.com/Gq46Bqjw - untested so far, basically just added NE to those which look like they would deal with special chars [17:45:03] paravoid, something like that? [17:50:32] RECOVERY - mysqld processes on es1 is OK: PROCS OK: 1 process with command name mysqld [17:54:13] New patchset: awjrichards; "Update technical feedback email address for mobile site contact page" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34105 [17:55:38] New patchset: awjrichards; "Update technical feedback email address for mobile site contact page" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34105 [17:57:09] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/34105 [17:57:40] eww, I shouldn't have hurried with it [17:58:38] Reedy, it's your time - should I revert and resubmit ^^^ after it or you don't need to change settings so it won't disrupt you? [17:58:52] Go ahead if you want [17:59:03] I've only got to change 1 line in wikiversions.dat for this deploy [17:59:29] you mean go ahead and push it? [18:00:20] yeah [18:06:34] oh, who is the ops on duty today ? [18:06:43] this week [18:08:35] LeslieCarr, topic says it's you:) [18:08:39] hehe [18:08:42] that's a lie [18:12:32] LeslieCarr: I was waiting until the ops meeting :) [18:12:37] Krenair: I'm Faidon. [18:12:37] oh [18:12:37] hehe [18:13:24] ... ah. hi :) [18:14:08] hi :) [18:14:41]