[00:44:31] PROBLEM - Memcached on srv255 is CRITICAL: Connection refused [01:34:49] meh [01:35:03] will the out times stop somewhen? ;) [01:35:35] have repeatedly no css or get a index.php download popup when clicking a link at commons [01:41:37] I'm getting "No data received" occasionally in Google Chrome on https://en.wikipedia.org/. [01:41:40] Hi woosters. [01:44:21] Hmm, also getting edit conflicts with myself. [01:44:26] Anyone else having problems? [01:51:49] PROBLEM - Memcached on srv257 is CRITICAL: Connection refused [01:55:01] Joan: yes, same here, Firefox, Unitymedia (ISP) [01:55:28] since an hours or so [01:55:38] On https or http? [01:55:41] sometimes also pages delivered incompletely [01:55:44] https, yes [01:55:50] don't tried http [01:56:06] on Commons [01:56:10] Ryan_Lane, ^ [01:56:23] thx [01:56:25] hm [01:58:30] this isn't an https issue [01:58:39] https either works or doesn't [01:58:46] it's a transparent proxy [01:59:19] PROBLEM - Memcached on srv254 is CRITICAL: Connection refused [02:00:19] PROBLEM - Memcached on srv256 is CRITICAL: Connection refused [02:05:15] New patchset: Diederik; "Project structure and initial commit." [analytics] (master) - https://gerrit.wikimedia.org/r/2044 [02:06:35] !log LocalisationUpdate completed (1.18) at Mon Jan 23 02:06:35 UTC 2012 [02:06:37] Logged the message, Master [02:11:43] Ryan_Lane: in case that helps: http://pastebin.com/mK9bj8zd reply header of a failed answer (only html, no css), firefox reports "loading of source failed" (in german) [02:13:22] hm. it's weird, but it isn't https [02:14:08] Ryan_Lane: before that request there was a 302 at my POST (made a comment at this talk page). Probably that is normal that there is a 302 then, right? [02:15:49] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1377s [02:17:54] New review: Diederik; "(no comment)" [analytics] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2044 [02:18:45] New review: Diederik; "(no comment)" [analytics] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2044 [02:19:05] New review: Diederik; "(no comment)" [analytics] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2044 [02:19:06] Change merged: Diederik; [analytics] (master) - https://gerrit.wikimedia.org/r/2044 [02:19:19] now it is borked on every request [02:19:22] no css [02:19:25] that's me... [02:19:38] I just broke something [02:19:41] Lost images. [02:19:50] are you using ipv6? [02:19:58] * matthewrbowker doesn't know, 1 sec [02:20:00] Enwiki is working for me. [02:20:22] I don't think so, but ... [02:20:35] no ipv4 only (AFAIK) [02:20:40] pretty sure :D [02:20:54] but: may be of relevance [02:20:58] i am in Germany [02:21:10] OK, I'm getting images again on refresh. [02:21:23] yeah, should be working now [02:21:38] works [02:21:52] !log installed new version of nginx in pmtpa and esams [02:21:53] Logged the message, Master [02:22:00] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1747s [02:22:01] !log restarted nginx in pmtpa and esams [02:22:03] Logged the message, Master [02:22:09] !log installing new version of nginx in eqiad [02:22:10] Logged the message, Master [02:22:45] Sorry, client fail :( [02:23:04] again? [02:23:52] 13:10 Is one of the SSL nodes playing up? Today I've been getting intermittent errors where https://en.wikipedia.org/ bombs out strangely 13:19 twb: same situation here [02:24:59] sorry…. [02:25:00] heh [02:25:14] we're doing some upgrades, and running into some annoying issues [02:25:33] No problem, just making sure you're aware of the symptoms [02:25:42] yep [02:26:06] Thanks, keep up the good work :-) [02:26:51] working now? [02:29:21] seems so [02:30:01] man, I really do not understand gridiron [02:30:12] what a weird game [02:34:29] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 8.78355825688 (gt 8.0) [02:36:09] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:38:39] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 9.62220981651 (gt 8.0) [02:42:09] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:21:21] RECOVERY - Disk space on es1004 is OK: DISK OK [04:24:42] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:36:42] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:57:12] PROBLEM - Packetloss_Average on emery is CRITICAL: CRITICAL: packet_loss_average is 8.30751299065 (gt 8.0) [05:14:11] someone called? [05:15:08] jeremyb, ? [05:15:12] hi [05:15:22] you mailed about the list migration [05:15:26] yup [05:15:34] i followed your google link and was unable to find any problems [05:15:42] can you give more details? [05:15:49] hmm [05:16:44] ah, maybe it is https which isnt working [05:16:51] i use HTTPS Everywhere extension [05:17:03] and it worked before? [05:17:08] * jay|busy fires up Opera [05:17:30] we can just fix https everywhere to point people at the right domain [05:17:50] or we can not care because the domain is deprecated [05:19:12] jay|busy: http://hurl.it/ can be your friend ;) [05:19:36] $ wget https://mail.wikipedia.org/pipermail/wikipedia-l/2006-October/045481.html [05:19:37] --2012-01-23 16:18:31-- https://mail.wikipedia.org/pipermail/wikipedia-l/2006-October/045481.html [05:19:37] Resolving mail.wikipedia.org... 208.80.152.133 [05:19:37] Connecting to mail.wikipedia.org|208.80.152.133|:443... failed: Connection refused. [05:20:07] sure, but 1) can you say for certain that's a regression? and 2) do we care? [05:20:59] meh. I care enough to complain. its up to devs whether they care enough to do something about it [05:21:57] none of the URLs advertised in old archives you found in the google search are actually SSL, right? [05:22:21] ah, John Du Hart thinks it can be fixed in the DNS which would be simple [05:22:44] = johnduhart who is here [05:23:09] idk what points to mail.wm.o [05:23:23] (maybe it's an MX record for something?) [06:25:41] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 86, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/2/1: down - Core: cr1-sdtpa:xe-0/0/1 (Level3/FPL, CV71026) {#2008} [10Gbps wave]BR [06:27:21] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Core: cr1-eqiad:xe-5/2/1 (FPL/GBLX, CV71026) [10Gbps wave]BR [07:33:12] Hmundol: ping [08:04:59] PROBLEM - Lucene on search3 is CRITICAL: Connection timed out [08:52:40] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 3.9968854902 [08:58:00] RECOVERY - Packetloss_Average on emery is OK: OK: packet_loss_average is 3.0833322 [09:04:30] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [09:12:30] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [09:20:49] hey is there anyone who can help out with a config change to support an outreach event that we're doing the day after tomorrow? [09:24:55] What type of config change is it? They generally should be filed in bugzilla [09:26:43] it's to remove the limit of not allowing more than 7 new users to create their usernames from a single IP address in a 24 hour period. given we're doing an editing workshop, we want to encourage as many of them to create usernames there and then. it's only for a few (6 hours at most; workshop is on for 3 hours but I'm just putting in a little buffer in case it starts late or ends late) [09:26:57] expecting potentially 20-30 people attending [09:27:18] Yeap, just file that in bugzilla and then poke in here with a link to the bug [09:27:47] got it. thanks! [09:38:58] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [09:39:58] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 88, down: 0, dormant: 0, excluded: 0, unused: 0 [09:52:18] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 453303 MB (3% inode=99%): [09:53:39] Hmundol: the wikimedia.in wiki doesn't appear to be hosted by the foundation [09:54:13] it's the Wikimedia India Chapter's website. [09:54:38] it's hosted by WM-Switzerland [09:54:39] yes, but it doesn't appear to be hosted by the foundation (not all chapter websites are) [09:55:01] so we can't do site configs for them, you will need to contact your local wiki tech team [09:55:32] unless you want the limits raised somewhere else [09:55:37] no, but this config setting changes are not to edit wikimedia.in but to edit en-wp... [09:55:38] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 411038 MB (3% inode=99%): [09:56:09] and bengali wikipedia (bn-wp) [10:03:08] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 103 MB (1% inode=60%): /var/lib/ureadahead/debugfs 103 MB (1% inode=60%): [10:13:08] RECOVERY - Disk space on srv222 is OK: DISK OK [10:15:10] Hmundol, or you can take help from account creator. [10:15:58] They will create the account and will mail the password to their e-mails. All they have to do to fill-up the a small form. [10:18:28] Astatine: that's one way but we'd rather have them create their own user accounts. makes it so much real. [10:18:40] so much "more" real [10:18:48] * Astatine nods. [10:51:37] RECOVERY - MySQL slave status on es1004 is OK: OK: [11:29:54] PROBLEM - LVS Lucene on search-pool1.svc.pmtpa.wmnet is CRITICAL: Connection timed out [12:31:35] RECOVERY - LVS Lucene on search-pool1.svc.pmtpa.wmnet is OK: TCP OK - 0.002 second response time on port 8123 [12:46:15] PROBLEM - Puppet freshness on srv199 is CRITICAL: Puppet has not run in the last 10 hours [12:52:28] �� [13:37:46] RECOVERY - Lucene on search3 is OK: TCP OK - 0.002 second response time on port 8123 [15:15:42] apergos: You there? [15:15:50] yes [15:16:02] but going to take a break soon, what's up? [15:16:12] I spotted a few errors with the POTY and archives you just posted on dumps.wikimedia.org [15:16:32] archives, not really error, I think [15:16:42] let's hear it [15:16:54] but the POTY index, 2010 deserves its own line :P [15:17:02] woops [15:17:07] I'll fix that [15:17:20] for archives, those dumps with the "new style" page points to the Internet Archive [15:17:42] you have to give me a link [15:17:49] yeah, finding it [15:18:02] browsing through, found this error, forgot where it is :P [15:18:21] and now I am getting 403s [15:18:25] I don't see where the 2010 index page is not on its own line? [15:18:28] what page is that? [15:18:47] hold, I am forbidden to access currently [15:18:55] so I can't really see... [15:19:08] downloading two other things are we? :-P [15:19:20] not really, everything finished [15:19:45] oh dear, whatever happened [15:20:22] but AFAIK, http://dumps.wikimedia.org/other/poty should have 2010 in its own line [15:20:52] it does when I look at it [15:21:00] http://dumps..wikimedia.org/archives points to wrong page of the incremental dumps [15:21:15] what nice timing to be getting 403... haiz [15:21:21] f5 in your browser maybe? [15:21:29] cause my copy of the poty page looks fine [15:21:46] yeah, but whats wrong with the server to block me now... [15:22:52] are you downloading two things in another window? [15:23:14] no, I am doing nothing [15:23:36] ah, back [15:23:53] yeah, I still see 2010 together in the same line as 2009 [15:24:31] you forgot the
tag between 2009 and 2010 [15:24:52] it's in my copy [15:24:59] and I f5ed and everything [15:25:30] hmm, still there for me [15:25:41] as for the other thing: http://dumps.wikimedia.org/archive/enwiki/20080103/ [15:25:45] its pointing to the IA [15:25:51] wayback for links [15:26:10] oh, that one I got form them [15:26:24] I should force a link to the directory by moving that page out of the way [15:26:26] I'll do that [15:26:58] :) [15:27:37] check the archive page link again please [15:28:07] that link ^ [15:28:08] ? [15:28:20] no [15:28:29] the main page of archives? [15:28:45] yeah, I see the link to the incrementals fixed [15:29:28] chekc the poty and the other one [15:29:51] poty fixed [15:29:55] other one... [15:30:03] yep [15:30:06] οκ [15:30:09] now its a directory listing [15:30:12] yep [15:30:15] and that's just fine [15:30:20] one more funny one... [15:30:30] Broken: http://dumps.wikimedia.org/archive/2006/2006-12/enwiki/20061130/ [15:30:37] I mean, dumps failed lol [15:31:11] but overall, thanks for the archives :) [15:31:32] there's a good 2006 in the other directory :-P [15:31:38] and you're welcome [15:32:17] I especially had fun looking at the 2002 ones [15:32:20] now just attempting to move them to IA [15:32:28] did you get these files from IA itself? [15:32:33] most not [15:32:43] I see [15:32:44] the 2008 yes [15:32:54] the 2007 maybe from people who hoarded them [15:32:56] for enwiki? [15:33:08] the 2002 through 2005 from an old hard drive [15:33:30] * Hydriz feels dumb, asking if 2008 means enwiki when there is only enwiki [15:33:32] I'm hoping to get more (across projects for a couple times in 2007 2008) from someone else, dunno though [15:33:36] heh [15:33:47] I just missed emijrp, haiz [15:34:12] need to talk to him about these dumps too :P [15:35:28] are you/will you be compiling poty pictures for 2008 and 2011 too? [15:35:44] will get around to it [15:36:22] no hurry, just hoping to see it available :) [15:36:54] I was also asking for old dumps of the Wikimedia Incubator too [15:37:08] like, those dumps of wikis that were deleted [15:37:15] though we lost quite a bit of them [15:37:25] I don't know if we can get those [15:37:40] this is probably it for this month, we'll see what next month brings [15:37:58] speaking of deleted wikis... [15:38:18] I recalls seeing some dumps of wikis that were entirely deleted from Wikimedia [15:38:44] eg: http://dumps.wikimedia.org/dkwikibooks/ [15:38:49] uh huh [15:39:34] like, is it linked from anywhere? [15:39:48] ? [15:40:05] I didn't understand the question [15:40:27] no, is there any interest in these deleted wikis' dumps? [15:41:16] oh [15:41:29] I think it's fine for the last one of each of those to get saved to the archive for sure [15:41:31] why not? [15:41:48] yeah, so lets just move it to the archive :) [15:42:40] I feel like I have lots of questions to ask you but just don't want to disturb your rest :P [15:43:07] hahaha [15:43:18] you could write up a list of qs and put it [15:43:19] hmmm [15:43:24] on some page... [15:43:27] hmmm [15:43:47] is there a page for you guys on meta? [15:43:59] about the dumps? [15:44:05] no [15:44:13] about the archive team [15:44:21] as related to wm files [15:44:56] wait, can you rephrase that? I don't quite get you :( [15:47:31] you have a pile of qs for me [15:47:41] I assume they mostly have to do with moving stuff to the archives and such? [15:48:33] no, its just a feeling lol [15:48:50] but yeah, some :P [15:49:18] like, "can I burn your server?" :P [15:49:36] so if the archive team had a page on meta you could leave me questions on the talk page there (maybe tell me in irc that there's something) [15:49:56] but actually I didn't burn *your* server, but IA's ones [15:50:13] if you were to ask me that q I would have to tell you "no" [15:50:16] so best not to ask :-P [15:50:29] not to worry, I won't :P [15:50:46] I tried to, but its the IA that failed first [15:51:02] spam throwing 500s just a short while after I started [15:51:30] ah hah [15:57:14] oh yes, just remembered [15:57:24] do you have old image dumps of 2005? [15:58:21] no, there's one on the archive though [15:58:35] yeah, I know [15:58:43] but I was hoping if you had other wikis [15:58:47] ah [15:58:52] not from 2005 afaik [15:58:58] I have other plans about the iimages though [15:59:06] parallel? :P [15:59:07] though again nothing is likely to happen for another month [15:59:10] nooooooo [15:59:18] we are not going to generate a pile of current image dumps [15:59:38] so what plans do you have? [15:59:47] hold yer horses :-P [16:00:09] can't give up all my secrets! [16:00:16] * Hydriz sees a bright future in image dumps [16:00:18] (also can't g around building up false expectations) [16:01:25] * Hydriz holds back WikiTeam's plans to grab Wikimedia Commons [16:01:26] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/1958 [16:01:26] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1958 [16:02:24] grab away [16:02:29] but you have to be nice to the servers [16:03:18] yeah, but the database is constantly updating, so thats the irritating part [16:03:29] yes, it is [16:03:38] lots of busy uploaders :-) [16:04:09] if we were to really grab that site, we would also need the list of changes that happened while we are grabbing [16:04:20] that is what apis are for [16:04:21] which, is a work in progress [16:04:22] it's easy [16:04:29] yeah, I know [16:04:36] we are still working on our grabber [16:04:56] how we grab, when we grab and what we do with the result/grab [16:05:00] uh huh [16:05:36] lets just hope that SOPA doesn't pass [16:05:44] or else we would be too late [16:05:55] well right now they have been pulled from the agenda [16:06:00] not scheduled [16:06:16] thats nice [16:06:24] i sure is [16:06:38] if the success rate is higher, then we would have to rush our project [16:06:53] no [16:06:57] you would send us some disks [16:07:02] and we would do some copies [16:07:10] lol [16:07:19] and pass it would be passed back to us [16:07:29] s/pass//g [16:07:32] but we would insist you have a pubically accessible site to put them on [16:07:49] * Hydriz looks at the IA [16:07:54] no [16:08:26] as in we will go through the pain of manually copying the stuff if the disks will go to a mirror site directly [16:08:48] about that... [16:09:07] thats not really our priority if we can't get the files first lol [16:09:23] though we would love you to copy all that images [16:09:28] and PDFs [16:09:31] and OGGs [16:09:53] then you'll just have to wait for my evil plans to hatch :-P [16:09:58] heh [16:11:25] so how is dataset1001? [16:11:36] making me very very very sad [16:11:51] I am waiting for rob to wake up so I can ask him to order a different raid card [16:12:08] s/card/controller card/ [16:12:36] lol waiting for him to wake up? [16:12:46] its Tuesday here [16:13:42] not here [16:14:12] yeah [16:14:28] I just passed midnight [17:18:51] could one delete /stats on any wikipedia? It's pointing to knams which has been killed a while back [17:24:51] heh, though I see offending lines [17:24:56] I have to remember how to push them out :) [17:26:41] gonna go grab some breakfast [17:26:49] I'm new at this office, can get lost easily [17:44:44] !log reedy synchronized wmf-config/InitialiseSettings.php 'Push changes for aswiki by Santhosh' [17:44:46] Logged the message, Master [18:04:27] !log nikerabbit synchronized php-1.18/extensions/CodeReview/ui/CodeRevisionView.php 'i18ndeploy r109836' [18:04:28] Logged the message, Master [18:05:33] !log nikerabbit synchronized php-1.18/extensions/WebFonts/ 'i18ndeploy r109836' [18:05:34] Logged the message, Master [19:15:17] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [19:22:07] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [20:23:17] New patchset: Pyoungmeister; "adding check commands for udp2log. if this works correctly, I will add a contact group for consumers of this who have shells" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2049 [20:33:02] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2049 [20:33:50] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2049 [20:34:05] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33166 - Creation of a new namespace for Malagasy Wiktionary' [20:34:07] Logged the message, Master [20:39:03] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33841 - Re-point $wgLogo to on zhwiki' [20:39:04] Logged the message, Master [20:44:45] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33899 - Request for Narayam in outreach.wikimedia.org' [20:44:46] Logged the message, Master [21:07:10] New patchset: Diederik; "Initial commit of pipeline.py" [analytics] (master) - https://gerrit.wikimedia.org/r/2050 [21:12:44] New review: Ottomata; "I'm working on documentation for this stuff, and will have lots of comments shortly." [analytics] (master) C: 1; - https://gerrit.wikimedia.org/r/2050 [21:14:11] New review: Diederik; "We are happy." [analytics] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2050 [21:14:12] Change merged: Diederik; [analytics] (master) - https://gerrit.wikimedia.org/r/2050 [21:15:45] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33864 - Flood flag on sr.wiki' [21:15:47] Logged the message, Master [21:18:16] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33864 - Flood flag on sr.wiki' [21:18:17] Logged the message, Master [21:26:55] !log reedy synchronized wmf-config/InitialiseSettings.php 'Wrapping more long lines' [21:26:56] Logged the message, Master [21:54:15] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [21:54:28] ohoh [21:54:42] New patchset: Pyoungmeister; "some cleanup and adding diederik/analytics group to alerts for udp2log" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2051 [21:55:42] Reedy: did yo see nagios' megassage about the empty / of srv223? [21:55:47] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2051 [21:55:47] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2051 [21:59:19] DaBPunkt, have now. I can't fix them though :( [21:59:40] ok [22:24:15] RECOVERY - Disk space on srv223 is OK: DISK OK [22:50:30] hi sysops [22:50:50] who really like videos and university outreach? [22:52:30] o0 [22:52:45] Pharos, are you after someone to bulk upload stuff? : [22:52:46] :p [22:52:53] i am [22:53:09] stuff that's >100 MB [22:53:16] are they hosted somewhere? [22:53:22] and how many? [22:53:24] they could be [22:53:36] maybe a dozen [22:54:36] Easiest way of dealing with this is a bugizlla request.. If you can prepare all the description files... Then put the videos somewhere we can download them to the cluster from.. [23:00:11] PROBLEM - Puppet freshness on srv199 is CRITICAL: Puppet has not run in the last 10 hours [23:01:09] nighty [23:03:51] RECOVERY - Puppet freshness on cp1044 is OK: puppet ran at Mon Jan 23 23:03:21 UTC 2012 [23:06:08] New patchset: RobH; "more servers to decom" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2052 [23:07:40] New review: RobH; "I'm the best at what I do." [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2052 [23:07:41] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2052 [23:10:56] !log preilly synchronized wmf-config/CommonSettings.php 'add MobileFrontend custom logo support' [23:10:57] Logged the message, Master [23:11:18] !log preilly synchronized wmf-config/InitialiseSettings.php 'add MobileFrontend custom logo support' [23:11:19] Logged the message, Master [23:12:23] !log push config change for custom logos [23:12:25] Logged the message, Master [23:16:02] !log preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to Mobile Frontend for custom logo support' [23:16:03] Logged the message, Master [23:20:50] preilly: hello :) Someone complained about MobileFrontend hiding the Z3988 CSS class [23:20:54] preilly: bug report is https://bugzilla.wikimedia.org/33906 [23:21:05] hashar: yes, I saw that [23:21:10] great :) [23:21:15] hashar: the fix should go out tomorrow [23:21:23] wonderful!! [23:21:26]