[00:23:48] New patchset: Lcarr; "correcting smtp server for formey post-commit hook" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2005 [00:24:05] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2005 [00:24:23] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2005 [00:24:23] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2005 [00:51:35] !log Resent mediawiki-cvs commit emails from r109549 through r109704 [00:51:36] Logged the message, Master [01:11:32] New patchset: pugmajere; "Rework the JunOS firewall creation tools." [operations/software] (master) - https://gerrit.wikimedia.org/r/2009 [01:11:33] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2009 [01:16:58] New patchset: pugmajere; "Clean up some bad whitespace unintentionally introduced in 15e696f" [operations/software] (master) - https://gerrit.wikimedia.org/r/2010 [01:19:34] New review: Lcarr; "whitespaces fixed in change-id I95f19aa6" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2009 [01:19:34] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2009 [01:19:44] New review: Lcarr; "(no comment)" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2010 [01:19:44] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2010 [02:05:22] !log LocalisationUpdate completed (1.18) at Sun Jan 22 02:05:22 UTC 2012 [02:05:25] Logged the message, Master [02:21:50] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1630s [02:25:40] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1861s [02:35:46] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:35:56] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [02:41:57] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:19:03] RECOVERY - Disk space on es1004 is OK: DISK OK [04:24:53] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:36:43] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:44:25] !log depooling ssl1004 [05:44:26] Logged the message, Master [06:24:18] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 86, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/2/1: down - Core: cr1-sdtpa:xe-0/0/1 (Level3/FPL, CV71026) {#2008} [10Gbps wave]BR [06:28:08] PROBLEM - Router interfaces on cr1-sdtpa is CRITICAL: CRITICAL: host 208.80.152.196, interfaces up: 76, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Core: cr1-eqiad:xe-5/2/1 (FPL/GBLX, CV71026) [10Gbps wave]BR [06:35:43] log repooling ssl1004, depooling ssl4 [06:36:17] Ryan_Lane: ! [06:36:21] ? [06:36:26] forgot the ! [06:36:29] ah [06:36:33] !log repooling ssl1004, depooling ssl4 [06:36:34] thanks [06:36:34] Logged the message, Master [07:00:55] New patchset: Catrope; "Adding .gitreview file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2014 [07:01:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2014 [07:23:08] RoanKattouw: did you catch the fixme on the rev that introduced loader.setBlocking ? I wonder how we're gonna fix that one [07:23:42] I did not, link? [07:24:22] RoanKattouw: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/108184#code-comments [07:24:25] last cmt [07:25:49] load() returns before the load.php calls are entirely loaded and parsed and executed [07:25:53] Yeah [07:25:59] Asynchronousity [07:26:02] Dammit [07:26:03] which makes sense in a way, since its blocking the page not JS itself at that poiny [07:26:11] point [07:26:23] Passing the variable around in the load() call was uglier, so I decided against [07:26:24] it [07:26:27] But it's safer, I *knew* it [07:26:40] It felt safer at least [07:26:44] I think we'll have to pass blocking to load() which will do a promise() type of thing (when all is done set blocking off( [07:27:07] A better fix would be to pass the blocking parameter to load() and pass it through the entire call stack [07:27:25] Not nice but at least that way it's preserved across async calls, and things that shouldn't be blocking aren't made blocking [07:27:43] before you knock yourself out though, i suggest you/me first go through paused/stepped javascript and verify that that's the issue [07:28:27] That's simple [07:28:28] Chrome does that nicely, per statement stepping through with scopes listed around it [07:28:30] Set two breakpoints [07:28:38] oh ofcourse [07:28:40] One inside setBlocking(), one inside addScript() [07:28:53] They will race each other and it'll be quite obvious to you which one wins [07:29:03] k, gotta go. just finished an allnighter. gonna get some sleep now [07:29:08] Oh right [07:29:10] :D [07:29:12] I was thinking of going to sleep [07:29:16] But at least it's evening here [07:29:21] It's friggin 8:30am where you are :O [07:29:25] yeah [07:29:34] I got up at 1pm though [07:29:46] Ah OK [07:29:53] k, byte [07:29:56] bye [07:29:59] Good night [10:05:36] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 389336 MB (3% inode=99%): [10:06:55] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 385133 MB (3% inode=99%): [10:15:45] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [10:25:45] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.031 second response time [10:51:05] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:45:22] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [12:46:19] Hello :) [12:47:25] I'm from Spanish Wikipedia. I'd like ask something... [12:53:28] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [12:56:42] ask away, if no one answers right away just be patient [13:00:22] Ok, thank you [13:01:00] First of all, sorry for my English level... [13:01:29] no worries, my spanish level is 0 :-( [13:01:50] Ok, no worries :) [13:02:22] I'd like ask if Wikimedia servers are ready to support a solar storm [13:02:43] are we expecting one shortly? [13:02:44] :-) [13:03:10] (that's not a trick question actually. ) [13:03:38] yes we do [13:03:48] apergos: there is ment to be a few this year actually [13:03:48] time frame? [13:03:50] a moderate solar storm will hit earth today [13:03:54] ah today [13:04:03] it takes about 24-48 hours for a solar storm to come from the sun and hit us [13:04:19] we have entered the 11th year of suns magnetic shifts [13:04:37] every 11 years the magnetic field of the sun shifts creating massive solar flares [13:04:43] We don't know when it can produce, but there is a danger, and Wikimedia must be ready [13:04:55] we do know when it happens [13:05:03] it is a matter of following the sun spots [13:05:08] I rather expect eqiad to be in good shape as far as that goes (even if we would up out of service I expect the data to be protected) but we're not fully operational there yet [13:05:22] should something like the solar flare like 1850 happen there would be no electronics left [13:05:24] we do however have full copies of images and all database there [13:05:32] solar flares can corrupt data beyond recovery [13:05:41] well... if all electronics are gone whatsoever [13:05:54] I guess people not being able to access wikipedia will be a moot point [13:06:04] because what would they access it with?? [13:06:05] the 1850s flares caused auroras all the way down to rome [13:06:13] electronics could be rebuilt [13:06:20] Yes, of course [13:06:36] the idea is having a copy of the data in a shielded area [13:06:42] There is an "easy" solution - a Faraday cage [13:06:48] I am sure they could but I'm equally sure than any manufacruting plant righ tnow relies on electronic [13:06:50] s [13:06:51] Invadinado not really [13:06:57] the highly charged particles would be everywhere [13:06:58] so it might take a good while [13:07:06] faradays cage would work agains something like a lightning [13:07:24] if you have lightning at everywhere at once faradays cage wont work [13:08:24] apergos indeed the catastrophic damage would take ages tor ecover form but I genuinlely feel a shielded dump may be worth the while. [13:08:38] http://sidc.be/ according to them we had a strong storm in September of last year [13:09:09] A catastrophic solar flare is a low likely hood high risk kind of a situation [13:09:11] apergos that wasnt the storm season [13:09:17] 2012 is the 11th year of the cycle [13:09:27] it could be calm too [13:09:39] but the risk exists for every 11 years. [13:10:21] the real threat is if earths magnetic field and suns solar burst are in resonance [13:10:43] IB. Solar Activity Forecast: Solar activity is expected to be low through the period (22 - 24 January) with a chance for an isolated M-class flare. [13:10:50] http://www.swpc.noaa.gov/forecast.html [13:11:10] http://en.wikipedia.org/wiki/Solar_storm_of_1859 [13:11:31] apergos the prediction range is short [13:11:39] consider february march [13:11:47] well someone mentioned "today" [13:11:48] Yes, I've watched some videos of NASA about it [13:11:54] I asked about a time frame for a reason [13:12:23] eqiad is as state of the art as there is in data centers [13:13:07] Today can be... But I think it can be any day... [13:13:21] apergos it can hit today [13:13:35] it is not like it hits and thats it [13:13:42] the interaction with the magnetic field is complex [13:14:02] it will keep bouncing back and forth in the magnetic field [13:14:08] it can be harmless or more harmful [13:14:28] well, we have the forecasts we have and we have the data centers we have [13:14:33] none of that is going to change in a day [13:14:43] over the next while eqiad will come fully on line [13:15:12] Ok, thank you very much for the information :) [13:15:19] Then, are we ready? [13:15:21] apergos I still suggest it is unwise to go on interstellar travel this week so I strogly suggest staying on the planet, ok? :) [13:15:36] I'll bear that in mind [13:15:43] * apergos looks at their agenda [13:15:56] also no foundation satelite launches please. [13:16:14] I'm afraid I don't have any control over that [13:16:19] JimboSat [13:16:19] it's a different department [13:16:33] Well, we will do an effort, and we will wait to go on interstellar travel... :) [13:16:35] ApergosSat? [13:16:42] so getting back to your question Invadinado: [13:17:46] we have replicated copies of images and text and metadata in eqiad, so if a disaster hits us that is survivable by a data center then we should have our data intact [13:18:19] for true disaster scenarios, all bets are off, but then we will all have much more to worry about (rebuilding industries from the ground up, etc.) [13:19:17] apergos btw: http://www.spaceweather.com/ [13:19:27] Ok, perfect, we are ready. Already I stay calm. :) [13:19:27] ya the arival seems to be tonight [13:20:18] Oh... [13:21:07] Well, thank you for your help, and for your information [13:21:22] apergos where is the data though [13:21:30] I guess those were from c-class flares [13:21:33] I hope we have a server around the equator [13:21:48] apergos the class is half the story :) [13:22:10] . We expect at most a glancing blow from the ICME, with geomagnetic conditions up to the minor or moderate storm level. [13:22:43] Thanks again [13:22:52] I have to return to es.wikipedia... [13:22:58] (that's from the solar in fluences site [13:23:02] See you! :) [13:23:04] you're welcome [13:23:06] have a nice day [13:23:17] apergos I am not talking short term here [13:23:40] Good luck! Bye! :) [13:23:45] :-) [13:23:51] fox news reported "Massive solar flare headed toward Earth, may spark celestial light show" so I expect a small storm. [13:23:58] fox news? [13:24:01] yes [13:24:03] *eyeroll* [13:24:12] they are reliable, I just believe the opposite they say [13:24:29] if I have to choose between fox news and scientific data centers, I know who to rely on :-P [13:24:52] if fox news reports snow, I will rush to the beach to catch the right spot as it will be sunny [13:24:59] nice [13:25:02] speaking of which [13:25:09] I need to go do some errands while it is sunny [13:25:13] ya [13:25:13] it's going to rain later [13:25:24] its gonna rain cats, dogs and mice [13:25:27] and I just got to a break in my documentation cleanup [13:25:47] RECOVERY - Router interfaces on cr1-sdtpa is OK: OK: host 208.80.152.196, interfaces up: 78, down: 0, dormant: 0, excluded: 0, unused: 0 [13:25:49] so I'm heading out for awhile, see folks later [13:28:17] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 88, down: 0, dormant: 0, excluded: 0, unused: 0 [13:29:18] at least we are not dealing with a GRB [14:54:29] hexmode: With regard to JS on smaller wikis, is there any chance that the process of checking for missing dependencies could be automated? [16:36:10] Hey, having a strange problem. Whether I'm logged in or not, and I've tried clearing my chache, clicking anywhere on the page http://en.wikipedia.org/wiki/Talk:Whiplash_%28Selena_Gomez_%26_the_Scene_song%29/GA2 redirects me to the admins' noticeboard; specifically, a section on Dream Focus [16:36:12] I can't work out how or why this is happening [16:36:37] Using Firefox [16:36:42] It doesn't happen on IE [16:36:51] I've tried closing the browser, still happens [16:37:23] DOES happen on Chrome [16:37:44] Does anyone else have the same problem? Anyone have any idea why it's happening or how to fix it? [16:38:38] J_Milburn: it's [16:38:45] related to a page move protection [16:38:55] Just purged the page [16:38:59] That seems to have fixed it [16:39:01] How weird [16:39:13] Jarry, is this something that has happened elsewhere? [16:40:34] Not that I'm aware of. [16:41:31] J_Milburn, I can't reproduce on Chromium 15.0.874.106 Ubuntu 11.10. You could try right clicking anywhere and click 'inspect element' and see if it leads to an element with a href="something ANI like" [16:41:39]

[16:41:44] Is the offending HTML [16:42:05] Should I send an error report about this? [16:42:55] Ah, got it [16:42:56] https://en.wikipedia.org/w/index.php?title=Template:Al&diff=472630772&oldid=457917066 [16:43:04] ^ The offending edit [16:43:45] nice job [16:44:37] Wow, good catch [16:44:43] I'll protect that thing [16:44:48] guess they must be pissedoff or something [16:45:09] JM: just did [16:45:15] Sorry, didn't see your note. [16:45:28] Ah, that's fine [16:45:36] Anyway, 1 year semi, should be adequate. [16:48:58] J Milburn: not sure if it's worth following up (SPI?_ [16:49:14] It was a TOR IP [16:49:20] So they've been blocked for two years [16:49:26] And we'll never be able to work out who it was [16:49:41] There's always going to be stuff like this surrounding high-profile blocks... [16:50:11] Who has been blocked for the past two years? [16:50:34] Oh, sorry [16:50:36] Misread you [16:50:48] Yes, they've now been blocked. Case closed, I guess. [16:51:00] It /would/ be worth leaving a note on AN [16:51:05] So that if it happens to others... [16:51:11] But that would be a bit BEANS-y [16:51:16] Just a little. [16:51:21] Probably RBI. [16:52:33] RBI? [16:53:54] Revert, Block, Ignore [16:54:02] oh [17:09:37] PROBLEM - Puppet freshness on ssl4 is CRITICAL: Puppet has not run in the last 10 hours [17:33:15] apergos the servers would probably survive a GRB. [17:33:27] they are in amsterdam and florida right? [17:53:52] Hi, I have a question, did you already know about this bug? http://meta.wikimedia.org/wiki/Special:GlobalUsers/Global_rollback (and this is for all global groups) [17:54:15] except for stewards, because that works without using letters in capital [17:54:23] see http://meta.wikimedia.org/wiki/Special:GlobalUsers/steward [18:10:37] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 86, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-5/2/1: down - Core: cr1-sdtpa:xe-0/0/1 (Level3/FPL, CV71026) {#2008} [10Gbps wave]BR [18:20:37] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 88, down: 0, dormant: 0, excluded: 0, unused: 0 [18:32:22] !log running apt-get upgrade on snapshot1 to get wikimedia version of php-wikidiff2 [18:32:24] Logged the message, Master [18:39:05] !log running apt-get upgrade on snapshot2 and snapshot4 [18:39:06] Logged the message, Master [18:39:31] New patchset: Mark Bergsma; "Explicitly pin the Wikimedia version of php-wikidiff2 to prevent problems in the future" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2016 [18:41:00] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2016 [18:41:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2016 [18:42:03] !log running apt-get upgrade on searchidx2 [18:42:04] Logged the message, Master [18:56:23] New patchset: Catrope; "Adding .gitreview file" [labs/private] (master) - https://gerrit.wikimedia.org/r/2017 [19:23:01] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2014 [19:23:02] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2014 [19:23:11] New review: Ryan Lane; "(no comment)" [labs/private] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2017 [19:23:11] Change merged: Ryan Lane; [labs/private] (master) - https://gerrit.wikimedia.org/r/2017 [19:24:15] New patchset: Catrope; "Add .gitreview file" [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/2018 [19:24:39] New review: Ryan Lane; "(no comment)" [integration/jenkins] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2018 [19:24:40] Change merged: Ryan Lane; [integration/jenkins] (master) - https://gerrit.wikimedia.org/r/2018 [20:12:34] New patchset: Domas; "PEP8 sans line lengths, fixed the 'no database' condition to be somewhat nicer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2021 [20:12:51] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2021 [20:14:53] New patchset: Domas; "uh oh, tabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2022 [20:16:57] New review: Domas; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2021 [20:16:57] Change merged: Domas; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2021 [20:17:13] New review: Domas; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2022 [20:17:14] Change merged: Domas; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2022 [21:38:33] !log preilly synchronized php-1.18/extensions/MobileFrontend/MobileFrontend.php 'update to mobile frontend to remove sopa banner' [21:38:34] Logged the message, Master [21:38:54] !log remove SOPA banner [21:38:55] Logged the message, Master [21:41:28] zzz =_= [22:08:08] Hi [22:08:43] I'm getting a 503 server error @ the mobile version of the dutch wikipedia [22:08:44] Error 503 Service Unavailable Service UnavailableGuru Meditation: XID: Varnish cache server [22:09:18] Error 503 Service Unavailable [22:14:36] Guest72801: okay, looking into it [22:14:52] Thanks [22:15:18] Guest72801: is it working now? [22:15:30] Sec. 3G is slow.. [22:16:25] No [22:16:39] But another XID now. Need that? [22:19:45] Brb, switching to wifi.. Will ping out [22:30:34] New patchset: pugmajere; "Upate firewall creation tools to support protocol." [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:30:35] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2032 [22:33:56] New patchset: pugmajere; "Upate firewall creation tools to support protocol." [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:33:57] New review: gerrit2; "Lint check passed." [operations/software] (master); V: 1 - https://gerrit.wikimedia.org/r/2032 [22:35:24] New review: Lcarr; "Looks better than good, looks awesome!!!" [operations/software] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2032 [22:35:24] Change merged: Lcarr; [operations/software] (master) - https://gerrit.wikimedia.org/r/2032 [22:35:32] Well [22:35:37] It's fixed [22:53:12] New patchset: Domas; "Add atop to all servers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2034 [22:53:28] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2034 [22:54:17] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2034 [22:54:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2034 [22:54:32] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [22:58:58] gn8 folks [23:02:42] PROBLEM - Puppet freshness on cp1044 is CRITICAL: Puppet has not run in the last 10 hours [23:07:33] New patchset: Domas; "use all servers in srv250-257 range for memcached" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2035 [23:07:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2035 [23:08:04] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2035 [23:08:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2035 [23:11:23] PROBLEM - mobile traffic loggers on cp1042 is CRITICAL: PROCS CRITICAL: 0 processes with command name varnishncsa [23:26:09] !log testing new nginx package on ssl4 [23:26:10] Logged the message, Master [23:31:02] !log pushed out unstripped version of wmerrors [23:31:04] Logged the message, Master [23:33:55] New patchset: Lcarr; "Add a script to run a local parse validate of the puppet configs." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2040 [23:34:11] New patchset: Lcarr; "Hide warnings about storeconfigs when running a local lint." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2041 [23:34:26] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2040 [23:34:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2040 [23:34:27] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2040 [23:34:27] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2040 [23:34:29] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2041 [23:34:29] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2041 [23:35:22] LeslieCarr: still working? :) [23:35:35] hackathon :) [23:35:37] yep [23:35:37] hehe [23:35:45] ahhh right [23:36:09] RECOVERY - mobile traffic loggers on cp1042 is OK: PROCS OK: 2 processes with command name varnishncsa [23:36:18] * werdna is getting to work and watching to see if SF will be in the superbowl [23:36:39] https://en.wikipedia.org/wiki/Hoodie isn't loading [23:37:15] what's wrong? [23:37:59] hm [23:38:07] well, I just deployed a new version of nginx [23:38:14] pir^2: are you in europe? [23:38:27] why? [23:38:31] because it's working for u [23:38:32] *us [23:38:35] and we're in the US [23:38:45] * matthewrbowker doesn't work for me... [23:38:49] you would be hitting a different datacenter [23:38:52] yeah? [23:39:03] This is a major problem then? [23:39:06] can you *please* tell me where you are located? [23:39:10] seriously, it's info I need [23:39:18] Ryan_Lane, WFM [23:39:24] * Ryan_Lane stabs [23:39:27] New patchset: Catrope; "ROCK!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2042 [23:39:30] so that's europe apparently fine ;) [23:39:35] cool [23:39:35] US [23:39:35] ok [23:39:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2042 [23:39:57] !log restarting nginx servers [23:39:58] Logged the message, Master [23:40:01] occasionally reload has issues [23:40:25] works for me [23:40:35] it's using the SH scheduler [23:40:44] so, it is likely one or two servers acting up [23:40:55] https://en.wikipedia is apparently having issues, in case it hasn't been reported. [23:41:13] give me a min, then I'll ask you if it's working [23:41:19] Change abandoned: Catrope; "Demo change" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2042 [23:41:46] k [23:41:54] try now [23:42:10] fine now, thanks [23:42:13] great [23:43:07] works [23:43:49] Thanks for quickly fixing it (noticed the old school design on Commons) :-) [23:44:24] yw [23:44:53] <3 Ryan_Lane [23:45:18] is alolita around at the hackathon? [23:45:40] !log repooling ssl4 [23:45:42] Logged the message, Master [23:46:05] !log changing nginx config to use the escaped useragent [23:46:06] Logged the message, Master [23:46:47] werdna: Yes she is [23:47:07] RoanKattouw: reckon you could poke her for me? [23:47:23] New patchset: Ryan Lane; "Making https servers use the new encoded user agent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2043 [23:47:38] werdna: In ~30 [23:47:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2043 [23:47:45] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2043 [23:47:45] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2043 [23:47:52] RoanKattouw: nw [23:47:55] I'll be around all day [23:48:59] RECOVERY - Puppet freshness on ssl4 is OK: puppet ran at Sun Jan 22 23:48:58 UTC 2012 [23:49:41] I am supposed to be judging now [23:54:01] well, somebody else could poke her if somebody is in poking distance [23:54:54] robla: are you in alolita-poking range? [23:55:55] werdna: looks like she's busy with someone else [23:58:47] aha