[01:25:21] I get "unable to connect" when I try to hit http://ganglia.wmflabs.org/ (Firefox can't establish a connection to the server at ganglia.wmflabs.org.) - known problem? [01:28:56] Ryan_Lane: http://ganglia.wikimedia.org is responsive, but http://ganglia.wmflabs.org/ isn't (the latter is the one linked to from the monthly report/status update) [01:29:11] oh [01:29:13] gimme a sec [01:29:48] heh [01:29:51] seems the instance is down [01:31:07] g'night [01:31:15] * Ryan_Lane waves [02:13:30] !log LocalisationUpdate completed (1.20wmf1) at Wed May 2 02:13:30 UTC 2012 [02:13:34] Logged the message, Master [02:32:39] !log updated production civicrm to r1723 [02:32:43] Logged the message, Master [02:35:51] !log LocalisationUpdate completed (1.20wmf2) at Wed May 2 02:35:51 UTC 2012 [02:35:54] Logged the message, Master [08:20:59] foobar dologmsg broken? [08:24:52] hashar: using dologmsg on fenari [08:26:16] !log hashar: using my own logging script [08:26:18] Logged the message, Master [08:40:01] !log hashar synchronized php-1.20wmf2/includes/GitInfo.php 'Fix Special:Version for 1.20wmf2 (commit ae12df0)' [08:40:04] Logged the message, Master [09:46:26] bonjour les triso :) [09:49:45] ca va bien les triso ? [09:54:11] bonjour whoose triso [09:55:24] je vous traite de triso et vous reagisser meme pas ha ha mdr [09:56:38] bonjour rr0 triso mdr :p [09:57:47] triso un jour,triso toujours [10:00:04] triso [10:00:09] triso [10:00:10] triso [10:00:12] triso [10:00:14] triso [10:00:15] triso [10:00:19] ha ha [10:01:42] triso [10:01:43] triso [10:01:44] triso [10:01:46] triso [10:01:47] trisotriso [10:01:50] trisotrisotriso [10:01:51] triso [10:01:53] triso [10:01:56] triso [10:01:58] !ops flooding [10:05:48] triso [10:05:49] triso [10:05:50] triso [10:05:52] triso [10:06:01] triso [10:06:06] héhé [10:08:51] Kooo: bonjour [10:12:56] Kooo: c'est un canal sérieux ici :-] [10:13:07] merci [10:18:49] j'arrete si vous me donnez les droits d'operateur du canal [10:19:15] a prendre ou a laisser... [10:20:57] alors ... ? [10:22:02] ca me va [10:22:23] nous sommes toujours à la recherche de volontaires pour nous aider à gérer le site :-) [10:22:53] ;-) [10:23:50] j'arrête de vous embêter alors :) [10:25:32] Kooo: n'hésite pas à éditer le site! [10:37:41] bonjour frakir [10:37:50] Salut Ko__ [12:39:55] Reedy, about https://bugzilla.wikimedia.org/show_bug.cgi?id=28026 – what about changing the default for new users now? [12:40:36] markers don't seem to have increased load (or anyway we could measure it now), so now we can see what's the impact of some more emails [13:58:48] gsdfhj [13:58:49] hdfhxfkijgdj [13:58:50] ggdxg,jdgljyk [13:58:51] jfd^kjhdhdf [13:58:51] fggf [13:58:52] k [13:58:52] hgvsqfgd [13:58:53] htk [13:58:53] hikgjbb [13:58:54] nvhk,d [13:58:54] cfh [13:58:54] dfh [13:58:55] xfb [13:58:56] cjkgjhgffdf [13:58:56] jfdt [13:58:56] h [13:58:57] dcvb [13:58:57] nh [13:58:58] gfbdcs [13:58:58] dcf [14:01:52] bghwgwdh [14:01:57] haha [14:02:22] right [14:02:27] Lsifjdihg: agree! [14:02:42] i don't spêak english ! [14:03:05] bah, this client doesnt autoban modes on kick, lame [14:07:28] +o me and I will ban 4 u :p [14:08:18] I Love Rock'n'roll yeah [14:09:16] i am the winner, winner tralala [14:35:15] * DarkoNeko blinks [15:26:19] css stuff seems to be having problems [15:27:43] (bits is giving 503 again) [15:29:51] something gone skewy with bits.wm? [15:30:10] Reedy: ^ [15:30:12] I am getting display issues for aswiki, barwiki (preferences) [15:30:36] and diff issues at enwikisource (and just started looking wider) [15:31:08] ugh [15:31:10] in meta too [15:31:10] thanks [15:31:10] sDrewth: Varnish seems to run wild, so JS and CSS wont get server properly atm... that's why... [15:31:19] that's a global problem, btw [15:31:37] Seems to be loading fine for me [15:31:44] sounds like some of our users :-/ [15:33:28] works again, as it seems [15:33:57] a few purges has me back up to speed too [15:34:08] ack, sporadic [15:34:34] https://as.wikipedia.org/wiki/%E0%A6%AC%E0%A6%BF%E0%A6%B6%E0%A7%87%E0%A6%B7:%E0%A6%B6%E0%A7%87%E0%A6%B9%E0%A6%A4%E0%A7%80%E0%A6%AF%E0%A6%BC%E0%A6%BE_%E0%A6%95%E0%A6%BE%E0%A6%AE is not playing nicely [15:35:37] one refresh out of 5 shows unstyled pages [15:36:07] beau, I am running higher than that, but won't quibble [15:38:54] and seeing it at meta [15:39:28] I can see it on pl.wikipedia, en.wikisource etc. [15:51:03] Was having trouble loading css [15:51:20] bits gave 503 [15:51:59] * apergos grits teeth [15:52:10] still happening for me as stated above [15:53:07] js as well [16:00:00] they all come from the same place [16:03:58] omg sylelessness [16:04:00] * ToAruShiroiNeko panics [16:05:56] Hello -tech… you are aware of bits throwing 503 errors instead of stylesheets? [16:06:23] where are you located? [16:06:44] VA/US [16:06:45] europe or outside of europe? [16:06:48] ok [16:06:50] outside of europe [16:07:08] * Beau_ is in Europe. [16:08:17] * Alchimista in europe too [16:10:11] outside of europe? huh [16:10:14] oh both [16:10:18] even better :-( [16:11:00] I saw a 503 error a few minutes ago. Now, there is no problem. [16:11:35] still heap big problem [16:11:40]

Error 503 Service Unavailable

[16:11:41]

Service Unavailable

[16:11:43]

Guru Meditation:

[16:11:44]

XID: 1426629465

[16:11:45]

[16:11:46]

Varnish cache server

[16:13:52] wait outside of europe = what continent though? [16:14:01] Beau_: Alchimista [16:14:11] apergos: NA [16:14:12] sorry, I mean livitup [16:14:17] thanks [16:14:21] ugh [16:14:50] apergos: i'm in europe, not outside :P [16:15:01] yeah, I said sorry :-P [16:15:28] working *better* now, but still failing 10% of requests [16:17:12] Bits error (JS/CSS not loading). Known issue? [16:17:26] guillom: yeah, for about 50 minutes ;) [16:17:33] ok [16:17:45] I just got online so I didn't notice before :) [16:18:53] guillom: france? [16:19:26] Beau_: has it been an uninterrupted issue for an hour? [16:19:26] jeremyb: yep [16:20:07] guillom: i'm not seeing it yet but supposedly it's effecting US+europe [16:20:20] i've also only been looking for a few mins [16:20:36] err, affecting* [16:21:38] jeremyb: HTTPS only? [16:21:51] guillom: i've no idea. i'm HTTPS only [16:22:20] guillom: i don't see much info on the issue. unless i'm missing something. just scroll up and you'll know as much as i do ;) [16:22:34] I'm not and got the problems as well (from Germany) [16:22:34] right :) [16:22:42] first mention seems to be 02 15:26:19 < Alchimista> css stuff seems to be having problems [16:22:47] UTC [16:23:22] It's weird, I get no CSS in FF (where I'm logged in and on HTTPS) but it works fine in Chromium (where I'm logged out, on either HTTP or HTTPS) [16:23:29] (even after force-reloading in FF) [16:24:39] guillom: it does not happen every time you refresh the page [16:24:48] HTTP here and fails about 20% of the time [16:25:53] I'm getting 503s again [16:26:06] Just got worse AFAIS [16:27:46] My skin keeps randomly changing to Nostalgia [16:28:22] omtsh: that's just without css [16:28:25] The skin isn't changing, it's just broking :P [16:28:48] broking... what :P * broken [16:29:30] What's up with the CSS :( [16:29:39] omtsh: bitss problems [16:29:42] *bits [16:33:18] ok does it work better now? [16:33:27] checking [16:33:51] it seems ok on ptwiki [16:34:07] looks ok [16:34:29] yup, ok [16:38:05] Seems okay now, thanks [16:51:59] !log Changing docroot/bits/skins-1.19 and other 1.19 symlinks to point to the 1.20wmf1 tree instead. This is needed because we're still getting requests for magnify-clip.png at the 1.19 URL from cached HTML [16:52:02] Logged the message, Mr. Obvious [17:14:01] !log catrope synchronized php-1.20wmf2/skins/vector/ 'Deploying 7260cc5fe4071e03241378ba1a48bc0b6f188948' [17:14:03] Logged the message, Master [17:28:50] !log preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test' [17:28:52] Logged the message, Master [17:30:57] !log preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test' [17:30:59] Logged the message, Master [17:36:26] !log aaron synchronized php-1.20wmf2/includes/specials/SpecialContributions.php 'Deployed 799998c3a160ef6dd3b926b7d6fec223682b788c' [17:36:28] Logged the message, Master [17:37:16] preilly: my git pull also picked up some zerorated stuff fyi, not synced though [17:39:02] preilly: though it could mean you thought you pulled stuff you didn't [17:39:43] PHP Warning: in_array() expects parameter 2 to be array, string given in /usr/local/apache/common-local/php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php on line 156 [17:40:20] not related probably, but in the logs nonetheless [17:41:15] o hai AaronSchulz [17:46:06] !log updated production civicrm to r1726 [17:46:08] Logged the message, Master [17:46:47] preilly: you want that in wmf2? [17:47:52] seems ok [17:56:48] !log preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test' [17:56:51] Logged the message, Master [17:57:11] !log preilly synchronized php-1.20wmf2/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test' [17:57:14] Logged the message, Master [18:09:19] !log preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI' [18:09:21] Logged the message, Master [18:15:27] robla1: enwikinews? [18:15:47] sure [18:15:49] maybe we should make a script that uses a dblist and mt_rand() to index into the array to pick a wiki [18:16:07] I'd prefer a physical wheel we spin [18:16:10] * AaronSchulz puts that at the top of his todo list [18:16:15] maybe hook it up via arduino [18:17:07] * robla files a bug [18:18:06] AaronSchulz if you have time could you tell me if cascaded protection is something you worked on? [18:18:06] https://bugzilla.wikimedia.org/show_bug.cgi?id=22521 [18:18:38] I've touched a lot of stuff, but I think werdna is the main guy there [18:19:17] OK, I'll ask werdna then [18:20:44] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2 [18:20:46] Logged the message, Master [18:22:56] preilly: did you sync that fix yet? [18:23:03] I still see those warnings [18:25:53] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikinews to 1.20wmf2 [18:25:55] Logged the message, Master [18:26:08] there we go :) [18:31:03] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: enwikisource to 1.20wmf2 [18:31:06] Logged the message, Master [18:39:21] AaronSchulz: *.wikinews.org now? [18:44:35] index.php?title=Specjalna:BannerController&cache=/cn.js&303-4() [18:44:37] ops [18:44:42] umm... has someone upgraded centralnotice ? on pl.wikipedia on Firefox I get: jQuery is not defined [18:44:58] hrm...nope [18:44:59] :O [18:45:04] That's ... extremely concernign [18:45:32] Beau_: I don't see that [18:45:40] But only on Firefox, maybe it is related to RL asynchronousnes... [18:45:40] Beau_: Do you see this if you're logged out too? [18:45:48] RoanKattouw: I am logged out [18:46:07] no scaping now [18:46:16] I am too and I'm not getting the error [18:46:38] does central notice uses geolocation to display banners? [18:46:40] Beau_: Do you have Firebug turned on? Do you see red things in the Net panel? [18:46:47] I guess that's possible [18:47:23] But "jQuery is not defined" is an extremely scary error, unless it's caused by jQuery not being delivered, which you would see as a bad error code on the net panel [18:48:09] 200 OK [18:48:37] I would love to be able to reproduce this but I can't [18:48:51] Also it can't be RL async's fault, because RL itself depends on jQuery [18:49:46] The above 852 wiki DBs are missing wikiversion rows. [18:49:47] Re-configured 0 wiki(s) from php-1.20wmf1 to php-1.20wmf2. [18:49:49] RoanKattouw: lol [18:49:55] * AaronSchulz makes a newline fix [18:50:08] RoanKattouw: I will try with a new firefox profile [18:50:13] OK [18:50:47] If you get it again, please show me Firebug's net panel [18:50:55] (as a screenshot I mean) [18:51:58] !log asher synchronized wmf-config/db.php 'added ES cluster23 to templateOverridesByCluster but not activating' [18:52:00] Logged the message, Master [18:52:17] AaronSchulz: yes [18:53:05] RoanKattouw: sorry for the false alarm, it seems some extension was gettin in the way, on clean profile all works [18:53:15] OK [18:57:12] * robla hovers over AaronSchulz's shoulder, and finds out he's testing some stuff before doing the *.wikinews.org deploy [18:58:56] PHP Notice: Undefined index: X-Carrier in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2347 [18:59:00] * AaronSchulz blames preilly :) [18:59:12] AaronSchulz: yeap [19:00:23] !log preilly synchronized wmf-config/CommonSettings.php 'only remove images for DIGI' [19:00:24] AaronSchulz: - if ( $_SERVER['X-Carrier'] == 'DIGI' ) { [19:00:25] + if ( isset( $_SERVER['X-Carrier'] ) && $_SERVER['X-Carrier'] == 'DIGI' ) { [19:00:25] Logged the message, Master [19:03:21] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikinews sites to 1.20wmf2 [19:03:24] Logged the message, Master [19:04:00] robla: good to know that the "each wiki has a version" sanity check works [19:04:29] yay for process improvement! :) [19:04:46] when you get "The above 852 wiki DBs are missing wikiversion rows." you know something is a little off [19:08:30] all of the homepages load. Special:Version spot checked on a few [19:08:34] looks good [19:08:40] onward to *.wikisource.org [19:11:20] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikisource sites to 1.20wmf2 [19:11:23] Logged the message, Master [19:13:02] it seems oldwikisource was left out again [19:14:23] interesting. AaronSchulz, I take it oldwikisource must not be in the wikisource.dblist [19:14:54] nope [19:18:02] funny, it's in wikipedia.dblist [19:18:26] that's part of a lot of weirdnesses it has [19:19:01] for instance "interwikis" pointed to Wikipedia until recently [19:19:09] and there's still https://bugzilla.wikimedia.org/show_bug.cgi?id=32189 btw [19:20:48] !log preilly synchronized php-1.20wmf1/extensions/ZeroRatedMobileAccess/ 'zero weekly carrier test' [19:20:50] Logged the message, Master [19:21:30] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved sourceswiki to 1.20wmf2 [19:21:32] Logged the message, Master [19:23:56] alright enwikibooks, then wikiquote, wikiversity... [19:25:15] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved enwikibooks to 1.20wmf2 [19:25:18] Logged the message, Master [19:28:57] * AaronSchulz sees some failed NFS thumb deletes [19:31:16] AaronSchulz: deployment related or something else? [19:31:23] wikibooks looks fine to me [19:31:37] enwikibooks that is. AaronSchulz: *.wikibooks now? [19:31:38] errors only on commonswiki [19:32:05] I wonder if a bot is moving something [19:34:46] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved all wikibooks to 1.20wmf2 [19:34:48] Logged the message, Master [19:35:45] !log preilly synchronized php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array' [19:35:48] Logged the message, Master [19:36:05] !log preilly synchronized php-1.20wmf2/extensions/MobileFrontend/skins/SkinMobile.php 'fix php notice for in_array' [19:36:07] Logged the message, Master [19:36:26] http://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=move&user=TUBS&page=&year=&month=-1&tagfilter= [19:36:30] just a bunch of moves [19:36:34] !log fix for PHP Warning: in_array() expects parameter 2 to be array, string given in /usr/local/apache/common-local/php-1.20wmf1/extensions/MobileFrontend/skins/SkinMobile.php on line 156 [19:36:37] Logged the message, Master [19:36:38] sad that NFS is that fragile [19:38:54] werdna: PHP Fatal error: [] operator not supported for strings in /usr/local/apache/common-local/php-1.20wmf1/extensions/GlobalBlocking/GlobalBlocking.class.php on line 20 [19:39:42] PHP Fatal error: Allowed memory size of 125829120 bytes exhausted (tried to allocate 4730216 bytes) in /usr/local/apache/common-local/php-1.20wmf1/includes/ChangesList.php on line 1189 [19:40:45] commons coming up next... [19:40:45] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: commonswiki to 1.20wmf2 [19:40:48] Logged the message, Master [19:40:51] speak of the devil [19:45:46] preilly: the GlobalBlocking one has been around for like 5 versions ;) [19:46:02] AaronSchulz: can't that be fixed? [19:46:44] AaronSchulz: meta now? [19:46:45] it is a distinct theoretical possibility, yes [19:47:48] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: metawiki to 1.20wmf2 [19:47:50] Logged the message, Master [19:51:22] rsync: write failed on "/usr/local/apache/common-local/wikiversions.dat": No space left on device (28) [19:51:26] RoanKattouw: srv 219 & 224 [19:51:31] though df -h looks fine [19:51:34] on a/ [19:51:56] Well /a sure [19:51:58] Looking [19:52:11] Hah and / has space [19:52:30] It looks like it should work [19:52:31] all others partitions too (219 at least) [19:52:36] May be transient /tmp stuff [19:52:53] looking [19:52:53] who's stuff, rsyncs? for two files? [19:53:35] I bet it's /tmp all right [19:53:42] same filesystem. [19:53:55] we clear out old files every 5 mins I think so ... [19:55:07] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: [19:55:10] Logged the message, Master [19:55:16] yeah, transient [19:59:59] !log asher synchronized wmf-config/db.php 'adding dbs 52,53,57 to s2 at lower weights' [20:00:01] Logged the message, Master [20:04:33] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved special, wikimedia, wikiquote, wikiversity, and wiktionary wikis to 1.20wmf2 [20:04:35] Logged the message, Master [20:06:05] robla: maybe it's http://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=move&user=TUBS&page=&year=&month=-1&tagfilter= :) [20:06:30] new thumbs involve temp files, which may be filling up space for brief moments, hard to say for sure [20:07:52] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: [20:07:55] Logged the message, Master [20:10:14] for example srv220 is full right this second [20:10:21] same with 219 [20:10:27] steady at 100% [20:10:38] I just cleared mw-cache-1.19 off srv219 [20:10:42] srv220: /dev/sda1 7.4G 7.0G 0 100% / [20:10:42] not completely full but full enough, if you get my drift [20:10:42] Unfortunately that was only 18M [20:10:46] yeah [20:10:49] The scalers badly need more disk space [20:11:00] RoanKattouw: did you do 220? [20:11:01] we need to redo those partitions generally [20:11:27] and now they are all good again [20:11:30] for a few minutes [20:12:19] !log aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Re-pushing for srv219 and srv220 [20:12:22] Logged the message, Master [20:12:23] ok, done [20:16:02] did my intarweb connection survive my laptop move? [20:16:03] :) [20:18:05] yes! [20:24:53] !log hashar: updated TestSwarm to distribute tests to Firefox 12 users. [20:24:56] Logged the message, Master [20:25:32] is blue background in chrome the zoom bug? or i'm confusing with something else? [20:25:48] I just tried to create an account and got a "Database error" page with the following technical details: [20:25:49] Last attempted database query: (SQL query hidden) [20:25:51] Function: User::saveOptions [20:25:52] MySQL error: 1213: Deadlock found when trying to get lock; try restarting transaction (10.0.6.48) [20:26:37] did you repeat? [20:26:43] @info 10.0.6.48 [20:26:44] jeremyb: Unknown identifier (10.0.6.48) [20:26:56] I did not yet but I can [20:27:35] @info db38 [20:27:35] jeremyb: [db38: s1] 10.0.6.48 [20:27:40] PHP Fatal error: Invalid host name (wikipedia), can't determine language.#012 in /usr/local/apache/common-local/multiversion/MWMultiVersion.php on line 342 [20:27:44] hrm, bad urls? [20:27:52] Hmm...this time I got a big red box saying "Login error The user account was not created, as we could not confirm its source. Ensure you have cookies enabled, reload this page and try again." [20:28:11] and again? ;) [20:29:23] (that's the master btw. but it should be because this is a write) [20:30:18] Huh...this time "The requested username is already taken in the unified login system. Please choose a different name."...which is somewhat scary seeing as I'm trying to create Ks0spy [20:32:03] https://en.wikipedia.org/wiki/Special:CentralAuth/Ks0spy [20:32:32] O.o It's not in any of the user creation logs... [20:32:42] Or wasn't when I checked a minute ago [20:32:58] i have to assume that centralauth succeeded and then enwikip failed [20:33:11] and that left you with a centralauth [20:33:27] Hmm...so what do I do now? [20:33:34] you could try just logging in to the account (and not creating a new one) [20:33:59] @info centralauth [20:34:00] jeremyb: [centralauth: s7] db37: 10.0.6.47, db16: 10.0.6.26, db56: 10.0.6.66, db26: 10.0.6.36 [20:34:34] or try creating one since the centralauth seems to have fixed itself? [20:35:46] It let me log in but it's still not showing up in the user creation log [20:37:44] i don't see how that's possible... [20:37:52] it's saying there's no global account [20:38:02] what wiki were you on? [20:38:03] [06:25] is blue background in chrome the zoom bug? p858snake|l: danke [20:38:43] jeremyb: The https:// English Wikipedia [20:40:05] hrmmm, you do seem to exist... [20:40:23] this should prolly get a look in the db [20:42:02] I was logged in to Ks0stm when creating the account and it's not in https://en.wikipedia.org/w/index.php?title=Special%3ALog&type=newusers&user=Ks0stm&hide_patrol_log=1&hide_review_log=1 nor is it showing up at http://en.wikipedia.org/w/index.php?title=Special%3ALog&type=newusers&user=&page=User%3AKs0spy&year=&month=-1&tagfilter= [20:45:20] jeremyb: I also notice it's not allowing me to log in to one account on the https version and a different one on the http version anymore [20:45:31]