[00:50:14] is bugzilla down? [00:53:13] apparently, gives TimStarling a little nudge [00:54:44] !log kaulen is not responding on ssh, web down, rebooting [00:54:46] Logged the message, Master [00:57:14] https://en.wikipedia.org/wiki/User_talk:68.59.212.62 [00:57:33] Entries in the page table without corresponding revisions. Suppression? [00:59:34] !log after kaulen came back up, it was immediately overloaded with jsonrpc.cgi. Stopped apache. [00:59:36] Logged the message, Master [01:07:22] Good night! [01:15:11] !log started apache with MaxClients=30 [01:15:15] Logged the message, Master [01:17:19] someone is DDoSing us from site JS on wikipedia [01:18:08] :( [01:18:13] <[Haekchen]> Bugzilla seems to be down; or is it just me? [01:20:06] why is the javascript with debug=true completely useless and different to the javascript with debug=false? [01:24:26] [Haekchen]: not just you [01:24:27] <[Haekchen]> TimStarling: The debug js will load each script module separately to debug syntax errors. Why do you think that is useless? [01:24:37] I have the same problem [01:37:20] is Edokter on IRC? [01:37:28] User:Edokter [01:38:17] well, "Edokter" isn't a registered nick [01:39:54] TimStarling: I don't think he uses IRC. [01:40:04] He's rpetty quick to respond to talk page messages if he's around, though. [01:40:08] Also, pretty. [01:42:23] someone should probably tell him to lie low for a few days [01:43:01] because this is taking a lot of time to fix [01:45:43] anyway it's up at https://bugzilla.wikimedia.org:444/ now [01:46:16] but not port 80? [01:46:24] ok [01:46:48] !log temporarily moved bugzilla to port 444 until the connection flood (~1k req/s) subsides [01:46:49] Logged the message, Master [01:47:14] I can probably fix the redirect from port 80 [01:48:57] That gadget is already used enough to cause overload? [01:50:35] he enabled it by default [01:50:48] so yeah, 300M users is enough to cause an overload [01:50:56] Ah, nice. [01:55:41] http://ganglia.wikimedia.org/latest/graph.php?r=hour&z=large&c=Miscellaneous+pmtpa&h=kaulen.wikimedia.org&v=358.60&m=pkts_in&jr=&js=&vl=packets%2Fsec&ti=Packets+Received [02:14:04] does anyone here know how to configure varnish? [02:17:50] !log LocalisationUpdate completed (1.18) at Mon Feb 13 02:17:50 UTC 2012 [02:17:52] Logged the message, Master [02:21:09] !log on kaulen: restored MaxClients [02:21:10] Logged the message, Master [02:23:47] !log bugzilla is mostly working now, although it's very slow. The DDoS requests are blocked after connection setup using [02:23:49] Logged the message, Master [02:32:42] !log on kaulen: increased MaxClients to 500 to better deal with the connection flood [02:32:43] Logged the message, Master [02:34:18] Bugzilla seems to be working on port 80 [02:34:53] Oh, that's http not https [02:35:18] I and other users (from Germany and Austria) experienced at least two times only half or 15% displaying thumbnails (specifically the version on the file page in commons). [02:35:19] I (Germany) and someone else (Austria) just had it at https://commons.wikimedia.org/wiki/File:SYDNEY_AIRPORT_AMO_2011_01January.png while a third use from Germany saw it correctly - a purge fixed it. [02:36:16] all users accessed via https [02:36:24] I'm going for lunch now [02:36:29] TimStarling, did you put bugzilla back to port 443 [02:36:35] yes [02:59:17] TimStarling: Could the BZ issues be search engine spiders? although would of probably happened earlier if that was the case [03:54:12] TimStarling: Could the BZ issues be search engine spiders? [03:54:42] no, the BZ issues were due to this: https://en.wikipedia.org/w/index.php?title=MediaWiki:Gadgets-definition&diff=476551858&oldid=472069181 [08:35:21] zz =_= [08:59:58] Ryan_Lane: unable to login on wmflabs [09:00:11] Failed opening required '/usr/local/apache/common/live/extensions/MobileFrontend/MobileFrontend.php' [10:24:06] It seems that there has been a problem with the image scalers. I noticed this since the Extension:Collection (~ PDF export) lately reports problems with corrupt images. Check for instance: http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Russia_China_Locator.png/1200px-Russia_China_Locator.png The images gets scaled correctly now (e.g. when changing the width from 1200px to 1199px: http://upload.wikimedia.org/wikipedia/commons/thumb/a/af [10:24:06] /Russia_China_Locator.png/1199px-Russia_China_Locator.png). The problem right now is that there seem to be a lot of corrupt images that are cached. Is this a know problem? Can the cache be invalidated? [10:25:24] the second link got split. here's the correct version: http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Russia_China_Locator.png/1199px-Russia_China_Locator.png [10:56:17] It seems that there has been a problem with the image scalers. I noticed this since the Extension:Collection (~ PDF export) lately reports problems with corrupt images. Check for instance: http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Russia_China_Locator.png/1200px-Russia_China_Locator.png The images gets scaled correctly now [10:56:23] (e.g. when changing the width from 1200px to http://upload.wikimedia.org/wikipedia/commons/thumb/a/af/Russia_China_Locator.png/1199px-Russia_China_Locator.png). The problem right now is that there seem to be a lot of corrupt images that are cached. Is this a know problem? Can the cache be invalidated? [11:48:21] Failed to load resource: the server responded with a status of 403 (Forbidden) [11:48:25] bugzilla [11:48:38] <^demon> WFM. [11:50:09] ^demon: https://bugzilla.wikimedia.org/34366 [11:50:57] <^demon> Ah, hrm. [11:52:58] Tim-away probably did that earlier [11:53:03] when en.wiki was DDoSing it [11:53:29] <^demon> Yeah, should've read my e-mail :) [11:54:13] didn't got that mail I think [12:01:23] New patchset: Mark Bergsma; "Cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2531 [12:01:48] New patchset: Mark Bergsma; "Move monitor_service to the service class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2532 [12:02:09] New patchset: Mark Bergsma; "Pull LVS service IPs from lvs.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2533 [12:02:30] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2531 [12:02:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2531 [12:02:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2532 [12:02:31] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2532 [12:02:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2532 [12:02:46] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2533 [12:02:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2533 [12:06:57] New patchset: Mark Bergsma; "Pull in LVS service IPs from lvs.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2534 [12:08:09] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2534 [12:08:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2534 [12:17:19] New patchset: Mark Bergsma; "Scoped variable lookups" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2535 [12:17:41] New patchset: Mark Bergsma; "Make varnish_xff_sources a class parameter" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2536 [12:18:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2535 [12:18:01] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2535 [12:18:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2535 [12:18:34] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2536 [12:18:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2536 [12:24:44] New patchset: Mark Bergsma; "Use class parameters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2537 [12:25:15] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2537 [12:25:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2537 [12:55:10] New review: Demon; "Are we going to be doing python things in jenkins?" [integration/jenkins] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2289 [13:28:01] New patchset: Mark Bergsma; "Use an upstart manifest for managing varnish udp loggers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2538 [13:29:07] New patchset: Mark Bergsma; "Use an upstart manifest for managing varnish udp loggers" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2538 [13:30:25] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2538 [13:30:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2538 [13:32:53] New patchset: Mark Bergsma; "Move out upstart_job dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2539 [13:33:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2539 [13:33:43] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2539 [13:38:31] New patchset: Mark Bergsma; "Using a service type causes conflicts, try exec instead" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2540 [13:39:02] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2540 [13:39:02] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2540 [13:40:26] New patchset: Mark Bergsma; "Fix dependency" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2541 [13:40:58] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2541 [13:40:59] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2541 [13:44:23] !log reedy synchronized php-1.19/extensions/MobileFrontend [13:44:25] Logged the message, Master [13:48:17] New patchset: Mark Bergsma; "Don't start if already running" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2542 [13:49:16] New patchset: Mark Bergsma; "Don't start if already running" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2542 [13:49:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2542 [13:49:47] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2542 [13:49:48] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2542 [13:50:13] New patchset: Hashar; "gitreview file" [operations/software] (master) - https://gerrit.wikimedia.org/r/2543 [13:52:03] New patchset: Mark Bergsma; "su forks" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2544 [13:52:34] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2544 [13:52:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2544 [14:00:58] New patchset: Mark Bergsma; "Make varnish_instance default undefined, never mind pid files" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2545 [14:02:09] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2545 [14:02:09] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2545 [14:04:13] New review: Hashar; "I have looked at the jenkins gerrit plugin, patchsets are polled by jenkins so we do need a "patchse..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/2495 [14:05:28] New review: Demon; "(no comment)" [operations/software] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2543 [14:05:28] Change merged: Demon; [operations/software] (master) - https://gerrit.wikimedia.org/r/2543 [14:06:13] New patchset: Demon; "Chad's the best" [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2546 [14:13:17] Change abandoned: Demon; "Was just testing git-review" [test/mediawiki/core2] (master) - https://gerrit.wikimedia.org/r/2546 [14:15:56] New patchset: Mark Bergsma; "Fix su invocation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2547 [14:16:26] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2547 [14:16:27] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2547 [14:19:54] New patchset: Mark Bergsma; "Fix status command" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2548 [14:20:26] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2548 [14:20:27] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2548 [14:22:17] New patchset: Mark Bergsma; "Specify path" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2549 [14:22:49] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2549 [14:22:50] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2549 [14:26:34] New patchset: Mark Bergsma; "Remove status" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2550 [14:26:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2550 [14:27:06] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2550 [14:27:06] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2550 [14:27:25] !log reedy synchronized 1.17.dblist '1.17-phase1.dblist 1.17-phase2.dblist all.dblist big.dblist closed.dblist deleted.dblist fishbowl.dblist flaggedrevs.dblist news.dblist new_wiktionaries.dblist pmtpa.dblist pmtpa-dump1.dblist pmtpa-dump2.dblist pmtpa-dump3.dblist private.dblist readonly.dblist s1.dblist s2.dblist s2-fixed.dblist s3.dblist s3-fixed.dblist s4.dblist s5.dblist s6.dblist s7.dblist small.dblist special.dblist switchover-ju [14:27:27] Logged the message, Master [14:27:37] Well that's a fucking lie [14:28:08] !log reedy synchronized all.dblist 'Fix double bewikimedia' [14:28:10] Logged the message, Master [14:28:30] !log reedy synchronized pmtpa.dblist 'Fix double bewikimedia' [14:28:31] Logged the message, Master [14:28:47] !log reedy synchronized s3.dblist 'Fix double bewikimedia' [14:28:49] Logged the message, Master [14:29:06] !log reedy synchronized wikimedia.dblist 'Fix double bewikimedia' [14:29:08] Logged the message, Master [14:31:07] New patchset: Hashar; "Force Jenkins request through HTTPS" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2551 [14:45:22] New review: Demon; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2551 [14:59:39] hyexmode: now we have milestones, maybe we can close the 1.19 tracking bug https://bugzilla.wikimedia.org/show_bug.cgi?id=29097 :) [15:01:55] help [15:03:04] !log reedy synchronizing Wikimedia installation... : Reverting roans live hacks for bug 31576 [15:03:06] Logged the message, Master [15:04:17] anybody home? [15:05:58] John_plaut: what is happening? [15:06:25] hi [15:06:28] sync done. [15:06:46] hashar a Q please [15:07:41] * hashar gives a Q to John_plaut [15:07:45] here you have :-] [15:07:51] have being working with a bot since hours...and I get "pausing 300 seconds [15:08:07] due to DB server lag" [15:08:16] @replag [15:08:27] !replag [15:08:27] Deadbot is dead [15:08:33] :-[ [15:08:40] John_plaut: which wiki? [15:08:45] es [15:08:48] http://noc.wikimedia.org/dbtree/ [15:08:49] http://toolserver.org/~bryan/stats/replag/ [15:08:55] eswiki? [15:08:58] yup [15:09:05] 27 [15:09:06] *s7 [15:09:20] ah that is on the main cluster? [15:09:21] db18 is 1147.60 lagged [15:09:24] That's gonna be the issue [15:11:00] !log reedy synchronized php-1.18//includes/ 'Bringing across 1.18wmf1 livehacks' [15:11:02] Logged the message, Master [15:12:01] Ah [15:12:12] hashar: looks like asher is running rev_sha1 patch on the host [15:12:46] so it was put out of replication? [15:13:06] No, I don't think so [15:13:15] Still seems to be replicating, I think it's just acklogged [15:13:21] or is that the sha1 inserts flood that makes it lags out? [15:13:26] btw [15:13:32] Hey Ariel [15:13:34] New patchset: Mark Bergsma; "Upstart cookbook recommends using start-stop-daemon" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2552 [15:13:50] while that is going around, it seems that it doesn't quite do all revs [15:14:02] ? [15:14:06] if there are revs with text still in the archive table (instead of in the text table) [15:14:11] then it doesn't get those, [15:14:12] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2552 [15:14:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2552 [15:14:26] this was reported by Christian (contractor working on xml dumps test suite) [15:14:33] Isn't there a seperate ar_sha1 patch to come? [15:14:59] I don't know about patches, this was running the populateRevisionSha1.php script [15:14:59] yeah, there is... [15:15:07] Oh [15:15:08] which in theory does both [15:15:10] right? [15:15:11] Tell Aaron off then [15:15:19] ok, I told him to get in touch [15:15:28] if I see Aaron I'll poke him about it [15:15:41] yeah... Code looks like it should do both [15:17:02] Any idea if we can take db18 out of rotation for the time being? [15:17:40] Might just be worth logging a bug and assigning it to Aaron [15:17:51] As there may be a point was want to run that on the cluster (that'll be much fun) [15:19:38] Hmmm [15:19:39] Lag is down to 108 now [15:19:43] I think Christian is goingto do that (bugzilla it) [15:20:06] we should spot check after the ar_sha1 patch goes and make sure all rev texts actually have a sha1 [15:20:07] Fair enough [15:20:27] if it's dropping then meh [15:20:32] might as well leave it in [15:20:50] I was thinking that [15:20:59] It's gone from 1500 to 100 in the last few minutes [15:21:03] ok [15:25:35] New patchset: Mark Bergsma; "With start-stop-daemon, varnishncsa needs to fork" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2553 [15:26:16] New patchset: Mark Bergsma; "With start-stop-daemon, varnishncsa needs to fork" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2553 [15:26:53] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2553 [15:26:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2553 [15:31:23] New patchset: Mark Bergsma; "Try expect fork" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2554 [15:32:09] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2554 [15:32:10] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2554 [15:52:50] New patchset: Mark Bergsma; "Comment for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2555 [15:53:26] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2555 [15:53:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2555 [16:19:07] !log reedy synchronized php-1.19/extensions/CategoryTree/ 'r111382' [16:19:09] Logged the message, Master [16:19:59] !log reedy synchronized php-1.19/includes/ 'r111382' [16:20:01] Logged the message, Master [16:28:02] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2551 [16:28:02] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2551 [16:30:59] !log reedy synchronized php-1.19/extensions/PagedTiffHandler/ 'r111385' [16:31:01] Logged the message, Master [16:31:36] !log reedy synchronized php-1.19/extensions/OggHandler/ 'r111385' [16:31:37] Logged the message, Master [16:32:04] New review: Dzahn; "confirmed. jenkins on http://integration.mediawiki.org/ci/ now forces HTTPS" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2551 [17:12:02] !log mailman: deleting test-list [17:12:04] Logged the message, Master [17:37:28] !log reedy synchronizing Wikimedia installation... : Rebuilt trusted-xff.cdb [17:37:30] Logged the message, Master [17:40:01] sync done. [19:30:19] pictures not fully shown, anyone any idea why? http://commons.wikimedia.org/wiki/File:Maria_Cristina_Falls_Iligan_City.jpg [19:30:25] http://commons.wikimedia.org/wiki/File:Mimbalut_Falls_Iligan_City_01.JPG [19:34:40] hrm, looks like a resizer error ? maplebed have any ideas? ^^ [19:34:54] hmm. [19:35:14] for me it looks like with the 450px version i only see the top 1/5th or so of the image [19:36:06] I asked the resizers to make a new version and it looks ok. My suggestion: purge the cache of the image to force recreation of all the thumbnails. [19:36:10] creating a new image looks good though (451) [19:36:10] hehe [19:36:18] maplebed: want to show me how to do that ? [19:36:25] sure! [19:38:39] AaronSchulz: ?action=purge is the right magic to purge an image from commons, right? [19:38:47] eg commons.wikimedia.org/wiki/File:Mimbalut_Falls_Iligan_City_01.JPG?action=purge [19:38:55] yes [19:39:02] try it. no dice. [19:39:09] no cigars either? [19:39:10] I'm even logged in. [19:39:25] good thing too. smoke in the office? [19:39:53] weird, for unlogged in it gave me a confirmation page, but still did not actually purge [19:40:06] confirmation for anons is normal [19:40:20] and if you're logged in it just happens (with no user feedback)? [19:40:33] yes [19:40:50] lame. [19:40:59] http://upload.wikimedia.org/wikipedia/commons/thumb/e/e4/Mimbalut_Falls_Iligan_City_01.JPG/90px-Mimbalut_Falls_Iligan_City_01.JPG [19:41:00] <^demon> It doesn't require confirmation on POST, or did that change? [19:41:01] Unexpected error HTTP Error 502: Bad Gateway [19:41:31] weird, because for creating 451px it took a while but worked [19:43:29] I need some sysadmin to fix a bug ... [19:43:49] axpde: explain bug plz, outcome and steps to reproduce [19:44:19] I wanted to do some steward action on ru.wikibooks and uz.wikiquote [19:44:26] ^demon: it does, but I fixed that in 1.19 ;) [19:44:34] that was a regression [19:44:44] but both server didn't accept my login [19:44:56] can you open up a bug in bugzilla ? [19:45:07] on ru.wikibooks I was able to request a temp password [19:45:35] and so I was able to log in and manually unify this account to my SUL [19:45:53] but on uz.wikiquote I get the message: "no email address" [19:46:18] LeslieCarr I have a bugzilla account [19:48:05] but I'm not sure whether this bugf is reproducible [19:49:03] LeslieCarr do you think this needs to be taken to bugzilla? [19:49:17] believe so [19:50:10] can you (or someone else) check the database entry of my account? [19:51:05] axpde: the password is not stored in clear text and is differend on each wiki [19:51:19] different? [19:51:44] But I thought it's the same on every wiki I come accross! [19:51:55] axpde: sorry: The data is stored differend, not the pasword itself [19:52:14] all I need is an email address entry [19:52:24] axpde, are you sure it's not an input method thing? [19:52:34] on uz.wikiquote to have a temp password sent to me [19:53:00] Nemo_bis: do speak of a misstyped password? [19:53:15] or whatever else you supposedly typed in [19:55:26] but I didn't type anything! [19:55:43] As usual I went to some page on uz.wikiquote [19:56:36] usually when visiting a page on a wiki I haven't been before, the local account is set up and all I need to do is set the local language to german [19:57:01] This has happened to me before, but can't remember what I was doing wrong [19:57:08] I don't need to do type any password, my SUL account makes it unneccessary [19:57:22] neither can I dp [19:57:24] neither can I do [19:58:05] As said before, I did the same I always do when visiting a (wmf) wiki I haven't been before [19:59:11] LeslieCarr or DaBPunkt is there anyone able to fix the local account settings and add an email address? [19:59:29] axpde: every server-guy with db-access [19:59:36] sadly not me :( [20:00:11] DaBPunkt ... but where are those server-guys when needed ?-) [20:01:09] New review: Hashar; "Thanks for the merge. Jenkins has an other issue though, its cookie is not secure so browsing to ht..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2551 [20:20:40] ^demon: have you rebuild the git repo this afternoon? [20:20:47] the mediawiki/core2 one I mean [20:20:51] <^demon> Yeah I just rebuild core2. [20:20:57] <^demon> Best to probably rm and re-clone [20:21:26] git fetch is redownloading everything :-) [20:21:57] <^demon> I figured it might :) [20:22:31] New patchset: Lcarr; "Ensuring tcp setting tweaks are on all serving platforms" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2556 [20:23:00] <^demon> hashar: What I want to try now is a git commit that doesn't happen in svn. [20:23:06] <^demon> Then try to pull stuff from svn again [20:23:09] <^demon> And see how it behaves. [20:23:38] conflicts! [20:23:50] then you will get a git and a svn forks [20:24:24] <^demon> I figured :\ [20:24:45] aren't we switching this week? [20:25:16] <^demon> Heck, I could switch *right now* [20:25:22] go ahead [20:25:30] <^demon> People will complain :p [20:25:31] and make an announcement in our weekly conf call :D [20:25:58] well it is not like the mediawiki / developer community is going to make any more news than the enwiki community [20:26:05] so I guess it is safe to annoy them a bit [20:26:35] more seriously, we need a new way to deploy changes too [20:29:05] ^demon: the core2 repo is more than 350MB now :/ [20:29:17] <^demon> No, it's ~100M for me. [20:29:23] <^demon> Try repacking it. [20:29:25] $ du -sk core{,2}/.git [20:29:26] 137908 core/.git [20:29:26] 336432 core2/.git [20:29:50] <^demon> `git repack -a -d -f --depth=250 --window=250` [20:30:11] aren't you repacking it server side already? [20:31:20] ahh got it [20:31:23] I have two copies [20:31:38] New patchset: Pyoungmeister; "making new lucene classes and configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2557 [20:35:16] New patchset: Pyoungmeister; "making new lucene classes and configs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2557 [20:38:00] ^demon could you look at https://bugzilla.wikimedia.org/show_bug.cgi?id=34381 [20:38:23] or hashar? [20:39:05] * axpde sighs deeply. [20:40:08] looks like there is an axpde username on uz.wikiquote.org :) [20:40:21] yep, but look at the date [20:40:38] It's logged today! [20:40:49] 13.2.2012, 18:33 [20:42:25] Any idea what happened this evening, hashar? [21:06:50] New patchset: Demon; "Adding .gitreview" [mediawiki/extensions/Translate] (master) - https://gerrit.wikimedia.org/r/2558 [21:07:20] New review: Demon; "(no comment)" [mediawiki/extensions/Translate] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2558 [21:07:20] Change merged: Demon; [mediawiki/extensions/Translate] (master) - https://gerrit.wikimedia.org/r/2558 [21:08:47] http://noc.wikimedia.org/cgi-bin/ng/report.py?db=thumb [21:09:01] maplebed: why so much ddjvu? Is that normal? [21:09:23] AaronSchulz: any chance you could avoid jumping from channel to channel? [21:10:04] :P [21:10:08] oh, I guess i did ask you here earlier. [21:10:19] grumblegrumblegrumble. [21:10:22] I was looking that backscroll, likes like its in ops now [21:10:22] * axpde thinks channel hopping makes things more complicated. [21:10:35] see, this is why I hate being in a million channels [21:11:00] it's not bad with a decent client. [21:38:51] hexmode: will you be available to refresh the 1.19 Revision report? [21:40:08] hashar: yes, but it looks like db on toolserver is slow right now so the report is timing out [21:40:19] oohh [21:41:21] hexmode: it is not that much important anyway [21:53:44] hi TimStarling do you have a few minutes to talk about the Messages*.php discussion from last week? I have a few questions about discrepancies between 1.18 and 1.19 [21:54:58] hashar: it is updated [22:01:24] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2557 [22:01:25] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2557 [22:01:28] hexmode: thanks [22:09:37] Ryan_Lane: still recovering from the trip? [22:10:23] yep. jet lag is making my brain worthless [22:10:39] :( [22:15:04] New patchset: Diederik; "Check if country codes are valid and finish renaming stuff" [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2559 [22:16:37] New patchset: Diederik; "Adding ip-filtering support (not working)" [analytics/udp-filters] (refactoring) - https://gerrit.wikimedia.org/r/2560 [22:20:18] !log tstarling synchronized wmf-config/InitialiseSettings.php 'disabling the collection extension due to image scaler overload' [22:20:20] Logged the message, Master [22:29:26] !log on pdf1: killed a convert process that had been running since Jan 6 [22:29:28] Logged the message, Master [22:31:54] :o [22:33:45] !log reedy synchronized php-1.19/includes/api/ApiWatch.php 'r111422' [22:33:46] Logged the message, Master [22:43:34] New patchset: Tim Starling; "Increasing MaxClients slightly on the image scalers. They can probably handle a lot more than this without risking OOM, but it's not clear whether ms5 can handle the extra load." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2563 [22:44:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2563 [22:44:17] New review: Tim Starling; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2563 [22:44:18] Change merged: Tim Starling; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2563 [22:45:08] Is the API unusually overloaded or something? [22:45:14] I've been getting lag e-mails from cron all day. [22:45:38] Joan: From TS? That one got it's own problems... [22:45:51] Yeah, the scripts run on the Toolserver. [22:46:02] "Unable to run job: error: no suitable queues. [22:46:02] Exiting." [22:46:07] Like that? [22:46:07] I don't see how server lag would be a TS issue. [22:46:21] No, this is "Server lag, sleeping for 120 seconds". [22:46:44] Which is some output from wikitools indicating that the API is lagged. [22:46:48] New patchset: Pyoungmeister; "small fix and redoing one of my worst naming decisions ever" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2564 [22:47:11] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2564 [22:47:34] if error == "maxlag": [22:47:35] lagtime = int(re.search("(\d+) seconds", content['error']['info']).group(1)) [22:47:37] if lagtime > self.wiki.maxwaittime: [22:47:40] lagtime = self.wiki.maxwaittime [22:47:43] print("Server lag, sleeping for "+str(lagtime)+" seconds") [22:47:44] So it's hitting that branch of the code, repeatedly. [22:47:59] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2564 [22:47:59] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2564 [22:48:14] Joan: Ah, got that as well (on my own system) [22:48:34] http://zh.wikiquote.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=1 [22:48:44] a few days back... db25 was the one to blame, I think [22:48:53] http://en.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=dbrepllag&sishowalldb=1 [22:49:07] I'm not editing zh.wikiquote. ;-) But the lags all look fine now... [22:49:30] Joan: so it should work atm ... [22:49:36]