[00:00:10] I imagine it also becomes a "why are you doing it for them and not us" problem for them [00:00:30] maybe if they did it they would be afraid of accidentally licensing all their patents to us [00:00:51] Do we know how Wordpress do it? [00:01:06] As IIRC, WP is already in the scary IIS installer thing [00:01:11] Reedy: I was planning on downloading their package and snooping around [00:01:33] I seem to recall poking at this 18 months or so ago for wordpress [00:01:41] and the answer was "fuck it, let's use linux" [00:01:49] lol [00:02:15] Grace Francisco from MS put in the request to Sumana and I - so.. [00:03:07] she followed up again this week - so I felt inclined to look into it more and give her a more detailed response than our initial "hmmm" [00:03:13] Is the 1.19 deployment stage 3 happening tonight or is it being moved forward because commons was done later than scheduled? [00:05:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:01] robla: ^ [00:06:29] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2729 [00:06:30] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2729 [00:06:31] Krenair: we had already rescheduled stage 3 for tomorrow night [00:06:44] Krenair: is there a calendar/wiki page somewhere we didn't update? [00:07:06] robla, looks like it: https://www.mediawiki.org/wiki/MediaWiki_1.19/Roadmap [00:07:12] TimStarling: Should this change be propagated to other wikis which still have mwEmbed enabled as a gadget? https://commons.wikimedia.org/?diff=prev&oldid=67397013 [00:07:43] yes, if they use prototype.wikimedia.org [00:07:58] Hmm ok thanks [00:08:13] I'll request a change on ptwiki.. [00:08:21] Krenair: Thursday, February 23 (-24), 23:00-03:00 UTC (3pm-7pm PDT): [00:08:21] MediaWiki 1.19 stage 3 deployment to: [00:08:22] (all projects except Wikipedia) [00:08:52] (make that PST...hrm) [00:08:59] oh right, my bad [00:11:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.044 seconds [00:14:10] TimStarling: Is it urgent to disable that? I noticed the script is being used in a lot of wikis. E.g.: http://toolserver.org/~hoo/globalPageHistory.php?page_namespace=8&page_title=Gadget-mwEmbed.js&rev_count=3&project%5B%5D=wikipedia [00:14:19] Maybe someone with global access could help? [00:14:20] it's not urgent [00:15:06] helderwiki: do you know somebody who uses it? [00:15:06] New patchset: Diederik; "Improvements: 1) IP range filtering and regular expression now work. 2) Started with unit-tests 3) Major refactoring" [analytics/udp-filters] (refactoring) - https://gerrit.wikimedia.org/r/2698 [00:15:12] helderwiki: does it work at all? [00:16:09] I tested it a few months ago... but I don't know if it is still working. [00:16:45] it is documented at http://en.wikipedia.org/wiki/Wikipedia:Multimedia_beta [00:16:58] subtitles do not work anymore [00:17:01] since months [00:17:14] at least in firefox - did not test somehwere else [00:19:08] New patchset: Diederik; "Improvements:" [analytics/udp-filters] (refactoring) - https://gerrit.wikimedia.org/r/2698 [00:20:10] Saibo: From https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)/Archive_97#Playing_sound.2Fvideos it seems the player may not be working either [00:20:33] I'll be back in 10-15 min-ish [00:24:53] helderwiki: the player works for me [00:25:07] however, there is no advantage compared to FF's default player [00:25:26] Hmm.. [00:34:59] New patchset: Lcarr; "Bringing neon back to life" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2730 [00:37:45] New patchset: Ryan Lane; "Use the simple scheduler, rather than chance." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2731 [00:38:20] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2731 [00:38:21] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2731 [00:40:09] PROBLEM - Host cp1017 is DOWN: PING CRITICAL - Packet loss = 100% [00:41:57] RECOVERY - Host cp1017 is UP: PING OK - Packet loss = 0%, RTA = 26.43 ms [00:44:57] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:46:00] PROBLEM - Backend Squid HTTP on cp1017 is CRITICAL: Connection refused [00:46:45] PROBLEM - Frontend Squid HTTP on cp1017 is CRITICAL: Connection refused [00:49:16] 󠁟 󠁟 󠁟 󠁟 󠁟^ÂÊÎÔÛâêîôûĈĉĜĝĤĥĴĵŜŝŴŵŶŷˆ̭̂᷍ḒḓḘḙḼḽṊṋṰṱṶṷẐẑẤấẦầẨẩẪẫẬậẾếỀềỂểỄễỆệỐốỒồỔổỖỗỘộ⨣⨶⩯ꞈ [00:50:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.045 seconds [00:55:20] New patchset: Bhartshorne; "syntax corrections, size correction, partition number correction" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2732 [00:55:47] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2732 [00:56:22] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2732 [00:59:36] New patchset: Ryan Lane; "Making labstore1-4 LDAP clients" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2733 [00:59:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2733 [01:02:39] RECOVERY - Frontend Squid HTTP on cp1017 is OK: HTTP OK HTTP/1.0 200 OK - 27545 bytes in 0.126 seconds [01:03:51] RECOVERY - Backend Squid HTTP on cp1017 is OK: HTTP OK HTTP/1.0 200 OK - 27400 bytes in 0.178 seconds [01:04:42] !log reedy synchronized php-1.19/resources/ 'r112174' [01:04:44] Logged the message, Master [01:11:43] Since I did not mention it already (regarding the 1.19 update at Commons): ***Good job!*** :-) Third time lucky ;) [01:12:20] New patchset: Ryan Lane; "Adding git and mysql-client to stat1" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2734 [01:12:55] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2734 [01:13:18] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2733 [01:13:19] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2734 [01:13:19] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2733 [01:19:05] New patchset: Ryan Lane; "Adding geoip libraries to ensure it's managed by puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2735 [01:24:42] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2735 [01:24:42] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2735 [01:24:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.064 seconds [01:35:57] New patchset: Bhartshorne; "adding class to install pip for stat1 - dev use only." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2736 [01:37:39] !log Updated and synchronized fraud prevention settings on the payments cluster. [01:37:55] Logged the message, Master [01:47:19] New patchset: Bhartshorne; "adding class to install pip for stat1 - dev use only." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2736 [01:50:58] New patchset: Bhartshorne; "adding class to install pip for stat1 - dev use only." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2736 [01:51:53] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2736 [01:51:54] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2736 [01:56:12] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 635s [01:57:06] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 689s [02:04:25] New patchset: Bhartshorne; "puppet exec enforces full paths; that's cool." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2737 [02:04:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:06:30] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2737 [02:06:31] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2737 [02:10:54] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 5.475 seconds [02:18:24] !log LocalisationUpdate completed (1.18) at Thu Feb 23 02:18:24 UTC 2012 [02:18:26] Logged the message, Master [02:27:00] New patchset: Bhartshorne; "trying to remove 4700 lines of spam as stat1 tries to manage /a for ezachte." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2738 [02:27:23] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2738 [02:27:23] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2738 [02:37:13] !log LocalisationUpdate completed (1.19) at Thu Feb 23 02:37:12 UTC 2012 [02:37:15] Logged the message, Master [02:42:33] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [03:08:14] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 23s [03:09:17] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Thu Feb 23 03:09:00 UTC 2012 [03:09:35] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 1s [03:12:26] RECOVERY - Puppet freshness on db46 is OK: puppet ran at Thu Feb 23 03:12:05 UTC 2012 [04:19:07] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 213 seconds [04:19:52] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 232 seconds [04:23:19] PROBLEM - MySQL Replication Heartbeat on db1047 is CRITICAL: CRIT replication delay 202 seconds [04:23:46] PROBLEM - MySQL Slave Delay on db1047 is CRITICAL: CRIT replication delay 228 seconds [06:08:29] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [06:14:29] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [06:14:29] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [07:42:06] RECOVERY - MySQL Replication Heartbeat on db1047 is OK: OK replication delay 0 seconds [07:42:15] RECOVERY - MySQL Slave Delay on db1047 is OK: OK replication delay 2 seconds [08:15:51] sidebar on wikitech wiki is broken (reverted to defaults) [08:16:06] PROBLEM - Lucene on search9 is CRITICAL: Connection timed out [08:16:36] purged sidebar message [08:21:48] RECOVERY - Lucene on search9 is OK: TCP OK - 0.023 second response time on port 8123 [09:36:30] !log tstarling synchronized php-1.18/includes/LocalisationCache.php 'revert live hack' [09:36:35] Logged the message, Master [10:27:42] New patchset: ArielGlenn; "verify objects; no-derive tag for uploads; success/failure message." [operations/dumps] (ariel) - https://gerrit.wikimedia.org/r/2739 [10:27:46] New review: gerrit2; "Lint check passed." [operations/dumps] (ariel); V: 1 - https://gerrit.wikimedia.org/r/2739 [11:26:24] !log tstarling synchronizing Wikimedia installation... : scap speed test [11:26:26] Logged the message, Master [11:33:54] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:35:51] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [11:40:58] sync done. [11:41:16] 15 seconds [11:42:10] are you sure? [11:42:57] no [11:43:18] how long did it take? [11:46:16] 20 minutes [11:46:32] painful [11:48:22] I think it's mostly rsyncd [11:48:57] it was 6 minutes of local stuff and 14 minutes of rsync [11:49:29] good night [11:49:32] night [12:06:16] should use bittorrent based sync! [12:12:29] domas: no! proper multicast! [12:43:47] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [13:48:54] PROBLEM - RAID on search7 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:50:42] RECOVERY - RAID on search7 is OK: OK: 1 logical device(s) checked [14:40:23] !log reedy synchronized wmf-config/flaggedrevs.php 'Bug 33273 - Enable FlaggedRevs on Ukrainian Wikipedia' [14:40:26] Logged the message, Master [14:41:22] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34615 - Disable Moodbar on Tamil Wikipedia' [14:41:24] Logged the message, Master [14:45:08] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34570 - Set wgBabelMainCategory to false for dewiki' [14:45:11] Logged the message, Master [14:47:33] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34476 - Set $wgLogo value for fawikisource' [14:47:35] Logged the message, Master [14:50:51] !log reedy synchronized wmf-config/InitialiseSettings.php 'bump AF lottery odds for ptwikipedia' [14:50:53] Logged the message, Master [14:52:56] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34566 - Modify the URL to upload files in eswiki' [14:52:59] Logged the message, Master [14:56:49] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34407 - New aliases for ko.wikipedia' [14:56:51] Logged the message, Master [15:17:54] https://commons.wikimedia.org/wiki/File:St_Joachim_Old_Mines_1.jpg the 800px thumb was broken (only ~75% of the image visible) and purging the file page did not help. In JS error console: Fehler: Image corrupt or truncated: https://upload.wikimedia.org/wikipedia/commons/thumb/9/9c/St_Joachim_Old_Mines_1.jpg/800px-St_Joachim_Old_Mines_1.jpg [15:17:54] Shortly accessed a *wrong* (wrong directory) thumb URL: https://upload.wikimedia.org/wikipedia/commons/thumb/9/9d/St_Joachim_Old_Mines_1.jpg/800px-St_Joachim_Old_Mines_1.jpg And now it also works at the right thumb URL?! [15:42:39] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia' [15:42:42] Logged the message, Master [15:43:24] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia' [15:43:27] Logged the message, Master [15:45:07] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 34401 - Change sitename and a namespace for Inuktitut Wikipedia' [15:45:10] Logged the message, Master [16:09:55] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [16:15:55] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [16:15:55] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [16:47:20] Hello. Can a sys admin install extension:MoodBar to fr.wikisource with the same configuration as en.wikipedia : anyone can view Feedback dashboard, any user can answer, moodbar-admin right for the admin, wgMoodBarCutoffTime set a month ago. [16:47:44] Here is the vote on the installation of the extension : https://fr.wikisource.org/wiki/Wikisource:Votes/MoodBar [16:48:22] <^demon> Tpt: File a request in Bugzilla with all that information. "Wikimedia" product, "Extension requests" component. [16:48:36] Ok. [17:04:37] ^demon: Here is the request https://bugzilla.wikimedia.org/show_bug.cgi?id=34618 [17:17:31] New patchset: Andre Engels; "Porting my changes to the new version of the code. Main changes: * Including traits.py, containing a number of 'standard' traits, enabling them to be re-used without having to be recopied or rewritten * Including selectors.py, which does the same for " [analytics/reportcard] (andre/mobile) - https://gerrit.wikimedia.org/r/2740 [18:03:33] zzz [18:17:43] aaron cleared profiling data [18:34:25] !log aaron synchronized php-1.19/includes/parser/Parser.php 'disabled per-template profiling' [18:34:28] Logged the message, Master [18:34:30] aaron cleared profiling data [18:34:57] http://live.wikimedia.in/ [18:54:14] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2730 [18:54:15] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2730 [18:56:04] New patchset: Lcarr; "Changing neon to public machine" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2741 [18:56:41] PROBLEM - Packetloss_Average on locke is CRITICAL: CRITICAL: packet_loss_average is 12.6867269565 (gt 8.0) [18:56:41] New patchset: Ryan Lane; "Increasing max-cores for the simple scheduler" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2742 [18:57:12] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2741 [18:57:12] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2741 [18:59:22] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2742 [18:59:23] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2742 [19:02:40] RECOVERY - Packetloss_Average on locke is OK: OK: packet_loss_average is 2.28160850877 [19:31:18] whoopsey. [19:39:13] New review: Ottomata; "Hey Andre, looks good. I've got a few comments and requests for changes. I put '-1' only because I..." [analytics/reportcard] (andre/mobile); V: 0 C: -1; - https://gerrit.wikimedia.org/r/2740 [19:43:38] ACKNOWLEDGEMENT - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 84, down: 2, dormant: 0, excluded: 0, unused: 0BRae3: down - BRae4: down - BR LeslieCarr not actual problem [19:55:37] !log aaron synchronized wmf-config/StartProfiler.php 'removed commented out code and w/s cleanups' [19:55:41] Logged the message, Master [20:01:00] !log reedy synchronized php-1.19/extensions/UploadWizard/includes/specials/SpecialUploadWizard.php 'r112235' [20:01:03] Logged the message, Master [20:17:07] domas: do we still need the 'incubatorslowness' profiling url param? :) [20:34:59] !log aaron synchronized wmf-config/StartProfiler.php 'Sample thumb.php traffic properly. Removed 'bigpage' and 'incubatorslowness' hacks. Split thumbnail group by MW version and added some more code comments.' [20:35:01] Logged the message, Master [20:42:45] !log aaron synchronized wmf-config/StartProfiler.php 'updated profiling class code paths to post-1.17 locations' [20:42:48] Logged the message, Master [20:50:20] !log aaron synchronized wmf-config/StartProfiler.php 'Sample thumb.php traffic properly. Removed 'bigpage' and 'incubatorslowness' hacks. Split thumbnail group by MW version and added some more code comments.' [20:50:23] Logged the message, Master [20:50:33] ugh, wrong summary, meh [20:51:22] New patchset: Lcarr; "Fixing neon cronspam and neon nagios" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2743 [20:52:13] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2743 [20:52:14] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2743 [20:54:35] !log aaron synchronized php-1.19/StartProfiler.php 'require() wmf-config profiler' [20:54:38] Logged the message, Master [20:54:43] New patchset: Lcarr; "Removing ensure => absent" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2744 [20:55:00] !log aaron synchronized php-1.19/StartProfiler.php [20:55:03] Logged the message, Master [20:55:10] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2744 [20:55:11] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2744 [21:00:47] !log aaron synchronized php-1.19/StartProfiler.php [21:00:49] Logged the message, Master [21:01:32] !log aaron synchronized wmf-config/StartProfiler.php 'added empty arrays to constructors as needed' [21:01:34] Logged the message, Master [21:02:23] !log aaron synchronized php-1.18/StartProfiler.php 'require() wmf-config profiler' [21:02:25] Logged the message, Master [21:02:27] New review: Danakim; "Got it, will summarize the commit message and wrap it from now. Will also keep tabulations from now ..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2682 [21:02:32] aaron cleared profiling data [21:03:01] hrmph [21:06:29] oh, nvm :) [21:09:31] domas: is cpu use down? [21:12:01] New patchset: Lcarr; "Fixing type in nrpe.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2745 [21:12:44] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2745 [21:12:45] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2745 [21:22:30] hehe, the previous bias was distorting graphite too ;) [21:22:49] hi! could you look at https://bugzilla.wikimedia.org/show_bug.cgi?id=34268 [Polish Planet Wikimedia stopped updating] - I've reported this bug 2 weeks ago and still not fixed :/ [21:30:53] New patchset: Ottomata; "Adding David Schoonever account 'dsc', including dsc on stat1.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2746 [21:41:53] New patchset: Lcarr; "changing /usr/lib/nagios to /usr/lib/nagios3" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2747 [21:42:40] New patchset: Ottomata; "Renaming my user 'otto'. Deleting 'aotto'. I'm currently only installed on stat1.wikimedia.org, so it is better to do this now rather than later. I hope this doesn't piss anyone off. Thank you!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2748 [21:43:38] New patchset: Lcarr; "changing /usr/lib/nagios to /usr/lib/nagios3" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2747 [21:44:54] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2747 [21:44:55] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2747 [21:48:32] PROBLEM - Host cadmium is DOWN: PING CRITICAL - Packet loss = 100% [21:56:47] RECOVERY - Host cadmium is UP: PING OK - Packet loss = 0%, RTA = 30.88 ms [22:00:52] Still an hour to go [22:08:24] New patchset: Lcarr; "ensuring /usr/local/nagios exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2749 [22:10:01] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2749 [22:10:02] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2749 [22:13:14] !log reedy synchronized php-1.19/resources/mediawiki/mediawiki.feedback.css 'r112249' [22:13:16] Logged the message, Master [22:14:11] !log reedy synchronized php-1.19/includes/resourceloader/ResourceLoaderStartUpModule.php 'r112249' [22:14:13] Logged the message, Master [22:24:39] !r 112249 [22:24:45] meh [22:26:42] !bug [22:26:42] https://bugzilla.wikimedia.org/show_bug.cgi?id=$1 [22:27:23] !r is https://www.mediawiki.org/wiki/Special:Code/MediaWiki/$1 [22:27:23] Key was added! [22:27:29] !r 112249 [22:27:29] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/112249 [22:33:01] New patchset: Lcarr; "ensuring directory exists + a bit of cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2750 [22:35:11] !log reedy synchronized php-1.19/extensions/UploadWizard/ 'r112256' [22:35:15] Logged the message, Master [22:35:57] !log reedy synchronized php-1.19/resources/startup.js 'touch' [22:35:59] Logged the message, Master [22:38:23] New patchset: Lcarr; "ensuring directory exists + a bit of cleanup" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2750 [22:38:58] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2750 [22:38:58] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2750 [22:43:25] thumbnail of ogg videos are not produced. 500 server error [22:43:33] just called a video with working thumbs [22:43:35] purged [22:43:42] → broken [22:44:43] link? [22:44:44] this one: https://commons.wikimedia.org/wiki/File:Kaiserwagen_einfahrt_vohwinkel.ogg [22:44:50] * robla looks [22:45:08] https://commons.wikimedia.org/wiki/File:Volxbibel_-_Die_Bergpredigt_.ogv works [22:45:13] have a look and then purge [22:45:39] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [22:46:05] or not, we want some working examples left ;) [22:46:16] it is all videos [22:46:17] :D [22:46:22] we have many of them ;) [22:47:01] bummer [22:47:31] maplebed: looks like we have a problem with video thumbnailing [22:47:52] yeah? [22:47:58] see ^ [22:48:20] https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Kaiserwagen_einfahrt_vohwinkel.ogg/mid-Kaiserwagen_einfahrt_vohwinkel.ogg.jpg [22:48:41] it seems to got broken between 23 Feb 2012 07:18:05 +0000 and 23 Feb 2012 11:58:48 +0000 [22:48:43] * maplebed runs a test. [22:48:53] according to thumbs of recently uploaded videos [22:48:55] could that be rewrite.py giving 404 in response to 500 from thumb.php? [22:49:18] likely [22:49:21] if so, rewrite.py should really be using a 500 error code [22:50:01] ms5 is returning the 500. [22:50:09] on that URL that TimStarling just posted. [22:50:29] well, ms5 does correctly propagate error codes from apache [22:50:38] like what I'm saying rewrite.py should do [22:50:39] (what I mean is it's not swift generating the 500) [22:51:08] TimStarling: I'm assuming the 500->404 thing in rewrite.py is a secondary problem and not actually a possible root cause, correct? [22:51:37] (not that we shouldn't fix that) [22:51:55] oh, I see the 500 -> 404 thing. sorry, I was looking for the source of teh 500. [22:52:01] ha [22:52:07] RoanKattouw: btw, sure are lots of UploadStash lines in exception.log [22:52:09] yeah, we should probably fix that too. [22:52:10] thumb.php gives a 404 apparently [22:52:20] so it's 404 -> 500 -> 404 [22:52:25] nice. [22:52:32] The source file for the specified thumbnail does not exist. [22:52:43] AaronSchulz: Sample error? [22:52:52] hmm [22:52:57] ms5 gives 500 [22:52:58] /w/api.php Exception from line 103 of /usr/local/apache/common-local/php-1.19/includes/upload/UploadStash.php: key '10dkh954uso4.2v3yjl.1870110.jpg' not found in stash [22:53:25] ms5 output: http://paste.tstarling.com/p/SxMfzZ.html [22:53:59] just my test was wrong, thumb.php is giving 500 as expected [22:54:04] Maybe this is DB lag? [22:54:08] * robla assumes we want to get a better handle on this issue before plowing ahead with a deploy [22:54:11] and rewrite.py is failing to propagate both the response code and the error message [22:54:18] it's not DB lag [23:00:54] New patchset: Lcarr; "Changing static /etc/nagios to variable of config directory needed for nagios3 compatibility" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2751 [23:01:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2751 [23:01:30] did anyone recently change the oggvideotools package? [23:04:55] $status = $this->runOggThumb( $file->getPath(), $dstPath, $thumbTime ); [23:05:06] likewise for runFFmpeg() [23:05:27] that will give an mwstore:// path to some shell script [23:05:39] that's not gunna work [23:06:07] are you fixing it? [23:06:17] sounds like a simple getLocalRefPath() drop in [23:06:38] I think that's a "yes" [23:09:29] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2751 [23:09:29] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2751 [23:09:38] I'm going to just jinx us right now by putting up the banner for deployment now [23:10:51] I am deploying those clicktracking changes now [23:11:04] To both versions [23:11:14] I said I'd do it 30 mins or so ago but got distracted [23:12:33] !log catrope synchronized php-1.18/extensions/WikiEditor 'r112264' [23:12:35] Logged the message, Master [23:12:41] did anyone back out those namespace changes? [23:12:51] Which ns changes? [23:12:52] !log catrope synchronized php-1.18/extensions/ClickTracking 'r112264' [23:12:52] !log aaron synchronized php-1.19/extensions/OggHandler 'deployed r112264' [23:12:52] which changes? [23:12:54] Logged the message, Master [23:12:56] Logged the message, Master [23:12:59] RoanKattouw: :) [23:13:02] lol [23:13:14] !log catrope synchronized php-1.19/extensions/WikiEditor 'r112264' [23:13:16] Logged the message, Master [23:13:25] Ones siebrand made.. [23:13:28] good [23:13:39] !log catrope synchronized php-1.19/extensions/ClickTracking 'r112264' [23:13:40] Saibo: looks like AaronSchulz fixed it [23:13:41] Logged the message, Master [23:14:04] r107309 and it looks like the answer is "no" [23:14:15] !r 107309 [23:14:15] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/107309 [23:14:17] RoanKattouw: any clue about those stash errors? [23:14:41] TimStarling: Not been backed out I don't think no [23:14:41] TimStarling: are you doing it now? Or do you want me to do them? [23:14:42] TimStarling: maybe you should have responded? :D [23:14:43] AaronSchulz: nfi [23:15:00] it'll take a few minutes for my svn mirror to get synced [23:15:04] AaronSchulz: If it says "stash key missing" then something weird is going on in UploadStash itself [23:15:09] so it's probably quicker if you do it, Reedy [23:15:13] Alright [23:15:21] New patchset: Bhartshorne; "when the back end returns something other than a 404, use that HTTP status code rather than 404." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2752 [23:15:23] RoanKattouw: like in the DB? [23:15:29] and the Swift/FileRepo/thumb.php layers wouldn't be at fault [23:15:31] robla: meeebeee [23:15:31] Yeah [23:15:34] TimStarling: when you're not busy: https://gerrit.wikimedia.org/r/2752 [23:16:00] New review: Catrope; "(no comment)" [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2752 [23:16:17] Reedy: are you fully armed and operational? [23:16:34] robla: AaronSchulz: thank you! Confirm: video thumbs work again [23:16:34] it's going on now... [23:17:36] TimStarling: guess we need to run scap to get the messages properly updated? [23:18:34] svn up run on fenari for r112266 [23:18:47] yeah, I think it can be sped up by removing the l10nupdate-*.cache files [23:19:05] RoanKattouw / TimStarling / Reedy : why do they need to be backed out? I replied both on wikitech-l as well as in the commit why there's no need to panic at all. [23:19:17] I don't know, ask Tim [23:19:21] ^ [23:19:26] hmm, never mind [23:19:27] RoanKattouw: well, TimStarling is not responding. [23:19:54] I thought I explained in great detail why they needed to be backed out [23:20:01] Reedy: Hang on I've got one more rev to merge in, doing it now [23:20:07] in email and IRC [23:20:24] TimStarling: I may have missed that. In that case a apologize. Can you please point me to the mail once more? [23:21:20] TimStarling: you have seen http://lists.wikimedia.org/pipermail/wikitech-l/2012-February/058189.html, right? [23:21:34] how about I just explain it [23:21:50] WTF [23:21:52] catrope@fenari:/home/wikipedia/common$ svn up php-1.19/extensions/CustomUserSignup/ [23:21:54] Skipped 'php-1.19/extensions/CustomUserSignup/CustomUserSignup.php' [23:22:00] TimStarling: sure [23:22:13] Conflicted state [23:22:26] !r 112266 [23:22:26] https://www.mediawiki.org/wiki/Special:Code/MediaWiki/112266 [23:22:31] But no conflict markers? WTF [23:22:39] OK that did it [23:22:43] in the 1.18 deployment we had a number of namespace changes which took both us and the wikipedia communities in question by surprise, and they had to be reverted immediately after deployment [23:22:54] there are two issues with namespace changes: [23:23:08] 1. we have to run namespaceDupes.php after they are deployed [23:23:18] Reedy: OK I'm done [23:23:29] and svn up-ped to [23:23:30] 2. they are disruptive and properly require some kind of review or community involvement [23:23:30] o [23:24:00] that's why I want them backed out now and deployed individually in response to shell requests or whatever [23:24:38] hopefully they can all be deployed before the 1.20 branch point and then they can stay in trunk [23:27:14] I'm not convinced, but I'm not going to argue. I think we're showing a lack of agf by doing this. I expect backend devs to pick up swiftly whatever requests come into bugzilla, and I think an announcement of this policy for Wikimedia would be appropriate. [23:27:21] 00:30 here. Time for bed. [23:27:24] away zzz... [23:27:47] good night [23:28:16] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/2752 [23:28:30] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2752 [23:29:22] I'll delete the cache files and run scap then [23:30:03]