[00:00:03] Krinkle: It's not a bug yet, but the message name from mwalker is right [00:00:10] mw.org is working fine [00:00:42] > return wfMsg('multimediaviewer-viewfile-link'); [00:00:42] <multimediaviewer-viewfile-link> [00:00:53] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu Jun 12 00:00:48 UTC 2014 [00:01:12] Krinkle: Weird. [00:01:41] marktraceur: is this message available on all wikis? [00:01:49] or a subset or certain mwversion? [00:01:51] Only wmf8 [00:01:53] ok [00:01:56] I was testing a wikipedia [00:02:01] mw.org works OK [00:02:02] mwscript extensions/WikimediaMaintenance/refreshMessageBlobs.php --wiki="$wiki" [00:02:06] commons's API works fine [00:02:08] mediawikiwiki [00:02:08] > return wfMsg('multimediaviewer-viewfile-link'); [00:02:08] View original file [00:02:33] and it works from eval.php as well, if it were failing there, it would rule out any rl_cache or bits cache [00:02:40] yeah [00:03:09] I'm not super heartbroken about this [00:03:13] messagecache is the same between mw and commons, verified via cli to confirm, but yeah, message works on both. [00:03:24] marktraceur: If you invalidate the bits url but keep debug=false, is the message there? [00:03:26] It has to be RL related [00:03:27] Then you may need to touch a file in the module [00:03:33] No touching yet [00:03:51] (I mean, we could, but I'd rather find the cause in the few minutes we have) [00:03:54] Krinkle: How do I invalidate the bits url? [00:04:03] marktraceur: Did you find the url that has the message error on it? [00:04:30] Uh, no? [00:04:40] When you load your code on a page, there is a load.php url in the stack that will have the function, css text and json message blob [00:04:45] Krinkle: in debug mode, this is the outdated script: [00:04:49] https://bits.wikimedia.org/commons.wikimedia.org/load.php?debug=true&lang=en&modules=jquery.color%2CcolorUtil%2Cfullscreen%7Cmmv%7Cmmv.ThumbnailWidthCalculator%2Capi%2Clightboximage%2Clightboxinterface%2Cmodel%2Cperformance%2Cprovider%2Crouting%2Cui%7Cmmv.model.FileUsage%2CImage%2CLicense%2CRepo%2CTaskQueue%2CThumbnail%2CThumbnailWidth%2CUser%7Cmmv.ui.canvas%2CcanvasButtons%2Ccategories%2Cdescription%2CfileUsage%2CmetadataPanel%2Cpermission%2Cprogr [00:04:49] *nod* [00:05:02] in there you will most certainly find mw.load.implement( yourmodule, function(){}, css, and { key: ' not value' } [00:05:07] !log aaron Synchronized php-1.24wmf8/includes/EditPage.php: e11d41dd366b039bff79e247368b6bff1245ea5e (duration: 00m 07s) [00:05:12] Logged the message, Master [00:05:56] tgr: What is the timestamp at the end of that URL? (It got cut off) [00:05:57] url cut off, none of those modules contain multimediaviewer-viewfile-link [00:06:05] qchris: ping? can I haz new repo? :) mediawiki/extensions/UrlShortener? [00:06:23] it's in mmv.ui.canvasButtons, right? [00:06:24] YuviPanda: Sure. [00:06:31] https://bits.wikimedia.org/commons.wikimedia.org/load.php?debug=true&lang=en&modules=mmv.ui.canvasButtons [00:06:32] YuviPanda: Are we gonna have both UrlShortener and ShortUrl? [00:06:45] So it works there [00:06:48] RoanKattouw: indeed. currently I'm calling it ShortUrlv5, but beyond the joke value that sounds even more confusing [00:06:57] http://pastebin.com/raw.php?i=JEAHFF87 [00:07:03] Krinkle: Well yes. It's almost certainly an outdated Varnish cache entry [00:07:12] this means we're most definitely hitting the race condition that is caused by scap and having live servers in rotation. The timestamped url got cached before the message was there I guess. [00:07:14] RoanKattouw: I'm implementing whatever was defined in the RfC. And we can't fully get rid of the current shorturls since they are already deploye in a few places and shared too [00:07:15] Yep [00:07:16] &version=20140611T231430Z&* [00:07:17] See [00:07:23] That's almost an hour ago [00:07:43] Actually, no it ins't a scap race condition. It didn't get bumped at all, and even if it was, message cache is synced before any thing else [00:07:50] this means the module tiemstamp didn't get increased at all [00:08:01] Also that URL WFM [00:08:02] YuviPanda: Since I was about to ask the same thing as Ro-anKattouw ... UrlShortener it is? [00:08:16] YuviPanda: OK, just asking [00:08:18] marktraceur: touch a file in the mmv.ui.canvasButtons module and sync-file it [00:08:30] I'm not deploying, I'm not even logged into the server [00:08:34] qchris: can't think of another name that's pointing out that these are different and also that they do similar things [00:08:39] mwalker: Are you still around? [00:08:42] yepyep [00:08:43] YuviPanda: k. [00:08:49] mwalker: * [00:08:51] Ahm [00:08:52] Guyts [00:08:53] PROBLEM - MySQL Processlist on db1019 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 145 statistics [00:09:01] RoanKattouw: indeed, that URL works now [00:09:02] I don't know why, but tgr's link doesn't produce broken output for me [00:09:06] I get the message just fine [00:09:10] Is this even still a bug? [00:09:19] heh; well; the clearMessageBlobs script is still running... [00:09:20] Message is working [00:09:24] Or is it "just" in browser cache [00:09:28] On Commons [00:09:28] So [00:09:31] RoanKattouw: tgr's url doesn't contain the message, the url got cut off before the relevant modules query part [00:09:33] Maybe it got fixed [00:09:37] Commons itself works now [00:09:38] YuviPanda: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/UrlShortener [00:09:40] mwalker, I usually just touch all extension's files and sync-dir [00:09:50] Krinkle: He pastebinned the full thing [00:09:52] qchris: woo! ty! :) [00:09:53] RECOVERY - MySQL Processlist on db1019 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [00:09:58] YuviPanda: yw. [00:10:02] Yeah, btu that's debug mode. [00:10:04] MaxSem, that certainly is faster than this script [00:10:05] so fixed by the clearMessageBlobs script? [00:10:09] that's only ached for a few minutes [00:10:23] No [00:10:52] Krinkle: Same thing for debug=false [00:10:54] it was already fixed before we investigated anything, and it's still broken for users not using debug mode. [00:12:10] it is fixed in non-debug mode now [00:12:42] the pastebined url with debug=false in place has [00:12:42] {"multimediaviewer-viewfile-link":"\u003Cmultimediaviewer-viewfile-link\u003E"} [00:12:46] I checked around :45 I think, and it was broken in both modes then [00:13:08] https://gist.githubusercontent.com/Krinkle/c9ddbfe7736fe1473b15/raw [00:13:27] well as you said it has an old timestamp [00:13:30] Hah indeed it does [00:13:33] So yeah, old timestamp [00:13:39] So we need to touch a file in that module [00:13:41] even when bypassing browser cache, that one remains broken and will forever remain so until we update the module [00:14:14] * Krinkle yields and goes to look into the commit that changes it to see why it didn't cause a change event [00:15:46] ok -- so I'm going to let the script continue on it's merry way; but I'm going to touch and sync a/common/php-1.24wmf8/extensions/MultimediaViewer/resources/mmv/ui/mmv.ui.canvasButtons.js [00:16:04] Krinkle, ^ that sound OK? or do you want me to hold off until you finish your investigation? [00:16:15] Yep [00:16:18] go ahead [00:16:41] the fix is going to be the same either way, I'm doing postfix investigation already basically [00:17:11] RoanKattouw: Maybe json messages don't have a good message updated timestamp [00:17:14] !log mwalker Synchronized php-1.24wmf8/extensions/MultimediaViewer/resources/mmv/ui/mmv.ui.canvasButtons.js: poking cache for multimediaviewer messages (duration: 00m 04s) [00:17:18] message and use was added in the same commit: [00:17:19] https://github.com/wikimedia/mediawiki-extensions-MultimediaViewer/commit/b2039ad7eb8a994d78ab45761d8dae405e729ac9 [00:17:19] Logged the message, Master [00:17:32] so the typical "add message key that was updated before last module update timestamp" doeesn't apply [00:17:35] they're bothnew [00:17:51] (03PS2) 10Ori.livneh: mw/apache 2.4 compat: remove DefaultType directive [operations/puppet] - 10https://gerrit.wikimedia.org/r/138891 [00:18:03] PROBLEM - Puppet freshness on palladium is CRITICAL: Last successful Puppet run was Wed 11 Jun 2014 18:16:14 UTC [00:18:24] I think clearMessageBlobs is supposed to take care of this, but it's slow [00:18:33] Maybe there was a race condition? [00:18:47] soooo slow; it's still running [00:18:56] it would take care of it by giving all messages globally the current timestamp [00:18:57] I'm tempted to ctrl+c this [00:19:04] I mean it sounds like there must have been, we generated a response with the new code and without the message [00:19:06] Oh, oh, hold on! [00:19:09] Remember [00:19:22] There was a cached RL response with "messages":{"foo":""}} [00:19:26] So there is no failure on RL's part! [00:19:35] The l10n cache hadn't been rebuilt yet [00:19:44] RL was trying to include the message, but wfMsg('foo') was failing [00:19:50] I know [00:19:59] mwalker: Kill it [00:20:04] but scap does l10n update before rsyncing actual code [00:20:18] so why would rl be doing wfMsg(newkey) already? [00:20:25] !log clearMessageBlobs.php killed because we fixed the problem in a more different way [00:20:30] Logged the message, Master [00:20:59] it does l10n update for all caches, and only then start syncing to the first app server, right? [00:21:33] *nods* [00:21:37] but, it syncs code first [00:21:39] then i18n [00:21:56] wat? [00:22:10] ah, the file cache [00:22:13] wait; I'm misreading this [00:22:36] http://pastebin.com/1pXq4mrx [00:23:20] I think that's a regression in scap [00:23:24] scap-rebuild-cdbs happens before sync_wikiversions, but I'm not sure sync_wikiversions is what deploys the code [00:23:31] I think sync-common deploys the code [00:23:35] I'm pretty sure it didn't used to start with rsync common/ and *then* do l10n update [00:23:41] sync-common gets the code there, yeah [00:23:54] wikiversion just updates the versions we point to [00:24:10] so, for non-new-versions, sync-common is it [00:24:20] I reckon that output is just lieing [00:24:48] it if would really rsync first and then generate new wikiversions.cdb and l10n cache, then those files wouldn't be synced [00:25:14] yeah, not sure what that first step is, honestly [00:25:22] lines 3 and 4 [00:25:23] to tin.eqiad.wmnet from tin [00:25:35] ah that's /a/common to /usr/local/apache [00:25:46] still doesn't make sense though [00:27:12] RoanKattouw: Yeah, so it is what I thought. We no longer (or never did) rsync l10n cache to the app servers first. So while one single server will (aside from a very short window) have new code +new i18n or old code +new i18n, there can be an index.php request or startupmodule request handled by a new server, and generate a versioned url handled by an old server [00:27:23] the classic rl race condition that we see from time to time [00:27:27] Yeah [00:27:38] still weird that it had the new js but old message though [00:27:50] So what you're saying is we need to change our scripts to first sync the l10n cache, then the code? [00:27:51] do we put message cache somewhere central or per apache? [00:28:02] well, that might cause the opposite isssue [00:28:08] IIRC it's built centrally, then synced to each Apache [00:28:19] Krinkle: No it wouldn't, because of timestamps [00:28:21] Right, but it's not like sql [00:28:36] it's not accessing it from somewhere outside of the apache server [00:28:42] No, it's local only [00:28:44] (memcached shar on another host, or db slave) [00:28:46] ah, wait [00:28:47] Every Apache has its own local copy [00:28:49] it is [00:28:56] message *timestamps* are in sql [00:29:04] Yeah..... [00:29:07] Oh! [00:29:14] No but message contents are there too [00:29:18] So that should be fine, right? [00:29:31] RoanKattouw: No [00:29:33] message_blobs [00:29:35] because of something else [00:29:44] Or ahm, I guess it's called msg_resource [00:29:47] RoanKattouw: remember module definition summary hash ? [00:29:56] Yeaaaah? [00:29:57] the memcached cache key used to be modulename + rlcontext hash [00:30:15] with value { hash: md5(summary), timestamp: now() } [00:30:24] we changed it to modulename + rlcontext hash + md5(summary) [00:30:33] because during scap it keeps changing back and forth in memcached [00:30:42] and which ever wins last is kept [00:30:45] that can very well be an old one [00:31:06] I suspect that might be happening in sql rl message blob [00:31:40] depending on how we update the value [00:31:54] do we always update the blob if we had to regenerate the response, or only if timestamp is higiher [00:32:07] i hope we already update so that we can rollback [00:32:20] but that does mean it can be subject to this [00:34:06] I forget how this all works exactly [00:35:36] (03PS1) 10MaxSem: Disable PageImages on Wikibooks and Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139048 (https://bugzilla.wikimedia.org/66455) [00:43:43] RECOVERY - Puppet freshness on palladium is OK: puppet ran at Thu Jun 12 00:43:39 UTC 2014 [00:44:05] !log ran "puppetca -s palladium.eqiad.wmnet" on palladium to get agent running again, someone borked/regenerated the key there 6 hours ago? [00:44:10] Logged the message, Master [00:52:53] PROBLEM - Disk space on analytics1020 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/g 73191 MB (3% inode=99%): [00:59:55] (03CR) 10BBlack: [C: 031] Replace references to /etc/apache2/wmf symlink with its link target [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138858 (owner: 10Ori.livneh) [01:00:02] (03CR) 10BBlack: [C: 031] mediawiki/apache: load all.conf from canonical path rather than symlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/138877 (owner: 10Ori.livneh) [01:00:23] (03CR) 10BBlack: [C: 031] 2.4 compat: load mod_filter for AddOutputFilterByType [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138889 (owner: 10Ori.livneh) [01:01:08] (03CR) 10Kaldari: [C: 031] "Looks good. Feel free to merge when ready to deploy." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139048 (https://bugzilla.wikimedia.org/66455) (owner: 10MaxSem) [01:18:37] (03CR) 10BBlack: "As we discussed a bit in IRC, I'm a little wary of how this changes behavior for the existing apache 2.2 servers. I'm not even sure what " [operations/puppet] - 10https://gerrit.wikimedia.org/r/138891 (owner: 10Ori.livneh) [01:30:09] (03CR) 10Ori.livneh: "Fair point. I'll sample the logs to see if i can determine what requests that default is being applied to and how they might be effected." [operations/puppet] - 10https://gerrit.wikimedia.org/r/138891 (owner: 10Ori.livneh) [01:33:23] (03PS1) 10Krinkle: mwgrep: Add namespace prefix in output [operations/puppet] - 10https://gerrit.wikimedia.org/r/139059 [01:34:58] (03CR) 10Ori.livneh: [C: 032] 2.4 compat: load mod_filter for AddOutputFilterByType [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138889 (owner: 10Ori.livneh) [01:35:12] (03CR) 10Hoo man: [C: 031] "Makes sense :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/139059 (owner: 10Krinkle) [01:47:15] !log graceful'd appservers for I0e66ee0a1: 2.4 compat: load mod_filter for AddOutputFilterByType [01:47:23] Logged the message, Master [02:13:04] (03PS1) 10Tim Starling: .bashrc: fix error on machines without MW [operations/puppet] - 10https://gerrit.wikimedia.org/r/139060 [02:14:08] (03CR) 10Tim Starling: [C: 032] .bashrc: fix error on machines without MW [operations/puppet] - 10https://gerrit.wikimedia.org/r/139060 (owner: 10Tim Starling) [02:16:52] @info 10.64.16.27 [02:16:52] Krinkle: [10.64.16.27: s3 (DEFAULT)] db1038 [02:16:58] @replag s3 [02:16:58] Krinkle: [s3: aawiki] db1038: 0s, db1035: 0s, db1003: 0s, db1019: 0s, db1027: 0s [02:20:03] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [02:26:14] Hm.. I don't know how many Redis servers we have, but I'm seeing 2 or 3 in the logs that get a lot of connection errors [02:26:15] https://logstash.wikimedia.org/#dashboard/temp/XBWricGwS1Gr8yv3sqCAZg [02:26:48] And while doing interface message clean up on liwiktionary on tin, I'm consistently getting a connection time out from the mysql link [02:26:53] Caught exception DBConnectionError: DB connection error: Lost connection to MySQL server at 'reading authorization packet', system error: 0 (10.64.16.27) [02:27:03] making an edit on that wiki seems to work fine though [02:33:10] !log LocalisationUpdate completed (1.24wmf7) at 2014-06-12 02:32:07+00:00 [02:33:16] Logged the message, Master [02:35:01] TimStarling: Might be interesting to look into ^^ g2g [02:55:41] (03PS1) 10Springle: depool db1051 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139063 [02:56:14] (03CR) 10Springle: [C: 032] depool db1051 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139063 (owner: 10Springle) [02:56:20] (03Merged) 10jenkins-bot: depool db1051 for schema changes [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139063 (owner: 10Springle) [02:58:03] PROBLEM - Puppet freshness on mw1149 is CRITICAL: Last successful Puppet run was Wed 11 Jun 2014 23:57:00 UTC [02:58:05] !log springle Synchronized wmf-config/db-eqiad.php: depool db1051 (duration: 01m 08s) [02:58:09] Logged the message, Master [03:01:03] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Thu 12 Jun 2014 00:00:48 UTC [03:03:13] !log LocalisationUpdate completed (1.24wmf8) at 2014-06-12 03:02:09+00:00 [03:03:18] Logged the message, Master [03:18:03] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Thu 12 Jun 2014 00:17:12 UTC [03:25:49] (03CR) 10Mattflaschen: "I'm not sure which part was taken to email, and which list/where." [operations/puppet] - 10https://gerrit.wikimedia.org/r/49678 (owner: 10Ottomata) [03:30:53] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Thu Jun 12 03:30:46 UTC 2014 [03:40:14] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Jun 12 03:39:07 UTC 2014 (duration 39m 6s) [03:40:18] Logged the message, Master [03:56:57] (03CR) 10Ori.livneh: [C: 032] Replace references to /etc/apache2/wmf symlink with its link target [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138858 (owner: 10Ori.livneh) [04:16:13] (03CR) 10MaxSem: "> However, if I recall correctly, views by logged in users always trigger a parse. I would appreciate it if someone could confirm." [operations/puppet] - 10https://gerrit.wikimedia.org/r/49678 (owner: 10Ottomata) [04:16:53] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:43] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Thu Jun 12 04:17:34 UTC 2014 [04:17:43] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.003 second response time [04:21:34] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Thu Jun 12 04:21:32 UTC 2014 [04:25:45] (03CR) 10MaxSem: "As of correllation, this is an interesting task, actually: observers sniffing HTTPS traffic know times each request was made, their durati" [operations/puppet] - 10https://gerrit.wikimedia.org/r/49678 (owner: 10Ottomata) [04:27:03] (03CR) 10Jforrester: [C: 031] "Now good to go." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138345 (owner: 10Jforrester) [04:41:59] (03PS3) 10Ori.livneh: mediawiki/apache: load all.conf from canonical path rather than symlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/138877 [04:42:01] (03PS1) 10Ori.livneh: mediawiki: small clean-ups (wip) [operations/puppet] - 10https://gerrit.wikimedia.org/r/139065 [04:44:54] (03PS4) 10Ori.livneh: mediawiki/apache: load all.conf from canonical path rather than symlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/138877 [04:46:03] * ori will wait for euro-opsen to wake up before deploying that [04:46:33] * YuviPanda should go to sleep at some point [04:46:42] heh [04:47:13] ori: got a minute? [04:47:21] literally just [04:47:27] ori: ah, then 'tis k :) [04:47:38] i'll be back in an hour or so [04:47:41] but you should sleep [04:47:45] I should, yeah. [04:47:46] should.. [04:48:00] * YuviPanda is writing an implementation of the URL Shortner RfC [05:21:04] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [05:46:29] !log amslvs[1234]: stopping pybal [05:46:33] Logged the message, Master [05:52:17] !log cr1-esams/cr2-knams: dismantling amslvs BGP peerings [05:52:21] Logged the message, Master [05:54:39] <_joe_> paravoid: wow we're moving everything to the new load balancers? [05:55:00] we already did, these were just left as backups [05:55:21] <_joe_> oh ok [06:00:27] _joe_: fwiw, replaced by lvs300[1234] [06:00:35] bblack has been pulling all the work there [06:00:45] <_joe_> yeah I noticed [06:00:56] <_joe_> I just thought he was not 100% done yet [06:01:04] <_joe_> more like 99% done [06:01:06] it's 99% [06:01:08] yeah [06:01:15] he tried 3.13 yesterday [06:01:32] <_joe_> ok, still it's a 99% that allows going to prod [06:01:34] <_joe_> after they fixed the ipvs6 fiasco? [06:01:39] yup [06:01:48] <_joe_> that was just abysmal [06:01:49] and 3.13 fixes XPS for bnx2x [06:02:16] so I *think* what's left is rolling out 3.13 (and maybe trusty) to all of them and fix the python script to configure XPS [06:02:21] he'd know better :) [06:06:48] (03PS1) 10Physikerwelt: Disable MathML rendering option [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139068 (https://bugzilla.wikimedia.org/66492) [06:09:59] ori: awjr_away wait what's the name of the varnish role for vagrant we had? [06:10:17] ? [06:10:55] ori: hmm, at the zurich hackathon, didn't someone create a varnish role for mediawiki-vagrant? [06:10:59] or am I confusing nginx with varnish? [06:11:30] ottomata made the nginx module from operations/puppet a submodule and introduced it to vagrant [06:11:37] and awjr added an SSL role on top of it [06:11:46] ah, right. I... should sleep soon. [06:11:49] but no varnish yet [06:12:00] * YuviPanda needs to figure out some way of testing varnish, since he has to test purges [06:12:25] it's extremely annoying to compile VMODs [06:12:34] heh, thought so [06:12:41] but I need to test just HTCP purges... [06:12:54] what about labs? [06:13:04] ori: yeah, that's where I'll end up doing it, most probably [06:13:09] the varnish setup in labs is pretty true to production [06:13:23] and while you can't completely set fire to it it's not the end of the world if it goes down for a minute or two [06:13:24] ori: https://gerrit.wikimedia.org/r/#/c/139054/7 which is the beginning of a full implementation of https://www.mediawiki.org/wiki/Requests_for_comment/URL_shortener [06:13:57] so need to figure out a way to make sure that regular hits just hit varnish instead of apache, and also that negative cache results are purged when appropriate. [06:14:56] why do you need to purge? [06:15:57] ori: I want to cache negative hits as well. say I get /adf cached as 'sorry this does not exist', and sometime in the future a short url with that key *does* exist, so it should be purged from the cache to make sure that the redirect happens properly [06:16:52] serve a synthetic response from vcl_miss [06:17:56] ori: I haven't checked out VMODs at all, so that flew over my head a bit. [06:18:01] https://github.com/wikimedia/operations-puppet/blob/production/templates/varnish/errorpage.inc.vcl.erb [06:19:25] <_joe_> ori: I am struggling with mwprof/reporter - Its .gitreview is probably wrong? http://git.wikimedia.org/blob/operations%2Fsoftware%2Fmwprof%2Freporter.git/b26e2b2806b323b66a914e5cc0628467a6830b3b/.gitreview [06:19:43] ori: hmm, ok. I think I'll need to look at these with a slightly more awake brain, so am off to sleep now. [06:20:06] YuviPanda: after varnish looks for a suitable response object in its cache and fails to find one, right before it forwards the request to a backend, you have a chance to intervene [06:20:31] you can decline to forward the request to a backend and serve a static response generated in varnish [06:20:56] _joe_: sec, i'll look [06:21:55] ori: ah, hmm. also this will be a Special page, so I guess by default it won't emit headers that make varnish cache it, and it's something I must specify [06:22:52] <_joe_> ori: project points at mwprof instead than at mwprof/reporter - it seems wrong, but I may just be missing something. Take your time, not in a hurry [06:23:08] _joe_: that's almost certainly just me being an idiot [06:23:12] it sounds like a mistake [06:23:16] i probably created one, then the other [06:23:27] want me to fix? [06:24:00] YuviPanda: but why do you need all the machinery of special pages when it's basically a 404? [06:24:13] ori: not for the 'not found' case, but for the 'found' case. [06:24:32] but you don't need to purge those [06:24:48] oh, you're wondering how to get varnish to cache it [06:25:48] <_joe_> ori: no nevermind, I'll do that [06:26:07] _joe_: thanks, sorry about that [06:26:50] ori: yeah, how to get varnish to cache it [06:41:20] akosiaris: manutius only left with ganglia_new::monitor::aggregator, where should that move to? [06:45:48] netmon1001? [06:46:02] I guess so [06:49:55] (03PS1) 10Matanya: netmon1001: add ganglia_new::monitor::aggregator [operations/puppet] - 10https://gerrit.wikimedia.org/r/139071 [06:57:53] PROBLEM - Disk space on analytics1020 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/k 72305 MB (3% inode=99%): /var/lib/hadoop/data/j 107784 MB (5% inode=99%): [07:04:59] (03PS1) 10Matanya: manutius: decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/139072 [07:14:22] (03PS1) 10Matanya: manutius: decom, left mgmt [operations/dns] - 10https://gerrit.wikimedia.org/r/139075 [07:18:04] (03CR) 10Ori.livneh: [C: 032] mediawiki/apache: load all.conf from canonical path rather than symlink [operations/puppet] - 10https://gerrit.wikimedia.org/r/138877 (owner: 10Ori.livneh) [07:25:53] RECOVERY - Disk space on analytics1020 is OK: DISK OK [07:36:59] (03PS1) 10Giuseppe Lavagetto: access-request: grant Bernd Sitzmann access [operations/puppet] - 10https://gerrit.wikimedia.org/r/139079 [07:45:40] (03PS1) 10Faidon Liambotis: Allocate an IP for cobalt.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/139080 [07:48:15] (03PS1) 10Faidon Liambotis: install-server: add server cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/139081 [07:48:39] (03CR) 10Faidon Liambotis: [C: 032] Allocate an IP for cobalt.wikimedia.org [operations/dns] - 10https://gerrit.wikimedia.org/r/139080 (owner: 10Faidon Liambotis) [07:49:00] (03CR) 10Faidon Liambotis: [C: 032] install-server: add server cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/139081 (owner: 10Faidon Liambotis) [07:49:36] (03CR) 10Faidon Liambotis: [V: 032] install-server: add server cobalt [operations/puppet] - 10https://gerrit.wikimedia.org/r/139081 (owner: 10Faidon Liambotis) [07:49:43] oh jenkins [07:49:51] haha [07:52:27] (03CR) 10Faidon Liambotis: [C: 04-1] "Order of inclusion shouldn't matter on properly written manifests, this sounds like an awful hack to me..." [operations/puppet] - 10https://gerrit.wikimedia.org/r/138804 (owner: 10Hashar) [07:54:38] paravoid: yeah that patch ( ^^^^ ) is awful :( [07:54:56] <_joe_> hashar: also, does not do whatever you wanted to do [07:55:13] it is all because gallium ends up calling mediawiki::packages and webserver::php5 which both rely on the same apache packages [07:55:18] <_joe_> hashar: you could re-define packages defs in the parent classes in a better way [07:55:19] and I am not sure how to fix it [07:56:01] <_joe_> hashar: ideally, there would be a class like php5::packages that would be included by both [07:56:02] _joe_: yup I replied to you about how I have no clue how the dependence works (i.e. Class -> Class ) [07:56:17] <_joe_> hashar: it is documented to work as you expect [07:56:47] <_joe_> but in reality using "include" (which is encouraged by puppetlabs itself) defies that dependency chain [07:57:09] <_joe_> it will NOT apply the class on the left hand side before the one on the right-hand side [07:57:32] so with: Class['role::ci::slave'] -> Class['role::ci::website'] [07:57:38] that would apply role::ci::website first right? [07:58:02] <_joe_> I discovered it while trying to find decent solution [07:58:06] <_joe_> no [07:58:15] ah NOT [07:58:20] I am not paying attention [07:58:25] <_joe_> it will just ensure the second is defined if the first one is defined as well [07:58:31] <_joe_> maybe I wrote it wrong [07:58:45] <_joe_> let me make an example [07:59:27] honestly, lets just throw the class dependency idea to the bin and use something else :D [08:00:10] I thought about either 1) adding if ! defined Package 2) creating a tiny wrapping class like apache::php5 to include [08:00:29] but iirc we don't really want tiny wrapper all other the place. I had the case with some other packages definition preivously [08:00:42] <_joe_> hashar: http://paste.debian.net/104585/ [08:01:10] <_joe_> I bet you expected the output to be 'bar' before 'foo' [08:01:13] <_joe_> and it's not [08:01:21] <_joe_> I expected it as well [08:01:59] <_joe_> the funny part is that if you declare class {'foo': } class {'bar': } in the node, it will behave as expected [08:02:34] <_joe_> "a fractal of bad design" [08:02:56] <_joe_> well, in the case of puppet, s/design/implementation/ [08:04:09] could it be that the notice{} are realized before? [08:07:05] _joe_: since it does not work, I am wondering which other solution could be used [08:07:18] I am out of ideas [08:12:39] <_joe_> hashar: as I stated earlier, build a class that both classes could include [08:20:22] <_joe_> hashar: do you know how can I seach a change on gerrit by its subject? [08:20:45] _joe_: I am not sure you can [08:20:51] <_joe_> eww [08:20:57] https://gerrit.wikimedia.org/r/Documentation/user-search.html might have some clue [08:21:09] <_joe_> so I'm left with looking at the git log of site.pp [08:21:12] <_joe_> shit [08:21:27] but git log can probably grep [08:22:03] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [08:22:58] _joe_: git log --grep // Limit the commits output to ones with log message that matches the specified regular expression [08:23:11] <_joe_> hashar: yes I did that, thanks [08:24:06] _joe_: in Gerrit that would be: message:'something' [08:24:14] Changes that match MESSAGE arbitrary string in the commit message body. [08:27:18] bah libapache2-mod-php5 is used all other the place :( [08:27:40] <_joe_> ok thanks :) [08:45:42] pffffff [08:46:23] apache::mod::php provides a default php template bah [08:56:12] (03PS3) 10Hashar: contint: reduce duplication with mediawiki::packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/138804 [09:00:09] (03CR) 10Hashar: "I factored out the installation of packages apache2-mpm-prefork and libapache2-mod-php5 to a new php5::apache2packages class." [operations/puppet] - 10https://gerrit.wikimedia.org/r/138804 (owner: 10Hashar) [09:00:23] I hate puppet sometime [09:05:24] <_joe_> hashar: sometimes? then you're still a puppet newbie [09:05:37] <_joe_> :) [09:05:53] I am [09:05:57] I should get a course maybe [09:06:07] <_joe_> not really [09:06:28] <_joe_> the doc is good enough [09:06:38] <_joe_> or, buy/download/whatever "pro puppet" [09:06:48] <_joe_> dunno how up to date is it [09:07:58] "How To Become a Puppet Master In A Week" [09:08:16] oh I see puppet as the gate protecting protection. A gate I want to avoid approaching :] [09:20:24] (03PS2) 10Filippo Giunchedi: keep 5 days worth of diamond.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/138789 [09:20:34] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] keep 5 days worth of diamond.log [operations/puppet] - 10https://gerrit.wikimedia.org/r/138789 (owner: 10Filippo Giunchedi) [09:43:24] (03CR) 10Filippo Giunchedi: "mhh the bug mentiones a similar problem than what we had and the fact that having the default sample rate low for high-frequency metrics, " [operations/puppet] - 10https://gerrit.wikimedia.org/r/138574 (owner: 10Filippo Giunchedi) [10:04:03] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Thu 12 Jun 2014 07:03:00 UTC [10:06:59] (03PS1) 10Giuseppe Lavagetto: mediawiki: re-include deployment users [operations/puppet] - 10https://gerrit.wikimedia.org/r/139089 [10:07:22] mediawiki? not appserver? [10:08:11] <_joe_> it was renamed... I was in doubt [10:08:18] <_joe_> yes, it's the appservers [10:08:37] <_joe_> but now the roles are all 'mediawiki::something' [10:08:48] <_joe_> btw not sure I included the correct group [10:11:09] oh are they? :( [10:11:24] I was hoping we'd finally make a distinction between mediawiki the software and our appserver stack [10:11:27] oh well [10:12:08] <_joe_> in other news, something turned puppet back on on the puppet-compiler02 [10:12:26] <_joe_> which made it not working again [10:12:27] <_joe_> jesus. [10:22:14] (03CR) 10Tobias Gritschacher: [C: 031] "@Faidon: is this ok now and we can move forward? Or are there still some issues with the script from your side?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [10:33:10] (03PS1) 10Giuseppe Lavagetto: profiler-to-carbon: set a timeout on the connection [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/139093 [10:33:23] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Thu Jun 12 10:33:16 UTC 2014 [10:33:59] (03PS2) 10Giuseppe Lavagetto: profiler-to-carbon: set a timeout on the connection [operations/software/mwprof/reporter] - 10https://gerrit.wikimedia.org/r/139093 [10:47:40] (03PS1) 10Nikerabbit: cxserver configuration for beta labs [operations/puppet] - 10https://gerrit.wikimedia.org/r/139095 [11:18:37] !log Gerrit: created mediawiki/services/cxserver/deploy repository for Nikerabbit and kart_ [11:18:43] Logged the message, Master [11:19:02] whii [11:23:03] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [11:47:43] _joe_: topic says you're on rt duty. Could you please help me with an alert around eventlogging on vanadium (I have no access there)? [11:48:04] Could you check whether or not under /var/log/eventlogging [11:48:35] There are files "all-events", "client-side-events", "server-side-events" getting written [11:48:39] with current information? [11:48:52] (Like tailing them and seeing a timestamp not older than 5 minutes) [11:51:07] ori: ^ You around? [11:52:05] apergos: Could you please help me check three files on vanadium? [11:52:52] sure [11:52:55] qchris: [11:53:07] ah I see, let me look [11:53:14] It's about /var/log/eventlogging [11:53:59] Could you please check whether or not "all-events", "client-side-events", "server-side-events" getting lines appended with recent timestamps, like not older than 5 minutes? [11:54:00] yes, I see it in the backread [11:54:02] just a sec [11:54:04] Thanks. [11:54:55] they all have a last modified of now [11:54:59] let me check the contents [11:57:19] yes, all of them have entries within the last minute [11:57:30]