[00:00:13] it's in SwiftFileBackend now (plus it uses a curl multi client in core in /libs) [00:01:11] (03PS2) 10Faidon Liambotis: Make temp URLs actually work (e.g. for private containers) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106635 (owner: 10Aaron Schulz) [00:01:26] alrighty, I'm deploying some lightnings [00:01:33] the funny thing is that SwiftFileBackend actually has slightly less code in it ;) [00:01:44] :) [00:01:45] MaxSem: just one, right? ;) [00:01:54] yup [00:01:58] (03CR) 10Faidon Liambotis: [C: 032] Swift: make temp URLs actually work [operations/puppet] - 10https://gerrit.wikimedia.org/r/106635 (owner: 10Aaron Schulz) [00:02:26] gerrit is awfully slow [00:03:23] maybe we should use methods = GET HEAD in the conf too [00:03:30] y so sloww, dear gerrit? [00:03:32] it's slightly scary that PUT is in there by default [00:03:55] some sort of temp key leak could lead to data loss via overwriting files [00:04:27] under [filter:tempurl] that is [00:05:16] (03CR) 10MaxSem: [C: 032] Enable new diff everywhere now that wikidiff2 has been upgraded [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106629 (owner: 10MaxSem) [00:05:21] we filter methods in varnish too [00:05:23] but it's a good point [00:05:41] and we will probably never have a use case for PUT anyway [00:05:50] paravoid: I'll make a patch [00:05:52] (03Merged) 10jenkins-bot: Enable new diff everywhere now that wikidiff2 has been upgraded [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106629 (owner: 10MaxSem) [00:06:03] I was about to, but gerrit is getting on my nerves [00:07:02] AaronSchulz: do you know if swift sends a proper Expires header for the URL? [00:08:37] !log maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/106629' [00:08:44] Logged the message, Master [00:08:48] MaxSem: so what is this new diff feature? :) [00:09:46] paravoid, https://en.m.wikipedia.org/wiki/Special:MobileDiff/589997839...589997897?mobileaction=beta [00:10:13] * AaronSchulz checks [00:10:54] (03PS1) 10Aaron Schulz: Added a [filter:tempurl] section to the swift proxy [operations/puppet] - 10https://gerrit.wikimedia.org/r/106637 [00:11:11] okay, I'm done [00:11:20] AaronSchulz: do these temp urls go through img_auth.php? or are they served to users via varnish too? [00:11:57] they just get used by avconv internally at the moment [00:12:03] ok [00:12:04] there are no plans for giving them to users [00:12:10] ok, good :) [00:12:15] which wouldn't work anyway due to the private dns name [00:12:40] so the cache headers probably don't matter then [00:14:51] I wonder why all of our other filters have a use = egg... [00:15:00] the docs say it's not needed [00:24:31] paravoid, re backporting- in addition to node 0.10 (doneish) and textlive we'll need a recent phantomjs [00:24:39] *texlive [00:24:55] 1.9.0? [00:25:20] yes, afaik that is the latest in debian [00:25:43] at least what I have locally [00:25:50] tracking unstable [00:26:05] (03PS4) 10BryanDavis: [WIP] Add logstash config for udp2log [operations/puppet] - 10https://gerrit.wikimedia.org/r/106154 [00:27:18] paravoid, nm- just checked on ruthenium and realized that 1.9.0 is already available [00:30:48] "Changes newer than 29 seconds may not be shown in this list. " [00:30:51] Hmm [00:31:00] "Due to high database server lag, changes newer than 43 seconds may not be shown in this list. : [00:31:05] Oh my [00:31:10] And it's rising it seems [00:31:18] Okay it stopped [00:34:09] 2014-01-09 21:34:50 mw1108 mediawikiwiki: Memcached error for key "flow-tree:rootpath:050dad18cf43f4081a6a90b11c2793de" on server "127.0.0.1:11211": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY [00:34:52] Does this mean $wgMemc->get( key ) would fail? [00:35:32] found why RT is broken [00:35:50] bleerghh [00:37:00] bsitu: it means all memc ops would fail for a while on that server [00:37:56] pecl avoids "memcached" for php in apache, but it's really avoiding twemproxy, which proxies to all the mc* boxes [00:38:06] AaronSchulz: ah, thx [00:38:06] *pecl memcached [00:38:21] and by "that server" I mean mw1108 [00:38:52] paravoid: so....how about that swift cluster in tampa? :) [00:39:47] give ma sec, I'm doing a bazillion things atm [00:39:55] me a sec* [00:42:45] (03CR) 10Faidon Liambotis: "This is not what this commit does. The previous commit had the same commit message, never do that. The commit should had been titled "rt: " [operations/puppet] - 10https://gerrit.wikimedia.org/r/106114 (owner: 10RobH) [00:42:48] RobH: ^ [00:43:07] (I have a reason for looking at your commits and this got me confused -- RT is broken atm) [00:43:23] oh, it was workign post commit and apache restart... i thought [00:43:32] i tested, did my stuff break or something else? [00:43:41] it's not your fault exactly [00:43:53] the mail->RT is broken [00:43:59] we have mails in the queue from vendors etc. too :) [00:44:09] it's because we have two certificates now, so we're doing SNI [00:44:27] and the perl libraries in precise don't do SNI [00:44:44] so your comment on my commit message is what, it shoudl read something like 'install the rt.wikimedia.org' rather than replace? [00:44:52] since replace i guess means the apache file? [00:44:57] no [00:45:20] grr gerrit is slow [00:45:27] so 8ddbd0760987b2b63681b0ffcfcba235a6db4ee8 is fine [00:45:31] yea, im trying to look and its reallllly slow [00:45:57] but then you did another fixup [00:46:04] to move install_certificate from inside the if to outside [00:46:15] oh, so my first was fine? i thought it wasnt working =P [00:46:18] so i moved it due to that [00:46:26] second guessed myself [00:46:28] no, the commit message for the first is fine I mean [00:46:39] it was buggy, and your second commit fixed it [00:46:47] but your second commit's message should reflect that [00:46:55] and referenced the other one somehow? [00:46:57] say "Fixup on the previous commit" or something like that [00:47:01] ok [00:47:18] because I was looking at the logs, and found the "replace commit", viewed that, and it was all confusing [00:47:26] yea, i understand what you mean [00:47:27] I was like "this doesn't replace anything!?" [00:47:35] took me a while :) [00:47:47] so now its also broken due to certs? [00:47:53] kind of, yes [00:48:13] .....shit, uhh, does the mailserver use the wildcard? [00:48:16] cuz i never touched it. [00:48:28] (or unrelated, but it just occured to me now) [00:48:47] You don't need to explain what is up with it now if you are busy fixing it, I can wait =] [00:50:23] yup, give me a sec [00:58:10] RobH: can you look at gerrit in the meantime? [00:58:30] checkin [01:00:23] its maxed out its memory with some java process [01:00:56] which is of course, well, gerrit.. [01:01:34] Have you tried turning it off and on again? [01:01:58] im going to restart the process yea, but was trying to see if i could determine what its doing first [01:02:04] well restart gerrit that is [01:02:21] i hope that doesnt break shit worse [01:02:44] I presume you can restart it gracefully? (ie now kill -9 etC) [01:02:47] !log gerrit is slowed down and memory is maxed out on system, restarting gerrit to see if it helps [01:02:53] yea, its restarting gracefully [01:02:53] Logged the message, RobH [01:03:11] its stopping... i imagine this part takes awhile [01:03:20] oh awesome [01:03:28] it cleared, but its clear came through AFTER i restarted [01:03:32] so its just gonna do what its doin. [01:03:36] and its back [01:03:54] and still slow, wooo [01:04:11] =/ [01:04:40] It should be alright when it warms up and consumes moar memory [01:05:07] yea, its no longer maxed out atleast [01:05:37] ok, its faster now [01:05:59] !log gerrit seems faster again for now [01:06:06] Logged the message, RobH [01:14:10] (03PS1) 10Faidon Liambotis: Add SSLCACertificatePath for rt & magnesium [operations/puppet] - 10https://gerrit.wikimedia.org/r/106640 [01:16:17] (03PS2) 10Faidon Liambotis: Add SSLCACertificatePath for rt & magnesium [operations/puppet] - 10https://gerrit.wikimedia.org/r/106640 [01:16:18] (03PS2) 10Faidon Liambotis: swift: don't allow PUT for temp URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/106637 (owner: 10Aaron Schulz) [01:16:31] gerrit still seems really happy now, and my bloodsugar is craptastic, im afk for a short bit getting food. [01:16:49] (03CR) 10Faidon Liambotis: [C: 032] swift: don't allow PUT for temp URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/106637 (owner: 10Aaron Schulz) [01:17:02] RobH: ping me later if you want to know what was broken with RT (it's complicated) [01:17:16] (03CR) 10Faidon Liambotis: [V: 032] swift: don't allow PUT for temp URLs [operations/puppet] - 10https://gerrit.wikimedia.org/r/106637 (owner: 10Aaron Schulz) [01:17:33] (03CR) 10Faidon Liambotis: [C: 032] Add SSLCACertificatePath for rt & magnesium [operations/puppet] - 10https://gerrit.wikimedia.org/r/106640 (owner: 10Faidon Liambotis) [01:19:51] !log apt: installing trusty's libio-socket-ssl-perl into precise-wikimedia; SNI-capable [01:19:59] Logged the message, Master [01:21:09] !log restoring RT's mail interface; expect delayed RT email to arrive soon [01:21:16] Logged the message, Master [01:29:03] ottomata: hey [01:31:58] (03PS1) 10Springle: Reduce LB general traffic during reindexing/partitioning on groupLoadsBySection slaves. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106644 [01:32:44] (03CR) 10Springle: [C: 032] Reduce LB general traffic during reindexing/partitioning on groupLoadsBySection slaves. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106644 (owner: 10Springle) [01:32:52] (03Merged) 10jenkins-bot: Reduce LB general traffic during reindexing/partitioning on groupLoadsBySection slaves. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106644 (owner: 10Springle) [01:34:00] !log springle synchronized wmf-config/db-eqiad.php 'Reduce LB general traffic during reindexing/partitioning on groupLoadsBySection slaves' [01:34:06] Logged the message, Master [01:36:08] AaronSchulz: I'm restarting all Swift frontends for tempurl [01:36:24] ok [01:36:32] I was too quick on the depool/restart/pool one of them and it spammed exception.log, ignore that [01:37:16] ok, it's done [01:41:21] !log maxsem synchronized php-1.23wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/106639/' [01:41:30] Logged the message, Master [01:42:24] paravoid: confirmed [01:43:12] and fudging the url/sig slightly gives a 401 so that's good [01:43:36] I need to do a swift sprint sometime soon [01:43:54] upgrade to latest version, switch to using a proper puppet module, remove the whole syslog crap etc. [01:43:57] so how much do we are about tampa being well synced? [01:45:05] * AaronSchulz made https://gerrit.wikimedia.org/r/#/c/106638/ to help with that, since the timestamp based mode is slow and sub-optimal after switching around "masters" [01:45:21] RECOVERY - Disk space on ms-be11 is OK: DISK OK [01:45:41] of course swiftrepl could do this now (e.g. clean up any discrepancies) [01:45:42] !log maxsem synchronized php-1.23wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/#/c/106639/' [01:45:49] Logged the message, Master [01:46:24] !log swift: set_weight 0 to ms-be11/sde1, disk failed [01:46:30] Logged the message, Master [01:47:12] wait, swift returns md5s on the listings? [01:47:21] holy crap [01:47:21] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:47:21] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:47:38] springle: ^^ [01:48:13] research slave [01:49:11] RECOVERY - Disk space on wtp1014 is OK: DISK OK [01:49:11] RECOVERY - Disk space on wtp1019 is OK: DISK OK [01:49:11] RECOVERY - Parsoid on wtp1024 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.004 second response time [01:49:12] RECOVERY - Disk space on wtp1004 is OK: DISK OK [01:49:21] RECOVERY - Disk space on wtp1007 is OK: DISK OK [01:49:22] RECOVERY - Parsoid on wtp1010 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.003 second response time [01:49:31] RECOVERY - Disk space on wtp1024 is OK: DISK OK [01:49:51] RECOVERY - Disk space on wtp1010 is OK: DISK OK [01:50:11] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:50:11] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [01:50:22] looks like parsoid had filled the disks with logs [01:50:44] yes, three days ago [01:50:52] I pinged you back then but didn't follow up [01:50:57] I just cleaned them up [01:51:04] thanks [01:51:27] with the upstart and log rotation stuff in puppet we'll hopefully end that misery soon [01:51:33] yup [01:51:47] I'm not complaining :) [01:52:06] this is something that we long should had done within ops [01:52:22] we should had done it long time ago I mean [01:52:45] well, either way we are getting there ;) [01:53:41] (03CR) 10Faidon Liambotis: "Brandon, what do you think? Can/should we fix this in the vmod?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/102887 (owner: 10Yurik) [01:55:35] (03CR) 10Faidon Liambotis: "I too find the abstraction a bit too much." [operations/puppet] - 10https://gerrit.wikimedia.org/r/83768 (owner: 10Dzahn) [01:59:11] RECOVERY - Parsoid on wtp1014 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.007 second response time [02:01:31] RECOVERY - Parsoid on wtp1019 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.006 second response time [02:05:05] AaronSchulz: you have a change for 1.23wmf10, were you planning on deploying it? [02:06:59] (03CR) 10Faidon Liambotis: [C: 04-1] "Pretty good. Inline comments." (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106471 (owner: 10Subramanya Sastry) [02:07:55] robla the Flow fix is just a sync-file, but I understand its bad form to leave another's commit on tin undeployed. What should we do? [02:08:41] AaronSchulz: ping [02:09:26] paravoid: do you know if the change AaronSchulz is working on is to fix an urgent issue? [02:09:58] which change is that? [02:10:12] RECOVERY - Parsoid on wtp1004 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.003 second response time [02:10:21] robla: I don't know, no [02:10:52] paravoid: https://gerrit.wikimedia.org/r/#/c/106638/ [02:11:07] he self-merged it, backported to wmf10, and apparently it's on tin now [02:11:11] oh, this is not urgent at all [02:11:15] but it can be deployed [02:11:22] copyFileBackend runs on a cron [02:11:46] er, not even that, it doesn't even run until we run it [02:11:56] ah, ok [02:12:24] it's for copying files between filebackends, i.e. between pmtpa & eqiad right now [02:12:48] which right now means purely reconciling differences for data integrity reasons [02:13:09] spage: is the change on tin some variant of https://gerrit.wikimedia.org/r/#/c/106638/ or something else? [02:13:31] RECOVERY - Parsoid on wtp1007 is OK: HTTP OK: HTTP/1.1 200 OK - 970 bytes in 0.007 second response time [02:13:47] robla yes, that's it, only changes maintenance/copyFileBackend.php [02:13:58] ok...go ahead and sync-file your change [02:14:49] robla OK, so that one file will be on tin but not on cluster after our sync-file. I'll e-mail Aaron Schulz and Greg-G. Thanks a bunch [02:14:59] spage: no prob [02:15:25] paravoid: thanks for clearing that up. I didn't notice it was in the maintenance directory as I skimmed it [02:18:16] robla: I don't think that was a self-merge, BTW. [02:18:22] It looks like Bryan D. +2'd it. [02:19:16] Gloria: you sure about that? Bryan is on the list, but Aaron has the checkmark [02:19:34] robla: Read the comments, which is basically the history. :P [02:19:39] Gerrit is stupid. [02:19:56] It has no actual history. You're just supposed to guess at what happened or pray for an e-mail. [02:20:06] It --> Gerrit, to be clear. [02:20:22] ah, I see now [02:20:52] it was +2 by bryan, +1 by parent5446, rebased and self-merged [02:21:29] which isn't stupid; the rebase *is* a different commit [02:21:41] The stupid part is Gerrit's lack of an actual changeset history. [02:21:48] It says all that, but in the comments. [02:22:00] As a hack, as far as I'm concerned. [02:22:38] I don't gerrit is that bad there [02:22:53] reading the comments is easy enough...unless you have like 50 versions [02:23:05] Well, it also aggressively collapses the comments. [02:23:11] And is generally hostile toward conversation. [02:23:20] So maybe that's the intent, heh. That the comments are from Gerrit alone. [02:23:27] It could use love. [02:23:41] PROBLEM - puppetmaster https on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:53] AaronSchulz: if we were doing automated stats on self-review, that would have been caught up in the net [02:25:19] !log spage synchronized php-1.23wmf10/extensions/Flow/includes/Repository/TreeRepository.php 'Flow cache key fix to 1.23wmf10' [02:25:27] short of parsing the comments, we'd have no way of knowing what was a true self-review, and what was a manual merge after a peer review [02:25:45] robla: FWIW the sync-common on mw1017 for final sanity check of the fix took "ages". (time flies when you're having fun :) ) [02:25:58] testing fix in prod now [02:25:59] Stats are often wrong. [02:28:26] !log LocalisationUpdate completed (1.23wmf9) at Fri Jan 10 02:28:25 UTC 2014 [02:28:31] PROBLEM - HTTP on virt0 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:25] Perhaps morebots shouldn't be on wikitech. [02:29:51] robla: the reviewers +2ed the change, which is the codediff + parents, though. you can't assume that the reviewers are ok with +2ing the rebase [02:30:27] robla: so it *is* a self-review, just with good reason. which is okay :) [02:30:43] Gloria: yeah, see #-labs for what's happening with virt0 [02:30:52] I just saw. I heh'd pretty hard. [02:31:48] robla our Flow fix helped, thanks soooo much! [02:33:32] so did my engineering list post just go to dev/null? [02:35:36] hmm, it went to engineering@wikimedia.org instead of engineering@list.wikimedia.org [02:35:54] paravoid: arg, what is the former? [02:35:56] AaronSchulz: reading it now. [02:36:14] ok, so that worked, maybe I just don't get me own messages back from there [02:41:45] AaronSchulz: did you see ^ about your maintenance/copyFileBackend.php change? I updated 1.23wmf10 on tin to sync one file and that showed up. [02:43:12] yeah you can ignore that [02:43:48] it was taking jenkins a while to merge...and then I forgot what I was waiting on and got a burrito [02:46:21] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:46:21] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:47:11] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [02:47:11] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [02:50:10] researchers sure know how to hammer a poor defenceless db slave [02:50:17] hehe [02:55:47] (03CR) 10Chad: [C: 032] Turn off automatically searching commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106537 (owner: 10Manybubbles) [02:55:51] hey, we're monitoring logs and seeing steady exceptions from "ApiFormatXml::recXmlPrint: (P248, ...)" (looks like Wikidata 'snaks'), is this a known issue [02:55:57] (03Merged) 10jenkins-bot: Turn off automatically searching commons [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/106537 (owner: 10Manybubbles) [02:57:25] aude, Lydia_WMDE : ^ ? [02:58:01] !log demon synchronized wmf-config/CirrusSearch-common.php 'Disable commons search results for performance.' [02:58:24] <^d> !log That was supposed to go out during today's LD before I dove down a rabbit hole and forgot [02:58:40] !log LocalisationUpdate completed (1.23wmf10) at Fri Jan 10 02:58:40 UTC 2014 [03:01:31] RECOVERY - HTTP on virt0 is OK: HTTP OK: HTTP/1.1 302 Found - 457 bytes in 0.072 second response time [03:01:31] RECOVERY - puppetmaster https on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 3.823 second response time [03:15:28] paravoid: so is there any kind of long term plan for tampa yet? [03:15:53] long term, it's going to be shut down [03:16:03] (:P) [03:16:17] but in the meantime, the current thinking is to keep a copy of all data [03:16:28] and start moving them to the new DC [03:29:23] paravoid: do we want swift failover support now or just to have copies around? [03:30:14] and would just having swiftrepl on cron be good enough? [03:32:04] !log LocalisationUpdate ResourceLoader cache refresh completed at Fri Jan 10 03:32:03 UTC 2014 [03:32:10] Logged the message, Master [04:41:59] (03CR) 10Subramanya Sastry: "Thanks for flagging the bootstrapping issue." [operations/puppet] - 10https://gerrit.wikimedia.org/r/106471 (owner: 10Subramanya Sastry) [05:45:51] (03PS1) 10Springle: Assign db1055 to S1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106660 [05:47:15] (03CR) 10Springle: [C: 032] Assign db1055 to S1 [operations/puppet] - 10https://gerrit.wikimedia.org/r/106660 (owner: 10Springle) [05:55:57] !log xtrabackup clone db1050 to db1055 [05:56:04] Logged the message, Master [06:06:59] PROBLEM - mysqld processes on db1055 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [06:48:46] ori: can you please explian what is somehow? https://gerrit.wikimedia.org/r/#/c/100760/8/modules/subversion/manifests/init.pp [06:57:32] (03CR) 10Faidon Liambotis: [C: 04-1] "LGTM in general, very minor comments." (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106170 (owner: 10BryanDavis) [06:59:15] (03CR) 10Faidon Liambotis: "I think I'd prefer just calling it logstash, since that's how everyone's calling it. If alternative interfaces emerge, we probably host th" [operations/dns] - 10https://gerrit.wikimedia.org/r/105105 (owner: 10BryanDavis) [07:09:57] (03CR) 10Faidon Liambotis: "A few minor comments inline." (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106169 (owner: 10BryanDavis) [07:10:04] (03CR) 10Faidon Liambotis: [C: 04-1] Kibana puppet class [operations/puppet] - 10https://gerrit.wikimedia.org/r/106169 (owner: 10BryanDavis) [07:29:29] spage: thx - aude was working on that [07:34:42] paravoid: https://bugzilla.wikimedia.org/show_bug.cgi?id=59894 hrm :/ [07:34:56] curl gives me '401 Unauthorized: Temp URL invalid' [07:35:41] odd since that worked find in eval.php earlier today [07:39:44] heh, still works in eval [07:42:24] ottomata: did you note: https://gerrit.wikimedia.org/r/#/c/106505/ ? [07:49:57] hm, some kind of encoding issue, 7/74/SA_challenge_2013.jpg works fine [07:51:04] (03PS1) 10Springle: Assign db1060 to S2 (future master) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106663 [07:52:58] (03CR) 10Springle: [C: 032] Assign db1060 to S2 (future master) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106663 (owner: 10Springle) [07:56:16] (03PS2) 10Faidon Liambotis: Sync EQIAD before PMTPA [operations/puppet] - 10https://gerrit.wikimedia.org/r/105006 (owner: 10Reedy) [07:56:48] (03CR) 10Faidon Liambotis: [C: 032 V: 032] "Despite my objections, it doesn't hurt, so merging." [operations/puppet] - 10https://gerrit.wikimedia.org/r/105006 (owner: 10Reedy) [07:57:16] heh, I wondering about the shuffling :) [07:57:24] hey [07:57:36] though not all the rsync stuff uses that [07:57:54] that's what my objections were about :) [07:57:57] see comments [07:58:48] !log xtrabackup clone db1018 to db1060 [07:58:54] Logged the message, Master [07:59:10] (03PS2) 10Faidon Liambotis: remove chapter domains from DNS that are not owned by the WMF [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn) [07:59:19] (03PS3) 10Faidon Liambotis: Remove chapter domains that aren't delegated to us [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn) [08:00:17] (03CR) 10Faidon Liambotis: [C: 032] Remove chapter domains that aren't delegated to us [operations/dns] - 10https://gerrit.wikimedia.org/r/86659 (owner: 10Dzahn) [08:02:17] (03CR) 10Faidon Liambotis: [C: 04-2] "I don't think we need this. This was proposed during a bits outage that was because origin started throwing 40x because of a code deploys " [operations/puppet] - 10https://gerrit.wikimedia.org/r/95534 (owner: 10Mark Bergsma) [08:02:28] (03Abandoned) 10Faidon Liambotis: Double session limit for bits caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/95534 (owner: 10Mark Bergsma) [08:44:21] hello [08:44:27] hi [08:44:29] feeling better? [08:48:17] yeah a bit [08:48:37] basically slept non stop for the last 36 hours or so and barely eat anything [08:48:40] goood time [09:33:13] (03PS1) 10Dzahn: shell and deployment rights for Kartik Mistry [operations/puppet] - 10https://gerrit.wikimedia.org/r/106668 [09:34:13] paravoid: akosiaris , feel like checking that? [09:34:22] i suppose you also know Kartik from Debian? [09:37:19] mutante: LGTM [09:37:33] akosiaris: thx [09:37:39] but no... I dont know kartik :-( [09:37:53] paravoid probably does though. He seems to know everybody [09:38:22] i just assume it if somebody is a Debian Dev [09:38:26] :) [09:38:38] mutante: sent you an email [09:39:44] matanya: looking [10:02:23] (03PS5) 10Faidon Liambotis: Varnish: don't mobile redirect www.$project.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/89879 (owner: 10JanZerebecki) [10:02:29] (03PS1) 10Faidon Liambotis: varnish: simplify the mobile redirect regexp [operations/puppet] - 10https://gerrit.wikimedia.org/r/106669 [10:02:40] MaxSem: can I use your insight for a moment? :) [10:02:48] sure [10:03:01] do I have any credits left or should I help you with WAP first? :P [10:03:29] i was wondering about WAP [10:03:37] 5003 [10:04:17] MaxSem: it's the two patchsets above [10:05:55] (03CR) 10Alexandros Kosiaris: svn: convert into a module (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [10:13:58] paravoid, at least it was possible to unit-test the redirector:P [10:19:52] (03CR) 10Alexandros Kosiaris: [C: 032] Removed the unneeded now wikidiff2.ini override [operations/puppet] - 10https://gerrit.wikimedia.org/r/106510 (owner: 10Alexandros Kosiaris) [10:21:19] (03CR) 10MaxSem: [C: 031] Varnish: don't mobile redirect www.$project.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/89879 (owner: 10JanZerebecki) [10:23:34] (03CR) 10MaxSem: [C: 031] "Fuck hell yeah." [operations/puppet] - 10https://gerrit.wikimedia.org/r/106669 (owner: 10Faidon Liambotis) [10:29:32] paravoid, but yeah - vetting my WAP plans would be muchly appreciated:) [10:36:50] (03CR) 10Matanya: svn: convert into a module (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100760 (owner: 10Matanya) [10:38:23] akosiaris: did you understand what ori meant about moving '/etc/apache2/sites-available/svn' to the role? [10:40:42] matanya: yes. That manifests/role/subversion should have that resource [10:41:07] yeah, but why [10:41:36] hmm. So the general idea is that we should strive to create modules that are not wmf specific [10:41:37] MaxSem: yeah, I'm terribly sorry about the delay... it hasn't been easy. [10:41:41] if possible that is [10:41:52] not everyone loves that idea btw [10:42:13] it's not just not specific to wmf, it's also not specific to deployment [10:42:25] what if you want to set up an svn server for a different purpose, not svn.wm.org, for instance? [10:42:31] it's an unlikely scenario nowadays, but these things happen [10:43:25] that was my second point [10:49:16] (03PS3) 10Alexandros Kosiaris: applicationserver: pass puppetlint / retab [operations/puppet] - 10https://gerrit.wikimedia.org/r/104920 (owner: 10Hashar) [11:04:50] (03CR) 10Alexandros Kosiaris: [C: 032] applicationserver: pass puppetlint / retab [operations/puppet] - 10https://gerrit.wikimedia.org/r/104920 (owner: 10Hashar) [11:11:36] (03CR) 10Faidon Liambotis: [C: 04-1] Collection Renderer (Now a module!) (0316 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/102352 (owner: 10Mwalker) [11:15:37] (03PS2) 10Dzahn: shell and deployment rights for Kartik Mistry [operations/puppet] - 10https://gerrit.wikimedia.org/r/106668 [11:23:09] paravoid: ori, re: gdash migration. is this one also fixed? (#5922: fenari used as a proxy for graphite installation) [11:23:14] yes [11:23:22] cool, closing another one:) [11:23:25] thanks [11:39:31] (03CR) 10Dzahn: [C: 032] "< akosiaris> mutante: LGTM | < apergos> indentation :-/ (fixed in PS2) < apergos> otherwise lgtm | looks good" [operations/puppet] - 10https://gerrit.wikimedia.org/r/106668 (owner: 10Dzahn) [11:52:06] !log welcome new software deployer Kartik from Language Engineering [11:52:12] Logged the message, Master [12:18:49] kart_: welcome :) [12:48:08] PROBLEM - MySQL Processlist on db1059 is CRITICAL: CRIT 0 unauthenticated, 66 locked, 0 copy to table, 0 statistics [12:58:08] RECOVERY - MySQL Processlist on db1059 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 0 statistics [13:14:18] PROBLEM - MySQL Processlist on db1006 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 119 statistics [13:15:18] RECOVERY - MySQL Processlist on db1006 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 3 statistics [13:56:41] !log Gerrit: adding WikidataJenkins user to groups "Non-Interactive Users" and "Wikidata" [13:56:48] Logged the message, Master [13:59:57] RobH: wanna complete the integration/doc migration ? [14:00:03] RobH: I went sick yesterday sorry :( [14:16:38] (03PS1) 10Faidon Liambotis: Add librenms module and role class & apply it [operations/puppet] - 10https://gerrit.wikimedia.org/r/106694 [14:17:45] mark: wanna have a look ^^ ? [14:17:53] was already on it [14:17:58] I didn't configure SSL, are we going to do misc-web-lb? [14:18:07] maybe we shouldn't, like for other monitoring tools [14:19:02] your defs support it nicely I see, it shouldn't be a big deal [14:23:54] yeah perhaps not here [14:46:32] (03PS2) 10Faidon Liambotis: Add librenms module and role class & apply it [operations/puppet] - 10https://gerrit.wikimedia.org/r/106694 [14:46:34] now I just have to learn the deployment system [14:46:34] fun [14:46:34] (03PS1) 10Faidon Liambotis: webserver::apache: misc SSL fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/106700 [14:46:36] (03PS1) 10Faidon Liambotis: librenms: install SSL certificate & enable vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/106701 [14:47:07] (03CR) 10jenkins-bot: [V: 04-1] Add librenms module and role class & apply it [operations/puppet] - 10https://gerrit.wikimedia.org/r/106694 (owner: 10Faidon Liambotis) [14:47:21] (03CR) 10jenkins-bot: [V: 04-1] librenms: install SSL certificate & enable vhost [operations/puppet] - 10https://gerrit.wikimedia.org/r/106701 (owner: 10Faidon Liambotis) [14:47:28] oh [14:47:36] (03CR) 10jenkins-bot: [V: 04-1] webserver::apache: misc SSL fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/106700 (owner: 10Faidon Liambotis) [14:47:36] is it your intention to use $hostname where it's confusing with $::hostname? :) [14:48:04] -1? [14:48:15] yes, it is [14:48:20] my intention [14:49:00] (03CR) 10Mark Bergsma: Add librenms module and role class & apply it (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/106694 (owner: 10Faidon Liambotis) [14:49:12] why not use another name? [14:50:28]