[01:16:09] (03PS1) 10TTO: Add suppressredirect user group for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) [01:33:32] (03CR) 10TTO: [C: 031] "Any problem here? The FeaturedFeeds patch has been merged." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/136316 (https://bugzilla.wikimedia.org/66015) (owner: 10Whym) [01:40:54] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 29 Jun 2014 23:40:27 UTC [01:41:32] !log springle Synchronized wmf-config/db-eqiad.php: depool db1061 during schema changes (duration: 00m 07s) [01:41:41] Logged the message, Master [01:46:58] (03PS1) 10TTO: Allow subpages in template namespace of frwikiversity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142980 (https://bugzilla.wikimedia.org/57487) [01:49:41] (03PS2) 10Springle: Deploy db1072 and db1073 as future S1. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142487 [01:50:06] (03CR) 10Springle: [C: 032] Deploy db1072 and db1073 as future S1. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142487 (owner: 10Springle) [01:59:49] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jun 30 01:59:37 UTC 2014 [02:15:27] !log LocalisationUpdate completed (1.24wmf10) at 2014-06-30 02:14:23+00:00 [02:15:38] Logged the message, Master [02:24:59] !log LocalisationUpdate completed (1.24wmf11) at 2014-06-30 02:23:56+00:00 [02:25:04] Logged the message, Master [02:28:57] !log springle Synchronized wmf-config/db-eqiad.php: repool db1061, warm up (duration: 00m 07s) [02:29:03] Logged the message, Master [02:49:34] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 30 02:48:28 UTC 2014 (duration 48m 27s) [02:49:39] Logged the message, Master [03:32:22] (03PS1) 10Springle: switch tendril backend to db1011 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142982 [03:37:56] (03CR) 10Springle: [C: 032] switch tendril backend to db1011 [operations/puppet] - 10https://gerrit.wikimedia.org/r/142982 (owner: 10Springle) [03:39:42] (03PS1) 10Ori.livneh: Apache config for Wikivoyage using mod_proxy_fcgi [operations/apache-config] - 10https://gerrit.wikimedia.org/r/142983 [03:40:02] TimStarling: a pache for your thoughts [03:40:36] also, good afternoon [03:54:09] (03PS1) 10Springle: add db1044 to mariadb 10 s1 for HA tests [operations/puppet] - 10https://gerrit.wikimedia.org/r/142985 [03:57:41] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 30 Jun 2014 01:56:49 UTC [03:58:48] (03CR) 10Springle: [C: 032] add db1044 to mariadb 10 s1 for HA tests [operations/puppet] - 10https://gerrit.wikimedia.org/r/142985 (owner: 10Springle) [04:17:53] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jun 30 04:17:43 UTC 2014 [04:49:41] (03CR) 10Calak: [C: 031] Add suppressredirect user group for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) (owner: 10TTO) [05:45:18] (03CR) 10Calak: "TTO, admins should be able to add/remove this group; you have forgot it!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) (owner: 10TTO) [05:54:03] (03PS2) 10TTO: Add suppressredirect user group for ckbwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) [05:55:33] (03CR) 10TTO: "Sorry! It's been so long since I did one of these." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) (owner: 10TTO) [06:27:08] <_joe_> ori: still around? [06:32:02] akosiaris: morning, can we have a look at the patch again [07:30:47] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "seems legit, but some food for thought." (031 comment) [operations/apache-config] - 10https://gerrit.wikimedia.org/r/142983 (owner: 10Ori.livneh) [07:34:00] Hey anyone out there? [07:34:35] Just looking for some further details about this: https://www.mediawiki.org/wiki/Manual:Database_access#Lock_contention [07:53:22] !log upgrading ms-be300[2-4] to swift icehouse [07:53:27] Logged the message, Master [08:03:54] (03PS1) 10Filippo Giunchedi: swift: add container-sync config section [operations/puppet] - 10https://gerrit.wikimedia.org/r/142989 [08:07:55] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: add container-sync config section [operations/puppet] - 10https://gerrit.wikimedia.org/r/142989 (owner: 10Filippo Giunchedi) [08:23:10] PROBLEM - swift-container-server on ms-be3003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server [08:23:10] PROBLEM - swift-account-auditor on ms-be3003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-auditor [08:23:19] PROBLEM - swift-container-updater on ms-be3003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-container-updater [08:23:19] PROBLEM - swift-account-reaper on ms-be3003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [08:24:10] RECOVERY - swift-container-server on ms-be3003 is OK: PROCS OK: 25 processes with regex args ^/usr/bin/python /usr/bin/swift-container-server [08:24:10] RECOVERY - swift-account-auditor on ms-be3003 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-auditor [08:24:19] RECOVERY - swift-container-updater on ms-be3003 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-container-updater [08:24:19] RECOVERY - swift-account-reaper on ms-be3003 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/swift-account-reaper [08:24:39] RECOVERY - Disk space on ms-be3003 is OK: DISK OK [08:27:04] <_joe_> .win 12 [08:34:03] _joe_: hey! [08:34:23] <_joe_> YuviPanda: hey [08:34:41] _joe_: switched over the labs proxis to use the nginx module [08:35:05] so can test pfs either with that patch or with the nginx::ssl thing [08:35:19] <_joe_> YuviPanda: oh, great [08:35:38] <_joe_> the nginx::ssl thing has been a creation of mine last week to ease transitions [08:35:51] yeah noticed :) [08:36:05] is prod nginx using the module? [08:36:24] also i got a few commits merged on the nginx module as well [08:37:55] we don't have throughput monitoring for the labs nginx yet tho [08:40:55] <_joe_> not really, no [08:41:14] <_joe_> just the new service rcstream and a few other things do [08:41:40] would prod switch evetually? [08:48:56] <_joe_> not at present [08:49:09] <_joe_> no idea if it ever will, it should probably [09:25:20] (03CR) 10Calak: [C: 031] "Thank you!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142979 (https://bugzilla.wikimedia.org/67278) (owner: 10TTO) [09:54:03] <_joe_> Nemo_bis: fancy a new edition of the PFS gazette? :) [09:55:15] hashar: there? I'm going to merge 141488 141487 and 141924 [09:59:25] _joe_: yess [09:59:58] * Nemo_bis is always too lazy for tab-complete, fingers refuse to type more than 2 letters but jo* doesn't suffice [10:03:40] (03CR) 10Filippo Giunchedi: Add Grafana module & role (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh) [10:07:04] _joe_: morning, so PFS is a go? it was published on tech news, which means it must be delayed, tech news is never right about timing :) [10:10:10] godog: hi. Sorry I wasn't around :-D [10:10:47] matanya: [citation needed] [10:10:55] hashar: np, I think we're good to merge [10:11:07] +1 Nemo_bis [10:11:10] godog: ah the zuul puppet patches. I completely forgot about them over the weekend [10:11:17] godog: yeah those three are trivial enough [10:11:42] hashar: no worries, I've been away too on fri/weekend [10:12:01] godog: the rest of patches I would need some input / design idea / experience whatever :D [10:15:12] <_joe_> matanya: did not know about that [10:15:21] <_joe_> for me it's a go [10:15:57] I have subscribed to TechNews RSS feed https://meta.wikimedia.org/w/api.php?action=featuredfeed&feed=technews&feedformat=rss [10:16:05] that is handy sometime :) [10:16:09] godog: merge time? :-D [10:16:22] godog: I am on gallium ready to manually launch puppet [10:16:33] hashar: yup [10:16:55] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] zuul: migrate server definitions to server.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/141488 (owner: 10Hashar) [10:17:08] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] zuul: migrate merger definitions to merger.pp [operations/puppet] - 10https://gerrit.wikimedia.org/r/141487 (owner: 10Hashar) [10:17:21] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] zuul: get rid of git_dir and zuul_url in server conf [operations/puppet] - 10https://gerrit.wikimedia.org/r/141924 (owner: 10Hashar) [10:17:27] (03PS1) 10Aude: Set internalEntitySerializerClass Wikibase setting [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142998 [10:18:37] hashar: done [10:18:49] running puppet [10:20:25] !log restarting zuul after a puppet change for /etc/zuul/zuul.conf [10:20:31] Logged the message, Master [10:29:01] (03PS3) 10Hashar: zuul: prefix server default template with 'zuul.' [operations/puppet] - 10https://gerrit.wikimedia.org/r/141501 [10:29:09] (03CR) 10Hashar: "rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141501 (owner: 10Hashar) [10:29:52] (03PS4) 10Hashar: zuul: merger now has its own default file [operations/puppet] - 10https://gerrit.wikimedia.org/r/141502 [10:30:00] (03CR) 10Hashar: "rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/141502 (owner: 10Hashar) [10:30:51] godog: the next two one split the init script default files for the two services running: the server and the merger. [10:31:01] godog: https://gerrit.wikimedia.org/r/#/c/141501/ and https://gerrit.wikimedia.org/r/#/c/141502/ :-] [10:41:03] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 30 Jun 2014 08:40:26 UTC [10:47:32] hashar: yep looks good [10:47:43] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] zuul: prefix server default template with 'zuul.' [operations/puppet] - 10https://gerrit.wikimedia.org/r/141501 (owner: 10Hashar) [10:47:54] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] zuul: merger now has its own default file [operations/puppet] - 10https://gerrit.wikimedia.org/r/141502 (owner: 10Hashar) [10:48:23] hashar: merged! [10:48:38] \O/ [10:50:59] <_joe_> ok, if someone knows how I can check if the submodule change I did in a patch has already been merged in prod [10:51:09] <_joe_> and more importantly, how do I rebase? [10:51:14] <_joe_> shit I'm hating this. [10:51:45] git branch --all --contains ? [10:51:59] would let you know whether some sha1 is in the local repo [10:52:07] or drop --all for the current branch (i.e. HEAD) [10:52:11] <_joe_> it's a submodule [10:52:23] <_joe_> I really really hate this [10:52:46] cd to the submodule ? :-D [10:53:04] <_joe_> can't see how we're benefiting from this. Submodules may be nice in general (maybe), but having to work with gerrit it's a real mess [10:53:12] <_joe_> hashar: do you get how worng that is? [10:53:22] <_joe_> I'm sure you are [10:54:08] <_joe_> and rebasing does strange things in this situation. [11:00:40] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jun 30 11:00:31 UTC 2014 [11:01:27] (03PS4) 10Giuseppe Lavagetto: rcstream: add SSL support [operations/puppet] - 10https://gerrit.wikimedia.org/r/141931 [11:06:53] (03PS5) 10Giuseppe Lavagetto: rcstream: add SSL support [operations/puppet] - 10https://gerrit.wikimedia.org/r/141931 [11:07:13] (03CR) 10Giuseppe Lavagetto: [C: 032] rcstream: add SSL support [operations/puppet] - 10https://gerrit.wikimedia.org/r/141931 (owner: 10Giuseppe Lavagetto) [11:08:27] (03CR) 10Physikerwelt: "Which WMF version is currently being used?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/139421 (https://bugzilla.wikimedia.org/66587) (owner: 10Reedy) [11:27:55] <_joe_> lunch, bbl [12:09:17] _joe_: you can do git submodule in the ops repo to get the hash at which that submodule is pinned, and then in the submodule repo do a git log [12:09:37] _joe_: and if this is about your nginx::ssl change then yes, it is in prod now [12:19:47] <_joe_> YuviPanda: oh yes I long solved that :) thanks :) [12:20:36] _joe_: :) do you want to test the PFS patches on the labs proxies today? [12:20:59] <_joe_> YuviPanda: if you feel like it, yes [12:21:10] <_joe_> but don't cut out ie6 :) [12:21:14] _joe_: sure! can you merge? let me find patches [12:21:39] (03PS2) 10Yuvipanda: dynamicproxy: Tweak SSL config to match prod [operations/puppet] - 10https://gerrit.wikimedia.org/r/142207 [12:21:48] _joe_: ^ [12:21:57] <_joe_> YuviPanda: ok, will do [12:22:29] _joe_: still supports sslv3 so ie6 is ok... for now :) [12:23:05] (03CR) 10Giuseppe Lavagetto: [C: 032] dynamicproxy: Tweak SSL config to match prod [operations/puppet] - 10https://gerrit.wikimedia.org/r/142207 (owner: 10Yuvipanda) [12:23:24] <_joe_> 5 mins and I'll merge [12:24:30] _joe_: ok! I can run puppet on the box when appropriate [12:26:46] <_joe_> merged [12:28:03] any paravoid seen? [12:28:54] _joe_: done! [12:29:18] _joe_: https://sugarfrosties.wmflabs.org/wiki/Main_Page is sample URL [12:30:04] <_joe_> The connection is encrypted and authenticated using AES_128_GCM and uses ECDHE_RSA as the key exchange mechanism [12:30:08] <_joe_> :) [12:30:42] matanya: yes [12:31:36] _joe_: I see a CPU spike http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1404131481.593&from=00%3A00_20140626&until=23%3A59_20140630&target=project-proxy.dynamicproxy-gateway.cpu.total.system.value&target=project-proxy.dynamicproxy-gateway.cpu.total.user.value [12:32:00] let [12:32:05] 's see if that is just momentary [12:32:08] paravoid: have you seen OTRS admins complaint ? [12:32:11] <_joe_> YuviPanda: which is expected, it's still very low [12:32:28] <_joe_> YuviPanda: the point is not the cpu alone, but the throughput [12:32:39] <_joe_> which is not greatly affected [12:32:45] _joe_: right, which we don't track yet, sadly [12:32:51] _joe_: yeah, 9% seems ok :) [12:33:15] but it is almost a 2.5x spike, and this box has low load [12:33:19] <_joe_> YuviPanda: also, that includes the puppet run I think [12:33:32] _joe_: yeah, am waiting for it to stabilize. [12:34:16] <_joe_> YuviPanda: http://graphite.wmflabs.org/render/?width=586&height=308&_salt=1404131481.593&from=09%3A00_20140630&until=23%3A59_20140630&target=project-proxy.dynamicproxy-gateway.cpu.total.system.value&target=project-proxy.dynamicproxy-gateway.cpu.total.user.value [12:34:25] <_joe_> this shows the last few hours [12:34:32] <_joe_> and the spikes are all puppet runs [12:34:34] <_joe_> :) [12:34:37] _joe_: ah, makes sense :) [12:34:46] <_joe_> so the cpu increase is not really there [12:34:53] (03PS2) 10Yuvipanda: toollabs: Tweak SSL config to match prod [operations/puppet] - 10https://gerrit.wikimedia.org/r/142208 [12:35:00] _joe_: yeah, seems so. [12:35:03] <_joe_> (I predict a 20%-30% increase) [12:35:26] <_joe_> but, that server is literally doing nothing :) [12:35:27] _joe_: ^ is toollabs, which has more traffic. we should have throughput measurements in about a week, so I'm ok if you want to wait for that [12:35:34] _joe_: yeah [12:35:43] _joe_: most of labs traffic is to toollabs :) [12:35:59] paravoid: https://gerrit.wikimedia.org/r/#/c/141919/ for ref [12:36:13] <_joe_> YuviPanda: we actually wanted to go live tomorrow :) [12:36:19] _joe_: aah :) [12:36:27] _joe_: feel free to merge then :) [13:26:23] (03CR) 10Filippo Giunchedi: [C: 04-1] initial commit (0311 comments) [operations/debs/php-mailparse] (review) - 10https://gerrit.wikimedia.org/r/142751 (owner: 1020after4) [13:31:54] !log upgrade ms-fe300[12] to swift icehouse [13:31:59] Logged the message, Master [13:33:36] (03CR) 10Anomie: Disable local uploads on Malay Wiktionary (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) (owner: 10Odder) [13:33:49] (03CR) 10Anomie: [C: 04-1] Disable local uploads on Malay Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) (owner: 10Odder) [13:34:01] woohoo [13:35:05] <_joe_> wow [13:35:57] so far so good! [14:00:42] (03CR) 10Manybubbles: [C: 031] "Load tested this this morning and saw a 2-5% spike in load when I enabled it as the cache filled. After about two minutes the load died d" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142590 (owner: 10Chad) [14:03:41] (03PS2) 10Odder: Disable local uploads on Malay Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) [14:05:58] godog: how's swift? [14:07:50] paravoid: aside from that sdk problem in esams (that gets unmounted/remounted) I think it is fine, I see some bandwith going on now between the backends tho [14:11:37] godog: you can just set the weight for that disk at 0 [14:11:44] on the rings [14:11:51] we should anyway, it's a failed disk [14:12:55] paravoid: yup, I'm not sure it can replicate it without filling up the disks though [14:13:12] well, what's remaining, ~100G per disk [14:13:42] more 180G [14:14:53] yeah [14:22:08] mhh ok I'll try to decrease the weight by 30% or so and see what swift thinks [14:23:34] (03CR) 10Manybubbles: [C: 031] Disable local uploads on Malay Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) (owner: 10Odder) [14:24:19] (03CR) 10Manybubbles: [C: 031] Remove completed surveys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138634 (owner: 10MarkTraceur) [14:27:16] (03CR) 10Manybubbles: "Does this have to wait for the next commit in the chain (I26a5971a4ce388c81ead4f8a0af8d0754b3d945c) or is it safe?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137804 (owner: 10Awight) [14:29:16] anomie: I can swat today if you ant [14:29:18] want [14:30:33] manybubbles: I'd have no problem SWATting today either, but go ahead if you want [14:30:52] anomie: I want to redeem myself after the last time I failed to deploy and broke cirrus [14:31:01] I need to get my deploy legs back under me, I guess [14:31:20] that, and we're deploying again at 1 any way [14:41:20] !log Cleared out a watchlist with 126652 entries on warwiki to resolve https://bugzilla.wikimedia.org/show_bug.cgi?id=67123 [14:41:25] Logged the message, Master [14:42:24] paravoid: did you see my messages ? [14:42:43] I did [14:42:48] Jeff_Green is working on it [14:43:02] thanks. [14:43:13] fwiw, the correct response is to file an RT ticket [14:43:29] it should be fixed temporarily, I'm working on the permanent fix [14:44:05] thank you both, i'll do so next time. [14:44:26] :) [14:44:38] thanks for relaying the message :) [14:45:23] hoo is enjoying himself lately, I notice. [14:45:48] twkozlowski: I love killing people's wathclists, yeah :P [14:46:03] What was that in total? 200k+? [14:46:29] That bug was about 3 watchlists... biggest one was 800k+ entries [14:46:46] so 1M+ [14:46:52] JUST DON'T WATCH SO MANY PAGES PEEPS [14:46:53] In total? Yeah [14:47:33] <_joe_> 800k pages? [14:47:36] <_joe_> wtf? [14:47:52] yep, on shwiktionary, he watched 819k+ pages [14:48:23] !log installed new swift ring on esams, decrease ms-be3003/sdk1 weight [14:48:28] Logged the message, Master [14:49:44] But why watching 800k pages? You can't even make so many edits in your whole life... [14:51:08] twkozlowski: I've no idea how he managed to do that... he "only" got 450k edits over ther [14:51:08] e [14:52:32] 800k pages ????? [14:52:42] you can do that with a bot [14:52:46] Yep... biggest watchlist I've ever seen [14:52:54] :o [14:53:20] suppose it's watch everything i edit, then they go crazy with huggle or such [14:53:29] could do on wikidata [14:53:34] he got 450k edits on taht wiki [14:53:43] ooo [14:53:54] it's easy, when you operate a bot on the account [14:55:41] * aude has almost 4000 pages on my wikidata watchlist [14:56:19] oO [14:57:10] paravoid: I think we have a permissions snafu on polonium as well [14:57:19] could be [14:57:38] polonium:/var/lib/spamassassin is debian-spamd.debian-spamd but the proc runs as spamd.spamd, there are no bayes* files [14:57:38] aude: I got 65 pages... mostly because I watch virtually no items [14:58:01] ah [14:58:06] i have the setting enabled [14:58:23] some point have to clear and/or disable it [14:58:39] hoo see bug [14:59:04] How is that possible with only 63,000 edits [14:59:08] Oh [14:59:12] I see. [14:59:19] Kolega2357: ? [14:59:37] hoo https://bugzilla.wikimedia.org/show_bug.cgi?id=67123 [14:59:37] x2 right? [14:59:38] it would be a problem to have soooo many and then ask for maximum changes to show [15:00:04] manybubbles, anomie: The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140630T1500) [15:00:12] (03PS1) 10Jgreen: make iodine (otrs) spamassassin daemon run as default user (spamd) [operations/puppet] - 10https://gerrit.wikimedia.org/r/143033 [15:00:48] twkozlowski: around to verify your SWAT patch? [15:00:48] Kolega2357: What's with that? Should be fine now [15:01:07] Jeff_Green: please fix polonium too :) [15:01:10] hoo Yes all is a fine [15:01:14] we should probably run it as debian-spamd everywhere [15:01:15] marktraceur: same for you - around so you can verify your patch? [15:01:25] Hi! Yeah, I am for a bit [15:01:28] paravoid: yeah [15:01:30] Should be pretty quick [15:01:41] apergos: dataset alert? :) [15:02:02] paravoid: i'll figure it out, and make it adjust the directory ownership accordingly [15:02:14] ok [15:02:16] thanks :) [15:02:27] sorry for breaking it [15:02:31] did the deb create that directory? [15:02:36] marktraceur: ok -you'll go first then [15:02:40] /var/lib/spamassassin? iirc yes [15:02:46] Sweet [15:02:52] paravoid: ok [15:03:00] (03CR) 10Jgreen: [C: 032 V: 031] make iodine (otrs) spamassassin daemon run as default user (spamd) [operations/puppet] - 10https://gerrit.wikimedia.org/r/143033 (owner: 10Jgreen) [15:03:07] (03CR) 10Manybubbles: [C: 032] Remove completed surveys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138634 (owner: 10MarkTraceur) [15:03:27] manybubbles: Also the ProofreadPage thing, right? [15:03:34] * marktraceur being helpful to some community person [15:03:36] marktraceur: yup - both [15:03:41] ottomata/manybubbles: there's a "Slow CirrusSearch query rate" alert [15:04:04] Coren: labstore1001 has /exp/dumps with 0% free space [15:04:26] paravoid: thanks - I'll have a look at it in a bit - I imagine it is caused by the performance testing I was doing this morning - I wanted to push the system hard enough that it'd cause that [15:04:35] manybubbles: 1d 13h old [15:04:36] (03Merged) 10jenkins-bot: Remove completed surveys [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138634 (owner: 10MarkTraceur) [15:04:43] paravoid: then not so good [15:04:52] I'll investigate after swat [15:05:11] k [15:05:13] thx :) [15:05:15] there are unstaged changes on tin! [15:05:42] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:05:43] wmf-config/db-eqiad.php [15:06:04] I'm pretty sure that means I need to not deploy until someone decides what to do with them [15:06:14] committer? [15:06:15] springle? [15:06:22] unstaged - so no committer [15:06:39] manually changed you mean? [15:06:49] I thought merged but not deployed [15:07:12] paravoid: nope - manually changed but not committed [15:07:27] its in the middle springle's changes, but can't be sure who did it [15:07:54] I should be able to work around it if I'm careful but it'll be a pain [15:08:37] <_joe_> manybubbles: we should verify if that has been distributed [15:08:49] _joe_: sure - let me go check [15:08:50] <_joe_> if not, we should just revert the file IMO and save a stash [15:09:24] (03CR) 10BryanDavis: Add Grafana module & role (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh) [15:10:00] _joe_: looks to have been synced [15:10:29] <_joe_> ... [15:10:41] yeah! [15:10:43] (03CR) 10Andrew Bogott: [C: 04-1] "Thanks for doing this!" (035 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 (owner: 10Scottlee) [15:10:59] marktraceur and _joe_: I'm going to sync the proofreadpage change because this one won't be in the way [15:11:19] <_joe_> manybubbles: I don't know what to advise you to do [15:11:44] <_joe_> surely do not stash that change; I'd probably commit it right away since it's already in prod [15:11:57] <_joe_> but, you may have your ideas about that [15:12:00] _joe_: me neither - I might just commit it with a snarky message. I'm not sure what the db-eqiad file configures because I haven't looked but I can guess. [15:12:24] well, I know its synced to mw1001 - its probably on all the others, but I'm just assuming [15:12:29] Sean uses it to pool/depool/other stuff EQIAD machines [15:14:02] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: Fetching readonly [15:15:16] !log manybubbles Synchronized php-1.24wmf10/extensions/ProofreadPage: SWAT - fix ProofreadPage number of pages (duration: 00m 09s) [15:15:21] Logged the message, Master [15:15:29] marktraceur: ^^^^ that was proofreadpage [15:15:32] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.007 second response time [15:15:49] manybubbles: ugh, sorry re uncommited but sync'd changes [15:16:10] paravoid: is there any particular reason the spamassassin module only manages the user if the user is 'spamd' ? [15:16:17] greg-g: advice on should commit it, work around it carefully, or stop? [15:16:23] I imagine I should commit it [15:16:28] lemme read SAL [15:16:43] ah, yeah, that [15:16:59] is the diff basically: " springle Synchronized wmf-config/db-eqiad.php: repool db1061, " [15:17:03] ? [15:17:43] greg-g: I can't tell if its to repool it or repool it with a lower weight [15:17:49] I imagine its to lower the weight [15:19:03] <_joe_> !log restarted profiler-to-carbon, stuck since _9_ days, will see that my patch gets deployed. [15:19:08] Logged the message, Master [15:20:29] (03PS1) 10Manybubbles: Lower load factor on db-eqiad [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/143036 [15:20:49] manybubbles: well, the previous one was a depool, so I bet we can assume we can commit it with a snarky message ;) [15:22:33] (03CR) 10Manybubbles: [C: 032] Lower load factor on db-eqiad [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/143036 (owner: 10Manybubbles) [15:22:39] (03Merged) 10jenkins-bot: Lower load factor on db-eqiad [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/143036 (owner: 10Manybubbles) [15:23:50] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: Fetching readonly [15:23:55] !log manybubbles Synchronized wmf-config/: SWAT - remove completed mediaviewer surveys (duration: 00m 04s) [15:23:57] marktraceur: ^^^^^ [15:24:00] Logged the message, Master [15:24:07] now blame and shame :) [15:24:12] (03PS1) 10Jgreen: run spamd as user debian-spamd and manage that user+homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/143039 [15:24:32] I committed it and made sure that is was synced everywhere [15:25:03] progress [15:25:07] ok, moving on [15:25:39] twkozlowski: want to do yours next? [15:25:41] Yes [15:25:41] thanks manybubbles [15:25:42] (03CR) 10Jgreen: [C: 032 V: 031] run spamd as user debian-spamd and manage that user+homedir [operations/puppet] - 10https://gerrit.wikimedia.org/r/143039 (owner: 10Jgreen) [15:25:48] (03CR) 10Manybubbles: [C: 032] Disable local uploads on Malay Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) (owner: 10Odder) [15:26:00] wow jenkins review is speedy now! [15:26:10] (03Merged) 10jenkins-bot: Disable local uploads on Malay Wiktionary [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142318 (https://bugzilla.wikimedia.org/67152) (owner: 10Odder) [15:26:28] hashar: ^^ re Jeff's comment [15:26:44] !log manybubbles Synchronized wmf-config/: SWAT - disable local uploads on Malay Wiktionary (duration: 00m 04s) [15:26:45] twkozlowski: ^^^^^^ [15:26:50] Logged the message, Master [15:27:09] * twkozlowski nods [15:27:17] James_F and aude: next? [15:27:22] Sure. [15:27:28] I don't see awight [15:27:41] here [15:28:10] also, I was a bit worried about awight's one because it only had Nemo's blessing (but I love Nemo) and wasn't sure if that was enough. also, it has a subsequent commit and I'm not sure if that needs to go too. [15:28:20] James_F: then you can be next! [15:29:01] manybubbles: Yeah, given the follow-up commit is -2'ed as not working let's leave it? [15:29:08] the followup is a completely distinct thing [15:29:43] <mathshowimage> [15:29:51] Nemo_bis: It does look distinct - just accidentally a follow up. I'd like awight to be around though [15:29:54] Looks like a cache issue or whatever? [15:30:02] twkozlowski: math [15:30:24] Yes, but why doesn't it appear correctly? [15:30:27] I don't see anything on the great eye of logstash [15:30:56] heya ori, lemme know when youa re around [15:30:59] (03PS2) 10BryanDavis: scap: /usr/local/bin/sync-common-file is unused [operations/puppet] - 10https://gerrit.wikimedia.org/r/135925 [15:31:00] nuria and i have a q for you [15:32:27] (03CR) 10BryanDavis: [C: 031] "It's been a couple of weeks since I448f574 was merged, so cleaning up the ensure=>absent should be safe." [operations/puppet] - 10https://gerrit.wikimedia.org/r/135925 (owner: 10BryanDavis) [15:32:30] oh, sorry, nuria and oyu have already chatted abou tthis, maybe we can figure it out... [15:35:27] !log manybubbles Synchronized php-1.24wmf11/extensions/VisualEditor/: SWAT Correctly VisualEditor - update full size in MediaSizeWidget (duration: 00m 07s) [15:35:29] James_F: ^^^^^ [15:35:32] Logged the message, Master [15:35:52] Thanks manybubbles. [15:36:09] sure! is better? [15:36:12] ready [15:36:22] aude: merging [15:36:29] manybubbles: Will take a while to get through the bits cache, IME. [15:36:48] ok [15:38:00] (03PS4) 10BryanDavis: Set wgGitInfoCacheDirectory to point to scap managed location [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142320 (https://bugzilla.wikimedia.org/53972) [15:39:29] opinions on the awight patch? Without him around I'm not supposed to deploy it. I imagine Nemo_bis could verify that it is working though. Is it something I can just revert if it is broken or will it leave pages in a busted state? I don't know that part of the code at all. [15:40:18] no clue about that code [15:40:55] it's only triggered on edit, no retroactive effects [15:41:37] if you're unsure you can wait for Nikerabbit to +1 https://gerrit.wikimedia.org/r/#q,137804,n,z [15:43:02] !log manybubbles Synchronized php-1.24wmf11/extensions/Wikidata/: (no message) (duration: 00m 09s) [15:43:05] aude: ^^^^^ [15:43:07] Logged the message, Master [15:43:31] Nemo_bis: I'll push it then - just to be safe - I broke things badly last week and don't want to push my luck this week [15:43:34] ok [15:44:04] it's certainly not urgent [15:44:27] manybubbles: looks good [15:44:42] aude: sweet - done SWATing for today [15:44:56] :) [15:45:26] manybubbles: Thanks! [15:53:52] (03CR) 10coren: [C: 032] "Yeah, that makes sense." [operations/puppet] - 10https://gerrit.wikimedia.org/r/142208 (owner: 10Yuvipanda) [15:54:09] _joe_: ^ merged for toollabs :) [15:54:26] <_joe_> ok [15:54:36] <_joe_> great! [15:57:03] <_joe_> I'll be back later [16:02:50] Coren: ? [16:03:53] paravoid: ! [16:03:58] 18:04 < paravoid> Coren: labstore1001 has /exp/dumps with 0% free space [16:04:08] * Coren facepalms. [16:04:55] :) [16:04:59] Yeah, the RT ticket we were hoping would reach consensus hasn't. I'll carve out some space out of the general redundant array for the read stats, that'll allow us more time. [16:05:11] also escalate this into our meeting today [16:05:17] Yeah, good idea. [16:05:32] :) [16:05:33] The fundamental problem remains that dumps growth is, by definition, unbounded. [16:08:09] anybody working on the elasticsearch cache warming? [16:20:41] (03CR) 10Rush: "re: Too lazy to setup a trebuchet target for the graphite project in labs, so I just cloned the appropriate repo into the appropriate loc" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/133274 (owner: 10Ori.livneh) [16:48:32] godog: oh btw, there's another sort-of related action item that I've forgot to tell you [16:48:47] esams ms-fes are loadbalanced using DNS roundrobin, which is very suboptimal [16:49:13] at the time we didn't have LVS servers that sat on the esams internal VLAN or even supported multiple VLANs [16:49:30] now we do, so this should probably be converted to use an LVS service IP, like eqiad [16:49:36] bblack can help you with that in my absence :) [16:49:36] (03PS1) 10Jgreen: modify train_spamassassin to operate as user debian-spamd [operations/puppet] - 10https://gerrit.wikimedia.org/r/143048 [16:51:39] ^ :P [16:51:40] paravoid: oh ok, didn't notice that detail, will add it to the TODO [16:52:06] (03CR) 10Jgreen: [C: 032 V: 031] modify train_spamassassin to operate as user debian-spamd [operations/puppet] - 10https://gerrit.wikimedia.org/r/143048 (owner: 10Jgreen) [16:52:10] dns roundrobin would be awesome if caches were better at it [16:52:21] nah, we don't use ms-fe esams for real traffic [16:52:36] but even for swiftrepl (a custom replicator that mark wrote), it's a problem [16:52:42] it opens multiple persistent connections [16:52:54] (I meant if dns caches were better at it) [16:53:04] but doesn't do separate DNS queries when it does that, iirc [16:53:06] bblack: real caches or stuff like os stub resolvers too? [16:53:34] depends. for a big client app, you probably want your own resolver anyways. [16:53:49] but mostly real caches are bad at it [16:54:01] (03PS5) 10Scottlee: Fixed spacing and lint rules for manifests files. [operations/puppet] - 10https://gerrit.wikimedia.org/r/142479 [16:54:52] * bblack adds writing an awesome dns cache to the tail of his ever growing to-do list, in the section that will still be there long after he expires. [16:55:21] andrewbogott: could you review and ? [16:55:38] hehe I thought unbound might do bblack ? [16:55:40] bblack: is that where the geoip patch is? :D [16:55:48] ori: yes [16:55:54] heheh [16:56:19] ori: Yes, sorry -- I had already selected "+2" for that first patch on Friday when my wifi stopped working [16:56:34] bblack: amateurish C code at the front line of a top-10 website, what could go wrong?! [16:56:39] godog: well, anything's better than BIND [16:57:06] (03PS10) 10Faidon Liambotis: Improve nginx TLS cipher list & session timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [16:57:43] (03CR) 10Andrew Bogott: [C: 032] mediawiki_singlenode: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142206 (owner: 10Ori.livneh) [16:58:52] (03CR) 10Faidon Liambotis: [C: 032] Improve nginx TLS cipher list & session timeout [operations/puppet] - 10https://gerrit.wikimedia.org/r/132393 (https://bugzilla.wikimedia.org/53259) (owner: 10JanZerebecki) [16:59:14] (03CR) 10Andrew Bogott: [C: 032] role::deployment: port apache::vhost to apache::site [operations/puppet] - 10https://gerrit.wikimedia.org/r/142205 (owner: 10Ori.livneh) [16:59:42] andrewbogott: much obliged [17:00:04] manybubbles, ^d: The time is nigh to deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140630T1700) [17:00:09] dogeydogey: I can deploy if you want [17:00:30] (03CR) 10Manybubbles: [C: 032] Move about half of pool 4 lsearchd wikis to CirrusSearch [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/142590 (owner: 10Chad) [17:00:50] manybubbles deploy what? [17:00:52]