[00:03:28] (03PS1) 10Ori.livneh: Disable module storage on enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100933 [00:03:30] y'all, https://www.mediawiki.org/w/index.php?title=Talk:Flow&workflow=050b8fe7deb710412404782bcb0873b9 Sincerely <3 \o/ [00:04:05] (03CR) 10Ori.livneh: [C: 032] Disable module storage on enwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100933 (owner: 10Ori.livneh) [00:04:15] !log ori updated /a/common to {{Gerrit|I62541406b}}: Disable module storage on enwiki [00:04:31] Logged the message, Master [00:04:55] !log ori synchronized wmf-config/InitialiseSettings.php 'I62541406b: Disable module storage on enwiki' [00:05:11] Logged the message, Master [00:06:48] (03CR) 10Dzahn: [C: 032] add mw1-16 (pmtpa inactive job runners) to dsh apaches [operations/puppet] - 10https://gerrit.wikimedia.org/r/100759 (owner: 10ArielGlenn) [00:08:10] ori-l: Are you done LDing? [00:08:28] RoanKattouw: yes, go ahead; I was sneaking this in before LD [00:08:33] OK no worries [00:09:53] (03CR) 10Dzahn: "better sync one too many than the other way around and until they are actually shut down, yea" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100759 (owner: 10ArielGlenn) [00:11:33] !log catrope updated /a/common/php-1.23wmf6 to {{Gerrit|I70eaddd39}}: Update VisualEditor to wmf6 branch for cherry-picks [00:11:50] Logged the message, Master [00:12:20] !log catrope synchronized php-1.23wmf6/includes/libs/CSSMin.php 'Fix for bug 58338 (@import URL mangling)' [00:12:37] Logged the message, Master [00:12:52] !log catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks' [00:13:08] Logged the message, Master [00:13:14] (03CR) 10Dzahn: [C: 031] "looks like a no-op. you just did tabs->spaces, right" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100790 (owner: 10Matanya) [00:18:03] (03CR) 10Dzahn: "line 52: is this right? = ::labs_mediawiki_hostname ? no $?" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/100790 (owner: 10Matanya) [00:20:25] (03CR) 10Dzahn: "please also put your pubkey on your user page on office wiki, thx" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100923 (owner: 10Andrew Bogott) [00:27:13] (03CR) 10Akosiaris: [C: 032] Backup geowiki's data-private bare repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/98499 (owner: 10QChris) [00:27:24] (03CR) 10Akosiaris: [V: 032] Backup geowiki's data-private bare repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/98499 (owner: 10QChris) [00:30:54] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [00:58:34] !log catrope synchronized php-1.23wmf6/extensions/VisualEditor/ 'Cherry-picks' [00:58:48] Logged the message, Master [01:15:02] (03PS1) 10Bsitu: Fix Flow config setting for beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 [01:15:08] (03CR) 10jenkins-bot: [V: 04-1] Fix Flow config setting for beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 (owner: 10Bsitu) [01:15:59] (03PS2) 10Bsitu: Fix Flow config setting for beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 [01:36:26] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 10:33:16 AM UTC [01:37:52] i need DateTime.pm , i wonder if i can get it from one of all those "libdatetime-*" packages [01:38:18] but there are so many [01:38:41] mutante: yes you can [01:39:08] mutante: heh, there are just two :)) [01:39:21] one for i386 and one for amd64 [01:39:36] s/i386/x86/ [01:40:12] apt-cache search libdatetime | wc -l [01:40:13] 31 [01:40:20] libdatetime-perl [01:40:32] mutante: I've used it for jenkins [01:40:42] oh, libdatetime-set-perl is probably all unrelated [01:40:46] average: k, thanks! [01:41:09] :) np [01:41:59] i see, that would pull in a bunch of other lib*-perl [01:42:08] but yea [01:42:43] mutante: and for the general solution https://metacpan.org/pod/release/JKUTEJ/Debian-Apt-PM-0.09/script/apt-pm [01:43:07] "apt-pm - locate Perl Modules in Debian repositories" [01:44:10] average: nice! ah [01:44:15] :) [01:54:04] (03PS2) 10Gage: Add gage as a root [operations/puppet] - 10https://gerrit.wikimedia.org/r/100923 (owner: 10Andrew Bogott) [01:54:08] (03PS1) 10Dzahn: add package libdatetime-perl for bugzilla [operations/puppet] - 10https://gerrit.wikimedia.org/r/100947 [01:59:45] (03PS1) 10Tholam: Update favicon wiktionary/si.ico [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100949 [02:01:03] (03CR) 10Dzahn: [C: 032] "welcome to root - https://office.wikimedia.org/wiki/User:JGerard_%28WMF%29" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100923 (owner: 10Andrew Bogott) [02:05:54] (03Abandoned) 10BryanDavis: Hack: cron job to clean up files orphaned by UploadFromUrl [operations/puppet] - 10https://gerrit.wikimedia.org/r/100928 (owner: 10BryanDavis) [02:18:04] !log LocalisationUpdate completed (1.23wmf6) at Thu Dec 12 02:18:04 UTC 2013 [02:18:21] Logged the message, Master [02:21:10] (03PS2) 10Mattflaschen: fix routing of non-wikipedia on beta cluster [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100573 (owner: 10Hashar) [02:29:47] greg-g, et al, do I need a window to deploy something that should only affect Beta (https://gerrit.wikimedia.org/r/#/c/100573/)? [02:38:50] !log LocalisationUpdate completed (1.23wmf5) at Thu Dec 12 02:38:50 UTC 2013 [02:39:08] Logged the message, Master [02:46:41] (03CR) 10Mattflaschen: [C: 031] "This looks good. I just need to check if I need a window to deploy this." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100573 (owner: 10Hashar) [03:25:15] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec 12 03:25:15 UTC 2013 [03:25:32] Logged the message, Master [03:30:05] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [04:36:45] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 10:33:16 AM UTC [04:53:06] superm401: You don't need a window, you can just deploy that. [04:53:57] Oh, I thought that was -labs.php files. [04:55:16] That whole script is a bit terrifying. :-/ [05:01:29] * gwicke lols at http://www.joachim-breitner.de/heisse-news/news_27.xml [05:03:41] heh [05:05:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:07:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:09:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:11:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:13:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:15:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:17:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:19:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:21:25] if you had asked me a month ago if it were possible for puppet freshness alerts to be more annoying and less useful, I would have said no [05:21:48] and yet here we are [05:21:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:23:38] Gloria: you should write a bot [05:23:53] that produces dada-style puppet alerts [05:23:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:25:16] Puppet freshness is POINTY [05:25:50] ori-l: I think you're the only person who sees them. [05:25:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:26:09] gwicke: Ich lachte, but mainly at "Systemd is outdo in fact no init system , but a secretly by Linus Torvalds" [05:26:10] My tolerance for stupidity is pretty low. I ignored the bot. [05:27:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:00:04 AM UTC [05:28:22] spagewmf: ;) [05:29:13] ori-l: I liked my puppet idea better, in any case. ;-) [05:29:49] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Thu Dec 12 05:29:44 UTC 2013 [05:31:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:29:44 AM UTC [05:33:59] PROBLEM - Puppet freshness on mw1036 is CRITICAL: Last successful Puppet run was Thu 12 Dec 2013 05:29:44 AM UTC [05:59:32] RECOVERY - Puppet freshness on mw1036 is OK: puppet ran at Thu Dec 12 05:59:30 UTC 2013 [06:28:10] PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:30:10] RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active [07:28:45] (03PS1) 10Ori.livneh: Decouple Nginx configs from graphite::web & relegate to subclass [operations/puppet] - 10https://gerrit.wikimedia.org/r/100957 [07:30:26] (03PS1) 10Nemo bis: Preference "Email me when a page or file on my watchlist is changed" explicitly false [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100958 [07:37:33] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 10:33:16 AM UTC [07:41:26] (03CR) 10Spage: [C: 031] "Looks more right than before ;) . This change only affects labs, but as I understand it ops doesn't like undeployed changes hanging around" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 (owner: 10Bsitu) [08:00:14] (03PS1) 10ArielGlenn: stat1002 access for ironholds, rt #6452 [operations/puppet] - 10https://gerrit.wikimedia.org/r/100960 [08:05:58] (03PS2) 10ArielGlenn: stat1002 access for ironholds, rt #6452 [operations/puppet] - 10https://gerrit.wikimedia.org/r/100960 [08:07:31] (03CR) 10ArielGlenn: [C: 032] stat1002 access for ironholds, rt #6452 [operations/puppet] - 10https://gerrit.wikimedia.org/r/100960 (owner: 10ArielGlenn) [08:58:07] goood morning [08:58:44] hi hashar [09:12:41] hunting the 503 errors we got on beta [09:12:49] somehow varnish can't access the backends :( [09:14:03] morning [09:19:55] apergos: have you ever imported huge files on commons using mediawiki import scripts ? [09:20:01] no [09:20:03] apergos: was wondering if I should do it from tin.eqiad.wmnet [09:20:10] roan did that for a time [09:20:42] how huge is huge? [09:20:51] ~15GB [09:20:56] files at http://lan.80686.net [09:21:05] tin as enough disk space apparently [09:21:16] hashar@tin:~$ mwscript [09:21:17] This script can only be run from the command line. [09:21:18] hehe [09:22:20] h but any one file is 2gb or so? [09:22:32] https://commons.wikimedia.org/wiki/Help:Server-side_upload#See_also [09:22:35] hm 2.8 still ok [09:23:02] hashar: if you harrse Reedy, he should give you pointers [09:23:04] ah yeah one is 2.4G [09:23:11] the files are listed at http://lan.80686.net [09:23:19] Nemo_bis: thank you :-] [09:24:53] ah has reedy been doing them? awesome [09:25:02] we should really have that on wikitech someplace [09:25:59] apergos: any clue how I can check the max size a FS supports ? [09:28:27] if you are thinking of / on tin (ext4) it's fine [09:28:35] nice thanks [09:29:09] Max. file size 16 TiB (for 4k block filesystem) [09:29:25] import something that big into commons and we will kill you [09:29:32] before you get anywhere near finishing :-P [09:30:17] apergos: that would fill Swift right? [09:30:26] ahh [09:30:40] and tin doesn't have access to internet grblblb [09:30:49] internal ip [09:31:43] apergos: At this stage, You probably just need to convert Reedy into CAL and attach his brain to wikitech, to retain his knowledge [09:31:53] I'll even copy-edit [09:32:18] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [09:32:50] https://wikitech.wikimedia.org/wiki/Uploading_large_files well looky there, one whole line [09:32:51] :-D [09:34:20] :-D [09:35:35] gotta find out from which host I will wget though [09:37:48] we usually rely on brewster as proxy server [09:38:28] ahh that is it [09:38:31] forgot about that host [09:38:40] is that simply a http proxy or does it have some cache as well ? [09:39:01] there's squid over there [09:39:54] lemme see which ports etc [09:40:46] 8080 [09:40:49] brewster:8080 indeed [09:40:54] gotta find out how to pass that to wget :D [09:41:08] http_proxy=http://brewster.wikimedia.org:8080 [09:41:11] export http_proxy [09:41:14] should get it [09:41:23] ah lowercase [09:41:24] damn [09:41:31] 0% [ ] 10,625,229 1.15M/s eta 45m 17s [09:41:41] 45 mins [09:41:44] yawn [09:42:09] !log on tin, started a screened wget to download materials that will be uploaded to commons {{bug|58155}} [09:42:25] Logged the message, Master [09:43:36] will let it run and figure out later on how to run the import [09:44:26] that one-liner tells you [09:44:38] well assuming you put your stuff in the right directory [09:45:08] but we still should have more of a description, you are right [09:47:41] 16 FetchError c backend write error: 11 (Resource temporarily unavailable) [09:47:45] ah and here are my 503 [09:47:50] somehow apaches backend don't respond [09:48:17] it's repeatable? [09:48:35] I suspect the apache pool is too small [09:48:45] ah so a percent of them fail? [09:48:50] so they would refuse connections when they have too many connected [09:49:00] how is maxclients on them? [09:49:28] for that matter how many connections are in use at a given time? do we have that in the local ganglia? [09:49:35] I have no clue [09:49:38] (the prod cluster shows it) [09:50:42] ahh we have apache metrics \O/ [09:50:45] http://ganglia.wmflabs.org/latest/?r=hour&cs=&ce=&m=load_one&s=by+name&c=deployment-prep&h=deployment-apache32&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4 [09:50:59] day view v [09:51:00] http://ganglia.wmflabs.org/latest/?r=day&cs=&ce=&c=deployment-prep&h=deployment-apache32&tab=m&vn=&mc=2&z=small&metric_group=ALLGROUPS [09:51:06] lots of idle threads [09:51:46] maxclients isn't the issue [09:51:49] I really got to cleanup the apache conf and add them to the git repo [09:52:23] well I say that but I don't know what the setting is [09:52:46] maybe it's 20 [09:53:51] apache2.conf:MaxClients 40 [09:53:52] apache2.conf:MaxClients 5 [09:53:54] :D [09:54:54] er? :-D [09:55:16] yeah maxclients 5 is defined when there is a define SLWO [09:55:18] SLOW [09:55:23] so that should be MaxClients 40 [09:57:11] nothing related in apache error logs [09:58:57] the apache2.conf comes from puppet so that should be fine hopefully [10:00:41] right [10:01:50] what does apache status say, anything remotely useful? [10:05:11] it is working [10:05:22] I would need some extended error logs though [10:05:26] to find out why it reject connections [10:07:48] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [10:08:12] I can look with you in a little while (hopefully) [10:10:53] ah at least I found out the reason for SIGTERM [10:10:54] notice: /Stage[main]/Applicationserver::Service/Service[apache]: Triggered 'refresh' from 1 events [10:11:00] thanks puppet [10:11:26] is there anything more complex and confusing in the universe than LDAP? [10:11:27] heh [10:11:42] *cough* x.509 + asn1 [10:11:50] worst. idea. ever. [10:12:06] heh, good call [10:12:09] but ldap is way up there, I gotta give you that [10:12:39] does all this also relate to the fact that oauth is not working ? [10:19:59] GerardM: no [10:20:52] giving up on apache restarting, will RT it [10:25:43] hashar: bonjour. The Flow deploy broke Flow on beta labs. We think we have a fix for wmf-config/InitialiseSettings-labs.php in https://gerrit.wikimedia.org/r/#/c/100945/ . Do you usually +2 and deploy these, or is it only Demon/Reedy ? [10:28:49] spagewmf: anyone with +2 access would randomly approve the -labs.php changes [10:29:09] spagewmf: I do self merge often :-D [10:29:23] though only if I am 1000% sure it is not going to impact production hehe [10:30:28] hashar right, but ^demon & Roan say not to leave merged config patches undeployed [10:30:34] (03CR) 10Hashar: [C: 032] "+2 and it will self deploy on beta. Then one need to git pull on tin.eqiad.wmnet to avoid people freaking out because something did not g" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 (owner: 10Bsitu) [10:30:38] spagewmf: yeah replied there [10:30:40] so basically : +2 [10:30:43] (03Merged) 10jenkins-bot: Fix Flow config setting for beta labs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/100945 (owner: 10Bsitu) [10:30:46] jenkins will deploy on beta cluster [10:31:01] then I go on tin.eqiad.wmnet and pull the beta only change [10:31:09] this way people get a clean git status on tin [10:31:17] doing it right now [10:32:10] hashar, thanks! I didn't know that jenkins deploys config changes as well as core and extension updates. [10:32:12] ori-l: what ldap thing are you working on? [10:32:20] (out of curiosity) [10:32:24] graphite [10:32:39] access control? [10:32:42] spagewmf: well Jenkins report so on the change :-D [10:32:48] yeah [10:32:52] sorry... [10:33:00] spagewmf: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update/1659/console : Change has been deployed on the beta cluster in 13s [10:33:14] spagewmf: the related documentation is https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated [10:33:29] spagewmf: anyway change got deployed and tin is up to date :-] So you are safe now [10:33:50] apergos: hm? [10:34:03] oh that you get stuck with that [10:34:42] hashar: right, it's the "101% sure it is not going to impact production" that gets me. We deployed other labs changes in our deployment window (that didn't quite work, hence this patch :) ) [10:34:45] i sort of did it to myself :) [10:34:47] anyway, thanks [10:35:06] spagewmf: and if in doubt, get review from other folks as usual :-] [10:36:02] the best lack all conviction [10:38:32] PROBLEM - Puppet freshness on elastic1007 is CRITICAL: Last successful Puppet run was Wed 11 Dec 2013 10:33:16 AM UTC [10:39:10] doo dee doo dee doo [10:39:21] * apergos is not loving ruby [10:39:34] oo oo what ruby? [10:39:39] meh [10:39:44] 'doo dee doo dee doo' is probably valid ruby [10:39:49] :-D [10:40:18] oh just trying to beat a script into submission [10:40:38] I'm close to winning, just not excited about it [10:42:04] paravoid: akosiaris: apergos: any clue why a pure virtual package ends up not being marked as installed ? [10:42:22] what test are you using? [10:42:33] We have a package { 'fonts-sil-yi': ensure => present } [10:42:36] but that is never installed [10:42:59] so puppet reinstall it and triggers an event that escalate up to Service['apache'] which get restarted :( [10:43:14] it reinstalls every run? [10:43:21] yep [10:43:21] well, pretends to reinstall [10:43:31] puppet log shows no complaints about it? [10:44:19] * apergos is going to get downright crabby about this ruby thang [10:45:18] na it keep reinstalling it [10:45:27] apparently purely virtual package are some kind of redirection [10:45:35] the real package being installed is fonts-sil-nuosusil [10:45:48] (03CR) 10Addshore: "2 different ways as in the single repo on labs and multiple repos on prod?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/95996 (owner: 10Aude) [10:47:01] hashar: http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA working again, merci beaucoup [10:49:22] spagewmf: bravo! [10:49:45] (03PS1) 10Hashar: imagescaler: replace virtual font package with real package [operations/puppet] - 10https://gerrit.wikimedia.org/r/100970 [10:51:26] apergos: https://gerrit.wikimedia.org/r/100970 removes the purely virtual package and use the real one instead [10:52:44] ok but doesn't .. I mean dpkg -l doesn't show the virtual package installed? [10:53:02] pasted on https://rt.wikimedia.org/Ticket/Display.html?id=6500 [10:53:09] it does: [10:53:10] dpkg-query -W --showformat '${Status} ${Package} ${Version}\n' fonts-sil-yi [10:53:13] err puppet do that [10:53:18] and that returns unknown ok not-installed fonts-sil-yi [10:53:30] so version being 'not-installed' puppet install it as a good citizen [10:53:35] the package is http://packages.ubuntu.com/precise/fonts-sil-yi [10:53:42] apparently the font got renamed [10:53:51] it was probably valid in Lucid, no more the case today [10:54:00] we probably have had that issue since we migrated to Precise :-D [10:56:23] I am not sure it happens in production though [10:59:59] away for some coding dojo [11:00:04] bb in 2 hours [11:00:06] ok [11:00:49] * apergos goes back to kicking ruby [11:09:33] !log labsdb1002:3307 db instance crash, page corruption, now assertion failures. restoring from upstream db snapshot [11:09:50] Logged the message, Master [11:11:26] bah nodojo today apparently :D [11:20:50] !gerrit I6b9c47055e7ee232cec3fb1f9f5d3a15d4ad392d [11:20:51] https://gerrit.wikimedia.org/ [11:21:00] !gerrit I6b9c47055e7ee232cec3fb1f9f5d3a15d4ad392d [11:21:01] https://gerrit.wikimedia.org/ [11:21:03] .. [11:21:10] !alias gerrit [11:21:12] !help gerrit [11:21:13] !documentation for labs !wm-bot for bot [11:21:16] !gerrit [11:21:17] https://gerrit.wikimedia.org/ [11:22:22] (03PS2) 10Hashar: imagescaler: replace virtual font package with real package [operations/puppet] - 10https://gerrit.wikimedia.org/r/100970 [11:22:55] (03CR) 10Hashar: "added reference to I6b9c47055e7ee232cec3fb1f9f5d3a15d4ad392d by Reedy : Update font packages to not use virtual packages" [operations/puppet] - 10https://gerrit.wikimedia.org/r/100970 (owner: 10Hashar) [11:23:09] apergos: I think you can go ahead and merge that change :D [11:23:27] anyway, will be back in 2 hours :-D [11:23:47] I want to see what's going on in production first [11:23:53] sure [11:24:07] couldn't find an occurrence of SIGTERM in apache log, but I might be looking at the wrong file [11:24:25] I have amended the change to refer to a previous commit that changed the fonts package to no more point to virtual packages [11:24:25] I will do it in a while, just... still in my ruby death match, this is dragging on much longer than expected [11:24:31] right [11:24:31] good luck with ruby :-D [11:24:32] off ! [11:24:33] thanks [11:24:35] see ya [11:42:48] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [11:46:29] (03PS1) 10Springle: assign db1017 to s5 for analytics [operations/puppet] - 10https://gerrit.wikimedia.org/r/100973 [11:48:18] (03CR) 10Springle: [C: 032] assign db1017 to s5 for analytics [operations/puppet] - 10https://gerrit.wikimedia.org/r/100973 (owner: 10Springle) [11:57:12] !log xtrabackup clone s5 db1005 to db1017 [11:57:29] Logged the message, Master [11:58:32] (03CR) 10Ori.livneh: [C: 032] Decouple Nginx configs from graphite::web & relegate to subclass [operations/puppet] - 10https://gerrit.wikimedia.org/r/100957 (owner: 10Ori.livneh) [11:59:35] (03CR) 10Akosiaris: [C: 032] include the bugzilla config in puppet [operations/puppet] - 10https://gerrit.wikimedia.org/r/100752 (owner: 10Dzahn) [12:00:12] (03PS1) 10Ori.livneh: Use Apache to serve Graphite [operations/puppet] - 10https://gerrit.wikimedia.org/r/100974 [12:01:21] (03PS2) 10Ori.livneh: Use Apache to serve Graphite [operations/puppet] - 10https://gerrit.wikimedia.org/r/100974 [12:01:43] mutante: hi [12:01:54] mutante: I have a small question about a repo that disappeared from gerrit [12:02:14] (03CR) 10Ori.livneh: [C: 032 V: 032] Use Apache to serve Graphite [operations/puppet] - 10https://gerrit.wikimedia.org/r/100974 (owner: 10Ori.livneh) [12:02:17] mutante: actually, some changes that disappeared from gerrit and together with them, also the repos on git.wikimedia.org [12:02:25] mutante: ofc not all, just one in particular [12:03:33] mutante: is there any reason why a gerrit change would just vanish ? [12:03:46] mutante: this is the gerrit change http://web.archiveorange.com/archive/v/RcW9ZCDoEjVWWzbob2aI [12:04:04] mutante: https://gerrit.wikimedia.org/r/73860 [12:04:59] and the underlying repo also disappeared gerrit.wikimedia.org:29418/analytics/dclass [12:05:16] fortunately, I still have copies of this on my machine [12:05:50] the gerrit change disappearing is probably the result of the repo not being there [12:06:03] why the repo is not there ... i do not know at this time [12:06:29] akosiaris: that's ok, this is not urgent [12:06:48] PROBLEM - mysqld processes on db1017 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [12:06:55] akosiaris , mutante when you find something please let me know [12:07:56] akosiaris: I would suspect that gerrit has a log somewhere with who performed what operations [12:08:55] akosiaris: but it's also a policy question. do we keep just the latest X months/days of gerrit merged changes ? or do we keep all of them ? [12:09:10] all of them [12:10:16] when did the repo disappear ? [12:10:36] (03PS1) 10Ori.livneh: Decouple Nginx from Gdash module; serve via Apache instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/100975 [12:11:11] (03CR) 10Ori.livneh: [C: 032 V: 032] Decouple Nginx from Gdash module; serve via Apache instead [operations/puppet] - 10https://gerrit.wikimedia.org/r/100975 (owner: 10Ori.livneh) [12:11:18] RECOVERY - mysqld processes on labsdb1003 is OK: PROCS OK: 3 processes with command name mysqld [12:11:55] akosiaris: I first noticed about 1-2 months ago [12:12:18] heh... logs go back 10 days... [12:12:54] (03PS1) 10Ori.livneh: Serve gdash.wikimedia.org via Apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/100976 [12:13:17] akosiaris: I see.. [12:13:39] SAL also does not have any entry for dclass being deleted [12:14:17] akosiaris: how about the git repo ? [12:14:20] (03PS2) 10Ori.livneh: Serve gdash.wikimedia.org via Apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/100976 [12:14:35] (03CR) 10Ori.livneh: [C: 032 V: 032] Serve gdash.wikimedia.org via Apache [operations/puppet] - 10https://gerrit.wikimedia.org/r/100976 (owner: 10Ori.livneh) [12:14:44] akosiaris: ssh://gerrit.wikimedia.org:29418/analytics/dclass [12:15:15] not there [12:16:28] (03PS1) 10Ori.livneh: Correct typo in template path reference introduced in Ib87eb5dbc [operations/puppet] - 10https://gerrit.wikimedia.org/r/100977 [12:17:26] anyone online i might be able to chat with about throttling job queue jobs? [12:17:36] (03CR) 10Ori.livneh: [C: 032] Correct typo in template path reference introduced in Ib87eb5dbc [operations/puppet] - 10https://gerrit.wikimedia.org/r/100977 (owner: 10Ori.livneh) [12:18:37] sigh, i really shouldn't rush [12:19:32] (03PS1) 10Ori.livneh: Correct file reference in Ib87eb5db [operations/puppet] - 10https://gerrit.wikimedia.org/r/100978 [12:19:39] sorry for git spam :/ [12:19:47] (03CR) 10Ori.livneh: [C: 032 V: 032] Correct file reference in Ib87eb5db [operations/puppet] - 10https://gerrit.wikimedia.org/r/100978 (owner: 10Ori.livneh) [12:21:49] PROBLEM - DPKG on tungsten is CRITICAL: DPKG CRITICAL dpkg reports broken packages [12:22:48] RECOVERY - DPKG on tungsten is OK: All packages OK [12:25:53] (03PS1) 10ArielGlenn: write out attributes for a nagios service in fixed order [operations/puppet] - 10https://gerrit.wikimedia.org/r/100979 [12:26:05] (03PS2) 10ArielGlenn: write out attributes for a nagios service in fixed order [operations/puppet] - 10https://gerrit.wikimedia.org/r/100979 [12:26:42] gerrit rebase button, best thing since sliced bread [12:28:05] (03CR) 10ArielGlenn: [C: 032] write out attributes for a nagios service in fixed order [operations/puppet] - 10https://gerrit.wikimedia.org/r/100979 (owner: 10ArielGlenn) [12:29:44] apergos: @parameters.sort.map { |param,value| [12:29:55] sorts by key anyway [12:30:26] this is directly from their branch however [12:31:11] well... :) [12:31:33] I was wasting a lot of time trying to get the to_s to work before realizng that they had redefined it :-/ [12:33:09] I now have to do two puppet runs on neon just to confirm (though I did some testing already writing to files in a different location) [12:33:21] and neon puppet runs are still the slowest except maybe bast1001 [12:34:16] is map faster than each and retrieval of the value as they do here? [12:34:25] (might as well learn something while we're at it) [12:34:32] no, just more idiomatic if you need the value too [12:34:54] actually, durr, each can do |k,v| too [12:34:59] yes it can [12:35:00] that's even more straightforward [12:35:12] I don't know why they didn't here [12:36:18] each can do |k,v| ? [12:36:22] isn't that each_pair ? [12:36:54] I don't know each_pair, but I know each does it [12:37:22] * apergos looks it up [12:38:37] http://www.ruby-doc.org/core-1.8.7/Hash.html#method-i-each [12:38:42] seems like the same [12:38:47] it does, doesn't it [12:39:02] marginally more efficient [12:39:07] whatever that means [12:39:12] meh [12:39:44] heh... I 've never used each for hashes... I always used each_pair [12:40:17] which btw returns a different order of items in every execution of puppet [12:40:34] yes, we know :-D [12:40:41] ruby 1.8 ftw (not!) [12:40:53] I"m fixing one of those right now, hopefully [12:41:10] yeah i fixed one of those yesterday... [12:41:41] meh.. the ruby 1.8 => 1.9 change was too big for a minor version number [12:42:00] what else got changed? [12:42:09]