[00:08:05] New review: Dzahn; "the current version in the repo does not use "infile" at all anymore. that part now looks more like:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37570 [00:16:50] New patchset: Odder; "(bug 27911) Standardise Apple Touch icon, take 2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60777 [00:17:39] Change abandoned: Odder; "The philosophy is changed with I2f0990fa087a7b2969c982d05187331e7005b64a" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60773 [00:18:43] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 212 seconds [00:20:41] odder: still dealing with all that? [00:20:54] thanks :) [00:21:31] New patchset: Odder; "(bug 27911) Standardise Apple Touch icon, take 2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60777 [00:21:48] odder: fwiw, I think that saying to enwiktionary people that they can't have their own apple touch icon because of technical limitations [00:21:56] and that maybe they should just agree on one for the whole project [00:22:00] is a perfectly fine answer [00:22:04] paravoid: Krinkle had good arguments to keep the icons in docroot [00:22:26] defining them via *should* work too... [00:22:47] ... but since this !@##$% system will hit apple-touch-icon.png anyway... :-) [00:22:59] paravoid: I do not want to start WW3 over that little detail. [00:23:04] heh [00:24:00] * odder wonders where to document why we're keeping those icons in docroot [00:24:44] you could use steghide to put text into the icons themselves .. ;) /me hides [00:25:06] odder: README? [00:25:11] there is one in the repo [00:25:18] or am I the only one writing/reading that? [00:25:38] it can be extended with information about favicons as well [00:25:43] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [00:26:26] Oh. There is indeed a README there. [00:29:26] !log DNS update - removing internproxy, removing tesla (minus mgmt) [00:29:34] Logged the message, Master [00:30:38] wheee [00:36:45] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [00:38:05] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.68 ms [00:38:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 224 seconds [00:39:45] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [00:39:45] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [00:39:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 4 seconds [00:40:05] PROBLEM - Apache HTTP on mw31 is CRITICAL: Connection refused [00:44:40] !log upgrading packages on manutius [00:44:51] Logged the message, Master [00:44:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 218 seconds [00:45:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [00:49:05] RECOVERY - Apache HTTP on mw31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.622 second response time [00:59:55] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 182 seconds [01:00:55] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [01:28:52] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 224 seconds [01:29:32] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [01:31:52] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [01:44:52] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 209 seconds [01:48:53] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [02:08:53] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 190 seconds [02:11:22] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [02:12:53] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [02:14:21] !log LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013 [02:14:30] Logged the message, Master [02:23:03] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [02:25:03] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [02:28:53] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 202 seconds [02:31:52] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [02:35:07] PROBLEM - SSH on amslvs1 is CRITICAL: Server answer: [02:36:07] RECOVERY - SSH on amslvs1 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [02:38:56] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 201 seconds [02:40:56] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 22 seconds [02:42:31] New review: Krinkle; "The current repo also says wikibugs is in #wikimedia-dev but it is in #mediawiki still." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/37570 [03:13:49] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [03:14:48] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 26 seconds [03:24:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 210 seconds [03:28:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [03:33:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [03:39:45] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013 [03:39:52] Logged the message, Master [03:41:51] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [03:48:51] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 188 seconds [03:51:51] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 23 seconds [04:13:49] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 225 seconds [04:14:48] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 11 seconds [04:23:48] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 200 seconds [04:24:48] RECOVERY - MySQL Slave Delay on [06:34:50] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 209 seconds [06:36:29] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 30 seconds [09:47:18] New patchset: Matthias Mullie; "Config var was confusing; all others are ArticleFeedbackv5 prefixeg" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60803 [10:39:52] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [10:39:52] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:19] New patchset: Mark Bergsma; "Remove temporary hack to ensure X-Analytics header on responses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60814 [11:22:40] ~. [11:23:54] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60814 [11:30:23] PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours [11:31:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:32:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [11:52:23] New patchset: Krinkle; "Move Apple Touch icons to docroots" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60777 [11:52:31] Change merged: Krinkle; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60777 [11:55:32] !log krinkle synchronized docroot [11:55:41] Logged the message, Master [11:56:15] !log krinkle synchronized wmf-config/InitialiseSettings.php 'I2f0990fa087a' [11:56:23] Logged the message, Master [12:01:57] New patchset: Mark Bergsma; "Sync new ganglia clusters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60823 [12:04:02] jenkins [12:04:05] where aaaaaare you [12:04:28] maybe you should ask jeeves [12:04:55] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60823 [12:12:03] PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours [12:12:27] New patchset: Mark Bergsma; "Spaces to tabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60824 [12:13:58] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60824 [12:14:02] jenkins went to bed i think [12:14:17] it's an overworked dude, should have some rest [12:17:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:18:34] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.120 second response time [12:20:19] New patchset: Mark Bergsma; "Use manutius as first ganglia aggregator host on pmtpa bits caches" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60826 [12:20:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60826 [12:31:06] New patchset: Mark Bergsma; "Add pmtpa suffix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60827 [12:31:27] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60827 [12:37:13] New patchset: Mark Bergsma; "Add nickel as gmetad host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60828 [12:37:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60828 [12:39:06] Krinkle: whee, no more 404s [12:39:18] and smaller html :) [12:39:37] Yes :) [12:39:50] 0.2%, wasn't it? :) [12:43:38] nice job :) [13:16:28] New patchset: Mark Bergsma; "Restart aggregator instances on corresponding config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60834 [13:17:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60834 [13:25:34] New patchset: Mark Bergsma; "Test the new ganglia monitor classes with pmtpa bits caches" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60835 [13:26:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60835 [13:30:40] New patchset: Mark Bergsma; "Add file type for python_modules directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60836 [13:31:26] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60836 [13:32:33] @seen Isarra [13:32:33] odder: Last time I saw Isarra they were quitting the network with reason: *.net *.split N/A at 4/25/2013 11:29:17 AM (02:03:16.0109100 ago) [13:32:41] osome [13:33:31] @notify Isarra [13:33:31] I will notify you, when I see Isarra around here [13:33:51] New patchset: Mark Bergsma; "Add file type for /etc/ganglia/conf.d as well" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60837 [13:34:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60837 [13:37:35] New patchset: Mark Bergsma; "Add alias gmond for ganglia-monitor service" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60838 [13:37:53] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60838 [13:58:29] James_F: will you give me a poke when deployment is done? [13:58:55] Lydia_WMDE: Of course. [13:59:00] :) [13:59:10] currently preparing Kurier article based on your blog [14:12:03] James_F: //en.wikipedia.org/w/index.php?title=Special:UserLogin&type=signup&campaign=VE in the blog should be changed to de.wp and should work right? [14:12:12] Lydia_WMDE: Yeah. [14:12:17] thx [14:12:27] Lydia_WMDE: The "campaign=VE" stuff works everywhere. [14:12:32] cool [14:13:34] Lydia_WMDE: And can be used for "campaign=Wikidata" if you want. :-) [14:13:40] hehe [14:15:54] James_F: the page for feedback will be Wikipedia:VisualEditor/Feedback? [14:17:29] Lydia_WMDE: We can set it to what you want. [14:17:39] Lydia_WMDE: But yes, that was what was agreed for dewiki. :-) [14:20:39] New review: Ottomata; "> Should we call the kafka::server class kafka::broker instead? I'm not sure." [operations/puppet/kafka] (master) - https://gerrit.wikimedia.org/r/50385 [14:21:25] New patchset: Ottomata; "Initial commit of Kafka Module." [operations/puppet/kafka] (master) - https://gerrit.wikimedia.org/r/50385 [14:22:13] k just making sure it links to the right page [14:26:08] New patchset: Mark Bergsma; "Add a field "sites" to the ganglia cluster configuration" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60841 [14:26:08] New patchset: Mark Bergsma; "Support aggregators responsible for multiple sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60842 [14:27:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60841 [14:31:22] New patchset: Mark Bergsma; "Support aggregators responsible for multiple sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60842 [14:31:22] New patchset: Mark Bergsma; "Resolve ganglia resource conflicts on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60845 [14:32:42] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60845 [14:35:49] New patchset: Mark Bergsma; "Support aggregators responsible for multiple sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60842 [14:38:34] New patchset: Mark Bergsma; "Support aggregators responsible for multiple sites" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60842 [14:40:31] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60842 [14:42:19] New patchset: Demon; "Add second key for myself" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60846 [14:43:05] New patchset: Demon; "Add second key for myself" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60846 [14:44:12] New patchset: Hashar; "contint: 100% passing puppet-lint" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60851 [14:46:03] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [14:46:03] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [14:46:03] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [14:50:03] New patchset: Mark Bergsma; "Add missing parser function suffix() from puppet stdlib 4.x" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60853 [14:51:34] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60853 [14:52:21] !log demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'Debugging' [14:52:29] Logged the message, Master [14:53:22] !log demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log on' [14:53:35] Logged the message, Master [14:54:31] New patchset: Mark Bergsma; "Define $cluster variable before reference" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60854 [14:55:35] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60854 [14:55:45] !log demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log back off, its not phps fault' [14:55:53] Logged the message, Master [14:56:38] !log demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'rm debug hack' [14:56:45] Logged the message, Master [14:56:51] Krinkle: https://s3.amazonaws.com/f.cl.ly/items/0U2B2m0T3J1V2O2y2o2x/2013-04-25%2009.52.04.png [14:57:00] legoktm: you're famous now [14:57:06] o.O [14:57:46] odder: yes? [14:58:15] Krinkle: legoktm was so kind as to provide an example screenshot of how the apple-touch-icon looks like [14:58:22] ok [14:58:25] this is the 144px icon scaled down to 72px [14:58:49] both? [14:59:09] errrm — 114px [14:59:51] odder: both commons and enwiki are resized? [15:00:09] Not sure what you want to indicate? Good? bad? [15:00:16] Krinkle: the Commons icon is actually the mobile app [15:00:23] ok [15:00:32] currently the Commons icon is 57x57 [15:00:39] Krinkle: looks good to me... [15:00:57] odder: right [15:01:17] and it's just 5985 bytes [15:01:24] https://bugzilla.wikimedia.org/show_bug.cgi?id=27911 [15:01:40] [that's what prompted me to ask legoktm to test it in the first place] [15:04:22] New patchset: Mark Bergsma; "Strip the site suffix from the title to retrieve the cluster name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60855 [15:06:16] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60855 [15:08:58] New patchset: Mark Bergsma; "Run aggregator instances for eqiad ganglia clusters on manutius" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60857 [15:09:15] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60857 [15:12:31] New patchset: Mark Bergsma; "Rename $site_port_prefix to $id_prefix, and use it for gmond instance IDs as well" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60859 [15:13:03] New patchset: Demon; "Return empty searches with HTTP code 200" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [15:13:48] PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours [15:14:01] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60859 [15:15:32] New review: Demon; "This is untested beyond "does it compile" (it does)." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [15:15:57] (it doesn't) would have been funnier [15:16:19] easily fixable:P [15:16:48] <^demon> xyzram: could you take a look at https://gerrit.wikimedia.org/r/#/c/60860/? [15:17:40] ^demon: sure [15:18:16] New patchset: Mark Bergsma; "Operator precedence!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60861 [15:18:43] is it possible to do https://gerrit.wikimedia.org/r/#/c/55155/ in a shorter way (and a way that actually works without parameters)? [15:21:04] New patchset: Mark Bergsma; "Expressions and selectors do not go along" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60861 [15:22:35] <^demon> mark: Damn, that was exactly what I wanted to get away with doing: https://gerrit.wikimedia.org/r/#/c/60302/1/manifests/generic-definitions.pp [15:23:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60861 [15:23:33] you can't do that simply because puppet doesn't allow you to redefine variables [15:23:57] if you make that $newname = ... it would work [15:26:07] there we go... [15:28:07] New patchset: Mark Bergsma; "Calculate the id/portnr on the monitors matching the aggregators" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60865 [15:29:33] New patchset: Demon; "Deprecate $name param to systemuser in favor of $title" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60302 [15:30:17] New patchset: Hashar; "Jenkins job validation (DO NOT SUBMIT)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60866 [15:33:04] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60865 [15:40:52] New patchset: Hashar; "Jenkins job validation (DO NOT SUBMIT)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60866 [15:46:25] New patchset: Hashar; "Jenkins job validation (DO NOT SUBMIT)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60866 [15:47:29] New review: Ram; "Since the top-level 'if' checks that res is not null, adding the same check in the newly added code ..." [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [15:48:36] ^demon: Just added a comment to that patch. [15:49:32] Change abandoned: Hashar; "(no reason)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60866 [15:51:36] I am about to push out new Parsoid code, so please ignore Parsoid-related warnings in the next minutes [15:55:49] went well and without warnings [16:05:18] we love you hashar [16:05:52] New patchset: Ottomata; "Initial debian packaging using git-buildpackage" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [16:06:05] mark: :-] [16:06:41] +1 [16:08:28] !log Zuul git operations are slow again :/ [16:08:35] Logged the message, Master [16:09:49] PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours [16:09:49] PROBLEM - Puppet freshness on magnesium is CRITICAL: No successful Puppet run in the last 10 hours [16:11:18] yeahhh that's awesome hashar! [16:11:33] that is really basic erb linting though [16:11:43] we will see next week how well it went [16:14:34] hashar, are you dealing with -%> ? [16:14:46] i've have trouble using ERb as is on templates with -%> [16:15:25] i had to modify your expanderb.rb script with this to make it not die: [16:15:25] def render_erb(template, locals) [16:15:25] # make sure we can render templates with -%> closing tags. [16:15:26] # Note that this does not actually remove newlines after -%>, it just [16:15:26] # avoids a syntax error. [16:15:26] ERB.new(template, nil, '%<>-').result(OpenStruct.new(locals).instance_eval { binding }) [16:15:26] end [16:17:09] ottomata: feel free to push a change to ops/puppet.git and find out what the result is [16:17:11] :) [16:17:17] I barely tested it [16:19:00] JENKINS I AM gOING TO KiLL Youuuuuu [16:19:01] :( [16:19:23] <^demon> I think hashar needs the rage hat now. [16:19:38] !log Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/ Should take roughly 40 minutes (no idea when it started) [16:19:45] Logged the message, Master [16:21:33] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:21:33] <^demon> xyzram: If res is null, won't we get a NPE in the else block anyway? ;-) [16:22:07] I am off for now [16:22:11] gallium would be fine :-] [16:22:12] <^demon> Oh, I guess the larger catch for Exception gets it, but bleh. [16:22:19] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.133 second response time [16:28:04] ^demon: yes. Not clear if res can ever be null (most places seem to allocate a new results object and return that) but I didn't do an exhaustive analysis since there are so many places. [16:28:21] <^demon> Fair enough, easy to check for. [16:28:23] <^demon> Will amend now [16:29:44] New patchset: Demon; "Return empty searches with HTTP code 200" [operations/debs/lucene-search-2] (master) - https://gerrit.wikimedia.org/r/60860 [16:36:58] New patchset: Ottomata; "Initial commit of Kafka Module." [operations/puppet/kafka] (master) - https://gerrit.wikimedia.org/r/50385 [16:37:48] !log catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master' [16:38:00] Logged the message, Master [16:38:29] New patchset: Ottomata; "Initial commit of Kafka Module." [operations/puppet/kafka] (master) - https://gerrit.wikimedia.org/r/50385 [16:40:53] average, around? [16:42:43] Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60293 [16:45:41] New patchset: MaxSem; "Restore the ability to use misc::maintenance classes without parameters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60879 [16:49:50] !log catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 14 more Wikipedias' [16:49:57] Logged the message, Master [16:56:46] New patchset: MaxSem; "Restore the ability to use misc::maintenance classes without parameters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60879 [16:56:52] can someone please take a look at ^^^ ? [16:59:05] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/58523 [17:00:04] !log demon synchronized wmf-config/CommonSettings.php 'Removing oldskin notice' [17:00:12] Logged the message, Master [17:02:41] New patchset: Ryan Lane; "Initial puppetization for git.wikimedia.org" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59857 [17:04:03] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59857 [17:05:07] New patchset: Ryan Lane; "Remove old github detection hack, that's what remoteNameStyle is for" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59928 [17:05:35] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59928 [17:05:46] New patchset: Ryan Lane; "Begin replicating all repositories to antimony" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59927 [17:05:55] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/59927 [17:06:08] New patchset: Ryan Lane; "Swap gerrit to new gerrit.wikimedia.org certificate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60688 [17:08:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [17:09:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time [17:10:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [17:17:56] New patchset: Ottomata; "Initial debian packaging using git-buildpackage" [operations/debs/kafka] (master) - https://gerrit.wikimedia.org/r/53170 [17:19:49] New patchset: Demon; "Swap gerrit to new gerrit.wikimedia.org certificate" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60688 [17:29:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 231 seconds [17:30:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 19 seconds [17:54:25] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:55:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time [17:57:31] New review: Ram; "Looks good." [operations/debs/lucene-search-2] (master) C: 1; - https://gerrit.wikimedia.org/r/60860 [17:58:26] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:59:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [18:17:30] Reedy: i saw an old "samreed" directory on fenari - i'mg uessing old username for you ? [18:18:04] I think so.. Not sure I ever logged on as that account hough [18:18:23] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 199 seconds [18:18:46] okay [18:18:47] BALEETED! [18:18:54] !log cleaning up old home directories on fenari [18:19:02] Logged the message, Mistress of the network gear. [18:19:56] New patchset: Bsitu; "Fix echo digest email cron script path" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60893 [18:20:23] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 3 seconds [18:26:13] RECOVERY - NTP on magnesium is OK: NTP OK: Offset -0.01098251343 secs [18:28:00] New review: Bsitu; "This change should only be deployed after https://gerrit.wikimedia.org/r/#/c/52745/ is deployed" [operations/puppet] (production) C: -1; - https://gerrit.wikimedia.org/r/60893 [18:43:23] New patchset: Reedy; "Lossless png compression" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60897 [18:43:35] Change abandoned: Reedy; "Replaced by https://gerrit.wikimedia.org/r/60897" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60380 [18:45:09] New review: Reedy; "reedy@ubuntu64-web-esxi:~/git/operations/mediawiki-config$ git pull ssh://reedy@gerrit.wikimedia.org..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60897 [18:45:54] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60897 [18:46:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: Connection refused by host [18:47:24] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [18:47:29] mutante, can you review https://gerrit.wikimedia.org/r/#/c/60879/ plz? [18:52:23] !log reedy synchronized docroot [18:52:31] Logged the message, Master [18:52:35] MaxSem: is "needed by labs" more important than having nicer parameterized classes. the change you say broke it seems to be useful for the terbium/hume replacement. like what does it break currently [18:53:02] wish py would be here to comment [18:53:43] mutante, it doesn't remove the parameterization, it simply makes default parameter values not to cause puppet to die a horrible death [18:57:01] notpeter, what do you think of https://gerrit.wikimedia.org/r/#/c/60879/ ? ^^^ [18:59:24] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 195 seconds [19:00:24] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [19:08:22] Change merged: Kaldari; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60743 [19:11:54] !log kaldari synchronized wmf-config/CommonSettings.php 'updating for new schema number' [19:12:01] Logged the message, Master [19:14:29] heya paravoid, you around? i know you are real busy, but I [19:14:44] i'm supposed to start working on the puppet cdh4 stuff, and I can work on it while we wait for you to have time for kafka review [19:16:03] but, before I delve into it, i would appreciate a quick brain bounce as to how to structure it. mainly: should I actually have a cdh4 module with all the subcomponents, or a buncha smaller modules [19:22:38] ottomata: umm, I can't tell without knowing more... [19:23:32] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 191 seconds [19:24:15] right would like to brain bounce with you about it [19:24:30] i'm not entirely sure either, i just need talk about the setup and what i'm thinking [19:24:32] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 14 seconds [19:24:48] we can always change later i guess, i just want to bounce the design around before I get too far into it all by myself [19:31:23] New patchset: Odder; "Add Orikrin1998 to the French Planet Wikimedia" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/60902 [19:46:48] didn't we switch to mariaDB? [19:46:55] need to update icinga messages :) [19:47:11] we have both [19:47:51] guess it should be M*DB than :) [19:49:19] my vote is for "an error has occured" [19:49:37] nonosql [19:50:31] db1025 is still mysql IIRC [19:50:46] yeah it's the facebook version [19:50:50] and it's being stupid [19:51:04] ack. 5.1.53-fb3753-wm1 [19:51:20] Jeff_Green: did you forget to tell it how you were feeling? [19:51:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:36] Jeff_Green is annoyed at db1025 -- feeling pissed [19:51:52] pgehres: i yelled at the ssh terminal, does that count? [19:52:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.137 second response time [19:57:38] about to run scap [20:04:18] MaxSem: got a host/build where it is dying ? [20:04:27] i'd like to see if i can fix this without a major revert [20:06:24] LeslieCarr, I just need to use it w/o parameters - a repro would be to apply use any of these classes unparameterized on labs [20:07:43] can i use one of your instances (run puppet on it) where this is occuring ? [20:08:08] a hackier way to fix would be to change it to inline_template("<%= \"$::site\" == \"$::mw_primary\" %>") but it's scary enough already [20:08:42] hah, scary [20:09:19] 1sec, preparing an instance [20:09:54] RECOVERY - Puppet freshness on magnesium is OK: puppet ran at Thu Apr 25 20:09:49 UTC 2013 [20:10:24] LeslieCarr, try on deployment-el-test [20:10:49] it reproduces the error on current production tip now [20:11:24] cool [20:11:30] err: Could not retrieve catalog from remote server: wrong header line format ? [20:12:38] yes [20:13:27] New patchset: Jforrester; "Clean up VisualEditor settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60911 [20:13:28] forgot, the ^^ hack didn't actually work in my testing:P [20:13:33] New review: Ori.livneh; "This looks good. Thanks for parametrizing the rsync destination, that's a nice improvement." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/60680 [20:14:21] ...probably because template() always returns strings [20:14:52] New patchset: Matmarex; "(bug 45979) Set $wgCategoryCollation to 'uca-vi' on all Vietnamese-language wikis" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/60705 [20:16:07] or possibly One of your templates contains invalid syntax. Look for unintentional or invalid "<%" and ">%" strings. If you need to put these strings in templates you can use "<%%" and ">%%". ? [20:16:25] !log kaldari Started syncing Wikimedia installation... : [20:16:33] Logged the message, Master [20:17:05] inline templates are interesting ... [20:17:46] LeslieCarr, that's because it expands to pmtpa == [20:18:12] yes, pmtpa not quoted and compared to nothing because the variable name is wrong [20:19:08] yeah, i don't see a primary_site variable yet … [20:19:10] in my looking [20:19:28] cause it's called mw_primary;) [20:19:40] ah [20:20:08] well that would be a problem :) [20:20:28] first, i'm going to fix up that [20:20:56] i will make this work properly if it's the last thing i do! (ok, technically last thing i do will be the reverting.. but i figure it's worth trying ot make this work) [20:21:18] no it's not:) [20:21:22] haha [20:22:19] MaxSem: what are you hoping to accomplish with that patchset? [20:22:29] you can already use them without params [20:22:48] notpeter: it is actually breaking labs [20:22:57] err: Could not retrieve catalog from remote server: wrong header line format [20:23:58] when trying to call with enabled defined? [20:24:59] why not just define primary_site for the labs realm as well? [20:25:14] where's the patch that breaks? [20:25:15] especially if we're eventually going to migrate labs to eqiad [20:25:34] first of all, there's no primary_site on prod either, it's called mw_rpimary [20:26:12] second, if you set it you'll get an error saying that variable 'pmtpa' is not set [20:27:15] MaxSem: ...which patch / line / file? [20:27:15] uh [20:27:40] I see $mw_primary on line 38 of realm.pp of the current master [20:27:51] ori-l, https://gerrit.wikimedia.org/r/#/c/55155/ - my fix is at https://gerrit.wikimedia.org/r/#/c/60879/ [20:28:01] so, I have no idea where you're getting mw_rpimary [20:28:25] rpimary is a typo:) [20:28:26] that was just a typo, but it does appear to be mw_primary instead of primary_site [20:29:00] and it looks like it is then trying to evaluate whatever $site is as a variable instead of changing the value of $site [20:29:26] ok, this should be mw_primary, for sure [20:29:36] and actually just making a change for that now [20:29:54] MaxSem: do the crons not work in labs if you set enabled => true ? [20:30:07] and now … i think that we can actually do this without inline template …. because the class will be set at the node level [20:30:18] notpeter: the problem is the class won't compile [20:30:38] notpeter, that requires a self-hosted puppetmaster. it has to be usable by just applying a class [20:31:36] alright, whatevs [20:31:37]