[00:00:05] <jouncebot>	 RoanKattouw, ^d, marktraceur, MaxSem, kaldari: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141204T0000).
[00:00:36] <MaxSem>	 I can do it
[00:03:02] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Setting $wgMFEnableWikiGrokOnAllDevices to true on en Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177153 (owner: 10Kaldari)
[00:03:17] <grrrit-wm>	 (03Merged) 10jenkins-bot: Setting $wgMFEnableWikiGrokOnAllDevices to true on en Beta Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177153 (owner: 10Kaldari)
[00:04:22] <logmsgbot>	 !log maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/177153/ , noop in production (duration: 00m 07s)
[00:04:25] <morebots>	 Logged the message, Master
[00:04:26] <MaxSem>	 kaldari|2, ^^^
[00:04:51] <kaldari|2>	 MaxSem: testing...
[00:05:08] <MaxSem>	 tgr, yt?
[00:05:22] <MaxSem>	 legoktm, yt?
[00:05:31] <legoktm>	 MaxSem: hey
[00:05:48] <legoktm>	 my backports are just adding a maint. script
[00:10:53] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf10/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/177393 (duration: 00m 08s)
[00:10:57] <morebots>	 Logged the message, Master
[00:11:07] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf11/extensions/WikiGrok/: https://gerrit.wikimedia.org/r/177393 (duration: 00m 05s)
[00:11:09] <morebots>	 Logged the message, Master
[00:11:26] <hoo>	 greg-g: ori: TimStarling: We found the problems with sites in Wikibase. Patches are merged. Do you want the backport now or next week, or... ?
[00:11:31] <hoo>	 Two small patches
[00:11:54] <ori>	 how safe are the patches?
[00:11:56] <kaldari|2>	 MaxSem: that config change doesn't seem to have had any effect :(
[00:12:11] <Krinkle>	 Reedy: Did you stash the error log somewhere? (I assume it's disabled again)
[00:12:12] <hoo>	 One is extremely trivial
[00:12:16] <hoo>	 the other one rather trivial
[00:12:27] <Reedy>	 Krinkle: They've been rotated, so they're in the archive folder
[00:12:29] <hoo>	 ori: https://gerrit.wikimedia.org/r/177429
[00:12:32] * ori looks
[00:12:34] <andrewbogott>	 bd808: now labs_vagrant says "Fatal error: Class 'MWLogger' not found in /srv/vagrant/mediawiki/includes/GlobalFunctions.php on line 1107"
[00:12:38] <andrewbogott>	 Is that another role I need to add?
[00:12:39] <hoo>	 that's the not-that-trivial one
[00:12:43] <Krinkle>	 Reedy: OK
[00:12:46] <Krinkle>	 Reedy: And logstash?
[00:12:54] <Reedy>	 Not done yet for the error log
[00:13:05] <Krinkle>	 Reedy: look at git-status in wmf9 on tin.
[00:13:08] <Reedy>	 /a/mw-log/archive/error.log-20141118.gz
[00:13:11] <bd808>	 andrewbogott: you need to run composer I bet
[00:13:22] <ori>	 hoo: do it, imo
[00:13:25] <andrewbogott>	 ...ok...
[00:13:26] <hoo>	 Ok
[00:13:34] <andrewbogott>	 in vagrant?  I've never done that
[00:13:42] <Reedy>	 Krinkle: lol. We're not using wmf9 now at least. But yes, that really sucks
[00:13:49] <bd808>	 andrewbogott: Do you have a /vagrant/mediawiki/vendor directory?
[00:14:16] <andrewbogott>	 I don't
[00:14:51] <bd808>	 andrewbogott: The master branch of mw-vagrant would take care of this for you (at least on the first install) but the older branch doesn't know anything about composer dependencies
[00:15:45] <hoo>	 greg-g: you saw that?
[00:16:32] <andrewbogott>	 bd808: So vagrant is no longer maintained for precise at all, huh?
[00:17:19] <bd808>	 andrewbogott: nope
[00:17:33] <MaxSem>	 andrewbogott, no - you just need to run vagrant git-update instead of git pull
[00:18:05] <bd808>	 andrewbogott: The easier thing for you than using composer would be to clone mediawiki/vendor.git into your mediawiki checkout
[00:18:26] <bd808>	 MaxSem: He's on a 12.04 labs box
[00:19:24] <MaxSem>	 blegrgh
[00:19:31] <MaxSem>	 then vagrant destroy
[00:21:03] <grrrit-wm>	 (03CR) 10MaxSem: [C: 032] Force CommonsMetadata on beta to recalculate data from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177271 (owner: 10Gergő Tisza)
[00:21:25] <grrrit-wm>	 (03Merged) 10jenkins-bot: Force CommonsMetadata on beta to recalculate data from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177271 (owner: 10Gergő Tisza)
[00:21:54] <logmsgbot>	 !log maxsem Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/177271/ (duration: 00m 07s)
[00:21:59] <morebots>	 Logged the message, Master
[00:22:04] <MaxSem>	 tgr, please test ^^^
[00:22:47] <andrewbogott>	 this is broken in every way :(
[00:24:27] <MaxSem>	 omgbutwhy anyone couldpotentiallyneedprecise
[00:25:00] <Krinkle>	 Reedy: (sorry, I dropped for a sec, did you reply?)
[00:25:14] <Reedy>	 [00:13:42] <Reedy> Krinkle: lol. We're not using wmf9 now at least. But yes, that really sucks
[00:25:28] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf10/maintenance/removeInvalidEmails.php: https://gerrit.wikimedia.org/r/#/c/177021/ (duration: 00m 05s)
[00:25:30] <morebots>	 Logged the message, Master
[00:25:31] <Krinkle>	 k :)
[00:25:48] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf11/maintenance/removeInvalidEmails.php: https://gerrit.wikimedia.org/r/#/c/177021/ (duration: 00m 05s)
[00:25:50] <morebots>	 Logged the message, Master
[00:25:54] <MaxSem>	 legoktm, ^^^
[00:26:07] <legoktm>	 thanks!
[00:26:19] <tgr>	 MaxSem: I can't find a non-cached image to test on right now, but the change only affects beta, so no huge deal if it doesn't work
[00:26:21] <legoktm>	 I'll run the script later on
[00:26:30] <MaxSem>	 ok
[00:26:31] <tgr>	 thanks for deploying!
[00:26:39] <MaxSem>	 :)
[00:28:43] <MaxSem>	 Notice: Unable to unserialize: [a:12:{s:5:"width";i:1004;s:6:"height";i:1416... <pile of pdf stuff>
[00:30:53] <hoo>	 MaxSem: Want to push a Wikidata update in a bit? :P
[00:31:09] <MaxSem>	 if I have a permssion;)
[00:31:18] <hoo>	 Well, guess I can also do it myself :P
[00:31:29] <hoo>	 Last minute patching, ftw
[00:35:09] <hoo>	 MaxSem: https://gerrit.wikimedia.org/r/177446 and https://gerrit.wikimedia.org/r/177447 ? :)
[00:36:11] <MaxSem>	 still need someone to give a permission
[00:36:22] <MaxSem>	 ah whatever
[00:36:24] <MaxSem>	 SWAT
[00:36:33] <MaxSem>	 hoo, please add to calendar
[00:36:35] <hoo>	 I can +2 myself if that makes oyu more comfortable
[00:36:40] <hoo>	 will do
[00:37:28] <MaxSem>	 hoo, next time please mention commit being cherrypicked in summary
[00:37:39] <hoo>	 Yeah, forgot that :S
[00:37:45] <hoo>	 Will mention it on wiki
[00:39:41] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf10/extensions/Wikidata/: (no message) (duration: 00m 12s)
[00:39:44] <morebots>	 Logged the message, Master
[00:40:00] <logmsgbot>	 !log maxsem Synchronized php-1.25wmf11/extensions/Wikidata/: (no message) (duration: 00m 13s)
[00:40:03] <morebots>	 Logged the message, Master
[00:40:06] <MaxSem>	 hoo, ^^^
[00:40:19] <hoo>	 watching the graphs
[00:41:15] <hoo>	 awesome: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Memcached%20eqiad&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2&st=1417641558&g=network_report&z=large
[00:52:35] <AaronS>	 springle: is there any page table schema change going on?
[00:54:13] <springle>	 AaronS: none. last one was ~month ago
[00:55:18] <AaronS>	 huh, I'm seeing slow selects for page rows in it
[00:55:29] <AaronS>	 springle: e.g. go to a random page with ?forceprofile=true
[00:57:29] <springle>	 AaronS: i'm not seeing much difference with forceprofile. certain?
[00:58:27] <AaronS>	 do you see the output in view source?
[01:00:35] <AaronS>	 I  keep seeing 200-300ms for WikiPage::loadPageData ... though some of that may be hooks (though I don't see offenders)
[01:01:20] <grrrit-wm>	 (03PS2) 10EBernhardson: Disable LQT on office wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175563 
[01:01:39] <MaxSem>	 when I first saw the xhprof stats I thought they're bugged. still not sure they're not:P
[01:01:45] <springle>	 AaronS: if i'm reading the html commentcorrectly, i see about 30ms for WikiPage::loadPageData
[01:02:26] <AaronS>	 77.27% 223.074     19 - wfRunHooks on https://en.wikipedia.org/wiki/Domitianus_II?forceprofile=true
[01:02:39] <AaronS>	 I mean  83.83% 399.348      1 - WikiPage::pageData
[01:03:11] <MaxSem>	 91.70% 299.163      1 - MediaWiki::getAction <-- BS
[01:05:44] <AaronS>	 that ends up calling WikiPage::factory( $title ), which calls methods on the page, which make it load and do the query
[01:05:49] <AaronS>	 please actually follow the call chain
[01:06:11] <AaronS>	 I thought actionName looked odd too, but the chain makes sense
[01:07:19] * AaronS goes afk for a bit
[01:08:05] <icinga-wm>	 PROBLEM - HHVM busy threads on mw1232 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [115.2]  
[01:08:06] <MaxSem>	 would be cool to have something like https://lh4.googleusercontent.com/MppDN5ETv7QGO0WoKn7DIH83r364dl2HTEN35X_sgSevRTk5m19W4EhYi4SDVCjj6EVLbNeE7CaOajmYP9nEJl9vnbXMVGyKA9tkGApe3YUzCi2i77SyPbhugqU4r5CN-A
[01:09:47] <ori>	 hoo: !! nice!
[01:10:06] <hoo>	 ori: More in the pipeline: https://gerrit.wikimedia.org/r/177416
[01:10:56] <icinga-wm>	 RECOVERY - HHVM busy threads on mw1232 is OK: OK: Less than 30.00% above the threshold [76.8]  
[01:15:59] <grrrit-wm>	 (03PS1) 10Ori.livneh: Enable $wgAjaxEditStash on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177454 
[01:16:19] <grrrit-wm>	 (03PS2) 10Ori.livneh: Enable $wgAjaxEditStash on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177454 
[01:16:27] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Enable $wgAjaxEditStash on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177454 (owner: 10Ori.livneh)
[01:16:38] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable $wgAjaxEditStash on Beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177454 (owner: 10Ori.livneh)
[01:17:06] <ori>	 ^ AaronS, fyi
[01:21:11] <bd808>	 MaxSem: A flame graph would be cool. I think there are some projects that have started working on that for the raw xhprof data.
[01:21:50] <icinga-wm>	 PROBLEM - puppet last run on virt1007 is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:22:11] <MaxSem>	 bd808, though it could be ridiculously tall with preprocessor calls:)
[01:22:12] <icinga-wm>	 PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:22:29] <icinga-wm>	 PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: Puppet has 1 failures  
[01:22:33] <bd808>	 MaxSem: http://nathanielhoag.com/blog/2014/11/23/flamegraphs-show-why-fast-404-is-important-for-drupal-performance/
[01:22:59] <bd808>	 They use sample mode :(
[01:26:26] <springle>	 !log labsdb1003 upgrade & reboot
[01:26:30] <morebots>	 Logged the message, Master
[01:30:26] <icinga-wm>	 RECOVERY - puppet last run on virt1007 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures  
[01:30:56] <grrrit-wm>	 (03PS1) 10Krinkle: Minor code clean up PHP and HTML files in w/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177459 
[01:31:07] <icinga-wm>	 RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures  
[01:31:25] <Krinkle>	 Reedy: Fancy a quick CR on ^
[01:31:26] <icinga-wm>	 RECOVERY - puppet last run on mw1029 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures  
[01:32:53] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 04-2] "It's a dependency for php-memcached on Precise, and php-memcached breaks without it." [puppet] - 10https://gerrit.wikimedia.org/r/177431 (owner: 10MaxSem)
[01:34:49] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Minor code clean up PHP and HTML files in w/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177459 (owner: 10Krinkle)
[01:35:20] <grrrit-wm>	 (03CR) 10Reedy: [C: 031] Minor code clean up PHP and HTML files in w/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177459 (owner: 10Krinkle)
[01:38:49] <grrrit-wm>	 (03Abandoned) 10MaxSem: Remove igbinary everywhere [puppet] - 10https://gerrit.wikimedia.org/r/177431 (owner: 10MaxSem)
[01:40:15] <^d>	 oh yeah I forgot about that dependency whoops.
[01:40:40] <^d>	 I guess we should make double certain we don't have any stale api apaches writing memcached with igbinary :)
[01:41:48] <MaxSem>	 just fucking migrate everywhere:P
[01:42:04] <ori>	 MaxSem: yeah
[01:56:57] <springle>	 !log labsdb1002 upgrade & reboot
[01:57:03] <morebots>	 Logged the message, Master
[02:01:10] <icinga-wm>	 PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 1 failures  
[02:09:46] <springle>	 !log labsdb1001 upgrade & reboot
[02:09:49] <morebots>	 Logged the message, Master
[02:09:50] <Krinkle>	 Reedy: Thanks for touch.php, didn't know about that one.
[02:11:02] <Reedy>	 It was there for a while, I just forgot to plumb it into the apache config
[02:12:59] <icinga-wm>	 RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[02:20:30] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Minor code clean up PHP and HTML files in w/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177459 (owner: 10Krinkle)
[02:20:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Minor code clean up PHP and HTML files in w/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177459 (owner: 10Krinkle)
[02:20:43] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf10/cache/l10n: (no message) (duration: 00m 01s)
[02:20:47] <morebots>	 Logged the message, Master
[02:20:47] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf10) at 2014-12-04 02:20:47+00:00
[02:20:52] <morebots>	 Logged the message, Master
[02:22:01] <Reedy>	 lies all lies logmsgbot
[02:23:49] <jackmcbarn>	 can someone see if the backtrace for 79208ee3 on enwiki is the same as https://phabricator.wikimedia.org/T76609 ?
[02:25:26] <^d>	 jackmcbarn: Looks like it, yep.
[02:25:37] <jackmcbarn>	 good, thanks.
[02:25:43] <^d>	 yw
[02:32:12] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf11/cache/l10n: (no message) (duration: 00m 01s)
[02:32:17] <morebots>	 Logged the message, Master
[02:32:18] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf11) at 2014-12-04 02:32:18+00:00
[02:32:21] <morebots>	 Logged the message, Master
[02:40:17] <grrrit-wm>	 (03PS1) 10Krinkle: Remove w/COPYING and w/CREDITS dead symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177476 
[02:40:47] <grrrit-wm>	 (03CR) 10Krinkle: "They're dead in git, and dead on tin as well (it's not a local thing that was half in git). http://en.wikipedia.org/w/COPYING is 404 over " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177476 (owner: 10Krinkle)
[02:40:56] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] Remove w/COPYING and w/CREDITS dead symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177476 (owner: 10Krinkle)
[02:41:06] <grrrit-wm>	 (03Merged) 10jenkins-bot: Remove w/COPYING and w/CREDITS dead symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177476 (owner: 10Krinkle)
[02:42:47] <spagewmf>	 heya grrrit Accesspeople ^d AzaToth ori legoktm ... , do I ask for a grrrit deploy ( https://gerrit.wikimedia.org/r/#/c/177371/ ) here or request it in a SWAT window?
[02:43:18] <^d>	 that's a tool thing, should be deployable anytime by someone who knows how.
[02:43:22] <^d>	 no need for a window.
[02:44:08] <spagewmf>	 ^d thanks. I'll ask in #wikimedia-labs 
[02:45:47] <grrrit-wm>	 (03CR) 10Awight: [C: 031] "+2 -- but leaving merge for whoever deploys it." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177278 (owner: 10Ejegg)
[02:55:05] <logmsgbot>	 !log krinkle Synchronized w/: Ifbfb7dfd8fc0cd822b0 and I6594bc82b9de (duration: 00m 05s)
[02:55:07] <morebots>	 Logged the message, Master
[03:21:37] <grrrit-wm>	 (03PS1) 10Aaron Schulz: Declare profiler in CLI mode (though it is not used) to support --profiler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177481 
[03:52:41] <Fiona>	 _joe_, ori, et al.: Congrats on the transition to HHVM! :-)
[04:01:52] <TimStarling>	 !log on mw1189: restarting hhvm
[04:01:59] <morebots>	 Logged the message, Master
[04:06:28] <icinga-wm>	 RECOVERY - HHVM rendering on mw1189 is OK: HTTP OK: HTTP/1.1 200 OK - 68195 bytes in 1.321 second response time  
[04:06:49] <icinga-wm>	 RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[04:07:19] <icinga-wm>	 RECOVERY - Apache HTTP on mw1189 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.124 second response time  
[04:23:27] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Dec  4 04:23:26 UTC 2014 (duration 23m 25s)
[04:23:31] <morebots>	 Logged the message, Master
[04:39:03] <grrrit-wm>	 (03CR) 10KartikMistry: "cx_ tables are added in wikishared DB now, so this patch can go ahead if everything is OK!" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry)
[05:01:58] <icinga-wm>	 PROBLEM - puppet last run on tungsten is CRITICAL: CRITICAL: Puppet has 1 failures  
[05:07:58] <grrrit-wm>	 (03PS1) 10Chad: Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 
[05:11:32] <grrrit-wm>	 (03PS1) 10Chad: Disable ConfirmEdit on private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177495 
[05:11:55] <grrrit-wm>	 (03PS2) 10Chad: Disable ConfirmEdit on closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177495 
[05:12:17] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Disable ConfirmEdit on closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177495 (owner: 10Chad)
[05:12:25] <grrrit-wm>	 (03Merged) 10jenkins-bot: Disable ConfirmEdit on closed wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177495 (owner: 10Chad)
[05:12:56] <icinga-wm>	 RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[05:13:00] <logmsgbot>	 !log demon Synchronized wmf-config/InitialiseSettings.php: confirmedit disabled on closed wikis (duration: 00m 05s)
[05:13:04] <morebots>	 Logged the message, Master
[05:13:05] <^d>	 127 less wikis with confirmedit :p
[05:19:21] <Fiona>	 Nice.
[05:23:34] <grrrit-wm>	 (03CR) 10MZMcBride: "Thanks for this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[05:26:31] <^d>	 I thought group0.dblist got added.
[05:28:59] <^d>	 Ah, it does but not in commonsettings.
[05:32:39] <ori>	 ^d: +1 for the patch apart from the technical bit
[05:33:13] <^d>	 We can add group0 to the commonsettings bit easily enough.
[05:34:49] <grrrit-wm>	 (03PS2) 10Chad: Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 
[05:35:25] <ori>	 ^d: does CirrusSearch pay any attention to searchNs* user preferences? (which indicate whether a namespace should be searched by default for some user)
[05:35:39] <^d>	 That's a MW core thing.
[05:36:07] <^d>	 We just search the namespaces given to us, Special:Search has already figured out prefs/ticked boxes by that one.
[05:36:16] <^d>	 s/one/point/
[05:36:22] <ori>	 aha, ok
[05:36:29] <^d>	 Why you ask? :)
[05:36:48] <^d>	 Fiona: Did we have a bug filed somewhere about cleaning up dblists to some subdirectory?
[05:37:17] <ori>	 i have a patch open for it and i boldly declared my intents to see it through, but i haven't actually done anything
[05:37:28] <ori>	 (cleaning up dblists)
[05:37:47] <^d>	 For some reason I found myself grepping about for them now. We'll need symlinks for some.
[05:37:53] <^d>	 Lots of puppet. Couple of extensions.
[05:38:55] <ori>	 re: search preferences -- the scheme for search preferences is silly; each namespace gets a preference, and the value is boolean, which is unfortunately verbose when encoded as json
[05:39:26] <ori>	 so for example on every page on enwiki this is included for me in the html output:
[05:39:28] <ori>	 searchNs1":false,"searchNs2":false,"searchNs3":false,"searchNs4":false,"searchNs5":false,"searchNs6":false,"searchNs7":false,"searchNs8":false,"searchNs9":false,"searchNs10":
[05:39:28] <ori>	 false,"searchNs11":false,"searchNs12":false,"searchNs13":false,"searchNs14":false,"searchNs15":false,"searchNs100":false,"searchNs101":false,"searchNs108":false,"searchNs109":false,"searchNs118":false,"searchNs119":false,"searchNs446":false,"searchNs447":false,"searchNs710":false,"searchNs711":false,"searchNs828":false,"searchNs829":false,"searchNs2600":false
[05:39:50] <^d>	 Yes, it's an awful preference.
[05:40:09] <^d>	 Also there's some bug about negative numbers being used.
[05:40:29] <ori>	 i was going to fix it but then i suddenly had a lazy thought that maybe cirrus had obsoleted these and they could just be removed
[05:40:42] <ori>	 but i guess not
[05:41:02] <^d>	 !phab T54777
[05:41:10] <^d>	 No phab links? Boo.
[05:41:28] <^d>	 !phab is https://phabricator.wikimedia.org/`1
[05:41:28] <wm-bot>	 Unable to modify db, access denied, link to database isn't valid
[05:41:33] <^d>	 Oh boo
[05:41:42] <^d>	 Anyway, that task.
[05:42:23] <ori>	 yeah reading it
[05:42:31] <ori>	 user prefs can only be boolean, right?
[05:44:10] <Fiona>	 What?
[05:44:42] <ori>	 yeah that's not right
[05:44:56] <Fiona>	 ori: Does that extra HTML output matter?
[05:44:57] <^d>	 No, they can be other things.
[05:46:17] <Fiona>	 Not that I disagree with killing overly expressive cruft in the HTML page source, but in terms of performance, I imagine that code isn't really hurting us.
[05:47:55] <ori>	 it's half a kilobyte per page, ~150 bytes w/gzip. so no, probably not. but interesting it's a database issue as well
[05:48:15] <legoktm>	 there's a bug about the user_properties table being bloated
[05:48:23] <ori>	 that's the one that ^d just linked to
[05:48:30] <Fiona>	 https://phabricator.wikimedia.org/T54777
[05:48:41] <ori>	 i haven't customized my default search namespaces at all so i shouldn't be seeing any of them
[05:49:08] <legoktm>	 oh
[05:49:27] <Fiona>	 Hmmm.
[05:49:36] <Fiona>	 I thought I fixed the trailing newline for en.wikipedia.org.
[05:51:40] <ori>	 what the hell?
[05:51:59] <ori>	 $defaultPreferences['searchNs' . $n] = array( 'type' => 'api' );
[05:52:05] <ori>	 no wonder
[05:52:24] <Fiona>	 </html>
[05:52:24] <Fiona>	 <?php
[05:52:28] <Fiona>	 Does PHP eat the newline?
[05:52:38] <^d>	 I'm experimenting with folding that into one preference.
[05:52:52] <^d>	 And only saving true values, not false.
[05:52:56] <^d>	 Should cut down on db bloat.
[05:53:18] <ori>	 ^d: there's a simpler fix, it seems
[05:53:37] <^d>	 Hmm?
[05:53:44] <ori>	 the default preferences set each searchable searchNsN to array( 'type' => 'api' )
[05:54:16] <ori>	 but the user default preferences set it to a boolean value
[05:54:31] <ori>	 so you end up thinking the user has customized a lot more than she or he actually has
[05:55:37] <ori>	 the preference-per-namespace scheme is not awful if you only store preferences that differ from the default
[05:56:48] <Fiona>	 Okay, so I see "</html>\n<?php".
[05:56:52] <Fiona>	 Where does the \n go...
[05:57:56] <ori>	 ^d: you merged it! :P https://gerrit.wikimedia.org/r/#/c/131727/
[05:58:18] <^d>	 Yeah I know. It used to be a ton of checkboxes on the prefs page.
[05:58:20] <^d>	 And was ugly :)
[05:59:21] * ori fixes
[06:00:08] <ori>	 Nemo_bis: what is the type => api thing?
[06:00:37] <legoktm>	 API means it doesn't show up on Special:Prefs, but can be set via API:Options
[06:01:00] <legoktm>	 https://www.mediawiki.org/wiki/Manual:Hooks/GetPreferences#API_preferences
[06:02:30] <Nemo_bis>	 Yes, did we still not document that
[06:03:10] <ori>	 so it's tri-state, with possible values being: true, false, and array( 'type' => 'api' ) ?
[06:06:13] <ori>	 <legoktm> API means it doesn't show up on Special:Prefs, but can be set via API:Options
[06:06:32] <ori>	 what is the actual preference value it's array( 'type' => 'api' )?
[06:06:40] <ori>	 *when it's
[06:06:50] <ori>	 should it be searched or not?
[06:07:51] <Fiona>	 I think so?
[06:08:17] <ori>	 Fiona: valid answers are "Yes", "No", or "Array"
[06:08:57] <Fiona>	 !tias
[06:10:31] <ori>	 so, really, there should be a config option for which namespaces are shown on Special:Preferences and which are settable via the API alone
[06:10:41] <ori>	 and it should have nothing to do with the actual preference value
[06:12:20] <Fiona>	 I think you mean Special:Search?
[06:12:41] <Fiona>	 The namespace checkboxes are no longer at Special:Preferences, I believe.
[06:12:54] <^d>	 There's a lot of types.
[06:12:58] <^d>	 Not just boolean or api.
[06:14:12] <^d>	 Type is mostly stuff like multiselect/radio/etc for the special:preferences page.
[06:14:34] <ori>	 aha
[06:14:34] <^d>	 type = api is for things that are explicitly *not* shown on special:preferences and are only available internally and via the API.
[06:14:49] <^d>	 mw core is not the only user of this, there's a few extensions if memory serves too.
[06:15:30] <^d>	 To make it extra fun, we also have stuff like int and float.
[06:15:36] <Fiona>	 If you're looking for preferences work, I have some design tasks.
[06:16:29] <Fiona>	 The HHVM beta feature is still enabled?
[06:17:09] <ori>	 d'oh, is it?
[06:17:21] <ori>	 i think we should just change:
[06:17:24] <Fiona>	 https://www.mediawiki.org/wiki/Special:Preferences#mw-prefsection-betafeatures
[06:17:27] <ori>	 $defOpt['searchNs' . $nsnum] = !empty( $wgNamespacesToBeSearchedDefault[$nsnum] ); 
[06:17:42] <ori>	 to:
[06:18:42] <Fiona>	 I just burped and it tasted like sushi.
[06:18:49] <ori>	 $defOpt['searchNs' . $nsnum] = empty( $wgNamespacesToBeSearchedDefault[$nsnum] ) ? array( 'type' => 'api' ) : true;
[06:19:16] <ori>	 that should fix it
[06:19:42] <icinga-wm>	 PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 8 below the confidence bounds  
[06:20:07] <Fiona>	 Patch or it never happened.
[06:26:30] <ori>	 yeah i'm on it
[06:30:43] <ori>	 Nemo_bis: since the search namespace preferences are no longer at Special:Preferences anyway, and only at Special:Search (when you click 'advanced'), is there really a need to make them api-only?
[06:32:03] <^d>	 Why would you want to set the default preference to an array?
[06:32:13] <^d>	 The default prefs in User are not the same thing as type.
[06:32:56] <ori>	 yeah, i just realized that
[06:34:22] <icinga-wm>	 PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:35:23] <icinga-wm>	 PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:35:24] <icinga-wm>	 PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 3 failures  
[06:36:10] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:36:13] <icinga-wm>	 PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 2 failures  
[06:44:32] <grrrit-wm>	 (03CR) 10MZMcBride: [C: 031] "I think this is fine to try. Just be sure to sync CommonSettings.php first. :-)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[06:45:31] <grrrit-wm>	 (03CR) 10Jforrester: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[06:45:40] <icinga-wm>	 RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures  
[06:46:11] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures  
[06:46:22] <Nemo_bis>	 ori: so the context is that you're trying to optimise the huge <script> blurb in the <head>?
[06:46:29] <icinga-wm>	 RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures  
[06:48:00] <icinga-wm>	 RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[06:48:59] <icinga-wm>	 RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures  
[06:50:44] <Nemo_bis>	 In my case it's 7325 characters , only 10 % are for search preferences (and this is Meta, which has many additional namespaces).
[06:50:56] <Nemo_bis>	 The simplest solution is probably to remove some namespace. :D
[06:55:12] <icinga-wm>	 PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures  
[07:01:25] <icinga-wm>	 PROBLEM - puppet last run on db1045 is CRITICAL: CRITICAL: Puppet has 2 failures  
[07:02:08] <grrrit-wm>	 (03CR) 10PleaseStand: [C: 04-1] Declare profiler in CLI mode (though it is not used) to support --profiler (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177481 (owner: 10Aaron Schulz)
[07:05:52] <_joe_>	 morning
[07:06:22] <grrrit-wm>	 (03CR) 10Florianschmidtwelzow: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[07:06:55] <icinga-wm>	 RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures  
[07:15:53] <icinga-wm>	 RECOVERY - puppet last run on db1045 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures  
[07:23:02] <grrrit-wm>	 (03CR) 10Swalling: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[07:37:38] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Allow OCG to read the contents of its output/temp/postmortem directories. [puppet] - 10https://gerrit.wikimedia.org/r/177384 (owner: 10Cscott)
[08:03:23] <grrrit-wm>	 (03CR) 10Bartosz Dziewoński: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[08:13:23] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[08:13:23] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: Partitioning scheme for cassandra hosts [puppet] - 10https://gerrit.wikimedia.org/r/177504 
[08:13:23] <akosiaris>	 !log disable puppet on carbon. Playing with partman :)
[08:13:23] <morebots>	 Logged the message, Master
[08:17:55] <icinga-wm>	 PROBLEM - Host cerium is DOWN: PING CRITICAL - Packet loss = 100%  
[08:25:05] <icinga-wm>	 RECOVERY - Host cerium is UP: PING OK - Packet loss = 0%, RTA = 3.54 ms  
[08:25:43] <icinga-wm>	 ACKNOWLEDGEMENT - DPKG on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:43] <icinga-wm>	 ACKNOWLEDGEMENT - Disk space on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:43] <icinga-wm>	 ACKNOWLEDGEMENT - NTP on cerium is CRITICAL: NTP CRITICAL: No response from NTP server alexandros kosiaris alex playing around
[08:25:44] <icinga-wm>	 ACKNOWLEDGEMENT - RAID on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:44] <icinga-wm>	 ACKNOWLEDGEMENT - SSH on cerium is CRITICAL: Connection refused alexandros kosiaris alex playing around
[08:25:44] <icinga-wm>	 ACKNOWLEDGEMENT - check configured eth on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:44] <icinga-wm>	 ACKNOWLEDGEMENT - check if dhclient is running on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:45] <icinga-wm>	 ACKNOWLEDGEMENT - check if salt-minion is running on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:25:45] <icinga-wm>	 ACKNOWLEDGEMENT - puppet last run on cerium is CRITICAL: Connection refused by host alexandros kosiaris alex playing around
[08:27:19] <gwicke>	 akosiaris: hey
[08:28:34] <gwicke>	 akosiaris: if you re-image the nodes one at a time we can see if any requests are dropped in the process
[08:29:03] <gwicke>	 (not that it matters, just a fun thought)
[08:29:31] <akosiaris>	 gwicke: hey. Yeah, for now playing a bit with partman and cerium
[08:29:42] <akosiaris>	 but I can reimage them one at a time, no worries
[08:29:58] <gwicke>	 I'm currently pulling enwiki through xenon
[08:30:20] <akosiaris>	 but I don't have yet a partman recipe doing what we want working so it is going to take me some time
[08:30:51] <gwicke>	 akosiaris: I'm also happy to set up the partitions for the SSDs manually
[08:31:28] <gwicke>	 just having the OS on the non-SSD disks would be enough
[08:31:48] <akosiaris>	 yeah, I thought about that too. I decided to torture myself a bit first . Better than RT duty ;-)
[08:31:56] <gwicke>	 okay ;)
[08:32:37] <gwicke>	 I was playing with the restbase graphite metrics: http://bit.ly/1AjdAPS
[08:33:35] <akosiaris>	 that explains why you are up at this hour :-)
[08:33:57] <gwicke>	 yeah, graphs are addictive
[08:34:21] <gwicke>	 but I think I'll head to bed now
[08:34:29] <gwicke>	 see you later today!
[08:35:36] <akosiaris>	 good night, sleep tight!
[08:35:40] <icinga-wm>	 PROBLEM - Host cerium is DOWN: PING CRITICAL - Packet loss = 100%  
[08:37:59] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Expand ConfirmEdit disabling test to all group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[08:40:13] <grrrit-wm>	 (03CR) 10Nemo bis: [C: 031] "One-size-fits-all spambots certainly target mediawiki.org, so we can see how many we currently stop." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177494 (owner: 10Chad)
[08:42:54] <icinga-wm>	 RECOVERY - Host cerium is UP: PING OK - Packet loss = 0%, RTA = 2.23 ms  
[08:53:51] <AzaToth>	 imo, all "Bug NNN has been marked as a duplicate of this bug. ***" messages in phabricator should be purged
[08:54:51] <ori>	 AzaToth: makes sense; file a bug :)
[08:55:14] <AzaToth>	 heh
[08:55:51] * AzaToth tried to figure out how to file a new bug in phabricator...
[08:55:53] <AzaToth>	 tries*
[09:01:06] <icinga-wm>	 PROBLEM - SSH on cerium is CRITICAL: Connection refused  
[09:01:12] <AzaToth>	 ori: manage to file https://phabricator.wikimedia.org/T76698 now ツ
[09:04:21] <Nemo_bis>	 AzaToth: https://phabricator.wikimedia.org/T687
[09:04:23] <icinga-wm>	 PROBLEM - puppet last run on cp3017 is CRITICAL: CRITICAL: puppet fail  
[09:13:08] <grrrit-wm>	 (03PS9) 10Nikerabbit: Enable ContentTranslation shared database on labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry)
[09:13:12] <grrrit-wm>	 (03PS10) 10Nikerabbit: Enable ContentTranslation shared database on labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry)
[09:13:27] <grrrit-wm>	 (03CR) 10Nikerabbit: [C: 032] "Only affects labs" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry)
[09:13:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: Enable ContentTranslation shared database on labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/175979 (owner: 10KartikMistry)
[09:18:55] <icinga-wm>	 RECOVERY - puppet last run on cp3017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[09:40:47] <icinga-wm>	 PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/).  
[09:46:58] <godog>	 !log upload diamond 3.5-1wmf1 to trusty-wikimedia
[09:47:03] <morebots>	 Logged the message, Master
[10:01:31] <ori>	 !log disabling puppet on mw1081 and restarting hhvm with hhvm.server.stat_cache	=true to observe impact
[10:01:33] <morebots>	 Logged the message, Master
[10:02:57] <paravoid>	 ori: did you see hoo's work?
[10:03:08] <paravoid>	 wikidata-tech
[10:03:22] <ori>	 i haven't seen his e-mail, but i watched his deploys and the ganglia graphs
[10:03:33] <paravoid>	 good stuff
[10:03:33] <ori>	 it looks less joyous if you look at the one year graph, but yeah, it's good news
[10:03:42] <paravoid>	 yeah I thought the same
[10:04:16] <paravoid>	 SiteList is still & by far the first in bandwidth
[10:04:40] <paravoid>	 wmf11 switches completely away from memcache as I understand it though
[10:05:09] <paravoid>	 https://gerrit.wikimedia.org/r/#/c/177416/ is also pending review
[10:05:53] <paravoid>	 and https://gerrit.wikimedia.org/r/#/c/174874/ is the wmf11 change
[10:05:57] <paravoid>	 I'm hopeful :)
[10:08:01] <ori>	 me too, but i'm still a bit worried about the overall complexity and poor optics into resource utilization
[10:09:14] <ori>	 once it was determined that it was the site list key, and that the data it encodes is mostly static (changes only when we add new wikis), the fix should have been trivial
[10:09:45] <ori>	 instead it took a series of patches that were entirely too subtle for me to actually follow
[10:10:58] <Nikerabbit>	 dunno if you can do anything about this, but it's troubling: https://integration.wikimedia.org/zuul/
[10:13:19] <ori>	 Nikerabbit: hrm, the top change stuck in that queue is interesting: https://gerrit.wikimedia.org/r/#/c/177271
[10:15:14] <ori>	 Nikerabbit: https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/5921/ has been running for 10 hours
[10:16:47] <Nikerabbit>	 which says it's waiting for next available executor
[10:32:44] <ori>	 _joe_: i'm off to bed, but i'd like to keep puppet disabled on mw1081; hhvm.server.stat_cache	=true looks pretty good: https://graphite.wikimedia.org/render/?width=586&height=308&_salt=1417689131.682&from=-2hours&lineMode=connected&target=movingAverage(removeBelowValue(servers.mw1081.cpu.total.system.value%2C2)%2C5)
[10:33:35] <godog>	 !log test-upgrade diamond 3.5 on swift in codfw
[10:33:41] <morebots>	 Logged the message, Master
[10:34:33] <_joe_>	 ori: how long does that stat cache live?
[10:34:41] <_joe_>	 well I'll look into the code
[10:34:48] <_joe_>	 because I fear scaps :)
[10:35:17] <_joe_>	 (I'll merge changes that will make it easy to do such tests without breaking puppet btw)
[10:35:40] <_joe_>	 but yeah it looks good
[10:36:19] <paravoid>	 what exactly is the stat cache?
[10:37:28] <ori>	 it caches the results of lstat and readlink calls
[10:37:37] <ori>	 and uses inotify to invalidate
[10:37:48] <paravoid>	 heh, I was asking exactly that yesterday, funny
[10:38:03] <ori>	 _joe_: i don't think there's a time-based expiry; it uses inotify to invalidate entries
[10:38:09] <paravoid>	 23:12 < bblack> I wonder why sqlite container for the files? as I see it the main benefit of repoauthorative is that it doesn't stat() the code files all the time and try to reload them on the fly.  couldn't that  also be done with code on a regular filesystem and a flag that says "load all the code now, don't ever check it on disk again until told to?"
[10:38:14] <paravoid>	 23:12 < bblack> and leave everything on the fs?
[10:38:16] <paravoid>	 23:12 < paravoid> or, you know, inotify 
[10:39:54] <ori>	 "couldn't that  also be done with code on a regular filesystem and a flag that says "load all the code now, don't ever check it on disk again until told to?""
[10:40:00] <ori>	 that's RepoAuthoritative
[10:40:11] <ori>	 the sqlite container doesn't contain the files; it contains IR byte code
[10:40:14] <icinga-wm>	 RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected  
[10:40:21] <paravoid>	 yes I pointed that out later in the conversation
[10:41:11] <ori>	 the repo is populated by translating files on disk, and when you set RepoAuthoritative, you're telling it exactly that: don't ever check if the files on disk have changed until I tell you to
[10:42:08] <paravoid>	 as far as I understand it, the point of RepoAuthoritative is to do code swaps in their entirety, no?
[10:42:12] <paravoid>	 instead of file-by-file
[10:43:23] <ori>	 no, it's to avoid having to constantly hit the disk to check if the byte code is in sync with the PHP code
[10:43:49] <paravoid>	 with stat_cache you don't need to hit the disk either though, no?
[10:44:23] <ori>	 yeah, i'm not sure why brett hadn't mentioned it (or tim for that matter, since he knows his way around the code base pretty well by now)
[10:44:29] <godog>	 !log upgrade diamond to 3.5 on all trusty machines in codfw
[10:44:33] <ori>	 i came across it for the first time about an hour ago
[10:44:35] <paravoid>	 you don't really need sqlite for this
[10:44:35] <morebots>	 Logged the message, Master
[10:44:56] <paravoid>	 the problem with what we have now (with or without the cache) is that code swaps aren't atomic
[10:45:18] <ori>	 that's rarely an issue in practice
[10:45:21] <paravoid>	 Ryan's vision for fixing this with trebuchet was to deploy in two different directories ("slots" he called them)
[10:45:31] <ori>	 yes, and to swap a symlink
[10:45:49] <ori>	 the problem with that was that zend's apc cache caches by inode rather than content hash
[10:45:53] <paravoid>	 yup
[10:46:32] <paravoid>	 that could have been fixed with per-file hardlinks but I don't think anyone bothered
[10:47:19] <ori>	 at one point bd808 and i also realized that there's an rsync option that stages all the changes on the target system and then swaps them all at once
[10:47:26] <ori>	 we're using it now IIRC
[10:47:56] <ori>	 that's still not atomic, but it reduces the time the system is in an inconsistent state to a flash
[10:48:13] <paravoid>	 nod
[10:49:03] <ori>	 yeah, we do both --delete-delay and --delay-updates
[10:49:18] <paravoid>	 so if it's not really a problem in practice
[10:49:32] <paravoid>	 what would repoauthoritative give us?
[10:49:43] <_joe_>	 I'm pretty sure repoauth has other advantages in how JIT is done
[10:49:48] <paravoid>	 compared to stat_cache
[10:49:50] <_joe_>	 I remember brett mentioning this
[10:50:01] <_joe_>	 I'm asking myself the same question :)
[10:50:16] <ori>	 yeah, HHVM just skips certain optimizations that it reasons are too expensive to perform at runtime
[10:50:32] <ori>	 there isn't really a hard reason for it, brett conceded that non-repo-auth could be made a lot more performant
[10:50:32] <paravoid>	 https://github.com/facebook/hhvm/blob/master/hphp/doc/repo
[10:50:44] <ori>	 but it was added as a developer convenience feature really
[10:50:45] <paravoid>	 hhvm utilizes SQLite transactions to ensure that the repo is always in a
[10:50:48] <paravoid>	 consistent state
[10:51:35] <ori>	 yes, but it doesn't know when the files on disk will stop changing
[10:51:37] <paravoid>	 so I guess consistency is a goal of theirs as well
[10:51:49] <ori>	 they restart HHVM with each deploy
[10:51:57] <paravoid>	 right
[10:52:40] <paravoid>	 I can't say that I find the idea of deploying large bulks of sqlite sounds too appealing to me
[10:52:59] <paravoid>	 sync-file can be pretty useful
[10:53:14] <paravoid>	 but I rarely deploy code, so what do I care :P
[10:53:20] <ori>	 no, it's true
[10:53:27] <_joe_>	 paravoid: no one likes that I think, if stat_cache is good enough, we should stick to it
[10:53:59] <ori>	 well, there are other issues
[10:55:09] <ori>	 hhvm initially runs code in the interpreter, and we have run into bugs before where JITed code behaves subtly differently than interpreted code
[10:55:25] <paravoid>	 oops
[10:55:25] <_joe_>	 for one, I'll check how they use inotify and if that creates issues with symlinks
[10:55:27] <paravoid>	 heisenbugs
[10:55:49] <ori>	 it's less of a problem now because we get enough traffic that everything is almost always warm
[10:57:00] <ori>	 but earlier on we were restarting the servers frequently enough (and the volume of requests was low enough) that one of the first things to figure out whenever we spotted a crash was whether it happens just w/the JIT or not
[10:57:53] <grrrit-wm>	 (03PS5) 10Giuseppe Lavagetto: hiera: role-based backend, role keyword [puppet] - 10https://gerrit.wikimedia.org/r/176334 
[10:58:07] <_joe_>	 godog: ^^ now with tests!
[10:58:32] <godog>	 _joe_: sweet! I'll take a look
[10:58:43] <ori>	 3am, bed time for realz. _joe_, is it alright to leave mw1081 in this state?
[10:58:47] <_joe_>	 (and I found a bug that I inherited from the original puppet yaml backend, meh. Not that we ever use array lookup, still...)
[10:58:54] <paravoid>	 night
[10:59:22] <_joe_>	 ori: yep, I'll amend my patches to allow special ini values along your suggestions last week, and use that to deploy stat_cache = true there :)
[10:59:28] <_joe_>	 good night
[10:59:32] <ori>	 thanks, good night
[11:00:22] <_joe_>	 does jenkins run rspec tests for puppet?
[11:00:26] <_joe_>	 I don't think so
[11:00:37] <_joe_>	 we should start doing that maybe
[11:00:49] <paravoid>	 it used to
[11:01:04] <paravoid>	 I recall akosiaris working with hashar for that
[11:01:25] <paravoid>	 operations-puppet-spec Experimental unit tests, please ignore. in 22s (non-voting)
[11:01:29] <_joe_>	 I imagined, the only modules with tests are created by alex
[11:01:32] <_joe_>	 ok
[11:02:50] <_joe_>	 10:58:23 Don't know how to build task 'spec_standalone'
[11:02:53] <_joe_>	 ook
[11:04:24] <akosiaris>	 _joe_: actually it does
[11:04:39] <akosiaris>	 just not voting as paravoid mentioned
[11:04:54] <akosiaris>	 and that is mostly my fault... I need to conform my Rakefiles to what jenkins expects
[11:05:06] * akosiaris likes his Rakefiles better :-)
[11:05:21] <_joe_>	 akosiaris: I cowardly copied yours, damn
[11:05:26] <_joe_>	 :)
[11:05:45] <_joe_>	 I thought I could exploit some of the wisdom of the seniors
[11:05:52] <_joe_>	 <grin>
[11:06:35] <akosiaris>	 there you go... now I will invoke that wisdom and force hashar to adhere to my Rakefiles!!!
[11:06:48] <akosiaris>	 muahahaahaha (evil laugh)
[11:16:25] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 031] "minor comment, I'm not a ruby expert but what I can make from the code LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/176334 (owner: 10Giuseppe Lavagetto)
[11:17:27] <_joe_>	 godog: eheh good comment
[11:18:02] <godog>	 angry-based messaging
[11:18:30] <godog>	 puppet's karma is already low enough there
[11:18:45] <_joe_>	 godog: or, I could just make the user angrier by throwing something like "Unexpected T_PAAMAYIM_NEKUDOTAYIM"
[11:19:18] <godog>	 hahah or fail silently
[11:20:14] <_joe_>	 (obligatory reference to http://phpsadness.com/)
[11:21:35] <godog>	 hehehe no shortage of examples
[11:24:07] <godog>	 !log upgrade to diamond 3.5 on trusty hosts in ulsfo
[11:24:12] <morebots>	 Logged the message, Master
[11:25:07] <godog>	 alas, no trusty hosts in ulsfo
[11:29:31] <grrrit-wm>	 (03PS6) 10Giuseppe Lavagetto: hiera: role-based backend, role keyword [puppet] - 10https://gerrit.wikimedia.org/r/176334 
[11:48:23] <kart__>	 Can anyone kick stuck mediawiki-config deployment in Beta?
[11:48:35] <kart__>	 https://integration.wikimedia.org/zuul/
[12:09:30] <paravoid>	 !log powercycling rhenium, kernel locked up
[12:09:33] <morebots>	 Logged the message, Master
[12:12:27] <icinga-wm>	 RECOVERY - DPKG on rhenium is OK: All packages OK  
[12:13:14] <icinga-wm>	 RECOVERY - Disk space on rhenium is OK: DISK OK  
[12:13:39] <icinga-wm>	 RECOVERY - RAID on rhenium is OK: OK: Active: 6, Working: 6, Failed: 0, Spare: 0  
[12:13:59] <icinga-wm>	 RECOVERY - SSH on rhenium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0)  
[12:13:59] <icinga-wm>	 RECOVERY - check configured eth on rhenium is OK: NRPE: Unable to read output  
[12:14:16] <icinga-wm>	 RECOVERY - check if salt-minion is running on rhenium is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion  
[12:14:28] <icinga-wm>	 RECOVERY - check if dhclient is running on rhenium is OK: PROCS OK: 0 processes with command name dhclient  
[12:14:34] <icinga-wm>	 RECOVERY - puppet last run on rhenium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[12:16:26] <icinga-wm>	 RECOVERY - NTP on rhenium is OK: NTP OK: Offset -0.03571105003 secs  
[12:17:16] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Remove alert for hhvm-appservers.svc.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/177518 
[12:18:07] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Remove hhvm-appservers.svc.eqiad.wmnet service IP [dns] - 10https://gerrit.wikimedia.org/r/177519 
[12:18:09] <paravoid>	 _joe_: ^^
[12:18:10] <paravoid>	 those two
[12:19:04] <_joe_>	 paravoid: eh it was on my todo, thanks
[12:24:53] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] Remove alert for hhvm-appservers.svc.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/177518 (owner: 10Faidon Liambotis)
[12:33:30] <grrrit-wm>	 (03PS4) 10Giuseppe Lavagetto: Allow host-specific HHVM config overrides via Hiera [puppet] - 10https://gerrit.wikimedia.org/r/175425 
[12:36:44] <grrrit-wm>	 (03PS5) 10Giuseppe Lavagetto: Allow host-specific HHVM config overrides via Hiera [puppet] - 10https://gerrit.wikimedia.org/r/175425 
[12:40:12] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] Allow host-specific HHVM config overrides via Hiera [puppet] - 10https://gerrit.wikimedia.org/r/175425 (owner: 10Giuseppe Lavagetto)
[12:41:17] <grrrit-wm>	 (03PS2) 10Alexandros Kosiaris: Partitioning scheme for cassandra hosts [puppet] - 10https://gerrit.wikimedia.org/r/177504 
[12:45:25] <grrrit-wm>	 (03PS1) 10Giuseppe Lavagetto: HAT: test hhvm.server.stat_cache on mw1081 [puppet] - 10https://gerrit.wikimedia.org/r/177520 
[12:46:07] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Remove hhvm-appservers.svc.eqiad.wmnet service IP [dns] - 10https://gerrit.wikimedia.org/r/177519 (owner: 10Faidon Liambotis)
[12:46:33] <_joe_>	 there was the api endpoint as well I guess
[12:46:44] <paravoid>	 I didn't remove that
[12:46:50] <paravoid>	 not sure if you still need it
[12:46:54] <_joe_>	 no
[12:48:48] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Partitioning scheme for cassandra hosts [puppet] - 10https://gerrit.wikimedia.org/r/177504 (owner: 10Alexandros Kosiaris)
[12:52:37] <icinga-wm>	 PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:53:35] <grrrit-wm>	 (03PS2) 10Giuseppe Lavagetto: HAT: test hhvm.server.stat_cache on mw1081 [puppet] - 10https://gerrit.wikimedia.org/r/177520 
[12:53:57] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 032] HAT: test hhvm.server.stat_cache on mw1081 [puppet] - 10https://gerrit.wikimedia.org/r/177520 (owner: 10Giuseppe Lavagetto)
[12:54:12] <icinga-wm>	 PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:54:22] <icinga-wm>	 PROBLEM - puppet last run on mw1153 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:54:23] <icinga-wm>	 PROBLEM - puppet last run on cp1049 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:21] <icinga-wm>	 PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:27] <icinga-wm>	 PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:37] <icinga-wm>	 PROBLEM - puppet last run on search1010 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:37] <icinga-wm>	 PROBLEM - puppet last run on elastic1012 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:44] <grrrit-wm>	 (03PS1) 10Ori.livneh: Emit alert when Ori commits on a weekend [puppet] - 10https://gerrit.wikimedia.org/r/177521 
[12:55:47] <icinga-wm>	 PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:55:55] <icinga-wm>	 PROBLEM - puppet last run on elastic1030 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:56:00] <mark>	 that's hilarious, ori
[12:56:03] <mark>	 i was thinking yesterday in fact
[12:56:07] <mark>	 facebook has this "domas finder"
[12:56:07] <icinga-wm>	 PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:56:14] <mark>	 and I was thinking, we need an ori alerter
[12:56:16] <icinga-wm>	 PROBLEM - puppet last run on labstore1003 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:56:36] <icinga-wm>	 PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:56:41] <_joe_>	 (note he promised to go to bed like 3 hours ago)
[12:56:43] * ori self-flaggelates.
[12:56:44] <mark>	 i think it needs to involve hardware with thumbscrews and solenoids
[12:56:48] <_joe_>	 ^^ apt-get failures
[12:56:54] <_joe_>	 ahah
[12:57:07] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:57:16] <_joe_>	 electrocution is not a nice thing :)
[12:57:20] <matanya>	 lol @ ori 
[12:57:25] <icinga-wm>	 PROBLEM - puppet last run on mc1012 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:57:29] <mark>	 a solenoid doesn't electrocute
[12:57:37] <icinga-wm>	 PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:57:40] <_joe_>	 yeah I miss the term in english
[12:57:55] <icinga-wm>	 PROBLEM - puppet last run on db2007 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:06] <icinga-wm>	 PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:31] <icinga-wm>	 PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:32] <icinga-wm>	 PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:55] <icinga-wm>	 PROBLEM - puppet last run on install2001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:58] <icinga-wm>	 PROBLEM - puppet last run on labstore1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:59] <icinga-wm>	 PROBLEM - puppet last run on tin is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:58:59] <icinga-wm>	 PROBLEM - puppet last run on es1007 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:59:41] <mark>	 anyway, i'm not really sure how emitting an alert and forcing us to react during our weekends is really a solution
[12:59:41] <akosiaris>	 magnetize ? 
[12:59:47] <icinga-wm>	 PROBLEM - puppet last run on wtp1012 is CRITICAL: CRITICAL: Puppet has 1 failures  
[12:59:52] <mark>	 it's kind of part of the problem
[12:59:59] <icinga-wm>	 PROBLEM - puppet last run on ms-be2005 is CRITICAL: CRITICAL: Puppet has 1 failures  
[13:00:07] <icinga-wm>	 PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 1 failures  
[13:00:22] <icinga-wm>	 PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 1 failures  
[13:00:49] <icinga-wm>	 PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: Puppet has 1 failures  
[13:01:08] <icinga-wm>	 PROBLEM - puppet last run on amssq40 is CRITICAL: CRITICAL: Puppet has 1 failures  
[13:01:43] <paravoid>	 should this be a parameterized class or are we supposed to accept that ori is one of a kind
[13:02:18] <mark>	 ori should be a native puppet resource
[13:02:25] <ori>	 i would be pretty ashamed if it were ever triggered
[13:02:46] <mark>	 you say that
[13:03:27] <ori>	 i'm not good with electrocution devices
[13:03:33] <ori>	 but i can see if amazon has anything
[13:03:38] <mark>	 solenoids are just mechanical
[13:04:05] <mark>	 linear stepper motors
[13:04:18] <icinga-wm>	 RECOVERY - puppet last run on cp1070 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:04:25] <ori>	 http://www.magnetschultz.co.uk/uploads/images/product_images/box-frame-miniature-solenoid-l.jpg
[13:04:29] <ori>	 this looks dangerous
[13:04:34] <paravoid>	 it's called a pager
[13:04:39] <icinga-wm>	 RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[13:04:51] <paravoid>	 encourage more of this, and give ori a pager
[13:05:05] <paravoid>	 I told you the fun story of Etsy
[13:05:09] <icinga-wm>	 RECOVERY - puppet last run on labstore1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:05:14] <paravoid>	 one page every 1.5 minutes, on average
[13:05:25] <mark>	 mean time to sleep
[13:05:28] <paravoid>	 yes, that too
[13:05:28] <icinga-wm>	 RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures  
[13:05:30] <icinga-wm>	 RECOVERY - puppet last run on elastic1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:05:38] <icinga-wm>	 RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures  
[13:05:38] <paravoid>	 that fits really well here doesn't it
[13:05:46] <paravoid>	 ori: https://github.com/etsy/opsweekly
[13:05:47] <icinga-wm>	 RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures  
[13:05:49] <icinga-wm>	 RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures  
[13:05:52] <paravoid>	 ori: search for "sleep tracking"
[13:06:01] <icinga-wm>	 RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures  
[13:06:06] <paravoid>	 they have graphs for Mean time to sleep (MTTS)
[13:06:07] <_joe_>	 I need that as well
[13:06:10] <icinga-wm>	 RECOVERY - puppet last run on mw1153 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:06:14] <ori>	 https://gerrit.wikimedia.org/r/#/c/177501/ shaves 4kb from each page in varnish
[13:06:20] <_joe_>	 I slept ~ 4 hours in average per day since sunday
[13:06:49] <ori>	 i should really go to bed. bye!
[13:06:53] <_joe_>	 it's the effect of HHVM on young people's minds
[13:06:59] <icinga-wm>	 RECOVERY - puppet last run on search1010 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[13:07:36] <_joe_>	 oh snap, mw1081 misses hyperthreading
[13:07:55] <icinga-wm>	 RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[13:08:04] <icinga-wm>	 RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:08:08] <icinga-wm>	 RECOVERY - puppet last run on elastic1030 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:08:25] <icinga-wm>	 RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:08:25] <icinga-wm>	 RECOVERY - puppet last run on wtp1012 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures  
[13:08:35] <icinga-wm>	 RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:08:35] <icinga-wm>	 RECOVERY - puppet last run on es1007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:08:55] <icinga-wm>	 RECOVERY - puppet last run on mc1012 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:09:15] <icinga-wm>	 RECOVERY - puppet last run on labstore1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:09:25] <icinga-wm>	 RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:09:35] <icinga-wm>	 RECOVERY - puppet last run on install2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:09:55] <icinga-wm>	 RECOVERY - puppet last run on db2007 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[13:09:55] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures  
[13:10:05] <icinga-wm>	 RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:10:05] <icinga-wm>	 RECOVERY - puppet last run on tin is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:10:27] <icinga-wm>	 RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:10:45] <_joe_>	 !log depooling mw1081 to activate hyperthreading
[13:10:53] <morebots>	 Logged the message, Master
[13:10:54] <icinga-wm>	 RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[13:13:06] <icinga-wm>	 RECOVERY - puppet last run on ms-be2005 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures  
[13:13:08] <icinga-wm>	 RECOVERY - puppet last run on amssq40 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures  
[13:16:34] <icinga-wm>	 PROBLEM - DPKG on mw1081 is CRITICAL: DPKG CRITICAL dpkg reports broken packages  
[13:24:58] <icinga-wm>	 PROBLEM - very high load average likely xfs on ms-be2014 is CRITICAL: CRITICAL - load average: 100.70, 100.42, 83.91  
[13:26:57] <godog>	 that's me btw
[13:40:18] <ottomata>	 hey godog, any plans for upgrading graphite? :D
[13:40:35] <ottomata>	 i want to be able to use some functions that we don't have in our version
[13:40:40] <ottomata>	 specifically
[13:40:41] <ottomata>	 perSecond()
[13:41:54] <godog>	 what version has that? we're running 0.9.12 afaict
[13:42:32] <_joe_>	 we should really do something better about our graphite structure
[13:42:59] <_joe_>	 meaning the hierarchy of metrics
[13:43:09] <_joe_>	 before we think of moving other parts
[13:44:20] <ottomata>	 godog, i guess 1.0
[13:44:38] <ottomata>	 http://graphite.readthedocs.org/en/latest/functions.html#graphite.render.functions.perSecond
[13:44:46] <ottomata>	 hm, no
[13:45:38] <_joe_>	 !log repooling mw1081
[13:45:44] <morebots>	 Logged the message, Master
[13:46:32] <icinga-wm>	 RECOVERY - DPKG on mw1081 is OK: All packages OK  
[13:47:26] <paravoid>	 ottomata: a friend was just telling how they plan to switch from logstash (on every node) to syslog-ng + librdkafka
[13:47:34] <paravoid>	 apparently syslog-ng got librdkafka support
[13:48:21] <ottomata>	 huh.  cool!  seems logstash would give them more stuff, but thats super cool
[13:48:26] <ottomata>	 easy integration with lots more things
[13:48:31] <paravoid>	 that's for log shipping
[13:48:40] <paravoid>	 not for the interface
[13:48:48] <ottomata>	 aye
[13:49:03] <paravoid>	 anyway, he submitted an ITP (Intent to Package) for kafkacat
[13:49:12] <paravoid>	 saw it, responded and looped in Magnus
[13:49:27] <ottomata>	 ha, i have been meaning to do that!
[13:49:44] <paravoid>	 I proposed that we maintain both librdkafka & kafkacat together
[13:49:47] <paravoid>	 he likes that ;)
[13:49:51] <ottomata>	 :)
[13:50:09] <paravoid>	 so yeah
[13:50:11] <paravoid>	 good stuff
[14:03:25] <ottomata>	 cool, looking good qchris
[14:03:26] <ottomata>	 thanks
[14:03:29] <ottomata>	 oops, wrong room
[14:03:34] <ottomata>	 anyway ^ :)
[14:04:21] <ottomata>	 _joe_: i'm about to start sending lots of metrics to graphite.  is there a hierarchy you had in mind I should consider?
[14:05:02] <_joe_>	 ottomata: 1) coordinate with godog to be sure you're not killing our current graphite install
[14:05:14] <_joe_>	 2) are those per-server or aggregated?
[14:05:24] <paravoid>	 the already killed graphite install :P
[14:05:34] <_joe_>	 yes, exactly
[14:05:36] <_joe_>	 :)
[14:05:50] <godog>	 ottomata: how many is lots? :)
[14:06:12] <icinga-wm>	 PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Puppet has 1 failures  
[14:06:51] <ottomata>	 HMMmmmI I can hold off i suppose
[14:06:54] <ottomata>	 yes, per server
[14:07:02] <ottomata>	 the kafka ones are a few, but not so many.
[14:07:06] <ottomata>	 the varnishkafka ones are a lot
[14:07:53] <icinga-wm>	 RECOVERY - very high load average likely xfs on ms-be2014 is OK: OK - load average: 14.14, 54.80, 79.68  
[14:08:01] <ottomata>	 um, what, about 90 varnishes?
[14:08:13] <ottomata>	 ohhh, that would be about 40K new metrics.
[14:08:42] <ottomata>	 godog ^ i assume that counts as logs.
[14:08:44] <ottomata>	 lots*
[14:08:59] <godog>	 hehe, how often are they pushed?
[14:09:12] <ottomata>	 configurable...i think i'd have to run this in a cron, so once a minute?
[14:09:31] <ottomata>	 but, actually, i might want to just hold off anyway, these aren't very useful for me unless I can do perSecond
[14:09:37] <ottomata>	 they are running total sums 
[14:09:41] <ottomata>	 always increasing
[14:09:47] <ottomata>	 and I want to have a rate
[14:10:10] <ottomata>	 well, many of them are, anyway
[14:10:21] <ottomata>	 there are some that are more like real gauge values, but most are increasing
[14:12:36] <hashar>	 !log Jenkins in deadlock , restarting it ( https://phabricator.wikimedia.org/T72597 )
[14:12:40] <morebots>	 Logged the message, Master
[14:12:54] <godog>	 ottomata: there seem to be a workaround for persecond() https://github.com/graphite-project/graphite-web/issues/428
[14:13:05] <godog>	 also supposedly included in 0.9.12, bah
[14:13:47] <ottomata>	 oo, ok
[14:13:57] <ottomata>	 we are 0.9.11?
[14:14:05] <hashar>	 !log Jenkins: haven't had to restart it, I cancelled a few jobs and it went back up processing jobs..
[14:14:08] <morebots>	 Logged the message, Master
[14:16:21] <godog>	 ottomata: 0.9.12+debian-1 at the moment
[14:23:57] <icinga-wm>	 RECOVERY - puppet last run on cerium is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures  
[14:31:23] <godog>	 !log upgrade to diamond 3.5 on trusty hosts in esams
[14:31:26] <morebots>	 Logged the message, Master
[14:38:28] <_joe_>	 godog: I got a dpkg error when upgrading manually on mw1081, FYI
[14:39:33] <godog>	 _joe_: yeah I'm about to upgrade eqiad too
[14:40:03] <godog>	 !log upgrade to diamond 3.5 on trusty hosts in esams
[14:47:30] <icinga-wm>	 PROBLEM - puppet last run on wtp1021 is CRITICAL: CRITICAL: Puppet has 1 failures  
[14:52:39] <icinga-wm>	 PROBLEM - puppet last run on netmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[14:53:52] <grrrit-wm>	 (03PS1) 10Ottomata: Rename all webrequest varnishkafka instances as 'webrequest' [puppet] - 10https://gerrit.wikimedia.org/r/177546 
[14:54:16] <ottomata>	 ori: ^
[14:58:16] <icinga-wm>	 PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: Puppet has 1 failures  
[15:02:17] <icinga-wm>	 RECOVERY - puppet last run on wtp1021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[15:04:32] <icinga-wm>	 RECOVERY - puppet last run on netmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[15:09:42] <icinga-wm>	 PROBLEM - udp2log log age for lucene on oxygen is CRITICAL: CRITICAL: log files /a/log/lucene/lucene.log,  have not been written in a critical amount of time.  For most logs, this is 4 hours.  For slow logs, this is 4 days.  
[15:10:12] <icinga-wm>	 RECOVERY - puppet last run on labmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[15:21:00] <ottomata>	 godog: so, tell me not to send 40K metrics to statsd/graphite at this time.
[15:22:35] <godog>	 ottomata: hehe, yeah that's a bit too much, we have a machine with ssd meant to be replacing tungsten on order though
[15:23:10] <_joe_>	 godog: still one machine?
[15:24:25] <godog>	 yep, one in eqiad and one in codfw, with 4x ssd each should be able to handle it
[15:26:17] <ottomata>	 ok
[15:26:33] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: fix silly typo in bigbrother [puppet] - 10https://gerrit.wikimedia.org/r/177549 
[15:27:58] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Typo fix" [puppet] - 10https://gerrit.wikimedia.org/r/177549 (owner: 10coren)
[15:31:00] <grrrit-wm>	 (03PS1) 10Cscott: More AppArmor fixes for OCG. [puppet] - 10https://gerrit.wikimedia.org/r/177550 
[15:31:43] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] Initial Debian packaging for apertium-urd-hin [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/176924 (owner: 10KartikMistry)
[15:31:51] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [V: 032] Initial Debian packaging for apertium-urd-hin [debs/contenttranslation/apertium-urd-hin] - 10https://gerrit.wikimedia.org/r/176924 (owner: 10KartikMistry)
[15:32:17] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Added initial Debian packaging for apertium-id-ms [debs/contenttranslation/apertium-id-ms] - 10https://gerrit.wikimedia.org/r/176889 (owner: 10KartikMistry)
[15:33:51] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Added initial Debian packaging [debs/contenttranslation/apertium-eo-en] - 10https://gerrit.wikimedia.org/r/176886 (owner: 10KartikMistry)
[15:40:10] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] More AppArmor fixes for OCG. [puppet] - 10https://gerrit.wikimedia.org/r/177550 (owner: 10Cscott)
[15:41:17] <marktraceur>	 Who's doing them thar SWAT?
[15:50:28] * anomie sees nothing for SWAT this morning
[16:00:04] <jouncebot>	 manybubbles, anomie, ^d, marktraceur: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141204T1600).
[16:00:20] * anomie still sees nothing for SWAT this morning
[16:00:40] <marktraceur>	 *sigh* fine, I'll do it.
[16:00:45] * marktraceur declares SWAT closed
[16:00:47] <Reedy>	 gj marktraceur
[16:00:57] <marktraceur>	 Thanks Reedy, I try
[16:00:58] <^d>	 best swat ever
[16:01:14] <marktraceur>	 Where's quips.cgi when you need it
[16:01:14] <manybubbles>	 nice
[16:01:27] <Reedy>	 https://old-bugzilla.wikimedia.org/
[16:01:31] <Reedy>	 it still works ;)
[16:01:44] <Reedy>	 wut
[16:01:58] <Reedy>	 it's been emptied
[16:02:04] <Reedy>	 "No new entries may be submitted at this time."
[16:02:06] <Reedy>	 bleugh
[16:02:10] <Reedy>	 damn you andre__ for fixing it
[16:02:12] <Reedy>	 :P
[16:02:27] <marktraceur>	 Curses.
[16:02:33] <marktraceur>	 Wait, are all the old quips gone!?
[16:02:54] <marktraceur>	 Obviously today's SWAT should be focused on recovering them.
[16:02:55] <Reedy>	 Nope, they're there on edit
[16:02:56] <Reedy>	 https://old-bugzilla.wikimedia.org/quips.cgi?action=show
[16:03:03] <Reedy>	 mutante|travel pastebinned them too
[16:06:10] <andre__>	 Reedy: sorry for closing down your playground ;)
[16:08:19] <marktraceur>	 Maybe we should just put them on mw.org?
[16:28:53] <bd808>	 !log restarted elasticsearch on logstash1001.
[16:29:00] <morebots>	 Logged the message, Master
[16:29:22] <icinga-wm>	 PROBLEM - RAID on analytics1021 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[16:29:23] <jgage>	 bd808, another oom? :(
[16:29:50] <bd808>	 jgage: corrupt index :/
[16:30:10] <bd808>	 org.apache.lucene.index.CorruptIndexException: [logstash-2014.11.26][0] Corrupted index [corrupted_f1vhJijhRvmkw4dyyz36zw] caused by: CorruptIndexException[codec footer mismatch: actual footer=962802488 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/var/lib/elasticsearch/production-logstash-eqiad/nodes/0/indices/logstash-2014.11.26/0/index/_8nq.fdt"))]
[16:30:17] <bd808>	 and restart did not fix
[16:30:26] <jgage>	 blah, wonder how that happens
[16:30:46] <jgage>	 my ssh to analytics1021 is timing out and i have a meeting starting now
[16:30:54] <jgage>	 will investigate after unless someone beats me to it
[16:32:01] <icinga-wm>	 RECOVERY - RAID on analytics1021 is OK: OK: no disks configured for RAID  
[16:34:38] <icinga-wm>	 PROBLEM - Kafka Broker Server on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties  
[16:35:23] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on analytics1018 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 14.0  
[16:35:41] <ottomata>	 uhhhhHHHHH
[16:35:43] <ottomata>	 ok!
[16:36:47] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on analytics1012 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 18.0  
[16:37:23] <_joe_>	 ottomata: I got paged, do you need help or are you handling this?
[16:37:28] <_joe_>	 looks bad
[16:38:27] <ottomata>	 yeah looks bad.
[16:38:29] <ottomata>	 i am looking
[16:38:34] <ottomata>	 looks like sdj has a big ol' problem
[16:38:36] <icinga-wm>	 PROBLEM - Kafka Broker Under Replicated Partitions on analytics1022 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 13.0  
[16:38:39] <ottomata>	 the other kafka servers should be fine...
[16:38:56] <ottomata>	 under replicated is ok.  it just means that they don't have full replication
[16:39:02] <ottomata>	 because one of the replicas is down
[16:39:12] <ottomata>	 not sure why there are so many delivery errors from varnishkafka right now
[16:40:37] <chasemp>	 well a counter will just average and a guage will replace w/ last value and it's really graphite where you would see the difference as statsd only holds state for teh flush interval...which I think is 60 seconds
[16:40:43] <chasemp>	 oops that was in response to long ago text 
[16:40:55] <chasemp>	 hooray for text caching in my irc bar
[16:41:28] <ottomata>	 i will ask you qs about that shortly, chasemp
[16:42:00] <ottomata>	 hm
[16:42:01] <ottomata>	 ext4_discard_preallocations:3894: comm java: Error loading buddy information for
[16:42:07] <_joe_>	 ok, my next question would be "why did I got paged then?", but I'll leave that for later
[16:43:27] <ottomata>	 _joe_ there is a pageable problem :)
[16:43:31] <ottomata>	 disk failuer on analytics1021
[16:43:47] <ottomata>	 cmjohnson: yt?
[16:44:37] <cmjohnson>	 i'm here
[16:44:51] <jgage>	 omg what if a slowly failing disk has been the root cause of problems iwth that box for the past several months
[16:45:15] <ottomata>	 haha, maybe!
[16:45:23] <Reedy>	 jgage: computers suck
[16:45:26] <ottomata>	 cmjohnson: got a failed disk on a kafka broker, can you be a hero?
[16:45:36] <cmjohnson>	 ottomata: on 1021?
[16:45:38] <ottomata>	 yes
[16:45:49] <ottomata>	 this isn't super duper urgent, as it look slike the rest of the system is ok
[16:45:55] <ottomata>	 but the sooner the better
[16:46:50] <cmjohnson>	 it's a warranty issue. I will submit ticket to dell now
[16:46:56] <ottomata>	 yeah, cmjohnson analytics1021, sdj
[16:47:01] <cmjohnson>	 yep..i see it
[16:47:06] <ottomata>	 k
[16:47:10] <ottomata>	 got any spares?
[16:47:19] <cmjohnson>	 cat /proc/mdstat doesn't show an error though 
[16:47:25] <ottomata>	 it isn't in raid
[16:47:27] <cmjohnson>	 no, we don't have spares of disks under warranty 
[16:47:31] <ottomata>	 oo
[16:49:34] <ottomata>	 cmjohnson: so, we have to wait for a disk to be delivered?
[16:50:05] <cmjohnson>	 yep..it should be here tomorrow unless they have a questoin
[16:50:05] <cmjohnson>	 typically not
[16:50:11] <ottomata>	 hmmmMMM
[16:50:12] <ottomata>	 ok.
[16:50:22] <ottomata>	 this is gonna be interesting
[16:50:52] <ottomata>	 don't think I've had to restore a broker that will be so many hours behind....I can do it :)
[16:51:01] <ottomata>	 i think I can bump up the speed it will consume and replicate 
[16:51:06] <ottomata>	 until it catches up.
[16:51:17] <ottomata>	 hm, cmjohnson, seems like we should ahve spares on hand though, right?
[16:51:44] <ottomata>	 cmjohnson: should I make an RT to track this?
[16:51:59] <ottomata>	 (yes, yes?)
[16:52:06] <cmjohnson>	 ottomata: that has been ongoing discussion...Dell won't send us spares for "just in case" 
[16:52:12] <cmjohnson>	 yes please submit a ticket
[16:53:37] <ottomata>	 cmjohnson: https://rt.wikimedia.org/Ticket/Display.html?id=8978
[16:53:39] <cmjohnson>	 ottomata: the work order for dell has been submitted.  I will udpate as soon as they approve
[16:54:17] <icinga-wm>	 ACKNOWLEDGEMENT - Kafka Broker Under Replicated Partitions on analytics1012 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 18.0 ottomata sdj died on analytics1021.  Waiting for replacement: https://rt.wikimedia.org/Ticket/Display.html?id=8978
[16:54:17] <icinga-wm>	 ACKNOWLEDGEMENT - Kafka Broker Under Replicated Partitions on analytics1018 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 14.0 ottomata sdj died on analytics1021.  Waiting for replacement: https://rt.wikimedia.org/Ticket/Display.html?id=8978
[16:54:17] <icinga-wm>	 ACKNOWLEDGEMENT - Kafka Broker Server on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties ottomata sdj died on analytics1021.  Waiting for replacement: https://rt.wikimedia.org/Ticket/Display.html?id=8978
[16:54:28] <ottomata>	 cmjohnson: should we order our own spares then, for just in case?
[16:54:34] <icinga-wm>	 ACKNOWLEDGEMENT - Kafka Broker Under Replicated Partitions on analytics1022 is CRITICAL: kafka.server.ReplicaManager.UnderReplicatedPartitions.Value CRITICAL: 13.0 ottomata sdj died on analytics1021.  Waiting for replacement: https://rt.wikimedia.org/Ticket/Display.html?id=8978
[16:55:44] <cmjohnson>	 ottomata: I don't think that is necessary and we haven't done it in the past w/ exception of ceph but they were failing at a high rate at the time. 
[16:56:21] <ottomata>	 hmmm, oooooook
[16:57:13] <cmjohnson>	 feel free to appeal to mark 
[16:57:42] <grrrit-wm>	 (03PS1) 10Ottomata: [WIP] Use logster to parse mobile vanrishkafka logs and send to statsd [puppet] - 10https://gerrit.wikimedia.org/r/177562 
[16:57:54] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 04-1] Change ru.wikinews.org to HTTPS only. (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[16:58:12] <grrrit-wm>	 (03CR) 10Ottomata: [C: 04-2] "Graphite isn't ready for this jelly." [puppet] - 10https://gerrit.wikimedia.org/r/177562 (owner: 10Ottomata)
[16:59:05] <icinga-wm>	 RECOVERY - Kafka Broker Server on analytics1021 is OK: PROCS OK: 1 process with command name java, args kafka.Kafka /etc/kafka/server.properties  
[16:59:41] <_joe_>	 ottomata: I am being flooded with pages
[16:59:46] <_joe_>	 why is that?
[16:59:51] <_joe_>	 we never do this for a single host
[16:59:57] <_joe_>	 unless it's vital to prod
[17:00:07] <jgage>	 huh really? i only got 3.
[17:00:30] <_joe_>	 jgage: which, given I get at most 2 when wikipedia goes down, is a lot :)
[17:00:34] <jgage>	 down, ack, up
[17:03:17] <ottomata>	 hm
[17:03:23] <ottomata>	 ah, its puppet maybe?
[17:03:28] <ottomata>	 trying to start kafka, but then kafka dies
[17:03:33] <ottomata>	 checking
[17:03:58] <icinga-wm>	 RECOVERY - puppet last run on analytics1027 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures  
[17:04:40] <ottomata>	 ah, i unmounted sdj and it is now writing to root partition! stopping, and disabling puppet, and acknoweding
[17:09:19] <grrrit-wm>	 (03PS1) 10Ottomata: Disable log age monitoring for udp2log-lucene now that most searches are in elasticsearch [puppet] - 10https://gerrit.wikimedia.org/r/177565 
[17:09:50] <grrrit-wm>	 (03PS2) 10Ottomata: Disable log age monitoring for udp2log-lucene now that most searches are in elasticsearch [puppet] - 10https://gerrit.wikimedia.org/r/177565 
[17:10:57] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 031] "Actually, my memory failed me: RewriteRules are evaluated before ProxyPass is. So I guess it's fine." [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[17:11:51] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Disable log age monitoring for udp2log-lucene now that most searches are in elasticsearch [puppet] - 10https://gerrit.wikimedia.org/r/177565 (owner: 10Ottomata)
[17:13:12] <ottomata>	 _joe_, i'm not sure what pages you got
[17:13:30] <ottomata>	 nor am I very familiar with how pages get generated.
[17:13:59] <ottomata>	 does a page go out whenever there is a critical alert in icinga?
[17:14:20] <icinga-wm>	 RECOVERY - udp2log log age for lucene on oxygen is OK: OK: all log files active  
[17:15:24] <bblack>	 ottomata: yeah kafka paged me too
[17:15:40] <bblack>	 but I think only like trice?
[17:15:58] <bblack>	 *thrice
[17:16:44] <bblack>	 he's in the same sms contactgroup we're all in, I'd think we'd all get the same pages.
[17:17:07] <ottomata>	 it should probably page if the kafka process dies, which are the ones jgage says he got
[17:17:18] <ottomata>	 but, not for the others (underreplicated partitions, etc.)
[17:20:00] <_joe_>	 ottomata: nagios_critical => true
[17:25:24] <Lydia_WMDE>	 jzerebecki: manybubbles: meet each other to talk query meeting timing :D
[17:26:17] <jzerebecki>	 suggestions?
[17:26:24] <ottomata>	 ok,yeah, _joe_ it looks lke that is only set for the kafka process
[17:26:26] <ottomata>	 to make sure it is running
[17:26:29] <ottomata>	 is that ok?
[17:26:42] <manybubbles>	 jzerebecki: we have a 2x a week get together to talk about technical issues and progress.  Its in 3 and a half hours on Monday and Thursday
[17:27:02] <manybubbles>	 it was a convenient time for me and the folks in SF but I imagine its pretty late for you
[17:27:44] <jzerebecki>	 yea 2200 is not that convenient
[17:28:45] <bd808>	 !log restarted elasticsearch on logstash1003 to see if the missing indices would recover
[17:28:47] <morebots>	 Logged the message, Master
[17:30:40] <jzerebecki>	 manybubbles: how about 1000 sf time, thats 1900 berlin
[17:48:16] <awight>	 Hey uh I thought "retest" was supposed to trigger magic zuul testing in every repo
[17:48:27] <awight>	 But when I tried it here, nothing happened: https://gerrit.wikimedia.org/r/#/c/177455/
[17:48:59] <bblack>	 maybe it takes a bit?
[17:49:17] <bblack>	 or is it recheck?
[17:49:57] <grrrit-wm>	 (03PS1) 10BBlack: Fix PURGE regex for betalabs upload servers [puppet] - 10https://gerrit.wikimedia.org/r/177576 
[17:50:00] <Reedy>	 recheck
[17:54:41] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] Fix PURGE regex for betalabs upload servers [puppet] - 10https://gerrit.wikimedia.org/r/177576 (owner: 10BBlack)
[17:57:27] <awight>	 that's funny.  Only half joking, it might be nice to liberally accept spelling and other brain errors.
[18:00:41] <grrrit-wm>	 (03CR) 10BBlack: [C: 031] "My +1 is conditional on you waiting to commit this until about 2am Friday night US time :)" [puppet] - 10https://gerrit.wikimedia.org/r/177521 (owner: 10Ori.livneh)
[18:15:19] <grrrit-wm>	 (03PS2) 10Aaron Schulz: Declare profiler in CLI mode (though it is not used) to support --profiler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177481 
[18:22:45] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).  
[18:23:12] <bd808>	 !log restarted elasticsearch on logstash1002
[18:23:19] <morebots>	 Logged the message, Master
[18:23:53] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet).  
[18:28:47] <bd808>	 !log Rolling restart of the elasticsearch cluster for logstash did not fix corrupted logstash-2014.11.30 index. It was worth a shot.
[18:28:51] <morebots>	 Logged the message, Master
[18:33:41] <legoktm>	 !log ran removeInvalidEmails.php on testwiki
[18:33:43] <morebots>	 Logged the message, Master
[18:35:47] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge.  
[18:37:39] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 
[18:37:42] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge.  
[18:38:23] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 (owner: 10Andrew Bogott)
[18:43:52] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 
[18:44:40] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 (owner: 10Andrew Bogott)
[18:46:35] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 
[18:47:17] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 (owner: 10Andrew Bogott)
[18:48:10] <grrrit-wm>	 (03PS4) 10Andrew Bogott: Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 
[18:48:14] <andrewbogott>	 omg I'm so dumb today
[18:49:15] <bd808>	 jgage: That index is just gone. I may have made it all worse by doing the rolling restart but there are now no data files in any of the hosts. :( I'm going to drop the empty index so the cluster health comes back to green.
[18:51:43] <jgage>	 that is.. troubling :(
[18:52:09] <jgage>	 for the record i haven't taken any actions which might have contributed to this situation
[18:52:39] <jgage>	 i struggle to understand how it could be totally gone when we store orig + 2 replicas
[18:53:52] <bd808>	 Maybe my restart made it try to sync the brokenness? I'm not sure either
[18:54:33] <bd808>	 There is one shard of the logstash-2014.11.26 index that seems similarly broken, but the other 2 nodes for it are fine
[18:54:50] <bd808>	 It is not empty on disk though, just showing as down and testing as corrupt
[18:55:21] <grrrit-wm>	 (03PS5) 10Tim Landscheidt: Add an 'apache' user to eqiad labstores. [puppet] - 10https://gerrit.wikimedia.org/r/177584 (owner: 10Andrew Bogott)
[18:55:21] <jgage>	 i wonder if manybubbles might have some ideas
[18:55:29] <manybubbles>	 ?
[18:55:32] <manybubbles>	 reading
[18:55:40] <bd808>	 This blog post had the magic command that I should have run before doing any restarts -- http://joejulian.name/blog/how-to-check-and-fix-indexes-for-elasticsearchlogstash/
[18:56:22] <jgage>	 hm interesting *adds to notes*
[18:56:32] <grrrit-wm>	 (03CR) 10Chad: [C: 032] Declare profiler in CLI mode (though it is not used) to support --profiler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177481 (owner: 10Aaron Schulz)
[18:56:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: Declare profiler in CLI mode (though it is not used) to support --profiler [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177481 (owner: 10Aaron Schulz)
[18:57:05] <manybubbles>	 I've never had to deal with a corrupt index before
[18:57:36] <logmsgbot>	 !log demon Synchronized wmf-config/StartProfiler.php: (no message) (duration: 00m 05s)
[18:57:40] <manybubbles>	 so you had a corrupt shard and you think elasticsearch synced the corruption everywhere
[18:57:41] <morebots>	 Logged the message, Master
[18:57:46] * jgage afk for meeting
[18:57:51] <grrrit-wm>	 (03PS1) 10Ottomata: Use bucketType 'g' for all Kafka jmxtrans metrics sent to statsd [puppet/kafka] - 10https://gerrit.wikimedia.org/r/177591 
[18:58:40] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Use bucketType 'g' for all Kafka jmxtrans metrics sent to statsd [puppet/kafka] - 10https://gerrit.wikimedia.org/r/177591 (owner: 10Ottomata)
[18:58:44] <bd808>	 manybubbles: Well I actually saw messages about the corruption in all 3 server logs before I started poking
[18:58:55] <manybubbles>	 hmm
[18:59:33] <bd808>	 manybubbles: org.apache.lucene.index.CorruptIndexException: [logstash-2014.11.26][0] Corrupted index [corrupted_f1vhJijhRvmkw4dyyz36zw] caused by: CorruptIndexException[codec footer mismatch: actual footer=962802488 vs expected footer=-1071082520 (resource: NIOFSIndexInput(path="/var/lib/elasticsearch/production-logstash-eqiad/nodes/0/indices/logstash-2014.11.26/0/index/_8nq.fdt"))]
[18:59:41] <manybubbles>	 bd808: I see it
[18:59:49] <icinga-wm>	 RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge.  
[19:00:35] <grrrit-wm>	 (03PS1) 10Ottomata: Update kafka module with all jmxtrans metrics in statsd as gauge type [puppet] - 10https://gerrit.wikimedia.org/r/177593 
[19:00:48] <grrrit-wm>	 (03PS2) 10Ottomata: Update kafka module with all jmxtrans metrics in statsd as gauge type [puppet] - 10https://gerrit.wikimedia.org/r/177593 
[19:01:00] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Update kafka module with all jmxtrans metrics in statsd as gauge type [puppet] - 10https://gerrit.wikimedia.org/r/177593 (owner: 10Ottomata)
[19:01:18] <bd808>	 oh... that's the one that's still there
[19:01:33] <manybubbles>	 bd808: so it _looks like_ elasticsearch will refuse replicate a corrupt index
[19:02:24] <bd808>	 the 11.30 thing may be a red herring -- I mean it's gone but that message is about the one that is still around
[19:02:28] * bd808 reads logs again
[19:02:33] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Switch Gerrit's 'Report Bug' url to Phabricator [puppet] - 10https://gerrit.wikimedia.org/r/176267 (owner: 10QChris)
[19:03:29] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Drop 'Phabricator' suffix from gerrit's ITS actions [puppet] - 10https://gerrit.wikimedia.org/r/176266 (owner: 10QChris)
[19:03:50] <grrrit-wm>	 (03PS2) 10Ottomata: Move gerrit's remaining ITS templates into gerrit module [puppet] - 10https://gerrit.wikimedia.org/r/176264 (owner: 10QChris)
[19:04:12] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Move gerrit's remaining ITS templates into gerrit module [puppet] - 10https://gerrit.wikimedia.org/r/176264 (owner: 10QChris)
[19:04:33] <grrrit-wm>	 (03CR) 10Ottomata: [V: 032] Move gerrit's remaining ITS templates into gerrit module [puppet] - 10https://gerrit.wikimedia.org/r/176264 (owner: 10QChris)
[19:04:44] <grrrit-wm>	 (03PS2) 10Ottomata: Remove hooks-bugzilla configuration [puppet] - 10https://gerrit.wikimedia.org/r/176265 (owner: 10QChris)
[19:04:59] <bd808>	 jgage, manybubbles: so the only mention of the 11.30 index in any logs is "[2014-11-30 03:50:49,771][INFO ][cluster.metadata         ] [logstash1002] [logstash-2014.11.30] creating index, cause [auto(bulk api)], shards [1]/[2], mappings [_default_]"
[19:05:37] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Remove hooks-bugzilla configuration [puppet] - 10https://gerrit.wikimedia.org/r/176265 (owner: 10QChris)
[19:05:43] <grrrit-wm>	 (03PS2) 10Ottomata: Drop 'Phabricator' suffix from gerrit's ITS actions [puppet] - 10https://gerrit.wikimedia.org/r/176266 (owner: 10QChris)
[19:06:07] <grrrit-wm>	 (03CR) 10Ottomata: [V: 032] Drop 'Phabricator' suffix from gerrit's ITS actions [puppet] - 10https://gerrit.wikimedia.org/r/176266 (owner: 10QChris)
[19:06:17] <grrrit-wm>	 (03PS2) 10Ottomata: Switch Gerrit's 'Report Bug' url to Phabricator [puppet] - 10https://gerrit.wikimedia.org/r/176267 (owner: 10QChris)
[19:06:30] <grrrit-wm>	 (03CR) 10Ottomata: [V: 032] Switch Gerrit's 'Report Bug' url to Phabricator [puppet] - 10https://gerrit.wikimedia.org/r/176267 (owner: 10QChris)
[19:07:12] <bd808>	 Was logstash broken over the weekend? Maybe it was empty because the collectors were all borked.
[19:07:48] <bblack>	 could be related to fallout from the eqiad rack outage
[19:07:58] <bblack>	 I think logstash machines were among the ones knocked off the network
[19:08:09] <bd808>	 2014-12-01 SAL says -- 15:49 bd808: restarted logstash on logstash1001; log2udp events were not being processed
[19:08:28] <bd808>	 So maybe the mystery is not so mysterious
[19:08:33] <bblack>	 :)
[19:08:44] <bblack>	 and yeah, logstash100[123] were offline for 6.5 hours on Sunday
[19:08:45] <bblack>	 https://wikitech.wikimedia.org/wiki/Incident_documentation/20141130-Eqiad-Rack-C4
[19:09:05] <bd808>	 bam. thanks bblack 
[19:11:50] <grrrit-wm>	 (03PS1) 10Ejegg: Make spiders ignore BannerLoader and RecordImpression [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177598 
[19:12:43] <manybubbles>	 bd808: I'm going to move the copy of 11.26 aside on logstash1001 (the only one that isn't pick it up) and bounce the elasticsearch process.  it should reload it from 1003 which seems to think it isn't corrupted.
[19:13:12] <bd808>	 manybubbles: cool. I was thinking about doing the same thing
[19:13:37] <bd808>	 I tried fixing it with org.apache.lucene.index.CheckIndex and that didn't seem to make it happy
[19:13:38] <manybubbles>	 1002 complained about seeing corruption as well but seems to have picked up the index ok this time
[19:16:32] <legoktm>	 !log running removeInvalidEmails.php across all wikis
[19:16:37] <morebots>	 Logged the message, Master
[19:19:09] <grrrit-wm>	 (03PS3) 10BBlack: Change ru.wikinews.org to HTTPS only. [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[19:19:17] <manybubbles>	 bd808: that didn't make it happy. it still claims to be "initializing" it
[19:20:25] <bblack>	 gerrit's 503-ing :/
[19:20:32] <RoanKattouw>	 Gerrit down?
[19:21:20] <bblack>	 and it's back magically!
[19:21:26] <manybubbles>	 works for me
[19:21:27] <manybubbles>	 oh
[19:21:28] <manybubbles>	 yeah
[19:21:32] <manybubbles>	 you too
[19:22:36] <bblack>	 hmm I wonder what's up with apache lints recently failing with: 19:19:20 ERROR:zuul.Repo:Unable to initialize repo for https://gerrit.wikimedia.org/r/p/operations/mediawiki-config
[19:23:28] <grrrit-wm>	 (03CR) 10BBlack: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[19:25:10] <RoanKattouw>	 bblack: Probably caused by that Gerrit outage
[19:25:12] <RoanKattouw>	 ?
[19:25:44] <bblack>	 well it's still happening, and I saw it on another change earlier today as well (which luckily wasn't an apache change anyways)
[19:26:10] <bblack>	 19:23:34 IOError: Lock for file '/srv/ssd/jenkins-slave/workspace/operations-apache-config-lint/src/operations/mediawiki-config/.git/config' did already exist, delete '/srv/ssd/jenkins-slave/workspace/operations-apache-config-lint/src/operations/mediawiki-config/.git/config.lock' in case the lock is illegal
[19:26:26] <bblack>	 probably some racy code left a lock that never gets cleaned up
[19:27:55] <grrrit-wm>	 (03CR) 10BBlack: "rehceck" [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[19:30:17] <bblack>	 (and yeah the lock was over a day old, I killed it)
[19:33:25] <grrrit-wm>	 (03CR) 10Awight: [C: 031] "Great! Leaving merge for deployment" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177598 (owner: 10Ejegg)
[19:35:25] <grrrit-wm>	 (03CR) 10BBlack: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[19:35:41] <bblack>	 ... do something zuul!
[19:37:58] <bblack>	 19:35:34 AttributeError: 'IterableList' object has no attribute 'origin'
[19:38:01] <bblack>	 19:35:34 Build step 'Execute shell' marked build as failure
[19:38:24] <bblack>	 I really don't know wtf is up with zuul + apache-lint + mediawiki-config.  something has changed for the worse recently.
[19:38:40] <bblack>	 after fixing the broken lock: https://integration.wikimedia.org/ci/job/operations-apache-config-lint/4690/console
[19:38:56] <grrrit-wm>	 (03PS1) 10MaxSem: Disable gadgets caching [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177600 
[19:39:18] <bblack>	 anyways, merging without it, since it was passing the lint before recently trivial changes / rebases :p
[19:39:41] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] Change ru.wikinews.org to HTTPS only. [puppet] - 10https://gerrit.wikimedia.org/r/173078 (owner: 10JanZerebecki)
[19:40:00] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1387.133301  
[19:44:38] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0  
[19:49:57] <icinga-wm>	 PROBLEM - check_puppetrun on backup4001 is CRITICAL: CRITICAL: puppet fail  
[19:51:24] <grrrit-wm>	 (03PS1) 10Milimetric: Add job to crunch the Edit team's EL data [puppet] - 10https://gerrit.wikimedia.org/r/177602 
[19:51:51] <grrrit-wm>	 (03PS2) 10Milimetric: Add job to crunch the Edit team's EL data [puppet] - 10https://gerrit.wikimedia.org/r/177602 
[19:52:45] <grrrit-wm>	 (03CR) 10Milimetric: [C: 04-1] "the contents of the related data repo (referenced by convention here) need to be vetted by Edit team first. I will +1 when this can be me" [puppet] - 10https://gerrit.wikimedia.org/r/177602 (owner: 10Milimetric)
[19:55:02] <icinga-wm>	 PROBLEM - check_puppetrun on backup4001 is CRITICAL: CRITICAL: puppet fail  
[19:55:12] <icinga-wm>	 PROBLEM - check_puppetrun on indium is CRITICAL: CRITICAL: puppet fail  
[19:55:13] <icinga-wm>	 PROBLEM - check_puppetrun on db1025 is CRITICAL: CRITICAL: puppet fail  
[19:55:13] <icinga-wm>	 PROBLEM - check_puppetrun on silicon is CRITICAL: CRITICAL: puppet fail  
[19:55:13] <icinga-wm>	 PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: puppet fail  
[19:55:19] <icinga-wm>	 PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: puppet fail  
[19:55:20] <icinga-wm>	 PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: puppet fail  
[19:59:10] <jzerebecki>	 bblack: mh i think _joe_ was right, it seems as if rewrite rules are evaluated _after_ ProxyPass; which means quite a few rewrite rules were broken when ProxyPass was added; i'm unsure if it is the file order though, i remember reading something about ProxyPass always happening before rewrite. ideas?
[19:59:59] <_joe_>	 jzerebecki: how did you test that?
[20:00:03] <icinga-wm>	 RECOVERY - check_puppetrun on backup4001 is OK: OK: Puppet is currently enabled, last run 264 seconds ago with 0 failures  
[20:00:15] <icinga-wm>	 PROBLEM - check_puppetrun on indium is CRITICAL: CRITICAL: puppet fail  
[20:00:15] <icinga-wm>	 PROBLEM - check_puppetrun on db1025 is CRITICAL: CRITICAL: puppet fail  
[20:00:15] <icinga-wm>	 PROBLEM - check_puppetrun on silicon is CRITICAL: CRITICAL: puppet fail  
[20:00:15] <icinga-wm>	 PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: puppet fail  
[20:00:15] <icinga-wm>	 PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: puppet fail  
[20:00:16] <icinga-wm>	 PROBLEM - check_puppetrun on db1008 is CRITICAL: CRITICAL: puppet fail  
[20:00:19] <jzerebecki>	 _joe_: wget --debug -O /dev/null ru.wikinews.org
[20:00:33] <_joe_>	 jzerebecki: varnish cache?
[20:00:49] <jzerebecki>	 it does -> Mainpage -> https instead of the other way around
[20:00:55] <jzerebecki>	 _joe_: could be
[20:01:24] <_joe_>	 let me check
[20:01:29] <grrrit-wm>	 (03CR) 10Milimetric: [C: 031] Add job to crunch the Edit team's EL data [puppet] - 10https://gerrit.wikimedia.org/r/177602 (owner: 10Milimetric)
[20:01:41] <_joe_>	 oh, did you reload apache across the cluster?
[20:03:41] <_joe_>	 jzerebecki: it works on the backend, it's just varnish caches
[20:03:54] <jzerebecki>	 _joe_: thx
[20:04:43] <_joe_>	 and btw, I should update the banner for static content, right now it's X-Powered-By: HHVM/3.3.0-static, while our HHVM says 3.3.1 :)
[20:04:51] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Add job to crunch the Edit team's EL data [puppet] - 10https://gerrit.wikimedia.org/r/177602 (owner: 10Milimetric)
[20:05:12] <icinga-wm>	 PROBLEM - check_puppetrun on indium is CRITICAL: CRITICAL: puppet fail  
[20:05:15] <icinga-wm>	 RECOVERY - check_puppetrun on silicon is OK: OK: Puppet is currently enabled, last run 283 seconds ago with 0 failures  
[20:05:15] <icinga-wm>	 RECOVERY - check_puppetrun on db1025 is OK: OK: Puppet is currently enabled, last run 295 seconds ago with 0 failures  
[20:05:15] <icinga-wm>	 RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 262 seconds ago with 0 failures  
[20:05:19] <icinga-wm>	 PROBLEM - check_puppetrun on payments1002 is CRITICAL: CRITICAL: puppet fail  
[20:05:20] <icinga-wm>	 RECOVERY - check_puppetrun on db1008 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures  
[20:10:12] <icinga-wm>	 RECOVERY - check_puppetrun on indium is OK: OK: Puppet is currently enabled, last run 250 seconds ago with 0 failures  
[20:10:26] <icinga-wm>	 RECOVERY - check_puppetrun on payments1002 is OK: OK: Puppet is currently enabled, last run 234 seconds ago with 0 failures  
[20:15:55] <grrrit-wm>	 (03CR) 10Awight: [C: 032] Make spiders ignore BannerLoader and RecordImpression [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177598 (owner: 10Ejegg)
[20:16:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Make spiders ignore BannerLoader and RecordImpression [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177598 (owner: 10Ejegg)
[20:17:23] <logmsgbot>	 !log awight Synchronized robots.txt: Disallow banner stuff in robots.txt (duration: 00m 07s)
[20:17:28] <morebots>	 Logged the message, Master
[20:21:59] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Rename all webrequest varnishkafka instances as 'webrequest' [puppet] - 10https://gerrit.wikimedia.org/r/177546 (owner: 10Ottomata)
[20:23:08] <grrrit-wm>	 (03CR) 10Ottomata: [C: 04-2] "Cool." [puppet] - 10https://gerrit.wikimedia.org/r/177546 (owner: 10Ottomata)
[20:30:34] <grrrit-wm>	 (03CR) 10JanZerebecki: "The redirect was deployed. Can someone pick this up for deployment?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173083 (owner: 10JanZerebecki)
[20:31:39] <grrrit-wm>	 (03CR) 10MaxSem: "Ading to today's SWAT." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173083 (owner: 10JanZerebecki)
[20:34:59] <bblack>	 jzerebecki / _joe_ : and no, I didn't restart apache.  I kinda figured there might be a step like that.
[20:35:22] <_joe_>	 bblack: it did restart automagically in fact
[20:35:27] <bblack>	 ok
[20:35:32] <_joe_>	 its working on the backend
[20:35:35] <bblack>	 so yeah then the next thing is purging the caches
[20:37:41] <ori>	 _joe_: some awesome discoveries in #hhvm-dev, i have to paste the log for you
[20:38:00] <_joe_>	 ori: oh yes, thanks please
[20:38:14] <_joe_>	 I should remember to join
[20:38:34] <_joe_>	 (good morning)
[20:39:27] <_joe_>	 btw, mw1081 had hyperthreading disabled, I enabled it today, so cpu figures are basically cut in half, but the system cpu reduction is still there
[20:40:29] <MaxSem>	 hmm, HHVM takes advantage of HT?
[20:40:49] <_joe_>	 MaxSem: yes
[20:41:08] <_joe_>	 MaxSem: most software do with modern kernels
[20:42:14] <ori>	 _joe_: https://dpaste.de/75gT/raw
[20:45:58] <bblack>	 jzerebecki: caches cleared for ru.wn.o now, it should reliably behave properly from a public pov
[20:49:32] <grrrit-wm>	 (03CR) 10BBlack: [C: 031] Change ru.wikinews.org to HTTPS only. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/173083 (owner: 10JanZerebecki)
[20:53:03] <jzerebecki>	 bblack: thx. works as expected. i guess there is no harm in not clearing the caches for the next site we switch.
[20:53:23] <grrrit-wm>	 (03PS1) 10Ottomata: Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 
[20:53:49] <bblack>	 for smaller traffic loads it's fine.  it's not ideal if we wipe large portions of the shared cache.
[20:53:52] <grrrit-wm>	 (03PS2) 10Ottomata: Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 
[20:54:25] <bblack>	 for some of the larger sites, when we get to those, we may have to use some other strategy.  either do it relatively-slowly, or deploy a JS workaround like ru.wn.o did first and then let the caches expire naturally, or something.
[20:54:36] <ottomata>	 ori: would you take a look at this too? 
[20:54:36] <ottomata>	 https://gerrit.wikimedia.org/r/#/c/177616/2/manifests/role/analytics/kafka.pp
[20:54:38] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 (owner: 10Ottomata)
[20:54:43] <ottomata>	 oop
[20:54:50] <ori>	 ottomata: sure
[20:55:21] <grrrit-wm>	 (03PS3) 10Ottomata: Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 
[20:55:44] <bblack>	 jzerebecki: (probably the former actually, I think we could do a wipe over the course of several hours slowly for most things.  the only really painful one would be major chunks of upload multimedia content)
[20:56:33] <grrrit-wm>	 (03PS1) 10Ori.livneh: mw1081: disable StatCache and toggle hhvm.check_sym_link to compare impact [puppet] - 10https://gerrit.wikimedia.org/r/177617 
[20:56:55] <legoktm>	 !log removeInvalidEmails.php finished, removed a total of 218,598 emails
[20:56:58] <morebots>	 Logged the message, Master
[20:57:38] <ori>	 _joe_: ^^
[20:59:32] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 (owner: 10Ottomata)
[21:00:01] <_joe_>	 ori: not sure that's a good idea
[21:00:29] <ori>	 _joe_: the bulk of code is loaded via the autoloader, rather than include / require
[21:00:42] <ori>	 we only use symlinks for a few entry-points which consist of very little code
[21:00:46] <ori>	 let's try it on one host?
[21:01:00] <_joe_>	 ori: uhm and the autoloader doesn't use absolute paths, right?
[21:01:23] <_joe_>	 yeah worst that can happen is hhvm finishing the cache and crashing
[21:01:38] <ori>	 we can track that in ganglia
[21:01:45] <ori>	 the cache sizes are plotted
[21:01:45] <_joe_>	 which we would notice
[21:01:53] <_joe_>	 yes
[21:02:12] <ori>	 and even if it crashed without us noticing, it'd be one app server, and it'd get restarted, so nothing too horrible
[21:02:39] <ori>	 i thought the idea of the hiera overrides was to make this sort of experimentation possible :)
[21:02:42] <grrrit-wm>	 (03CR) 10Giuseppe Lavagetto: [C: 031] "Let's try this, but keep an eye in particular to the cache sizes." [puppet] - 10https://gerrit.wikimedia.org/r/177617 (owner: 10Ori.livneh)
[21:02:48] <ori>	 :)
[21:02:59] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] mw1081: disable StatCache and toggle hhvm.check_sym_link to compare impact [puppet] - 10https://gerrit.wikimedia.org/r/177617 (owner: 10Ori.livneh)
[21:03:28] <_joe_>	 ori: yeah, I just want to be reasonably sure we're not doing something obviously wrong or dangerous
[21:03:37] <ori>	 yeah makes sense
[21:04:03] <ori>	 hmm, puppet-merge hangs on 'Fetching new commits from https://gerrit.wikimedia.org/r/p/operations/puppet '
[21:05:00] <YuviPanda>	 hmm, no coren.
[21:05:20] <grrrit-wm>	 (03PS4) 10Ottomata: Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 
[21:05:27] <_joe_>	 while you're at it, restart ganglia-monitor on that host. For some reason, it's not picking up hhvm stats
[21:05:31] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Add anomaly detection for # of messages sent to Kafka [puppet] - 10https://gerrit.wikimedia.org/r/177616 (owner: 10Ottomata)
[21:05:36] <ori>	 _joe_: will do
[21:06:03] <ottomata>	 ori, can I merge yours too?
[21:06:05] <ottomata>	 or, you can merge mine.
[21:06:06] <ottomata>	 :)
[21:06:13] <ori>	 ottomata: yes, please merge mine
[21:06:19] <ottomata>	 done
[21:08:55] <ori>	 !log restarted ganglia-monitor on mw1081
[21:08:59] <morebots>	 Logged the message, Master
[21:12:00] <grrrit-wm>	 (03PS1) 10Ori.livneh: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 
[21:12:05] <ori>	 ottomata: ^ as you initially suggested. you were right
[21:12:08] <ottomata>	 :)
[21:12:21] <YuviPanda>	 hmm, so..
[21:12:28] <YuviPanda>	 how do I prevent transient puppet failures from spamming everyone?
[21:12:30] <_joe_>	 ori: maybe you tied the hhvm memory stats on ganglia to servers in the "hhvm" group?
[21:12:32] <grrrit-wm>	 (03CR) 10Ottomata: [C: 031] stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 (owner: 10Ori.livneh)
[21:12:37] <YuviPanda>	 like, apt-get failing, or gerrit 503'ing for a few mins
[21:12:57] <YuviPanda>	 while *also* not ignoring errors that are vital, like a service not restarting because a config script was wrong
[21:13:03] <YuviPanda>	 that'll make puppet fail only once, I think.
[21:13:21] <YuviPanda>	 hmm, so how to differentiate between failures that we care about and ones we don't.
[21:13:30] <ori>	 _joe_: did i? i'll check
[21:13:47] <ori>	 YuviPanda: you can't. i think the better approach is to aggregate alerts
[21:13:47] <_joe_>	 ori: newly imaged servers don't have the stats
[21:14:24] <YuviPanda>	 ori: in what sense? 'alert X from hosts Y, Z, C, X, 9, 42'?
[21:14:28] <ori>	 so instead of puppet failure on mw1081, puppet failure on mw1082, puppet failure on mw1083, repeated alerts become: Puppet failure on N hosts
[21:14:42] <ori>	 yeah, maybe you don't even have to enumerate each host, but just link to icinga or something
[21:14:44] <YuviPanda>	 right.
[21:15:07] <ori>	 i think either syslog or apache do something like
[21:15:14] <ori>	 'last message repeated NNN times'
[21:15:16] <ori>	 or something like that
[21:15:32] <YuviPanda>	 hmmm. but hosts, etc are different so would need some parsing.
[21:15:36] <ori>	 _joe_: maybe ganglia-monitor wasn't refreshed, hm
[21:15:51] <_joe_>	 now they're showing up, nevermind
[21:15:58] <YuviPanda>	 perhaps we make icinga output json or something and make some other service handle actual notifications, but complexity penalty
[21:16:01] <ori>	 odd
[21:16:26] <logmsgbot>	 !log awight Synchronized php-1.25wmf10/extensions/CentralNotice: Hide CentralNotice banners from Google (duration: 00m 06s)
[21:16:28] <ori>	 YuviPanda: https://en.wikipedia.org/wiki/Levenshtein_distance :)
[21:16:28] <_joe_>	 ori: right after you restarted it
[21:16:31] <morebots>	 Logged the message, Master
[21:16:38] <logmsgbot>	 !log awight Synchronized php-1.25wmf11/extensions/CentralNotice: Hide CentralNotice banners from Google (duration: 00m 07s)
[21:16:41] <morebots>	 Logged the message, Master
[21:16:47] <ori>	 _joe_: so maybe the ganglia-monitor service isn't being refreshed
[21:16:53] <YuviPanda>	 ori: well, still won't work for icinga since each alert doesn't know about any other alerts. need something to keep state.
[21:17:09] <_joe_>	 but that server was rebooted this morning...
[21:17:10] <ori>	 YuviPanda: the alerts themselves can pile up, as long as the email and irc relays can batch and aggregate
[21:17:20] <YuviPanda>	 right, that's what I meant
[21:17:29] <YuviPanda>	 we need a relay to handle email as well - right now that's just /usr/bin/mail 
[21:17:37] <ori>	 _joe_: hm, maybe the script doesn't handle the case gracefully where hhvm is not up yet
[21:17:53] <YuviPanda>	 it'll also need to handle actual paging.
[21:18:08] <_joe_>	 YuviPanda: give up
[21:18:25] <_joe_>	 YuviPanda: you can maybe the bot here in chat do that
[21:18:30] <ori>	 _joe_: i'd give the same advice to most other people, but YuviPanda could actually pull of something brilliant and insane here :P
[21:18:40] <_joe_>	 but let alone the icinga alerting
[21:18:44] <YuviPanda>	 but a monitoring system that sends alerts you can ignore seems a bit useless.
[21:18:55] <_joe_>	 ori: no please, you don't mess with paging, even when you're very good
[21:19:10] <_joe_>	 YuviPanda: so we may need to change that alarm ;)
[21:19:16] <ori>	 _joe_: btw yes, i think the gmond module would crash if hhvm is not up and just linger
[21:19:18] <ori>	 i'll fix
[21:19:28] <_joe_>	 ori: thanks
[21:19:45] <YuviPanda>	 _joe_: yeah, that was my first thought, but I can't find an easy way to differentiate between 'well, apt-get update failed, not a big deal' vs 'well, nginx restart failed, so you are running old config, haha'
[21:19:55] <YuviPanda>	 since both are transient failures that'll show up only in one run
[21:20:07] <YuviPanda>	 wait, batching wont' actually fix *that*
[21:20:24] <YuviPanda>	 only thing batching will fix is our current 'well, mod_passenger is dead, heh' errors
[21:20:45] <_joe_>	 that would be fixed by a service dependency
[21:20:55] <_joe_>	 so that all the puppet failure alarm
[21:21:05] <_joe_>	 depend on the https check on strontium and palladium
[21:21:23] <YuviPanda>	 _joe_: ah, I have that in shinken :) except of course the checks run on different times, and when shinken does check the puppetmaster things are ok there.
[21:21:32] <_joe_>	 and won't fire up if the check upper in the chain is failing
[21:21:44] <YuviPanda>	 hmm, perhaps I need to average out the puppetmaster check as well.
[21:22:28] <YuviPanda>	 heh, no way to check why apt-get update failed during a puppet run
[21:22:34] <YuviPanda>	 which is the majority of transient failures on labs
[21:22:39] <YuviPanda>	 maybe I should just play whack a mole
[21:22:55] <YuviPanda>	 like, fixing the apt-get issue would fix a good chunk of labs spam
[21:23:15] <_joe_>	 YuviPanda: or, you check that apt-get update works on the shinken server, and establish another service dependency
[21:23:26] <_joe_>	 :)
[21:23:53] <YuviPanda>	 ori: fwiw, I already have a working IRC relay via JSON data packaged and puppetized on labs (ircyall) that can easily also write events elsewhere (ES/LS, wherever)
[21:24:15] <YuviPanda>	 ori: will probably move shinken to it at some point, and perhaps over time logmsgbot or whatever. 
[21:24:31] <YuviPanda>	 _joe_: heh, but it's *transient*, so it works one time and doesn't the next run and vice versa.
[21:24:45] <YuviPanda>	 like, all of these recover the second run.
[21:25:14] <YuviPanda>	 hmmm, I *could* count 'failed-to-restart' events!
[21:25:22] <YuviPanda>	 and setup a *separate* more stringient alert for those
[21:25:32] <YuviPanda>	 and just make regular puppet failures alert only when they fail twice in a row
[21:28:03] <grrrit-wm>	 (03PS1) 10Legoktm: Set $wgCentralAuthPreventUnattached = true on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177660 
[21:29:03] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] Disable gadgets caching [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177600 (owner: 10MaxSem)
[21:31:37] <ori>	 bblack: would it be OK with you to merge https://gerrit.wikimedia.org/r/#/c/177634/ ? it changes bits vcl from "if (req.url ~ "^/(event\.gif|statsv/)") { ..." to "if (req.url ~ "^/(event\.gif|statsv)") { ..." (no trailing slash). and no, we don't have bits.wikimedia.org/statsverydifferentthing :)
[21:33:50] <ottomata>	 ori, maybe: ^/(event\.gif|statsv$|statsv/)
[21:33:51] <ottomata>	 ?
[21:34:12] <grrrit-wm>	 (03PS1) 10Ottomata: Move Kafka MessagesIn anomaly alert to role::graphite::production [puppet] - 10https://gerrit.wikimedia.org/r/177662 
[21:34:22] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Move Kafka MessagesIn anomaly alert to role::graphite::production [puppet] - 10https://gerrit.wikimedia.org/r/177662 (owner: 10Ottomata)
[21:34:39] <ori>	 ottomata: it'd have to be [/?]
[21:34:49] <ori>	 varnish considers the query string part of the URL
[21:34:57] <ori>	 actually i'm not sure of that
[21:35:18] <gwicke>	 YuviPanda: are we going to stay with txstatsd for the foreseeable future?
[21:35:22] <grrrit-wm>	 (03PS1) 10Ori.livneh: hhvm: Make Ganglia monitoring scripts resilient to transient failures [puppet] - 10https://gerrit.wikimedia.org/r/177663 
[21:35:24] <grrrit-wm>	 (03PS2) 10Awight: Change banner hide cookie duration to one week [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177278 (owner: 10Ejegg)
[21:35:29] <grrrit-wm>	 (03CR) 10Awight: [C: 032] Change banner hide cookie duration to one week [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177278 (owner: 10Ejegg)
[21:35:31] <YuviPanda>	 gwicke: godog would know, I think :)
[21:35:39] <ori>	 yes, it does
[21:35:41] <grrrit-wm>	 (03Merged) 10jenkins-bot: Change banner hide cookie duration to one week [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177278 (owner: 10Ejegg)
[21:35:54] <gwicke>	 YuviPanda: k, thx; was wondering about set support
[21:36:13] <YuviPanda>	 gwicke: I recently saw a deb repo created in gerrit for the node version.
[21:36:14] <ori>	 awight: is google aware of the don't-show-banners-to-google thing? don't they penalize you for showing different content to googlebot?
[21:36:17] <YuviPanda>	 gwicke: but that's as much as I know
[21:36:28] <grrrit-wm>	 (03PS2) 10Ottomata: Move Kafka MessagesIn anomaly alert to role::graphite::production [puppet] - 10https://gerrit.wikimedia.org/r/177662 
[21:36:38] <gwicke>	 YuviPanda: ahh, that's a strong hint ;)
[21:36:38] <logmsgbot>	 !log awight Synchronized wmf-config: Shortening CentralNotice close box to 1 week (duration: 00m 07s)
[21:36:42] <morebots>	 Logged the message, Master
[21:36:50] <YuviPanda>	 gwicke: not necessarily! :)
[21:37:01] <awight>	 ori: a few upstairs people are calling friends at Google about it...
[21:37:09] <ottomata>	 ah yeah, 
[21:37:30] <awight>	 ori: but yeah it's nerve-wracking cos none of our so-called fixes are officially supported.
[21:37:56] <awight>	 ori: oh, now I see what you're saying--the bot test might be a problem.
[21:38:03] <ottomata>	 ^/(event\.gif|statsv[\?\/])
[21:38:04] <ottomata>	 ?
[21:38:10] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] hhvm: Make Ganglia monitoring scripts resilient to transient failures [puppet] - 10https://gerrit.wikimedia.org/r/177663 (owner: 10Ori.livneh)
[21:38:21] <ori>	 ottomata: yeah
[21:38:26] <ori>	 [/\?]
[21:38:31] <grrrit-wm>	 (03PS3) 10Ottomata: Move Kafka MessagesIn anomaly alert to role::graphite::production [puppet] - 10https://gerrit.wikimedia.org/r/177662 
[21:38:40] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Move Kafka MessagesIn anomaly alert to role::graphite::production [puppet] - 10https://gerrit.wikimedia.org/r/177662 (owner: 10Ottomata)
[21:38:53] <ori>	 awight: yeah
[21:39:19] <ottomata>	 oh / doesn't need \
[21:39:22] <ottomata>	 ja
[21:39:31] <ori>	 awight: it's a common SEO tactic to hide promotional content from googlebot so your site's content-to-ads ratio looks better
[21:41:12] <awight>	 yikes
[21:43:19] <AndyRussG>	 Hummm. Which is essentially what we're trying to do
[21:43:27] <grrrit-wm>	 (03PS2) 10Ori.livneh: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 
[21:43:46] <ori>	 AndyRussG, awight: https://support.google.com/webmasters/answer/66355?hl=en
[21:44:01] <ori>	 "Cloaking refers to the practice of presenting different content or URLs to human users and search engines. Cloaking is considered a violation of Google’s Webmaster Guidelines because it provides our users with different results than they expected."
[21:45:15] <ori>	 awight, K4-713 ^ can you acknowledge awareness of that?
[21:45:38] <AndyRussG>	 ori: on it
[21:45:49] <AndyRussG>	 Sounds to me like rollback time
[21:46:09] <ori>	 be sure to coordinate, because it sounds like we have multiple fronts here
[21:47:56] <K4-713>	 ori: Definitely many conversations happening at the same time right now. As we're still waiting to hear back from their people, I don't think we want to roll anything back before we actually talk to them. 
[21:48:22] <ori>	 K4-713: *nod* I didn't suggest a roll-back; I just wanted an ACK, which you just gave. Thanks :)
[21:49:46] <K4-713>	 ori: Thank you for bringing that up. I'll make sure it's a question we add to the stack. 
[21:54:04] <grrrit-wm>	 (03PS1) 10MaxSem: search-redirect: check for parameter existence consistently [mediawiki-config] - 10https://gerrit.wikimedia.org/r/177665 
[21:54:24] <bblack>	 ori: shouldn't it be [?/#] at the end there, if you're looking for all end-of-url-path? (also, I don't think ? needs escaping within [])
[21:55:28] <ori>	 bblack: i think you're right, but i followed existing usage: text-frontend.inc.vcl.erb:48:		&& req.url ~ "^/(wiki|(gan|ike|iu|kk|ku|shi|sr|tg|uz|zh)(-[a-z]+)?)[/\?]") {
[21:55:33] <bblack>	 although, without a $ at the end, I don't see what difference this is making at all
[21:55:47] <ori>	 bblack: it excludes /statsverydifferent
[21:55:57] <bblack>	 oh right
[21:56:00] <Nemo_bis>	 ori: maybe paste your comments above on https://phabricator.wikimedia.org/T76743
[21:56:25] <ori>	 Nemo_bis: could you? doing eight different things
[21:56:49] <ori>	 Nemo_bis: by the way, is it actually true that people have stopped you on the street in Milan to ask you about the banners? If you weren't kidding, I think that's awesome and touching and hilarious.
[21:58:03] <ori>	 bblack: clients are not supposed to send fragment identifiers (#...) to the server
[21:58:06] <Nemo_bis>	 People who had ways to know about my volunteer projects of course
[21:58:22] <ori>	 Nemo_bis: yes of course I didn't think that random strangers just ran up to you :)
[21:58:24] <Nemo_bis>	 Milan sometimes looks smaller than it is
[21:58:35] <_joe_>	 Milan is small
[21:58:43] <bblack>	 oh really? I never in my life of parsing http-ish things even considered that or noticed it, but I guess it makes sense :)
[21:59:14] <Nemo_bis>	 _joe_: and sometimes feels even smaller :)
[22:00:54] <grrrit-wm>	 (03PS3) 10Ori.livneh: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 
[22:01:13] <YuviPanda>	 !log nodejs sucks.
[22:01:16] <grrrit-wm>	 (03PS4) 10Ori.livneh: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 
[22:01:17] <morebots>	 Logged the message, Master
[22:01:29] <grrrit-wm>	 (03CR) 10Ori.livneh: "Removed redundant escape" [puppet] - 10https://gerrit.wikimedia.org/r/177634 (owner: 10Ori.livneh)
[22:01:58] <_joe_>	 YuviPanda: you know logs go to twitter, right? :P
[22:02:02] <bblack>	 ori: there one in the other file too :)
[22:02:06] <YuviPanda>	 oh, damn.
[22:02:16] <YuviPanda>	 _joe_: oh well :)
[22:02:39] <ori>	 bblack: heh.
[22:02:43] <^d>	 _joe_: I don't !log unless it's worth putting on twitter anyway :p
[22:02:45] <chasemp>	 just made my day YuviPanda
[22:02:51] <grrrit-wm>	 (03PS5) 10Ori.livneh: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 
[22:03:02] <bblack>	 if only that regex was a piece of hiera data, it wouldn't be redundant :)
[22:03:23] <_joe_>	 https://twitter.com/wikimediatech/status/540626921210798080
[22:03:31] <YuviPanda>	 chasemp: heh, ran into a (apparently very common) npm install race condition issue in writing and reading files where the solution is 'we think we fixed all of those, just use npm to upgrade npm'
[22:03:35] <bblack>	 well not really
[22:03:54] <ori>	 bblack: careful not to suggest puppet refactoring projects to me or you'll wake up one day with very scary diffs in your gerrit inbox :P
[22:04:05] <awight>	 greg-g: fyi, we're grabbing another hour of deployment, unless anyone else wants to jump in?
[22:04:09] <bblack>	 which I'll blithely ignore!
[22:04:23] <greg-g>	 awight: god speed good sir
[22:04:27] <YuviPanda>	 yeah, the b in bblack stands for 'blithely'
[22:04:36] <ori>	 YuviPanda: WHICH ONE
[22:04:41] <bblack>	 both of them!
[22:04:47] <YuviPanda>	 ALL OF THEM
[22:04:51] <ori>	 so one is redundant?
[22:05:06] <greg-g>	 good luck figuring out which one's the redundant one
[22:05:13] <^d>	 Sheesh guys, character duplication.
[22:05:13] <ori>	 greg-g: it's the 'b'
[22:05:17] <bblack>	 no, it just means I'm very blithe about being blithely about things
[22:05:33] <YuviPanda>	 well, blithely blithely lacak
[22:05:35] <YuviPanda>	 *lack
[22:05:43] <YuviPanda>	 so clearly you blithely lack blith?
[22:05:55] <bblack>	 sometimes!
[22:06:16] <YuviPanda>	 clearly not blithely enough
[22:06:40] <greg-g>	 "Error: Could not retrieve catalog from remote server: Error 400 on SERVER: keys():  Requires hash to work with at /etc/puppet/manifests/role/cache.pp:388 on node  i-0000007f.eqiad.wmflabs"
[22:06:47] <bblack>	 well if I were less blithe I'd bother to explain when sometimes is
[22:06:52] <ori>	 yes, ottomata ^^^ i just saw that.
[22:06:59] <ori>	 looks like your change?
[22:08:46] <YuviPanda>	 hmmm, I wonder if someone will start beating me up if I start sending binary true/false values to graphite
[22:08:54] <YuviPanda>	 like, 'puppet AST failure' 0/1
[22:09:07] <ori>	 ottomata: $cluster_config in kafka.pp has 'main' and 'external' rather than 'eqiad'
[22:10:26] <greg-g>	 ori: re the keys error, could it be due to:
[22:10:27] <greg-g>	 $kafka_brokers = keys($role::analytics::kafka::config::cluster_config['eqiad'])
[22:10:30] <greg-g>	 in https://github.com/wikimedia/operations-puppet/commit/b27fb0d40f681615d9899f8de12aecfd76ac696e
[22:10:41] <greg-g>	 (that's from bd808, I'm just relaying/cross pollinating)
[22:10:59] <greg-g>	 rephrase: the keys error in beta cluster
[22:11:06] <ori>	 yes
[22:11:09] <ori>	 hence
[22:11:10] <ori>	 14:09 <ori> ottomata: $cluster_config in kafka.pp has 'main' and 'external' rather than 'eqiad'
[22:11:12] <ori>	 ;)
[22:11:31] <greg-g>	 oh, I read that backwards somehow
[22:12:22] <grrrit-wm>	 (03PS6) 10BBlack: stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 (owner: 10Ori.livneh)
[22:12:31] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 (owner: 10Ori.livneh)
[22:12:49] <grrrit-wm>	 (03CR) 10BBlack: [V: 032] stats bits endpoint: don't require a trailing slash [puppet] - 10https://gerrit.wikimedia.org/r/177634 (owner: 10Ori.livneh)
[22:13:02] <ori>	 bblack: hahahaha. i'm not sure if the double +2 was intentional but it worked beautifully. thanks :)
[22:13:23] <bblack>	 C/V, I didn't want to wait on jenkins again :)
[22:14:25] <grrrit-wm>	 (03PS1) 10Ori.livneh: Kafka $cluster_config on labs: replace 'main' key with 'eqiad' [puppet] - 10https://gerrit.wikimedia.org/r/177670 
[22:14:31] <bblack>	 people seem to have a habit of SMSing me within ~30s of puppet-merge.  I always think it's an icinga alert I just triggered :p
[22:14:31] <ori>	 ottomata, greg-g ^
[22:15:08] <ori>	 i'll cherry-pick it on labs
[22:15:13] <ori>	 since that's where it counts, really
[22:16:17] <ottomata>	 reading...
[22:16:24] <greg-g>	 ori: thanks
[22:16:43] <greg-g>	 I just reported https://phabricator.wikimedia.org/T76799 because, yeah
[22:17:30] <ottomata>	 eh?
[22:17:34] <ottomata>	 confused
[22:18:18] <ottomata>	 ohhhh
[22:18:21] <ottomata>	 hm
[22:18:22] <ori>	 $kafka_brokers = keys($role::analytics::kafka::config::cluster_config['eqiad'])
[22:18:26] <ottomata>	 i see
[22:18:27] <grrrit-wm>	 (03PS2) 10Greg Grossmeier: Kafka $cluster_config on labs: replace 'main' key with 'eqiad' [puppet] - 10https://gerrit.wikimedia.org/r/177670 (owner: 10Ori.livneh)
[22:18:37] <ottomata>	 needs  a stupid $realm wrapper too?
[22:18:50] <ottomata>	 ehhhh
[22:18:53] <ori>	 well, why is it 'main' on labs?
[22:18:54] <ottomata>	 that is a very hacky fix.
[22:18:57] <ottomata>	 hm
[22:19:07] <bblack>	 needs hiera!
[22:19:08] <bd808>	 *cough* hiera *cough*
[22:19:11] <ottomata>	 indeed!
[22:19:15] <ori>	 ottomata: i have no affection for it, feel free to abandon
[22:19:24] <ori>	 moar indirection!
[22:19:26] <ottomata>	 lemme see...
[22:19:37] <bd808>	 separate code from configuration
[22:19:51] <ottomata>	 shhh this is pre-heira days
[22:20:14] <ori>	 speaking of separating code from configuration
[22:20:18] <ori>	 once this is fixed, puppet will still fail
[22:20:20] <ori>	 because:
[22:20:24] <ori>	 Error: Failed to apply catalog: Could not find dependency Package[ganglia-monitor] for File[/usr/lib/ganglia/python_modules/varnishkafka.py] at /etc/puppet/modules/varnishkafka/manifests/monitor.pp:22 
[22:20:36] <ori>	 and that's because ganglia.pp has:
[22:20:39] <ori>	     # FIXME: remove after the ganglia module migration
[22:20:39] <ori>	     if $::realm == 'labs' or ($::hostname in ['netmon1001'] or $::site == 'esams' or $::site == 'codfw') {
[22:20:55] <ori>	 which bypasses the rest of the module and includes some other module instead
[22:20:59] <ori>	 which doesn't declare the package, i guess
[22:21:00] <ottomata>	 this si $::realm == 'beta'?
[22:21:05] <ori>	 labs
[22:21:07] <ottomata>	 hm
[22:21:37] <bblack>	 beta would make much more sense
[22:21:53] <ottomata>	 ah, ori, i'm going to wrap the varnish::kafka::statsv include ina  realm conditional
[22:21:57] <ottomata>	 liek the other varnishkafka class includes
[22:22:09] <bd808>	 $::realm=='labs' is for all of labs. We don't have a realm for beta separately
[22:24:24] <grrrit-wm>	 (03PS3) 10Ottomata: Move include of role::cache::varnish::kafka::statsv into prodution realm conditional [puppet] - 10https://gerrit.wikimedia.org/r/177670 (owner: 10Ori.livneh)
[22:24:56] <ottomata>	 ori ^
[22:25:09] <cmjohnson>	 ottomata: fyi your disk shipped 
[22:25:12] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] "grumble grumble" [puppet] - 10https://gerrit.wikimedia.org/r/173758 (owner: 10Ori.livneh)
[22:25:13] <ottomata>	 cool!
[22:25:39] <hierapanda>	 ori: ty
[22:25:40] <grrrit-wm>	 (03PS6) 10Yuvipanda: eventlogging: couple less tightly to ganglia [puppet] - 10https://gerrit.wikimedia.org/r/173758 (owner: 10Ori.livneh)
[22:26:06] <grrrit-wm>	 (03PS4) 10Ottomata: Move include of role::cache::varnish::kafka::statsv into prodution realm conditional [puppet] - 10https://gerrit.wikimedia.org/r/177670 (owner: 10Ori.livneh)
[22:26:15] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Move include of role::cache::varnish::kafka::statsv into prodution realm conditional [puppet] - 10https://gerrit.wikimedia.org/r/177670 (owner: 10Ori.livneh)
[22:26:33] <ottomata>	 greg-g: try now.
[22:26:44] <ottomata>	 or bd808
[22:27:17] <bd808>	 ottomata: I'll force a run
[22:27:33] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] eventlogging: couple less tightly to ganglia [puppet] - 10https://gerrit.wikimedia.org/r/173758 (owner: 10Ori.livneh)
[22:28:08] <bd808>	 ottomata: ori wins a prize -- Error: Failed to apply catalog: Could not find dependency Package[ganglia-monitor] for File[/usr/lib/ganglia/python_modules/varnishkafka.py] at /etc/puppet/modules/varnishkafka/manifests/monitor.pp:22
[22:28:31] <ori>	 *cough* hiera *cough*
[22:29:05] <ori>	 ganglia is *really* slow
[22:29:10] <ori>	 shouldn't it be faster, since it's on better hardware?
[22:29:29] <hierapanda>	 bd808: rebasing deployment-salt puppet repo manually to drop older version of an EL patch that just got merged. are you doing things to it?
[22:29:39] <bd808>	 hierapanda: nope
[22:29:42] <CloudPanda>	 bd808: cool
[22:29:52] <MaxSem>	 sometimes, I start to think that things would be so much more transparent and manageable if we just replaced puppet with a bunch of shell scripts... :P
[22:30:02] <bblack>	 you'd be surprised how often hardware is very secondary to efficient design.  I think ganglia's one of those cases
[22:30:02] <bd808>	 git rebase --interactive is so fun
[22:30:42] <CloudPanda>	 hmm, 72a36d6a4f3bb883472af5e5f3beac49b37a95a9 fails to apply (ori's tiny little patch to varnish)
[22:30:44] <bblack>	 git rebase -i is probably my most-frequently-typed command on my dev box.  I couldn't function without it
[22:31:35] <icinga-wm>	 PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: puppet fail  
[22:31:49] <CloudPanda>	 bblack: heh, I do reflog with an equal frequencey.
[22:31:53] <CloudPanda>	 but my favorite is still commit -p
[22:32:12] <ori>	 oh, looking
[22:32:25] <bblack>	 well yeah, reflog saves my bacon a lot too.  usually just to extract something or other from a "lost" previous version of a commit
[22:32:57] <bblack>	 but getting used to that gives you a lot of freedom in aggressive refactoring locally, since you have reflog as a fallback option.
[22:32:57] <ori>	 CloudPanda: where did it fail?
[22:33:06] <ori>	 i looked at cp1056 and it did the right thing
[22:33:17] <ottomata>	 wait, but, who cares, we aren't even including varnishkaffka anymore
[22:33:20] <ottomata>	 unless in production
[22:33:23] <ottomata>	 so why would that matter?
[22:33:24] <ottomata>	 ori?
[22:33:47] <CloudPanda>	 ori: oh, it just failed rebase.
[22:33:54] <ori>	 dunno off the top of my head, probably some crazy chain of includes
[22:34:00] <ori>	 (@ottomata)
[22:34:01] <CloudPanda>	 is  +      if (req.url ~ "^/(event\.gif|statsv[/?])") { the correct version?
[22:34:06] <CloudPanda>	 or +       if (req.url ~ "^/(event\.gif|statsv[/\?])") {
[22:34:07] <CloudPanda>	 ?
[22:34:07] <bblack>	 yes
[22:34:13] <bblack>	 the first
[22:34:15] <CloudPanda>	 ah, ok
[22:35:23] <icinga-wm>	 PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail  
[22:35:51] <logmsgbot>	 !log awight Synchronized php-1.25wmf10/extensions/CentralNotice: push CentralNotice features (duration: 00m 08s)
[22:35:53] <CloudPanda>	 hmm.
[22:35:54] <morebots>	 Logged the message, Master
[22:36:08] <logmsgbot>	 !log awight Synchronized php-1.25wmf11/extensions/CentralNotice: push CentralNotice features (duration: 00m 06s)
[22:36:10] <morebots>	 Logged the message, Master
[22:36:21] <ori>	 CloudPanda: hafnium is your EL change
[22:36:40] <CloudPanda>	 was there an alert?
[22:36:41] * CloudPanda looks
[22:36:55] <YuviPanda>	 oh, right.
[22:38:58] <ori>	 ottomata: I think you had a snippet of code you used to test the statsv endpoint. Do you still have it?
[22:42:15] <YuviPanda>	 ori: was a transient failure, I think. Running puppet again makes it happy
[22:42:39] <ottomata>	 ori
[22:42:43] <ottomata>	 ctrl-r to the rescue
[22:42:44] <ottomata>	 while true; do curl --head -H 'Host: bits.wikimedia.org'  "http://localhost:80/statsv/?hi=there&ts=$(date +%s)"; sleep 1; done
[22:43:02] <ottomata>	 and if you want to test consuming
[22:43:05] <ottomata>	 log into a kafka broker
[22:43:07] <ottomata>	 and do
[22:43:17] <ottomata>	 kafka console-consumer --topic statsv
[22:43:37] <ori>	 aha! it was the latter bit that i was looking for
[22:44:51] <icinga-wm>	 RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[22:45:49] <ori>	 ottomata: which endpoint should the real consumer connect to? should zookeeper be used?
[22:47:02] <icinga-wm>	 RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[22:50:48] <ottomata>	 eh?
[22:50:50] <ottomata>	 ummm
[22:51:05] <ottomata>	 i am not sure, ori, i think it doesn't ahve to.
[22:51:15] <ottomata>	 i am not very familiar with consumer api
[22:51:22] <ottomata>	 are you writing a consumer?
[22:55:10] <ottomata>	 laters all!
[22:55:40] <icinga-wm>	 PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet).  
[22:56:37] <YuviPanda>	 hmm
[22:56:42] <YuviPanda>	 that was mine, but I merged it in palladium
[22:57:04] <icinga-wm>	 PROBLEM - puppet last run on hafnium is CRITICAL: CRITICAL: puppet fail  
[22:57:10] <YuviPanda>	 wat
[22:58:51] <icinga-wm>	 RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge.  
[23:19:01] <grrrit-wm>	 (03PS1) 10Yuvipanda: eventlogging: Explicitly include monitoring class when we require it [puppet] - 10https://gerrit.wikimedia.org/r/177687 
[23:19:50] <grrrit-wm>	 (03PS2) 10Yuvipanda: eventlogging: Explicitly include monitoring class when we require it [puppet] - 10https://gerrit.wikimedia.org/r/177687 
[23:19:52] <YuviPanda>	 ori: heh, it wasn't transient, but somehow a single puppet run succeeded. ^ should fix, CR?
[23:20:12] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Add a partman recipe for the new HP virt servers. [puppet] - 10https://gerrit.wikimedia.org/r/177688 
[23:20:38] <YuviPanda>	 ori: hmm, on second thought, I'm somewhat surprised, since this is prod and monitoring *should* be included anyway...
[23:25:15] <grrrit-wm>	 (03PS2) 10Andrew Bogott: Add a partman recipe for the new HP virt servers. [puppet] - 10https://gerrit.wikimedia.org/r/177688 
[23:28:28] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] eventlogging: Explicitly include monitoring class when we require it [puppet] - 10https://gerrit.wikimedia.org/r/177687 (owner: 10Yuvipanda)
[23:32:21] <icinga-wm>	 RECOVERY - puppet last run on hafnium is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures  
[23:32:45] <grrrit-wm>	 (03PS1) 10Yuvipanda: beta: Add debug=true for shinken monitoring check [puppet] - 10https://gerrit.wikimedia.org/r/177692 
[23:36:37] <grrrit-wm>	 (03PS2) 10Yuvipanda: beta: Add debug=true for shinken monitoring check [puppet] - 10https://gerrit.wikimedia.org/r/177692 
[23:39:07] <grrrit-wm>	 (03PS3) 10Andrew Bogott: Add a partman recipe for the new HP virt servers. [puppet] - 10https://gerrit.wikimedia.org/r/177688 
[23:40:50] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Add a partman recipe for the new HP virt servers. [puppet] - 10https://gerrit.wikimedia.org/r/177688 (owner: 10Andrew Bogott)
[23:40:55] <icinga-wm>	 PROBLEM - Kafka Broker Messages In Per Second on tungsten is CRITICAL: CRITICAL: Anomaly detected: 0 data above and 45 below the confidence bounds  
[23:41:02] <icinga-wm>	 PROBLEM - Kafka Broker Messages In Per Second on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 0 data above and 45 below the confidence bounds  
[23:41:47] <jgage>	 what does "0 data above and 45 below" mean?
[23:42:47] <^d>	 jgage: Soon as you find out let me know so I can decipher my own Cirrus slow query logs :)
[23:42:52] <jgage>	 heh ok
[23:43:09] <jgage>	 it seems a value is out of bounds, but it's unclear to me what the value or bounds are
[23:43:25] <^d>	 Maybe it means you have 45 data points below the confidence interval?
[23:43:33] <^d>	 $whateverThatIsForYourCheck
[23:44:47] <jgage>	 ah yeah, that deciphers logically. "45 data points out of bounds, 0 within bounds"
[23:45:21] <YuviPanda>	 we really need to make check_graphite messages not suck
[23:51:09] <grrrit-wm>	 (03PS3) 10Yuvipanda: beta: Add debug=true for shinken monitoring check [puppet] - 10https://gerrit.wikimedia.org/r/177692 
[23:54:21] <awight>	 DB heads... I just deployed a patch which issues a SET SESSION query, any thoughts about why that is failing?
[23:54:47] <awight>	 springle: ^