[00:00:16] !log ori Synchronized php-1.24wmf18/extensions/WikimediaEvents: Update WikimediaEvents for cherry-picks (duration: 00m 03s) [00:00:22] Logged the message, Master [00:00:38] PROBLEM - OCG health on ocg1001 is CRITICAL: CRITICAL: /mnt/tmpfs 0B: /srv/deployment/ocg/output 5.5GB (= 5.0GB critical): /srv/deployment/ocg/postmortem 990482B: ocg_job_status 9897 msg: ocg_render_job_queue 0 msg [00:01:11] (03PS1) 10coren: Tool Labs: add php5-xdebug to dev environment [puppet] - 10https://gerrit.wikimedia.org/r/158032 [00:01:14] aww, OCG ? [00:05:18] disks are cheap, why not give it a terabyte /tmp? :P [00:06:21] they are not _that_ cheap if you get them for actual servers in data centers [00:06:30] as opposed to home computers [00:07:51] but i guess.. if it needs it ,it needs it [00:08:28] icinga-wm: that looks resolved already? what? [00:08:50] the_nobodies: yet when checking on ocg1001 it looks already resolved.. and .. well .. "tmp" [00:09:30] (03PS2) 10coren: Tool Labs: add php5-xdebug to dev environment [puppet] - 10https://gerrit.wikimedia.org/r/158032 (https://bugzilla.wikimedia.org/70313) [00:09:34] oh, or it's the output size [00:09:40] 5.6G /srv/deployment/ocg/output/ [00:10:52] dat oits. [00:11:13] (03PS1) 10Legoktm: Install "php5-xdebug" on tool labs [puppet] - 10https://gerrit.wikimedia.org/r/158033 (https://bugzilla.wikimedia.org/70313) [00:11:28] :OO [00:11:31] * legoktm hugs Coren [00:12:16] (03PS9) 10Ori.livneh: Clean up salt::minion [puppet] - 10https://gerrit.wikimedia.org/r/153727 [00:12:25] Coren: does installing it in dev also make it available on the exec nodes? [00:12:51] legoktm: No; it's the other way 'round. [00:13:03] ok, well I need it on the exec nodes [00:13:11] ... I'm not sure I like that idea. [00:13:26] https://phpunit.de/manual/3.7/en/code-coverage-analysis.html requires it [00:13:43] Ah. Ew. [00:14:01] I don't actually need it for debugging, just as a dependency :P [00:14:56] (03PS3) 10coren: Tool Labs: add php5-xdebug to exec environment [puppet] - 10https://gerrit.wikimedia.org/r/158032 (https://bugzilla.wikimedia.org/70313) [00:15:17] (03Abandoned) 10Legoktm: Install "php5-xdebug" on tool labs [puppet] - 10https://gerrit.wikimedia.org/r/158033 (https://bugzilla.wikimedia.org/70313) (owner: 10Legoktm) [00:15:38] legoktm: Dude, I already got a patch. :-) [00:15:49] I wasn't paying attention :P [00:15:58] Clearly. :-P [00:16:02] (03CR) 10Legoktm: [C: 031] "Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/158032 (https://bugzilla.wikimedia.org/70313) (owner: 10coren) [00:16:13] ori: any luck with virt1000? [00:16:37] RECOVERY - OCG health on ocg1001 is OK: OK: /mnt/tmpfs 0B: /srv/deployment/ocg/output 3711649432B: /srv/deployment/ocg/postmortem 990482B: ocg_job_status 9897 msg: ocg_render_job_queue 0 msg [00:16:52] Reedy: i didn't get around to looking; will do now. just a sec [00:17:00] np, thanks :) [00:17:50] Reedy: it's the wrong server [00:17:58] eh? [00:18:01] you have it both listening on 11212 and connecting to 11212 [00:18:06] !log deleted PDF files older than 3d and a huge 1G one on ocg1001 in reaction to monitoring complaints [00:18:13] Logged the message, Master [00:18:21] why do we monitor that being over 5G? [00:18:26] since we have > 450G of space [00:18:40] i guess first there was monitoring, then new disk? [00:18:43] ori: eh? :/ [00:18:53] listen => '127.0.0.1:11212', [00:18:57] '127.0.0.1:11211:1', [00:19:04] 'servers' => array( '127.0.0.1:11212' ), [00:19:32] (03PS1) 10Ori.livneh: wikitech: specify correct port for memcached instance [puppet] - 10https://gerrit.wikimedia.org/r/158034 [00:19:37] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [00:19:37] ^ Reedy [00:20:26] https://github.com/wikimedia/operations-puppet/blob/production/manifests/role/mediawiki.pp#L13 [00:20:27] and Coren, if you can +1 [00:20:34] It's 11211 on the others [00:20:49] well, andrewbogott_afk doesn't want to change it [00:21:19] which is fine [00:21:37] the following will be true for all app servers, wikitech included: [00:21:39] andrewbogott_afk: is the clock on your laptop/pc out by a few days? :/ [00:21:53] * mediawiki connects to memcached on port 11212 [00:22:05] * nutcracker runs locally and listens on port 11212 [00:22:08] (03CR) 10Reedy: [C: 031] "Per https://gerrit.wikimedia.org/r/#/c/158001/1/wmf-config/wikitech.php,unified" [puppet] - 10https://gerrit.wikimedia.org/r/158034 (owner: 10Ori.livneh) [00:22:34] on wikitech, nutcracker will proxy to localhost:11000 (the local memcached instance) [00:22:35] (03CR) 10coren: [C: 032] "The number, she is better that way." [puppet] - 10https://gerrit.wikimedia.org/r/158034 (owner: 10Ori.livneh) [00:22:42] right [00:22:43] yadda yadda yadda, thanks :) [00:23:00] (03CR) 10coren: [C: 032] Tool Labs: add php5-xdebug to exec environment [puppet] - 10https://gerrit.wikimedia.org/r/158032 (https://bugzilla.wikimedia.org/70313) (owner: 10coren) [00:23:07] note puppet is disabled on virt1000 atm [00:23:11] so might want to fix it manually ;) [00:23:16] sure, easy to do [00:23:54] Reedy: try now [00:24:29] A database query error has occurred. This may indicate a bug in the software. [00:24:30] Function: AccountAudit::updateLastLogin [00:24:30] Error: 1146 Table 'labswiki.accountaudit_login' doesn't exist (208.80.154.18) [00:24:30] hahaha [00:24:38] more fscking extensions [00:25:09] disable the extension, or run update.php? [00:25:40] it's on the same version as wikitech, so at worst, update.php is only going to add any needed extension tables [00:25:53] update.php IMHO [00:25:59] mind running it please? [00:26:10] but yeah, login works now. CC andrewbogott_afk :D [00:26:12] thanks [00:26:41] (03PS2) 10Reedy: Merge virt1000 apache config back into wikitech apache config [puppet] - 10https://gerrit.wikimedia.org/r/157853 [00:26:44] Reedy: how should i run it? the mwscript wrapper isn't present afaict [00:27:00] php multiversion/MWMultiVersion.php update.php --wiki=labswiki [00:27:01] I think [00:27:15] oh, no [00:27:18] MWScript.php [00:27:31] php multiversion/MWScript.php update.php --wiki=labswiki [00:27:43] obviously from /usr/local/apache/common [00:29:20] are you taking responsibility for this? i'm not sure who needs to sign off or whatever [00:29:46] let's get another opinion from Coren maybe? [00:29:53] Hmm? [00:30:18] Error: 1146 Table 'labswiki.accountaudit_login' doesn't exist (208.80.154.18) [00:30:21] disable the extension, or run update.php? [00:30:29] it's on the same version as wikitech, so at worst, update.php is only going to add any needed extension tables [00:30:43] (i agree, fwiw) [00:30:58] Should be, though tha table doesn't ring a bell. [00:31:10] it records the last login time [00:31:22] it was installed a while back as part of the SUL switch over [00:31:34] so that very inactive accounts could be identified [00:31:58] so we probably don't need it on wikitechwiki, do we? [00:32:17] * jamesofur shrugs [00:32:22] possibly not [00:32:22] update.php sounds safer, io [00:32:23] imo [00:32:38] though I've got a feeling we might end up in circle disabling these extensions that cause no harm being there [00:32:41] it's fairly light weight, it likely doesn't hurt much [00:33:05] (03PS1) 10Reedy: Merge virt1000.wikimedia.org back into wikitech.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158035 [00:33:33] ok i'll run it in 3 mins if nobody speaks up against the idea [00:34:05] * ori is surprised that update.php doesn't have a --dry-run option [00:34:43] heh [00:34:47] might be worth BZ-ing that one [00:35:00] if you BZ, i'll run :P [00:36:10] ah, dupe [00:36:11] https://bugzilla.wikimedia.org/show_bug.cgi?id=59857 [00:37:23] ori: ah [00:37:33] (03PS1) 10Dzahn: OCG-raise monitor threshold for output file size [puppet] - 10https://gerrit.wikimedia.org/r/158036 [00:37:34] --schema [00:37:42] "There is a --schema option, to this file is only written, when there actually changes, so having a empty file means no changes." [00:37:43] Reedy: not the same, really [00:37:56] gives you an idea at least [00:38:08] oh, that's --quiet [00:38:10] well, you really want to know what *would* change when there are changes to make [00:38:11] sort of [00:39:02] SAL doesnt have CSS anymore? [00:39:19] wikitech works but no skin? i mean [00:39:36] Reedy: known issue related to the ongoing thing? [00:40:56] ooh [00:41:04] that was working fine not so long ago... [00:41:40] I see the css/js returning but it's not applied? [00:42:39] Resource interpreted as Document but transferred with MIME type text/javascript: "https://wikitech.wikimedia.org/w/load.php?debug=false&lang=en&modules=startup&only=scripts&skin=vector&*". [00:43:46] (03CR) 10Krinkle: "fixme: Tabs instead of spaces." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157993 (owner: 10Reedy) [00:44:55] (03CR) 10Dzahn: [C: 032] "00:18 mutante: deleted PDF files older than 3d and a huge 1G one on ocg1001 in reaction to monitoring complaints" [puppet] - 10https://gerrit.wikimedia.org/r/158036 (owner: 10Dzahn) [00:45:41] (03CR) 10Dzahn: "/dev/md2 456G 3.8G 452G 1% /srv" [puppet] - 10https://gerrit.wikimedia.org/r/158036 (owner: 10Dzahn) [00:45:52] bd808: That MIME is false positive. Only happens when the chrome dev tools are open. Harmless either way (the js request is fine, css is missing) [00:45:54] been in chrome for years. [00:45:56] https://wikitech.wikimedia.org/w/load.php?debug=false&lang=en&modules=site&only=styles&skin=vector&* [00:46:00] that looks garbled [00:46:02] mysql chartype? [00:46:52] Related to running update.php maybe? [00:47:11] I think it might be memcached [00:47:21] if virt1000 and wikitech are using the same keys [00:47:27] but different access libraries [00:47:30] No [00:47:34] it's not just css [00:47:45] oh nvm, it is just css [00:47:55] I thought it was Vector missing, but our base styles just lok better these days [00:48:18] Reedy: One with compression and one without or something? [00:48:27] https://wikitech.wikimedia.org/w/load.php?debug=false&lang=en&modules=ext.echo.badge%7Cext.gadget.enwp-boxes%7Cext.visualEditor.viewPageTarget.noscript%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.interface%7Cmediawiki.ui.button%7Cskins.vector.styles&only=styles&skin=vector&* [00:48:31] That's the skin css request [00:48:34] linked from html [00:48:38] also contains garbled characters [00:48:47] ori: Does that ring any bells? mixed memcached client libs causing corruption? [00:49:01] so it's not just for modules with contents from wiki pages (mysql), but indeed memcached, php or apache in general [00:49:24] k, leaving for real this time [00:50:09] https://wikitech.wikimedia.org/wiki/Main_Page?debug=true looks right [00:50:27] without debug not so much [00:51:02] https://noc.wikimedia.org/conf/highlight.php?file=mc.php [00:51:18] versus the stuff using our php memcached client etc [00:54:26] yes, i've seen that before [00:54:27] we could switch the normal wikitech over to using the newer memcached config via nutcracker etc [00:54:37] why is it not using the same memcached driver? [00:54:37] How do we get the right things back into cache? Touch all the resource files in the "real" file tree for wikitech? [00:54:37] but that's more fiddling around [00:55:20] (03CR) 10Dzahn: [C: 031] "thanks, this came up because i recently wanted to add the same thing. then ended up wondering why we _don't_ set those. misc hostgroups in" [puppet] - 10https://gerrit.wikimedia.org/r/157658 (owner: 10Alexandros Kosiaris) [00:55:28] ori: wikitech is using the old one... to save even more config variation on virt1000 (ie going forward), I updated it to using the newer config etc [00:58:14] if we switch back to CACHE_MEMCACHED on the various $wg, and then set $wgMemCachedServers = array( '127.0.0.1:11000' ); ... [01:01:00] (03PS1) 10Reedy: Use old style memcached access on wikitech to stop cache pollution [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158038 [01:01:03] ala that [01:04:30] why not newer style on old wikitech? [01:07:13] i suggested that originally ;) [01:07:23] just I can't do it, so someone else would have to [01:16:21] Error: 1146 Table 'labswiki.betafeatures_user_counts' doesn't exist (208.80.154.18) [01:16:23] another one ;) [01:19:11] andrewbogott_afk: Coren load on labmon seems ok [01:19:26] I see 3 - 5% waiting on IO now with everything sending them stats [01:19:29] which is fine [01:19:42] Reedy: ^d: http://www.bullshit.wiki/api.php :) [01:20:19] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00332225913621 [01:21:07] (03PS1) 10Rush: iridium exim roled for phab [puppet] - 10https://gerrit.wikimedia.org/r/158039 [01:25:17] (03PS1) 10Rush: phab::mail renamed to phab::mailrelay [puppet] - 10https://gerrit.wikimedia.org/r/158040 [01:25:23] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [01:25:46] (03CR) 10Rush: [C: 032] "already there, was removed erroneously previously" [puppet] - 10https://gerrit.wikimedia.org/r/158039 (owner: 10Rush) [01:26:13] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [01:26:30] (03CR) 10Rush: [C: 032] "mail is a terrible name as this just sets up a relay for ticket interaction not actual mail" [puppet] - 10https://gerrit.wikimedia.org/r/158040 (owner: 10Rush) [01:32:41] I'm done for today [01:32:53] (03CR) 10Dzahn: "why "certainly not via templates"? it's a config file, don't we usually use puppet templates for config files?" [puppet] - 10https://gerrit.wikimedia.org/r/157294 (https://bugzilla.wikimedia.org/69979) (owner: 10Dzahn) [01:38:18] (03CR) 10Dzahn: "i see manifests/init.pp: core_dump_report_directory => '/var/log/hhvm'," [puppet] - 10https://gerrit.wikimedia.org/r/157294 (https://bugzilla.wikimedia.org/69979) (owner: 10Dzahn) [01:38:23] (03Abandoned) 10Dzahn: hhvm - make debug path configurable [puppet] - 10https://gerrit.wikimedia.org/r/157294 (https://bugzilla.wikimedia.org/69979) (owner: 10Dzahn) [01:39:53] (03CR) 10Dzahn: "don't know if there is another labs solution, bug doesnt say such thing.." [puppet] - 10https://gerrit.wikimedia.org/r/157294 (https://bugzilla.wikimedia.org/69979) (owner: 10Dzahn) [01:40:32] (03PS1) 10Rush: phabricator email relay to task creation [puppet] - 10https://gerrit.wikimedia.org/r/158042 [01:41:33] (03CR) 10Rush: [C: 032] phabricator email relay to task creation [puppet] - 10https://gerrit.wikimedia.org/r/158042 (owner: 10Rush) [01:50:05] (03PS1) 10Reedy: Add comma between active MW versions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/158043 [02:11:13] PROBLEM - Disk space on virt0 is CRITICAL: DISK CRITICAL - free space: /a 3614 MB (3% inode=99%): [02:16:39] (03CR) 10BBlack: [C: 04-1] "In both cases this stuff should be conditional as labs-only, I think." [puppet] - 10https://gerrit.wikimedia.org/r/158016 (https://bugzilla.wikimedia.org/70181) (owner: 10Dduvall) [02:20:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [02:29:53] RECOVERY - swift eqiad-prod object availability on tungsten is OK: OK: Less than 1.00% under the threshold [95.0] [02:39:56] Is there a bug tracking the wikitech CSS thing? [02:40:25] I think Reedy was working on something to do with that earlier. [02:40:54] But he's probably asleep now. Sorry scfc_de [02:42:39] Yep, that's why I'm interested in something I can subscribe to so that I can sleep as well :-). [02:43:27] !log LocalisationUpdate completed (1.24wmf15) at 2014-09-03 02:42:24+00:00 [02:43:33] Logged the message, Master [03:00:13] RECOVERY - Disk space on virt0 is OK: DISK OK [03:07:53] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [03:07:53] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [03:09:03] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 1 failures [03:16:14] PROBLEM - puppet last run on mw1142 is CRITICAL: CRITICAL: Puppet has 1 failures [03:17:40] !log LocalisationUpdate completed (1.24wmf18) at 2014-09-03 03:16:37+00:00 [03:17:47] Logged the message, Master [03:24:53] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:25:53] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [03:27:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [03:27:03] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [03:33:59] Krenair: Which CSS thing? [03:34:14] RECOVERY - puppet last run on mw1142 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:34:26] Oh, hmm. [03:34:28] Carmela, read up the chat log [03:34:39] Less exciting than I'd hoped. [03:36:03] PROBLEM - puppet last run on mw1091 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:13] we could throw in some helvetica, make it interesting [03:37:37] Uh, you mean Helvetica Neue. [03:37:46] helvetica neue light [03:38:14] PROBLEM - puppet last run on mw1154 is CRITICAL: CRITICAL: Puppet has 1 failures [03:45:03] PROBLEM - puppet last run on mw1197 is CRITICAL: CRITICAL: Puppet has 1 failures [03:46:23] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 1 failures [03:46:53] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 1 failures [03:51:20] !log LocalisationUpdate completed (1.24wmf19) at 2014-09-03 03:50:17+00:00 [03:51:26] Logged the message, Master [03:51:43] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: Puppet has 1 failures [03:55:03] RECOVERY - puppet last run on mw1091 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [03:56:14] RECOVERY - puppet last run on mw1154 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [04:03:03] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:03:53] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [04:04:23] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:09:43] RECOVERY - puppet last run on mw1079 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [04:21:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [04:54:56] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Sep 3 04:53:50 UTC 2014 (duration 53m 49s) [04:55:02] Logged the message, Master [04:59:13] (03PS2) 10Giuseppe Lavagetto: Add pcre overflow patch that should prevent beta from crashing regularly [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157790 [05:01:55] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] "Made the patch apply cleanly so that the debian build system won't argue about it" [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157790 (owner: 10Giuseppe Lavagetto) [05:21:30] (03PS1) 10Springle: ignore oai.% for s3 sanitarium replication [puppet] - 10https://gerrit.wikimedia.org/r/158053 [05:22:34] (03CR) 10Springle: [C: 032] ignore oai.% for s3 sanitarium replication [puppet] - 10https://gerrit.wikimedia.org/r/158053 (owner: 10Springle) [05:28:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [05:32:03] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [06:22:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [06:28:24] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:24] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Epic puppet fail [06:28:44] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:04] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:23] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:23] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:23] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:33] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:33] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:03] PROBLEM - Disk space on elastic1004 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [06:42:24] PROBLEM - puppet last run on db1007 is CRITICAL: CRITICAL: Puppet has 2 failures [06:43:34] (03CR) 10Alexandros Kosiaris: [C: 032] "After some discussion in IRC with Giuseppe we concluded that previous behavior was old and the new one seems better. So merging this." [puppet] - 10https://gerrit.wikimedia.org/r/157658 (owner: 10Alexandros Kosiaris) [06:45:33] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:46:03] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [06:46:03] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:23] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:24] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:46:56] damn you passenger... [06:47:19] it's punctual [06:47:34] yes... on apache logrotate [06:47:43] despire graceful [06:47:47] despite* [06:48:17] why don't we use http://cronolog.org/ [06:48:25] godog mentioned it recently so i read up about it [06:48:35] Wikitech looks a bit silly at the moment. [06:49:01] Carmela: indeed [06:49:12] it's cache pollution from the migration work [06:49:28] both old (current) and new wikis are using the same memcached infrastructure [06:49:57] And you can't just flush memcached? [06:50:58] we probably could, but if it goes south i don't want to be on the hook [06:51:16] i guess i can tcpdump memcached keys and reset the resourceloader ones [06:52:24] * ori keeps this one-liner handy for such occasions [06:52:25] tcpdump -i eth0 -s 65535 -A -ttt port 11211 | cut -c 9- | grep -i '^get' | cut -d' ' -f2 [06:55:35] it's better now [06:56:07] Ah, nice. Thanks. [06:56:57] !log restarted memcached on virt1000 due to cache pollution from migration (different memc drivers w/different encoding) [06:57:03] Logged the message, Master [06:57:58] thanks ori [06:58:41] np. [06:58:43] * ori sleeps [07:00:24] RECOVERY - puppet last run on db1007 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [07:00:29] <_joe_> night ori [07:04:11] and the new cluster is named [07:04:19] sca - from service cluster a [07:04:47] mathoid probably going into production early next week [07:11:05] yep cronolog might be worth a try [07:11:44] kind of hesitant tbh. I have seen piped logs in apache misbehaving... [07:12:12] ending up with a couple or more lines of garbled logs [07:12:26] which is not much [07:12:49] but still. Plus we would do that to avoid sending a signal to passenger ? [07:13:09] well the entire apache + mod_passenger to be more precise [07:13:29] ah! I've never seen apache mispiping but it might have been luck [07:14:28] yeah that'd avoid apache doing the rotation, well anything that fixes the puppet o'clock alert storm would do :) [07:14:46] possibly trusty's version of mod_passenger is better? [07:14:57] could be. [07:15:04] it is worth a shot anyway [07:15:13] and reading up on passenger docs as well [07:15:29] which I intend to do as soon as I get some time [07:16:27] (03PS2) 10Giuseppe Lavagetto: Fix startup of hhvm [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 [07:16:59] <_joe_> or, we could upgrade to trusty, ditch apache on the puppetmasters and use nginx [07:17:09] indeed, I mentioned mod_passenger because when it crashed a while ago it seemed a lot of people were seeing the same crashes [07:17:24] <_joe_> (never had a problem with passenger+nginx managing a node app in prod) [07:17:48] <_joe_> godog: ^^ thanks for the CR on that btw [07:17:51] indeed that would work too [07:18:09] I actually dislike mod_passenger [07:18:31] <_joe_> passenger in its most recent incarnation is actually nice [07:18:41] <_joe_> we were managing nodejs with it [07:18:56] I meant the apache module [07:19:12] <_joe_> well, I dislike apache in general :) [07:19:17] ahahahaha [07:19:29] well somebody has to :P [07:19:31] <_joe_> but you know that from my interview :P [07:19:42] :-) [07:20:02] <_joe_> well, it's unfair to say I dislike it. [07:20:17] <_joe_> I prefer nginx for 99.9% of interesting things you can do [07:21:14] (03CR) 10Alexandros Kosiaris: "@Daniel. Seems like they are in use since puppet-master is complaining about non @-prepended variables contained in these files in logs." [puppet] - 10https://gerrit.wikimedia.org/r/157685 (owner: 10Matanya) [07:21:14] <_joe_> also, once you get used to the nginx paradigm, you'll end up thinking "how could I live with all those messy rewrites all this time?" [07:21:38] (03CR) 10Filippo Giunchedi: "we went from 4*1G to 4*10G networking on the frontends, to 10x increase (in theory, in practice I don't think swift could do 10G before ru" [puppet] - 10https://gerrit.wikimedia.org/r/157678 (owner: 10Filippo Giunchedi) [07:21:45] (03PS3) 10Alexandros Kosiaris: deployment: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/157685 (owner: 10Matanya) [07:22:13] (03CR) 10Giuseppe Lavagetto: Fix startup of hhvm (033 comments) [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 (owner: 10Giuseppe Lavagetto) [07:22:47] (03PS3) 10Giuseppe Lavagetto: Fix startup of hhvm [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 [07:22:56] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Fix startup of hhvm [debs/hhvm] - 10https://gerrit.wikimedia.org/r/157798 (owner: 10Giuseppe Lavagetto) [07:23:19] (03CR) 10Alexandros Kosiaris: [C: 032] "Ran through catalogcompiler, noop" [puppet] - 10https://gerrit.wikimedia.org/r/157685 (owner: 10Matanya) [07:23:49] <_joe_> akosiaris: someone fixed the compiler? [07:23:52] <_joe_> nice to know [07:23:59] <_joe_> I need to add a cleanup job [07:24:07] eeeh no [07:24:21] I fell back to the old one [07:24:34] what is there needed to be done about that one ? [07:25:20] <_joe_> cleaning of old jobs leftovers [07:29:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [07:31:12] <_joe_> done [07:34:20] (03CR) 10Alexandros Kosiaris: [C: 04-1] "A couple of missing stuff, otherwise looks pretty good" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/157856 (owner: 10Matanya) [07:39:02] (03CR) 10Filippo Giunchedi: mediawiki::monitoring::errors: report to statsd (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/158008 (owner: 10Ori.livneh) [07:40:35] (03PS5) 10Filippo Giunchedi: elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 [07:40:41] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] elasticsearch: handle request timeout and increase timeout [puppet] - 10https://gerrit.wikimedia.org/r/157805 (owner: 10Filippo Giunchedi) [07:42:30] (03PS2) 10Matanya: protoproxy: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/157856 [07:49:32] (03PS1) 10Giuseppe Lavagetto: Version bump for repackaging. [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158069 [07:50:43] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Version bump for repackaging. [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158069 (owner: 10Giuseppe Lavagetto) [07:55:26] <_joe_> !log re-enabling mw1192, what we were seeing was probably load and not anything else [07:55:33] Logged the message, Master [08:05:20] gerrit doesn't seem to have anchors to single comments I could link to? sadness [08:08:10] (03PS1) 10Matanya: gerrit: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/158071 [08:13:53] <_joe_> is anyone around here proficient with cmake? [08:14:13] PROBLEM - puppet last run on amslvs3 is CRITICAL: CRITICAL: Puppet has 1 failures [08:23:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [08:23:03] PROBLEM - Puppet freshness on elastic1004 is CRITICAL: Last successful Puppet run was Wed 03 Sep 2014 06:22:45 UTC [08:31:13] RECOVERY - puppet last run on amslvs3 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [08:42:22] (03PS1) 10Giuseppe Lavagetto: fix install path [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158072 [08:45:00] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] fix install path [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158072 (owner: 10Giuseppe Lavagetto) [09:05:08] (03PS2) 10Filippo Giunchedi: swift: check high load average on backend machines [puppet] - 10https://gerrit.wikimedia.org/r/157672 [09:05:15] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: check high load average on backend machines [puppet] - 10https://gerrit.wikimedia.org/r/157672 (owner: 10Filippo Giunchedi) [09:07:33] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] releases: do not include hostname in sudoers.d [puppet] - 10https://gerrit.wikimedia.org/r/157797 (owner: 10Filippo Giunchedi) [09:30:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [09:32:13] (03PS1) 10Giuseppe Lavagetto: Use ini-style comments [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158075 [09:33:52] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Use ini-style comments [debs/hhvm] - 10https://gerrit.wikimedia.org/r/158075 (owner: 10Giuseppe Lavagetto) [09:36:22] sigh, I'm taking a look at cronspam on tin [09:43:00] (03PS1) 10Filippo Giunchedi: releases: fully qualify sudo command [puppet] - 10https://gerrit.wikimedia.org/r/158076 [09:43:39] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] releases: fully qualify sudo command [puppet] - 10https://gerrit.wikimedia.org/r/158076 (owner: 10Filippo Giunchedi) [09:48:37] #thereifixedit [09:53:03] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [09:53:39] icinga-wm: shush! [09:54:03] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [10:23:43] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet last ran 14445 seconds ago, expected 14400 [10:24:03] PROBLEM - Puppet freshness on elastic1004 is CRITICAL: Last successful Puppet run was Wed 03 Sep 2014 06:22:45 UTC [10:24:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [10:45:47] (03PS4) 10Giuseppe Lavagetto: beta: use HHVM for all requests [puppet] - 10https://gerrit.wikimedia.org/r/157823 [10:46:28] (03PS1) 10JanZerebecki: icinga check wikidata: escape arguments [puppet] - 10https://gerrit.wikimedia.org/r/158081 [10:49:23] anyone up for merging a 1 line patch that hopefully corrects an icinga check so it goes green?^^ [10:49:45] <_joe_> jzerebecki: let me see [10:50:56] (03CR) 10Giuseppe Lavagetto: [C: 032] icinga check wikidata: escape arguments [puppet] - 10https://gerrit.wikimedia.org/r/158081 (owner: 10JanZerebecki) [10:51:05] <_joe_> simple enough [10:52:27] <_joe_> jzerebecki: and thanks once again :) [10:54:06] thank you :) [11:00:38] _joe_: can I trouble you please ? [11:06:13] <_joe_> matanya: yes [11:06:16] <_joe_> :) [11:06:26] can you please update https://etherpad.wikimedia.org/p/puppet3 [11:06:35] with up to date data at some point ? [11:06:59] <_joe_> yes! sorry I left that back [11:07:39] no rush, i still have some modules i didn't finish [11:07:50] was very busy lately [11:17:19] !log run gmond on elastic1001 manually to debug ES collector issues [11:17:24] Logged the message, Master [11:21:31] <_joe_> matanya: done btw [11:21:38] thank you! [11:22:00] <_joe_> and I'll be back later :) [11:22:25] <_joe_> we're down to less than 500 warnings, we had more than 700 [11:22:48] <_joe_> I'll work on this sooner than later [11:23:23] akosiaris: Hi, are there any updates regarding the Mathoid deployment? [11:23:48] !log run gmond on elastic1002 manually to debug ES collector issues [11:23:54] Logged the message, Master [11:24:14] physikerwelt: yes, machines are almost assigned. Probable names sca1001, sca1002 [11:24:58] physikerwelt: almost == baring any unexpected changes [11:27:39] akosiaris: Thank you very much for supporting Mathoid. Just ping me if it makes sense to update https://gerrit.wikimedia.org/r/#/c/156576/ [11:29:32] physikerwelt: ok, will do [11:31:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [11:54:49] (03PS1) 10Matanya: pybal: qualify vars [puppet] - 10https://gerrit.wikimedia.org/r/158086 [12:25:03] PROBLEM - Puppet freshness on elastic1004 is CRITICAL: Last successful Puppet run was Wed 03 Sep 2014 06:22:45 UTC [12:25:03] PROBLEM - Puppet freshness on mw1053 is CRITICAL: Last successful Puppet run was Sun 31 Aug 2014 01:50:18 UTC [12:41:07] <_joe_> !log mw1120: remove from pybal, schedule downtime, reimage to HAT [12:41:13] Logged the message, Master [12:42:03] <_joe_> !log typo: mw1020, not mw1120 [12:42:10] Logged the message, Master [12:46:36] anyone any idea why my icinga check gives me: 'The command defined for service ... does not exist' https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=wikidata&service=check+if+wikidata.org+dispatch+lag+is+higher+than+2+minutes [12:53:03] <_joe_> jzerebecki: sorry, can't help you now [12:54:50] Did wikitech CSS re-break? [12:55:06] Reedy: Ori restarted memcache, but it seems re-broken. [12:55:16] It's quite likely [12:55:22] If people hit virt1000 and pollute the cache [12:57:29] (03PS1) 10Giuseppe Lavagetto: reimage: fix unbound variable [puppet] - 10https://gerrit.wikimedia.org/r/158089 [12:59:14] Hopefully we can get it finished up today [13:00:05] K4-713: Respected human, time to deploy Fundraising (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140903T1300). Please do the needful. [13:13:11] (03CR) 10Giuseppe Lavagetto: [C: 032] reimage: fix unbound variable [puppet] - 10https://gerrit.wikimedia.org/r/158089 (owner: 10Giuseppe Lavagetto) [13:30:20] nostalgia skin on wikitech wiki? [13:30:35] aude: borked css [13:30:40] :( [13:30:48] virt1000 is seemingly polluting memcached due to the different driver used etc [13:31:04] though, bar missing tables for extensions, I think virt1000 is ready to go [13:31:12] <_joe_> I actually prefer it this way, btw [13:31:49] heh [13:32:03] PROBLEM - Puppet freshness on virt1000 is CRITICAL: Last successful Puppet run was Tue 02 Sep 2014 23:24:49 UTC [13:33:07] Didn't we have a skin for this before? ;) [13:36:27]