[00:00:04] (03CR) 10Chad: [C: 032] clean.py: Rework command execution, reduce code dupe [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339032 (owner: 10Chad) [00:00:04] addshore, hashar, anomie, ostriches, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, and thcipriani: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170222T0000). Please do the needful. [00:00:05] ebernhardson and tgr: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [00:00:28] \o [00:00:30] (03PS1) 10Ladsgroup: Enable ORES review tool in cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339090 (https://phabricator.wikimedia.org/T151611) [00:00:45] jouncebot: next [00:00:45] In 13 hour(s) and 59 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170222T1400) [00:01:19] (03PS3) 10Dzahn: sshd-phab service config needs to be a template [puppet] - 10https://gerrit.wikimedia.org/r/338302 (owner: 1020after4) [00:01:31] I can SWAT [00:01:31] tgr: damn that's a lot of config patches :P [00:01:31] tgr: Will leave comments on change [00:01:54] (03Merged) 10jenkins-bot: clean.py: Rework command execution, reduce code dupe [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339032 (owner: 10Chad) [00:02:08] Hey, I added one item to SWAT [00:02:13] sorry for being late [00:02:21] thcipriani: mine are both the same, javascript only changes which remove some test code [00:02:30] AB test code [00:02:50] okie doke [00:03:29] are we over the SWAT limit? I can remove some of those config changes [00:03:34] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339041 (https://phabricator.wikimedia.org/T158698) (owner: 10Gergő Tisza) [00:03:53] OTOH the first three are trivial beta-only changes that can be synced without looking [00:03:53] I *think* we'll be ok [00:04:21] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:04:29] (03CR) 10jenkins-bot: clean.py: Rework command execution, reduce code dupe [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339032 (owner: 10Chad) [00:04:41] Whoops, forgot it was swat time already [00:04:47] thcipriani: My sync will be done before your first merge lands [00:04:58] RainbowSprinkles: no problem, go for it [00:05:05] !log demon@tin Synchronized scap/plugins/clean.py: More code cleanup (duration: 00m 40s) [00:05:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:06:03] (03Merged) 10jenkins-bot: Fix PageViewInfo config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339041 (https://phabricator.wikimedia.org/T158698) (owner: 10Gergő Tisza) [00:06:35] 06Operations, 10Gerrit, 06Release-Engineering-Team: Decide weather to disable drafts in gerrit - https://phabricator.wikimedia.org/T158656#3045107 (10Dzahn) I have never used a draft and would not miss them. [00:07:01] PROBLEM - puppet last run on elastic1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:07:21] (03CR) 10Chad: clean.py: Fix up l10nupdate-owned files on masters (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339035 (owner: 10Chad) [00:08:42] (03CR) 10jenkins-bot: Fix PageViewInfo config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339041 (https://phabricator.wikimedia.org/T158698) (owner: 10Gergő Tisza) [00:08:56] (03PS2) 10Thcipriani: Fix Sentry URL scheme on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:09:15] (03CR) 10Thcipriani: Fix Sentry URL scheme on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:09:22] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:09:43] 06Operations, 10Gerrit, 06Release-Engineering-Team: Decide weather to disable drafts in gerrit - https://phabricator.wikimedia.org/T158656#3045110 (10Paladox) >>! In T158656#3045107, @Dzahn wrote: > I have never used a draft and would not miss them. There being replaced with private edits :) [00:10:10] (03PS1) 10BryanDavis: Use POST when fetching parsed HTML for wikitext [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339092 (https://phabricator.wikimedia.org/T158715) [00:11:10] (03Merged) 10jenkins-bot: Fix Sentry URL scheme on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:11:19] (03CR) 10jenkins-bot: Fix Sentry URL scheme on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339012 (owner: 10Gergő Tisza) [00:11:38] ebernhardson: your changes for both .12 and .13 are live on mwdebug1002 if there's anything to check there. [00:11:59] (03PS4) 10Thcipriani: Fix SiteConfiguration array merge syntax [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336747 (https://phabricator.wikimedia.org/T157656) (owner: 10Gergő Tisza) [00:12:08] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336747 (https://phabricator.wikimedia.org/T157656) (owner: 10Gergő Tisza) [00:13:05] !log smalyshev@tin Started deploy [wdqs/wdqs@7768422]: Deploy 2.1.5RC WAR on 2001 for testing [00:13:07] thcipriani: code seems to load and not complain, tis about all i can check [00:13:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:13:31] !log smalyshev@tin Finished deploy [wdqs/wdqs@7768422]: Deploy 2.1.5RC WAR on 2001 for testing (duration: 00m 25s) [00:13:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:13:47] ebernhardson: wfm, ok, going live wmf.13 first [00:13:52] (03PS5) 10Chad: Scap clean: Rework --l10n-only into --keep-static [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336730 (https://phabricator.wikimedia.org/T73313) [00:14:27] (03Merged) 10jenkins-bot: Fix SiteConfiguration array merge syntax [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336747 (https://phabricator.wikimedia.org/T157656) (owner: 10Gergő Tisza) [00:14:36] (03CR) 10jenkins-bot: Fix SiteConfiguration array merge syntax [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336747 (https://phabricator.wikimedia.org/T157656) (owner: 10Gergő Tisza) [00:15:01] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:15:08] (03CR) 10BryanDavis: [C: 032] Use POST when fetching parsed HTML for wikitext [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339092 (https://phabricator.wikimedia.org/T158715) (owner: 10BryanDavis) [00:16:00] !log thcipriani@tin Synchronized php-1.29.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:339011|Turn off sister search AB test.]] T157942 (duration: 00m 43s) [00:16:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:16:05] T157942: turn off sistersearch A/B/C test on 2017-02-21 - https://phabricator.wikimedia.org/T157942 [00:16:28] (03PS6) 10Chad: Scap clean: Rework --l10n-only into --keep-static [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336730 (https://phabricator.wikimedia.org/T73313) [00:17:07] !log thcipriani@tin Synchronized php-1.29.0-wmf.12/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: [[gerrit:339010|Turn off sister search AB test.]] T157942 (duration: 00m 39s) [00:17:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:17:37] (03CR) 10Chad: clean.py: Fix up l10nupdate-owned files on masters (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339035 (owner: 10Chad) [00:18:10] (03Merged) 10jenkins-bot: Use POST when fetching parsed HTML for wikitext [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339092 (https://phabricator.wikimedia.org/T158715) (owner: 10BryanDavis) [00:18:21] PROBLEM - Check systemd state on elastic1043 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [00:18:26] thcipriani: thanks! i'll keep an eye on the events coming in to make sure nothing crazy happens [00:18:31] PROBLEM - Elasticsearch HTTPS on elastic1043 is CRITICAL: SSL CRITICAL - failed to verify search.svc.eqiad.wmnet against elastic1043.eqiad.wmnet [00:20:40] thcipriani: the other conifg patches need testing on mwdebug; the two exception-related ones can go together [00:20:46] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [[gerrit:336747|Fix SiteConfiguration array merge syntax]] T157656 [[gerrit:339012|Fix Sentry URL scheme on beta]] [[gerrit:339041|Fix PageViewInfo config]] T158698 (beta-only changes) (duration: 00m 39s) [00:20:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:20:52] T158698: PageViewInfo does not work on beta - https://phabricator.wikimedia.org/T158698 [00:20:52] T157656: "Undefined index: wmgExtraLanguageNames" on beta - https://phabricator.wikimedia.org/T157656 [00:21:02] ^ 1043 seems to have gotten a bad certificate somehow :S will make a ticket for gehel shouldn't be a problem but needs to be depooled (if it is pooled) [00:21:20] tgr: yup, just looking at them now [00:22:05] (03PS10) 10Thcipriani: Set $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324215 (https://phabricator.wikimedia.org/T154698) (owner: 10Anomie) [00:22:13] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324215 (https://phabricator.wikimedia.org/T154698) (owner: 10Anomie) [00:24:25] (03Merged) 10jenkins-bot: Set $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324215 (https://phabricator.wikimedia.org/T154698) (owner: 10Anomie) [00:24:35] (03CR) 10jenkins-bot: Set $wgSoftBlockRanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/324215 (https://phabricator.wikimedia.org/T154698) (owner: 10Anomie) [00:25:32] (03PS3) 10BryanDavis: Use IB3 library [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339066 [00:26:26] (03PS2) 10Chad: clean.py: Fix up l10nupdate-owned files on masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339035 [00:26:28] tgr: Set $wgSoftBlockRanges is on mwdebug1002, check please [00:28:22] (03PS4) 10Thcipriani: Send 'exception' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323111 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:29:06] hm, either that's not working or I have misconceptions on how it can be tested [00:29:37] tgr: whoops, forgot to rebase, sorry :( [00:30:10] tgr: ok, try now [00:30:38] (03PS1) 10Chad: clean.py: Minor abstraction of param handling [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339094 [00:30:57] works, thx [00:31:35] ok, will go live with initialisesettings first and then all of wmf-config in 2 syncs [00:33:12] (03CR) 10BryanDavis: [C: 032] "Tested via cherry-pick." [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339066 (owner: 10BryanDavis) [00:33:42] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:324215|Set $wgSoftBlockRanges]] T154698 PART I (duration: 00m 40s) [00:33:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:33:47] T154698: Prevent contributions attributed to private and WMF IP addresses - https://phabricator.wikimedia.org/T154698 [00:33:59] (03Merged) 10jenkins-bot: Use IB3 library [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339066 (owner: 10BryanDavis) [00:34:53] !log thcipriani@tin Synchronized wmf-config: SWAT: [[gerrit:324215|Set $wgSoftBlockRanges]] T154698 PART II (duration: 00m 42s) [00:34:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:34:58] ^ tgr live now [00:35:03] RECOVERY - puppet last run on elastic1033 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:35:05] (03PS2) 10Chad: Scap clean: Automate purging of old deployment branches from gerrit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336901 [00:35:07] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323111 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:35:29] (03PS3) 10BryanDavis: Fix flake8 E128 warnings [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339067 [00:35:50] (03PS2) 10BryanDavis: Run flake8 on both python2 and python3 [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339076 [00:36:28] (03Merged) 10jenkins-bot: Send 'exception' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323111 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:36:44] (03PS3) 10Thcipriani: Do not send 'exception-json' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323330 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:36:51] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323330 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:37:37] (03CR) 10jenkins-bot: Send 'exception' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323111 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:37:52] tgr: send exception channel to logstash is on mwdebug1002 if you want to check it there. [00:38:18] (03Merged) 10jenkins-bot: Do not send 'exception-json' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323330 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:38:19] thx, I will [00:38:47] although probably better to check them together [00:39:16] tgr: ok, they are both there now [00:39:43] (03CR) 10jenkins-bot: Do not send 'exception-json' channel to logstash [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323330 (https://phabricator.wikimedia.org/T136849) (owner: 10Gergő Tisza) [00:40:32] (03CR) 10Dzahn: [C: 032] sshd-phab service config needs to be a template [puppet] - 10https://gerrit.wikimedia.org/r/338302 (owner: 1020after4) [00:44:01] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:44:56] thanks thcipriani, it seems to work [00:45:17] tgr: great! will sync [00:45:33] (03CR) 10Dzahn: "on iridium:" [puppet] - 10https://gerrit.wikimedia.org/r/338302 (owner: 1020after4) [00:46:07] (03CR) 10Dzahn: "should have fixed it on jessie systems. yea. 9 ExecStart=/usr/sbin/sshd -D -f/etc/ssh/sshd_config.phabricator" [puppet] - 10https://gerrit.wikimedia.org/r/338302 (owner: 1020after4) [00:48:08] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:323111|Send "exception" channel to logstash]] [[gerrit:323330|Do not send "exception-json" channel to logstash]] T136849 (duration: 00m 40s) [00:48:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:48:14] T136849: normalized_message is a JSON dump of the whole event for exceptions in beta logstash - https://phabricator.wikimedia.org/T136849 [00:48:14] ^ tgr live everywhere [00:48:21] thanks! [00:48:26] yw :) [00:48:44] (03CR) 10BryanDavis: [C: 032] Fix flake8 E128 warnings [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339067 (owner: 10BryanDavis) [00:49:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 5673 MB (12% inode=98%) [00:49:22] (03Merged) 10jenkins-bot: Fix flake8 E128 warnings [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339067 (owner: 10BryanDavis) [00:49:24] (03Merged) 10jenkins-bot: Run flake8 on both python2 and python3 [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/339076 (owner: 10BryanDavis) [00:49:32] (03PS2) 10Thcipriani: Enable ORES review tool in cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339090 (https://phabricator.wikimedia.org/T151611) (owner: 10Ladsgroup) [00:49:37] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339090 (https://phabricator.wikimedia.org/T151611) (owner: 10Ladsgroup) [00:49:42] (03PS3) 10Dzahn: contint: drop npm settings for precise [puppet] - 10https://gerrit.wikimedia.org/r/337203 (https://phabricator.wikimedia.org/T158652) [00:50:21] Amir1: trying to remember the order of operations for setting up ores on new wikis: create tables then run maintenance scripts then sync? [00:50:30] (03PS2) 10Dzahn: labs_vagrant: drop precise support [puppet] - 10https://gerrit.wikimedia.org/r/337205 (https://phabricator.wikimedia.org/T143349) [00:50:58] thcipriani: sync, create tables, run maintenance [00:51:12] RECOVERY - Disk space on elastic1035 is OK: DISK OK [00:51:14] (03PS2) 10Dzahn: toollabs: drop precise-related monitoring check [puppet] - 10https://gerrit.wikimedia.org/r/337207 (https://phabricator.wikimedia.org/T143349) [00:51:29] Amir1: ah, ok, so it doesn't freak out if it doesn't find the tables initially? [00:51:44] (03Merged) 10jenkins-bot: Enable ORES review tool in cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339090 (https://phabricator.wikimedia.org/T151611) (owner: 10Ladsgroup) [00:51:47] hmm [00:51:51] you're right [00:51:52] (03CR) 10jenkins-bot: Enable ORES review tool in cswiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339090 (https://phabricator.wikimedia.org/T151611) (owner: 10Ladsgroup) [00:52:15] create tables, run maintenance and then sync [00:52:25] (03CR) 10Dzahn: [C: 031] Introduce linters using rake [puppet/wikimetrics] - 10https://gerrit.wikimedia.org/r/338387 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [00:52:41] oh oh [00:52:53] (I was thinking without syncing, can not access to the maintenance scripts /sql files) [00:52:59] jouncebot's grid job is going nuts :/ [00:53:36] (03CR) 10Dzahn: [C: 031] Introduce linters using rake [puppet/kafka] - 10https://gerrit.wikimedia.org/r/338385 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [00:53:47] the nickser recover stuff seems to work though :) [00:55:36] Amir1: ok, tables created [00:56:04] Amir1: should I fetch over to mwdebug for you to check or maintenance first? [00:56:33] without syncing we can't test it [00:57:07] :) [00:57:41] Amir1: ok, on mwdebug1002, check please [00:58:09] Thanks! [01:00:51] thcipriani: good! [01:01:19] Amir1: cool, so I'll sync initialisesettings and run the maintenance scripts, sound right? [01:01:44] awesome [01:01:46] thanks [01:03:34] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: [[gerrit:339090|Enable ORES review tool in cswiki]] T151611 (duration: 00m 39s) [01:03:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:03:39] T151611: Enable ORES Review Tool on Czech Wikipedia - https://phabricator.wikimedia.org/T151611 [01:05:12] Amir1: ran checkmodelversions, running populatedatabase now [01:06:42] thcipriani: Awesome, Tested and works great [01:07:03] Amir1: cool, glad to hear it, thanks for testing :) [01:10:40] (03PS10) 10Madhuvishy: nfs: Snapshot backup device on secondary DC before replicating latest from remote [puppet] - 10https://gerrit.wikimedia.org/r/334692 (https://phabricator.wikimedia.org/T149870) [01:11:30] (03CR) 10Paladox: [C: 031] contint: drop npm settings for precise [puppet] - 10https://gerrit.wikimedia.org/r/337203 (https://phabricator.wikimedia.org/T158652) (owner: 10Dzahn) [01:12:11] (03CR) 10Krinkle: contint: drop npm settings for precise (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/337203 (https://phabricator.wikimedia.org/T158652) (owner: 10Dzahn) [01:13:23] (03PS5) 10Paladox: Phabricator: Add systemd template [puppet] - 10https://gerrit.wikimedia.org/r/338022 [01:13:35] Amir1: populatedatabase completed! [01:13:49] (03PS6) 10Paladox: Phabricator: Make ssh-phab port configurable [puppet] - 10https://gerrit.wikimedia.org/r/338294 [01:13:51] (03CR) 10jerkins-bot: [V: 04-1] nfs: Snapshot backup device on secondary DC before replicating latest from remote [puppet] - 10https://gerrit.wikimedia.org/r/334692 (https://phabricator.wikimedia.org/T149870) (owner: 10Madhuvishy) [01:14:36] thcipriani: Awesome [01:14:46] we are done now. I'll make the announcement [01:15:11] (03PS11) 10Madhuvishy: nfs: Snapshot backup device on secondary DC before replicating latest from remote [puppet] - 10https://gerrit.wikimedia.org/r/334692 (https://phabricator.wikimedia.org/T149870) [01:19:11] (03PS6) 10Tim Landscheidt: Tools: Make tools-clush-generator project-agnostic [puppet] - 10https://gerrit.wikimedia.org/r/326892 [01:19:13] (03PS4) 10Tim Landscheidt: Tools: Generate node sets dynamically [puppet] - 10https://gerrit.wikimedia.org/r/328030 [01:21:33] (03PS7) 10Paladox: Phabricator: Make ssh-phab port configurable [puppet] - 10https://gerrit.wikimedia.org/r/338294 [01:23:31] (03PS8) 10Paladox: Phabricator: Make ssh-phab port configurable [puppet] - 10https://gerrit.wikimedia.org/r/338294 [01:24:32] (03CR) 10Tim Landscheidt: "@chasemp: The on-demand query takes about 1.5 s (time /usr/local/sbin/tools-clush-generator map --observer-pass $OBSERVER_PASS --project t" [puppet] - 10https://gerrit.wikimedia.org/r/328030 (owner: 10Tim Landscheidt) [01:24:58] (03CR) 10Paladox: "I managed to get ssh working on a labs instance with phabricator installed :)." [puppet] - 10https://gerrit.wikimedia.org/r/338294 (owner: 10Paladox) [01:27:59] (03CR) 10Chad: "I guess I should also grant force push rights on extensions on wmf/* branches to deployers, otherwise nobody will be able to do this." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336901 (owner: 10Chad) [01:32:19] (03CR) 10Chad: "Oh, we already have it, nvm" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336901 (owner: 10Chad) [01:48:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 4362 MB (9% inode=98%) [01:51:51] PROBLEM - IPv6 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 22 probes of 264 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [01:54:11] RECOVERY - Disk space on elastic1035 is OK: DISK OK [01:56:51] PROBLEM - IPv4 ping to codfw on ripe-atlas-codfw is CRITICAL: CRITICAL - failed 21 probes of 415 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [01:56:51] RECOVERY - IPv6 ping to codfw on ripe-atlas-codfw is OK: OK - failed 16 probes of 264 (alerts on 19) - https://atlas.ripe.net/measurements/1791212/#!map [02:00:21] PROBLEM - puppet last run on cobalt is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:00:46] (03CR) 10Krinkle: [C: 031] Scap clean: Automate purging of old deployment branches from gerrit [mediawiki-config] - 10https://gerrit.wikimedia.org/r/336901 (owner: 10Chad) [02:01:26] Krinkle: Thx for the review. There's a whole chain of stuff to land here :) [02:01:31] Should be much nicer soon [02:01:51] RECOVERY - IPv4 ping to codfw on ripe-atlas-codfw is OK: OK - failed 3 probes of 415 (alerts on 19) - https://atlas.ripe.net/measurements/1791210/#!map [02:01:59] https://gerrit.wikimedia.org/r/#/q/topic:cleaner-scap+status:open is the full topic [02:04:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 596 MB (1% inode=98%) [02:07:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 3694 MB (8% inode=98%) [02:09:07] gehel: Got 2 more of those ^ [02:10:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 4631 MB (10% inode=98%) [02:11:23] RECOVERY - puppet last run on cobalt is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [02:12:09] elastic{1043,1039,1035} [02:12:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4207 MB (9% inode=98%) [02:12:31] RECOVERY - Disk space on elastic1044 is OK: DISK OK [02:12:41] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:14:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4753 MB (10% inode=98%) [02:28:11] RECOVERY - Disk space on elastic1035 is OK: DISK OK [02:29:21] PROBLEM - puppet last run on snapshot1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:29:21] RECOVERY - Disk space on elastic1039 is OK: DISK OK [02:32:42] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.12) (duration: 11m 48s) [02:32:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:33:09] (03CR) 10Subramanya Sastry: Ruthenium VisualDiff: Test w/ local Parsoid instead of prod Parsoid (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/338950 (owner: 10Subramanya Sastry) [02:37:01] PROBLEM - Check whether ferm is active by checking the default input chain on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:37:42] PROBLEM - Check systemd state on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:37:42] PROBLEM - Check size of conntrack table on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:37:51] PROBLEM - DPKG on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:37:51] PROBLEM - puppet last run on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:38:21] PROBLEM - SSH on bast3001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:41] PROBLEM - configured eth on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:38:41] PROBLEM - dhclient process on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:38:42] PROBLEM - salt-minion processes on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:39:31] RECOVERY - dhclient process on bast3001 is OK: PROCS OK: 0 processes with command name dhclient [02:39:31] RECOVERY - configured eth on bast3001 is OK: OK - interfaces up [02:39:32] RECOVERY - salt-minion processes on bast3001 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [02:39:41] RECOVERY - Check size of conntrack table on bast3001 is OK: OK: nf_conntrack is 0 % full [02:39:41] RECOVERY - DPKG on bast3001 is OK: All packages OK [02:39:42] RECOVERY - Check systemd state on bast3001 is OK: OK - running: The system is fully operational [02:40:01] RECOVERY - Check whether ferm is active by checking the default input chain on bast3001 is OK: OK ferm input default policy is set [02:40:21] RECOVERY - SSH on bast3001 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0) [02:40:41] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:40:51] RECOVERY - puppet last run on bast3001 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [02:44:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4391 MB (9% inode=98%) [02:46:21] RECOVERY - Disk space on elastic1039 is OK: DISK OK [02:46:42] PROBLEM - Check systemd state on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:46:51] PROBLEM - puppet last run on bast3001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:47:31] RECOVERY - Check systemd state on bast3001 is OK: OK - running: The system is fully operational [02:47:41] RECOVERY - puppet last run on bast3001 is OK: OK: Puppet is currently enabled, last run 31 minutes ago with 0 failures [02:54:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4626 MB (10% inode=98%) [02:55:21] RECOVERY - Disk space on elastic1039 is OK: DISK OK [02:58:21] RECOVERY - puppet last run on snapshot1001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:00:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 42 MB (0% inode=98%) [03:00:31] PROBLEM - puppet last run on analytics1045 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:04:27] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.13) (duration: 13m 54s) [03:04:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:10:13] !log l10nupdate@tin ResourceLoader cache refresh completed at Wed Feb 22 03:10:13 UTC 2017 (duration 5m 46s) [03:10:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:21] RECOVERY - Disk space on elastic1039 is OK: DISK OK [03:22:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 5701 MB (12% inode=98%) [03:24:31] RECOVERY - Disk space on elastic1044 is OK: DISK OK [03:28:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 4106 MB (9% inode=98%) [03:29:31] RECOVERY - puppet last run on analytics1045 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [03:33:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 4150 MB (9% inode=98%) [03:33:41] PROBLEM - puppet last run on db1090 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:35:12] RECOVERY - Disk space on elastic1035 is OK: DISK OK [03:35:31] RECOVERY - Disk space on elastic1044 is OK: DISK OK [03:49:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4916 MB (11% inode=98%) [04:01:41] RECOVERY - puppet last run on db1090 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:02:14] (03PS3) 10Andrew Bogott: Tools: Enable PHP module mcrypt on Trusty execution nodes [puppet] - 10https://gerrit.wikimedia.org/r/324957 (https://phabricator.wikimedia.org/T97857) (owner: 10Tim Landscheidt) [04:04:41] (03CR) 10Andrew Bogott: [C: 032] Tools: Enable PHP module mcrypt on Trusty execution nodes [puppet] - 10https://gerrit.wikimedia.org/r/324957 (https://phabricator.wikimedia.org/T97857) (owner: 10Tim Landscheidt) [04:14:51] PROBLEM - Unmerged changes on repository puppet on puppetmaster1001 is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet, ref HEAD..origin/production). [04:21:21] RECOVERY - Disk space on elastic1039 is OK: DISK OK [04:35:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 5612 MB (12% inode=98%) [04:45:31] RECOVERY - Disk space on elastic1044 is OK: DISK OK [05:02:11] (03CR) 1020after4: "So you want to abort instead of just continuing and skipping the problematic key? Does it really add any safety?" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/338984 (https://phabricator.wikimedia.org/T158660) (owner: 10Volans) [05:02:18] (03CR) 1020after4: [C: 031] Keyholder: fix filter of passwordless keys [puppet] - 10https://gerrit.wikimedia.org/r/338984 (https://phabricator.wikimedia.org/T158660) (owner: 10Volans) [05:03:25] (03CR) 1020after4: [C: 031] Keyholder: add support for ed25519 keys [puppet] - 10https://gerrit.wikimedia.org/r/339002 (https://phabricator.wikimedia.org/T158659) (owner: 10Volans) [05:04:01] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=1873.40 Read Requests/Sec=4170.70 Write Requests/Sec=8.20 KBytes Read/Sec=18360.40 KBytes_Written/Sec=82.40 [05:04:17] (03CR) 1020after4: [C: 031] Revert "Add replication client grants to phuser" [puppet] - 10https://gerrit.wikimedia.org/r/335554 (owner: 10Jcrespo) [05:17:01] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=164.80 Read Requests/Sec=203.90 Write Requests/Sec=64.80 KBytes Read/Sec=3082.40 KBytes_Written/Sec=377.60 [06:04:03] 06Operations, 10Continuous-Integration-Config, 06Release-Engineering-Team, 06Wikipedia-Android-App-Backlog, and 2 others: Investigate how to improve Android CI performance and stability - https://phabricator.wikimedia.org/T158014#3045538 (10Niedzielski) Hmm... I've been trying to debug this further but aft... [06:15:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 5689 MB (12% inode=98%) [06:19:31] RECOVERY - Disk space on elastic1044 is OK: DISK OK [06:46:20] (03PS2) 10Subramanya Sastry: Ruthenium VisualDiff: Test w/ local Parsoid instead of prod Parsoid [puppet] - 10https://gerrit.wikimedia.org/r/338950 [06:47:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 5456 MB (12% inode=98%) [06:49:21] PROBLEM - Disk space on elastic1039 is CRITICAL: DISK CRITICAL - free space: / 4963 MB (11% inode=98%) [06:49:31] PROBLEM - puppet last run on maps1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:49:31] PROBLEM - Disk space on elastic1044 is CRITICAL: DISK CRITICAL - free space: / 5311 MB (11% inode=98%) [07:04:01] PROBLEM - Start a job and verify on Precise on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/precise - 185 bytes in 0.257 second response time [07:07:11] RECOVERY - Disk space on elastic1035 is OK: DISK OK [07:09:01] RECOVERY - Start a job and verify on Precise on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.140 second response time [07:13:06] (03PS1) 10Marostegui: db-codfw.php: Repool db2055 restore db2048 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339118 (https://phabricator.wikimedia.org/T132416) [07:16:03] (03CR) 10Marostegui: [C: 032] db-codfw.php: Repool db2055 restore db2048 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339118 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:17:28] (03Merged) 10jenkins-bot: db-codfw.php: Repool db2055 restore db2048 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339118 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:17:31] RECOVERY - puppet last run on maps1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:17:36] (03CR) 10jenkins-bot: db-codfw.php: Repool db2055 restore db2048 weight [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339118 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:18:36] !log marostegui@tin Synchronized wmf-config/db-codfw.php: Repool db2055 - T132416 (duration: 00m 40s) [07:18:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:42] T132416: Rampant differences in indexes on enwiki.revision across the DB cluster - https://phabricator.wikimedia.org/T132416 [07:19:02] (03PS1) 10Marostegui: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339119 (https://phabricator.wikimedia.org/T132416) [07:21:23] (03CR) 10Marostegui: [C: 032] db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339119 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:22:32] (03Merged) 10jenkins-bot: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339119 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:22:40] (03CR) 10jenkins-bot: db-codfw.php: Depool db2062 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/339119 (https://phabricator.wikimedia.org/T132416) (owner: 10Marostegui) [07:23:17] !log Deploy alter table enwiki.revision db2062 - T132416 [07:23:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:23:31] !log marostegui@tin Synchronized wmf-config/db-codfw.php: Depool db2062 - T132416 (duration: 00m 40s) [07:23:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:24:01] PROBLEM - Start a job and verify on Precise on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/precise - 185 bytes in 1.361 second response time [07:25:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 5126 MB (11% inode=98%) [07:29:01] RECOVERY - Start a job and verify on Precise on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.128 second response time [07:30:11] PROBLEM - Disk space on elastic1035 is CRITICAL: DISK CRITICAL - free space: / 5314 MB (11% inode=98%) [07:43:18] !log trncating logs on elastic10(35|39|44) [07:43:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:44:57] !log restart elasticsearch on elastic1035 [07:45:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:47:11] RECOVERY - Disk space on elastic1035 is OK: DISK OK [07:57:09] (03PS2) 10Filippo Giunchedi: uwsgi: parametrize service settings [puppet] - 10https://gerrit.wikimedia.org/r/338804 [07:57:11] (03PS2) 10Filippo Giunchedi: coal: disable uwsgi autoload [puppet] - 10https://gerrit.wikimedia.org/r/338805 [07:57:13] (03PS1) 10Filippo Giunchedi: graphite: split checks for carbon-relay frontend/local drops [puppet] - 10https://gerrit.wikimedia.org/r/339122 [07:59:23] (03PS15) 10Giuseppe Lavagetto: Add schema support [software/conftool] - 10https://gerrit.wikimedia.org/r/288881 (https://phabricator.wikimedia.org/T155823) [08:02:28] (03CR) 10Giuseppe Lavagetto: Add schema support (032 comments) [software/conftool] - 10https://gerrit.wikimedia.org/r/288881 (https://phabricator.wikimedia.org/T155823) (owner: 10Giuseppe Lavagetto) [08:06:49] (03CR) 10Filippo Giunchedi: [C: 032] graphite: split checks for carbon-relay frontend/local drops [puppet] - 10https://gerrit.wikimedia.org/r/339122 (owner: 10Filippo Giunchedi) [08:07:35] !log upgrading openssl on redis clusters / various base service restarts [08:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:12:49] (03PS2) 10Filippo Giunchedi: graphite: split checks for carbon-relay frontend/local drops [puppet] - 10https://gerrit.wikimedia.org/r/339122 [08:12:51] (03PS3) 10Filippo Giunchedi: uwsgi: parametrize service settings [puppet] - 10https://gerrit.wikimedia.org/r/338804 [08:12:53] (03PS3) 10Filippo Giunchedi: coal: disable uwsgi autoload [puppet] - 10https://gerrit.wikimedia.org/r/338805 [08:17:07] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] graphite: split checks for carbon-relay frontend/local drops [puppet] - 10https://gerrit.wikimedia.org/r/339122 (owner: 10Filippo Giunchedi) [08:17:45] andrewbogott: I'm merging your change [08:18:05] and hope it doesn't break anything on tools [08:18:51] RECOVERY - Unmerged changes on repository puppet on puppetmaster1001 is OK: No changes to merge. [08:19:48] (03PS2) 10Filippo Giunchedi: Increase SWIFT_RETRIES in Thumbor [puppet] - 10https://gerrit.wikimedia.org/r/338957 (https://phabricator.wikimedia.org/T157949) (owner: 10Gilles) [08:22:01] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] Increase SWIFT_RETRIES in Thumbor [puppet] - 10https://gerrit.wikimedia.org/r/338957 (https://phabricator.wikimedia.org/T157949) (owner: 10Gilles) [08:28:15] (03PS3) 10Giuseppe Lavagetto: prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 [08:28:33] (03PS1) 10Elukey: Add the Apache Prometheus exporter to bohrium [puppet] - 10https://gerrit.wikimedia.org/r/339123 (https://phabricator.wikimedia.org/T154558) [08:28:41] 06Operations, 10Analytics: sync bohrium and apt.wikimedia.org piwik versions - https://phabricator.wikimedia.org/T149993#2771280 (10MoritzMuehlenhoff) That's still the case. Since piwik 2.16 is up and running on bohrium already, we could also simply use "dpkg-repack" to generate the deb from the installation o... [08:29:40] (03CR) 10jerkins-bot: [V: 04-1] prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 (owner: 10Giuseppe Lavagetto) [08:31:19] 06Operations, 10Analytics: sync bohrium and apt.wikimedia.org piwik versions - https://phabricator.wikimedia.org/T149993#3045761 (10elukey) We have also opened https://phabricator.wikimedia.org/T158322 to upgrade piwik to a more recent version, should we set up another component like we did for thirdparty/cdh? [08:31:50] (03CR) 10Elukey: [C: 032] "https://puppet-compiler.wmflabs.org/5521/bohrium.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/339123 (https://phabricator.wikimedia.org/T154558) (owner: 10Elukey) [08:33:38] (03CR) 10Filippo Giunchedi: [C: 031] "LGTM, minor nit on /metrics being the default" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/336852 (owner: 10Giuseppe Lavagetto) [08:34:11] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[openssl] [08:34:16] (03PS4) 10Giuseppe Lavagetto: prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 [08:35:13] 06Operations, 10Analytics: sync bohrium and apt.wikimedia.org piwik versions - https://phabricator.wikimedia.org/T149993#3045766 (10MoritzMuehlenhoff) Let's wait with adding new components until we have a plan how to move forward with the repository. piwik is a single package, so there's no risk of it overshad... [08:36:11] <_joe_> godog: heh found an issue with my PS with the switch to TLS [08:36:16] <_joe_> godog: fixing :) [08:37:49] hehe ok [08:43:32] (03CR) 10Hashar: "From their README, that has been made on purpose I guess for the purpose of scripting?" [puppet] - 10https://gerrit.wikimedia.org/r/338980 (https://phabricator.wikimedia.org/T158649) (owner: 10Hashar) [08:44:53] jynus: good morning. I had my coffee and brain is warmed up :} If you are still up for some puppet.git merges I am ready whenever you are. [08:45:17] hashar, give me 10 minutes [08:45:34] jynus: sure thing take all the time you need [08:45:55] (03PS3) 10Ema: cache: allow specifying applayer backend probes and probe piwik [puppet] - 10https://gerrit.wikimedia.org/r/338953 (https://phabricator.wikimedia.org/T154558) [08:48:02] (03PS5) 10Giuseppe Lavagetto: prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 [08:48:09] <_joe_> godog: I am going to merge this change, not sure if it will break things [08:48:22] <_joe_> as we never really tested prometheus::class_config [08:49:06] I might be interested to use it as well for bohrium :D [08:52:41] _joe_: ok, if the catalog compiles then the prometheus config itself isn't going to cause trouble I think [08:53:17] (03CR) 10Volans: "@twentyafterfour" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/338984 (https://phabricator.wikimedia.org/T158660) (owner: 10Volans) [08:53:29] <_joe_> godog: well the function is going to be called from the prometheus hosts [08:53:38] <_joe_> which are uncompilable atm [08:55:08] yep that's what I meant by "if the catalog compiles" [09:00:13] PROBLEM - puppet last run on db1043 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:02:03] RECOVERY - puppet last run on labnodepool1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [09:02:08] !log gehel@puppetmaster1001 conftool action : set/pooled=no; selector: name=elastic10(35|39|43|44).eqiad.wmnet [09:02:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:03:55] (03PS6) 10Giuseppe Lavagetto: prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 [09:04:08] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] prometheus: add etcd metrics [puppet] - 10https://gerrit.wikimedia.org/r/336852 (owner: 10Giuseppe Lavagetto) [09:04:58] !log rebuilding translation memories index - ETA ~4hours (from terbium, logs in ~dcausse/ttm-refresh) [09:05:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:05:11] (03PS2) 10Gehel: elasticsearch - reimage elastic10(35|39|43|44) to jessie and move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/339017 (https://phabricator.wikimedia.org/T151326) [09:06:19] <_joe_> godog: heh, it fails ofc [09:06:29] (03CR) 10Gehel: [C: 032] elasticsearch - reimage elastic10(35|39|43|44) to jessie and move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/339017 (https://phabricator.wikimedia.org/T151326) (owner: 10Gehel) [09:07:40] 06Operations, 10ops-eqiad, 06Discovery, 06Discovery-Search, and 2 others: rack/setup/install elastic1048-1052 - https://phabricator.wikimedia.org/T155790#3045854 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts: ``` ['elastic1043.eqiad.wmnet'] ``` The... [09:08:02] _joe_: of course [09:08:11] 06Operations, 10ops-eqiad, 06Discovery, 06Discovery-Search, and 2 others: rack/setup/install elastic1048-1052 - https://phabricator.wikimedia.org/T155790#3045863 (10ops-monitoring-bot) Script wmf_auto_reimage was launched by gehel on neodymium.eqiad.wmnet for hosts: ``` ['elastic1035.eqiad.wmnet'] ``` The... [09:08:33] PROBLEM - puppet last run on prometheus2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:08:38] <_joe_> godog: with an obscure message, will need to do debugging [09:08:53] <_joe_> I'll take a coffee first [09:09:28] hashar, we can probably go now? [09:09:51] sure thing :} [09:10:14] https://gerrit.wikimedia.org/r/#/c/338143/ is to instruct puppet-syntax to stop trying to puppet parser validate a bunch of stdlib manifests that are puppet 4 only [09:10:39] so merely just an ignore rule, and that is solely when running puppet syntax check locally on all manifests [09:10:45] 06Operations, 10ops-eqiad, 13Patch-For-Review: Suspected faulty SSD on graphite1001 - https://phabricator.wikimedia.org/T157022#3045883 (10fgiunchedi) 05Open>03Resolved Switchback to graphite1001 has been completed, I've updated {T88997} for followup on what services didn't follow the CNAME change correc... [09:11:02] later on I will find out a way to syntax check both with puppet 3.7 and puppet 4.8 but time hasn't come for that yet :) [09:11:05] yeah I'll take a break too [09:11:51] the three other changes are related to Jenkins configuration file. And if we feel brave we can try switching to systemd :} [09:12:03] PROBLEM - puppet last run on prometheus1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:12:33] PROBLEM - puppet last run on mc1031 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:13:02] the prometheus puppet failures are expected btw [09:14:13] (03PS5) 10Jcrespo: syntax: ignore stdlib Puppet 4 manifests [puppet] - 10https://gerrit.wikimedia.org/r/338143 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [09:14:34] (03PS1) 10Gehel: elasticsearch - reimage elastic10(45|46|47) to jessie and move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/339130 (https://phabricator.wikimedia.org/T151326) [09:14:35] !log Run pt-table-checksum on s2.nlwiki over some tables - T154485 [09:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:14:40] T154485: run pt-table-checksum before decommissioning db1015, db1035,db1044,db1038 - https://phabricator.wikimedia.org/T154485 [09:15:03] PROBLEM - DPKG on neodymium is CRITICAL: DPKG CRITICAL dpkg reports broken packages [09:16:03] RECOVERY - DPKG on neodymium is OK: All packages OK [09:18:03] PROBLEM - puppet last run on bast3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:18:30] Fetching source index from https://rubygems.org/... [09:19:33] PROBLEM - puppet last run on prometheus2002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:21:24] (03CR) 10Jcrespo: [C: 032] syntax: ignore stdlib Puppet 4 manifests [puppet] - 10https://gerrit.wikimedia.org/r/338143 (https://phabricator.wikimedia.org/T154894) (owner: 10Hashar) [09:25:04] (03PS7) 10Jcrespo: jenkins: allow access log to be flipped [puppet] - 10https://gerrit.wikimedia.org/r/337385 (owner: 10Hashar) [09:27:13] PROBLEM - puppet last run on prometheus1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:27:47] jynus: I will handle the puppet runs on the two hosts ( contint1001 and contint2001 ) [09:28:05] ok, should we disable them now? [09:28:13] RECOVERY - puppet last run on db1043 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [09:28:30] !log disable puppet on contint1001. Will use contint2001 as a canary [09:28:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:28:35] jynus: done :} [09:28:48] I will merge them separatedly [09:28:52] so if only one fails [09:29:00] we can revert only that one [09:29:09] (03CR) 10Jcrespo: [C: 032] jenkins: allow access log to be flipped [puppet] - 10https://gerrit.wikimedia.org/r/337385 (owner: 10Hashar) [09:29:25] yup seems easier to have a lot of incremental patches [09:29:42] (03PS10) 10Jcrespo: jenkins: allow changing the web service TCP port [puppet] - 10https://gerrit.wikimedia.org/r/337388 (owner: 10Hashar) [09:29:58] I have to praise your work, lot of puppet normally means you are doing your part on the cleaning up [09:30:20] 06Operations, 10ops-eqiad, 06Discovery, 06Discovery-Search, and 2 others: rack/setup/install elastic1048-1052 - https://phabricator.wikimedia.org/T155790#3045971 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['elastic1035.eqiad.wmnet'] ``` Of which those **FAILED**: ``` set(['elastic1035.eqi... [09:30:29] as first goal I want jenkins to be harnessed with systemd [09:30:33] PROBLEM - carbon-local-relay metric drops on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [100.0] [09:30:43] then be able to have multiple jenkins instances in parallel on the same host [09:30:43] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:30:47] (03CR) 10DCausse: WIP - elasticsearch: only send minimal logging to console (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/338998 (https://phabricator.wikimedia.org/T158664) (owner: 10Gehel) [09:31:03] and later starts puppetizing the crap of xml configuration files Jenkins has. But that part is a bit challenging [09:31:32] you can run puppet now [09:31:33] running puppet on contint2001 [09:31:52] will restart the service with the new conf and diff ps -u jenkins f [09:32:34] (03CR) 10Gehel: WIP - elasticsearch: only send minimal logging to console (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/338998 (https://phabricator.wikimedia.org/T158664) (owner: 10Gehel) [09:32:53] pfff [09:33:11]