[00:00:35] Dereckson: Don't forget, that VE patch doesn't do anything without the Flow patches [00:00:43] !log dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/Flow/modules/flow/ui/widgets/editor/editors/mw.flow.ui.VisualEditorWidget.js: Flow Fixes related to Visual Editor (T138356 and T139972) (duration: 00m 45s) [00:00:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:00:49] jinx [00:00:52] Dereckson: good to go [00:00:53] Sorry for being impatient :) [00:00:55] RoanKattouw: here you're [00:01:02] jdlrobson: ok [00:01:03] T138356: Editing comments in flow on mw.org does not work in IE 11 - https://phabricator.wikimedia.org/T138356 [00:01:03] T139972: [Regression wmf.17] "Uncaught TypeError: Cannot read property 'model' of undefined" on clicking 'Edit description' - https://phabricator.wikimedia.org/T139972 [00:01:41] (03CR) 10EBernhardson: Report partial result from mwgrep (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/307652 (https://phabricator.wikimedia.org/T127788) (owner: 10EBernhardson) [00:02:31] !log dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/MobileFrontend/resources/mobile.startup/Skin.js: Ensure lazy image placeholders without height can be loaded (T143768) (duration: 00m 46s) [00:02:32] T143768: Lazy loaded images experiment breaks math images - https://phabricator.wikimedia.org/T143768 [00:02:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:03:31] !log dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/ZeroBanner: Update router code (T143425) (duration: 00m 47s) [00:03:32] T143425: [Spike: 2 hours] Page actions bar not showing on Wikipedia Zero - https://phabricator.wikimedia.org/T143425 [00:03:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:03:49] Okay, wmf17 backports done. [00:03:54] Back to config. [00:06:02] ebernhardson: Disable phrase suggester for wikidata live on mw1099 (and was already there before) [00:06:52] (03PS3) 10Dereckson: End lazy loading reference experiments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307435 (https://phabricator.wikimedia.org/T144240) (owner: 10Jdlrobson) [00:07:48] Dereckson: :S the chrome plugin doesn't seem to work against wikidata.org, i'm getting responses from normal app servers...it's probably fine though [00:07:56] 06Operations: mx1001/2001 - Exim SMTP - Certificate expires Sep 22 2016 - https://phabricator.wikimedia.org/T144568#2603832 (10Dzahn) [00:07:59] ok [00:08:35] 06Operations, 10Mail: mx1001/2001 - Exim SMTP - Certificate expires Sep 22 2016 - https://phabricator.wikimedia.org/T144568#2603845 (10Dzahn) [00:09:09] (03PS2) 10EBernhardson: Report partial result from mwgrep [puppet] - 10https://gerrit.wikimedia.org/r/307652 (https://phabricator.wikimedia.org/T127788) [00:09:31] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Disable phrase suggester for wikidata (T143260, 1/2) (duration: 00m 46s) [00:09:32] T143260: Search Results: turn off 'did you mean' suggestion on Wikidata - https://phabricator.wikimedia.org/T143260 [00:09:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:10:30] !log dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Disable phrase suggester for wikidata (T143260, 2/2) (duration: 00m 46s) [00:10:31] ebernhardson: here you are, can you test? ^ [00:10:31] T143260: Search Results: turn off 'did you mean' suggestion on Wikidata - https://phabricator.wikimedia.org/T143260 [00:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:10:43] Dereckson: it's disabled as expected [00:11:17] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307435 (https://phabricator.wikimedia.org/T144240) (owner: 10Jdlrobson) [00:11:23] ebernhardson: good :) [00:11:42] (03Merged) 10jenkins-bot: End lazy loading reference experiments [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307435 (https://phabricator.wikimedia.org/T144240) (owner: 10Jdlrobson) [00:12:01] jdlrobson: End lazy loading reference experiments live on mw1099 [00:12:15] works Dereckson [00:12:18] thank you! [00:12:30] ack'ed, sending in prod [00:13:09] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: End lazy loading reference experiments (T144240) (duration: 00m 47s) [00:13:10] T144240: Disable lazy loaded references experiment in production - https://phabricator.wikimedia.org/T144240 [00:13:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:15:13] ebernhardson: thanks for the revert change [00:15:40] 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2603881 (10Dzahn) adding people who were on T103577 @Fjalapeno @JMinor @BGerstle-WMF as admins [00:15:51] Last change: 307494 math on wikitech [00:16:00] (03PS4) 10Dereckson: Enable Math on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307494 (https://phabricator.wikimedia.org/T126338) [00:16:14] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307494 (https://phabricator.wikimedia.org/T126338) (owner: 10Dereckson) [00:16:16] (03CR) 10Mobrovac: "Cherry-picked in beta, works like a charm." [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [00:16:42] (03Merged) 10jenkins-bot: Enable Math on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307494 (https://phabricator.wikimedia.org/T126338) (owner: 10Dereckson) [00:17:34] so we can't test that on mw1099, as wikitech is on silver only [00:17:52] hey ebernhardson [00:17:55] 18 Notice: Undefined variable: wmgCirrusSearchEnablePhraseSuggest in /srv/mediawiki/wmf-config/CirrusSearch-common.php on line 476 [00:18:23] strange: [00:18:23] 00:10 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Disable phrase suggester for wikidata (T143260, 2/2) (duration: 00m 46s) [00:18:24] T143260: Search Results: turn off 'did you mean' suggestion on Wikidata - https://phabricator.wikimedia.org/T143260 [00:18:27] 00:09 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Disable phrase suggester for wikidata (T143260, 1/2) (duration: 00m 46s) [00:18:27] T143260: Search Results: turn off 'did you mean' suggestion on Wikidata - https://phabricator.wikimedia.org/T143260 [00:18:38] we sent wmgCirrusSearchEnablePhraseSuggest before the new /srv/mediawiki/wmf-config/CirrusSearch-common.php [00:20:03] 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2603895 (10Dzahn) better source, we have the admin class "piwik-roots". 462 description: users who have root on analytics piwik servers 463 members: [@mforns, @milimetric,@nuria, @ottom... [00:21:13] Dereckson: strange, but seems to be a temporary and minor blip [00:21:29] okay, I sync math wikitech [00:21:58] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Math on wikitech (T126338) (duration: 00m 47s) [00:21:59] T126338: Enable math extension on wikitech - https://phabricator.wikimedia.org/T126338 [00:22:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [00:22:18] ebernhardson: could you check on logstash if it's not all on the same server? [00:22:50] godog: aïe, doesn't work, Failed to parse (Cannot store math image on filesystem.): \varphi(n) =n \prod_{p\mid n} \left(1-\frac{1}{p}\right), [00:23:08] godog: I wonder if we don't need to create manually the directory beforehand [00:23:39] Dereckson: came from a few instances, sec i'm going to update the default logstash dasboard to actually show the hosts involved... [00:23:52] Krenair: awight: ping? [00:23:57] pong [00:24:33] 06Operations, 10Domains, 10Traffic: Guapopedia - https://phabricator.wikimedia.org/T144276#2594645 (10Dzahn) I can't find the term "Guapopedia" at all with major search engines. You would expect it has been at least talked about _somewhere_. But no, all we have is El Guapo, the nickname for former major lea... [00:24:43] Krenair: I merged the Math configuration for Wikitech, but it doesn't seem to wish to save file [00:24:48] Oh. [00:24:54] CommonSettings.php: $wgMathDirectory = '/mnt/upload7/math'; // just for sanity [00:25:01] Yeah... not creating that [00:25:31] could do something like /srv/math-images on labswiki [00:25:48] Dereckson: came from 8 different hosts, so i dunno why :S [00:26:19] Krenair: pick the directory, I'm preparing the IS/CS config [00:26:50] ebernhardson: give me one of the host please [00:27:34] I suggest /srv/math-images [00:27:50] Dereckson: mw1286 had the most errors (8) [00:28:58] 06Operations, 10DNS, 10Traffic: Instead of url forward for wikipedia.in ... add server ip. - https://phabricator.wikimedia.org/T144508#2603910 (10Dzahn) [00:29:43] 06Operations, 10DNS, 10Traffic: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2603913 (10Dzahn) [00:29:48] ebernhardson: it has the right wmf-config/InitialiseSettings.php [00:30:27] so yes, we can consider it's not an issue [00:31:56] Krenair: could you also create it for labtestwiki? [00:32:05] yes [00:32:17] Thanks. [00:32:22] oh, wait [00:32:23] no [00:32:28] needs sudo [00:32:45] mutante: could you assist us please? ^ [00:32:55] I can do it on labtestweb2001 (labtestwiki), but not silver (labswiki) [00:33:15] (I only have root in labtest, not real-labs) [00:33:18] mutante: we would need a /srv/math-images directroy on silver [00:33:43] Krenair: which user should own it? [00:34:11] www-data [00:34:44] maybe it should be done with puppet [00:35:13] 06Operations, 10DNS, 10Traffic: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2602033 (10Dzahn) for context of this request and other ops Currently, in the DNS templates, wikipedia.in is a symlink to wikipedia.or... [00:37:06] Dereckson: no /srv/math-images on silver [00:37:19] (03PS1) 10Dereckson: Customize wgMathDirectory for wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308110 (https://phabricator.wikimedia.org/T126628) [00:39:26] that would be for labs::openstack::nova::manager [00:43:32] chmod 755 ? [00:46:02] (03PS1) 10BryanDavis: google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) [00:46:21] (03PS2) 10Dzahn: point wikipedia.in to 180.179.52.130 [dns] - 10https://gerrit.wikimedia.org/r/307959 (https://phabricator.wikimedia.org/T144508) [00:46:53] (03PS1) 10Dereckson: Create /srv/math-images on wikitech hosts [puppet] - 10https://gerrit.wikimedia.org/r/308112 (https://phabricator.wikimedia.org/T126628) [00:47:14] (03CR) 10jenkins-bot: [V: 04-1] google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [00:49:26] (03PS3) 10Paladox: gerrit: revert CSS change that fixed the diff being cutoff [puppet] - 10https://gerrit.wikimedia.org/r/308095 [00:49:33] (03PS2) 10BryanDavis: google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) [00:50:13] (03CR) 10Dzahn: "re: "can be reverted once upstream gerrit fixes it." but upstream has not fixed it yet it looks. we are reverting it anyways?" [puppet] - 10https://gerrit.wikimedia.org/r/308095 (owner: 10Paladox) [00:51:09] (03CR) 10jenkins-bot: [V: 04-1] google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [00:51:18] now what jerkins? [00:52:11] heh. json !== puppet syntax [00:52:33] (03PS3) 10BryanDavis: google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) [00:52:51] (03CR) 10Dzahn: [C: 032] Create /srv/math-images on wikitech hosts [puppet] - 10https://gerrit.wikimedia.org/r/308112 (https://phabricator.wikimedia.org/T126628) (owner: 10Dereckson) [00:53:42] (03CR) 10jenkins-bot: [V: 04-1] google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) (owner: 10BryanDavis) [00:54:21] 06Operations, 10DNS, 10Traffic, 06WMF-Legal, 13Patch-For-Review: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2603990 (10Peachey88) Probably want #wmf-legal to weigh in on it. But from my personal [non offic... [00:54:22] Dereckson: Notice: /Stage[main]/Role::Labs::Openstack::Nova::Manager/File[/srv/math-images]/ensure: created [00:54:36] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2603993 (10Peachey88) [00:55:01] (03CR) 10Dzahn: "confirmed it didn't exist on silver and puppet created it now, just an empty dir for now" [puppet] - 10https://gerrit.wikimedia.org/r/308112 (https://phabricator.wikimedia.org/T126628) (owner: 10Dereckson) [00:55:05] ok [00:55:34] it's appeared on silver too [00:55:42] puppet run finished [00:56:11] Puppet part done, now config part. [00:56:13] Krenair: https://gerrit.wikimedia.org/r/#/c/308110 looks good to you? [00:57:20] ok, if puppet part is done. i'm out for dinner and today [00:57:43] Dereckson, yep [00:57:45] thanks mutante [00:58:32] (03PS4) 10Paladox: gerrit: revert CSS change that fixed the diff being cutoff [puppet] - 10https://gerrit.wikimedia.org/r/308095 [00:58:37] (03PS5) 10Paladox: gerrit: revert CSS change that fixed the diff being cutoff [puppet] - 10https://gerrit.wikimedia.org/r/308095 [00:58:45] (03CR) 10Paladox: "@Dzahn yes please." [puppet] - 10https://gerrit.wikimedia.org/r/308095 (owner: 10Paladox) [00:58:53] (03PS4) 10BryanDavis: google_api_proxy: Add role for Google-api-proxy Labs project [puppet] - 10https://gerrit.wikimedia.org/r/308111 (https://phabricator.wikimedia.org/T144290) [00:59:01] Thanks mutante for the quick merge and for tin. [00:59:06] Bon appétit. [00:59:28] LOL [00:59:31] french [00:59:47] paladox... [01:00:03] (03CR) 10Dereckson: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308110 (https://phabricator.wikimedia.org/T126628) (owner: 10Dereckson) [01:00:11] Dereckson said Bon appétit. and i was saying it is french. [01:00:25] yes [01:00:32] (03Merged) 10jenkins-bot: Customize wgMathDirectory for wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308110 (https://phabricator.wikimedia.org/T126628) (owner: 10Dereckson) [01:00:43] Ok [01:01:47] mw1099: mwrepl enwiki [01:01:49] echo $wgMathDirectory [01:01:49] /mnt/upload7/math [01:01:49] Dereckson is in a French-speaking part of Belgium [01:02:02] so looks good to me [01:02:06] No need to "LOL" [01:05:38] !log dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Customize wgMathDirectory for wikitech (T126628, 1/2) (duration: 00m 47s) [01:05:39] T126628: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628 [01:05:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:06:35] !log dereckson@tin Synchronized wmf-config/CommonSettings.php: Customize wgMathDirectory for wikitech (T126628, 2/2) (duration: 00m 46s) [01:06:35] T126628: Allow wikitech to write files for Math - https://phabricator.wikimedia.org/T126628 [01:06:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:08:31] Still a Failed to parse (Cannot store math image on filesystem.) on wikitech [01:08:57] Oh ok [01:09:02] https://www.mediawiki.org/wiki/Manual:$wgMathDirectory [01:09:06] This feature was removed from MediaWiki core in version 1.18.0. Please see Extension:Math for an alternative way to use this feature. [01:10:12] If MediaWiki does not have write permissions to $wgUploadDirectory, see config items below $wgMathPath, $wgMathDirectory and $wgMathPath you will have to grant them with e.g. *) [01:11:58] Sorry for lol Kreanir i just thought it was good someone speaking french since it is mostly english. [01:12:22] I only know a few words like the one he said, which means something to do with enjoy your food [01:14:03] So it fails here: [01:14:05] # Create any containers/directories as needed... [01:14:05] if ( !$backend->prepare( [ 'dir' => $hashpath ] )->isOK() ) { [01:14:08] return $this->getError( 'math_output_error' ); [01:14:11] } [01:14:50] maybe the web server can't create directories in there? [01:14:51] let's see... [01:15:50] www-data@silver:/srv/math-images$ mkdir t [01:15:50] www-data@silver:/srv/math-images$ [01:15:51] nope, works [01:19:19] you should see in $tmpDir a subhash dir, with a png file inside [01:19:40] $tmpDir? [01:19:47] $tmpDir = wfTempDir(); [01:20:26] so /tmp [01:21:38] I don't see a subhash dir [01:21:45] I do see a couple of png files under /tmp though [01:24:44] 06Operations, 10Security-Reviews, 06Security-Team, 10Wikimedia-Site-requests: ACL configuration for url-downloader.wikimedia.org allowing upload.wikimedia.org - https://phabricator.wikimedia.org/T130695#2143360 (10Cwek) >>! In T130695#2587826, @TTO wrote: > Works for me: https://test.wikipedia.org/wiki/Fil... [01:25:18] hmmmm I'm looking the getBackend method of MathTexvc: if wgMathFileBackend is falsy, it creates a FSFileBackend called "math-backend" using wgMathDirectory [01:27:14] so perhaps localbut if a value is given, it doesn't uses wgMathDirectory: return FileBackendGroup::singleton()->get( $wgMathFileBackend ); [01:27:28] (-"so perhaps local") [01:28:28] so we want to set $wgMathFileBackend = false on wikitech? [01:28:56] yes, wgMathFileBackend should probably be false if we wish to use the $wgMathDirectory directory, and it's currently probably trying to store file in another directory. [01:29:25] https://github.com/wikimedia/mediawiki-extensions-Math/blob/master/MathTexvc.php#L318 [01:37:23] (03PS1) 10Dereckson: Set wgMathFileBackend to false for wikitech wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308117 (https://phabricator.wikimedia.org/T126628) [01:38:29] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2602033 (10Platonides) We (WM-ES) have requested the same thing for wikipedia.es a long time ago. It should... [01:43:20] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Instead of url forward for wikipedia.in ... add server ip. (point wikipedia.in to 180.179.52.130) - https://phabricator.wikimedia.org/T144508#2602033 (10BBlack) >>! In T144508#2604046, @Platonides wrote: > Security wise, these domains are not a probl... [02:38:35] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 12s) [02:38:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:43:50] !log l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 2 02:43:50 UTC 2016 (duration 5m 15s) [02:43:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:55:23] (03PS13) 1020after4: WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [04:15:54] RECOVERY - Host mw2190 is UP: PING OK - Packet loss = 0%, RTA = 37.09 ms [04:16:54] RECOVERY - Host mw2191 is UP: PING OK - Packet loss = 0%, RTA = 36.19 ms [04:35:31] !log tin - re-enabled puppet [04:35:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:40:28] (03CR) 10Dzahn: [C: 032] "per Roan "This change caused bugs with contextual commenting"" [puppet] - 10https://gerrit.wikimedia.org/r/308095 (owner: 10Paladox) [04:49:16] (03PS14) 1020after4: WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [04:54:42] 06Operations, 06WMF-NDA-Requests: Phabricator access for Manuel Arostegui to WMF-NDA and Ops-only procurement and other tasks - https://phabricator.wikimedia.org/T144496#2604185 (10jcrespo) @Aklapper Sorry, I searched from something like #phabricator-access-requests and couldn't find anything there. Thank you... [05:07:27] (03CR) 10Greg Grossmeier: [C: 031] Add a "now" command [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/308086 (owner: 10BryanDavis) [05:07:40] (03CR) 10Greg Grossmeier: [C: 031] Use normal messages rather than notices for help [wikimedia/bots/jouncebot] - 10https://gerrit.wikimedia.org/r/308087 (owner: 10BryanDavis) [06:27:30] (03PS2) 10Muehlenhoff: Remove decommissioned host rubidium from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/307917 (https://phabricator.wikimedia.org/T118213) [06:29:01] (03CR) 10Muehlenhoff: [C: 032] Remove decommissioned host rubidium from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/307917 (https://phabricator.wikimedia.org/T118213) (owner: 10Muehlenhoff) [06:30:50] (03PS1) 10Dzahn: delete misc::monitoring::view::analytics::data [puppet] - 10https://gerrit.wikimedia.org/r/308128 [06:32:41] (03CR) 10Alexandros Kosiaris: [C: 031] delete misc::monitoring::view::analytics::data [puppet] - 10https://gerrit.wikimedia.org/r/308128 (owner: 10Dzahn) [06:36:26] !log reimage mw2149-mw2151 to jessie [06:36:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [06:39:04] 06Operations, 10ops-codfw: Broken disk on wtp2016 - https://phabricator.wikimedia.org/T144260#2604262 (10akosiaris) After a powercycle and while logged in in the DRAC I get a: ``` No operating system is currently installed on this computer. ``` I 'll reimage it.. something went wrong with the disk change [06:40:07] !log reimage mw2101 to jessie [06:40:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [06:40:49] (03PS1) 10Dzahn: openstack::nova::instancersync fix class name [puppet] - 10https://gerrit.wikimedia.org/r/308130 [06:41:47] (03PS2) 10Dzahn: openstack::nova::instancersync fix define name [puppet] - 10https://gerrit.wikimedia.org/r/308130 [06:48:02] (03PS1) 10Dzahn: labstore: move resource classes into subdir, autolayout [puppet] - 10https://gerrit.wikimedia.org/r/308131 [06:48:52] (03PS2) 10Dzahn: labstore: move resource defines into subdir, layout warnings [puppet] - 10https://gerrit.wikimedia.org/r/308131 [06:55:13] 06Operations, 06WMF-NDA-Requests: Phabricator access for Manuel Arostegui to WMF-NDA and Ops-only procurement and other tasks - https://phabricator.wikimedia.org/T144496#2604269 (10Aklapper) No worries! :) And thanks also to RobH for quick handling. [07:01:33] (03PS1) 10Muehlenhoff: Provide systemd units for keyholder-agent and keyholder-proxy [puppet] - 10https://gerrit.wikimedia.org/r/308132 (https://phabricator.wikimedia.org/T144043) [07:03:47] PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 1 failures [07:09:16] RECOVERY - Host wtp2016 is UP: PING OK - Packet loss = 0%, RTA = 36.08 ms [07:14:35] PROBLEM - configured eth on wtp2016 is CRITICAL: Connection refused by host [07:14:52] PROBLEM - dhclient process on wtp2016 is CRITICAL: Connection refused by host [07:14:52] PROBLEM - Check size of conntrack table on wtp2016 is CRITICAL: Connection refused by host [07:14:52] PROBLEM - DPKG on wtp2016 is CRITICAL: Connection refused by host [07:15:07] PROBLEM - puppet last run on wtp2016 is CRITICAL: Connection refused by host [07:15:26] PROBLEM - salt-minion processes on wtp2016 is CRITICAL: Connection refused by host [07:15:28] PROBLEM - Disk space on wtp2016 is CRITICAL: Connection refused by host [07:16:32] 06Operations, 06WMF-NDA-Requests: Phabricator access for Manuel Arostegui to WMF-NDA and Ops-only procurement and other tasks - https://phabricator.wikimedia.org/T144496#2604288 (10Marostegui) Thanks everyone! [07:16:39] PROBLEM - parsoid on wtp2016 is CRITICAL: Connection refused [07:20:45] 06Operations: Blocked /etc/passwd on sca100[1234] hosts - https://phabricator.wikimedia.org/T144492#2604291 (10MoritzMuehlenhoff) The locking limitation is fixed in Linux 3.18: https://firejail.wordpress.com/support/known-problems/#removeblacklisted The sca cluster is currently the only cluster with long-runnin... [07:26:18] RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:30:07] PROBLEM - Host wtp2016 is DOWN: PING CRITICAL - Packet loss = 100% [07:32:58] 06Operations: Blocked /etc/passwd on sca100[1234] hosts - https://phabricator.wikimedia.org/T144492#2604300 (10akosiaris) None on my part [07:42:41] 06Operations, 07HHVM: Migrate deployment servers (tin/mira) to jessie - https://phabricator.wikimedia.org/T144578#2604309 (10MoritzMuehlenhoff) [07:51:21] PROBLEM - puppet last run on mw2134 is CRITICAL: CRITICAL: Puppet has 1 failures [07:56:04] (03PS9) 10Addshore: Add simple-json-datasource plugin to labs grafana [puppet] - 10https://gerrit.wikimedia.org/r/302119 (https://phabricator.wikimedia.org/T141636) [07:56:14] yuvipanda: regarding that one ^^ im back from vacation now :) [07:56:23] (03CR) 10Alexandros Kosiaris: "pedantic comment inline, otherwise LGTM" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308132 (https://phabricator.wikimedia.org/T144043) (owner: 10Muehlenhoff) [08:04:52] !log upgrading httpd on mw1290 to 2.4.10-10+deb8u6+wmf2 [08:04:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:07:28] (03Abandoned) 10Addshore: Enable mention status notifications on dewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304607 (https://phabricator.wikimedia.org/T143100) (owner: 10Addshore) [08:07:35] (03Abandoned) 10Addshore: Enable mention status notifications everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304608 (https://phabricator.wikimedia.org/T143101) (owner: 10Addshore) [08:08:18] (03Restored) 10Addshore: Enable mention status notifications everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304608 (https://phabricator.wikimedia.org/T143101) (owner: 10Addshore) [08:08:29] PROBLEM - parsoid on wtp2016 is CRITICAL: Connection refused [08:08:52] PROBLEM - puppet last run on wtp2016 is CRITICAL: CRITICAL: puppet fail [08:09:01] (03PS5) 10Addshore: Enable mention status notifications everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304608 (https://phabricator.wikimedia.org/T143101) [08:09:24] (03PS6) 10Addshore: Enable mention status notifications everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304608 (https://phabricator.wikimedia.org/T143101) [08:14:48] (03PS2) 10Addshore: Enable RevisionSlider BetaFeature on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/305653 (https://phabricator.wikimedia.org/T143421) [08:15:10] (03PS3) 10Addshore: Enable RevisionSlider BetaFeature on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/305653 (https://phabricator.wikimedia.org/T143421) [08:15:16] (03CR) 10Addshore: Enable RevisionSlider BetaFeature on all wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/305653 (https://phabricator.wikimedia.org/T143421) (owner: 10Addshore) [08:16:01] RECOVERY - parsoid on wtp2016 is OK: HTTP OK: HTTP/1.1 200 OK - 1514 bytes in 0.162 second response time [08:16:19] RECOVERY - puppet last run on wtp2016 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [08:18:48] RECOVERY - puppet last run on mw2134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:19:32] !log upgrading httpd on mw128[0124] to 2.4.10-10+deb8u6+wmf2 [08:19:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:19:51] 9 eqiad api servers to go [08:27:05] 06Operations, 06Discovery, 06Discovery-Search, 10Elasticsearch, 10Wikimedia-Logstash: Disable cron job to clear elasticsearch caches and validate that it does not have significant impact on GC - https://phabricator.wikimedia.org/T144396#2604386 (10Gehel) So far GC seems well under control and there is pl... [08:32:03] (03PS7) 10Gehel: maps - use new tileshell.js script to notify tilerator of expired tiles [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) [08:32:27] (03CR) 10Gehel: maps - use new tileshell.js script to notify tilerator of expired tiles (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) (owner: 10Gehel) [08:34:32] !log upgrading httpd on mw1289 to 2.4.10-10+deb8u6+wmf2 [08:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:38:35] !log reimaging mw2099, mw2117, mw2163, mw2164 to jessie [08:38:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:38:42] PROBLEM - puppet last run on mw1300 is CRITICAL: CRITICAL: Puppet last ran 19 hours ago [08:39:08] this was my fault --^ [08:39:12] just ran puppet [08:39:37] (03PS3) 10Gehel: wdqs - move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/308023 (https://phabricator.wikimedia.org/T144536) [08:41:18] RECOVERY - puppet last run on mw1300 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:43:58] PROBLEM - HHVM rendering on mw2117 is CRITICAL: Connection refused [08:44:41] PROBLEM - Apache HTTP on mw2117 is CRITICAL: Connection refused [08:45:13] (03PS4) 10Gehel: wdqs - move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/308023 (https://phabricator.wikimedia.org/T144536) [08:46:09] PROBLEM - Check size of conntrack table on mw2117 is CRITICAL: Connection refused by host [08:46:38] PROBLEM - nutcracker port on mw2117 is CRITICAL: Connection refused by host [08:46:40] PROBLEM - nutcracker process on mw2117 is CRITICAL: Connection refused by host [08:47:15] ^mw2117 is missing downtime, fixed [08:48:50] (03CR) 10Alexandros Kosiaris: [C: 04-1] "minor comments inline, otherwise LGTM" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) (owner: 10Gehel) [08:49:58] (03CR) 10Muehlenhoff: Provide systemd units for keyholder-agent and keyholder-proxy (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308132 (https://phabricator.wikimedia.org/T144043) (owner: 10Muehlenhoff) [08:55:46] (03PS2) 10Muehlenhoff: Provide systemd units for keyholder-agent and keyholder-proxy [puppet] - 10https://gerrit.wikimedia.org/r/308132 (https://phabricator.wikimedia.org/T144043) [08:57:04] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2604459 (10Aklapper) [08:57:55] (03PS8) 10Gehel: maps - use new tileshell.js script to notify tilerator of expired tiles [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) [09:03:06] !log reboot cp4006 for kernel upgrade [09:03:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:07:23] (03PS1) 10Elukey: Fix Apache cronspam from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/308142 (https://phabricator.wikimedia.org/T139943) [09:09:25] PROBLEM - mailman_queue_size on fermium is CRITICAL: CRITICAL: 1 mailman queue(s) above limits (thresholds: bounces: 25 in: 25 virgin: 25) [09:13:49] !log upgrading httpd on mw127[6789] to 2.4.10-10+deb8u6+wmf2 (eqiad api canaries) [09:13:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:14:26] RECOVERY - mailman_queue_size on fermium is OK: OK: mailman queues are below the limits. [09:25:08] (03CR) 10Alexandros Kosiaris: [C: 04-1] Fix Apache cronspam from netmon1001 (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308142 (https://phabricator.wikimedia.org/T139943) (owner: 10Elukey) [09:26:37] (03PS7) 10Gehel: Discovery stats module [puppet] - 10https://gerrit.wikimedia.org/r/305673 (https://phabricator.wikimedia.org/T143048) (owner: 10MaxSem) [09:26:43] (03CR) 10Alexandros Kosiaris: [C: 031] maps - use new tileshell.js script to notify tilerator of expired tiles [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) (owner: 10Gehel) [09:26:57] akosiaris: ack, makes sense :) [09:26:58] akosiaris: thanks for all the reviews! [09:27:10] gehel: you are welcome [09:29:22] (03CR) 10Gehel: [C: 032] Discovery stats module [puppet] - 10https://gerrit.wikimedia.org/r/305673 (https://phabricator.wikimedia.org/T143048) (owner: 10MaxSem) [09:30:31] (03PS2) 10Elukey: Fix Apache cronspam from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/308142 (https://phabricator.wikimedia.org/T139943) [09:31:27] conftool-merge just gave me a bunch of "codfw: Creating node mw2186.codfw.wmnet for cluster appserver/apache2" after a puppet merge [09:31:47] is that expected? My change had nothing to do with mediawiki... [09:31:50] yeah expected, these are reimaged hosts [09:32:06] double checking but I am almost sure [09:32:15] elukey: thanks! I suspected something like that, but better safe thansorry :P [09:32:52] https://etherpad.wikimedia.org/p/trusty-mw-reimage - I wouldn't have expected them to get cleared from conftool, but probably it is a consequence of puppet clean + first puppet run [09:33:02] elukey: https://phabricator.wikimedia.org/P3963 for the whole list... [09:34:04] so theoretically these warnings are about creating nodes to etcd eventually [09:34:22] but it is weird that a reimage triggers this [09:34:30] PROBLEM - salt-minion processes on mw2149 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [09:34:56] I bet that puppet clean is the responsible [09:35:13] the only thing that needs to be fixed is the weight [09:35:14] that's not because they're depooled, but due to Daniel removing them from conftool-data yesterday [09:35:27] that's not because they were reimaged, but due to Daniel removing them from conftool-data yesterday [09:35:29] ahhhh good! [09:35:32] this does make sense [09:35:42] PROBLEM - salt-minion processes on mw2117 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [09:35:43] gehel: don't listen to me, I keep telling you that [09:35:51] thanks moritzm [09:35:52] :) [09:35:57] moritzm: what was the reason to remove them? Is it ok to have them re-added? [09:36:01] PROBLEM - salt-minion processes on mw2164 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [09:36:18] elukey: I'll keep listening! But I might check what you are saying... [09:36:22] ahhaha [09:36:35] so we are re-imaging these hosts and most of them had an IPMI issue [09:36:41] PROBLEM - salt-minion processes on mw2163 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [09:36:45] so I guess that Daniel removed them for this reason [09:38:06] I see the log in SAL, but not much context... [09:38:08] the hosts were shut down since Papaul needed to fix hw settings and that slowed down a deployment window since scap tried to access the downed hosts [09:38:15] 06Operations, 10Traffic, 13Patch-For-Review: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502#2604493 (10ema) Due to multiple issues with Varnish 4 including T144257 and recurring 503 plateaus, we've decided to downgrade most of cache_upload in ulsfo to Varnish 3. We are going... [09:38:38] ok, so now that deployment is passed, it is not an issue to have them re-added. [09:38:49] he re-added them in a subsequent commit [09:38:56] (or rather reverted the old one) [09:39:32] I am a bit puzzled since I thought that now conftool was the souce of truth for scap dsh [09:40:04] it is, that's why I don't see the need for dropping them temporarily either [09:41:01] IIRC we'd need only to set them as inactive via conftool (awesome!) [09:41:19] will ask to daniel [09:42:22] RECOVERY - salt-minion processes on mw2149 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [09:42:58] elukey, moritzm: thanks for the explanation! [09:43:03] and even if scap runs into some timeout, it doesn't really matter since codfw doesn't serve traffic anyway... [09:48:42] !log Raised OSPF metrics on cr2-ulsfo<-->cr1-codfw link from 388 to 1000 in both directions [09:48:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:56:41] RECOVERY - salt-minion processes on mw2117 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [09:57:52] PROBLEM - puppet last run on elastic2020 is CRITICAL: CRITICAL: puppet fail [09:58:32] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 2 failures [10:02:40] RECOVERY - salt-minion processes on mw2164 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [10:03:11] RECOVERY - salt-minion processes on mw2163 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [10:10:23] !log reimaging mw216[5-8] to jessie (IPMI fixed) [10:10:28] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:10:34] (03CR) 10Alexandros Kosiaris: [C: 032] Fix Apache cronspam from netmon1001 [puppet] - 10https://gerrit.wikimedia.org/r/308142 (https://phabricator.wikimedia.org/T139943) (owner: 10Elukey) [10:17:41] !log reimaging mw2180-mw2183 to jessie [10:17:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [10:19:36] akosiaris: ran puppet on netmon1001, all good [10:21:43] 06Operations, 13Patch-For-Review: Tracking and Reducing cron-spam from root@ - https://phabricator.wikimedia.org/T132324#2604539 (10elukey) [10:21:45] 06Operations, 13Patch-For-Review: netmon1001 daily logrotate cronspam - https://phabricator.wikimedia.org/T139943#2604538 (10elukey) 05Open>03Resolved [10:22:18] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2602033 (10grin) >>! In T144508#2604050, @BBlack wrote: > I disagree. In general, we can't expect volunteer/chapter portals hosted elsewhere to... [10:24:14] RECOVERY - puppet last run on elastic2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:26:33] RECOVERY - puppet last run on ms-be1022 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [10:51:07] (03PS1) 10Muehlenhoff: Show latest 4.4.x release version in uname [debs/linux44] - 10https://gerrit.wikimedia.org/r/308151 [10:58:41] (03PS1) 10Hashar: aptly: move role to modules [puppet] - 10https://gerrit.wikimedia.org/r/308152 [11:03:34] PROBLEM - Apache HTTP on mw2181 is CRITICAL: Connection timed out [11:03:57] PROBLEM - Apache HTTP on mw2180 is CRITICAL: Connection timed out [11:06:00] PROBLEM - DPKG on mw2183 is CRITICAL: Timeout while attempting connection [11:06:00] PROBLEM - Check size of conntrack table on mw2182 is CRITICAL: Timeout while attempting connection [11:06:28] PROBLEM - Check size of conntrack table on mw2181 is CRITICAL: Timeout while attempting connection [11:06:28] PROBLEM - DPKG on mw2182 is CRITICAL: Timeout while attempting connection [11:06:28] PROBLEM - Disk space on mw2183 is CRITICAL: Timeout while attempting connection [11:06:57] PROBLEM - DPKG on mw2181 is CRITICAL: Timeout while attempting connection [11:06:57] PROBLEM - Check size of conntrack table on mw2180 is CRITICAL: Timeout while attempting connection [11:06:57] PROBLEM - Disk space on mw2182 is CRITICAL: Timeout while attempting connection [11:07:28] PROBLEM - Disk space on mw2181 is CRITICAL: Timeout while attempting connection [11:07:28] PROBLEM - DPKG on mw2180 is CRITICAL: Timeout while attempting connection [11:07:30] PROBLEM - Apache HTTP on mw2183 is CRITICAL: Connection timed out [11:07:38] PROBLEM - configured eth on mw2183 is CRITICAL: Timeout while attempting connection [11:07:38] PROBLEM - Disk space on mw2180 is CRITICAL: Timeout while attempting connection [11:07:48] PROBLEM - Apache HTTP on mw2182 is CRITICAL: Connection timed out [11:07:58] PROBLEM - configured eth on mw2182 is CRITICAL: Timeout while attempting connection [11:07:58] PROBLEM - dhclient process on mw2183 is CRITICAL: Timeout while attempting connection [11:08:18] PROBLEM - dhclient process on mw2182 is CRITICAL: Timeout while attempting connection [11:08:18] PROBLEM - configured eth on mw2181 is CRITICAL: Timeout while attempting connection [11:08:28] elukey, moritzm: something wrong with the reimaging? ^ [11:08:37] PROBLEM - configured eth on mw2180 is CRITICAL: Timeout while attempting connection [11:08:37] PROBLEM - dhclient process on mw2181 is CRITICAL: Timeout while attempting connection [11:08:37] PROBLEM - nutcracker port on mw2183 is CRITICAL: Timeout while attempting connection [11:08:50] PROBLEM - dhclient process on mw2180 is CRITICAL: Timeout while attempting connection [11:08:50] PROBLEM - nutcracker process on mw2183 is CRITICAL: Timeout while attempting connection [11:08:50] PROBLEM - nutcracker port on mw2182 is CRITICAL: Timeout while attempting connection [11:09:09] ema: no, that's ok, I'll silence: [11:09:17] PROBLEM - nutcracker process on mw2182 is CRITICAL: Timeout while attempting connection [11:09:17] PROBLEM - puppet last run on mw2183 is CRITICAL: Timeout while attempting connection [11:09:17] PROBLEM - nutcracker port on mw2181 is CRITICAL: Timeout while attempting connection [11:09:19] PROBLEM - salt-minion processes on mw2168 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [11:09:27] PROBLEM - nutcracker process on mw2181 is CRITICAL: Connection refused by host [11:09:29] PROBLEM - nutcracker port on mw2180 is CRITICAL: Connection refused by host [11:09:31] PROBLEM - puppet last run on mw2182 is CRITICAL: Timeout while attempting connection [11:09:31] PROBLEM - salt-minion processes on mw2183 is CRITICAL: Timeout while attempting connection [11:09:38] PROBLEM - salt-minion processes on mw2167 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [11:09:38] PROBLEM - nutcracker process on mw2180 is CRITICAL: Connection refused by host [11:09:38] PROBLEM - puppet last run on mw2181 is CRITICAL: Connection refused by host [11:09:48] PROBLEM - salt-minion processes on mw2182 is CRITICAL: Timeout while attempting connection [11:09:58] RECOVERY - Disk space on mw2181 is OK: DISK OK [11:09:58] PROBLEM - salt-minion processes on mw2166 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [11:09:59] RECOVERY - Apache HTTP on mw2183 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.075 second response time [11:10:09] RECOVERY - configured eth on mw2183 is OK: OK - interfaces up [11:10:09] RECOVERY - Disk space on mw2180 is OK: DISK OK [11:10:18] RECOVERY - Apache HTTP on mw2182 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.075 second response time [11:10:30] RECOVERY - configured eth on mw2182 is OK: OK - interfaces up [11:10:30] RECOVERY - dhclient process on mw2183 is OK: PROCS OK: 0 processes with command name dhclient [11:10:49] RECOVERY - dhclient process on mw2182 is OK: PROCS OK: 0 processes with command name dhclient [11:10:49] RECOVERY - configured eth on mw2181 is OK: OK - interfaces up [11:10:57] during the reimaging, the old host entry is removed (and the downtime set for those hosts is deleted along), so it needs to be re-added during a fairly brief window [11:10:58] RECOVERY - Apache HTTP on mw2181 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.074 second response time [11:10:58] RECOVERY - configured eth on mw2180 is OK: OK - interfaces up [11:10:58] RECOVERY - DPKG on mw2183 is OK: All packages OK [11:10:58] RECOVERY - nutcracker port on mw2183 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [11:10:58] RECOVERY - dhclient process on mw2181 is OK: PROCS OK: 0 processes with command name dhclient [11:10:59] RECOVERY - Check size of conntrack table on mw2182 is OK: OK: nf_conntrack is 0 % full [11:11:20] RECOVERY - nutcracker port on mw2182 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [11:11:20] RECOVERY - dhclient process on mw2180 is OK: PROCS OK: 0 processes with command name dhclient [11:11:20] RECOVERY - nutcracker process on mw2183 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name nutcracker [11:11:29] RECOVERY - Apache HTTP on mw2180 is OK: HTTP OK: HTTP/1.1 200 OK - 10975 bytes in 0.074 second response time [11:11:29] RECOVERY - Check size of conntrack table on mw2181 is OK: OK: nf_conntrack is 0 % full [11:11:29] RECOVERY - Disk space on mw2183 is OK: DISK OK [11:11:29] RECOVERY - DPKG on mw2182 is OK: All packages OK [11:11:40] RECOVERY - nutcracker port on mw2181 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [11:11:40] RECOVERY - nutcracker process on mw2182 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name nutcracker [11:11:59] RECOVERY - Check size of conntrack table on mw2180 is OK: OK: nf_conntrack is 0 % full [11:11:59] RECOVERY - DPKG on mw2181 is OK: All packages OK [11:11:59] RECOVERY - Disk space on mw2182 is OK: DISK OK [11:11:59] RECOVERY - nutcracker process on mw2181 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name nutcracker [11:12:00] RECOVERY - nutcracker port on mw2180 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 11212 [11:12:01] RECOVERY - salt-minion processes on mw2183 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [11:12:18] RECOVERY - nutcracker process on mw2180 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name nutcracker [11:12:19] RECOVERY - salt-minion processes on mw2182 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [11:12:29] RECOVERY - DPKG on mw2180 is OK: All packages OK [11:13:01] (03CR) 10Alexandros Kosiaris: [C: 032] aptly: move role to modules [puppet] - 10https://gerrit.wikimedia.org/r/308152 (owner: 10Hashar) [11:13:17] ;D [11:14:47] PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Puppet has 8 failures [11:16:57] PROBLEM - puppet last run on mw2168 is CRITICAL: CRITICAL: Puppet has 8 failures [11:17:18] PROBLEM - puppet last run on mw2167 is CRITICAL: CRITICAL: Puppet has 8 failures [11:24:14] (03CR) 10Alexandros Kosiaris: [C: 04-1] contint: tidy Nodepool slaves config history (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/300092 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [11:25:38] PROBLEM - Apache HTTP on mw2168 is CRITICAL: Connection refused [11:27:18] PROBLEM - HP RAID on ms-be1022 is CRITICAL: CHECK_NRPE: Socket timeout after 40 seconds. [11:27:30] ms-be1022 is me [11:29:46] 06Operations: Blocked /etc/passwd on sca100[1234] hosts - https://phabricator.wikimedia.org/T144492#2604583 (10mobrovac) +1. Thank you Moritz! [11:30:10] RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [11:30:48] RECOVERY - puppet last run on mw2181 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [11:31:27] PROBLEM - Apache HTTP on mw2167 is CRITICAL: Connection refused [11:32:27] RECOVERY - HP RAID on ms-be1022 is OK: OK: Slot 3: OK: 2I:4:2, 2I:4:1, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:1, 1I:1:2, 1I:1:3, 1I:1:4, 2I:2:1, 2I:2:2, 2I:2:3, 2I:2:4, Controller, Battery/Capacitor [11:33:08] RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [11:34:37] (03PS1) 10KartikMistry: apertium-ita: Initial Debian packaging [debs/contenttranslation/apertium-ita] - 10https://gerrit.wikimedia.org/r/308153 (https://phabricator.wikimedia.org/T144174) [11:34:41] sorry people my bad [11:34:41] PROBLEM - Apache HTTP on mw2166 is CRITICAL: Connection refused [11:35:00] I thought to be back in time to avoid this mess [11:35:42] usually I try to schedule downtime in icinga as soon as I see the first errors popping up [11:35:46] :/ [11:37:08] !log upgrading mw1283.eqiad.wmnet to the latest httpd version [11:37:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [11:39:20] (03PS1) 10KartikMistry: apertium-srd: Initial Debian packaging [debs/contenttranslation/apertium-srd] - 10https://gerrit.wikimedia.org/r/308154 (https://phabricator.wikimedia.org/T144174) [11:41:08] (03PS1) 10Muehlenhoff: rcstream_redis: Limit to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308155 [11:45:20] (03CR) 10Alexandros Kosiaris: [C: 031] rcstream_redis: Limit to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308155 (owner: 10Muehlenhoff) [11:46:45] (03PS3) 10Alexandros Kosiaris: ganglia: Make the ferm statements conditional [puppet] - 10https://gerrit.wikimedia.org/r/307303 [11:46:50] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] ganglia: Make the ferm statements conditional [puppet] - 10https://gerrit.wikimedia.org/r/307303 (owner: 10Alexandros Kosiaris) [11:48:05] (03PS3) 10Alexandros Kosiaris: ganglia: Define install2001 by FQDN, not IP [puppet] - 10https://gerrit.wikimedia.org/r/307305 [11:48:22] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] ganglia: Define install2001 by FQDN, not IP [puppet] - 10https://gerrit.wikimedia.org/r/307305 (owner: 10Alexandros Kosiaris) [11:48:38] (03PS3) 10Alexandros Kosiaris: ganglia: Remove nickel remnant [puppet] - 10https://gerrit.wikimedia.org/r/307306 [11:48:44] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] ganglia: Remove nickel remnant [puppet] - 10https://gerrit.wikimedia.org/r/307306 (owner: 10Alexandros Kosiaris) [11:48:48] RECOVERY - salt-minion processes on mw2168 is OK: PROCS OK: 2 processes with regex args ^/usr/bin/python /usr/bin/salt-minion [11:49:01] RECOVERY - salt-minion processes on mw2167 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [11:49:16] (03PS1) 10Ema: Revert "depool ulsfo, some kind of issues..." [dns] - 10https://gerrit.wikimedia.org/r/308156 [11:50:25] (03PS4) 10Alexandros Kosiaris: check_puppetrun: Improve full fail error message [puppet] - 10https://gerrit.wikimedia.org/r/305504 [11:50:35] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] check_puppetrun: Improve full fail error message [puppet] - 10https://gerrit.wikimedia.org/r/305504 (owner: 10Alexandros Kosiaris) [11:51:26] (03PS4) 10Alexandros Kosiaris: check_puppetrun: extra time to recently enabled hosts [puppet] - 10https://gerrit.wikimedia.org/r/305630 (https://phabricator.wikimedia.org/T143099) [11:51:31] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] check_puppetrun: extra time to recently enabled hosts [puppet] - 10https://gerrit.wikimedia.org/r/305630 (https://phabricator.wikimedia.org/T143099) (owner: 10Alexandros Kosiaris) [11:51:58] RECOVERY - salt-minion processes on mw2166 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [11:52:52] RECOVERY - Apache HTTP on mw2166 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.728 second response time [11:53:12] 06Operations, 10ops-codfw: Broken disk on wtp2016 - https://phabricator.wikimedia.org/T144260#2604622 (10akosiaris) 05Open>03Resolved Reimaged and now ok [11:53:58] RECOVERY - puppet last run on mw2167 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [11:54:14] let's see if our puppet check are now even a tad better [11:54:17] RECOVERY - puppet last run on mw2166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:54:38] RECOVERY - Apache HTTP on mw2168 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 614 bytes in 9.411 second response time [11:54:50] RECOVERY - Apache HTTP on mw2167 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.177 second response time [11:56:13] akosiaris: \o/ [11:59:45] (03PS3) 10Alexandros Kosiaris: ganglia: Remove the Corp OIT LDAP mirror cluster [puppet] - 10https://gerrit.wikimedia.org/r/307304 [12:00:45] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2604626 (10BBlack) >>! In T144508#2604540, @grin wrote: >>>! In T144508#2604050, @BBlack wrote: > >> I disagree. In general, we can't expect vol... [12:08:57] (03CR) 10Ema: [C: 032] Revert "depool ulsfo, some kind of issues..." [dns] - 10https://gerrit.wikimedia.org/r/308156 (owner: 10Ema) [12:09:08] !log pooling ulsfo [12:09:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:15:33] (03CR) 10Alexandros Kosiaris: [C: 032] ganglia: Remove the Corp OIT LDAP mirror cluster [puppet] - 10https://gerrit.wikimedia.org/r/307304 (owner: 10Alexandros Kosiaris) [12:24:06] ACKNOWLEDGEMENT - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 2 failures Gehel Probabls related to T143048 - checking - gehel [12:35:12] (03PS1) 10Gehel: discovery-stats - git::clone needs to be in an empty directory [puppet] - 10https://gerrit.wikimedia.org/r/308157 (https://phabricator.wikimedia.org/T143048) [12:38:29] (03CR) 10Gehel: [C: 032] discovery-stats - git::clone needs to be in an empty directory [puppet] - 10https://gerrit.wikimedia.org/r/308157 (https://phabricator.wikimedia.org/T143048) (owner: 10Gehel) [12:42:47] (03PS1) 10Gehel: discovery-analytics: fixing logrotate rule name [puppet] - 10https://gerrit.wikimedia.org/r/308158 (https://phabricator.wikimedia.org/T143048) [12:44:06] (03CR) 10Gehel: [C: 032] discovery-analytics: fixing logrotate rule name [puppet] - 10https://gerrit.wikimedia.org/r/308158 (https://phabricator.wikimedia.org/T143048) (owner: 10Gehel) [12:44:33] 06Operations, 10ArticlePlaceholder, 10Traffic, 10Wikidata: Performance and caching considerations for article placeholders accesses - https://phabricator.wikimedia.org/T142944#2604711 (10hoo) I've created {T144592} for a trial of submitting placeholders to search engines. Based on that, we can (probably) d... [12:47:23] (03PS1) 10Gehel: discovery-stats: fix script name [puppet] - 10https://gerrit.wikimedia.org/r/308159 (https://phabricator.wikimedia.org/T143048) [12:48:52] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:49:36] (03CR) 10Gehel: [C: 032] discovery-stats: fix script name [puppet] - 10https://gerrit.wikimedia.org/r/308159 (https://phabricator.wikimedia.org/T143048) (owner: 10Gehel) [12:51:36] 06Operations: Blocked /etc/passwd on sca100[1234] hosts - https://phabricator.wikimedia.org/T144492#2604716 (10MoritzMuehlenhoff) sca in codfw has been upgraded to the 4.4 kernel series, seems all fine by now. I'll upgrade eqiad on Monday. [12:52:41] RECOVERY - puppet last run on sca2002 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [12:53:13] thanks gehel for stat1002! [12:55:13] (03PS1) 10Ottomata: Install mc and unzip on stat1001 as per request from ezachte [puppet] - 10https://gerrit.wikimedia.org/r/308160 [12:55:38] elukey: i'm the one who broke it :( [12:55:58] well and then you fixed it, thanks :) [12:57:49] (03CR) 10DCausse: [C: 031] Report partial result from mwgrep (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/307652 (https://phabricator.wikimedia.org/T127788) (owner: 10EBernhardson) [12:59:48] elukey: you're welcomed! [13:05:57] (03PS3) 10Mobrovac: service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) [13:11:33] (03PS4) 10Mobrovac: service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) [13:12:45] (03CR) 10jenkins-bot: [V: 04-1] service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [13:13:35] (03CR) 10Anomie: [C: 031] "Looks good, haven't tested. I verified that the permissions that are being removed on non-central wikis aren't used on those wikis anyway " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307893 (https://phabricator.wikimedia.org/T143055) (owner: 10Gergő Tisza) [13:14:29] (03PS5) 10Mobrovac: service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) [13:15:37] (03CR) 10Ottomata: [C: 032] Install mc and unzip on stat1001 as per request from ezachte [puppet] - 10https://gerrit.wikimedia.org/r/308160 (owner: 10Ottomata) [13:19:31] (03PS6) 10Mobrovac: service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) [13:24:01] (03PS7) 10Mobrovac: service::node: Compile the file holding puppet-controlled vars [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) [13:26:22] 06Operations, 10Traffic: OpenSSL 1.1 deployment for cache clusters - https://phabricator.wikimedia.org/T144523#2604754 (10MoritzMuehlenhoff) Thanks for getting this started! My gut feeling is that we should wait for 1.1.0a . The release only happened a week ago and there will be inevitable bugs that only sho... [13:29:52] (03CR) 10Mobrovac: [C: 031] "Cherry-picked on beta, works as advertised. PS7 PCC looking good too - https://puppet-compiler.wmflabs.org/3922/" [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [13:31:47] (03Abandoned) 10Hashar: contint: tidy Nodepool slaves config history [puppet] - 10https://gerrit.wikimedia.org/r/300092 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [13:33:14] (03CR) 10Paladox: [C: 031] 0.1.1-wmf5: debian/gbp.conf upstream-tag = %(version)s [debs/nodepool] (debian) - 10https://gerrit.wikimedia.org/r/301891 (owner: 10Hashar) [13:48:28] !log rebooting labnet1002 for kernel update [13:48:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:55:53] (03PS1) 10Hashar: contint: tidy Nodepool slaves config history [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) [13:56:08] (03CR) 10Hashar: "Redone with find -delete and a cron job with https://gerrit.wikimedia.org/r/308165" [puppet] - 10https://gerrit.wikimedia.org/r/300092 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [13:57:06] (03CR) 10jenkins-bot: [V: 04-1] contint: tidy Nodepool slaves config history [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [13:57:16] 06Operations, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review, 07Wikimedia-Incident: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2604862 (10hashar) https://gerrit.wikimedia.org/r/#/c/300092/ had... [13:58:12] (03PS2) 10Hashar: contint: tidy Nodepool slaves config history [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) [14:05:38] 06Operations, 10ops-eqiad, 13Patch-For-Review: Broken memory on mw1217 - https://phabricator.wikimedia.org/T138925#2604897 (10MoritzMuehlenhoff) a:05Cmjohnson>03MoritzMuehlenhoff I'll decommission the host, then. [14:06:05] (03CR) 10Dzahn: [C: 031] rcstream_redis: Limit to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308155 (owner: 10Muehlenhoff) [14:07:35] (03CR) 10Dzahn: ":) nice" [puppet] - 10https://gerrit.wikimedia.org/r/308152 (owner: 10Hashar) [14:09:54] (03CR) 10Dzahn: [C: 032] contint: tidy Nodepool slaves config history [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [14:11:16] (03CR) 10Dzahn: "yea, cron is simpler" [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [14:12:08] (03CR) 10Dzahn: "on gallium: Notice: /Stage[main]/Role::Ci::Master/Cron[tidy_jenkins_ephemeral_nodes_configs]/ensure: created" [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [14:13:24] (03PS1) 10Alex Monk: openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) [14:13:52] hashar: running the command from cron on gallium [14:14:30] (03CR) 10jenkins-bot: [V: 04-1] openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) (owner: 10Alex Monk) [14:14:50] !log gallium deleting jenkins/config-history files older than an hour [14:14:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:15:00] hashar: works. before: 6409 after: 55 [14:15:37] (03CR) 10Dzahn: "ran the command manually once: before there were 6409 files, after there were 55 files left" [puppet] - 10https://gerrit.wikimedia.org/r/308165 (https://phabricator.wikimedia.org/T126552) (owner: 10Hashar) [14:16:41] (03PS2) 10Alex Monk: openstack: Import openstack-wikistatus nova plugin [puppet] - 10https://gerrit.wikimedia.org/r/308169 (https://phabricator.wikimedia.org/T144317) [14:17:16] (03PS1) 10Muehlenhoff: kibana: Restrict to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308171 [14:17:23] (03CR) 10DCausse: "Well... I'm also ok to try with 10, I believe that max_ngrams_extracted is really the one important param to control the explosion." [puppet] - 10https://gerrit.wikimedia.org/r/307652 (https://phabricator.wikimedia.org/T127788) (owner: 10EBernhardson) [14:17:34] (03PS2) 10Dzahn: delete misc::monitoring::view::analytics::data [puppet] - 10https://gerrit.wikimedia.org/r/308128 [14:17:54] (03CR) 10Dzahn: [C: 032] "not used. all other "misc::monitoring" is gone" [puppet] - 10https://gerrit.wikimedia.org/r/308128 (owner: 10Dzahn) [14:18:35] (03PS5) 10Alexandros Kosiaris: check_puppetrun: Add reportfile handling [puppet] - 10https://gerrit.wikimedia.org/r/305487 [14:19:01] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] "After setting a limit of 3 failed resources, merging. Let's evaluate" [puppet] - 10https://gerrit.wikimedia.org/r/305487 (owner: 10Alexandros Kosiaris) [14:19:18] (03PS6) 10Alexandros Kosiaris: check_puppetrun: Add reportfile handling [puppet] - 10https://gerrit.wikimedia.org/r/305487 [14:19:20] (03CR) 10Alexandros Kosiaris: [V: 032] check_puppetrun: Add reportfile handling [puppet] - 10https://gerrit.wikimedia.org/r/305487 (owner: 10Alexandros Kosiaris) [14:23:41] (03PS1) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [14:24:55] (03PS2) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [14:26:57] mutante: you are awesome :] [14:27:56] !log downgrading cp4007 to varnish 3 T131502 [14:27:57] T131502: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502 [14:28:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:28:01] 06Operations, 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review, 07Wikimedia-Incident: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2604970 (10hashar) 05Open>03Resolved From the Gerrit change:... [14:29:00] (03PS4) 10Alexandros Kosiaris: check_puppetrun: Add more info in failed resources message [puppet] - 10https://gerrit.wikimedia.org/r/305629 [14:29:25] hashar: :) cu later [14:32:16] (03PS5) 10Alexandros Kosiaris: check_puppetrun: Add more info in failed resources message [puppet] - 10https://gerrit.wikimedia.org/r/305629 [14:32:49] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] check_puppetrun: Add more info in failed resources message [puppet] - 10https://gerrit.wikimedia.org/r/305629 (owner: 10Alexandros Kosiaris) [14:35:11] (03PS1) 10KartikMistry: apertium-srd-ita: Initial Debian packaging [debs/contenttranslation/apertium-srd-ita] - 10https://gerrit.wikimedia.org/r/308179 (https://phabricator.wikimedia.org/T144174) [14:40:22] (03PS3) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [14:40:32] (03CR) 10jenkins-bot: [V: 04-1] Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 (owner: 10Elukey) [14:44:35] (03PS4) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [14:47:05] RECOVERY - puppet last run on ms-be3003 is OK: CRITICAL: Puppet has 1 failures. Last run 30 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdl] [14:47:15] (03CR) 10Elukey: "https://puppet-compiler.wmflabs.org/3923/" [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 (owner: 10Elukey) [14:50:21] RECOVERY - puppet last run on ms-be1021 is OK: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdh] [14:57:29] hm, that's a bug, /me fixing [14:58:27] (03PS1) 10Alexandros Kosiaris: check_puppetrun: Add omitted exit code [puppet] - 10https://gerrit.wikimedia.org/r/308182 [15:00:31] (03CR) 10Alexandros Kosiaris: [C: 031] kibana: Restrict to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308171 (owner: 10Muehlenhoff) [15:00:58] (03CR) 10Alexandros Kosiaris: [C: 032] check_puppetrun: Add omitted exit code [puppet] - 10https://gerrit.wikimedia.org/r/308182 (owner: 10Alexandros Kosiaris) [15:03:06] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2605044 (10grin) @BBlack thanks for the detailed reply. I try not to talk apart this task, so I try hard to be brief. >>! In T144508#2604626, @BB... [15:08:46] PROBLEM - puppet last run on ms-be3003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdl] [15:09:23] PROBLEM - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdh] [15:10:17] nice! [15:12:09] really nice alright! thanks akosiaris ! [15:13:03] godog: I suppose due to my mistake we 've lost some ACK [15:13:29] :-(. I 'll re-ACK those [15:13:56] ah, because of alert text change [15:14:01] yeah that's fine [15:14:08] no, not that [15:15:35] rather this https://gerrit.wikimedia.org/r/#/c/308182/ [15:15:38] :-( [15:16:30] ah! [15:17:38] ACKNOWLEDGEMENT - puppet last run on ms-be3003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdl] alexandros kosiaris https://phabricator.wikimedia.org/T83811 [15:18:43] ACKNOWLEDGEMENT - puppet last run on ms-be1021 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 14 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[parted-/dev/sdh] alexandros kosiaris https://phabricator.wikimedia.org/T140597 [15:18:54] godog: ok, tracked down the tasks and re-ACKed them [15:19:22] RECOVERY - Host mw2192 is UP: PING OK - Packet loss = 0%, RTA = 36.27 ms [15:19:51] RECOVERY - Host mw2193 is UP: PING OK - Packet loss = 0%, RTA = 36.54 ms [15:21:42] RECOVERY - Host mw2194 is UP: PING OK - Packet loss = 0%, RTA = 36.62 ms [15:22:01] RECOVERY - Host mw2195 is UP: PING OK - Packet loss = 0%, RTA = 38.22 ms [15:31:12] RECOVERY - Host mw2196 is UP: PING OK - Packet loss = 0%, RTA = 36.44 ms [15:31:13] RECOVERY - Host mw2197 is UP: PING OK - Packet loss = 0%, RTA = 36.23 ms [15:32:02] RECOVERY - Host mw2198 is UP: PING OK - Packet loss = 0%, RTA = 36.23 ms [15:32:51] (03Abandoned) 10Dzahn: point wikipedia.in to 180.179.52.130 [dns] - 10https://gerrit.wikimedia.org/r/307959 (https://phabricator.wikimedia.org/T144508) (owner: 10Dzahn) [15:33:01] RECOVERY - Host mw2199 is UP: PING OK - Packet loss = 0%, RTA = 36.71 ms [15:33:09] !log downgrading cp4013 to varnish 3 T131502 [15:33:10] T131502: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502 [15:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:33:44] (03Abandoned) 10Dzahn: openstack::nova::instancersync fix define name [puppet] - 10https://gerrit.wikimedia.org/r/308130 (owner: 10Dzahn) [15:35:10] akosiaris: sweet, thanks! [15:37:00] (03PS5) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [15:38:42] (03PS6) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [15:40:42] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 714 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5132188 keys - replication_delay is 714 [15:41:12] Hey, I have a strange problem. I'm trying to install via pip in stat1003 I did both https://wikitech.wikimedia.org/wiki/Http_proxy [15:41:34] when I curl https://pypi.python.org/simple/mwxml/ it works [15:41:40] but wget doesn't [15:41:51] and pip also fails saying can't connect [15:41:57] wget stalls like this [15:42:16] https://www.irccloud.com/pastebin/BCZacJ4h/ [15:42:26] what am I missing? [15:43:36] I exported both [15:45:24] Amir1: please dont install software manually [15:45:35] it's in virtualenv [15:45:36] mutante, we must for the work that we do [15:45:38] This is intended [15:45:48] I clean everything once it's done [15:47:42] It seems adding --proxy explicitly to pip works for all packages except numpy(???) [15:47:53] I'm installing numpy using wheels in research repo [15:48:01] ottomata, if you could comment, that would be welcome [15:51:02] mutante: afaik, they are running code as their own users in their home dirs in a virtualenv [15:51:05] (03PS7) 10Elukey: Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 [15:51:11] on stat boxes [15:51:31] (03CR) 10Ottomata: [C: 032] Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 (owner: 10Elukey) [15:51:35] (03CR) 10Ottomata: [C: 031] Allow a more custom set of arguments for the cleanup script [puppet/zookeeper] - 10https://gerrit.wikimedia.org/r/308173 (owner: 10Elukey) [15:51:36] i dont know, you'll know better [15:52:04] FFS, guys. I've re-fought this battle about researchers being able to install research-libraries on the research processing machines several times now. Are we going to do this again? [15:52:24] Sorry if I'm getting upset, but I swear that this conversation has come up and been settled 3 times already [15:53:15] just ignore my comment. it's an analytics system. [15:53:18] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2605156 (10Aklapper) This Phabricator task is //specifically// about pointing wikipedia.in to 180.179.52.130. For general high-level discussion... [15:53:22] \o/ [15:54:03] all i know is L3 says not to. what that applies to i dont know [15:54:39] We can't be disallowed from installing libraries on the research machines and still be able to do our job. [15:55:05] I'm glad to see that an exception can apply. [15:55:12] Maybe we can document this somewhere? [15:55:18] So it doesn't need to be re-hashed? [15:55:37] Like, maybe if I write a page up on wikitech and some authority says, "Yeah. This is cool" on it. [15:56:33] halfak: as long as you use a virtual env and you do the part of the good citizen, I am not seeing anything against that [15:56:41] analytics ops can handle that as they see fit, i dont make rules [15:56:50] (I am far from authoritative, just wanted to give me opinion :D [15:56:56] Great. I think that makes sense too elukey [15:57:13] virtualenv == can't really affect other users of the system [15:58:09] OK. I've got to run. I need switch locations. I'll add the wikitech docs to my to-do list. [15:58:22] you will not be 100% bullet proof because your user can consume tons of resources on the machine but this is the part of the "good citizen" [15:58:25] :) [15:58:30] it is true for all the stat boxes anyway [16:00:42] (03PS1) 10Alex Monk: openstack horizon puppettab: Put docs on a question mark next to the role name [puppet] - 10https://gerrit.wikimedia.org/r/308188 (https://phabricator.wikimedia.org/T91990) [16:06:56] (03PS1) 10Elukey: Elukey wants pages [puppet] - 10https://gerrit.wikimedia.org/r/308189 [16:08:27] (03CR) 10Elukey: [C: 032] Elukey wants pages [puppet] - 10https://gerrit.wikimedia.org/r/308189 (owner: 10Elukey) [16:08:33] "rm -rf virtualenv_folder" once it's done is a part of being good citizen :) [16:09:01] I think there is the need of a sanitization of the stat boxes [16:09:07] because they are a mess sometimes [16:09:47] but we want to support researchers of course. The only concern is not running a wild-west section of the wikimedia infrastructure [16:09:50] :) [16:10:30] agreed [16:10:40] !log downgrading cp4014 to varnish 3 T131502 [16:10:41] T131502: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502 [16:10:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:11:06] (03CR) 10BryanDavis: [C: 031] kibana: Restrict to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308171 (owner: 10Muehlenhoff) [16:12:45] (03PS1) 10Elukey: Replace elukey with ltoscano on contacts [puppet] - 10https://gerrit.wikimedia.org/r/308191 [16:14:59] (03Abandoned) 10Elukey: Replace elukey with ltoscano on contacts [puppet] - 10https://gerrit.wikimedia.org/r/308191 (owner: 10Elukey) [16:17:32] (03CR) 10Filippo Giunchedi: [C: 04-1] "LGTM overall, a few comments inline, also it is easier to understand (and rollback) if the change isn't coupled with changing mathoid too" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/308021 (https://phabricator.wikimedia.org/T144542) (owner: 10Mobrovac) [16:23:41] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5091317 keys - replication_delay is 30 [16:24:03] PROBLEM - Host bast3001 is DOWN: PING CRITICAL - Packet loss = 100% [16:24:57] ^ [16:25:04] I can't connect to stat1003 [16:25:22] I use bast3001 [16:27:11] mark: ^ work ongoing on bast3001 ? [16:27:34] Amir1: you should be able to use bast1001.wikimedia.org in its place [16:27:40] I'm routing using codfw [16:27:47] doesn't really matter I guess [16:28:20] just 50/80ms apart, if you don't mind that [16:28:27] I just wanted to say that it's broken [16:28:43] nah, I use iran internet my scale is seconds [16:28:45] :D [16:29:10] hehe I guess it doesn't really matter then no [16:29:13] various esams clusters in ganglia are down. 33% ping loss to maerlant.wikimedia.org (although that was the first attempt, the second one and all others have 0% ping loss) [16:29:56] godog: uhm [16:30:01] was it called slauerhoff once? :P [16:30:10] then yes :P [16:30:13] hehe possibly, checking [16:30:23] https://phabricator.wikimedia.org/T131543 [16:30:53] mark: yes that used to be slauerhoff [16:30:55] ooops [16:30:58] alright [16:31:01] we'll rerack it :P [16:31:04] almost wiped it :P [16:31:08] will relabel [16:31:22] gj ;) [16:31:30] ah yeah, what SPF|Cloud said [16:33:11] after we're done with your swift server [16:33:56] 06Operations, 10DNS, 10Domains, 10Traffic, and 2 others: Point wikipedia.in to 180.179.52.130 instead of URL forward - https://phabricator.wikimedia.org/T144508#2605282 (10BBlack) >>! In T144508#2605156, @Aklapper wrote: > This Phabricator task is //specifically// about pointing wikipedia.in to 180.179.52.... [16:38:06] mark, I think there were many issues with labeling. Can you check: https://phabricator.wikimedia.org/T136403 [16:40:16] !log restarting jenkins shortly for plugin upgrade [16:40:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:48:59] !log downgrading cp4015 to varnish 3 T131502 [16:49:00] T131502: Convert upload cluster to Varnish 4 - https://phabricator.wikimedia.org/T131502 [16:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:49:44] RECOVERY - Host bast3001 is UP: PING OK - Packet loss = 0%, RTA = 84.22 ms [16:50:12] !rebooting bast3001 [16:50:16] !log rebooting bast3001 [16:50:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:50:50] !log jenkins restarted [16:50:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:53:44] mark: confirmed we're back, thanks! also thanks for ms-be [16:55:35] PROBLEM - puppet last run on bast3001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Service[ganglia-monitor] [16:57:58] well [16:58:00] since we now wasted time [16:58:05] we're not gonna get it up :( [16:58:08] but at least it's racked now :D [16:59:00] hehe good enough, next time you are there works too [17:01:09] slauerhoff relabeled to bast3001 [17:01:17] racktables updated [17:02:58] 06Operations, 10ops-esams: update hostname label on system bast3001, was slauerhoff - https://phabricator.wikimedia.org/T131543#2605397 (10mark) 05Open>03Resolved Done! :D [17:06:46] 06Operations, 10Traffic: OpenSSL 1.1 deployment for cache clusters - https://phabricator.wikimedia.org/T144523#2605412 (10BBlack) >>! In T144523#2604754, @MoritzMuehlenhoff wrote: > As for re-enabling triple-des/rc4 I'm undecided with a tendency towards sticking with the openssl default (depending on when we g... [17:16:36] (03CR) 1020after4: [C: 031] WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) (owner: 1020after4) [17:29:55] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 771 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5098940 keys - replication_delay is 771 [17:31:02] (03PS15) 1020after4: WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [17:50:23] yuvipanda: any chance of a quota increase on reading-web-staging ? I need to create a new instance but I've run out of space. [17:51:08] jdlrobson there's a process for it now, https://phabricator.wikimedia.org/T140904 [17:51:30] * jdlrobson shakes his fists at the hear of the word process ;-) Thanks i will follow... [17:52:24] yw [17:53:48] (03PS9) 10Gehel: maps - use new tileshell.js script to notify tilerator of expired tiles [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) [17:55:03] (03CR) 10Gehel: [C: 032] maps - use new tileshell.js script to notify tilerator of expired tiles [puppet] - 10https://gerrit.wikimedia.org/r/307787 (https://phabricator.wikimedia.org/T139451) (owner: 10Gehel) [17:55:44] 06Operations, 10Traffic, 13Patch-For-Review: Planning for phasing out non-Forward-Secret TLS ciphers - https://phabricator.wikimedia.org/T118181#2605488 (10BBlack) From the CN subtask ( T144194 ), we've learned that CN won't be a useful tool for targeting the mass ancient browsers that matter. It may be abl... [17:55:51] yurik: ^ [17:57:51] yuvipanda: is there any issue with horizon right now? It tells me "nable to retrieve instances." [18:07:01] jdlrobson I just used it and seems ok, can you 1. try again and if not, 2. log out and back in? [18:07:12] sure [18:07:18] and if not, 3. file a bug? andrew is out today I think. [18:07:49] yuvipanda: yup its working again [18:07:51] w00t [18:10:03] yuvipanda: i'm guessing i cannot downgrade number of cpus on an existing box? [18:10:11] nope [18:13:08] yuvipanda: https://phabricator.wikimedia.org/T144623 [18:16:47] (03CR) 10Jforrester: [C: 031] "Thanks! Assuming the ltwiki community are happy with this. :-)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307455 (https://phabricator.wikimedia.org/T144118) (owner: 10Dereckson) [18:16:57] jdlrobson cool, thanks. note the last paragraph of https://phabricator.wikimedia.org/T140904 tho. you probably won't get it today. [18:17:07] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 5095712 keys - replication_delay is 0 [18:17:11] 06Operations, 10Ops-Access-Requests: Requesting access to stat1002 for ZZhou (WMF) - https://phabricator.wikimedia.org/T144624#2605535 (10ZhouZ) [18:17:15] yuvipanda: i figured when you said process :) [18:17:23] :) cool [18:17:52] apergos: Hi! If you're around, there was a question on labs-l about latest xml dumps not being available in tools - I understand it's rsync-ed over from dataset1001 to labstore1003, but the latest sept 1 dumps haven't showed up yet, even though I see it generated upstream. I was wondering if you'd know something about it [18:28:09] @seen Volker_E [18:28:09] mutante: I have never seen Volker_E [18:28:12] wm-bot: liar [18:28:40] maybe it's the underscore [18:29:07] @seen mutante [18:29:07] mutante: are you really looking for yourself? [18:29:45] (03PS3) 10Ppchelko: Change-Prop: Switch to new events. [puppet] - 10https://gerrit.wikimedia.org/r/308077 [18:29:47] @seen wm-bot [18:29:47] mutante: I am right here [18:30:11] Lies its an alien replica designed to fool you into thinking it is wm-bot [18:31:30] (03PS4) 10Dzahn: wikimedia.org: repeat hostname on each line for multi records [dns] - 10https://gerrit.wikimedia.org/r/304155 [18:34:37] 06Operations, 10Traffic: Strong cipher preference ordering for cache terminators - https://phabricator.wikimedia.org/T144626#2605597 (10BBlack) [18:35:43] !selfie is https://meta.wikimedia.org/wiki/Wm-bot#/media/File:Wm-bot.png [18:35:44] Key was added [18:36:48] This user is now online in #wikimedia-tech. I'll let you know when they show some activity (talk, etc.) [18:36:48] @notify Cameron11598 *g* [18:37:30] hmm, it knows about -tech but not about this channel [18:38:03] Hm [18:38:04] Cameron11598: it's a snitch and tells me which other channels you join [18:38:06] Also in -helpers [18:38:43] and #mediawiki [18:39:26] No results were found, remember, the bot is searching through content of keys and their names [18:39:26] @regsearch icinga [18:39:37] and -commons and -mobile [18:39:44] hmm, alright [18:40:04] It probably just picks the channel it first saw me in? [18:40:20] It did notify me when I joined wikipedia-help [18:40:38] but didn't tell me in which channel you were looking for me [18:40:43] yea, originally i was just curious why it says it never saw people who are right here [18:41:30] i should just use @notify actually [18:41:33] This user is now online in #wikimedia-dev. I'll let you know when they show some activity (talk, etc.) [18:41:33] @notify Volker_E [18:42:17] (yep, that's what i actually want) [18:44:53] (03PS6) 10Rush: labstore: run nfs-exports-daemon on all servers in a cluster [puppet] - 10https://gerrit.wikimedia.org/r/307629 (https://phabricator.wikimedia.org/T126083) [18:52:43] (03CR) 10Rush: [C: 032 V: 032] labstore: run nfs-exports-daemon on all servers in a cluster [puppet] - 10https://gerrit.wikimedia.org/r/307629 (https://phabricator.wikimedia.org/T126083) (owner: 10Rush) [19:05:29] 06Operations, 05Continuous-Integration-Scaling, 07Nodepool, 07WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#2605675 (10hashar) [19:09:23] LOL, so many errors in puppet [19:09:24] https://phabricator.wikimedia.org/P3965 [19:11:05] strange microsoft block https://bugs.debian.org/ [19:11:11] with a big fat red x [19:11:39] We recommend that you do not proceed to this website. It has been reported to Microsoft for containing threats to your computer that might reveal personal or financial information. [19:15:37] 06Operations, 05Continuous-Integration-Scaling, 07Nodepool, 07WorkType-NewFunctionality: Backport python-shade from debian/testing to jessie-wikimedia - https://phabricator.wikimedia.org/T107267#2605705 (10Paladox) @MoritzMuehlenhoff hi, could you do this please? [19:17:41] (03CR) 10Smalyshev: [C: 031] wdqs - move data to /srv [puppet] - 10https://gerrit.wikimedia.org/r/308023 (https://phabricator.wikimedia.org/T144536) (owner: 10Gehel) [19:28:31] LOL https://phabricator.wikimedia.org/F4426907 [19:29:38] (03PS2) 10Hashar: (wip) Add puppet-lint to Rakefile (wip) [puppet] - 10https://gerrit.wikimedia.org/r/288620 [19:29:40] (03PS2) 10Hashar: (wip) Speed up linting by only process HEAD (wip) [puppet] - 10https://gerrit.wikimedia.org/r/288629 [19:34:08] (03PS5) 10Gergő Tisza: Do not use $wgExtensionFunctions to set globals [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307893 (https://phabricator.wikimedia.org/T143055) [19:34:59] (03CR) 10jenkins-bot: [V: 04-1] (wip) Speed up linting by only process HEAD (wip) [puppet] - 10https://gerrit.wikimedia.org/r/288629 (owner: 10Hashar) [19:47:49] Rp [19:47:52] WOops [19:47:57] reported here https://github.com/Microsoft/BashOnWindows/issues/1044 [19:51:51] 06Operations, 06Labs: Set up monitoring for secondary labstore HA cluster - https://phabricator.wikimedia.org/T144633#2605766 (10madhuvishy) [19:58:44] (03PS16) 1020after4: WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [19:59:45] (03PS1) 10Ppchelko: Change-Prop: Rerender mobile-sections when wikidata item is changed [puppet] - 10https://gerrit.wikimedia.org/r/308230 (https://phabricator.wikimedia.org/T144632) [20:00:42] (03PS17) 1020after4: WIP: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [20:02:58] (03PS3) 10Hashar: (WIP) Add puppet-lint to Rakefile (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/288620 [20:03:00] (03PS3) 10Hashar: (wip) Speed up linting by only processing HEAD (wip) [puppet] - 10https://gerrit.wikimedia.org/r/288629 [20:04:43] (03CR) 10jenkins-bot: [V: 04-1] (wip) Speed up linting by only processing HEAD (wip) [puppet] - 10https://gerrit.wikimedia.org/r/288629 (owner: 10Hashar) [20:05:07] (03CR) 10BearND: [C: 031] "Thanks for the quick turnaround." [puppet] - 10https://gerrit.wikimedia.org/r/308230 (https://phabricator.wikimedia.org/T144632) (owner: 10Ppchelko) [20:09:57] (03PS18) 1020after4: Scap swat command [mediawiki-config] - 10https://gerrit.wikimedia.org/r/306259 (https://phabricator.wikimedia.org/T142880) [20:10:34] (03CR) 10Hashar: "Would probably timing out until we make puppet-lint to only process changes in HEAD which is https://gerrit.wikimedia.org/r/#/c/288629/" [puppet] - 10https://gerrit.wikimedia.org/r/288620 (owner: 10Hashar) [20:10:47] (03CR) 10Hashar: "BUGGED due to Jenkins doing a shallow clone" [puppet] - 10https://gerrit.wikimedia.org/r/288629 (owner: 10Hashar) [20:20:50] * LadyViolet wonders if there's any admin of mediawiki.org here for assistance [20:21:12] hi [20:21:31] LadyViolet, what do you need? [20:21:33] Krenair: I need an admin to grant a local IP block exemption to an user caught in a global rangeblock [20:21:41] ok [20:21:59] why the different nickname today? [20:22:09] User: IgorShishkin - Ticket#2016082710006695 [20:22:43] I haven't done OTRS stuff in like 16 months now [20:24:34] (03PS1) 10Gehel: elasticsearch: increase keepalive timeout to allow for better connection reuse. [puppet] - 10https://gerrit.wikimedia.org/r/308234 [20:26:03] I've left a note on https://www.mediawiki.org/wiki/Project:Requests#IgorShishkin [20:26:21] ticket is in our queue, so I don't know if you'd have access to it [20:26:35] I can handle that. [20:26:38] Uno momento [20:26:46] *un :) [20:26:53] Oh Krenair beat me [20:26:56] heh [20:27:00] LadyViolet: I don't speak spanish! :P [20:27:09] LadyViolet, steward queue? [20:27:15] I don't speak English either :P [20:27:18] Krenair: yep [20:27:33] yeah don't think I ever had the particular tech permission that'd grant access to that [20:27:59] Thank you all [20:28:06] I will inform the user. [20:30:29] hola! uno dos tres guacamole cerveza donde esta el baño [20:30:51] Nemo_bis: https://www.mediawiki.org/w/index.php?title=Project%3ARequests%2FUser_rights%2FPsychoslave&type=revision&diff=2229403&oldid=2212082 ;-) [20:30:52] :) [20:31:01] I'll take some guacamole. [20:31:11] We don't have that in here [20:31:25] sure you have, it's now world bobo/hipster food. [20:31:39] * MarcoAurelio enjoys Westminster debates and PMQs [20:32:02] Dereckson: probably there are some Mexican food store near me, but it's not typical of my country :) [20:32:16] Dereckson: Nah, you're getting that mixed up with kale and quinoa. [20:32:21] (03Draft2) 10Paladox: openstack: fix puppet lint in spriceproxy.pp [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:32:25] (03Draft1) 10Paladox: openstack: fix puppet lint in spriceproxy.pp [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:32:50] ostriches, it's a request for translation administrator rights [20:32:58] I'm well aware :p [20:33:00] (03PS3) 10Paladox: openstack: puppet-lint fixes in spiceproxy.pp [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:33:10] I'm just making fun of that batshit translation syntax [20:36:37] (03CR) 10EBernhardson: [C: 031] elasticsearch: increase keepalive timeout to allow for better connection reuse. [puppet] - 10https://gerrit.wikimedia.org/r/308234 (owner: 10Gehel) [20:37:48] (03Draft1) 10Paladox: Add --no-140chars-check to .puppet-lint.rc [puppet] - 10https://gerrit.wikimedia.org/r/308236 [20:38:23] (03PS2) 10Paladox: Add --no-140chars-check to .puppet-lint.rc [puppet] - 10https://gerrit.wikimedia.org/r/308236 [20:39:12] (03CR) 10jenkins-bot: [V: 04-1] Add --no-140chars-check to .puppet-lint.rc [puppet] - 10https://gerrit.wikimedia.org/r/308236 (owner: 10Paladox) [20:40:08] (03Abandoned) 10Paladox: Add --no-140chars-check to .puppet-lint.rc [puppet] - 10https://gerrit.wikimedia.org/r/308236 (owner: 10Paladox) [20:44:38] (03PS1) 10Gehel: discovery-stats: only collect maps usage hourly, not each minute [puppet] - 10https://gerrit.wikimedia.org/r/308237 [20:46:16] (03CR) 10MaxSem: [C: 031] discovery-stats: only collect maps usage hourly, not each minute [puppet] - 10https://gerrit.wikimedia.org/r/308237 (owner: 10Gehel) [20:48:51] (03CR) 10Gehel: [C: 032] discovery-stats: only collect maps usage hourly, not each minute [puppet] - 10https://gerrit.wikimedia.org/r/308237 (owner: 10Gehel) [20:49:42] (03PS2) 10Dzahn: rcstream_redis: Limit to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308155 (owner: 10Muehlenhoff) [20:50:03] (03CR) 10Dzahn: [C: 032] rcstream_redis: Limit to domain networks [puppet] - 10https://gerrit.wikimedia.org/r/308155 (owner: 10Muehlenhoff) [20:50:38] (03PS4) 10Paladox: openstack: puppet-lint fixes in conductor.pp, network.pp, scheduler.pp and spiceproxy.pp [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:50:48] (03PS5) 10Paladox: openstack: puppet-lint fixes in conductor.pp, network.pp, scheduler.pp and spiceproxy.pp [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:54:46] (03PS6) 10Paladox: openstack: puppet-lint fixes in /nova/ [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:55:02] (03PS7) 10Paladox: openstack: puppet-lint fixes in openstack [puppet] - 10https://gerrit.wikimedia.org/r/308235 [20:55:49] (03PS8) 10Paladox: openstack: lint, fix alignment [puppet] - 10https://gerrit.wikimedia.org/r/308235 [21:00:25] (03Draft1) 10Paladox: openstack: lint, fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308278 [21:00:50] . [21:02:07] , [21:03:02] i love krenair 💝💝💝 [21:03:51] ? [21:04:01] at least the trol decided to love you instead of start insulting, Krenair :P [21:04:04] old bans of the troll [21:04:11] Ah [21:04:17] he wanted to marry me last time [21:04:23] I love Krenair too ❤️ [21:04:24] to jem, too [21:04:35] lol [21:04:40] MarcoAurelio: and you broke his heart saying no? [21:04:49] Cameron11598: of course [21:05:04] also, what I'd say to my wife? [21:05:23] MarcoAurelio: I suppose thats fair, I hope you let him down gentlythough [21:05:53] I'm not good at this love affairs. I bluntly said no [21:06:00] (03PS2) 10Paladox: openstack: lint, fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308278 [21:07:12] my wife is called Solitude, and fidelity is important to us :P [21:07:29] What is fidelity? [21:07:58] https://www.fidelity.co.uk/investor/default.page [21:08:09] faithfulness to a person, cause, or belief, demonstrated by continuing loyalty and support [21:08:19] https://en.wiktionary.org/wiki/fidelity#Noun [21:08:22] Oh [21:08:27] thanks [21:10:34] What is this host host.redstation.co.uk [21:10:37] never herd of it [21:12:57] (03PS1) 10BBlack: ssl_ciphersuite: remove less-popular 3DES options [puppet] - 10https://gerrit.wikimedia.org/r/308279 [21:12:59] (03PS1) 10BBlack: ssl_ciphersuite: reorder to prefer ECDSA more-strongly [puppet] - 10https://gerrit.wikimedia.org/r/308280 [21:14:37] 06Operations, 10Analytics: Can't log into https://piwik.wikimedia.org/ - https://phabricator.wikimedia.org/T144326#2605979 (10Milimetric) I was logged in and it looks like my session is still good. But if I try to login with the credentials for piwik-wmf-admin, it fails authentication. Only two possibilities... [21:15:41] (03CR) 10BBlack: [C: 032 V: 032] ssl_ciphersuite: remove less-popular 3DES options [puppet] - 10https://gerrit.wikimedia.org/r/308279 (owner: 10BBlack) [21:17:07] (03CR) 10BBlack: [C: 032 V: 032] ssl_ciphersuite: reorder to prefer ECDSA more-strongly [puppet] - 10https://gerrit.wikimedia.org/r/308280 (owner: 10BBlack) [21:34:53] (03PS1) 10MarcoAurelio: Rename 'technican' and 'technician' to 'editinterface' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) [21:38:28] 06Operations, 10Traffic, 13Patch-For-Review: Decom bits.wikimedia.org hostname - https://phabricator.wikimedia.org/T107430#2606028 (10BBlack) FWIW, the iOS app change hit its first beta release 3 weeks ago in 5.1.0.900, and was released with 5.1.0.913.1 about two weeks ago. [21:39:03] (03Draft1) 10Paladox: swift: Fix WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308282 [21:40:35] (03CR) 10Dzahn: "i think you forgot to "git add" the file. it's empty." [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [21:40:49] (03PS2) 10Paladox: swift: Fix WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308282 [21:41:07] (03CR) 10Paladox: "@Dzahn yep sorry, I was doing it through gerrit's inline edit." [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [21:41:25] (03CR) 10MarcoAurelio: "NOTE TO DEPLOYER: After merging, maintenance script https://www.mediawiki.org/wiki/Manual:MigrateUserGroup.php needs to be run to migrate " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/308281 (https://phabricator.wikimedia.org/T144638) (owner: 10MarcoAurelio) [21:42:35] (03CR) 10Dzahn: [C: 031] "yea, the change is good, it's a good example for where we want to ignore warnings, "volatile" is special. but you should call this "ignore" [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [21:43:47] (03PS3) 10Alexandros Kosiaris: puppetmasters: support multiple git masters [puppet] - 10https://gerrit.wikimedia.org/r/307236 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto) [21:44:52] (03Abandoned) 10Alexandros Kosiaris: etherpad: Move role into module [puppet] - 10https://gerrit.wikimedia.org/r/220085 (owner: 10Alexandros Kosiaris) [21:53:38] (03PS4) 10Alexandros Kosiaris: puppetmasters: support multiple git masters [puppet] - 10https://gerrit.wikimedia.org/r/307236 (https://phabricator.wikimedia.org/T143869) (owner: 10Giuseppe Lavagetto) [21:53:41] jouncebot: next [21:53:41] In 63 hour(s) and 6 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160905T1300) [21:54:01] Dereckson: it's time to close wikimania2015wiki :) [21:54:07] will you take care of that? [21:54:12] deploy are usaually monday tuesday wednesday thursday [21:54:23] sorr for [21:54:26] joucebot [21:54:37] that should have been done yesterday [21:55:06] yep [21:55:10] I just remembered [21:55:13] it can wait until Monday, it's not an emergency [21:55:13] but then we has a very heavily loaded SWAT at evening [21:55:17] sure [21:55:50] the Monday first SWAT tend to be pretty busy too [21:56:59] So I suggest we plan that to morning or evening SWAT. [21:57:22] add it now :) [21:58:32] Added to evening SWAT. [22:01:21] (03PS1) 10Alexandros Kosiaris: Remove strontium remnant from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/308284 [22:01:23] (03PS1) 10Alexandros Kosiaris: puppetmaster200{1,2}.codfw.wmnet: Set up as puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/308285 [22:04:11] (03Abandoned) 10Dereckson: Test file backend for Math on labtestwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/270146 (https://phabricator.wikimedia.org/T126628) (owner: 10Dereckson) [22:05:05] (03PS1) 10Subramanya Sastry: Remove bunyan logger config from Parsoid RT config [puppet] - 10https://gerrit.wikimedia.org/r/308286 [22:07:23] (03CR) 10Arlolra: Remove bunyan logger config from Parsoid RT config (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/308286 (owner: 10Subramanya Sastry) [22:07:34] (03Draft1) 10Paladox: sslcert: ignore WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308287 [22:08:00] (03PS2) 10Paladox: sslcert: ignore WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308287 [22:09:55] (03PS2) 10Subramanya Sastry: Remove bunyan logger + statsd config from Parsoid RT config [puppet] - 10https://gerrit.wikimedia.org/r/308286 [22:10:51] (03CR) 10Arlolra: [C: 031] Remove bunyan logger + statsd config from Parsoid RT config [puppet] - 10https://gerrit.wikimedia.org/r/308286 (owner: 10Subramanya Sastry) [22:11:59] (03CR) 10Subramanya Sastry: "Note that this doesn't affect Parsoid production cluster. This only applies to ruthenium which we use for Parsoid's RT testing." [puppet] - 10https://gerrit.wikimedia.org/r/308286 (owner: 10Subramanya Sastry) [22:12:35] mutante, can you take a look at ^^ .. so we can unbreak rt testing there? [22:14:49] (03PS1) 10Alexandros Kosiaris: site.pp: Use role keyword as a function [puppet] - 10https://gerrit.wikimedia.org/r/308288 [22:15:35] (03CR) 10Alexandros Kosiaris: [C: 032] Remove strontium remnant from site.pp [puppet] - 10https://gerrit.wikimedia.org/r/308284 (owner: 10Alexandros Kosiaris) [22:15:53] (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster200{1,2}.codfw.wmnet: Set up as puppetmasters [puppet] - 10https://gerrit.wikimedia.org/r/308285 (owner: 10Alexandros Kosiaris) [22:16:44] or akosiaris if you are around .. https://gerrit.wikimedia.org/r/308286 [22:17:26] (03PS3) 10Alexandros Kosiaris: Remove bunyan logger + statsd config from Parsoid RT config [puppet] - 10https://gerrit.wikimedia.org/r/308286 (owner: 10Subramanya Sastry) [22:17:33] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Remove bunyan logger + statsd config from Parsoid RT config [puppet] - 10https://gerrit.wikimedia.org/r/308286 (owner: 10Subramanya Sastry) [22:17:39] thanks. :) [22:17:40] i i just.. [22:17:43] http://puppet-compiler.wmflabs.org/3928/ [22:17:52] cool :) ok [22:18:06] mutante: oops, sorry [22:18:15] no, thanks [22:18:20] i could have said something [22:18:22] it was clearly not for wtp boxes which is why I moved forward [22:18:32] all good [22:19:27] i forget what i run to update puppet now .. puppet -tv or something like that? [22:19:37] subbu: puppet agent -t -v [22:19:40] thanks. [22:19:49] mutante: this one is interesting https://gerrit.wikimedia.org/r/308288 [22:20:01] scap is complaining, Host key verification failed. [22:20:18] oh [22:20:19] arlolra: for which host ? [22:20:31] arlolra: what host? Something that was reimaged recently? [22:20:31] * akosiaris is pretty sure he is going to hear wtp2016 [22:20:35] deployment-parsoid09.deployment-prep.eqiad.wmflabs [22:20:39] akosiaris: wow, good to know! [22:20:40] ah labs [22:20:50] arlolra: E_WRONGCHANNEL :) [22:20:55] ha [22:20:56] ok [22:20:58] arlolra, #wikimedia-releng [22:21:09] thanks [22:21:33] bd808: xD [22:22:42] (03CR) 10Dzahn: [C: 031] "good, unless we actually want to move files/ssl/ to modules/sslcert/files/ which i'm unsure about" [puppet] - 10https://gerrit.wikimedia.org/r/308287 (owner: 10Paladox) [22:23:41] (03CR) 10Dzahn: [C: 032] swift: Fix WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [22:24:02] (03PS3) 10Dzahn: swift: Fix WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [22:24:29] (03PS4) 10Paladox: swift: ignore WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308282 [22:25:07] mutante ^^ thanks :) [22:25:11] (03CR) 10Dzahn: "it's not _actually_ fix, it's ignore, but since "volatile" is a special case that is alright here" [puppet] - 10https://gerrit.wikimedia.org/r/308282 (owner: 10Paladox) [22:25:43] 06Operations, 10ops-codfw, 05Puppet-infrastructure-modernization: rack/setup/deploy puppetmaster200[12] - https://phabricator.wikimedia.org/T143255#2606114 (10akosiaris) [22:28:00] (03PS9) 10Dzahn: openstack: lint, fix alignment [puppet] - 10https://gerrit.wikimedia.org/r/308235 (owner: 10Paladox) [22:28:38] (03PS2) 10Alexandros Kosiaris: site.pp: Use role keyword as a function [puppet] - 10https://gerrit.wikimedia.org/r/308288 [22:29:19] * akosiaris is going to submit a huge puppet compiler job soon. Sorry [22:30:29] (03CR) 10Dzahn: [C: 032] openstack: lint, fix alignment [puppet] - 10https://gerrit.wikimedia.org/r/308235 (owner: 10Paladox) [22:30:35] ^^ thanks [22:30:36] :) [22:30:40] hurries up :) [22:31:26] !log aaron@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: No-op sync - 880c7f9526610bfd396dddb14f7772f9247850cb (duration: 00m 49s) [22:31:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:31:38] (03CR) 10EBernhardson: Report partial result from mwgrep (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/307652 (https://phabricator.wikimedia.org/T127788) (owner: 10EBernhardson) [22:32:01] (03PS3) 10Dzahn: sslcert: ignore WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308287 (owner: 10Paladox) [22:33:15] (03CR) 10Dzahn: [C: 032] sslcert: ignore WARNING: puppet:// URL without modules/ found [puppet] - 10https://gerrit.wikimedia.org/r/308287 (owner: 10Paladox) [22:33:26] (03CR) 10Dzahn: "only comments" [puppet] - 10https://gerrit.wikimedia.org/r/308287 (owner: 10Paladox) [22:33:39] :p [22:36:41] madhuvishy: as jobs get done they should show up the next day, I think the rsync is once a day or so [22:39:03] (03CR) 10Dzahn: [C: 031] "yea, that's a newer puppet style convention / lint warning that all required parameters should be listed first. lgtm" [puppet] - 10https://gerrit.wikimedia.org/r/308278 (owner: 10Paladox) [22:40:13] (03CR) 10Dzahn: "20after4, ack? anytime and no service restart?" [puppet] - 10https://gerrit.wikimedia.org/r/307456 (owner: 10Paladox) [22:40:45] (03PS5) 10Paladox: phabricator: Add text/x-python to files.viewable-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/307456 [22:41:30] (03PS12) 10Thiemo Mättig (WMDE): Gerit: Rewrite outdated comment about Gerrit-Phabricator linking [puppet] - 10https://gerrit.wikimedia.org/r/256663 (https://phabricator.wikimedia.org/T75997) [22:42:46] (03PS3) 10Dzahn: admin: utility for a group/user matrix [puppet] - 10https://gerrit.wikimedia.org/r/288711 (https://phabricator.wikimedia.org/T135187) (owner: 10Hashar) [22:43:34] (03CR) 10Dzahn: [C: 032] admin: utility for a group/user matrix [puppet] - 10https://gerrit.wikimedia.org/r/288711 (https://phabricator.wikimedia.org/T135187) (owner: 10Hashar) [22:44:59] (03CR) 10Dzahn: "actually this is almost exactly what i wanted to have as well for another purpose. i think it's useful. thanks" [puppet] - 10https://gerrit.wikimedia.org/r/288711 (https://phabricator.wikimedia.org/T135187) (owner: 10Hashar) [22:45:10] apergos: all right, I'll watch out for it [22:45:13] thanks :) [22:45:30] sure [23:25:23] 06Operations, 06Release-Engineering-Team, 15User-greg, 07Wikimedia-Incident: Institute quarterly(?) review of incident reports and follow-up - https://phabricator.wikimedia.org/T141287#2606273 (10greg) >>! In T141287#2565426, @greg wrote: > One easy thing to do would be to ping all of the very old tasks (e... [23:32:18] (03CR) 1020after4: "This should be fine" [puppet] - 10https://gerrit.wikimedia.org/r/307456 (owner: 10Paladox) [23:32:23] (03CR) 1020after4: [C: 031] phabricator: Add text/x-python to files.viewable-mime-types [puppet] - 10https://gerrit.wikimedia.org/r/307456 (owner: 10Paladox) [23:32:46] (03CR) 10Alex Monk: [C: 031] openstack: lint, fix optional parameter listed before required parameter [puppet] - 10https://gerrit.wikimedia.org/r/308278 (owner: 10Paladox)