[00:29:31] PROBLEM - puppet last run on rdb1005 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [00:46:42] (03PS2) 10Aude: Move interwiki sorting orders to config [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323556 (https://phabricator.wikimedia.org/T111023) [00:57:31] RECOVERY - puppet last run on rdb1005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:34:51] PROBLEM - Postgres Replication Lag on maps1003 is CRITICAL: CRITICAL - Rep Delay is: 1811.526301 Seconds [01:35:51] RECOVERY - Postgres Replication Lag on maps1003 is OK: OK - Rep Delay is: 24.242038 Seconds [02:06:34] (03CR) 10Krinkle: [C: 031] Nginx timeout should be higher than thumbor subprocess timeout [puppet] - 10https://gerrit.wikimedia.org/r/323403 (https://phabricator.wikimedia.org/T151459) (owner: 10Gilles) [02:07:09] (03CR) 10Krinkle: Nginx timeout should be higher than thumbor subprocess timeout (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/323403 (https://phabricator.wikimedia.org/T151459) (owner: 10Gilles) [02:18:08] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.3) (duration: 06m 28s) [02:18:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:22:26] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Nov 26 02:22:26 UTC 2016 (duration 4m 18s) [02:22:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:24:01] PROBLEM - puppet last run on ms-fe3001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:53:01] RECOVERY - puppet last run on ms-fe3001 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [03:54:01] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_statistics_mediawiki] [04:00:21] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_statistics_mediawiki] [04:17:41] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=3481.70 Read Requests/Sec=520.60 Write Requests/Sec=5.20 KBytes Read/Sec=44885.60 KBytes_Written/Sec=156.40 [04:23:41] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=143.00 Read Requests/Sec=299.90 Write Requests/Sec=126.00 KBytes Read/Sec=4131.70 KBytes_Written/Sec=1000.00 [04:46:21] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 652 600 - REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3978921 keys, up 25 days 20 hours - replication_delay is 652 [04:47:21] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3961754 keys, up 25 days 20 hours - replication_delay is 0 [04:54:48] 06Operations, 10Traffic: repeated 503 errors for 90 minutes now - https://phabricator.wikimedia.org/T146451#2824148 (10doctaxon) 05Resolved>03Open Hi Joe! Now I get a very similar error since about two weeks (or longer), but this 502 Bad Gateway happens several time within 5 minutes(!) doing simple API qu... [04:55:59] 06Operations, 10Traffic: repeated 503 errors for 90 minutes now - https://phabricator.wikimedia.org/T146451#2824150 (10doctaxon) **Unbreak now! due to vulnerable security bug** (see above) [04:58:21] PROBLEM - puppet last run on neodymium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:21:21] RECOVERY - puppet last run on neodymium is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:31:11] PROBLEM - puppet last run on db1091 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:59:11] RECOVERY - puppet last run on db1091 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:19:11] PROBLEM - puppet last run on db1016 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:47:12] RECOVERY - puppet last run on db1016 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:51:51] PROBLEM - puppet last run on analytics1052 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[tzdata] [07:09:21] PROBLEM - puppet last run on cp3004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:19:51] RECOVERY - puppet last run on analytics1052 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [07:38:21] RECOVERY - puppet last run on cp3004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [08:32:10] (03PS2) 10Tim Landscheidt: toollabs: install opencv-data (trusty, jessie) [puppet] - 10https://gerrit.wikimedia.org/r/303416 (https://phabricator.wikimedia.org/T142321) (owner: 10Merlijn van Deen) [08:34:29] (03CR) 10Tim Landscheidt: [C: 031] toollabs: install opencv-data (trusty, jessie) [puppet] - 10https://gerrit.wikimedia.org/r/303416 (https://phabricator.wikimedia.org/T142321) (owner: 10Merlijn van Deen) [08:57:11] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:25:11] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [10:21:01] PROBLEM - puppet last run on ms-fe3002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:32:14] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 3297 MB (3% inode=99%) [10:36:14] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 2825 MB (2% inode=99%) [10:38:14] PROBLEM - Disk space on stat1002 is CRITICAL: DISK CRITICAL - free space: /tmp 3584 MB (3% inode=99%) [10:40:22] i am randomly getting https error (mentioning wrong configuration of the server, hsts) on wmf sites. is that a known issue? [10:44:59] elukey: I just had a quick look at stat1002. Most space on /tmp is taken by vectors_* files belonging to ellery. I have no idea what those are, so I'm not going to clean them. Do you have any idea? [10:49:05] RECOVERY - puppet last run on ms-fe3002 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [10:51:04] Danny_B|webgate: I've been getting such errors for several weeks now [10:51:38] There was a discussion on #wikimedia-tech [11:12:34] PROBLEM - puppet last run on labtestweb2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [11:42:34] RECOVERY - puppet last run on labtestweb2001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [12:11:54] PROBLEM - puppet last run on es1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:24:05] PROBLEM - puppet last run on maerlant is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:27:04] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [12:29:12] !log manually fixed the checkout of mediawiki core on stat1002 and stat1003 that was causing Puppet failing [12:29:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:24] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [12:39:54] RECOVERY - puppet last run on es1017 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [12:52:04] RECOVERY - puppet last run on maerlant is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [13:24:54] PROBLEM - puppet last run on db1070 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:33:15] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824383 (10Paladox) Adding #operations as this seems like this will require gerrit's server to be accessed to fix this. [13:36:24] !log ran refreshLinks on angwiki for T151584, it ran into issues with the EventBus extension at the links tables step [13:36:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [13:36:35] T151584: Old English Wikipedia (angwiki) reports good redirects as broken redirects - https://phabricator.wikimedia.org/T151584 [13:43:02] will debug later, probably just listens to the hook without taking into account that it can be run through maint script [13:45:24] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [13:52:09] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824399 (10Paladox) I found this https://bugs.chromium.org/p/gerrit/issues/detail?id=1954 and seems to be the error as we are having. the error should be in the logs. /var/l... [13:52:54] RECOVERY - puppet last run on db1070 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [13:59:12] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824401 (10Paladox) Re cloning worked for me. [14:13:24] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [14:14:44] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824410 (10Paladox) [14:15:49] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Paladox) There seems to be a fix for gc in jgit 4.3.1 that looks like something that will fix our problem. Note I am not 100% correct but hopefully it will. https:... [14:27:18] 06Operations, 10Traffic: repeated 503 errors for 90 minutes now - https://phabricator.wikimedia.org/T146451#2824419 (10Aklapper) 05Open>03Resolved > Now I get a very similar error since about two weeks (or longer), but this 502 Bad Gateway @DocTaxon: This task is about 503 errors. [[ https://mediawiki.org... [14:36:33] gehel: not at all, thanks for taking care of it. Please follow up with ellery since he was probably doing some work [14:36:52] I followed up with him in the past due to big home dir [14:37:12] maybe some specific user limit enforced could be good [14:44:03] elukey: I did not actually do anything (except send an email to ellery) [14:44:56] elukey: it looks like some of his files in /tmp are pretty old, my guess is that there some missing cleanup, but I did not really want to do that without knowing what this was about... [14:47:44] elukey, gehel: I've fixed puppet on stat1002/3 earlier, but was unrelated [14:47:53] just FYI [14:48:01] volans: thanks! [14:48:49] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824459 (10Reedy) Quick workaround is: ``` cp .git/config .git/config.backup git remote remove origin mv .git/config.backup .git/config git fetch ``` Thanks to @Volans on P4510 [14:50:46] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10ArielGlenn) guess I should add the info from the logs here: /var/lib/gerrit2/review_site/logs/error_log, the first relevant thing seems to be [2016-11-26 00:44:... [14:54:24] PROBLEM - puppet last run on dbproxy1009 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:01:28] paladox: re:paste yw, I've just searched a bit around for solutions ;) [15:03:35] volans :) [15:03:46] volans possibly updating to 2.13 will fix it [15:03:48] gerrit 2.13 [15:03:57] but it is a 50 50 chance [15:04:04] not a 100% sure. [15:04:09] lol [15:04:31] volans i found this [15:04:32] https://projects.eclipse.org/projects/technology.jgit/releases/4.3.1 [15:04:37] * Scan loose ref before packed in case gc is about to remove the loose refs [15:04:46] have no idea ^^ if that fix is what we need. [15:05:07] paladox: When you read the extended commit you linked me... [15:05:21] https://git.eclipse.org/c/jgit/jgit.git/commit/?id=05fd01656ee295e4a8d3c1efff0ca3a6f48c1f4c [15:05:30] yep [15:06:24] yep, looks like it might be related [15:06:51] volans we will be upgrading very soon [15:07:04] ostriches uploaded the patch for review, ive done the testing [15:07:07] with gerrit-test3 [15:07:16] all ssh keys are stored in git now. [15:08:02] 06Operations, 10DBA, 10Wikidata, 10netops, 07Performance: DispatchChanges: Avoid long-lasting connections to the master DB - https://phabricator.wikimedia.org/T151681#2824480 (10daniel) [15:13:04] volans apergos gave us some logs [15:13:05] https://phabricator.wikimedia.org/T151676#2824462 [15:13:23] Looking at it, it does make it looks like the fix in jgit 4.3.1 will fix it [15:13:42] lol [15:13:43] a602a8b2400730a8d9a0d20b79bfb55ec0de4d86 [15:13:47] that commit was abandoned [15:16:18] gehel: checking then [15:23:24] RECOVERY - puppet last run on dbproxy1009 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [15:30:26] gehel: will try to compress some files and see how it goes [15:31:34] PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:35:14] RECOVERY - Disk space on stat1002 is OK: DISK OK [15:35:53] !log deleted tmp files on stat1002's /tmp partition because of disk space consumption. Will follow up with the owner. [15:36:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:00:34] RECOVERY - puppet last run on cp3032 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:08:14] PROBLEM - puppet last run on elastic1029 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:20:22] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824506 (10Reedy) p:05Unbreak!>03High [16:20:44] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824332 (10Reedy) Lowered slightly; beta and CI have been unbroken, still some gerrit related stuff to be done [16:28:54] PROBLEM - puppet last run on elastic1042 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:37:14] RECOVERY - puppet last run on elastic1029 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:56:54] RECOVERY - puppet last run on elastic1042 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [17:03:35] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit: git pull on mw core broken on beta - https://phabricator.wikimedia.org/T151676#2824559 (10Paladox) @Reedy had to delete the local zuul clone for ci to unbreak it's self. Clones will take longer for all mw extensions then usual. [17:09:06] (03CR) 10Daniel Kinzler: "Wasn't langlink sorting factored out into a separate extension? Shouldn't we be using that?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/323556 (https://phabricator.wikimedia.org/T111023) (owner: 10Aude) [17:16:24] (03PS1) 10Reedy: Disable git gc as source of breakages [puppet] - 10https://gerrit.wikimedia.org/r/323655 (https://phabricator.wikimedia.org/T151676) [17:17:36] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824565 (10Reedy) [17:19:17] (03CR) 10Paladox: Disable git gc as source of breakages (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/323655 (https://phabricator.wikimedia.org/T151676) (owner: 10Reedy) [17:20:04] (03CR) 10Reedy: "Could do. But that's what git revert is for anyway ;)" [puppet] - 10https://gerrit.wikimedia.org/r/323655 (https://phabricator.wikimedia.org/T151676) (owner: 10Reedy) [17:21:20] (03CR) 10Paladox: [C: 031] "Oh ok." [puppet] - 10https://gerrit.wikimedia.org/r/323655 (https://phabricator.wikimedia.org/T151676) (owner: 10Reedy) [18:13:14] PROBLEM - puppet last run on mw1280 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:42:14] RECOVERY - puppet last run on mw1280 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [18:45:14] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:46:07] 06Operations, 10Traffic: several 502 Bad Gateway - https://phabricator.wikimedia.org/T151686#2824625 (10doctaxon) [18:47:00] (03CR) 10Chad: [C: 031] "Let's land this ASAP." [puppet] - 10https://gerrit.wikimedia.org/r/323655 (https://phabricator.wikimedia.org/T151676) (owner: 10Reedy) [18:53:36] 06Operations, 10Traffic: several 502 Bad Gateway - https://phabricator.wikimedia.org/T151686#2824659 (10doctaxon) p:05Triage>03High [19:00:24] PROBLEM - puppet last run on alsafi is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:06:48] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824660 (10demon) a:03demon [19:14:14] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [19:29:24] RECOVERY - puppet last run on alsafi is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [19:32:34] (03CR) 10MarcoAurelio: "> which specific rights?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322094 (https://phabricator.wikimedia.org/T150951) (owner: 10Reedy) [19:39:00] (03CR) 10Reedy: "Ditto. There's always gonna be more to add, and we can do that, not a problem." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322094 (https://phabricator.wikimedia.org/T150951) (owner: 10Reedy) [20:22:24] PROBLEM - puppet last run on mw1218 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:50:24] RECOVERY - puppet last run on mw1218 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [20:52:20] 06Operations, 10Beta-Cluster-Infrastructure, 10Gerrit, 13Patch-For-Review: gerrit jgit gc caused mediawiki/core repo problems - https://phabricator.wikimedia.org/T151676#2824703 (10demon) Ran `git fsck --full --strict` on all repos. Most were fine. A lot of these are probably fine too, but here's the resul... [21:21:15] (03CR) 10BryanDavis: "> Shouldn't it be the other way around?" [puppet] - 10https://gerrit.wikimedia.org/r/323351 (https://phabricator.wikimedia.org/T136849) (owner: 10BryanDavis) [21:33:03] 06Operations, 10Mail: not being able to send emails via Special:EmailUser (Yahoo / GMail) - https://phabricator.wikimedia.org/T137337#2824757 (10Legoktm) [21:33:14] 06Operations, 10Mail, 10Wikimedia-General-or-Unknown, 13Patch-For-Review, 05Security: DMARC: Users cannot send emails via a wiki's [[Special:EmailUser]] - https://phabricator.wikimedia.org/T66795#2824751 (10Legoktm) [21:33:24] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 603 600 - REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 4091913 keys, up 26 days 13 hours - replication_delay is 603 [21:36:24] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 4071357 keys, up 26 days 13 hours - replication_delay is 0 [21:49:34] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:17:34] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [22:18:58] (03CR) 10Gergő Tisza: "We'll probably want to keep the list around to flag log events (badpassword etc) as interesting, though." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322094 (https://phabricator.wikimedia.org/T150951) (owner: 10Reedy) [22:26:14] 06Operations, 10scap, 06Release-Engineering-Team (Long-Lived-Branches): Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824817 (10demon) I also want `git fsck --name-objects` [22:27:52] 06Operations, 10scap, 06Release-Engineering-Team (Long-Lived-Branches): Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824819 (10Paladox) We could just build git our selfs by creating a git repo in gerrit and then adding the debian/ folder to build a deb for Jessie... [22:39:05] 06Operations, 10scap, 06Release-Engineering-Team (Long-Lived-Branches): Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824822 (10demon) >>! In T140927#2824819, @Paladox wrote: > We could just build git our selfs by creating a git repo in gerrit and then adding the d... [22:39:59] 06Operations, 10scap, 06Release-Engineering-Team (Long-Lived-Branches): Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2824823 (10Paladox) >>! In T140927#2824822, @demon wrote: >>>! In T140927#2824819, @Paladox wrote: >> We could just build git our selfs by creating... [23:23:34] PROBLEM - puppet last run on cp3046 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:31:04] (03CR) 10Huji: [C: 04-1] "Comment out the line that references T14655 and then add a new line that would set the value to true referencing T66795. That way it would" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322243 (https://phabricator.wikimedia.org/T66795) (owner: 10Legoktm) [23:33:24] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:37:09] (03PS2) 10Huji: Set $wgUserEmailUseReplyTo = true; [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322243 (https://phabricator.wikimedia.org/T66795) (owner: 10Legoktm) [23:38:34] (03PS3) 10Huji: Set $wgUserEmailUseReplyTo = true; [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322243 (https://phabricator.wikimedia.org/T66795) (owner: 10Legoktm) [23:51:34] RECOVERY - puppet last run on cp3046 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [23:55:11] (03CR) 10Legoktm: [C: 04-1] "No, I'd rather not do that, if people want to look at the history, they can use git log or git blame, we shouldn't keep commented out old " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322243 (https://phabricator.wikimedia.org/T66795) (owner: 10Legoktm)