[00:25:25] RECOVERY - puppet last run on labnet1002 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:57:25] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:06:25] PROBLEM - puppet last run on ms-be1023 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:25:25] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:35:25] RECOVERY - puppet last run on ms-be1023 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [01:45:53] (03PS1) 10Reedy: Add python3-pil for ConfirmEdit captcha generation [puppet] - 10https://gerrit.wikimedia.org/r/337248 [01:47:06] (03CR) 10jerkins-bot: [V: 04-1] Add python3-pil for ConfirmEdit captcha generation [puppet] - 10https://gerrit.wikimedia.org/r/337248 (owner: 10Reedy) [01:49:23] (03PS2) 10Reedy: Add python3-pil for ConfirmEdit captcha generation [puppet] - 10https://gerrit.wikimedia.org/r/337248 [01:54:45] PROBLEM - puppet last run on iridium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:58:28] Puppet failed on phabricators host it seems ^^ [01:58:35] twentyafterfour ^^ [02:19:37] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.11) (duration: 07m 12s) [02:19:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:23:45] RECOVERY - puppet last run on iridium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [02:25:22] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 12 02:24:56 UTC 2017 (duration 5m 20s) [02:25:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:40:25] PROBLEM - puppet last run on elastic1044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:45:35] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:10:25] RECOVERY - puppet last run on elastic1044 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:14:35] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:32:55] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPRegion.dat.gz] [03:33:45] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIPRegion.dat.gz] [03:55:55] PROBLEM - puppet last run on db1072 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:00:46] RECOVERY - puppet last run on eeden is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:01:45] RECOVERY - puppet last run on analytics1037 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:15:15] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=511.60 Read Requests/Sec=347.70 Write Requests/Sec=27.20 KBytes Read/Sec=43124.00 KBytes_Written/Sec=217.20 [04:23:15] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=239.80 Read Requests/Sec=154.90 Write Requests/Sec=11.20 KBytes Read/Sec=3173.20 KBytes_Written/Sec=569.20 [04:24:55] RECOVERY - puppet last run on db1072 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [04:47:45] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:16:45] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:26:05] PROBLEM - citoid endpoints health on scb1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:26:05] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:26:15] PROBLEM - citoid endpoints health on scb1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:26:15] PROBLEM - restbase endpoints health on restbase1017 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:26:55] RECOVERY - citoid endpoints health on scb1003 is OK: All endpoints are healthy [06:26:55] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [06:27:05] RECOVERY - citoid endpoints health on scb1004 is OK: All endpoints are healthy [06:27:05] RECOVERY - restbase endpoints health on restbase1017 is OK: All endpoints are healthy [07:20:05] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2315 [07:25:05] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1089352 Threads: 1 Questions: 23185919 Slow queries: 5745 Opens: 8351 Flush tables: 1 Open tables: 575 Queries per second avg: 21.284 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [07:33:45] PROBLEM - puppet last run on rdb1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [07:43:35] PROBLEM - puppet last run on druid1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [08:01:45] RECOVERY - puppet last run on rdb1007 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [08:11:35] RECOVERY - puppet last run on druid1003 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [08:36:45] PROBLEM - puppet last run on elastic1033 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:05:45] RECOVERY - puppet last run on elastic1033 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [10:17:35] PROBLEM - puppet last run on lvs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:45:35] RECOVERY - puppet last run on lvs1004 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [11:52:05] Hi all, is it possible to proxy through bast1001 using PuTTY? Trying to connect to stat1003 [12:05:12] <_joe_> samtar: google has a lot of options for that, among which https://monkeyswithbuttons.wordpress.com/2010/10/01/ssh-proxycommand-and-putty/, but caveat emptor: it's 10 years I don't touch a windows machine if not for deinstalling the thing. [12:17:24] looks like I'm going to just end up using Linux in a vm, thanks anyway _joe_ [12:18:52] samtar hi, you can use git for windows. That includes ssh and it works really good for me. [12:25:46] paladox: okay so that works.. do you happen to know if/where it puts the .ssh/config? [12:30:10] samtar it should be /c/Users//.ssh/ [12:31:34] samtar do you have windows 10? [12:32:12] 8.1, but I've found it and it's pretty much working (.ssh/config file etc - just getting `no such identity` but that's probably down to my SSH key) [12:33:46] oh [12:34:03] samtar you should upgrade to windows 10 :), it includes ubuntu now [12:34:18] https://msdn.microsoft.com/en-gb/commandline/wsl/about [12:34:39] so no more vm's needed to run linux unless you want to have the gui. [12:35:30] https://www.microsoft.com/en-us/accessibility/windows10upgrade?tduid=(14c348e893a91089cb2980533c6ee123)(256380)(2459594)(TnL5HPStwNw-0FBZfNm6YPFXNhiu3o52rA)() [12:35:42] https://www.cnet.com/uk/how-to/microsoft-windows-10-free-upgrade-offer-assistive-features/ [12:36:17] ^ hey that's pretty neat.. [12:36:44] yep, samtar it was free for everyone at the begging of launch. [12:37:01] though last year the offer ended in july, but not for https://www.microsoft.com/en-us/accessibility/windows10upgrade?tduid=(14c348e893a91089cb2980533c6ee123)(256380)(2459594)(TnL5HPStwNw-0FBZfNm6YPFXNhiu3o52rA)() [12:37:19] microsoft is aware that everyone can still get it and havent made it harder to get :) [12:37:36] it's free and it's better then windows 8.1 with the start button back. [12:41:45] samtar just to also let you know about ubuntu, microsoft did not change anything in the os, it is the offical image from ubuntu :), you can use grep, ssh, most of the features too including sudo apt-get. [13:12:11] right that's all working, can SSH to stat1003.eqiad.wmnet through the bastion - get prompted for the passphrase for the SSH key all okay, but then it prompts for a normal `Password` [13:12:16] nothing I enter there works [13:28:38] samtar: ssh -vvv, see if it refused your key [13:29:20] -vvv is basically very very verbose [13:51:35] PROBLEM - puppet last run on db1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [14:05:08] (03PS1) 10Ladsgroup: dumps: Centeralize CSS in one file, make it wider and apply to more files [puppet] - 10https://gerrit.wikimedia.org/r/337264 (https://phabricator.wikimedia.org/T155697) [14:12:21] 06Operations, 10Ops-Access-Requests, 13Patch-For-Review: Request for access to stat1003 for Sam Tarling - https://phabricator.wikimedia.org/T157483#3020007 (10Samtar) 05Resolved>03Open @RobH looks like there may have been an issue with the patch. The above converted public key and the added key to https:... [14:19:35] RECOVERY - puppet last run on db1024 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:15:45] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:15:45] PROBLEM - puppet last run on mw1254 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:18:16] !log reedy@tin Synchronized php-1.29.0-wmf.11/extensions/ConfirmEdit/maintenance: Instrumentation to script (duration: 00m 41s) [15:18:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:29:15] PROBLEM - puppet last run on cp3036 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:40:15] PROBLEM - puppet last run on cp3044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:43:45] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [15:43:45] RECOVERY - puppet last run on mw1254 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:57:15] RECOVERY - puppet last run on cp3036 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [16:08:15] RECOVERY - puppet last run on cp3044 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:10:55] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:37:55] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [18:10:15] PROBLEM - puppet last run on cp3044 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [18:39:15] RECOVERY - puppet last run on cp3044 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [18:43:58] 06Operations, 06Labs: Mount /public/dumps for osmit project - https://phabricator.wikimedia.org/T156586#3020364 (10Sabas88) [19:50:05] 06Operations, 10MediaWiki-Cache, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020380 (10Reedy) [19:58:05] PROBLEM - puppet last run on cp3008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:09:48] (03PS1) 10Hashar: Role to provide Gerrit ssh host key on port 29418 [puppet] - 10https://gerrit.wikimedia.org/r/337283 (https://phabricator.wikimedia.org/T157912) [20:09:50] (03PS1) 10Hashar: wikidatabuilder: ship Gerrit ssh host key via a role [puppet] - 10https://gerrit.wikimedia.org/r/337284 [20:14:45] PROBLEM - puppet last run on relforge1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:16:10] (03CR) 10jerkins-bot: [V: 04-1] Role to provide Gerrit ssh host key on port 29418 [puppet] - 10https://gerrit.wikimedia.org/r/337283 (https://phabricator.wikimedia.org/T157912) (owner: 10Hashar) [20:20:59] (03PS1) 10Hashar: contint: remove /srv/ssd [puppet] - 10https://gerrit.wikimedia.org/r/337286 [20:27:05] RECOVERY - puppet last run on cp3008 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [20:31:45] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:32:08] (03PS1) 10Hashar: jenkins: merger user/group sub classes [puppet] - 10https://gerrit.wikimedia.org/r/337287 [20:43:45] RECOVERY - puppet last run on relforge1002 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:46:18] (03PS1) 10Hashar: jenkins: sync default file with upstream 1.651.3 [puppet] - 10https://gerrit.wikimedia.org/r/337289 [21:00:45] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [21:00:55] PROBLEM - puppet last run on labsdb1007 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:19:15] PROBLEM - puppet last run on cp4015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:19:28] (03PS1) 10Hashar: jenkins: support variable prefix setting [puppet] - 10https://gerrit.wikimedia.org/r/337307 [21:28:55] RECOVERY - puppet last run on labsdb1007 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [21:30:33] (03CR) 10Hashar: "Bah it fails https://puppet-compiler.wmflabs.org/5423/" [puppet] - 10https://gerrit.wikimedia.org/r/337307 (owner: 10Hashar) [21:35:45] PROBLEM - puppet last run on labvirt1008 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:44:45] (03PS1) 10ArielGlenn: little tool that displays the last page id in bz2 xml content file [dumps/mwbzutils] - 10https://gerrit.wikimedia.org/r/337341 [21:48:15] RECOVERY - puppet last run on cp4015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:00:06] PROBLEM - puppet last run on mw1259 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:03:46] RECOVERY - puppet last run on labvirt1008 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [22:09:05] PROBLEM - carbon-cache@c service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@c is inactive [22:09:45] PROBLEM - carbon-cache@h service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@h is inactive [22:09:45] PROBLEM - carbon-cache@d service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@d is inactive [22:09:45] PROBLEM - carbon-cache@a service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@a is inactive [22:09:45] PROBLEM - carbon-cache@e service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@e is inactive [22:09:45] PROBLEM - carbon-cache@b service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@b is inactive [22:09:55] PROBLEM - carbon-cache@g service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@g is inactive [22:09:55] PROBLEM - carbon-cache@f service on graphite1001 is CRITICAL: CRITICAL - Expecting active but unit carbon-cache@f is inactive [22:27:58] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020474 (10bd808) Is the problem primarily that the Main Page uses `[[{{LOCALDAY}}. {{LOCALMONTHNAME}}]] [[{{LOCALYEAR}}]]`? These are all magic words that... [22:30:05] RECOVERY - puppet last run on mw1259 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [22:32:45] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:32:50] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020477 (10Ijon) Thanks, @bd808! That's helpful insight. I guess the Estonian main page needs to be fixed not to use those. [22:33:09] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020478 (10bd808) trwiki is using `{{#time:Y-m-d}}` which again is incompatible with any type of caching. [22:35:11] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020481 (10bd808) I would suggest, lowering this from UNB! back to High and changing the task summery to make the goal being educating various wiki communi... [22:41:42] + [22:58:15] PROBLEM - Redis replication status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 607 600 - REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3606294 keys, up 104 days 14 hours - replication_delay is 607 [22:59:05] PROBLEM - Redis replication status tcp_6479 on rdb2005 is CRITICAL: CRITICAL: replication_delay is 658 600 - REDIS 2.8.17 on 10.192.32.133:6479 has 1 databases (db0) with 3606294 keys, up 104 days 14 hours - replication_delay is 658 [23:00:45] RECOVERY - puppet last run on elastic1028 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [23:01:05] RECOVERY - Redis replication status tcp_6479 on rdb2005 is OK: OK: REDIS 2.8.17 on 10.192.32.133:6479 has 1 databases (db0) with 3580845 keys, up 104 days 14 hours - replication_delay is 0 [23:01:15] RECOVERY - Redis replication status tcp_6479 on rdb2006 is OK: OK: REDIS 2.8.17 on 10.192.48.44:6479 has 1 databases (db0) with 3580571 keys, up 104 days 14 hours - replication_delay is 0 [23:04:08] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020503 (10MZMcBride) >>! In T119366#3020478, @bd808 wrote: > trwiki is using `{{#time:Y-m-d}}` which again is incompatible with any type of caching. Huh... [23:11:40] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020504 (10MZMcBride) >>! In T119366#3020503, @MZMcBride wrote: > I believe the day magic words/parser functions have some special logic that reduces the p... [23:26:13] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020512 (10kruusamagi) >>! In T119366#3020474, @bd808 wrote: > Is the problem primarily that the Main Page uses `[[{{LOCALDAY}}. {{LOCALMONTHNAME}}]] [[{{L... [23:45:27] 06Operations, 10Traffic, 10Wikimedia-General-or-Unknown: Disable caching on the main page for anonymous users - https://phabricator.wikimedia.org/T119366#3020514 (10bd808) >>! In T119366#3020504, @MZMcBride wrote: >>>! In T119366#3020503, @MZMcBride wrote: >> I believe the day magic words/parser functions ha...