[00:01:19] 06Labs: Track labs instances hanging - https://phabricator.wikimedia.org/T141673#2507307 (10bd808) Before deployment-logstash2 was restarted for the 2016-07-27 hang it had this console output via wikitech: {P3606} [05:21:04] PROBLEM - Puppet staleness on tools-worker-1008 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [05:27:03] PROBLEM - Puppet staleness on tools-worker-1010 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [05:29:35] PROBLEM - Puppet staleness on tools-worker-1005 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [05:30:25] PROBLEM - Puppet staleness on tools-worker-1019 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [43200.0] [05:30:59] PROBLEM - Puppet staleness on tools-worker-1004 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [05:31:05] PROBLEM - Puppet staleness on tools-worker-1015 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [05:31:35] PROBLEM - Puppet staleness on tools-worker-1014 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [05:37:40] PROBLEM - Puppet staleness on tools-worker-1007 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [05:42:22] PROBLEM - Puppet staleness on tools-worker-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [05:45:28] PROBLEM - Puppet staleness on tools-worker-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [43200.0] [05:45:28] PROBLEM - Puppet staleness on tools-worker-1017 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [43200.0] [05:45:34] PROBLEM - Puppet staleness on tools-worker-1016 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [43200.0] [05:45:48] PROBLEM - Puppet staleness on tools-worker-1011 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [05:45:52] PROBLEM - Puppet staleness on tools-worker-1018 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0] [05:47:03] PROBLEM - Puppet staleness on tools-worker-1012 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [05:47:52] PROBLEM - Puppet staleness on tools-worker-1009 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [43200.0] [05:51:54] PROBLEM - Puppet staleness on tools-docker-builder-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0] [07:05:45] ^ is known and accounted for, nobody panic. [21:05:31] 10Quarry: Make a Quarry automatically refresh on a set time interval - https://phabricator.wikimedia.org/T141698#2508803 (10Viswaprabha) [21:28:15] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Zppix was created, changed by Zppix link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Zppix edit summary: Created page with "{{Tools Access Request |Justification=Assisting, and creating bots that will help out mediawiki and/or the wikis ran by wikimedia foundation |Completed=false |User Name=Zppix }}" [22:34:28] 06Labs, 10Labs-Infrastructure, 07Tracking: Labs instances sometimes freeze - https://phabricator.wikimedia.org/T124133#2508932 (10hashar) [22:36:18] 06Labs, 10Labs-Infrastructure, 07Tracking: Labs instances sometimes freeze - https://phabricator.wikimedia.org/T124133#2508938 (10hashar) [23:38:11] PROBLEM - SSH on tools-docker-builder-03 is CRITICAL: Server answer