[00:00:05] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:01:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:01:45] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:02:05] RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:02:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 17.533333 [00:02:45] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.099998 [00:03:15] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 60.299999 [00:03:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 195.833328 [00:04:35] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:04:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 44.299999 [00:04:45] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:05:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 295.033325 [00:05:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 214.46666 [00:06:05] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 16.0 [00:06:45] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 38.066666 [00:07:45] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 187.133331 [00:08:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:09:15] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.333332 [00:09:25] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.466667 [00:10:05] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 155.066666 [00:10:15] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:12:05] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:12:05] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.733334 [00:12:15] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 41.299999 [00:12:45] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 27.466667 [00:14:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:15:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:15:15] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:15:56] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.233334 [00:15:56] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:15:56] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.066666 [00:18:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 81.033333 [00:19:45] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:19:45] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:20:15] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 27.566668 [00:22:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:23:45] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:24:15] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:24:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 251.766663 [00:25:05] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 65.866669 [00:25:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 297.333344 [00:25:35] RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:25:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 264.133331 [00:26:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:26:25] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 21.566668 [00:26:35] RECOVERY - Kafka Broker Server on analytics1021 is OK: PROCS OK: 1 process with command name java, args kafka.Kafka /etc/kafka/server.properties [00:26:45] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:26:45] RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:26:55] PROBLEM - Puppet freshness on elastic1017 is CRITICAL: Last successful Puppet run was Sat 07 Jun 2014 21:19:14 UTC [00:27:45] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:27:45] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:28:15] RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:29:35] PROBLEM - Kafka Broker Server on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties [00:29:45] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:30:05] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.0 [00:30:45] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 42.966667 [00:30:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.233334 [00:31:15] RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:32:45] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.700001 [00:33:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:33:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:33:45] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 47.066666 [00:34:15] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 37.533333 [00:34:25] RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:34:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:34:45] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:35:35] PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.5 [00:35:45] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:36:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 40.066666 [00:37:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:37:45] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 127.033333 [00:37:45] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 42.133335 [00:38:45] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:39:05] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:39:45] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 35.400002 [00:40:15] RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:41:15] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 11.0 [00:41:45] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 55.733334 [00:42:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 225.666672 [00:42:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 316.866669 [00:42:35] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:42:45] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 20.0 [00:44:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 298.366669 [00:46:15] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 41.166668 [00:46:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 94.900002 [00:47:05] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:47:25] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 33.599998 [00:48:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:49:05] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 44.133335 [00:49:45] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 228.800003 [00:49:45] PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 104.0 [00:50:15] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:50:45] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:51:16] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:51:16] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:51:45] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:52:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:52:15] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 12.9 [00:52:35] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 23.933332 [00:52:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:53:05] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:53:45] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:53:45] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:54:05] PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 71.800003 [00:55:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 40.833332 [00:56:05] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.866667 [00:56:45] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:57:25] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [00:59:15] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 7.666667 [01:00:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 273.866669 [01:00:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 299.033325 [01:00:45] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 29.333334 [01:01:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 327.700012 [01:01:45] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:02:45] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 32.099998 [01:03:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:05:05] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 14.866667 [01:05:15] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:06:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 58.266666 [01:06:45] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.200001 [01:08:05] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 225.733337 [01:08:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:08:25] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.5 [01:09:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:09:35] RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:09:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:10:45] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.066668 [01:10:45] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:11:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 23.166666 [01:12:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:12:45] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:13:45] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:14:45] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 197.100006 [01:15:15] RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:17:15] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 14.4 [01:17:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 371.966675 [01:18:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 279.100006 [01:19:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 287.700012 [01:19:45] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:20:35] PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 20.299999 [01:20:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 31.566668 [01:20:45] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 32.866665 [01:20:45] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1.6 [01:21:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:21:15] RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:22:25] RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:22:45] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:24:05] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:24:05] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.833334 [01:24:45] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:25:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:25:45] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:25:45] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 22.6 [01:26:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:27:15] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 17.566668 [01:27:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:27:45] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:28:15] RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:28:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:28:45] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.533333 [01:30:45] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 39.200001 [01:30:45] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:31:35] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:33:15] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.533333 [01:33:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 356.666656 [01:34:05] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 5.266667 [01:34:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 398.433319 [01:34:25] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 20.733334 [01:34:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 78.633331 [01:35:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 377.533325 [01:35:45] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.099998 [01:35:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.066666 [01:36:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:37:15] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:37:45] RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:39:45] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 55.466667 [01:40:05] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:40:15] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.533333 [01:40:35] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 20.033333 [01:41:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:42:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:42:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.766666 [01:43:05] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:43:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:45:45] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:46:45] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 223.199997 [01:47:25] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:47:45] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:48:15] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 19.066668 [01:50:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 311.266663 [01:50:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:51:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 244.233337 [01:51:45] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:52:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 257.0 [01:52:45] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:52:45] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:53:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:53:45] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 50.900002 [01:55:05] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 8.466666 [01:55:45] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 65.166664 [01:57:05] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 13.9 [01:57:15] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:57:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 8.366667 [01:57:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.066666 [01:57:45] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:58:05] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 203.399994 [01:58:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:58:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:58:25] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 9.4 [01:58:45] PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 92.933334 [01:58:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:58:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:58:45] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [01:59:55] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [02:01:45] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:02:45] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 60.5 [02:04:35] RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:05:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 36.5 [02:06:45] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 43.066666 [02:07:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 472.033325 [02:07:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 373.566681 [02:07:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 478.166656 [02:07:45] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 14.733334 [02:08:17] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 32.799999 [02:09:15] RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:09:45] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:11:45] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 10.2 [02:12:45] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:12:45] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:12:45] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 209.300003 [02:14:15] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:15:15] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:15:35] PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 12.633333 [02:15:45] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:15:49] !log LocalisationUpdate completed (1.24wmf7) at 2014-06-09 02:14:46+00:00 [02:15:57] Logged the message, Master [02:17:45] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:18:45] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 57.466667 [02:18:45] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:18:45] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:19:15] RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:19:45] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 60.099998 [02:19:45] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 108.433334 [02:20:05] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:20:25] RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:20:45] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:22:16] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 13.733334 [02:22:45] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:23:15] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 499.633331 [02:23:45] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 596.633362 [02:24:15] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 428.133331 [02:24:45] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:25:45] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 164.0 [02:27:45] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 50.133335 [02:28:05] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:28:05] RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:28:45] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 63.466667 [02:29:05] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:29:11] !log LocalisationUpdate completed (1.24wmf8) at 2014-06-09 02:28:08+00:00 [02:29:15] RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:29:16] Logged the message, Master [02:30:13] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:30:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:30:43] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 37.033333 [02:31:03] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1.2 [02:31:23] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:31:43] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:32:13] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:32:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:32:43] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.933332 [02:33:14] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 22.266666 [02:33:44] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:34:23] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.933333 [02:34:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:34:43] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:35:03] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.933332 [02:36:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:36:43] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:38:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:39:13] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 490.033325 [02:39:43] PROBLEM - Varnishkafka Delivery Errors on cp1056 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 418.133331 [02:40:13] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:40:23] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 517.06665 [02:40:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:40:43] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 108.833336 [02:41:13] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 10.0 [02:41:43] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:41:43] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 235.566666 [02:41:43] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 35.099998 [02:42:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:42:43] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:42:43] RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:43:13] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 12.266666 [02:43:43] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:43:43] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 59.700001 [02:44:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:45:03] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 233.666672 [02:45:13] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:45:43] RECOVERY - Varnishkafka Delivery Errors on cp1056 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:46:03] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:46:23] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:46:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:47:43] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.066667 [02:48:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:48:43] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 39.666668 [02:48:43] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:48:43] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:50:13] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 16.266666 [02:50:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:51:23] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:52:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:52:43] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 71.433334 [02:53:13] PROBLEM - Varnishkafka Delivery Errors on cp1057 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 371.166656 [02:53:43] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:54:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:54:43] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:55:14] RECOVERY - Varnishkafka Delivery Errors on cp1057 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:55:23] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 430.033325 [02:56:43] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 02:28:09 UTC [02:56:43] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 27.4 [02:57:43] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:57:53] RECOVERY - Puppet freshness on analytics1016 is OK: puppet ran at Mon Jun 9 02:57:47 UTC 2014 [02:58:13] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.466667 [02:58:43] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [02:59:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 40.966667 [03:03:17] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:03:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:03:27] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 11.133333 [03:03:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:03:47] PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 95.433334 [03:03:47] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 5.2 [03:04:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 46.533333 [03:06:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:06:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 79.133331 [03:06:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:07:47] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:08:47] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 223.366669 [03:09:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.0 [03:11:07] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:11:18] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 396.833344 [03:11:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2.666667 [03:12:17] RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:12:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:13:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:13:57] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:14:47] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:15:17] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 22.6 [03:15:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 74.300003 [03:16:07] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 88.26667 [03:17:47] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 10.6 [03:18:07] PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 73.199997 [03:19:17] RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:19:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:19:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 43.033333 [03:19:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:20:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 120.533333 [03:21:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:22:47] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:24:11] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jun 9 03:23:05 UTC 2014 (duration 23m 4s) [03:24:16] Logged the message, Master [03:24:17] PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 5.466667 [03:24:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:26:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 18.666666 [03:27:37] PROBLEM - Puppet freshness on elastic1017 is CRITICAL: Last successful Puppet run was Sat 07 Jun 2014 21:19:14 UTC [03:28:07] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:28:17] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 366.266663 [03:28:49] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 58.799999 [03:29:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:29:57] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 321.299988 [03:30:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:30:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 49.299999 [03:32:17] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.466667 [03:32:47] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 279.833344 [03:33:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:34:17] RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:34:27] RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:34:47] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 78.066666 [03:35:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:35:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:36:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 21.466667 [03:37:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 85.133331 [03:40:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 117.699997 [03:41:07] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 5.533333 [03:41:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:42:47] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:42:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 30.966667 [03:42:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:43:17] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 467.566681 [03:43:17] RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:44:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:45:47] RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:46:07] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:48:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 53.799999 [03:49:17] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.4 [03:49:27] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.0 [03:49:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 84.033333 [03:50:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:50:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 46.766666 [03:51:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:52:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:52:47] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:53:48] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:55:07] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:55:17] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:55:47] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 62.233334 [03:56:57] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:57:07] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1.833333 [03:57:17] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 10.533334 [03:58:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.466667 [03:58:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 80.133331 [03:58:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:58:47] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [03:59:07] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 54.333332 [03:59:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 45.033333 [04:00:17] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 482.333344 [04:02:07] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:02:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:02:47] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 242.699997 [04:03:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:05:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:06:17] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.733333 [04:06:47] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:06:47] PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 94.633331 [04:06:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 68.566666 [04:07:17] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:07:47] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2.066667 [04:08:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:09:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.233334 [04:10:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:10:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 91.533333 [04:10:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 52.633335 [04:12:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:13:57] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 250.199997 [04:14:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:16:07] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.733333 [04:17:05] (03PS1) 10Ori.livneh: Puppet compiler for Tim's redirects.dat DSL [operations/puppet] - 10https://gerrit.wikimedia.org/r/138292 [04:17:18] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 450.733337 [04:18:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 89.966667 [04:19:17] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 14.166667 [04:19:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 142.166672 [04:20:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 30.200001 [04:20:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:21:17] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:22:47] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:23:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:24:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:24:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:24:47] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 253.266663 [04:25:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:25:47] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:27:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 36.700001 [04:29:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 30.333334 [04:29:47] RECOVERY - Kafka Broker Server on analytics1021 is OK: PROCS OK: 1 process with command name java, args kafka.Kafka /etc/kafka/server.properties [04:30:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:30:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 55.799999 [04:30:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 16.666666 [04:32:17] RECOVERY - Varnishkafka Delivery Errors on amssq59 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:32:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:32:47] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 9.766666 [04:32:47] PROBLEM - Kafka Broker Server on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties [04:33:17] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3.366667 [04:34:17] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 386.700012 [04:35:47] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:36:07] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:37:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:37:47] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 40.333332 [04:37:54] hrerr drerr [04:38:17] RECOVERY - Varnishkafka Delivery Errors on amssq52 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:38:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 89.133331 [04:38:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 109.633331 [04:39:57] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:40:07] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.266666 [04:41:17] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:42:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:43:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.466667 [04:43:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:44:07] RECOVERY - Varnishkafka Delivery Errors on amssq56 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:44:15] !log umount /dev/sdf on analytics1021, fs in r/o mode, kafka broker not running. no checks yet [04:44:21] Logged the message, Master [04:44:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:45:17] PROBLEM - Varnishkafka Delivery Errors on amssq59 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 23.0 [04:45:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:47:07] RECOVERY - Varnishkafka Delivery Errors on cp1054 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:47:47] RECOVERY - Varnishkafka Delivery Errors on cp1065 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:47:56] (03CR) 10Ori.livneh: [C: 031] Allow searching NS_MODULE with mwgrep [operations/puppet] - 10https://gerrit.wikimedia.org/r/138216 (owner: 10Hoo man) [04:48:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:48:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 38.666668 [04:49:17] PROBLEM - Varnishkafka Delivery Errors on cp1070 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 465.066681 [04:50:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:50:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 70.366669 [04:50:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 34.299999 [04:50:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 97.466667 [04:52:18] PROBLEM - Varnishkafka Delivery Errors on amssq52 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1.266667 [04:52:18] RECOVERY - Varnishkafka Delivery Errors on cp1070 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:53:47] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 221.766663 [04:54:15] @externals [04:54:15] Krinkle: [operations/mediawiki-config.git] Checked out HEAD: e6bf6beb80d69110f91817fe05d7c2d9def43f85 - https://git.wikimedia.org/commit/operations%2Fmediawiki-config.git/e6bf6beb80d69110f91817fe05d7c2d9def43f85 [04:54:17] @replag [04:54:17] Krinkle: No replag currently. See also "replag all". [04:54:17] @replag all [04:54:19] Krinkle: [s1] db1052: 0s, db1055: 0s, db1051: 0s, db1061: 0s, db1062: 0s, db1065: 0s, db1066: 0s, db1070: 0s, db1071: 0s; [s2] db1024: 0s, db1018: 0s, db1002: 0s, db1009: 0s, db1036: 0s, db1060: 0s, db1063: 0s, db1067: 0s; [s3] db1038: 0s, db1035: 0s, db1003: 0s, db1019: 0s, db1027: 0s [04:54:20] Krinkle: [s4] db1040: 0s, db1042: 0s, db1004: 0s, db1049: 0s, db1056: 0s, db1059: 0s, db1064: 0s, db1068: 0s; [s5] db1058: 0s, db1005: 0s, db1026: 0s, db1021: 0s, db1037: 0s, db1045: 0s; [s6] db1023: 0s, db1022: 0s, db1030: 0s, db1010: 0s, db1015: 0s, db1006: 0s; [s7] db1033: 0s, db1041: 0s, db1028: 0s, db1034: 0s, db1039: 0s [04:54:57] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 288.333344 [04:55:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 102.533333 [04:55:47] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:56:07] PROBLEM - Varnishkafka Delivery Errors on amssq56 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 9.033334 [04:56:27] RECOVERY - Varnishkafka Delivery Errors on amssq55 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:56:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 43.633335 [04:56:47] RECOVERY - Kafka Broker Server on analytics1021 is OK: PROCS OK: 1 process with command name java, args kafka.Kafka /etc/kafka/server.properties [04:57:17] RECOVERY - Varnishkafka Delivery Errors on amssq49 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [04:58:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:00:37] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [05:01:43] (03PS1) 10Springle: Add a clean coredb-equivalent configuration MariaDB role. Use it on db1020 as a slave for m2 not handling real traffic. [operations/puppet] - 10https://gerrit.wikimedia.org/r/138296 [05:02:04] (03PS1) 10Ori.livneh: refreshDomainRedirects: fix off-by-one error [operations/apache-config] - 10https://gerrit.wikimedia.org/r/138297 [05:02:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:02:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:02:47] RECOVERY - Varnishkafka Delivery Errors on cp3011 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:03:07] RECOVERY - Varnishkafka Delivery Errors on amssq58 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:03:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:04:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 31.966667 [05:04:47] PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 0.0 [05:04:47] PROBLEM - Kafka Broker Replica Lag on analytics1021 is CRITICAL: kafka.server.ReplicaFetcherManager.Replica-MaxLag.Value CRITICAL: 1890022638.0 [05:04:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:06:17] RECOVERY - Varnishkafka Delivery Errors on amssq57 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:06:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:07:47] PROBLEM - Varnishkafka Delivery Errors on cp1065 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 88.633331 [05:08:25] (03CR) 10Springle: [C: 032] Add a clean coredb-equivalent configuration MariaDB role. Use it on db1020 as a slave for m2 not handling real traffic. [operations/puppet] - 10https://gerrit.wikimedia.org/r/138296 (owner: 10Springle) [05:08:47] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 44.444443 [05:08:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 24.433332 [05:09:47] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:10:47] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 55.933334 [05:10:47] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 27.433332 [05:11:27] PROBLEM - Varnishkafka Delivery Errors on amssq55 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 13.866667 [05:11:47] PROBLEM - Varnishkafka Delivery Errors on cp3011 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2.1 [05:12:07] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:12:17] PROBLEM - Varnishkafka Delivery Errors on amssq49 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 15.866667 [05:13:47] PROBLEM - Varnishkafka Delivery Errors on cp1053 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 376.433319 [05:14:07] PROBLEM - Varnishkafka Delivery Errors on amssq58 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.5 [05:14:47] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 87.833336 [05:15:07] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 27.466667 [05:15:16] !log xtrabackup clone db1046 to db1020 [05:15:20] Logged the message, Master [05:16:17] RECOVERY - Varnishkafka Delivery Errors on amssq51 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:16:47] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 104.199997 [05:21:17] PROBLEM - Varnishkafka Delivery Errors on amssq57 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.166667 [05:21:47] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:22:47] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:22:57] RECOVERY - Varnishkafka Delivery Errors on cp1066 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:23:07] RECOVERY - Varnishkafka Delivery Errors on amssq53 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:23:57] PROBLEM - Disk space on db1020 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=96%): [05:24:47] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:25:47] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:27:17] PROBLEM - Varnishkafka Delivery Errors on amssq51 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 0.6 [05:27:47] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:27:47] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 5.266667 [05:27:47] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:28:27] RECOVERY - Varnishkafka Delivery Errors on amssq54 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:28:47] RECOVERY - Varnishkafka Delivery Errors on cp1055 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:29:47] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 26.6 [05:31:41] RECOVERY - Varnishkafka Delivery Errors on cp1067 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:31:41] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:31:41] PROBLEM - Varnishkafka Delivery Errors on cp4018 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 28.5 [05:32:21] RECOVERY - Disk space on db1020 is OK: DISK OK [05:32:31] PROBLEM - mysqld processes on db1020 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [05:32:41] PROBLEM - Varnishkafka Delivery Errors on cp4010 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 20.566668 [05:32:42] PROBLEM - Varnishkafka Delivery Errors on cp4016 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 37.400002 [05:36:11] PROBLEM - Varnishkafka Delivery Errors on amssq53 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 4.2 [05:36:41] PROBLEM - Varnishkafka Delivery Errors on cp4008 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 80.633331 [05:36:51] RECOVERY - Varnishkafka Delivery Errors on amssq62 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:38:01] PROBLEM - Varnishkafka Delivery Errors on cp1054 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 51.433334 [05:38:41] PROBLEM - Varnishkafka Delivery Errors on cp4009 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 48.733334 [05:40:01] PROBLEM - Varnishkafka Delivery Errors on cp1066 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 496.799988 [05:40:21] PROBLEM - Varnishkafka Delivery Errors on amssq54 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.4 [05:43:41] PROBLEM - Varnishkafka Delivery Errors on cp1067 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 58.433334 [05:44:21] RECOVERY - Varnishkafka Delivery Errors on amssq50 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:44:41] RECOVERY - Varnishkafka Delivery Errors on cp4017 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:45:14] (03PS1) 10Faidon Liambotis: Remove varnishkafka delivery errors check [operations/puppet] - 10https://gerrit.wikimedia.org/r/138302 [05:45:37] (03CR) 10Faidon Liambotis: [C: 032] Remove varnishkafka delivery errors check [operations/puppet] - 10https://gerrit.wikimedia.org/r/138302 (owner: 10Faidon Liambotis) [05:46:41] RECOVERY - Varnishkafka Delivery Errors on cp4002 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:46:51] PROBLEM - Varnishkafka Delivery Errors on amssq62 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 7.533333 [05:47:11] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:48:41] RECOVERY - Varnishkafka Delivery Errors on cp1053 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:48:41] RECOVERY - Varnishkafka Delivery Errors on cp4018 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:48:49] (03CR) 10Faidon Liambotis: [C: 04-1] "I don't think we need multiple versions per package. I'm 100% sure no sysadmin would ever use that (or even look for it, as the standard D" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [05:49:41] RECOVERY - Varnishkafka Delivery Errors on cp4016 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:50:42] RECOVERY - Varnishkafka Delivery Errors on cp4008 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:50:42] RECOVERY - Varnishkafka Delivery Errors on cp4010 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:51:11] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 13.066667 [05:51:41] PROBLEM - Varnishkafka Delivery Errors on cp1055 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 156.233337 [05:52:41] PROBLEM - Varnishkafka Delivery Errors on cp4017 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 28.566668 [05:53:42] PROBLEM - Varnishkafka Delivery Errors on cp4002 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 3.2 [05:54:51] RECOVERY - Varnishkafka Delivery Errors on cp4009 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [05:56:17] PROBLEM - Varnishkafka Delivery Errors on amssq50 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 6.2 [05:59:55] bbye varnishkafka errors [06:00:11] * paravoid puts head in the sand [06:00:47] 'SPF|cloud' heh [06:01:35] Something wrong with it? ;) [06:03:46] nothing at all; it's fun :) [06:08:54] (03PS2) 10Matanya: labs: lint, clean and remove heartbleed leftover [operations/puppet] - 10https://gerrit.wikimedia.org/r/138282 [06:11:16] (03CR) 10Faidon Liambotis: "I'd like to hear from Tim, as I believe he was the one that made the MaxClient decisions in the past. +1 for actually documenting (in code" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137947 (owner: 10Ori.livneh) [06:14:34] (03CR) 10Faidon Liambotis: Icinga: Check Dispatch command for Wikidata notification (036 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [06:15:01] <_joe_> paravoid: shouldn't you be on holiday? [06:15:11] meh :) [06:16:41] <_joe_> :/ [06:28:07] PROBLEM - Puppet freshness on elastic1017 is CRITICAL: Last successful Puppet run was Sat 07 Jun 2014 21:19:14 UTC [06:50:21] ACKNOWLEDGEMENT - Puppet freshness on elastic1017 is CRITICAL: Last successful Puppet run was Sat 07 Jun 2014 21:19:14 UTC Giuseppe Lavagetto taken out of rotation due to io issues [06:51:02] ACKNOWLEDGEMENT - ElasticSearch health check on elastic1017 is CRITICAL: CRITICAL - Could not connect to server 10.64.48.39 Giuseppe Lavagetto taken out of rotation because of high io waits [07:07:37] PROBLEM - Disk space on analytics1021 is CRITICAL: DISK CRITICAL - free space: / 1034 MB (3% inode=96%): [07:14:37] RECOVERY - Disk space on analytics1021 is OK: DISK OK [07:14:54] !log disabled puppet on analytics1021 to avoid kafka broker restarting with missing mount [07:14:59] Logged the message, Master [07:15:30] (_joe_: yes, i'll ack that in icinga^ ;) [07:15:57] PROBLEM - Kafka Broker Server on analytics1021 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args kafka.Kafka /etc/kafka/server.properties [07:16:20] <_joe_> springle: :P [07:26:40] hi. sorry i missed the varnishkafka trouble. [07:27:50] <_joe_> jgage: hey, me too [07:27:51] hm still looks unhappy. *pokes* [07:28:12] <_joe_> I don't have any idea on how it works atm [07:28:17] <_joe_> so I won't touch it [07:29:30] jgage: it's definietly still unhappy. paravoid silenced the icinga noise, and i merely unmounted a failed disk on analytics1021 and stopped puppet and the broker doing a restart loop [07:29:49] nothing else has been done that i'm aware of [07:30:50] ouch, failed disk on broker [07:31:00] i guess that won't be getting fixed tonight [07:31:05] I like how we have these super resilient redundant systems [07:31:11] that crumble when a single disk fails :) [07:31:14] heh [07:31:27] :) [07:31:28] <_joe_> call me maybe [07:31:41] there is a plan to add more broker capacity, which would handle this case if i understand correctly [07:31:45] <_joe_> we don't even need a network partition [07:32:14] <_joe_> how many broker machines we do have? [07:32:18] 2 [07:32:19] well the second broker came up right? the disk wasn't the direct cause of the icinga noise [07:32:39] or s/came/stayed/ up [07:32:54] one broker does not have sufficient capacity [07:33:04] <_joe_> springle: if we need more that the capacity of one broker... [07:33:10] (03CR) 10Legoktm: Icinga: Check Dispatch command for Wikidata notification (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [07:33:11] <_joe_> as jgage said :) [07:33:14] ah ok [07:34:07] ok i've got a meeting in 8.5 hours, back to bed for me. l8r international crew! [07:34:42] <_joe_> bye [07:35:17] jgage: what's the verdict? [07:35:27] jgage: can you send a message to the list with your findings? [07:40:34] :/ [07:41:45] paravoid: i'll respond to greg-g's earlier email to the list [08:01:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [08:05:45] (03CR) 10Christopher Johnson (WMDE): Icinga: Check Dispatch command for Wikidata notification (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [08:15:24] (03PS9) 10Christopher Johnson (WMDE): Icinga: Check Dispatch command for Wikidata notification [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 [08:28:27] <_joe__> is there any op around (apart from me)? [08:28:35] yes? [08:29:57] no _joe__, you can't do the same thing you do every night… [08:30:21] he probably meant except those who should be resting on a beach or couch or whatever [08:31:09] <_joe__> :) [08:31:14] <_joe__> Nemo_bis: exactly [08:53:16] (03PS10) 10Christopher Johnson (WMDE): Icinga: Check Dispatch command for Wikidata notification [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 [09:24:37] RECOVERY - mysqld processes on db1020 is OK: PROCS OK: 1 process with command name mysqld [09:42:37] (03PS3) 10Filippo Giunchedi: add mini-dinstall to releases.wikimedia.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 [09:42:55] (03CR) 10Filippo Giunchedi: add mini-dinstall to releases.wikimedia.org (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [09:58:33] (03PS1) 10Giuseppe Lavagetto: monitoring: add check for puppet-merge [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 [10:24:04] (03CR) 10Filippo Giunchedi: [C: 04-1] monitoring: add check for puppet-merge (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 (owner: 10Giuseppe Lavagetto) [10:26:20] (03PS1) 10Giuseppe Lavagetto: redis: restart service upon first install [operations/puppet] - 10https://gerrit.wikimedia.org/r/138317 [10:30:30] (03CR) 10Giuseppe Lavagetto: monitoring: add check for puppet-merge (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 (owner: 10Giuseppe Lavagetto) [10:31:07] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 07:30:38 UTC [10:35:24] _joe_: hello, question - wasn't start cert been revoked ? [10:35:34] *star [10:35:59] <_joe_> matanya: probably the old one - we renewed that and the old one should be revoked [10:36:43] so star.wmflabs.org is still valid, right ? [10:37:41] <_joe_> I honestly have no idea, also it depends on what cert are you talking about [10:37:55] <_joe_> can I know the original problem you're trying to troubleshoot? [10:38:05] <_joe_> I could help you more, and faster [10:41:11] I'm not sure, looked at manifest/role/labsproxy.pp and saw it has this cert, i thought it was revoked, hence asking. [10:43:19] <_joe_> no idea honestly [10:43:41] thanks _joe_ i'll ask rohh [10:43:47] *robh [10:46:23] (03CR) 10Giuseppe Lavagetto: monitoring: add check for puppet-merge (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 (owner: 10Giuseppe Lavagetto) [10:46:41] the star wikimedia one was I believe [10:46:50] afaik labs has its own certs [10:47:08] which are completely separate [10:51:44] (03PS1) 10Matanya: lucene: admins on node level, remove redundant admin class [operations/puppet] - 10https://gerrit.wikimedia.org/r/138321 [10:52:20] they are p858snake|l_ but not sure which one was revoked [11:02:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [11:20:28] <_joe_> If I ever had to merge something in mw-config on tin, how would I do that? [11:20:39] <_joe_> is that done with a specific script? [11:21:03] <_joe_> (not that I want to do that, just to be sure I'm not missing pieces while working something out) [11:29:47] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Mon Jun 9 11:29:45 UTC 2014 [12:53:43] (03PS2) 10Giuseppe Lavagetto: monitoring: add check for git merging of important repos [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 [12:56:41] (03PS3) 10Giuseppe Lavagetto: monitoring: add check for git merging of important repos [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 [12:58:11] (03CR) 10Aude: [C: 031] "tried running the script on ubuntu and it works without me having to install anything additional." [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [13:30:08] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I think the idea is sound, but we still need to balance it." (032 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/137947 (owner: 10Ori.livneh) [13:37:40] (03PS2) 10Giuseppe Lavagetto: access_new_install: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/138277 (owner: 10Matanya) [13:40:20] (03CR) 10Filippo Giunchedi: monitoring: add check for git merging of important repos (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 (owner: 10Giuseppe Lavagetto) [13:40:51] (03CR) 10Giuseppe Lavagetto: [C: 032] access_new_install: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/138277 (owner: 10Matanya) [13:48:07] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 10:47:36 UTC [13:51:14] (03CR) 10Giuseppe Lavagetto: monitoring: add check for git merging of important repos (033 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 (owner: 10Giuseppe Lavagetto) [13:51:46] (03PS4) 10Giuseppe Lavagetto: monitoring: add check for git merging of important repos [operations/puppet] - 10https://gerrit.wikimedia.org/r/138313 [14:00:05] manybubbles, ^d: The time is nigh to deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140609T1400) [14:03:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [14:10:17] PROBLEM - MySQL Processlist on db1002 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 0 copy to table, 76 statistics [14:11:17] RECOVERY - MySQL Processlist on db1002 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 1 statistics [14:11:37] <_joe_> that has been faster than me :) [14:17:17] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Mon Jun 9 14:17:08 UTC 2014 [14:37:28] manybubbles: I'll take the SWAT this morning [14:37:33] anomie: thanks! [14:42:05] Hello! I'm running Wikimania this year, and I'd like to get Visual Editor deployed onto the Wikimania 2014 wiki, how can I go about this? (http://wikimania2014.wikimedia.org) [14:42:55] James_F: ^ [14:43:46] (03CR) 10Andrew Bogott: [C: 032] labs: lint, clean and remove heartbleed leftover [operations/puppet] - 10https://gerrit.wikimedia.org/r/138282 (owner: 10Matanya) [14:43:49] edsanders suggested I ask here. [14:44:09] edsaperia: James_F is the person to poke, I think :) [14:44:32] Yeah, he does this regularly [14:44:33] edsaperia: Hmm; thought I'd done that? Yeah, will do it now. [14:44:48] anomie: Config change coming for the SWAT. [14:45:10] ottomata: can you put elasticsearch 1.2.1 into apt? Its time to start the upgrade! [14:45:36] anomie: Bah, belay that, Parsoid needs config too. [14:45:48] edsaperia: This will have to take a day or so, sorry. [14:46:44] No super rush, but I will soon be messing around with some large and complex tables, so it'd be handy :) [14:47:25] yeah on it manybubbles... [14:47:28] James_F: Might be good to get a window for that anyway, enabling VE for a new wiki seems more complex than might be good for SWAT. [14:47:29] thanks! [14:47:59] anomie: Really? It's never needed a window before… [14:48:26] was verifying that the 1.2 url was good for 1.2.x [14:48:32] shoudl be [14:49:09] (03PS1) 10Ottomata: Include elasticsearch 1.2.x in reprepro updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/138343 [14:49:22] (03PS2) 10Ottomata: Include elasticsearch 1.2.x in reprepro updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/138343 [14:49:29] (03CR) 10Ottomata: [C: 032 V: 032] Include elasticsearch 1.2.x in reprepro updates [operations/puppet] - 10https://gerrit.wikimedia.org/r/138343 (owner: 10Ottomata) [14:50:27] aude: ping, SWAT in 10 minutes [14:52:27] paravoid: just to double-check… the [filter:tempauth] section in the swift proxy conf is the canonical representation of user accounts, right? [14:53:36] yay [14:53:53] there we go [14:53:54] manybubbles: http://apt.wikimedia.org/wikimedia/pool/main/e/elasticsearch/ [14:54:11] thanks! [14:54:26] !log starting Elasticsearch upgrade with elastic1001 [14:54:37] Logged the message, Master [14:56:28] (03CR) 10Manybubbles: [C: 032] Update plugins for Elasticsearch 1.2.1 [operations/software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/137376 (owner: 10Manybubbles) [14:56:32] (03CR) 10Manybubbles: [V: 032] Update plugins for Elasticsearch 1.2.1 [operations/software/elasticsearch/plugins] - 10https://gerrit.wikimedia.org/r/137376 (owner: 10Manybubbles) [14:57:00] !log syncing elasticsearch plugins for 1.2.1 - any elasticsearch restart from here on out needs to come with 1.2.1 or the node will break. [14:57:05] Logged the message, Master [14:57:17] manybubbles: poke about he.wiki regression, when you have time ... [14:57:52] matanya: yeah! I know about it..... sorry for not jumping on it. I'm going to do an upgrade today and be stuck in adminstrative hell, reviews, expense reports, I think. [14:58:00] after that, I'll crawl out and try to work on it [14:58:38] thanks a lot [14:58:55] James_F: outreach wiki is going away? [14:59:32] aude: No, just thought-o when I made that commit months ago. [14:59:42] s/months/weeks/ [15:00:02] ah, ok [15:00:04] manybubbles, anomie: The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140609T1500) [15:00:37] James_F: I'll do yours first, the bugfixes then the ptwiki config change. aude: Then yours after that. [15:00:41] * aude wouldn't mind if it got merged with meta wiki though [15:00:45] anomie: ok [15:00:56] anomie: Cool. [15:01:06] aude: Yeah, I think it's pretty useless, but… [15:01:10] !log successfully synced plugins, upgrading elastic1001 to make sure everything is working ok with it - then we'll run through the others more quickly [15:01:15] Logged the message, Master [15:04:03] (03PS1) 10Jforrester: Enable VisualEditor by default on Wikimania 2014 wiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138345 [15:05:43] !log anomie Synchronized php-1.24wmf8/extensions/VisualEditor/modules/ve-mw/: SWAT: VE fix for focus regression and alignment issues [[gerrit:137971]] [[gerrit:138122]] (duration: 00m 14s) [15:05:46] James_F: ^ Test please (wmf8) [15:05:47] Logged the message, Master [15:06:18] anomie: Doing. [15:06:50] !log beta updating all instances to puppet 3 via a cherry-pick of https://gerrit.wikimedia.org/r/#/c/137898/ on deployment-salt [15:06:55] Logged the message, Master [15:08:50] anomie: Sorry, still waiting for the new version to actually deploy. Yay for bits caching. :-( [15:09:08] !log anomie Synchronized php-1.24wmf7/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: VE fix for focus regression [[gerrit:137978]] (duration: 00m 15s) [15:09:13] Logged the message, Master [15:09:18] James_F: ^ wmf7 is ready to test too [15:09:59] anomie: (Finally!) Confirmed fixed in wmf8. [15:10:34] * anomie notes mw1151 is still "permission denied", assumes it's still the same brokenness as last week [15:10:49] anomie: And confirmed in wmf7. [15:10:58] James_F: ok! Moving on to the config patch [15:11:13] (03PS2) 10Anomie: Enable TemplateData GUI on Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137986 (https://bugzilla.wikimedia.org/66255) (owner: 10Jforrester) [15:11:18] (03CR) 10Anomie: [C: 032] Enable TemplateData GUI on Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137986 (https://bugzilla.wikimedia.org/66255) (owner: 10Jforrester) [15:11:24] (03Merged) 10jenkins-bot: Enable TemplateData GUI on Portuguese Wikipedia [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/137986 (https://bugzilla.wikimedia.org/66255) (owner: 10Jforrester) [15:12:11] !log anomie Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateData GUI on Portuguese Wikipedia [[gerrit:137986]] (duration: 00m 14s) [15:12:13] James_F: ^ Test please [15:12:16] Logged the message, Master [15:12:17] aude: You're next [15:12:34] !log mw1151 still "permission denied" during deploys [15:12:35] anomie: Working great. Thanks! [15:12:37] Logged the message, Master [15:12:51] ready to check [15:13:51] aude: FYI, as requested I'll touch all .js files in extensions/Wikidata before syncing it. [15:14:09] perfect [15:17:24] !log anomie Synchronized php-1.24wmf8/extensions/Wikidata: SWAT: Wikidata entity suggester bug fixes [[gerrit:138339]] (duration: 00m 16s) [15:17:27] aude: ^ Test please [15:17:30] Logged the message, Master [15:17:31] ok [15:18:11] both issues fixed [15:18:15] thanks! [15:18:19] * anomie is done with SWAT [15:18:25] :) [15:18:26] thanks anomie [15:21:22] greg-g: No problem. BTW, a question that came up earlier: would enabling VE on wikimania2014wiki be ok for SWAT, or is VE too complicated? I lean towards "too complicated", but I may be being overcautious. [15:22:02] James_F: ^ [15:22:18] (03PS3) 10QChris: Redirect https traffic from old metrics sites to wikimetrics [operations/puppet] - 10https://gerrit.wikimedia.org/r/133089 (https://bugzilla.wikimedia.org/64276) [15:22:53] those are getting easier, right? [15:23:08] those == enabling VE on wikis [15:23:25] (03CR) 10QChris: "> Qchris, is this ready to go?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/133089 (https://bugzilla.wikimedia.org/64276) (owner: 10QChris) [15:24:18] greg-g: It's a one-line change to visualeditor.dblist. [15:24:22] anomie: technical side I'm not too worried, community side definitely, but then for wikimaniawikis, that shouldn't be too hard to get consensus [15:25:59] greg-g: Ok, works for me. [15:28:32] !log elastic1001 went well, doing 1002 by hand again [15:28:37] Logged the message, Master [15:30:06] morning manybubbles, sorry about this weekend [15:30:17] greg-g: thanks for calling me - sorry to bother you [15:31:03] (03CR) 10Jforrester: [C: 04-1] "Depends on I9f8d3e3e in parsoid-deploy." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138345 (owner: 10Jforrester) [15:56:38] I'm going to head home from my coffee shop - the elasticearch upgrade is going fine - its doing 1002 now and I'll start the next one when I get back [15:58:37] (03PS11) 10Christopher Johnson (WMDE): Icinga: Check Dispatch command for Wikidata notification [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 [16:00:12] (03PS2) 10coren: labs: default all clients to puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137898 (owner: 10Giuseppe Lavagetto) [16:01:11] <_joe_> Coren: I'm going off in a few - if you need me for the puppet3 switch just poke me here, if I do not answer, my mobile is on :) [16:01:18] who fixed varnishkafka, and what was happening to it? (just out of curiosity) [16:01:29] _joe_: I just rebased it and was about to +2 it through. [16:01:44] But I don't think you need to just sit around. :-() [16:01:51] <_joe_> Coren: coordinate with andrewbogott [16:02:01] <_joe_> he is upgrading beta now I think [16:02:35] Coren: I just updated beta, want to wait an hour or so before merging on labs in general. [16:02:46] andrewbogott: kk; I'm standing by then. [16:02:49] <_joe_> ok, see you later/tomorrow guys, ikea furniture waits for me :/ [16:03:14] Just in case there are unintended consequences, give users a chance to squawk [16:04:06] _joe_: Happy shøppïng! [16:04:44] Coren: Meanwhile I'll build some fresh images [16:04:56] andrewbogott: w/ 3 by default? [16:05:00] yep [16:05:04] Good show. [16:06:55] hm, actually maybe that's easier if I wait until all instances are upgraded [16:09:46] (03CR) 10Anomie: "I don't really know python, but the logic seems ok to me." [operations/puppet] - 10https://gerrit.wikimedia.org/r/138216 (owner: 10Hoo man) [16:15:46] (03CR) 10GWicke: "@faidon: There are some practical use cases for multiple versions, both externally (nightlies) and potentially internally (rolling upgrade" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [16:26:17] (03CR) 10Aude: [C: 031] Icinga: Check Dispatch command for Wikidata notification [operations/puppet] - 10https://gerrit.wikimedia.org/r/136095 (owner: 10Christopher Johnson (WMDE)) [16:28:40] ori so I found this https://github.com/wikimedia/operations-puppet/blob/production/modules/gdash/lib/puppet/parser/functions/ordered_json.rb seems awesome. I struggled with this issue for awhile on friday, and came to much poorer solutions than that. any objection to moving this function to a more generic place so I can use it for phabricator? [16:29:45] chasemp: none at all, thanks for compliment [16:30:18] !log upgrading elastic1003 - upgrade is going well so far so I'm going to stop watching it as closely and let it be more automated [16:30:23] Logged the message, Master [16:30:35] it took me awhile to wrap my head around the fact that hashes in ruby 1.8 can't do this natively [16:31:10] chasemp: yeah. it's very annoying, no? [16:32:18] yes very and just downright strange [16:33:50] MatmaRex: disk failure on analytics1021, caused the disk to be remounted as read-only, java freaked b/c it wasn't able to write [16:34:16] MatmaRex: s.pringle and p.aravoid took it out of circulation and disabled the alert [16:34:37] heh [16:35:08] what does it do? it's clearly not very important if you could afford having it broken for two days :P [16:35:29] it's an analytics thing [16:35:48] it is the message bus for varnish req logs [16:36:53] * ori will brb [16:48:10] ottomata: with regard to elastic1017-1019, did they used to have ssds we might be able to get back in them? [16:48:29] THEY DID! [16:48:31] we took them out [16:48:37] WHY DID WE DO THAT!? [16:48:53] problem though: i think cmjohnson1 is in dallas this week [16:49:10] which is not good for those elastics or analytics1021 :/ [16:55:45] (03PS1) 10Mwalker: Add Farsi fonts for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138357 [16:56:45] (03PS1) 10John F. Lewis: Allow bureaucrats to add/remove TA on legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138358 (https://bugzilla.wikimedia.org/66377) [16:57:04] ottomata: meh, it can wait, I think [16:57:09] yeah [17:00:42] I hate searching the internet for my problems and finding only my own emails describing my problems..... [17:03:37] (03PS1) 10Mwalker: Add jpegtran dependency for metadata manipulation to OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138361 [17:04:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [17:04:09] !log switching labs to puppet3 [17:04:14] Logged the message, Master [17:04:41] (03CR) 10Andrew Bogott: [C: 032] labs: default all clients to puppet 3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137898 (owner: 10Giuseppe Lavagetto) [17:05:45] (03CR) 10Ori.livneh: [C: 031] Add Farsi fonts for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138357 (owner: 10Mwalker) [17:06:48] andrewbogott: Champagne ? [17:07:08] You and _joe_ deserve some! All I did was throw the switch [17:07:17] (well, and now I'm watching nervously, if that counts as a contribution) [17:07:36] what's the timeline for puppet3 in prod? [17:07:53] hold your horses :) [17:08:17] (03CR) 10Andrew Bogott: "Looks good, just a spot of red on the gerrit diff" (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138281 (owner: 10Matanya) [17:08:45] (03CR) 10Cscott: [C: 031] Add Farsi fonts for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138357 (owner: 10Mwalker) [17:10:11] (03PS2) 10Matanya: labslamp: includes in a sane way [operations/puppet] - 10https://gerrit.wikimedia.org/r/138281 [17:12:08] (03CR) 10Andrew Bogott: [C: 032] labslamp: includes in a sane way [operations/puppet] - 10https://gerrit.wikimedia.org/r/138281 (owner: 10Matanya) [17:14:29] _joe_ (or matanya): Does this look like anything to you: https://dpaste.de/7ip7 [17:14:38] I'm pretty sure that started happening w/puppet 3 just now [17:14:55] there are two others just like that one [17:20:33] (03PS1) 10Andrew Bogott: Specify puppet:///modules/labs_vmbuilder [operations/puppet] - 10https://gerrit.wikimedia.org/r/138365 [17:20:48] matanya: ^ look right to you? [17:22:25] (03CR) 10Andrew Bogott: [C: 032] Specify puppet:///modules/labs_vmbuilder [operations/puppet] - 10https://gerrit.wikimedia.org/r/138365 (owner: 10Andrew Bogott) [17:25:27] (03CR) 10Dzahn: [C: 031] beta: fix scap for videoscalers [operations/puppet] - 10https://gerrit.wikimedia.org/r/137274 (owner: 10BryanDavis) [17:26:52] (03CR) 10Dzahn: [C: 032] Add Farsi fonts for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138357 (owner: 10Mwalker) [17:27:58] yea andrewbogott it was wrong before [17:28:17] matanya: but worked with 2.7? [17:28:22] yes [17:28:34] ok… I wonder how many more of these I'll find today. [17:28:51] shouldn't be many, i fixed most of thise [17:28:56] *those [17:28:56] (03CR) 10Dzahn: [C: 032] "role does not appear to be used (at least in prod) and yep @ 'Firewalls go on node level, not role level'" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:29:28] andrewbogott: bear in mind you might also see modules/files/blah which work with 2.7 and not 3 [17:29:47] well /that/ I can grep for! [17:29:48] * andrewbogott greps [17:29:59] (03CR) 10Dzahn: "why is this still here when git log site.pp has a "Renaming role::haproxy to role::puppetproxy"?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:30:06] hm, nope [17:30:37] yeah, i greped that months ago :) [17:31:22] (03CR) 10Matanya: "see the class role name. that is what was renamed." [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:31:59] (03CR) 10Dzahn: "i see. it was once on brewster, but brewster is gone" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:32:39] (03CR) 10Dzahn: "i think it's confusing when file name does not match role name" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:33:47] (03CR) 10Dzahn: [C: 032] "yep, it's on antimony and the node has base::firewall" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138278 (owner: 10Matanya) [17:34:20] (03PS1) 10Matanya: puppetproxy: match role name to class name [operations/puppet] - 10https://gerrit.wikimedia.org/r/138370 [17:35:00] (03CR) 10Matanya: "https://gerrit.wikimedia.org/r/138370" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138279 (owner: 10Matanya) [17:36:04] (03CR) 10Dzahn: [C: 031] "cool, +1, added Alex" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138370 (owner: 10Matanya) [17:36:14] (03PS2) 10Matanya: puppetproxy: match role name to class name [operations/puppet] - 10https://gerrit.wikimedia.org/r/138370 [17:37:32] (03CR) 10Dzahn: [C: 032] labsbastion: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/138280 (owner: 10Matanya) [17:45:19] springle: what's the status of the indexes for https://gerrit.wikimedia.org/r/#/c/117373/ ? [17:45:45] mutante: does logstash collect dmesg ? [17:50:23] (03CR) 10Tim Landscheidt: labs: lint, clean and remove heartbleed leftover (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138282 (owner: 10Matanya) [17:51:23] (03CR) 10Matanya: labs: lint, clean and remove heartbleed leftover (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138282 (owner: 10Matanya) [17:58:56] !log elastic1004-1006 upgraded without trouble - cluster is working on filling elatic1006 before moving on to 1007, and the rest [17:59:01] Logged the message, Master [18:01:17] PROBLEM - Disk space on virt1000 is CRITICAL: DISK CRITICAL - free space: / 1953 MB (3% inode=89%): [18:01:34] (03CR) 10Dzahn: [C: 032] "replaced by admin yaml" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138321 (owner: 10Matanya) [18:02:11] andrewbogott: ^ [18:02:20] virt1000 disk [18:02:43] matanya: thanks, I will look... [18:04:04] <_joe_> andrewbogott: that should be fixed, atm [18:04:11] <_joe_> sorry, just seen [18:04:18] <_joe_> (I told to ping me in case) [18:04:31] _joe_ you mean the puppet:///modules/blahblah thing? [18:04:34] _joe_: any memcaches in pmtpa? [18:04:43] <_joe_> andrewbogott: yeah [18:04:50] <_joe_> what was your puppetmaster? [18:05:01] _joe_: virt1000 [18:05:06] What do you mean by 'should be fixed'? [18:05:09] <_joe_> andrewbogott: client? [18:05:18] <_joe_> andrewbogott: we had that problem, we fixed it [18:05:20] _joe_ I already patched it so you won't be able to see the issue any more... [18:05:28] Another lesson: When you think "git branch" should have an option to print the tag line of the top commit as well, just look in the man page: "git branch -v". Ah! [18:05:43] _joe_: although I'm seeing the same issue again in a different place [18:05:52] labs-vmbuilder-precise.eqiad.wmflabs [18:05:52] <_joe_> oh sorry, the vmbuilder issue? [18:06:03] Error: /Stage[main]/Passwords::Root/File[/etc/ssh/userkeys/root/.ssh/authorized_keys]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///private/ssh/root-authorized-keys: getaddrinfo: Name or service not known [18:06:14] I'm about to patch that but if you want to look first... [18:06:16] (03PS1) 10Ottomata: Add class role::analytics::refinery::data::drop [operations/puppet] - 10https://gerrit.wikimedia.org/r/138380 [18:06:24] <_joe_> getaddrinfo: Name or service not known [18:06:31] <_joe_> this is a dns problem of some kind [18:06:33] jgage: ^ got a sec to review that one? [18:06:36] https://gerrit.wikimedia.org/r/138380  [18:06:52] (03PS1) 10Dzahn: remove lucene::users adn ::admins remnants [operations/puppet] - 10https://gerrit.wikimedia.org/r/138381 [18:07:06] <_joe_> andrewbogott: sorry it's 8 PM here, if it's not something tragic, seek advice from other opsens if possible :) [18:07:30] (03PS2) 10Ottomata: Add class role::analytics::refinery::data::drop [operations/puppet] - 10https://gerrit.wikimedia.org/r/138380 [18:07:43] andrewbogott: not opsen, but where does it come from? [18:08:04] matanya: labs-vmbuilder-precise.eqiad.wmflabs [18:08:12] <_joe_> matanya: well, you are ;) [18:08:26] <_joe_> sorry, really gotta go [18:08:29] (03CR) 10jenkins-bot: [V: 04-1] remove lucene::users adn ::admins remnants [operations/puppet] - 10https://gerrit.wikimedia.org/r/138381 (owner: 10Dzahn) [18:08:31] (03CR) 10Ottomata: "Hashar, you seem to be the one with the most questions about this, so I'll wait for your +1 before merging." [operations/puppet] - 10https://gerrit.wikimedia.org/r/49678 (owner: 10Ottomata) [18:08:40] matanya: on labs-vmbuilder-precise [18:08:45] sorry if I'm repeating myself, IRC client crashed [18:08:55] i meant the puppet path [18:09:07] manifests/ ? [18:09:40] (03PS2) 10Dzahn: remove lucene::users adn ::admins remnants [operations/puppet] - 10https://gerrit.wikimedia.org/r/138381 [18:10:50] matanya: it's from including the private 'passwords' module [18:10:53] the labs one [18:11:01] oh [18:11:14] * AaronSchulz does whois on 62.99.71.196 [18:11:23] same IP keeps hitting the thumbnail rate limiting [18:12:27] RECOVERY - Disk space on analytics1015 is OK: DISK OK [18:12:38] (03PS1) 10Jgreen: add base::firewall to ocg role classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/138382 [18:12:49] matanya: isn't it just this? https://gerrit.wikimedia.org/r/#/c/138379/ [18:13:26] yes. seems so [18:13:27] RECOVERY - Disk space on analytics1020 is OK: DISK OK [18:14:22] (03CR) 10Dzahn: [C: 04-1] "please add base::fireall on node level, not in roles, add the ferm rules here though" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138382 (owner: 10Jgreen) [18:14:48] matanya: my only confusing is that _joe_ said it 'should be fixed' as though there's some server workaround for that bug... [18:14:57] which, of course it does no harm to make it right anyway [18:15:04] * andrewbogott tries it... [18:15:10] yep, fixed [18:15:20] yeah, from some reason i don't have this file on my puppet clone [18:15:26] matanya: https://gerrit.wikimedia.org/r/#/c/138382/ [18:15:58] mutante: took my words out of my keyboard [18:16:21] matanya: if you have yet more time… log into labs-vmbuilder-trusty-new and tell me what's happening there? [18:17:27] matanya: remember where we had the discussion about putting firewall on nodes ? [18:17:30] Jeff_Green: ^ [18:17:46] on IRC, about a month ago [18:18:00] ok, what's the executive summary [18:18:24] Jeff_Green: firewall is set to drop all [18:18:34] right? [18:18:36] ferm rules in roles, include base::firewall on nodes [18:18:53] if you set it on a role class one will have to dig through all role on the host to find what blocks him [18:19:00] akosiaris: right?^ [18:19:08] correct mutante [18:19:34] ah, right, because some boxes get more than one role [18:19:37] janky [18:19:39] yes [18:19:41] alright. i'll move it [18:19:44] moreover [18:19:56] you can create issues when removing a role [18:20:15] you might remove the forewall from a host which other roles rely on [18:20:20] then all of a sudden it does not include the base anymore for others [18:20:23] that [18:20:51] and things go cabush and tadam and icinga and all that jazz [18:21:02] ... and all that jazz! [18:21:10] matanya: on: https://gerrit.wikimedia.org/r/#/c/138381/2/manifests/role/lucene.pp .. somewhere there was lucene::users before [18:21:11] Jeff_Green: who is that as executive summary ? :) [18:21:25] matanya: good! [18:21:44] mutante: rush took care of it in the past iirc [18:21:48] andrewbogott: no permission [18:21:57] matanya: one sec [18:22:18] matanya: now? [18:22:32] (03CR) 10Dzahn: [C: 032] remove lucene::users adn ::admins remnants [operations/puppet] - 10https://gerrit.wikimedia.org/r/138381 (owner: 10Dzahn) [18:22:51] (03PS2) 10Jgreen: add base::firewall to ocg role classes [operations/puppet] - 10https://gerrit.wikimedia.org/r/138382 [18:23:05] andrewbogott: same bug [18:23:22] matanya: you see "Could not retrieve local facts: undefined method `to_a' for "00000000":String" [18:23:27] the same bug as what? [18:23:35] Error: Could not request certificate: getaddrinfo: Name or service not known [18:23:45] matanya: sudo? [18:23:54] * matanya facepalms [18:24:07] s'ok I was stuck with that same problem for 20 minutes earlier today [18:24:17] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:24:20] now same bug as before [18:24:29] Error: Could not retrieve local facts: undefined method `to_a' for "00000000":String [18:24:29] Error: Failed to apply catalog: Could not retrieve local facts: undefined method `to_a' for "00000000":String [18:24:41] right -- so, you've seen that and/or know how to fix it? [18:24:45] That's a brand new instance btw [18:25:19] * matanya scraches his head and thanks got his company migration to puppet3 went smoothly [18:25:24] *god [18:25:41] I saw that in the past, can't recall where or why [18:26:02] gahhh, stupid tabls vs spaces. my eyes. my eyes. [18:26:40] wait.. the "to_a" thing [18:26:45] that rings a bell [18:26:56] andrewbogott: [18:27:12] there was a change related to that.. hold on [18:27:14] I've seen it before too, but it's always gone away on a second try for me [18:27:39] (03CR) 10Jgreen: [C: 032 V: 031] "gotta get this out of the way b/c I forgot --amend on the spacing fix which will follow" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138382 (owner: 10Jgreen) [18:27:55] (03PS1) 10Jgreen: fix tabs vs spaces, put includes in alpha order [operations/puppet] - 10https://gerrit.wikimedia.org/r/138384 [18:28:14] Jeff_Green: commit message says something else now [18:28:46] yes it's a new commit [18:29:18] i went in to fix the spacing, and got annoyed that it's not alphabetical too :-) [18:29:38] it says you're adding it to role, but you don't [18:29:44] ok, ok [18:30:07] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.014 second response time [18:30:09] oh, i disbelieved that gerrit would tolerate me changing the commit string [18:30:16] so I didn't taunt it [18:30:47] Jeff_Green: you can change it as you like on --amend or click in gerrit web ui, if all you change is the message you can just do in web ui [18:30:52] andrewbogott: trying to find that change.. hrmmm [18:31:25] mutante: I don't trust gerrit not to freak out. [18:31:35] we do that all the time [18:32:04] (03CR) 10Jgreen: [C: 032 V: 031] fix tabs vs spaces, put includes in alpha order [operations/puppet] - 10https://gerrit.wikimedia.org/r/138384 (owner: 10Jgreen) [18:33:27] andrewbogott: modules/base/lib/facter/default_gateway.rb would be a good place to look at :) [18:34:45] andrewbogott: matanya https://gerrit.wikimedia.org/r/#/c/137940/ ? [18:34:52] "in labs I noticed that facter was failing on [18:34:52] this because ruby 1.9 removed "to_a" method from strings" [18:34:58] isn't that the issue you pasted above? [18:35:02] "to_a" method [18:35:12] i rest my case [18:35:23] that's unmerged.. i thought it was [18:35:31] matanya: hah [18:35:57] ruby --version [18:35:58] ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux] [18:36:24] yep, and 1.8.7 on prod it seems [18:36:35] and here is your someking gun. wish to fix it andrewbogott ? [18:37:45] (03PS2) 10Andrew Bogott: update default_gateway.rb module to ruby 1.9 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137940 (owner: 10Filippo Giunchedi) [18:37:47] (03CR) 10Dzahn: [C: 031] "andrewbogott> "Could not retrieve local facts: undefined method `to_a' for "00000000":String"" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137940 (owner: 10Filippo Giunchedi) [18:38:28] (03CR) 10Hoo man: [C: 031] update default_gateway.rb module to ruby 1.9 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137940 (owner: 10Filippo Giunchedi) [18:38:29] * andrewbogott rebases &c [18:39:03] (03CR) 10Dzahn: "@ "there is no 'jenkins-deploy' group in labs", can't we make it?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/135529 (owner: 10Hashar) [18:39:17] (03CR) 10Andrew Bogott: [C: 032] update default_gateway.rb module to ruby 1.9 [operations/puppet] - 10https://gerrit.wikimedia.org/r/137940 (owner: 10Filippo Giunchedi) [18:39:54] of course now the fact is local on that machine so fixing it in the puppet master doesn't help... [18:39:59] (03CR) 10Matanya: "This is not enough - we need to fix modules/stdlib/spec/unit/puppet/parser/functions/range_spec.rb too." [operations/puppet] - 10https://gerrit.wikimedia.org/r/137940 (owner: 10Filippo Giunchedi) [18:40:09] matanya: yus [18:41:19] andrewbogott: still not 100% ready: Error: /Stage[main]/Passwords::Root/File[/etc/ssh/userkeys/root/.ssh/authorized_keys]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/modules/ssh/root-authorized-keys [18:41:59] damn [18:42:10] well, wait, I thought that fix worked… on another box... [18:43:40] not the same error andrewbogott [18:44:03] that one was Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///private/ssh/root-authorized-keys: getaddrinfo: Name or service not known [18:44:13] matanya: https://github.com/puppetlabs/puppetlabs-rsync also needs to be updated [18:44:19] matanya: did you fix that fact on labs-vmbuilder-trusty-new by hand? [18:44:32] no, i haven't [18:44:40] just reran puppet [18:44:54] right hoo that to [18:44:57] o [18:45:19] * matanya sees a long evening ahead [18:46:22] andrewbogott: i'll go wash some dishes and bbl [18:46:32] poke if you wish [18:47:09] matanya: ok. I believe the fix to that is https://gerrit.wikimedia.org/r/#/c/138386/ [18:47:45] also: OMG that is a useless error message! [18:48:54] no andrewbogott same issue [18:49:10] yeah, that's 'cause I am also dumb [18:49:24] https://gerrit.wikimedia.org/r/#/c/138387/ [18:50:02] dammit! [18:50:03] still broken [18:54:27] RECOVERY - Disk space on analytics1013 is OK: DISK OK [18:59:58] !log decomissioning analytics1012 in hadoop cluster, this will become a Kafka broker [19:00:02] Logged the message, Master [19:05:47] PROBLEM - Hadoop NodeManager on analytics1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [19:07:47] RECOVERY - Hadoop NodeManager on analytics1012 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [19:11:21] matanya: New images are (I hope) building… I have to go renew my driver's license so I'll be back in 1-5 hours. [19:11:23] (03PS1) 10Dzahn: rm client/bz-mailer.php [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/138395 [19:13:58] andrewbogott: good luck! [19:21:47] PROBLEM - Hadoop NodeManager on analytics1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager [19:23:05] !log disabling puppet on analytics1012 [19:23:05] Logged the message, Master [19:24:25] (03Abandoned) 10Tim Landscheidt: Fix indentation in and lint role::labs::instance [operations/puppet] - 10https://gerrit.wikimedia.org/r/114734 (owner: 10Tim Landscheidt) [19:36:15] mutante: https://rt.wikimedia.org/Ticket/Display.html?id=7651 [19:36:43] while you are at it ;) [19:42:31] <_joe_> gwicke: if mutante cant handle that, I will do tomorrow morning my time [19:44:14] _joe_: awesome, thanks! [19:51:56] is beta-hhvm going to stay completely broken for a while? [19:52:50] jackmcbarn: yes [19:53:16] jackmcbarn: it's using the 3.1 packages from facebook, and we've caught and fixed many upstream bugs since then [19:53:20] jackmcbarn: but we don't have updated packages yet [19:53:53] why is it serving the contents of the php files right now? [19:54:25] dunno, i'll look [19:57:31] (03PS1) 10Cscott: Add libjpeg-progs for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138408 [19:57:55] gwicke: please give me "A terse phrase identifying this list. [19:58:05] (03CR) 10Cscott: [C: 031] Add jpegtran dependency for metadata manipulation to OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138361 (owner: 10Mwalker) [19:58:32] " services (SOA) related discussions " ? [19:58:45] (03Abandoned) 10Cscott: Add libjpeg-progs for OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138408 (owner: 10Cscott) [19:58:58] mutante: "Services (SOA) and REST API related discussions" [19:59:42] gwicke: thanks, you should have mail with a generated admin pass [20:00:04] gwicke, subbu: The time is nigh to deploy Parsoid (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140609T2000) [20:00:23] mutante: indeed, thanks! [20:01:09] satdav: Now you are here - andre__ - I think they want you :) [20:01:25] Hello is anyone a admin for the bugzilla site [20:01:37] _joe_, ^ [20:01:55] I see its not sending out emails to conform for new accounts [20:02:42] satdav: did you check your spam ? [20:02:52] yes and still not any email [20:03:07] satdav, I am a Bugzilla admin, and the usual first question is about the spam folder, yeah. [20:03:17] satdav, and second question in these days is "Yahoo account?" :) [20:03:35] No its a google apps [20:03:58] andre__, matanya ^ [20:04:00] satdav, can you private /msg me the email address you used for registering? [20:04:21] pm sent [20:04:29] <_joe_> andre__: thanks [20:04:45] _joe_: go to sleep [20:04:51] <_joe_> maybye I should do something like [20:05:03] <_joe|away> :) [20:05:03] satdav: hmm hmm, there is no such account registered yet, yeah [20:05:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [20:05:10] satdav, could you try registering again? :P [20:05:20] satdav, I don't have access to mail logs [20:05:35] Whoa when i tried to register I am getting You have requested an account token too recently to request another. Please wait 10 minutes then try again. [20:05:37] andre__: just tested it myself, and it works well [20:05:43] The work flow of an andre__: Checked Spam? Yahoo? Have you registered? :p [20:05:52] JohnLewis, basically yes :D [20:06:05] when i try and register [20:06:30] andre__, is their a channel or a person who can look at the logs well mail once [20:06:35] satdav: hmm, alright so a token should have been sent theoretically. Now I wonder if our mailservers are a bit busy :) [20:06:36] andre__ :D You deserve the 'debugger' award :p [20:07:01] JohnLewis: problem isn't complicated enough for that ;) [20:07:19] Yet. [20:07:19] andre__, could we be getting blocked [20:07:25] but yeah, really wondering then what's up with the mailservers, as Bugzilla thinks that the token has been sent [20:07:39] andre__: mail servers are fine [20:07:58] hmm, hmm, hmm. [20:08:03] registered and got a mail within seconds [20:08:04] !log deployed Parsoid 9b673587 (deploy sha 7d0097a1) [20:08:13] matanya, can you check the mail logs to see if any emails have been sent to david.weir@mozilla.org.uk [20:08:14] Logged the message, Master [20:08:36] satdav: i'm not even an employee of wmf :) [20:08:53] (03CR) 10Gage: [C: 032] "discussed on IRC, looks good to me" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138380 (owner: 10Ottomata) [20:08:56] whoa but can volunteers not check it [20:09:12] not, need access to servers [20:09:20] matanya: or a volunteer entrusted with access, not all sysadmins are staff :p [20:09:47] AFAIK that is. There are a few who aren't staff that I know of. [20:09:50] afaik, they are all current or previous [20:10:13] root access of course [20:10:44] chasemp: seems to be here, and he has root satdav [20:11:08] chasemp, can you check at your side on the mail servers [20:11:17] reading up [20:11:30] OK [20:12:34] whoa their is a 10 minute delay in the mail today [20:12:37] for google apps [20:12:41] satdav: andre__ , at least one mail has been sent to that address [20:12:52] seems so [20:13:06] thanks. so the problem is likely not on the WMF side [20:13:07] yes I just got it [20:13:18] andre__, I will report it to Mozilla [20:13:36] satdav, what exactly? [20:13:46] ah. google apps 10min. okay [20:13:49] about the delay on the mail [20:13:53] * andre__ shrugs :) [20:14:01] I don't think Mozilla will fix Google :) [20:14:15] mutante: mind helping me clear pmtpa from puppet ? [20:14:36] andre__: Secretly Mozilla is Google >:D [20:14:36] lol they don't but we have a good relatingship [20:15:47] Oh, I had the same problem, it [20:15:55] ... it is a Google delay, as far as I remember. [20:16:22] H=aspmx.l.google.com ..ack [20:17:39] could I get edit bugs if possible [20:18:02] satdav: you have it [20:18:37] oh when i add a bug it comes in as unconformed [20:19:11] I only have bz_canusewhines User can configure whine reports for self [20:19:36] satdav: what's your bugzilla email? [20:19:39] matanya: editbugs and canconfirm are no longer defaults anymore. [20:19:53] david.weir@mozilla.org.uk [20:20:21] satdav: done [20:20:30] thanks MatmaRex [20:21:07] by the way, everyone with editbugs can also add other users to the editbugs group (since last week) [20:21:39] orkut style permissions [20:21:47] MatmaRex: Thanks for the fact :) [20:21:50] thats a good idea [20:22:07] What other permissions do you give to contributors [20:22:08] * MatmaRex doesn't know who satdav is, but anyone smart enough to know what editbugs is definitely deserves it [20:22:16] satdav: on bugzilla? just this one [20:22:29] oh [20:22:33] satdav JohnLewis MatmaRex : http://lists.wikimedia.org/pipermail/wikitech-l/2014-May/076723.html [20:22:34] satdav: you're an unprivileged user, editbugs, admin, or blocked :) [20:23:11] we also have a security group ;) [20:23:34] btw, MatmaRex I hope you enjoy the new right i gave you :D [20:24:49] hoo: Let's keep that secret :p [20:24:54] matanya: hm? [20:25:06] edit interface [20:25:43] oh! sure i do [20:26:25] * MatmaRex has so many hats, i sometimes get lost [20:26:44] MatmaRex, I have been around for a while [20:27:01] is their a qa group [20:27:22] yes satdav [20:27:40] and also a ML for them if you are interested [20:28:15] https://lists.wikimedia.org/mailman/listinfo/qa [20:28:18] :) [20:28:39] no I am meaning on bugzilla [20:29:01] oh, yeah, Wikimedia -> QA [20:29:38] OK [20:29:49] greg-g: I didn't know that list existed; thanks for the link :p [20:29:54] np! [20:30:18] oh hey JohnLewis :) sorry I didn't get back to you last week. I haven't forgot about the NDA thing... [20:31:12] greg-g: It's alright :p [20:31:21] Poked robla yet though? [20:31:49] he's on vacation this week :/ [20:32:32] can I access the wikipedia irc [20:32:41] irc.wikimedia.org I believe [20:33:01] yeah, you just can't talk, it's announce only [20:33:11] Oh [20:33:19] whats the best channel on that to join [20:33:30] greg-g: Urg k :/ [20:33:47] satdav: #en.wikipedia for real spam :p [20:33:57] OK well for me to idle in [20:34:09] satdav: it's just the announcement of recent changes, why would you want to just idle? [20:34:24] (03PS1) 10Ottomata: Add partman recipe for 12 drive Kafka Brokers [operations/puppet] - 10https://gerrit.wikimedia.org/r/138451 [20:34:44] true [20:34:51] so why cant we add it to here [20:35:08] #wikimedia-operations, #wikimedia-tech, #wikimedia-dev are good ones on Freenode for WMF-specific development, also throw in #mediawiki and you're good to go :) [20:35:14] to *here*? [20:35:16] to where? [20:35:43] (03CR) 10Ottomata: "I have no idea if this will work! Could someone check this over big time for me?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138451 (owner: 10Ottomata) [20:36:12] maybe read https://meta.wikimedia.org/wiki/IRC and https://wikitech.wikimedia.org/wiki/Irc.wikimedia.org [20:36:15] greg-gL Why would people want to idle there? Give them something to laugh at with all the spam :D [20:36:25] JohnLewis: yeah, suppose so :) [20:36:46] Who doesn't like enwiki spam [20:38:10] #wikipedia-en [20:38:45] mutante: https://github.com/FluentTradeTechnologies/netconfigit should i post a question to ops list ? [20:39:49] mutante: got a sec to look over this partman recipe with me? [20:40:32] greg-g: i didn't poke you in a long time too [20:40:39] yes yes [20:40:44] :) :) [20:40:59] * matanya is a master in naging [20:42:13] !log upgraded elastic1007-elastic1010 without issue - starting elastic1010 [20:42:19] Logged the message, Master [20:42:47] greg-g: these upgrade days - they give me a good day to file expense reports and write reviews. [20:43:10] I can't really code while I'm doing it because I have to keep looking at the process, but I can do short stuff like this.... [20:43:21] :) [20:45:24] (03PS1) 10Ottomata: Include role::analytics::refinery::data::drop on analytics1026 [operations/puppet] - 10https://gerrit.wikimedia.org/r/138459 [20:49:30] ottomata: sorry, i dont have more for you than "copy and adjust an existing one, try and see" [20:49:42] yeah, that's what I did [20:49:43] well, there was ... [20:49:47] just gotta experiment, eh? [20:50:04] should I get decent error messages when I try if I tail syslog on carbon? [20:50:05] https://wikitech.wikimedia.org/wiki/Partman#Config_language [20:50:09] see the "dissecting" part [20:50:55] ok ja, i've skimmed that, will read deeper [20:51:08] my setup is more complicated than any I've seen yet: [20:51:08] ottomata: if it fails it would fail during intsaller and just stay a black screen, afair [20:51:13] 2 drives different than all the other drives [20:51:28] yay! black screen of 0 information! [20:51:30] ottomata: but in that case what you can do is drop to installer shell [20:51:50] Alt+F1? [20:52:19] hmm, ok [20:52:35] (03CR) 10Ottomata: [C: 032 V: 032] Include role::analytics::refinery::data::drop on analytics1026 [operations/puppet] - 10https://gerrit.wikimedia.org/r/138459 (owner: 10Ottomata) [20:53:30] matanya: yes! [20:59:01] (03PS1) 10Ottomata: Save hive-drop-webrequest-partition cron output in log_dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/138463 [20:59:20] (03CR) 10Ottomata: [C: 032 V: 032] Save hive-drop-webrequest-partition cron output in log_dir [operations/puppet] - 10https://gerrit.wikimedia.org/r/138463 (owner: 10Ottomata) [21:10:57] (03CR) 10Aklapper: [C: 031] rm client/bz-mailer.php [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/138395 (owner: 10Dzahn) [21:12:17] RECOVERY - Disk space on virt1000 is OK: DISK OK [21:13:46] Are we allowed to idle in here [21:14:06] satdav: Yeah of course [21:14:08] Lots of people do [21:16:34] (03CR) 10Dzahn: [C: 032] rm client/bz-mailer.php [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/138395 (owner: 10Dzahn) [21:18:17] (03CR) 10Dzahn: [V: 032] rm client/bz-mailer.php [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/138395 (owner: 10Dzahn) [21:20:03] (03PS2) 10Ottomata: Add partman recipe for 12 drive Kafka Brokers [operations/puppet] - 10https://gerrit.wikimedia.org/r/138451 [21:27:12] ori: seen http://developer.rackspace.com/blog/introducing-loggerfs.html [21:27:12] ? [21:27:24] seems rather cool, although I'm wary of Go + FUSE [22:00:00] (03CR) 10GWicke: "I just played a bit with aptly. Setting up a basic repo didn't take long. The extra features around mirroring & snapshotting are nice, and" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [22:00:53] why does Debian create dependency packages if there is only a single dependency? [22:01:13] and if so, should we use the former or the latter when installing it via puppet [22:01:56] f.e. libjpeg-progs (dependency package) -> libjpeg-turbo-progs (actual package) [22:07:07] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Last successful Puppet run was Mon 09 Jun 2014 19:06:33 UTC [22:09:39] mutante: there were probably several implementations at some point [22:10:03] a dependency package can depend on any of the alternatives [22:10:25] and they are just 12kb or so [22:14:41] (03PS2) 10Krinkle: webperf/deprecate: Log jqmigrate to statsd under mw.js.deprecate [operations/puppet] - 10https://gerrit.wikimedia.org/r/137484 [22:16:00] (03CR) 10jenkins-bot: [V: 04-1] webperf/deprecate: Log jqmigrate to statsd under mw.js.deprecate [operations/puppet] - 10https://gerrit.wikimedia.org/r/137484 (owner: 10Krinkle) [22:17:05] mutante: Did someone change something in master? I rebased that commit and getting pep8 warnings about modules/admin/files/home/rush/bin/fixup.py and modules/puppetmaster/files/puppetsigner.py [22:17:08] I dind't change those. [22:17:24] master is not passing, did someone bypass review? Or maybe pep8 got upgraded? [22:19:38] (03CR) 10LVilla (WMF): "I see that this is merged into the repo, but it doesn't look to be live - any idea when that will happen? If it needs to happen manually, " [operations/puppet] - 10https://gerrit.wikimedia.org/r/134121 (owner: 10Filippo Giunchedi) [22:21:46] (03PS2) 10Ori.livneh: Puppet compiler for Tim's redirects.dat DSL [operations/puppet] - 10https://gerrit.wikimedia.org/r/138292 [22:22:29] (03CR) 10Aklapper: [C: 04-1] "Heja. Sorry that this took me ages. :(" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/129671 (owner: 10Odder) [22:22:44] Krinkle: I can just remove fixup.py it was just utility [22:23:24] (03CR) 10Ori.livneh: "PS2 makes compile_redirects.rb invokable as a standalone CLI script" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138292 (owner: 10Ori.livneh) [22:24:42] (03PS1) 10Rush: chase remote utility regex replace script [operations/puppet] - 10https://gerrit.wikimedia.org/r/138479 [22:24:51] (03PS2) 10Rush: chase remote utility regex replace script [operations/puppet] - 10https://gerrit.wikimedia.org/r/138479 [22:24:59] (03CR) 10Rush: [C: 032 V: 032] chase remote utility regex replace script [operations/puppet] - 10https://gerrit.wikimedia.org/r/138479 (owner: 10Rush) [22:25:01] (03CR) 10Krinkle: "Please don't blindly bypass jenkins-bot without mitigating it in some way. Now every commit proposed against operations/puppet fails the t" [operations/puppet] - 10https://gerrit.wikimedia.org/r/137929 (owner: 10Giuseppe Lavagetto) [22:25:13] (03PS5) 10Tim Landscheidt: Tools: Unify Tools and Toolsbeta configuration [operations/puppet] - 10https://gerrit.wikimedia.org/r/102385 [22:27:19] mutante: https://gerrit.wikimedia.org/r/#/c/137669/ [22:27:49] maybe he forgot to merge that today [22:32:35] AaronSchulz: i wonder why monitoring for job queue is broken [22:32:40] is it really high again? [22:32:46] UNKNOWN: No valid datapoints found [22:32:48] on tungsten [22:36:10] (03CR) 10Dzahn: [C: 032] Use fraction sleep in jobs loop to keep the pipeline thicker [operations/puppet] - 10https://gerrit.wikimedia.org/r/137669 (owner: 10Aaron Schulz) [22:36:50] (03PS1) 10Tim Landscheidt: Tools: Remove last references to pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/138480 [22:38:44] mutante: https://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Miscellaneous%20eqiad&h=terbium.eqiad.wmnet&r=hour&z=default&jr=&js=&st=1400886240&v=1372767&m=Global_JobQueue_length&z=large [22:38:53] no, it's been going down the last few weeks actually [22:39:01] * AaronSchulz wonders how low it will get [22:39:24] that's mostly due to the auto-restarting fixing the pipeline "decay" bug [22:40:51] AaronSchulz: cool! [22:41:09] the icinga monitoring just doesn't get data for some unrelated reason [22:43:07] (03CR) 10Aklapper: "Confirming that both proposals by TTO work as expected (tested locally) and I'd be fine accepting that change (thanks for fiddling that ou" [wikimedia/bugzilla/modifications] - 10https://gerrit.wikimedia.org/r/124140 (https://bugzilla.wikimedia.org/62160) (owner: 1001tonythomas) [22:43:20] (03CR) 10Dzahn: [C: 031] Tools: Remove last references to pmtpa [operations/puppet] - 10https://gerrit.wikimedia.org/r/138480 (owner: 10Tim Landscheidt) [22:44:02] (03CR) 10Dzahn: [C: 032] Add jpegtran dependency for metadata manipulation to OCG [operations/puppet] - 10https://gerrit.wikimedia.org/r/138361 (owner: 10Mwalker) [23:00:05] mwalker, ori, MaxSem: The time is nigh to deploy SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20140609T2300) [23:00:14] * MaxSem will do [23:00:30] thanks [23:00:49] JohnLewis, yt? [23:00:50] *thumbs up* [23:01:01] MaxSem: Sure [23:01:15] will deploy your patch now [23:01:29] (03CR) 10MaxSem: [C: 032] Allow bureaucrats to add/remove TA on legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138358 (https://bugzilla.wikimedia.org/66377) (owner: 10John F. Lewis) [23:01:37] (03Merged) 10jenkins-bot: Allow bureaucrats to add/remove TA on legalteamwiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/138358 (https://bugzilla.wikimedia.org/66377) (owner: 10John F. Lewis) [23:03:24] !log maxsem Synchronized wmf-config/InitialiseSettings.php: https://bugzilla.wikimedia.org/66377 (duration: 00m 04s) [23:03:28] Logged the message, Master [23:03:45] JohnLewis, please test ^^^ :) [23:03:58] MaxSem: Right, this is where things get complicated :p [23:04:21] Said patch is deployed to the legal team's wiki and I'm not in LCA :p [23:04:37] lawl [23:04:55] okay, let's see if they scream bloody murder [23:05:14] I'll mail James now [23:06:07] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 03 Jun 2014 16:21:49 UTC [23:08:07] MaxSem: Mailed. [23:08:16] thanks:) [23:09:35] Told him I'd mark the bug as fixed and asked him to mark it as verified once he looks [23:11:13] !log maxsem Synchronized php-1.24wmf7/extensions/MobileApp: (no message) (duration: 00m 03s) [23:11:17] Logged the message, Master [23:12:49] !log maxsem Synchronized php-1.24wmf8/extensions/MobileApp: (no message) (duration: 00m 03s) [23:12:54] Logged the message, Master [23:17:04] MaxSem: James verified it :) [23:18:57] greg-g, I'm done [23:30:02] (03PS1) 10Dzahn: add haithams to analytics-users [operations/puppet] - 10https://gerrit.wikimedia.org/r/138495 [23:30:46] !log Reloading Zuul to deploy 6727b8b [23:31:01] Logged the message, Master [23:48:57] (03CR) 10Rush: [C: 031] "cool!" [operations/puppet] - 10https://gerrit.wikimedia.org/r/138495 (owner: 10Dzahn) [23:52:06] (03PS2) 10Dzahn: wikistats sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137988 (owner: 10Rush) [23:52:15] (03PS2) 10Dzahn: pmacct sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137989 (owner: 10Rush) [23:52:24] (03PS2) 10Dzahn: planet sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137990 (owner: 10Rush) [23:52:30] (03PS2) 10Dzahn: modules/mysql_multi_instance/ sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137991 (owner: 10Rush) [23:52:38] (03PS2) 10Dzahn: jenkins sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137992 (owner: 10Rush) [23:52:46] (03PS2) 10Dzahn: deployment sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137993 (owner: 10Rush) [23:52:52] (03PS2) 10Dzahn: modules/coredb_mysql/ sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137994 (owner: 10Rush) [23:52:58] (03PS3) 10Dzahn: bugzilla sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137995 (owner: 10Rush) [23:53:09] (03PS3) 10Dzahn: search sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137996 (owner: 10Rush) [23:53:11] (03PS3) 10Dzahn: parsoid sans systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/137997 (owner: 10Rush) [23:54:08] (03PS3) 10Dzahn: wikistats - replace generic::systemuser with user [operations/puppet] - 10https://gerrit.wikimedia.org/r/137988 (owner: 10Rush) [23:54:28] (03PS3) 10Dzahn: pmacct - replace generic::systemuser with user [operations/puppet] - 10https://gerrit.wikimedia.org/r/137989 (owner: 10Rush) [23:54:39] (03PS3) 10Dzahn: planet - replace generic::systemuser with user [operations/puppet] - 10https://gerrit.wikimedia.org/r/137990 (owner: 10Rush) [23:55:01] (03CR) 10Dduvall: [C: 04-1] "Overall, it looks good! I left some comments about style and a couple about implementation choices. Until we have a solid Ruby style guide" (038 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138292 (owner: 10Ori.livneh) [23:56:29] (03PS3) 10Dzahn: generic: remove systemuser definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/138011 (owner: 10Rush) [23:57:15] (03CR) 10Dzahn: generic: remove systemuser definition (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/138011 (owner: 10Rush) [23:57:49] !log upgrading elastic1015 [23:57:54] Logged the message, Master [23:58:10] !log lied - upgrading elastic1014 [23:58:15] Logged the message, Master [23:58:54] (03CR) 10GWicke: "Here's a parsoid test repo set up with aptly" [operations/puppet] - 10https://gerrit.wikimedia.org/r/136128 (owner: 10Filippo Giunchedi) [23:59:00] (03PS3) 10Dzahn: facilities, replace generic::systemuser with user [operations/puppet] - 10https://gerrit.wikimedia.org/r/138009 (owner: 10Rush) [23:59:14] (03PS3) 10Dzahn: gerrit - replace generic::systemuser with user [operations/puppet] - 10https://gerrit.wikimedia.org/r/138008 (owner: 10Rush) [23:59:38] (03PS3) 10Dzahn: fundraising, replace generic::systemuser [operations/puppet] - 10https://gerrit.wikimedia.org/r/138007 (owner: 10Rush)