[00:00:51] so now I'm wondering what happens when POSTing over a client keep-alive connection [00:01:34] it sounds like the new-connection part is about backend requests? [00:02:31] AaronSchulz: are you still around? [00:03:18] PROBLEM - Puppet freshness on db78 is CRITICAL: No successful Puppet run in the last 10 hours [00:05:48] !log installing package upgrades on iron [00:05:59] Logged the message, Master [00:15:47] !log swift install on iron (swift cleaner & utils only) had a depends issue python-swift (= 1.5.0-3) but 1.7.4-0ubuntu2+wmf1 is installed, upgraded anyway with -f [00:15:57] Logged the message, Master [00:26:36] created https://bugzilla.wikimedia.org/show_bug.cgi?id=51273 for the direct Parsoid connection to api.svc.pmtpa.wmnet [00:26:45] will look into it on Monday [00:33:52] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:39:35] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73373 [00:40:52] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:52] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:53] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:54] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:54] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:54] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:55] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:55] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:56] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:57] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:57] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:58] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:58] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:58] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [00:41:59] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:52] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:53] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:53] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:54] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [00:42:54] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:54] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:55] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:56] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:56] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:56] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:57] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:57] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [00:42:58] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:58] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [00:42:59] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:42:59] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:00] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:01] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:01] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:01] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:52] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:53] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:54] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:54] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:55] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:55] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:55] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:56] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:57] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:57] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:57] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:58] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:58] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [00:43:59] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:00] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:52] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:52] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:52] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [00:44:52] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [00:44:52] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:52] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:53] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:54] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:54] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:55] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:55] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:56] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:56] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:57] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:57] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:58] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:58] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:59] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [00:46:59] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:00] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:00] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:01] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:01] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:52] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:52] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:52] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:52] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [00:47:52] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on db65 is CRITICAL: No successful Puppet run in the last 10 hours [00:49:52] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:52] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:52] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:52] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [00:50:52] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:52] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:53] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:53] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:54] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:54] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:55] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:55] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:56] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:56] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:57] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:57] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:58] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:58] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:59] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [00:50:59] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [00:51:00] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:01] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:52] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:52] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [00:52:52] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:52] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:52] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:53] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:53] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:54] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:54] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:55] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:55] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:56] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:56] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:57] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [00:52:57] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:58] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:58] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [00:52:59] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [00:53:52] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [00:53:52] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [00:53:52] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:53:52] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [00:53:52] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:58] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:58] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:58] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:58] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [00:55:58] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:52] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:52] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:52] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:52] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:52] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:53] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:53] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:54] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:54] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:55] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:55] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [00:57:56] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:56] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:57] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:57] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:58] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:58] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:59] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [00:57:59] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [00:58:52] PROBLEM - Puppet freshness on db33 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:52] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:52] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:52] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:52] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [01:00:52] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on mw102 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:52] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:53] PROBLEM - Puppet freshness on srv241 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:53] PROBLEM - Puppet freshness on mw6 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:54] PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:54] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:55] PROBLEM - Puppet freshness on wtp1023 is CRITICAL: No successful Puppet run in the last 10 hours [01:01:55] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:52] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:52] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:52] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:52] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [01:02:52] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:52] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:52] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:52] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:52] PROBLEM - Puppet freshness on cp1045 is CRITICAL: No successful Puppet run in the last 10 hours [01:04:52] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:01] PROBLEM - Puppet freshness on cp1061 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:01] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:01] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:01] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:01] PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours [01:07:13] New patchset: Dzahn; "comment/FIXME duplicate apache definition when combining classes on a single node" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73552 [01:08:05] New review: Dzahn; "already defined by other class on this node" [operations/puppet] (production) C: 2; - https://gerrit.wikimedia.org/r/73552 [01:09:01] PROBLEM - Puppet freshness on cp3010 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:01] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:02] PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:02] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:03] PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:03] PROBLEM - Puppet freshness on mw1101 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:04] PROBLEM - Puppet freshness on mw1162 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:04] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:05] PROBLEM - Puppet freshness on mw1196 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:05] PROBLEM - Puppet freshness on mw5 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:06] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:06] PROBLEM - Puppet freshness on mw90 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:07] PROBLEM - Puppet freshness on search1014 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:07] PROBLEM - Puppet freshness on search21 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:08] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:08] PROBLEM - Puppet freshness on srv275 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:09] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:09] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [01:09:10] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:01] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:02] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:02] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:03] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:03] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:04] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:04] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:05] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:05] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:06] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:06] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:07] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:07] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:08] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [01:10:08] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:01] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:01] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:01] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:01] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [01:11:01] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on db1023 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:01] PROBLEM - Puppet freshness on es1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:02] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:02] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:03] PROBLEM - Puppet freshness on pc1003 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:03] PROBLEM - Puppet freshness on search1018 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:04] PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours [01:21:46] New review: Dzahn; "already defined by other class on this node, unbreak puppet now, real fix later" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73552 [01:21:49] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73552 [01:22:01] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:01] PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:01] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [01:22:01] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:01] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:01] PROBLEM - Puppet freshness on search14 is CRITICAL: No successful Puppet run in the last 10 hours [01:22:41] PROBLEM - RAID on ms-be10 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:23:08] (the icinga-wm messages are not real and totally unrelated, what i unbroke is just on a single host) bbl [01:29:13] New patchset: Dzahn; "add missing $etherpad_db_user after adding it to private repo. the user doesn't have actual grants yet but breaks if it doesn't exist" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73553 [01:29:58] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73553 [01:30:01] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [01:30:01] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [01:34:49] New patchset: Dzahn; "same for etherpad db_host, db_name" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73554 [01:38:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73554 [02:07:36] !log LocalisationUpdate completed (1.22wmf9) at Sat Jul 13 02:07:36 UTC 2013 [02:07:48] Logged the message, Master [02:13:55] !log LocalisationUpdate completed (1.22wmf10) at Sat Jul 13 02:13:55 UTC 2013 [02:14:06] Logged the message, Master [02:22:51] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jul 13 02:22:51 UTC 2013 [02:23:01] Logged the message, Master [02:28:51] !log copied etherpad-lite from lucid to precise [02:28:58] !log kernel upgrade, rebooting zirconium [02:29:01] Logged the message, Master [02:29:02] PROBLEM - Host zirconium is DOWN: PING CRITICAL - Packet loss = 100% [02:29:10] Logged the message, Master [02:31:32] RECOVERY - Host zirconium is UP: PING OK - Packet loss = 0%, RTA = 0.69 ms [02:34:12] PROBLEM - HTTP on zirconium is CRITICAL: Connection refused [02:35:46] New patchset: Dzahn; "zirconium apache - fix wrong path to ssl key" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73557 [02:38:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73557 [02:38:12] RECOVERY - HTTP on zirconium is OK: HTTP OK: HTTP/1.1 200 OK - 208444 bytes in 0.025 second response time [02:41:12] PROBLEM - HTTP on zirconium is CRITICAL: Connection refused [02:42:12] RECOVERY - HTTP on zirconium is OK: HTTP OK: HTTP/1.1 200 OK - 208444 bytes in 0.013 second response time [04:49:41] PROBLEM - Puppet freshness on grosley is CRITICAL: No successful Puppet run in the last 10 hours [04:57:41] PROBLEM - Puppet freshness on mw56 is CRITICAL: No successful Puppet run in the last 10 hours [06:19:47] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [07:12:01] PROBLEM - Disk space on analytics1006 is CRITICAL: DISK CRITICAL - free space: / 709 MB (3% inode=84%): [08:24:01] Nemo_bis: http://www.mediawiki.org/wiki/Talk:Wikimedia_Engineering/2013-14_Goals#.22Dozens_of_Toolserver_tools_have_moved_to_Labs.22 p858snake|l: yes, so? [08:25:30] so dozens is better than, since not all of those tools will be moving [08:28:01] better than what? [08:30:48] I kinda of misread your miss read your text there [08:31:28] but in the context that they have used "dozens" here its more of a unfixed number meaning, inregards to the keeping ts alive bit [08:33:31] PROBLEM - SSH on pdf3 is CRITICAL: Server answer: [08:35:31] RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [08:36:12] p858snake|l: it's still a ridiculous number [08:36:25] "hundreds" or "95 %" would make more [08:36:27] sense [08:48:51] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: No successful Puppet run in the last 10 hours [09:51:49] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:03:49] PROBLEM - Puppet freshness on db78 is CRITICAL: No successful Puppet run in the last 10 hours [10:07:13] New patchset: Jalexander; "PlanetWikimedia changes fr and en" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73564 [10:14:45] PROBLEM - RAID on mc1005 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:15:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [10:21:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.003 second response time on port 6379 [10:24:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [10:28:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [10:33:55] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:34:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [10:40:55] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:55] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:56] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:56] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:57] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:57] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:58] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:58] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:59] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:41:59] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:00] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:00] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:01] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:01] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:02] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:03] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [10:42:55] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:55] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:55] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:55] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:55] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:56] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:56] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [10:42:57] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:57] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:58] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:58] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:59] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [10:42:59] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:00] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:00] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:01] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:01] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:02] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [10:43:02] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:03] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:03] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:04] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:04] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:05] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [10:43:05] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:55] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:56] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:56] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:57] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:57] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:58] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:58] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:59] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [10:43:59] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:00] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:00] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:01] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:01] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:02] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:02] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:03] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:03] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.999 second response time on port 6379 [10:44:55] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:55] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:55] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [10:44:55] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [10:44:55] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:16] New patchset: Odder; "(bug 50929) Remove 'visualeditor-enable' from $wgHiddenPrefs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73565 [10:46:55] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:55] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:56] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:56] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:57] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:57] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:58] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:59] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:59] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [10:46:59] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:00] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:00] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:01] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:01] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:02] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:02] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:03] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:03] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:04] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:04] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [10:47:55] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:55] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:55] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:55] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:47:55] PROBLEM - Puppet freshness on analytics1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:49:55] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [10:49:55] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [10:49:55] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [10:49:55] PROBLEM - Puppet freshness on db65 is CRITICAL: No successful Puppet run in the last 10 hours [10:49:55] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:55] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:56] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:56] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:57] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:57] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:58] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:58] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:59] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [10:50:59] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:00] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:00] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:01] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:01] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [10:51:02] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:55] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:56] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:56] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:57] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:57] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:58] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:58] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:59] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [10:51:59] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:00] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:00] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:01] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:01] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:02] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:02] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:03] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:03] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:55] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:56] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:56] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:57] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:57] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:58] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:58] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [10:52:59] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [10:52:59] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:00] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:00] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:01] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:55] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:55] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:55] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:55] PROBLEM - Puppet freshness on db29 is CRITICAL: No successful Puppet run in the last 10 hours [10:53:55] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [10:55:55] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [10:55:55] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [10:55:55] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [10:55:55] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [10:55:55] PROBLEM - Puppet freshness on cp3021 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [10:57:55] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:55] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:56] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:56] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [10:57:57] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:57] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:58] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:58] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:59] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [10:57:59] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:00] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:00] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:01] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:01] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:55] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:55] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:55] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:55] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [10:58:55] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:55] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:55] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:55] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:55] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:00:55] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:13] I don't know why, but I have this feeling puppet is down, just some feeling I have [11:01:55] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:55] PROBLEM - Puppet freshness on mw102 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:56] PROBLEM - Puppet freshness on mw6 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:56] PROBLEM - Puppet freshness on srv241 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:57] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:57] PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:58] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [11:01:58] PROBLEM - Puppet freshness on wtp1023 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:04] Jamesofur: I don't know why you have that feeling at all, really :) [11:02:22] (ignore icinga! you can!) [11:02:43] I would but he keeps leaving ;) [11:02:55] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:55] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:55] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:55] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:55] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:02:58] I think he's trying to avoid me [11:03:18] 'the one with the power to vanquish icinga!' [11:03:28] there was a prophesy about you, I think :) [11:03:44] apergos: around? Can you run one more grep of the sampled logs for me? :D [11:04:15] PROBLEM - SSH on gadolinium is CRITICAL: Server answer: [11:04:55] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [11:04:55] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [11:04:55] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:04:55] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [11:04:55] PROBLEM - Puppet freshness on cp1045 is CRITICAL: No successful Puppet run in the last 10 hours [11:05:15] RECOVERY - SSH on gadolinium is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:07:15] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:15] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:15] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:15] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [11:07:15] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:15] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:15] PROBLEM - Puppet freshness on cp3010 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:15] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:15] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:15] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:16] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [11:09:16] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:17] PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:17] PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:18] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:18] PROBLEM - Puppet freshness on mw1101 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:19] PROBLEM - Puppet freshness on mw1162 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:19] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:20] PROBLEM - Puppet freshness on mw1196 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:20] PROBLEM - Puppet freshness on mw5 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:21] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:21] PROBLEM - Puppet freshness on mw90 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:22] PROBLEM - Puppet freshness on search1014 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:22] PROBLEM - Puppet freshness on search21 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:23] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:23] PROBLEM - Puppet freshness on srv275 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:24] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:24] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [11:09:25] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [11:09:25] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [11:10:15] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:15] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:15] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:15] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:15] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:16] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:16] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:17] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:17] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:18] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:18] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:19] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:19] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:20] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:20] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:21] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:21] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:22] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:22] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:23] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:23] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [11:10:25] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.58 ms [11:11:15] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [11:11:15] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [11:11:15] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:11:15] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [11:11:15] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [11:13:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.004 second response time on port 6379 [11:14:35] PROBLEM - SSH on lvs1001 is CRITICAL: Server answer: [11:15:35] RECOVERY - SSH on lvs1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:16:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [11:18:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [11:21:15] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on db1023 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on es1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:15] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:16] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:16] PROBLEM - Puppet freshness on pc1003 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:17] PROBLEM - Puppet freshness on search1018 is CRITICAL: No successful Puppet run in the last 10 hours [11:21:17] PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours [11:22:15] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [11:22:15] PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours [11:22:15] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [11:22:15] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [11:22:15] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [11:22:16] PROBLEM - Puppet freshness on search14 is CRITICAL: No successful Puppet run in the last 10 hours [11:24:00] New patchset: TTO; "change votewiki site name" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73568 [11:24:35] PROBLEM - NTP on mw31 is CRITICAL: NTP CRITICAL: Offset unknown [11:26:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [11:27:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [11:28:35] RECOVERY - NTP on mw31 is OK: NTP OK: Offset 0.001299977303 secs [11:30:15] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:15] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [11:30:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [11:34:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [11:37:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [11:49:15] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [11:52:25] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [11:52:25] PROBLEM - NTP on mc1005 is CRITICAL: NTP CRITICAL: No response from NTP server [12:24:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [12:27:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [12:39:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [12:42:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [12:46:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 1.003 second response time on port 6379 [12:49:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [12:51:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.002 second response time on port 6379 [12:54:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:04:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.004 second response time on port 6379 [13:09:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:20:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.998 second response time on port 6379 [13:25:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:36:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [13:42:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:44:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [13:47:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:49:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [13:52:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [13:56:04] New review: Dereckson; "(1 comment)" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/73568 [13:57:10] New patchset: Dereckson; "(bug 51264) Site name configuration for votewiki." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73568 [13:57:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.003 second response time on port 6379 [13:59:32] New review: Dereckson; "PS2: Fixing commit message to describe the modification" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/73568 [14:00:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:04:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.002 second response time on port 6379 [14:07:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:09:37] errm, what just happened Dereckson [14:10:13] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.998 second response time on port 6379 [14:13:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:21:13] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [14:24:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:38:23] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [14:41:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:42:15] Wonder why that's flapping [14:42:23] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.003 second response time on port 6379 [14:43:38] http://ganglia.wikimedia.org/latest/?c=Memcached%20eqiad&h=mc1005.eqiad.wmnet&m=cpu_report&r=hour&s=by%20name&hc=4&mc=2 [14:43:44] Constant wait and system load? :/ [14:45:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [14:50:04] PROBLEM - Puppet freshness on grosley is CRITICAL: No successful Puppet run in the last 10 hours [14:52:43] PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:33] RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [14:58:03] PROBLEM - Puppet freshness on mw56 is CRITICAL: No successful Puppet run in the last 10 hours [14:59:13] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.006 second response time on port 6379 [15:02:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [15:16:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 1.003 second response time on port 6379 [15:19:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [15:44:04] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73564 [16:19:57] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [16:34:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.998 second response time on port 6379 [16:37:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [16:41:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.998 second response time on port 6379 [16:47:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [16:55:28] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [16:58:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [17:06:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.005 second response time on port 6379 [17:09:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [17:21:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [17:24:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [17:35:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [17:38:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [17:46:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [17:49:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [17:52:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [17:58:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [18:19:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.998 second response time on port 6379 [18:22:20] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [18:26:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.999 second response time on port 6379 [18:29:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [18:49:09] PROBLEM - Puppet freshness on ms-be5 is CRITICAL: No successful Puppet run in the last 10 hours [18:51:17] New patchset: Platonides; "(Bug 51297) Enable Babel categories on es.wikinews" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73578 [18:53:02] I'm correct in my interpretation that https://it.wikipedia.org/robots.txt prevents Italian AfDs (PfCs?) from showing up in Google results, right? [18:53:02] however, I see that the pages do appear in search results, see . Is google not obeying the robots.txt? [18:53:33] This is creating problems for some subjects in OTRS where we promised the content wouldn't show up, yet they come back saying it is in fact available. [18:54:26] Oh, I see, Google will obnoxiously show the title of a page even if the page itself is removed :/ [18:55:13] New review: Odder; "shellpolicy" [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/73578 [18:57:06] lfaraone: #wikimedia-tech would be a better channel [18:57:46] Nemo_bis: ah, I wasn't too clear on what the difference was; I had mentioned this in that channel a while ago but didn't get a reply. → there, then. [18:58:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 1.000 second response time on port 6379 [19:01:19] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [19:04:19] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.004 second response time on port 6379 [19:07:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [19:19:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.000 second response time on port 6379 [19:22:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [19:27:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:28:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.124 second response time [19:35:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.006 second response time on port 6379 [19:38:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [19:47:21] RECOVERY - Redis on mc1005 is OK: TCP OK - 1.000 second response time on port 6379 [19:50:21] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [19:52:11] PROBLEM - Puppet freshness on mw1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:04:11] PROBLEM - Puppet freshness on db78 is CRITICAL: No successful Puppet run in the last 10 hours [20:23:31] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.002 second response time on port 6379 [20:26:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [20:34:03] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:41:03] PROBLEM - Puppet freshness on rubidium is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on ekrem is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on manganese is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on mw1007 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on mw1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on mw1043 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on mw1063 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:03] PROBLEM - Puppet freshness on mw1087 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:04] PROBLEM - Puppet freshness on mw1171 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:04] PROBLEM - Puppet freshness on mw1197 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:05] PROBLEM - Puppet freshness on mw121 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:06] PROBLEM - Puppet freshness on mw1210 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:06] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:06] PROBLEM - Puppet freshness on search1024 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:07] PROBLEM - Puppet freshness on search18 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:08] PROBLEM - Puppet freshness on solr1003 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:08] PROBLEM - Puppet freshness on solr3 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:08] PROBLEM - Puppet freshness on sq76 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:09] PROBLEM - Puppet freshness on srv292 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:09] PROBLEM - Puppet freshness on stat1 is CRITICAL: No successful Puppet run in the last 10 hours [20:42:10] PROBLEM - Puppet freshness on titanium is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on amssq53 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on cp1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on cp3009 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on cp3012 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on db1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:03] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:04] PROBLEM - Puppet freshness on db1044 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:05] PROBLEM - Puppet freshness on db39 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:05] PROBLEM - Puppet freshness on helium is CRITICAL: No successful Puppet run in the last 10 hours [20:43:05] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:06] PROBLEM - Puppet freshness on mc1007 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:07] PROBLEM - Puppet freshness on ms10 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:07] PROBLEM - Puppet freshness on mw1032 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:07] PROBLEM - Puppet freshness on mw124 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:08] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:08] PROBLEM - Puppet freshness on pc1 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:09] PROBLEM - Puppet freshness on potassium is CRITICAL: No successful Puppet run in the last 10 hours [20:43:10] PROBLEM - Puppet freshness on praseodymium is CRITICAL: No successful Puppet run in the last 10 hours [20:43:10] PROBLEM - Puppet freshness on rdb1002 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:10] PROBLEM - Puppet freshness on sq54 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:11] PROBLEM - Puppet freshness on sq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:11] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:12] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [20:43:12] PROBLEM - Puppet freshness on wtp1015 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on cp1010 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on db1022 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on ms-fe1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on mw1003 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on mw1033 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:03] PROBLEM - Puppet freshness on mw106 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:04] PROBLEM - Puppet freshness on mw1024 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:05] PROBLEM - Puppet freshness on mw1150 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:05] PROBLEM - Puppet freshness on mw1069 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:05] PROBLEM - Puppet freshness on mw1189 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:06] PROBLEM - Puppet freshness on mw1201 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:06] PROBLEM - Puppet freshness on mw2 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:07] PROBLEM - Puppet freshness on wtp1007 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:07] PROBLEM - Puppet freshness on mw79 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:08] PROBLEM - Puppet freshness on rdb1003 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:08] PROBLEM - Puppet freshness on mw1205 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:09] PROBLEM - Puppet freshness on mw35 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:09] PROBLEM - Puppet freshness on search33 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:10] PROBLEM - Puppet freshness on srv193 is CRITICAL: No successful Puppet run in the last 10 hours [20:44:10] PROBLEM - Puppet freshness on mw98 is CRITICAL: No successful Puppet run in the last 10 hours [20:45:03] PROBLEM - Puppet freshness on amslvs1 is CRITICAL: No successful Puppet run in the last 10 hours [20:45:03] PROBLEM - Puppet freshness on amssq48 is CRITICAL: No successful Puppet run in the last 10 hours [20:45:03] PROBLEM - Puppet freshness on analytics1004 is CRITICAL: No successful Puppet run in the last 10 hours [20:45:03] PROBLEM - Puppet freshness on calcium is CRITICAL: No successful Puppet run in the last 10 hours [20:45:03] PROBLEM - Puppet freshness on db1033 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:03] PROBLEM - Puppet freshness on antimony is CRITICAL: No successful Puppet run in the last 10 hours [20:47:03] PROBLEM - Puppet freshness on cp1058 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:03] PROBLEM - Puppet freshness on cp3011 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:03] PROBLEM - Puppet freshness on dataset1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:03] PROBLEM - Puppet freshness on db1002 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:04] PROBLEM - Puppet freshness on db1014 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:04] PROBLEM - Puppet freshness on labstore1 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:05] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:05] PROBLEM - Puppet freshness on labstore3 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:06] PROBLEM - Puppet freshness on mc1012 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:06] PROBLEM - Puppet freshness on ms-be12 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:07] PROBLEM - Puppet freshness on ms-fe1002 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:07] PROBLEM - Puppet freshness on mw1027 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:08] PROBLEM - Puppet freshness on mw1104 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:08] PROBLEM - Puppet freshness on mw1206 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:09] PROBLEM - Puppet freshness on mw1208 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:09] PROBLEM - Puppet freshness on mw1211 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:10] PROBLEM - Puppet freshness on mw42 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:10] PROBLEM - Puppet freshness on mw75 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:11] PROBLEM - Puppet freshness on search25 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:11] PROBLEM - Puppet freshness on solr1 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:12] PROBLEM - Puppet freshness on srv242 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:12] PROBLEM - Puppet freshness on ssl1 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:13] PROBLEM - Puppet freshness on virt11 is CRITICAL: No successful Puppet run in the last 10 hours [20:47:13] PROBLEM - Puppet freshness on wtp1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:48:03] PROBLEM - Puppet freshness on bast1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:48:03] PROBLEM - Puppet freshness on amssq56 is CRITICAL: No successful Puppet run in the last 10 hours [20:48:03] PROBLEM - Puppet freshness on amssq51 is CRITICAL: No successful Puppet run in the last 10 hours [20:48:03] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: No successful Puppet run in the last 10 hours [20:48:03] PROBLEM - Puppet freshness on cp1037 is CRITICAL: No successful Puppet run in the last 10 hours [20:50:03] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [20:50:03] PROBLEM - Puppet freshness on cp1038 is CRITICAL: No successful Puppet run in the last 10 hours [20:50:03] PROBLEM - Puppet freshness on db1058 is CRITICAL: No successful Puppet run in the last 10 hours [20:50:03] PROBLEM - Puppet freshness on db65 is CRITICAL: No successful Puppet run in the last 10 hours [20:50:03] PROBLEM - Puppet freshness on db77 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on brewster is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on cp1065 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on dataset2 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:03] PROBLEM - Puppet freshness on db1024 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:04] PROBLEM - Puppet freshness on db32 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:04] PROBLEM - Puppet freshness on db51 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:05] PROBLEM - Puppet freshness on db57 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:05] PROBLEM - Puppet freshness on es5 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:06] PROBLEM - Puppet freshness on mc1008 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:06] PROBLEM - Puppet freshness on mw1138 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:07] PROBLEM - Puppet freshness on mw26 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:07] PROBLEM - Puppet freshness on mw33 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:08] PROBLEM - Puppet freshness on mw36 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:08] PROBLEM - Puppet freshness on mw64 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:09] PROBLEM - Puppet freshness on sq50 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:09] PROBLEM - Puppet freshness on stat1002 is CRITICAL: No successful Puppet run in the last 10 hours [20:51:10] PROBLEM - Puppet freshness on tarin is CRITICAL: No successful Puppet run in the last 10 hours [20:51:10] PROBLEM - Puppet freshness on wtp1013 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:03] PROBLEM - Puppet freshness on aluminium is CRITICAL: No successful Puppet run in the last 10 hours [20:52:03] PROBLEM - Puppet freshness on db1010 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:03] PROBLEM - Puppet freshness on db46 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:03] PROBLEM - Puppet freshness on db55 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:03] PROBLEM - Puppet freshness on es1009 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:04] PROBLEM - Puppet freshness on labsdb1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:04] PROBLEM - Puppet freshness on mw1022 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:05] PROBLEM - Puppet freshness on mw1040 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:05] PROBLEM - Puppet freshness on mw1062 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:06] PROBLEM - Puppet freshness on mw107 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:06] PROBLEM - Puppet freshness on mw109 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:07] PROBLEM - Puppet freshness on mw1132 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:07] PROBLEM - Puppet freshness on mw1185 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:08] PROBLEM - Puppet freshness on mw1218 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:08] PROBLEM - Puppet freshness on mw40 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:09] PROBLEM - Puppet freshness on mw53 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:09] PROBLEM - Puppet freshness on mw70 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:10] PROBLEM - Puppet freshness on snapshot4 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:10] PROBLEM - Puppet freshness on sq55 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:11] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:11] PROBLEM - Puppet freshness on sq81 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:12] PROBLEM - Puppet freshness on srv296 is CRITICAL: No successful Puppet run in the last 10 hours [20:52:12] PROBLEM - Puppet freshness on wtp1021 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:03] PROBLEM - Puppet freshness on colby is CRITICAL: No successful Puppet run in the last 10 hours [20:53:03] PROBLEM - Puppet freshness on amssq59 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:03] PROBLEM - Puppet freshness on db31 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:03] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:03] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:04] PROBLEM - Puppet freshness on mc1011 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:04] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:05] PROBLEM - Puppet freshness on mw1072 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:05] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:06] PROBLEM - Puppet freshness on mw117 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:06] PROBLEM - Puppet freshness on mw1178 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:07] PROBLEM - Puppet freshness on mw1219 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:07] PROBLEM - Puppet freshness on mw87 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:08] PROBLEM - Puppet freshness on professor is CRITICAL: No successful Puppet run in the last 10 hours [20:53:08] PROBLEM - Puppet freshness on search1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:09] PROBLEM - Puppet freshness on search1011 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:09] PROBLEM - Puppet freshness on srv285 is CRITICAL: No successful Puppet run in the last 10 hours [20:53:10] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:03] PROBLEM - Puppet freshness on amssq32 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:03] PROBLEM - Puppet freshness on amssq43 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:03] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:03] PROBLEM - Puppet freshness on cp1015 is CRITICAL: No successful Puppet run in the last 10 hours [20:54:03] PROBLEM - Puppet freshness on db1011 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:03] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:03] PROBLEM - Puppet freshness on cp1001 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:03] PROBLEM - Puppet freshness on cp1039 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:03] PROBLEM - Puppet freshness on cp1054 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:03] PROBLEM - Puppet freshness on cp3019 is CRITICAL: No successful Puppet run in the last 10 hours [20:56:13] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.003 second response time on port 6379 [20:57:03] PROBLEM - Puppet freshness on amssq36 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:03] PROBLEM - Puppet freshness on cp1009 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:03] PROBLEM - Puppet freshness on cp1008 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:03] PROBLEM - Puppet freshness on cp1064 is CRITICAL: No successful Puppet run in the last 10 hours [20:57:03] PROBLEM - Puppet freshness on cp1068 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:03] PROBLEM - Puppet freshness on amssq41 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:03] PROBLEM - Puppet freshness on amssq60 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on analytics1006 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on analytics1024 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on cp1057 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on db1028 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on db38 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on db68 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:04] PROBLEM - Puppet freshness on ersch is CRITICAL: No successful Puppet run in the last 10 hours [20:58:05] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:05] PROBLEM - Puppet freshness on ms-be1005 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:06] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:06] PROBLEM - Puppet freshness on mw1039 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:07] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:07] PROBLEM - Puppet freshness on mw1184 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:08] PROBLEM - Puppet freshness on mw96 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:08] PROBLEM - Puppet freshness on mw120 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:09] PROBLEM - Puppet freshness on searchidx2 is CRITICAL: No successful Puppet run in the last 10 hours [20:58:09] PROBLEM - Puppet freshness on ssl1004 is CRITICAL: No successful Puppet run in the last 10 hours [20:59:03] PROBLEM - Puppet freshness on amssq58 is CRITICAL: No successful Puppet run in the last 10 hours [20:59:03] PROBLEM - Puppet freshness on cp1019 is CRITICAL: No successful Puppet run in the last 10 hours [20:59:03] PROBLEM - Puppet freshness on db1003 is CRITICAL: No successful Puppet run in the last 10 hours [20:59:03] PROBLEM - Puppet freshness on db1041 is CRITICAL: No successful Puppet run in the last 10 hours [20:59:03] PROBLEM - Puppet freshness on db1043 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:03] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:03] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:03] PROBLEM - Puppet freshness on analytics1026 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:03] PROBLEM - Puppet freshness on cp1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:03] PROBLEM - Puppet freshness on cp1011 is CRITICAL: No successful Puppet run in the last 10 hours [21:01:23] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [21:02:03] PROBLEM - Puppet freshness on amssq46 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:03] PROBLEM - Puppet freshness on amssq62 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:03] PROBLEM - Puppet freshness on cp1055 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:03] PROBLEM - Puppet freshness on db1053 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:03] PROBLEM - Puppet freshness on es8 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:04] PROBLEM - Puppet freshness on ms1002 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:04] PROBLEM - Puppet freshness on mw102 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:05] PROBLEM - Puppet freshness on mw6 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:05] PROBLEM - Puppet freshness on srv241 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:06] PROBLEM - Puppet freshness on ssl3002 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:06] PROBLEM - Puppet freshness on virt5 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:07] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [21:02:07] PROBLEM - Puppet freshness on wtp1023 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:03] PROBLEM - Puppet freshness on amssq38 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:03] PROBLEM - Puppet freshness on analytics1027 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:03] PROBLEM - Puppet freshness on cp1052 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:03] PROBLEM - Puppet freshness on cp1059 is CRITICAL: No successful Puppet run in the last 10 hours [21:03:03] PROBLEM - Puppet freshness on db1006 is CRITICAL: No successful Puppet run in the last 10 hours [21:05:03] PROBLEM - Puppet freshness on amssq39 is CRITICAL: No successful Puppet run in the last 10 hours [21:05:03] PROBLEM - Puppet freshness on amssq61 is CRITICAL: No successful Puppet run in the last 10 hours [21:05:03] PROBLEM - Puppet freshness on analytics1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:05:03] PROBLEM - Puppet freshness on analytics1025 is CRITICAL: No successful Puppet run in the last 10 hours [21:05:03] PROBLEM - Puppet freshness on cp1045 is CRITICAL: No successful Puppet run in the last 10 hours [21:08:07] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: No successful Puppet run in the last 10 hours [21:08:07] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: No successful Puppet run in the last 10 hours [21:08:07] PROBLEM - Puppet freshness on amssq52 is CRITICAL: No successful Puppet run in the last 10 hours [21:08:07] PROBLEM - Puppet freshness on cp1056 is CRITICAL: No successful Puppet run in the last 10 hours [21:08:07] PROBLEM - Puppet freshness on cp1007 is CRITICAL: No successful Puppet run in the last 10 hours [21:09:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [21:10:07] PROBLEM - Puppet freshness on cp1013 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:07] PROBLEM - Puppet freshness on db1036 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:07] PROBLEM - Puppet freshness on cp3010 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:07] PROBLEM - Puppet freshness on db1048 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:07] PROBLEM - Puppet freshness on gadolinium is CRITICAL: No successful Puppet run in the last 10 hours [21:10:08] PROBLEM - Puppet freshness on cp1050 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:08] PROBLEM - Puppet freshness on ms-be4 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:09] PROBLEM - Puppet freshness on lvs5 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:09] PROBLEM - Puppet freshness on mw1044 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:10] PROBLEM - Puppet freshness on ms5 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:10] PROBLEM - Puppet freshness on mw1101 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:11] PROBLEM - Puppet freshness on mw1162 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:11] PROBLEM - Puppet freshness on mw119 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:12] PROBLEM - Puppet freshness on mw1196 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:12] PROBLEM - Puppet freshness on mw5 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:13] PROBLEM - Puppet freshness on mw83 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:13] PROBLEM - Puppet freshness on search1014 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:14] PROBLEM - Puppet freshness on mw90 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:14] PROBLEM - Puppet freshness on search21 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:15] PROBLEM - Puppet freshness on sq79 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:15] PROBLEM - Puppet freshness on srv275 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:16] PROBLEM - Puppet freshness on srv290 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:16] PROBLEM - Puppet freshness on wtp1024 is CRITICAL: No successful Puppet run in the last 10 hours [21:10:17] PROBLEM - Puppet freshness on zirconium is CRITICAL: No successful Puppet run in the last 10 hours [21:11:07] PROBLEM - Puppet freshness on db1020 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:07] PROBLEM - Puppet freshness on mc1009 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:07] PROBLEM - Puppet freshness on lvs1 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:07] PROBLEM - Puppet freshness on ms-be1012 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:07] PROBLEM - Puppet freshness on mw1124 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:08] PROBLEM - Puppet freshness on mw1161 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:08] PROBLEM - Puppet freshness on mw1130 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:09] PROBLEM - Puppet freshness on mw1147 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:09] PROBLEM - Puppet freshness on mw1190 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:10] PROBLEM - Puppet freshness on mw1125 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:10] PROBLEM - Puppet freshness on mw1111 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:11] PROBLEM - Puppet freshness on pdf2 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:11] PROBLEM - Puppet freshness on mw8 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:12] PROBLEM - Puppet freshness on mw1214 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:12] PROBLEM - Puppet freshness on mw38 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:13] PROBLEM - Puppet freshness on search36 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:13] PROBLEM - Puppet freshness on solr2 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:14] PROBLEM - Puppet freshness on sq51 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:14] PROBLEM - Puppet freshness on srv301 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:15] PROBLEM - Puppet freshness on wtp1011 is CRITICAL: No successful Puppet run in the last 10 hours [21:11:15] PROBLEM - Puppet freshness on wtp1018 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:07] PROBLEM - Puppet freshness on amssq44 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:07] PROBLEM - Puppet freshness on analytics1023 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:07] PROBLEM - Puppet freshness on db1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:07] PROBLEM - Puppet freshness on cp1063 is CRITICAL: No successful Puppet run in the last 10 hours [21:12:07] PROBLEM - Puppet freshness on db43 is CRITICAL: No successful Puppet run in the last 10 hours [21:19:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.001 second response time on port 6379 [21:22:07] PROBLEM - Puppet freshness on cp1002 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on amssq34 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on cp1012 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on db1023 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on es1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on db34 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:07] PROBLEM - Puppet freshness on mc1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:08] PROBLEM - Puppet freshness on mw1139 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:08] PROBLEM - Puppet freshness on pc1003 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:09] PROBLEM - Puppet freshness on search1018 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:09] PROBLEM - Puppet freshness on srv258 is CRITICAL: No successful Puppet run in the last 10 hours [21:22:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [21:23:07] PROBLEM - Puppet freshness on cp1046 is CRITICAL: No successful Puppet run in the last 10 hours [21:23:07] PROBLEM - Puppet freshness on db40 is CRITICAL: No successful Puppet run in the last 10 hours [21:23:07] PROBLEM - Puppet freshness on fenari is CRITICAL: No successful Puppet run in the last 10 hours [21:23:07] PROBLEM - Puppet freshness on ms-be1 is CRITICAL: No successful Puppet run in the last 10 hours [21:23:07] PROBLEM - Puppet freshness on mw20 is CRITICAL: No successful Puppet run in the last 10 hours [21:23:07] PROBLEM - Puppet freshness on search14 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [21:31:07] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [21:37:32] New patchset: Andrew Bogott; "Catch tests up with local changes to this module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/73585 [21:44:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [21:49:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [21:52:17] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.005 second response time on port 6379 [22:01:27] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [22:11:35] PROBLEM - Disk space on mc1005 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:11:35] PROBLEM - DPKG on mc1005 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:11:35] PROBLEM - SSH on mc1005 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:24] New patchset: MZMcBride; "(bug 50929) Remove 'visualeditor-enable' from $wgHiddenPrefs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/73565 [22:31:34] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [22:33:26] Elsie: do I get to review my own commit now? [22:38:39] Dunno. [22:39:23] own commit review??? [22:39:24] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.000 second response time on port 11211 [22:39:54] I don't know what commit you're talking about, but it doesn't sound good [22:47:24] RECOVERY - Redis on mc1005 is OK: TCP OK - 3.002 second response time on port 6379 [22:52:24] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [23:01:14] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [23:02:34] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [23:06:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [23:10:28] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.002 second response time on port 11211 [23:18:38] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [23:19:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.001 second response time on port 6379 [23:22:28] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.002 second response time on port 11211 [23:25:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [23:33:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.999 second response time on port 6379 [23:36:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [23:38:38] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [23:40:28] RECOVERY - Memcached on mc1005 is OK: TCP OK - 0.999 second response time on port 11211 [23:52:18] RECOVERY - Redis on mc1005 is OK: TCP OK - 0.997 second response time on port 6379 [23:55:28] PROBLEM - Redis on mc1005 is CRITICAL: Connection timed out [23:55:38] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out [23:56:28] RECOVERY - Memcached on mc1005 is OK: TCP OK - 1.000 second response time on port 11211 [23:59:38] PROBLEM - Memcached on mc1005 is CRITICAL: Connection timed out