[00:00:42] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 21:59:41 UTC [00:19:59] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Jul 6 00:19:52 UTC 2014 [00:57:45] PROBLEM - SSH on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:57:55] PROBLEM - gdash.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:05] PROBLEM - MediaWiki profile collector on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:58:15] PROBLEM - Graphite Carbon on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:58:26] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:58:27] PROBLEM - puppet last run on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [00:58:35] RECOVERY - SSH on tungsten is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [00:58:45] RECOVERY - gdash.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 9055 bytes in 0.600 second response time [00:58:56] RECOVERY - MediaWiki profile collector on tungsten is OK: OK: All defined mwprof jobs are runnning. [00:59:06] RECOVERY - Graphite Carbon on tungsten is OK: OK: All defined Carbon jobs are runnning. [00:59:15] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 457 seconds ago with 0 failures [00:59:15] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [01:15:33] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [02:14:11] !log LocalisationUpdate completed (1.24wmf11) at 2014-07-06 02:13:07+00:00 [02:14:20] Logged the message, Master [02:20:19] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 06 Jul 2014 00:19:52 UTC [02:25:11] !log LocalisationUpdate completed (1.24wmf12) at 2014-07-06 02:24:08+00:00 [02:25:17] Logged the message, Master [02:40:21] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Jul 6 02:40:16 UTC 2014 [02:50:27] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Jul 6 02:49:21 UTC 2014 (duration 49m 20s) [02:50:32] Logged the message, Master [03:08:06] (03PS1) 10Yurik: LABS: Set domain override for zero banners [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/144339 [03:15:43] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [03:30:35] (03PS2) 10Yurik: LABS: Set domain override for zero banners [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/144339 [03:57:34] (03CR) 10Ori.livneh: [C: 04-1] "I have second thoughts about this patch as well. It performs the migration in a single swoop, which seems needlessly risky to me." (034 comments) [operations/puppet] - 10https://gerrit.wikimedia.org/r/143597 (owner: 10Giuseppe Lavagetto) [04:07:36] (03PS1) 10Ori.livneh: nutcracker migration: allow wikidev to (re-)start nutcracker [operations/puppet] - 10https://gerrit.wikimedia.org/r/144343 [04:08:55] PROBLEM - puppet last run on caesium is CRITICAL: CRITICAL: Puppet has 1 failures [04:08:55] PROBLEM - puppet last run on mw1174 is CRITICAL: CRITICAL: Puppet has 1 failures [04:08:55] PROBLEM - puppet last run on mw1187 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:25] PROBLEM - puppet last run on mw1026 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:25] PROBLEM - puppet last run on wtp1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:25] PROBLEM - puppet last run on amssq49 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:25] PROBLEM - puppet last run on analytics1025 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:25] PROBLEM - puppet last run on potassium is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:35] PROBLEM - puppet last run on search1016 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:35] PROBLEM - puppet last run on cp1039 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:45] PROBLEM - puppet last run on elastic1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:45] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:45] PROBLEM - puppet last run on wtp1020 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:45] PROBLEM - puppet last run on cp1055 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:45] PROBLEM - puppet last run on ms-be1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:46] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:55] PROBLEM - puppet last run on db1066 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:55] PROBLEM - puppet last run on lvs1002 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:55] PROBLEM - puppet last run on amssq32 is CRITICAL: CRITICAL: Puppet has 1 failures [04:09:55] PROBLEM - puppet last run on ms-be1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:05] PROBLEM - puppet last run on db1022 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:05] PROBLEM - puppet last run on helium is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:05] PROBLEM - puppet last run on elastic1004 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:06] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:15] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:15] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:15] PROBLEM - puppet last run on snapshot1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:25] PROBLEM - puppet last run on mc1006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:25] PROBLEM - puppet last run on mc1003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:25] PROBLEM - puppet last run on db1031 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:25] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:25] PROBLEM - puppet last run on db74 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:26] PROBLEM - puppet last run on amssq53 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:26] PROBLEM - puppet last run on db1050 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:27] PROBLEM - puppet last run on tridge is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:27] PROBLEM - puppet last run on ms-fe1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:28] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:28] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:29] PROBLEM - puppet last run on amssq61 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:35] PROBLEM - puppet last run on mw1082 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:55] PROBLEM - puppet last run on mc1002 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:55] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 1 failures [04:10:55] PROBLEM - puppet last run on lead is CRITICAL: CRITICAL: Puppet has 1 failures [04:11:05] PROBLEM - puppet last run on elastic1008 is CRITICAL: CRITICAL: Puppet has 1 failures [04:11:15] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [04:11:26] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [04:11:35] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:05] PROBLEM - puppet last run on mw1150 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:25] PROBLEM - puppet last run on mw1217 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:25] PROBLEM - puppet last run on lvs3001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:35] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:45] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:55] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:55] PROBLEM - puppet last run on mw1008 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:55] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:55] PROBLEM - puppet last run on mw1189 is CRITICAL: CRITICAL: Puppet has 1 failures [04:13:15] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [04:13:15] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [04:13:25] PROBLEM - puppet last run on mw1117 is CRITICAL: CRITICAL: Puppet has 1 failures [04:22:54] RECOVERY - puppet last run on caesium is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:23:24] RECOVERY - puppet last run on mc1006 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [04:23:24] RECOVERY - puppet last run on wtp1006 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [04:23:34] RECOVERY - puppet last run on search1016 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [04:23:44] RECOVERY - puppet last run on francium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:23:44] RECOVERY - puppet last run on elastic1001 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:23:44] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:23:44] RECOVERY - puppet last run on wtp1020 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:23:54] RECOVERY - puppet last run on elastic1007 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:23:54] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:23:54] RECOVERY - puppet last run on mw1174 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:23:54] RECOVERY - puppet last run on ms-be1006 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [04:23:55] RECOVERY - puppet last run on mw1187 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:24:04] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [04:24:14] RECOVERY - puppet last run on elastic1004 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:24:24] RECOVERY - puppet last run on mw1026 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:24:24] RECOVERY - puppet last run on db1031 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:24:24] RECOVERY - puppet last run on db74 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:24:24] RECOVERY - puppet last run on amssq49 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [04:24:24] RECOVERY - puppet last run on amssq53 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [04:24:25] RECOVERY - puppet last run on analytics1025 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:24:25] RECOVERY - puppet last run on tridge is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:24:26] RECOVERY - puppet last run on mw1160 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [04:24:34] RECOVERY - puppet last run on potassium is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [04:24:44] RECOVERY - puppet last run on cp1039 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [04:24:44] RECOVERY - puppet last run on ms-be1003 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [04:24:54] RECOVERY - puppet last run on db1066 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [04:24:54] RECOVERY - puppet last run on mc1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:25:04] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [04:25:04] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:04] RECOVERY - puppet last run on mw1150 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:25:14] PROBLEM - puppet last run on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:14] PROBLEM - SSH on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:24] RECOVERY - puppet last run on mc1003 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [04:25:24] RECOVERY - puppet last run on mw1217 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [04:25:24] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:25:24] RECOVERY - puppet last run on db1050 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:25:24] RECOVERY - puppet last run on ms-fe1001 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:25:34] RECOVERY - puppet last run on snapshot1003 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [04:25:44] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [04:25:44] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [04:25:44] RECOVERY - puppet last run on cp1055 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [04:25:54] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [04:25:54] RECOVERY - puppet last run on lvs1002 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:25:54] RECOVERY - puppet last run on amssq32 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:25:54] RECOVERY - puppet last run on mw1008 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [04:25:55] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:25:55] RECOVERY - puppet last run on lead is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [04:25:55] RECOVERY - puppet last run on mw1189 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [04:26:04] RECOVERY - puppet last run on db1022 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:26:04] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:26:04] RECOVERY - puppet last run on elastic1008 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:26:14] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [04:26:24] RECOVERY - puppet last run on mw1117 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:26:24] RECOVERY - puppet last run on lvs3001 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [04:26:24] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:26:34] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:26:34] RECOVERY - puppet last run on amssq61 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [04:26:34] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [04:26:34] RECOVERY - puppet last run on mw1082 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [04:26:34] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [04:26:54] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [04:26:54] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [04:27:04] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 955 seconds ago with 0 failures [04:27:04] RECOVERY - SSH on tungsten is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [04:40:40] (03PS11) 10Ori.livneh: nutcracker: move config in puppet, work with new packages [operations/puppet] - 10https://gerrit.wikimedia.org/r/143597 (owner: 10Giuseppe Lavagetto) [04:40:42] (03PS2) 10Ori.livneh: nutcracker migration: allow wikidev to (re-)start nutcracker [operations/puppet] - 10https://gerrit.wikimedia.org/r/144343 [04:42:56] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: Puppet has 1 failures [04:48:06] PROBLEM - puppet last run on cp3006 is CRITICAL: CRITICAL: Puppet has 1 failures [04:49:36] PROBLEM - puppet last run on amssq54 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:16] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Puppet has 1 failures [04:58:56] PROBLEM - puppet last run on virt1000 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:06] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:06] PROBLEM - puppet last run on cp1052 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:06] PROBLEM - puppet last run on mw1024 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:26] PROBLEM - puppet last run on db1038 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:26] PROBLEM - puppet last run on mw1033 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:26] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:26] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:49] PROBLEM - puppet last run on mw1001 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:59] PROBLEM - puppet last run on search1011 is CRITICAL: CRITICAL: Puppet has 1 failures [04:59:59] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [05:00:09] PROBLEM - puppet last run on cp1040 is CRITICAL: CRITICAL: Puppet has 1 failures [05:00:09] PROBLEM - puppet last run on osm-db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:02:59] RECOVERY - puppet last run on cp3006 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [05:03:39] RECOVERY - puppet last run on amssq54 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [05:12:09] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [05:13:00] RECOVERY - puppet last run on mw1043 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [05:13:19] RECOVERY - puppet last run on mw1033 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [05:13:29] RECOVERY - puppet last run on mw1121 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:13:49] RECOVERY - puppet last run on mw1001 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [05:13:59] RECOVERY - puppet last run on virt1000 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [05:14:09] RECOVERY - puppet last run on mw1024 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [05:14:09] RECOVERY - puppet last run on cp1052 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:14:19] RECOVERY - puppet last run on db1038 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:14:19] RECOVERY - puppet last run on ms-be1011 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:14:59] RECOVERY - puppet last run on search1011 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [05:15:09] RECOVERY - puppet last run on cp1040 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:15:09] RECOVERY - puppet last run on osm-db1002 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:16:39] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [05:26:24] (03CR) 10Ori.livneh: "I did some more research and became even more convinced that we should do this in stages, so I updated the patch. I hope you don't mind. I" [operations/puppet] - 10https://gerrit.wikimedia.org/r/143597 (owner: 10Giuseppe Lavagetto) [05:30:32] PROBLEM - puppet last run on wtp1016 is CRITICAL: CRITICAL: Puppet has 1 failures [05:30:52] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:02] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:02] PROBLEM - puppet last run on mw1173 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:02] PROBLEM - puppet last run on ruthenium is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:02] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:02] PROBLEM - puppet last run on mw1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:03] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:03] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:04] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:04] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:12] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:22] PROBLEM - puppet last run on db1040 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:22] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:22] PROBLEM - puppet last run on search1001 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:32] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:32] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:32] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:32] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:42] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 1 failures [05:31:42] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:02] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:02] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:12] PROBLEM - puppet last run on ms-fe1004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:12] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:12] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:12] PROBLEM - puppet last run on db1059 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:22] PROBLEM - puppet last run on db1042 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:32] PROBLEM - puppet last run on db1034 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:32] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:32] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:42] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:42] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:32:52] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [05:33:02] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [05:33:02] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures [05:33:02] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:33:32] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:02] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:02] PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:12] PROBLEM - puppet last run on mw1213 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:32] PROBLEM - puppet last run on mw1126 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:32] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:32] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:42] PROBLEM - puppet last run on mw1175 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:42] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:42] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [05:34:52] PROBLEM - puppet last run on amssq48 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:02] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:02] PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:02] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:12] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:12] PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:12] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:12] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:32] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [05:35:32] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [05:38:29] <_joe_> shit [05:40:17] when i ssh'd into one of the machines mentioned earliers and ran apt-get update by hand, it worked [05:40:44] i did see it pull a lot of changes, maybe there were many updates released at once or something [05:40:52] <_joe_> it's just security.ubuntu.com [05:41:00] <_joe_> having hiccups [05:41:15] increasing the timeout and adding a retry should take care of it, at least in the short term [05:43:39] or even just tries => 2, try_sleep => 10 [05:44:00] <_joe_> well, let's see if that stabilizes [05:45:32] RECOVERY - puppet last run on wtp1016 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:45:42] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [05:46:02] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:46:02] RECOVERY - puppet last run on mw1173 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:46:02] RECOVERY - puppet last run on mw1003 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [05:46:02] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:46:03] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:46:12] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:46:12] RECOVERY - puppet last run on ms-fe1004 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:46:12] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [05:46:12] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:46:12] RECOVERY - puppet last run on db1059 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:46:13] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:46:13] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:46:22] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [05:46:22] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:46:22] RECOVERY - puppet last run on db1040 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:46:22] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:46:22] RECOVERY - puppet last run on search1001 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:46:23] RECOVERY - puppet last run on db1042 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [05:46:32] RECOVERY - puppet last run on db1034 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:46:32] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:46:42] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [05:46:42] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [05:46:42] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [05:46:52] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:46:52] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:46:52] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [05:46:52] RECOVERY - puppet last run on amssq48 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [05:47:02] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:47:02] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [05:47:02] RECOVERY - puppet last run on ruthenium is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [05:47:02] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [05:47:02] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [05:47:03] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:47:03] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:47:04] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [05:47:04] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:47:05] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:47:05] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [05:47:06] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [05:47:06] RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:47:07] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:47:12] RECOVERY - puppet last run on mw1213 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [05:47:32] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [05:47:32] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [05:47:42] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:47:42] RECOVERY - puppet last run on mw1175 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [05:47:52] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [05:47:52] RECOVERY - puppet last run on mw1126 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:48:02] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [05:48:02] RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:48:12] RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [05:48:12] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [05:48:12] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [05:48:32] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [05:48:42] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:28:14] PROBLEM - puppet last run on lvs1005 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:44] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:54] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:54] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:04] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:04] PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:05] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:05] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:54] PROBLEM - puppet last run on db1003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:35:14] PROBLEM - puppet last run on ssl1001 is CRITICAL: CRITICAL: Puppet has 2 failures [06:45:17] RECOVERY - puppet last run on lvs1005 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:45:37] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:45:57] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:45:57] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:46:07] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:46:07] RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:47:07] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:47:07] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:47:57] RECOVERY - puppet last run on db1003 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:54:17] RECOVERY - puppet last run on ssl1001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [07:10:08] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:11:48] PROBLEM - DPKG on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:08] PROBLEM - uWSGI web apps on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:12:38] RECOVERY - DPKG on tungsten is OK: All packages OK [07:12:58] RECOVERY - uWSGI web apps on tungsten is OK: OK: All defined uWSGI apps are runnning. [07:12:58] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [07:16:48] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [07:28:25] (03PS1) 10Ori.livneh: Add twemproxy::decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144350 [07:28:27] (03PS1) 10Ori.livneh: nutcracker: delete twemproxy module and tie up loose ends [operations/puppet] - 10https://gerrit.wikimedia.org/r/144351 [07:31:10] (03Abandoned) 10Ori.livneh: nutcracker: delete twemproxy module and tie up loose ends [operations/puppet] - 10https://gerrit.wikimedia.org/r/144351 (owner: 10Ori.livneh) [07:33:07] (03PS2) 10Ori.livneh: Add twemproxy::decom [operations/puppet] - 10https://gerrit.wikimedia.org/r/144350 [09:10:04] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:10:05] PROBLEM - SSH on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:10:14] PROBLEM - puppet last run on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:10:54] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [09:10:54] RECOVERY - SSH on tungsten is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.4 (protocol 2.0) [09:11:14] RECOVERY - puppet last run on tungsten is OK: OK: Puppet is currently enabled, last run 1235 seconds ago with 0 failures [09:16:54] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [10:40:41] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [10:41:01] PROBLEM - puppet last run on virt1008 is CRITICAL: CRITICAL: Puppet has 1 failures [10:55:01] PROBLEM - DPKG on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:55:02] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:55:11] PROBLEM - gdash.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:55:41] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [10:55:51] RECOVERY - DPKG on tungsten is OK: All packages OK [10:55:51] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [10:56:01] RECOVERY - gdash.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 9055 bytes in 0.021 second response time [10:56:01] RECOVERY - puppet last run on virt1008 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [11:17:49] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [13:18:17] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [14:26:44] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:27:34] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.006 second response time [14:30:26] (03PS1) 10Yuvipanda: diamond: Make ensure => absent work [operations/puppet] - 10https://gerrit.wikimedia.org/r/144360 [14:30:27] scfc_de: ^ should reduce load on it by a fair bit, I hope [14:31:23] scfc_de: don't think I'll find anyone to merge, though. [14:50:50] (03CR) 10Tim Landscheidt: [C: 031] diamond: Make ensure => absent work [operations/puppet] - 10https://gerrit.wikimedia.org/r/144360 (owner: 10Yuvipanda) [15:18:28] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [15:27:13] (03PS1) 10Matanya: puppet: variable is in scope, no need for scope lookup [operations/puppet] - 10https://gerrit.wikimedia.org/r/144361 [15:34:36] (03PS1) 10Matanya: bastion: lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/144362 [15:44:43] (03PS1) 10Matanya: beta: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/144363 [16:14:58] (03PS1) 10Nemo bis: Allow Internet Archive's Wayback machine to get stuff from bits etc. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/144364 (https://bugzilla.wikimedia.org/65464) [16:18:50] * hoo rages [16:18:57] so many privacy policy violations [16:20:46] (03PS2) 10Nemo bis: Allow Internet Archive's Wayback machine to get stuff from bits etc. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/144364 (https://bugzilla.wikimedia.org/65464) [16:21:24] hoo, ? [16:21:30] hoo: what tools are you using nowadays to find those [16:21:37] Nemo_bis: mwgrep mostly [16:21:46] that needs knowing the domain though [16:21:53] Krenair: Various tools/ gadgets etc. importing stuff from github etc. [16:22:05] Nemo_bis: Yeah... already found enought that way [16:22:07] sigh [16:22:07] it's horrible [16:22:09] heh [16:22:34] should just start disabling everything that violates it, assuming you're not already [16:23:05] When they're on wiki, such stuff is removed on sight [16:23:21] Nemo_bis: Sadly not true [16:23:29] eswiki gadget from 2012 here [16:23:40] just killed an arwiki gadget from 2013 [16:25:08] I guess we should evaluate introducing Content Security Policies to prevent this effectively [16:25:18] although that's a technical solution for a social problem [16:28:03] could just take the ability for sysops to edit javascript away [16:30:46] Krenair: That would only help partly [16:31:18] there are still many pure user scripts being imported ... no way people can now about the 3rd party inclusions without reading up all dependencies [16:31:27] the whole concept of users adding own JS is kind of broken [16:32:12] yeah, I'm more concerned about gadget/theme/site JS than user JS [16:32:18] Sure [16:32:26] care to handle https://es.wikipedia.org/wiki/MediaWiki:Gadget-TemplateScript.js ? [16:32:53] I don't have global editinterface [16:33:28] And I don't know what you're doing with each bad script anyway [16:34:46] Krenair: Depends... eg. on arwiki I just killed one because the user already got warned about this before [16:34:52] no mercy second time [16:35:10] over there I'll probably get in touch with Pathos' to find a solution which is ok for eveyone [16:35:55] Sadly my day only has 24h and there's a lot of sick stuff going on in our JS [16:36:44] CSS/JS on WMF wikis is such a mess :-( needs more centralization, more eyes on the code, etc. [16:36:59] +1 [16:37:18] I don't have global editinterface << I think we should fix that [16:37:23] +2 [17:07:09] (03CR) 10BryanDavis: [C: 031] beta: minor lint [operations/puppet] - 10https://gerrit.wikimedia.org/r/144363 (owner: 10Matanya) [17:13:34] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [17:19:14] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [17:41:39] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [18:58:11] PROBLEM - Graphite Carbon on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:58:11] PROBLEM - RAID on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:58:31] PROBLEM - check configured eth on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:58:31] PROBLEM - gdash.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:58:31] PROBLEM - uWSGI web apps on tungsten is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:00:02] RECOVERY - RAID on tungsten is OK: OK: optimal, 1 logical, 2 physical [19:00:02] RECOVERY - Graphite Carbon on tungsten is OK: OK: All defined Carbon jobs are runnning. [19:00:07] Krenair: yes, apply please [19:00:22] RECOVERY - check configured eth on tungsten is OK: NRPE: Unable to read output [19:00:22] RECOVERY - gdash.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 9055 bytes in 0.015 second response time [19:00:22] RECOVERY - uWSGI web apps on tungsten is OK: OK: All defined uWSGI apps are runnning. [19:20:02] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [21:17:59] PROBLEM - Puppet freshness on db1009 is CRITICAL: Last successful Puppet run was Sun 06 Jul 2014 19:17:08 UTC [21:18:09] PROBLEM - puppet last run on mw1104 is CRITICAL: CRITICAL: Puppet has 1 failures [21:20:53] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC [21:36:13] RECOVERY - puppet last run on mw1104 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [22:09:18] (03CR) 10Aklapper: "Whitespace between parameters seems to be inconsistent - sometimes it's "1,22,0,0,out_buf" and sometimes it's "1, 22, 0, 0, out_buf"" [operations/dumps] (ariel) - 10https://gerrit.wikimedia.org/r/139413 (owner: 10Wpmirrordev) [22:37:02] RECOVERY - Puppet freshness on db1009 is OK: puppet ran at Sun Jul 6 22:36:54 UTC 2014 [23:21:51] PROBLEM - Puppet freshness on rhenium is CRITICAL: Last successful Puppet run was Sat 05 Jul 2014 05:09:48 UTC